Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with FitHiC implementation #78

Open
asgda opened this issue Oct 5, 2023 · 10 comments
Open

Issue with FitHiC implementation #78

asgda opened this issue Oct 5, 2023 · 10 comments

Comments

@asgda
Copy link

asgda commented Oct 5, 2023

I have got the results till -analyse part. But when I am trying to implement fithic, I am getting error of creating the bias file.

I am having issues in creating a bias file. I have the matrix files, bed files which i got after preprocessing of the .hic file.

The .hic file was generated using juicer tools.

I am not getting the exact method to get the interactions file, which is an input for the bias file generation. Can you please give steps of the same?

Thanks in advance!
@ay-lab
Copy link
Owner

ay-lab commented Oct 5, 2023 •
edited
Loading

Hi,

There is a script 'createFitHiCContacts.sh' in our https://github.com/ay-lab/fithic page, which can create interactions for input.

You can check this issue from fithic page ay-lab/fithic#53 for details.

You can use HiCKRy.py script from the same to calculate the bias.

Btw, if you're running the 'fithic' step, do you see any file under the "DifferentialResult/sample/fithic_run/" directory?

@asgda
Copy link
Author

asgda commented Oct 6, 2023

Dear Sir,

Thank you for your reply, I found the interactions and fragments file in "DifferentialResult/sample/fithic_run/" directory.

After that, I executed the HiCKRy.py file using appropriate commands and got the bias files. I stored the bias files in "biases" folder in the current directory and used the extension as mentioned in the instructions, using appropriate prefix.

But, I am getting an error:

Finding significant loops from intra sample  diff trf2si un  replicates
[1] "folder exists"
Fithic file already exists for  diff_100000 , skipping
[1] "folder exists"
Fithic file already exists for  trf2si_100000 , skipping
[1] "folder exists"
Fithic file already exists for  un_100000 , skipping
/home/cgntlab1/work/softwares/miniconda3/envs/dchic/bin/python /home/cgntlab1/work/softwares/fithic/fithic/fithic.py -i DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/interactions.txt.gz -f DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fragments.txt.gz -t /home/cgntlab1/work/igib/hic_shuvra/5y/biases/diff_100000.biases.gz -U 2000000 -o DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fithic_result -r 100000 


GIVEN FIT-HI-C ARGUMENTS
=========================
Reading fragments file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fragments.txt.gz
Reading interactions file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/interactions.txt.gz
Output path being used from DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fithic_result
Fixed size option detected... Fast version of FitHiC will be used
Resolution is 100.0 kb
Reading bias file from: /home/cgntlab1/work/igib/hic_shuvra/5y/biases/diff_100000.biases.gz
The number of spline passes is 1
The number of bins is 100
The number of reads required to consider an interaction is 1
The name of the library for outputted files will be FitHiC
Upper Distance threshold is 2000000
Lower Distance threshold is 0
Only intra-chromosomal regions will be analyzed
Lower bound of bias values is 0.5
Upper bound of bias values is 2
All arguments processed. Running FitHiC now...
=========================


Reading the contact counts file to generate bins...
Traceback (most recent call last):
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 1324, in <module>
    main()
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 317, in main
    (mainDic,observedInterAllCount,observedInterAllSum,observedIntraAllSum,observedIntraInRangeSum) = read_Interactions(contactCountsFile, biasFile)
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 429, in read_Interactions
    minObservedGenomicDist=min(minObservedGenomicDist,interxn.getDistance())
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 2953, in min
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/_methods.py", line 45, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
numpy.exceptions.AxisError: axis 100000 is out of bounds for array of dimension 0
/home/cgntlab1/work/softwares/miniconda3/envs/dchic/bin/python /home/cgntlab1/work/softwares/fithic/fithic/fithic.py -i DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/trf2si_100000_fithic/interactions.txt.gz -f DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/trf2si_100000_fithic/fragments.txt.gz -t /home/cgntlab1/work/igib/hic_shuvra/5y/biases/trf2si_100000.biases.gz -U 2000000 -o DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/trf2si_100000_fithic/fithic_result -r 100000 


GIVEN FIT-HI-C ARGUMENTS
=========================
Reading fragments file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/trf2si_100000_fithic/fragments.txt.gz
Reading interactions file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/trf2si_100000_fithic/interactions.txt.gz
Output path being used from DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/trf2si_100000_fithic/fithic_result
Fixed size option detected... Fast version of FitHiC will be used
Resolution is 100.0 kb
Reading bias file from: /home/cgntlab1/work/igib/hic_shuvra/5y/biases/trf2si_100000.biases.gz
The number of spline passes is 1
The number of bins is 100
The number of reads required to consider an interaction is 1
The name of the library for outputted files will be FitHiC
Upper Distance threshold is 2000000
Lower Distance threshold is 0
Only intra-chromosomal regions will be analyzed
Lower bound of bias values is 0.5
Upper bound of bias values is 2
All arguments processed. Running FitHiC now...
=========================


Reading the contact counts file to generate bins...
Traceback (most recent call last):
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 1324, in <module>
    main()
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 317, in main
    (mainDic,observedInterAllCount,observedInterAllSum,observedIntraAllSum,observedIntraInRangeSum) = read_Interactions(contactCountsFile, biasFile)
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 429, in read_Interactions
    minObservedGenomicDist=min(minObservedGenomicDist,interxn.getDistance())
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 2953, in min
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/_methods.py", line 45, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
numpy.exceptions.AxisError: axis 300000 is out of bounds for array of dimension 0
/home/cgntlab1/work/softwares/miniconda3/envs/dchic/bin/python /home/cgntlab1/work/softwares/fithic/fithic/fithic.py -i DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/un_100000_fithic/interactions.txt.gz -f DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/un_100000_fithic/fragments.txt.gz -t /home/cgntlab1/work/igib/hic_shuvra/5y/biases/un_100000.biases.gz -U 2000000 -o DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/un_100000_fithic/fithic_result -r 100000 


GIVEN FIT-HI-C ARGUMENTS
=========================
Reading fragments file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/un_100000_fithic/fragments.txt.gz
Reading interactions file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/un_100000_fithic/interactions.txt.gz
Output path being used from DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/un_100000_fithic/fithic_result
Fixed size option detected... Fast version of FitHiC will be used
Resolution is 100.0 kb
Reading bias file from: /home/cgntlab1/work/igib/hic_shuvra/5y/biases/un_100000.biases.gz
The number of spline passes is 1
The number of bins is 100
The number of reads required to consider an interaction is 1
The name of the library for outputted files will be FitHiC
Upper Distance threshold is 2000000
Lower Distance threshold is 0
Only intra-chromosomal regions will be analyzed
Lower bound of bias values is 0.5
Upper bound of bias values is 2
All arguments processed. Running FitHiC now...
=========================


Reading the contact counts file to generate bins...
Traceback (most recent call last):
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 1324, in <module>
    main()
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 317, in main
    (mainDic,observedInterAllCount,observedInterAllSum,observedIntraAllSum,observedIntraInRangeSum) = read_Interactions(contactCountsFile, biasFile)
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 429, in read_Interactions
    minObservedGenomicDist=min(minObservedGenomicDist,interxn.getDistance())
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 2953, in min
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/_methods.py", line 45, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
numpy.exceptions.AxisError: axis 300000 is out of bounds for array of dimension 0
[1] 1
Taking input= as a system command ('gzip -dc DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fithic_result/FitHiC.spline_pass1.res100000.significances.txt.gz') and a variable has been used in the expression passed to `input=`. Please use fread(cmd=...). There is a security concern if you are creating an app, and the app could have a malicious user, and the app is not running in a secure environment; e.g. the app is running as root. Please read item 5 in the NEWS file for v1.11.6 for more information and for the option to suppress this message.
gzip: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fithic_result/FitHiC.spline_pass1.res100000.significances.txt.gz: No such file or directory
Error in setnames(x, value) : 
  Can't assign 7 names to a 0 column data.table
Calls: fithicformat ... colnames<- -> names<- -> names<-.data.table -> setnames
In addition: Warning message:
In data.table::fread(paste0("gzip -dc ", diffdir, "/fithic_run/",  :
  File '/tmp/RtmplDNGy9/file175b453260a552' has size 0. Returning a NULL data.table.
Execution halted

Can you let me know how to deal with this error? Thank you in advance!

@ay-lab
Copy link
Owner

ay-lab commented Oct 6, 2023

I think this is an error arising from the Fithic code.
Just to make sure this is what it is, can you run this piece of line separately and see Fithic is creating a result file?

/home/cgntlab1/work/softwares/miniconda3/envs/dchic/bin/python /home/cgntlab1/work/softwares/fithic/fithic/fithic.py -i DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/interactions.txt.gz -f DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fragments.txt.gz -t /home/cgntlab1/work/igib/hic_shuvra/5y/biases/diff_100000.biases.gz -U 2000000 -o DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fithic_result -r 100000

@asgda
Copy link
Author

asgda commented Oct 8, 2023 •
edited
Loading

Sir, after running the above command, I got the following error:

GIVEN FIT-HI-C ARGUMENTS
=========================
Reading fragments file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fragments.txt.gz
Reading interactions file from: DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/interactions.txt.gz
Output path being used from DifferentialResult/un_vs_diff_vs_trf2si_100Kb/fithic_run/diff_100000_fithic/fithic_result
Fixed size option detected... Fast version of FitHiC will be used
Resolution is 100.0 kb
Reading bias file from: /home/cgntlab1/work/igib/hic_shuvra/5y/biases/diff_100000.biases.gz
The number of spline passes is 1
The number of bins is 100
The number of reads required to consider an interaction is 1
The name of the library for outputted files will be FitHiC
Upper Distance threshold is 2000000
Lower Distance threshold is 0
Only intra-chromosomal regions will be analyzed
Lower bound of bias values is 0.5
Upper bound of bias values is 2
All arguments processed. Running FitHiC now...
=========================


Reading the contact counts file to generate bins...
Traceback (most recent call last):
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 1324, in <module>
    main()
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 317, in main
    (mainDic,observedInterAllCount,observedInterAllSum,observedIntraAllSum,observedIntraInRangeSum) = read_Interactions(contactCountsFile, biasFile)
  File "/home/cgntlab1/work/softwares/fithic/fithic/fithic.py", line 429, in read_Interactions
    minObservedGenomicDist=min(minObservedGenomicDist,interxn.getDistance())
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 2953, in min
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return reduction(axis=axis, out=out, **passkwargs)
  File "/home/cgntlab1/work/softwares/miniconda3/envs/dchic/lib/python3.10/site-packages/numpy/core/_methods.py", line 45, in _amin
    return umr_minimum(a, axis, None, out, keepdims, initial, where)
numpy.exceptions.AxisError: axis 100000 is out of bounds for array of dimension 0

Also to add on, I also ran the test for FitHiC in dchic as mentioned in the readme and it showed that FitHiC is correctly installed.

How should I proceed next?
Thank you in advance!

@ay-lab
Copy link
Owner

ay-lab commented Oct 9, 2023

Can you check the input file formats and make sure it is right? You can check this to cross-check the interactions.txt.gz and fragments.txt.gz files look as described here https://github.com/ay-lab/fithic

Just to make sure, are you using createFitHiCContacts-hic.sh script to generate the inputs?
If it's a format issue, then I can check the way dchic is generating the input files, otherwise, it is a fithic issue.
I would also suggest generating an issue in fithic repository.

@asgda
Copy link
Author

asgda commented Oct 12, 2023

Dear sir,

I checked the input file formats:
interactions.txt:
chr1 50000 chr1 50000 20
chr1 50000 chr1 2350000 1
chr1 150000 chr1 150000 10
chr1 250000 chr1 250000 2
chr1 550000 chr1 550000 5
chr1 550000 chr1 850000 2
chr1 550000 chr1 950000 4
chr1 550000 chr1 2250000 1
chr1 550000 chr1 2350000 1
chr1 550000 chr1 3350000 1
chr1 650000 chr1 650000 24
chr1 650000 chr1 750000 1
chr1 650000 chr1 850000 3
chr1 650000 chr1 950000 4
chr1 650000 chr1 1050000 6
chr1 650000 chr1 1150000 2
chr1 650000 chr1 1350000 1
chr1 650000 chr1 1850000 3
chr1 650000 chr1 2050000 1.........
fragments.txt
chr1 0 50000 21 1
chr1 0 150000 10 1
chr1 0 250000 2 1
chr1 0 350000 0 1
chr1 0 450000 0 1
chr1 0 550000 14 1
chr1 0 650000 52 1
chr1 0 750000 291 1
chr1 0 850000 3357 1
chr1 0 950000 7015 1
chr1 0 1050000 7384 1
chr1 0 1150000 8098 1
chr1 0 1250000 7670 1
chr1 0 1350000 8056 1
chr1 0 1450000 7644 1.....

When I cross checked the formats I found that there is no "chr" in the reference manual, but here I am getting "chr" written.

Also, these files were already present after the analyze step and I didn't have to run createFitHiCContacts-hic.sh script.

Kindly let me know about it. Thank you in advance!

@ay-lab
Copy link
Owner

ay-lab commented Oct 12, 2023

Can you share one of the interactions, fragments and biases file with me? abhijit@lji.org
I am still not sure what is causing the issue. Need to dig deeper.

@asgda
Copy link
Author

asgda commented Oct 15, 2023

Respected sir, I have shared the files on email.

Thank you for your support!

@ay-lab
Copy link
Owner

ay-lab commented Oct 20, 2023

Thanks for sending the files.
I managed to successfully run the fithic v 2.0.7 on those, wasn't able to reproduce the error.
Are you using the same fithic version?

@asgda
Copy link
Author

asgda commented Oct 21, 2023

Dear sir,

Thank you so much for your reply. My FitHiC version is also 2.0.7, but I am facing the same error still. Now I will try with reinstalling FitHiC and check if it would make any difference and would let you know about the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ay-lab @asgda