RABIES preprocessing failure

[neginnes]

Hi,
I am writing this to explain the problem I am facing with the preprocessing of StandardRat dataset with RABIES. The dataset itself consists of 21 datasets, each with several subjects with a total of 209 rats. When running the preprocessing on the complete or part of the StandardRat dataset the process reports errors and crashes. This always starts with a warning followed by some computations as detailed in the run logs. This warning and computations do not appear in the run logs of runs that were completed correctly.

Also, sometimes I noticed the same problem even when running the script on a single dataset from StandardRat (including approximately 10 rats each with single run). Even when running the script using several datasets that individually completed preprocessing smoothly, correct running seems not to be guaranteed.
Unfortunately, I wasn't able to recognize what the first warning is about and why it is generated but the later errors seem to be either related to ITK or memory. I tested running preprocessing with all default values or running on my own PC or on Compute Canada, but all concluded with similar errors. In my most recent script I tried using min_memory scaling and increasing the number of nodes for a job but the problem still occurs in many of the cases with mostly the cases of multiple datasets preprocessing (more than 10 subjects) but as well in some single dataset preprocessing. I attached a single script that I ran in three different cases. I also attached the three associated logs. The three examples consist of: 1. The script was applied to 3 datasets (total of 30 subjects) which were previously preprocessed correctly dataset by dataset but failed to run when all three datasets were preprocessed together; 2. The script was applied to a single dataset (total of 10 subjects) and failed with errors;
3. The script was applied to 2 datasets (total of 20 subjects) and completed correctly.
I would appreciate it if someone could help me understand the reason for this warnings, errors, and failure of the script to complete correctly.
std_preproc2_4_on125.txt
rabies_preprocess_3.log
rabies_preprocess_2.log
rabies_preprocess_1.log

[gdevenyi]

Please provide the details on how things are run, and how RABIES is installed.

[gdevenyi]

Can you please provide the exact commands used.

[neginnes]

The std_preproc2_4_on125.txt file I uploaded doesn't carry the information? I copy the main lines below if you mean so. Please tell me if other information is needed.

#!/bin/bash
#SBATCH -J std-prepro # Job name
#SBATCH -A def-amirs # Account name
#SBATCH -n 1 # Number of tasks
#SBATCH --cpus-per-task=16 # Number of CPUs per task
#SBATCH --time=79:59:00 # Maximum runtime
#SBATCH --mem=512G # Memory per node
#SBATCH --mail-user=negin.esmaeilazdehkiabani@mail.mcgill.ca # Email address for notifications
#SBATCH --mail-type=ALL # Send email notifications for all job events (BEGIN, END, FAIL)

module load apptainer/1.2.4

BIDS_input='./Only_125_STD'
preprocess_outputs='./std_preproc_out2_4_tested_with_memory_scaling_without_verbose_on125'
singularity_path='apptainer'
template_dir='./sigma_wistar_rat_brain_templatesandatlases_version_2.0'
template='/template/SIGMA_Rat_Anatomical_Imaging/SIGMA_Rat_Anatomical_InVivo_Template/anatomy/SIGMA_InVivo_Anatomical_Brain_template.nii.gz'
mask='/template/SIGMA_Rat_Anatomical_Imaging/SIGMA_Rat_Anatomical_InVivo_Template/anatomy/SIGMA_InVivo_Anatomical_Brain_mask.nii.gz'
wm='/template/SIGMA_Rat_Anatomical_Imaging/SIGMA_Rat_Anatomical_InVivo_Template/anatomy/SIGMA_InVivo_Anatomical_Brain_wm_bin.nii.gz'
csf='/template/SIGMA_Rat_Anatomical_Imaging/SIGMA_Rat_Anatomical_InVivo_Template/anatomy/SIGMA_InVivo_Anatomical_Brain_csf_bin.nii.gz'
atlas='/template/SIGMA_Rat_Brain_Atlases/SIGMA_Anatomical_Atlas/InVivo_Atlas/SIGMA_InVivo_Anatomical_Brain_Atlas.nii.gz'

if [ ! -d "$preprocess_outputs" ]; then
mkdir -p "$preprocess_outputs"
fi

rabies_cmd="${singularity_path} run
-B ${template_dir}:/template
-B ${BIDS_input}:/BIDS_input:ro
-B ${preprocess_outputs}:/preprocess_outputs
/home/negines/rabies.sif -p MultiProc --local_threads 16 --scale_min_memory 20 --min_proc 2 preprocess /BIDS_input /preprocess_outputs
--apply_despiking
--TR 1
--anat_template ${template}
--brain_mask ${mask}
--WM_mask ${wm}
--CSF_mask ${csf}
--vascular_mask ${csf}
--labels ${atlas}
--anat_inho_cor method=SyN,otsu_thresh=2,multiotsu=true
--bold_inho_cor method=Rigid,otsu_thresh=2,multiotsu=true
--commonspace_reg masking=false,brain_extraction=false,template_registration=SyN,fast_commonspace=false"

${rabies_cmd}

[gdevenyi]

Thanks. The "block-o-text" you provided above did not clearly break-down exactly what you provide and what is what. Please refer to filenames when providing attachments.

Are you aware RABIES preprocessed StandardRat data is already available here, https://data.ru.nl/collections/di/dcmn/DSC_4180000.19_224

How did you decide upon the parameters you're attempting to use, I can see in the StandardRat published data that the parameters are quite different.

[neginnes]

I tried editing the previous comment to be the full script. Is it now informative? The attachments include std_preproc2_4_on125.txt file which is the script and the three corresponding logs for the three case examples described in the first note.

Yes, I am using different preprocessing parameters to set one for my own project. So I am testing things first on some open rat datasets. I ran the preprocessing script also on Boretius's Lab open rat dataset but facing the same issue. The memory error usually occurs when I am not using fast common space registration on a large dataset, no matter which dataset it is. However I am not sure how I can fix it to be able to make use of generation of unbiased template and more accurate registration.

Unlike the memory error, the ITK error which is occurring as example in rabies_preprocess_2.log, seems not to be related to common space registration.

[gdevenyi]

memory error

For memory, you will need to adjust how jobs are handled/submitted using the memory scaling options. The processes are not being assigned enough resources.

For other failing images, I first suggest you look at the input images and check if they are different from other successful images. Finally, you can take a look at the StandardRat dataset to see what the appropriate settings that worked in their case, and move towards those.

[Gab-D-G]

Hi @neginnes , have you inspected the preprocess_QC_report? In the log report 1), a single scan failed the BOLD inhomogeneity correction, whereas in log report 2), one scan failed registration for the anat inhomogeneity correction. The failed scans won't show up in the report, but a prior step in preprocessing could have caused error in 1), and you may obtain clues for why 2) happened if the anatomical scans have weird FOV.
Otherwise, have you tried re-running a second time with the same parameters? Only the failed step should be run if you didn't remove the outputs, and that may provide different results.