Submitting recon-all as a batch job

Single Subject

To recon a single subject on the SCC, we recommend creating a submission file. You can copy the example submission file below to your home directory by running the following command:

[scc4]$ cp /share/pkg.7/freesurfer/7.3.2/install/subjects/tutorial_data/recon_SS.qsub $HOME

This example qsub file will run recon-all on a single subject, with freesurfer version 7.3.2, while using FreeSurfer's parallel pipeline on 4 cores:

#!/bin/bash -l

# Set SCC project
#$ -P my_project

# Request 4 cores (2 per hemisphere) for open-mp called by -parellel flag.
#$ -pe omp 4

# Defaults to fair-share mem_per_core, which is at minimum 4GB. Uncomment to make active.
##$ -l mem_per_core=4GB

# Send an email when the job finishes or if it is aborted (by default no email is sent). Uncomment to make active.
##$ -m ae

# Give job a name
#$ -N recon-all

# Combine output and error files into a single file
#$ -j y

# Load desired version of freesurfer
module load freesurfer/7.3.2

# Set environment variables
export SUBJECTS_DIR=/projectnb/my_project/freesurfer

# Set recon-all variables
recon-all -all -subjid sub-001 -i /projectnb/test/sub-001/MPRAGE.nii.gz -parallel -openmp $NSLOTS

To submit the recon-all qsub file, use the qsub command:

[scc4]$ qsub recon-all.qsub

Multiple Subjects

To submit multiple subjects to the SCC for recon-all, we can use a custom bash script that loops through a list of subjects and submits each in a modified qsub file. This method will generate a unique jobID for each subject. All subjects will be submitted to the SCC queue instantly instead of waiting for one subject to finish before starting the next.

Again, you may copy the example list file, multi-subject qsub file, and bash script to your home directory with the following commands:

[scc4]$ cp /share/pkg.7/freesurfer/7.3.2/install/subjects/tutorial_data/recon_MS.qsub $HOME
[scc4]$ cp /share/pkg.7/freesurfer/7.3.2/install/subjects/tutorial_data/subj_list.txt $HOME
[scc4]$ cp /share/pkg.7/freesurfer/7.3.2/install/subjects/tutorial_data/recon_bash.sh $HOME

Your subject list file should include both the subject ID in column 1 and the path to the input T1w in column 2. An example of this subject list should look similar to the one below.

sub05676 /projectnb/test/freesurfer/sub05676/anat/mprage_anonymized.nii.gz
sub08224 /projectnb/test/freesurfer/sub08224/anat/mprage_anonymized.nii.gz
sub08889 /projectnb/test/freesurfer/sub08889/anat/mprage_anonymized.nii.gz
sub09607 /projectnb/test/freesurfer/sub09607/anat/mprage_anonymized.nii.gz

The multi-subject recon-all qsub file will use variable placeholders $1 for the -subjid and $2 for the -i. These placeholders will allow for the bash script to automatically fill the entries line by line from your subject list file. To do this, simply revise the recon-all command in your qsub file to the following:

recon-all -all -subjid $1 -i $2 -parallel -openmp $NSLOTS

The bash script below uses a for-loop to pull the subject ID and /path/to/input/T1w for each subject and submits them to the scheduler. Take caution to explicitly edit the subj_list variable in the below script to your subjects list.

# Set your subject list file
subj_list=/projectnb/test/freesurfer/subj_list.txt

# Loop and submit to the batch system using the modified recon_MS.qsub file.
count=$(cat $subj_list | wc -l)
for (( i=1; i<=$count; i++ ));
do
    subjid=$(cat $subj_list | head -$i | tail -1 | awk '{print $1}')
    input=$(cat $subj_list | head -$i | tail -1 | awk '{print $2}')
    qsub recon-all_multi.qsub $subjid $input
done

To actually run this bash script (i.e. named recon-all_bash.sh) from the command line:

[scc4]$ ./recon_bash.sh

Running recon-all on the SCC

The more specific your request of resources, the shorter the list of possible nodes that can accept your batch job. Therefore, if you are in a rush or have 100s of subjects to recon, you are best off requesting resources that can successfully perform recon-all and can be fulfilled by the most nodes (i.e. reducing queue time). Freesurfer recommends a minimum of 1-core and 4GB of memory for an absolute minimum of success.

We've found that the trade off between queue time and processing time peaks at 8-cores. Requesting more than 8-cores will not yield significant speed improvements but will significantly increase the time before your job lands on a node.

Depending on how many subjects you need to process and your time constraintes confer with the table below for choosing the appropriate resources:

# of Subjects	# of Cores (Recommended)
<10	8
<100	4
>100	1