Running MRIQC on the SCC

MRIQC is too heavy to run on a login node — always submit it to the queue.

Run it in two stages. The participant level computes the quality metrics and builds one report per scan; it is the expensive stage, and it is submitted as a job array so that every participant runs at the same time. The group level runs once afterwards and pools those metrics into summary reports.

The SCC uses the SGE scheduler, but many other clusters — and much of the field — use SLURM. Every scheduler-specific block below has a toggle, so you can read the same job as either. The MRIQC command itself never changes; only the directives and the submit commands around it do. Your choice is remembered as you move between blocks and pages.

For what the MRIQC options themselves mean, see Running MRIQC. For how many cores and how much memory to ask for, see the Appendix.

Participant Level

A job array submits one task per participant from a single script. The scheduler runs the tasks concurrently, so a study takes about as long as its slowest participant rather than the sum of them all.

1. Copy the script and the participant list

[scc4]$ cp /project/scv/examples/imaging/mriqc/mriqc.qsub .
[scc4]$ cp /project/scv/examples/imaging/mriqc/participant_list.txt .

[scc4]$ cp /project/scv/examples/imaging/mriqc/mriqc.sbatch .
[scc4]$ cp /project/scv/examples/imaging/mriqc/participant_list.txt .

participant_list.txt holds one participant per line and must sit in the same directory as the script:

sub-01
sub-02
sub-03

Build it from your own dataset:

[scc4]$ ls -d /projectnb/your_project/bids/sub-* | xargs -n1 basename > participant_list.txt

2. Set the number of tasks

The #$ -t 1-3#SBATCH --array=1-3 line must match the number of participants in the list. Count them:

[scc4]$ wc -l participant_list.txt

If the range is too short, participants at the end are silently skipped; if it is too long, the extra tasks fail with an empty participant label.

3. Fill in the script

#!/bin/bash -l

# Set your SCC project
#$ -P your_project

# Specify hard time limit for the job. This applies to EACH task in the array.
#   The task will be aborted if it runs longer than this time.
#   The default time is 12 hours
#$ -l h_rt=12:00:00

# Send an email when the job finishes or if it is aborted (by default no email is sent).
#$ -m ea

# Give job a name
#$ -N mriqc

# Combine output and error files into a single file
#$ -j y

# Write the job logs here. This directory must already exist -- the scheduler
# will not create it, and the job fails immediately if it is missing.
#$ -o /projectnb/your_project/logs/

#$ -pe omp 4
#$ -l mem_per_core=4G

# Run one task per participant. Change 3 to the number of lines in participant_list.txt
#$ -t 1-3

# Keep track of information related to the current job
echo "=========================================================="
echo "Start date : $(date)"
echo "Job name : $JOB_NAME"
echo "Job ID : $JOB_ID.$SGE_TASK_ID"
echo "=========================================================="

# load the mriqc module
module load mriqc/24.0.0

# specify where on the SCC your BIDS formatted data lives
BIDS_DIR='/projectnb/your_project/bids'

# specify where on the SCC you want the output reports to be written
OUTPUT_DIR='/projectnb/your_project/bids/derivatives/mriqc'

# specify where on the SCC you want temporary files to be written. Defaulting here to your job's temporary directory
WORK_DIR=$TMPDIR

# pick out this task's participant. participant_list.txt must sit in the same
# directory as this script, with one participant per line
readarray -t PARTICIPANT_LIST < "participant_list.txt"
PARTICIPANT=${PARTICIPANT_LIST[$(($SGE_TASK_ID-1))]}

# request computational resources
NPROCS=$NSLOTS
NTHREADS=$NSLOTS
MEM=16

# run MRIQC on this task's participant
mriqc $BIDS_DIR $OUTPUT_DIR participant --participant-label $PARTICIPANT -w $WORK_DIR --no-sub --nprocs $NPROCS --omp-nthreads $NTHREADS --mem $MEM

#!/bin/bash -l

# Set the account the job is charged to (SGE calls this the project)
#SBATCH --account=your_project

# Specify hard time limit for the job. This applies to EACH task in the array.
#   The task will be aborted if it runs longer than this time.
#SBATCH --time=12:00:00

# Send an email when the job finishes or if it fails. Add --mail-user=<address>
# if your site does not default to the submitting user.
#SBATCH --mail-type=END,FAIL

# Give job a name
#SBATCH --job-name=mriqc

# Write the job logs here. This directory must already exist -- the scheduler
# will not create it, and the job fails immediately if it is missing.
# %A is the array job ID and %a the task ID. Naming no --error file means
# errors are written into this same file.
#SBATCH --output=/projectnb/your_project/logs/mriqc-%A_%a.out

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4G

# Run one task per participant. Change 3 to the number of lines in participant_list.txt
#SBATCH --array=1-3

# Keep track of information related to the current job
echo "=========================================================="
echo "Start date : $(date)"
echo "Job name : $SLURM_JOB_NAME"
echo "Job ID : $SLURM_ARRAY_JOB_ID.$SLURM_ARRAY_TASK_ID"
echo "=========================================================="

# load the mriqc module
module load mriqc/24.0.0

# specify where your BIDS formatted data lives
BIDS_DIR='/projectnb/your_project/bids'

# specify where you want the output reports to be written
OUTPUT_DIR='/projectnb/your_project/bids/derivatives/mriqc'

# specify where you want temporary files to be written. Not every SLURM site
# defines TMPDIR on compute nodes -- check your local documentation
WORK_DIR=${TMPDIR:-/tmp}

# pick out this task's participant. participant_list.txt must sit in the same
# directory as this script, with one participant per line
readarray -t PARTICIPANT_LIST < "participant_list.txt"
PARTICIPANT=${PARTICIPANT_LIST[$(($SLURM_ARRAY_TASK_ID-1))]}

# request computational resources
NPROCS=$SLURM_CPUS_PER_TASK
NTHREADS=$SLURM_CPUS_PER_TASK
MEM=16

# run MRIQC on this task's participant
mriqc $BIDS_DIR $OUTPUT_DIR participant --participant-label $PARTICIPANT -w $WORK_DIR --no-sub --nprocs $NPROCS --omp-nthreads $NTHREADS --mem $MEM

4. Create the log directory and submit

The scheduler will not create the log directory for you, and the job fails straight away if it is missing:

[scc4]$ mkdir -p /projectnb/your_project/logs
[scc4]$ qsub mriqc.qsub

[scc4]$ mkdir -p /projectnb/your_project/logs
[scc4]$ sbatch mriqc.sbatch

Each task writes its own log there, named mriqc.o<job ID>.<task ID>mriqc-<job ID>_<task ID>.out, so a failed task is easy to isolate. Watch the array with:

[scc4]$ qstat -u <your BU login name>

[scc4]$ squeue -u <your login name>

Group Level

Once every participant task has finished, run the group level once. It reads the metrics already in your output directory and writes one summary report and one .tsv table per modality, which is what lets you spot a scan that is an outlier relative to the rest of your sample.

1. Copy the script

[scc4]$ cp /project/scv/examples/imaging/mriqc/mriqc_group.qsub .

[scc4]$ cp /project/scv/examples/imaging/mriqc/mriqc_group.sbatch .

2. Fill in the script

Point OUTPUT_DIR at the same directory the participant jobs wrote to.

#!/bin/bash -l

# Set your SCC project
#$ -P your_project

# Specify hard time limit for the job.
#   The job will be aborted if it runs longer than this time.
#$ -l h_rt=2:00:00

# Send an email when the job finishes or if it is aborted (by default no email is sent).
#$ -m ea

# Give job a name
#$ -N mriqc_group

# Combine output and error files into a single file
#$ -j y

# Write the job logs here. This directory must already exist -- the scheduler
# will not create it, and the job fails immediately if it is missing.
#$ -o /projectnb/your_project/logs/

#$ -pe omp 4
#$ -l mem_per_core=4G

# Keep track of information related to the current job
echo "=========================================================="
echo "Start date : $(date)"
echo "Job name : $JOB_NAME"
echo "Job ID : $JOB_ID"
echo "=========================================================="

# load the mriqc module
module load mriqc/24.0.0

# specify where on the SCC your BIDS formatted data lives
BIDS_DIR='/projectnb/your_project/bids'

# the SAME output directory the participant level jobs wrote to
OUTPUT_DIR='/projectnb/your_project/bids/derivatives/mriqc'

# specify where on the SCC you want temporary files to be written. Defaulting here to your job's temporary directory
WORK_DIR=$TMPDIR

# request computational resources
NPROCS=$NSLOTS
NTHREADS=$NSLOTS
MEM=16

# pool every participant's metrics into the group reports
mriqc $BIDS_DIR $OUTPUT_DIR group -w $WORK_DIR --no-sub --nprocs $NPROCS --omp-nthreads $NTHREADS --mem $MEM

#!/bin/bash -l

# Set the account the job is charged to (SGE calls this the project)
#SBATCH --account=your_project

# Specify hard time limit for the job.
#   The job will be aborted if it runs longer than this time.
#SBATCH --time=2:00:00

# Send an email when the job finishes or if it fails. Add --mail-user=<address>
# if your site does not default to the submitting user.
#SBATCH --mail-type=END,FAIL

# Give job a name
#SBATCH --job-name=mriqc_group

# Write the job logs here. This directory must already exist -- the scheduler
# will not create it, and the job fails immediately if it is missing.
# %j is the job ID. Naming no --error file means errors are written into this
# same file.
#SBATCH --output=/projectnb/your_project/logs/mriqc_group-%j.out

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4G

# Keep track of information related to the current job
echo "=========================================================="
echo "Start date : $(date)"
echo "Job name : $SLURM_JOB_NAME"
echo "Job ID : $SLURM_JOB_ID"
echo "=========================================================="

# load the mriqc module
module load mriqc/24.0.0

# specify where your BIDS formatted data lives
BIDS_DIR='/projectnb/your_project/bids'

# the SAME output directory the participant level jobs wrote to
OUTPUT_DIR='/projectnb/your_project/bids/derivatives/mriqc'

# specify where you want temporary files to be written. Not every SLURM site
# defines TMPDIR on compute nodes -- check your local documentation
WORK_DIR=${TMPDIR:-/tmp}

# request computational resources
NPROCS=$SLURM_CPUS_PER_TASK
NTHREADS=$SLURM_CPUS_PER_TASK
MEM=16

# pool every participant's metrics into the group reports
mriqc $BIDS_DIR $OUTPUT_DIR group -w $WORK_DIR --no-sub --nprocs $NPROCS --omp-nthreads $NTHREADS --mem $MEM

3. Submit it

[scc4]$ qsub mriqc_group.qsub

[scc4]$ sbatch mriqc_group.sbatch

The log lands in the same log directory, named mriqc_group.o<job ID>mriqc_group-<job ID>.out.

When it finishes, read the reports — see Checking Your Output.