Slurm Job Scripts — StudentHPC Portal

A job script is a shell script whose comment lines beginning #SBATCH are read by the scheduler before the shell ever sees them. They must appear before the first executable line — a directive placed after a command is silently ignored, which is the single most common reason a job lands in the wrong partition.

SBATCH directives #

The directives that matter for almost every job:

Directive	Meaning	Example
`--job-name`	Label shown in `squeue`	`--job-name=md-run7`
`--partition`	Which queue to run in	`--partition=compute`
`--nodes`	Whole nodes requested	`--nodes=4`
`--ntasks-per-node`	MPI ranks per node	`--ntasks-per-node=48`
`--cpus-per-task`	Threads per rank (OpenMP)	`--cpus-per-task=4`
`--mem`	Memory per node	`--mem=64G`
`--time`	Walltime limit, `D-HH:MM:SS`	`--time=2-00:00:00`
`--output`	stdout file (`%j` = job ID)	`--output=run-%j.out`
`--error`	stderr file, if kept separate	`--error=run-%j.err`
`--mail-type`	When to email you	`--mail-type=END,FAIL`

Walltime is a hard kill, not a hint

When --time expires Slurm terminates the job wherever it happens to be. Request more than you expect to need — but not wildly more, because the scheduler backfills short jobs into gaps, so an honest three-hour request often starts sooner than a padded two-day one.

MPI and multi-node jobs #

Each R640 node carries 48 physical cores, so a job asking for one full node should request 48 tasks. Launch MPI programs with srun rather than mpirun — srun inherits the allocation and binds ranks to cores without a hostfile.

gromacs-md.sh

#!/bin/bash
#SBATCH --job-name=md-production
#SBATCH --partition=compute
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=48
#SBATCH --cpus-per-task=1
#SBATCH --time=24:00:00
#SBATCH --output=md-%j.out
#SBATCH --mail-type=END,FAIL
#SBATCH [email protected]

# A batch job starts from a clean environment -- load modules here,
# not in your interactive shell.
module purge
module load gromacs/2023.3 openmpi/4.1.5

# Run from scratch: /home is not sized or tuned for job I/O.
cd /scratch/$USER/md-production

srun gmx_mpi mdrun -s topol.tpr -deffnm production

Hybrid MPI + OpenMP

For codes that thread within a rank, split the 48 cores between ranks and threads. The product of --ntasks-per-node and --cpus-per-task must not exceed 48, or ranks will contend for the same cores and the job will run slower than the single-threaded version.

hybrid.sh (excerpt)

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8      # 8 MPI ranks per node
#SBATCH --cpus-per-task=6       # x 6 threads = 48 cores. Exactly full.

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PROC_BIND=close
export OMP_PLACES=cores

srun ./my_hybrid_solver

Scale before you commit

Run the same input on 1, 2, and 4 nodes in the debug partition and compare the wall time. Most codes stop scaling well before the node count you hoped for, and discovering that in a five-minute test is cheaper than in a 24-hour one.

GPU jobs

Accelerated work goes to the gpu partition and must request GPUs explicitly with --gres. A job that omits --gres lands on a GPU node and sees no devices at all.

gpu-job.sh (excerpt)

#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:2          # 2 of the 4 GPUs on the node
#SBATCH --time=12:00:00

module load cuda apptainer

# --nv exposes the host GPU driver inside the container
apptainer exec --nv tensorflow.sif python train.py

Job arrays #

Running the same program over many inputs is a job array, not many jobs. One submission creates numbered tasks that the scheduler backfills independently, and $SLURM_ARRAY_TASK_ID tells each task which input is its own.

sweep.sh

#!/bin/bash
#SBATCH --job-name=sweep
#SBATCH --partition=compute
#SBATCH --array=1-100%10      # 100 tasks, at most 10 running at once
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=02:00:00
#SBATCH --output=sweep-%A_%a.out   # %A = array ID, %a = task index

module load python/3.11

INPUT=$(sed "${SLURM_ARRAY_TASK_ID}q;d" inputs.txt)
echo "Task $SLURM_ARRAY_TASK_ID processing $INPUT"

python simulate.py --config "$INPUT"

The %10 throttle matters. Without it, a 5,000-task array can occupy the whole partition and lock out every other researcher — on a shared facility that is the difference between using your allocation and losing it.

Monitoring and debugging #

bash

$ squeue --me                          # your queue
$ squeue -j 184203 --start            # estimated start time
$ scontrol show job 184203            # everything Slurm knows
$ scancel 184203                      # cancel one job
$ scancel --me --state=PENDING         # cancel all your queued jobs

After the job finishes

sacct is how you find out what a finished job actually used. Comparing MaxRSS against what you requested is the fastest way to stop over-asking for memory:

bash

$ sacct -j 184203 --format=JobID,JobName,State,Elapsed,MaxRSS,ReqMem
JobID        JobName      State    Elapsed  MaxRSS  ReqMem
------------ ---------- --------- -------- ------- -------
184203       md-produc+ COMPLETED 03:41:22            64Gn
184203.batch      batch COMPLETED 03:41:22  18.4G

Here the job reserved 64 GB and touched 18.4 GB. Dropping --mem to 24G would let it start sooner, because the scheduler can fit it into a smaller gap.

Common job states

State	Meaning	What to do
PD	Pending — waiting for resources	Check `REASON` in `squeue`
R	Running	Nothing
F	Failed — non-zero exit	Read the `--output` file first
TO	Timeout — hit `--time`	Raise walltime or checkpoint
OOM	Out of memory	Raise `--mem`, check for a leak

Do not run science on the login node

Login nodes are shared by everyone. A long job there degrades the cluster for every other user and will be killed without warning. If you need an interactive session, allocate one properly:

bash

$ srun --partition=debug --nodes=1 --ntasks=4 \
       --time=01:00:00 --pty bash