Requesting Computing Resources¶

Because CRCD operates a shared resource for the Pitt research community, there needs to be a tool that ensures fair and equitable access. CRCD uses the Slurm workload manager to accomplish this. Slurm is a batch queueing system that allocates resources based on defined policies, and it supports two ways of working:

Interactive or batch — which do I use?

Interactive — you get a compute node and type commands live. Best for testing, exploring data, and debugging.
Batch — you submit a script and Slurm runs it unattended when resources are available. Best for production and long-running work.

Throughout the examples, <variable> marks a placeholder you replace with your own value.

What you're requesting¶

Whether interactive or batch, you're asking Slurm for the same things: a cluster, optionally a partition within it, some number of cores and nodes, an amount of memory, a wall-time limit, and optionally GPUs. With the crc-interactive command, those map to the following flags:

To set…	Flag	Default
Cluster	`-s` smp · `-g` gpu · `-m` mpi · `-d` htc · `-e` teach	—
Partition	`-p <name>`	cluster dependent
Cores per node	`-c <n>`	1
Nodes	`-n <n>`	1
Memory (GB)	`-b <GB>`	partition dependent
Wall time	`-t <hours or HH:MM:SS>`	01:00:00
GPUs (with `-g`)	`-u <n>`	0
Account / allocation	`-a <group>`	your default allocation

Not sure which cluster or partition to pick? See the Hardware Profiles section. The same concepts map to #SBATCH directives in a batch script — those are shown in the batch section below.

What is the cost for my job?

Jobs debit Service Units from your research group's Resource Allocation. If you belong to more than one Resource Allocation, you can designate which to charge with the -a <group> (interactive) or #SBATCH --account=<group> (batch) options.

Interactive work¶

Request interactive resources through the Open OnDemand web portal or from the terminal with the crc-interactive command. The simplest possible request grabs the defaults — one core on one SMP node for one hour:

commandoutput

crc-interactive -s

[gnowmik@login1 ~]$ crc-interactive -s
srun: You have specified NO/WRONG partition, so defaulting to the smp partition.
srun: job 23645428 queued and waiting for resources
srun: job 23645428 has been allocated resources
[gnowmik@smp-n214 ~]$

When the session starts, your shell prompt changes from a login node to a compute node. See how the hostname in the output changes from login1 to smp-n214 once Slurm allocates the requested resources.

To request more resources, add the corresponding flags and desired values. For example, to request 8 cores and 256 GB of RAM on the high-mem partition of the SMP cluster for 12 hours:

commandoutput

crc-interactive -s -p high-mem -c 8 -b 256 -t 12:00:00

[gnowmik@login1 ~]$ crc-interactive -s -p high-mem -c 8 -b 256 -t 12:00:00
srun: job 23645429 queued and waiting for resources
srun: job 23645429 has been allocated resources
[gnowmik@smp-1024-n1 ~]$

Learn the underlying Slurm command

Add -z to any crc-interactive command to print the equivalent srun command without running it. This is a good way to learn the raw Slurm syntax.

Click for a full listing of crc-interactive options

usage: crc-interactive [-h] [-v] [-z] [-p PARTITION] [-s] [-g] [-m] [-i] [-d] [-e] [-b MEM] [-t TIME] [-n NUM_NODES] [-c NUM_CORES] [-u NUM_GPUS] [-a ACCOUNT] [-r RESERVATION]
                       [-l LICENSE] [-f FEATURE] [-o]

Launch an interactive Slurm session.

optional arguments:
  -h, --help                                 show this help message and exit
  -v, --version                              show program's version number and exit
  -z, --print-command                        print the equivalent slurm command and exit

Cluster Arguments:
  -p PARTITION, --partition PARTITION        run the session on a specific partition
  -s, --smp                                  launch a session on the smp cluster
  -g, --gpu                                  launch a session on the gpu cluster
  -m, --mpi                                  launch a session on the mpi cluster
  -i, --invest                               launch a session on the invest cluster
  -d, --htc                                  launch a session on the htc cluster
  -e, --teach                                launch a session on the teach cluster

Arguments for Increased Resources:
  -b MEM, --mem MEM                          memory in GB
  -t TIME, --time TIME                       run time in hours or hours:minutes [default: 01:00:00]
  -n NUM_NODES, --num-nodes NUM_NODES        number of nodes [default: 1]
  -c NUM_CORES, --num-cores NUM_CORES        number of cores per node [default: 1]
  -u NUM_GPUS, --num-gpus NUM_GPUS           if using -g, the number of GPUs [default: 0]

Additional Job Settings:
  -a ACCOUNT, --account ACCOUNT              specify a non-default account
  -r RESERVATION, --reservation RESERVATION  specify a reservation name
  -l LICENSE, --license LICENSE              specify a license
  -f FEATURE, --feature FEATURE              specify a feature, e.g. `ti` for GPUs
  -o, --openmp                               run using OpenMP style submission

Batch processing¶

Batch processing requires a script, which you submit to Slurm with sbatch <job_submission_script>. A submission script has three parts: a section of Slurm directives requesting resources, a section that loads the software environment, and a section that runs the software.

The example below runs Amber molecular dynamics on a GPU. Use the tabs to see the whole script, then each part explained.

Whole scriptPart 1: Slurm directivesPart 2: Load softwarePart 3: Run software

#!/usr/bin/env bash

## ------------------------------------------------------------------
## Slurm directives defining the resource request
## ------------------------------------------------------------------
#SBATCH --job-name=gpus-1
#SBATCH --output=gpus-1.out
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cluster=gpu
#SBATCH --partition=l40s
#SBATCH --gres=gpu:1
#SBATCH --time=24:00:00

## ---------------------------------------------------------------------
## Load software into environment
## ---------------------------------------------------------------------
module purge
module load amber/24

## ---------------------------------------------------------------------
## Setup software execution environment
## ---------------------------------------------------------------------
# Define environmental variables for Amber input/output files
INP=md.in
TOP=mocvnhlysm.top
CRD=mocvnhlysm.crd
OUT=mocvnhlysm

# Define software executable
SANDER=pmemd.cuda

# Display environmental variables to Slurm output file for diagnostics
echo AMBERHOME    $AMBERHOME
echo SLURM_NTASKS $SLURM_NTASKS
echo which SANDER `which $SANDER`
echo "Running on node:" `hostname`

# Display NVIDIA GPU information to Slurm output file
nvidia-smi

# Software execution line
$SANDER  -O     -i   $INP   -p   $TOP   -c   $CRD   -r   $OUT.rst \
                -o   $OUT.out   -e   $OUT.ene   -v   $OUT.vel   -inf $OUT.nfo   -x   $OUT.mdcrd

## ------------------------------------------------------------------
## Slurm directives defining the resource request
## ------------------------------------------------------------------
#SBATCH --job-name=gpus-1
#SBATCH --output=gpus-1.out
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cluster=gpu
#SBATCH --partition=l40s
#SBATCH --gres=gpu:1
#SBATCH --time=24:00:00

A Slurm directive begins with #SBATCH followed by --<variable>=<value>, where <variable> is one of the options defined for the sbatch command. The specific <value> is unique to how CRCD configured Slurm and is documented in the Slurm Batch Jobs section.

Which lines are GPU-specific?

--cluster=gpu, --partition=l40s, and --gres=gpu:1 request a GPU. A CPU-only job drops the --gres line and targets a CPU cluster instead, e.g. --cluster=smp --partition=smp. See Slurm Batch Jobs for a generic template and Basic Slurm Commands for the difference between --nodes, --ntasks-per-node, and --cpus-per-task.

Why do some # lines run and others don't?

Most Linux shells treat a line starting with # as a comment. Slurm, however, reads every #SBATCH line as a directive to it. Any other line starting with # (such as the ## separators above) is just a comment.

## ---------------------------------------------------------------------
## Load software into environment
## ---------------------------------------------------------------------
module purge
module load amber/24

Purge everything first, then load only what you need. Loading a module updates $PATH, $LD_LIBRARY_PATH, and other variables so the software is available. See Discovering Software for how to find modules and their dependencies.

## ---------------------------------------------------------------------
## Setup software execution environment
## ---------------------------------------------------------------------
# Define environmental variables for Amber input/output files
INP=md.in
TOP=mocvnhlysm.top
CRD=mocvnhlysm.crd
OUT=mocvnhlysm

# Define software executable
SANDER=pmemd.cuda

# Display environmental variables to Slurm output file for diagnostics
echo AMBERHOME    $AMBERHOME
echo SLURM_NTASKS $SLURM_NTASKS
echo which SANDER `which $SANDER`
echo "Running on node:" `hostname`

# Display NVIDIA GPU information to Slurm output file
nvidia-smi

# Software execution line
$SANDER  -O     -i   $INP   -p   $TOP   -c   $CRD   -r   $OUT.rst \
                -o   $OUT.out   -e   $OUT.ene   -v   $OUT.vel   -inf $OUT.nfo   -x   $OUT.mdcrd

This section contains the software-specific setup and execution lines. Typically you can copy the commands you'd run on your own laptop or desktop with only minor changes, translating the software's documented commands to work within the CRCD ecosystem.

Next steps¶

You've requested resources — the next stage is submitting and monitoring your work. Continue to Managing Jobs.

Write fuller job scripts

Common #SBATCH directives, CPU and GPU templates, and email notifications.

Slurm Batch Jobs
More on interactive jobs

Using srun, choosing an account, and interactive sessions on any cluster.

Interactive Jobs
Understand the cost

How Service Units are calculated and charged against your allocation.

Service Units