Athena GPU Cluster User Guide

In Athena (and SLURM in general), a partition is a logical group of compute nodes that share the same
hardware type, access policy, and scheduling rules. You can think of a partition as a queue with specific machines behind it.

tsinfo
tsinfo <partition_name>

tsinfo
tsinfo <partition_name>

CPU notation

SLURM reports CPU state as:

• A – Active
• I – Idle
• T – Total

CPUS(A/I/T): 36 / 220 / 256

This means 36 CPUs are currently in use and 220 are idle out of 256 total.

GPU notation

• gpu:nvidia_a100:2 (IDX:0,2) → GPUs 0 and 2 are allocated
• :0 → no GPUs in use

tsqr running
tsqr pending
tsqr NODELIST
tsqr -u <username>

tsqos
tsqos <qos_name>

Request CPU and memory proportional to GPU usage.

• Node has 8 GPUs → requesting 1 GPU ≈ request ~1/8 of the node’s CPUs and RAM.
• Adjust based on observed GPU utilization and your workload’s CPU/RAM needs.

export APPTAINER_BIND=$PRJ_WORKSPACE,$HOME

srun   --partition=h200-shared   --qos=2h_2g   --time=1:30:00   --gres=gpu:1   --ntasks=1   --cpus-per-task=16   --mem=40G   --pty   --container="/apps/apptainer/images/pytorch/2.7/torch2.7-tf2.20-cu128.sif"   /bin/bash

#!/bin/bash
#
# SBATCH Submission Script: Example GPU Workload
# Run with: sbatch example.sbatch
#

# --- Job Configuration ---
#SBATCH --job-name=gpu_workload
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=1

# --- Resource Allocation ---
#SBATCH --partition=h200-dds
#SBATCH --qos=contrib
#SBATCH --gres=gpu:nvidia_h200:1
#SBATCH --cpus-per-gpu=12
#SBATCH --time=12:00:00

# --- Container ---
#SBATCH --container=/apps/apptainer/images/pytorch/2.7/torch2.7-tf2.20-cu128.sif

set -e

echo "================================================================"
echo "Timestamp            : $(date)"
echo "Host                 : $(hostname)"
echo "SLURM Job ID         : $SLURM_JOB_ID"
echo "Apptainer Container  : $APPTAINER_CONTAINER"
echo "APPTAINER_BIND       : $APPTAINER_BIND"
echo "GPU(s) Allocated     : $CUDA_VISIBLE_DEVICES"
echo "================================================================"

nvidia-smi

# Your command here:
# cd /home/<user>/work/YourProject/
# python run.py

To run on a public partition, choose a *-public partition and an allowed QoS (e.g., 12h_4g), and verify availability with tsinfo.

# Show my jobs (formatted)
tsqr -u $USER

# Job details
scontrol show job <job_id>

# Cancel a job
scancel <job_id>

New to Persistent Overlays?
A Persistent Overlay is a writable layer that sits on top of a read-only .sif container image,
letting you install packages and save changes across runs – without modifying the shared base image.

Read more about Apptainer Persistent Overlays →

Best practice: use $TMPDIR (local NVMe storage; faster than network storage).

cd $TMPDIR
cp /apps/apptainer/images/pytorch/2.7/torch2.7-tf2.20-cu128.sif .

# Create overlay (size in MB)
apptainer overlay create --fakeroot --sparse --size 102400 torch2.7-tf2.20-cu128.sif

# Inspect partition layers
apptainer sif list torch2.7-tf2.20-cu128.sif

# Delete overlay if needed (example: delete partition index 5)
apptainer sif del 5 torch2.7-tf2.20-cu128.sif

# Enter writable container
apptainer exec   --writable   --fakeroot   --userns   --nvccli   torch2.7-tf2.20-cu128.sif   /bin/bash

/home/<USER>/work = $PRJ_WORKSPACE = /rg/<group_prj>/<USER>
This directory resides inside your group’s shared project space (group_prj)
and is accessible to all members of the same group. Be mindful of what you store there.

Additional shared storage quota can be purchased by your group’s Research PI through the official
Technion CIS Store.

# Copy a file to Athena
scp myfile.txt <USER>@athena.technion.ac.il:~/

# Sync a folder to Athena (recommended for large folders)
rsync -avP /local/data/ <USER>@athena.technion.ac.il:/path/on/athena/

• $TMPDIR is local NVMe (fast).
• Much faster than network-mounted storage.
• Ideal for temporary data, container modification, and intermediate outputs.

Verify your --partition and --qos are correct and compatible. Request fewer resources where possible. Set a shorter and more realistic --time. Check cluster availability with tsinfo. Use tsqr to inspect the queue — the output includes the reason your job is pending (e.g. Resources, Priority, QOSMaxJobsPerUserLimit).

QoS availability is determined by your account’s association tree. Run the following to see which QoS tiers are associated with your account:

tsassoc | grep $USER

Yes, subject to your QoS limits. MAXJOBSPU in tsqos output shows the maximum concurrent jobs per QoS. If you have reached the limit under one QoS, submitting under a different QoS has its own independent counter — allowing you to effectively run more jobs in parallel.

Run a short test first with a small subset of your data. Use that runtime to extrapolate. Always add a small buffer — but avoid over-estimating, as shorter --time values receive higher scheduling priority.

SLURM kills it immediately with no grace period. Always request enough time, and use checkpointing for long workloads so progress is not lost.

Enable checkpointing in your training code and save progress periodically. Shared workloads receive a 10-minute grace period before being re-queued — use it to save a final checkpoint on signal.

SLURM sends a SIGTERM signal to your job at the start of the grace period. Trap this signal in your script to trigger an immediate checkpoint save, giving you the full 10 minutes to write it cleanly:

trap 'echo "Preemption signal received — saving checkpoint"; python save_checkpoint.py; exit 0' SIGTERM

It will be re-queued automatically, but it will restart from the beginning unless your code resumes from a checkpoint. Implementing checkpoint/resume logic is strongly recommended for any workload running on shared partitions.

A good starting rule: if a node has 8 GPUs, requesting 1 GPU entitles you to roughly 1/8 of its CPUs and RAM. Run tsinfo <partition_name> to see total node resources, then divide proportionally by the number of GPUs you request. Adjust based on your workload’s observed utilization.

Run scontrol show job <job_id> for the full resource allocation of a running job, including actual CPU, memory, and GPU assignments. For your QoS-level limits (max GPUs, max runtime, max concurrent jobs), run tsqos.

Run nvidia-smi inside the container. You can also verify PyTorch sees the GPU:

python -c "import torch; print(torch.cuda.get_device_name(0))"

Docker is not available on Athena. Two alternatives exist: Apptainer, which can pull and convert Docker images directly and is the recommended approach:

apptainer pull docker://<image>

And Podman, which is available but functionally limited — its image storage defaults to $TMPDIR, a temporary per-job directory that is wiped when the job ends.

Use a persistent overlay — it adds a writable layer on top of the read-only container image, preserving installed packages across runs. See the overlay instructions in the Containers section above.

Check your APPTAINER_BIND variable. Your $HOME and $PRJ_WORKSPACE directories must be explicitly bound before the container can see them. Set the following before running srun or include it in your sbatch script:

export APPTAINER_BIND=$PRJ_WORKSPACE,$HOME

$TMPDIR is a per-job local NVMe directory on the compute node — significantly faster than network-mounted storage for intensive I/O. Use it for temporary files, container overlays, and intermediate outputs. Its contents are deleted when the job ends. Note that Athena’s current shared storage is NFS-based, which handles general file access well but is not optimized for the parallel I/O demands of HPC workloads — local NVMe is the practical workaround for I/O-heavy jobs for the time being.

By default, SLURM writes output to the directory from which you submitted the job, named slurm-<job_id>.out and slurm-<job_id>.err. Use #SBATCH --output and #SBATCH --error in your script to set explicit paths.

Yes — Odin is actually the preferred place for interactive long-running sessions. Unlike direct srun sessions on the command line, Odin manages the session lifecycle for you, making it more suitable for extended interactive work. For fully unattended workloads, sbatch remains the right choice.