Athena GPU Cluster User Guide

After successful registration to HPC services and receiving A confirmation email reply, Connect to Athena Login Node using SSH from a Technion network or VPN:

ssh username@dgx-master.technion.ac.il

Login nodes are protected behind the Technion firewall and therefor you must be connected to the Technion network before “SSHing” to dgx-master or remotely connected to Technion VPN.
The login node is the entry point to the Athena Cluster, where users can manage jobs, edit scripts, and transfer files. It is not meant for running computations – jobs must be submitted to SLURM to execute on compute nodes. Running heavy tasks on the login node can disrupt other users. Currently, dgx-master is the main Login node for Athena but in the near future there might be multiple servers that will act as the frontend of the cluster.

SSH Example

SSH using MobaXterm Windows client

Windows users should download two programs to access the cluster conveniently.
MobaXterm – An SSH client for remote connections.
WinSCP – An SCP client for remote file transfers.
After installing MobaXterm, open the application and select “Session,” then choose “SSH.” In the session settings, enter “dgx-master.technion.ac.il” under “Remote Host”. Check the “Specify username” box and input your Technion username (same as your email address). Set Port to 22 and proceed to connect.

After clicking OK, an Athena Cluster session will open. You will see a Linux terminal on the right and a file navigator on the left. The terminal allows you to execute commands, while the navigator helps with file transfers and management. However, all computations on Athena must be executed via the terminal using SLURM, as described in the following sections.

WinSCP provides a more convenient way to transfer files between your computer and your Athena home directory. To get started, open WinSCP and create a new session. In the connection settings, enter the following details:

• Host Name: dgx-master.technion.ac.il
• Port Number: 22
• Username: Your Technion username (same as your email’s)
• Password: Your Technion account password


Once logged in, you can easily transfer files by dragging and dropping them between your local machine and the cluster.

When you log-in to dgx-master a bash shell login session is opened, awaits your commands. Each Athena user is granted 300GB of storage assigned to his HOME directory (~/) and 2TB in the shared group (project) directory (~/work). Both directories are shared across all compute nodes.
To upload your data to the cluster storage, Use scp or WinSCP for secure file transfers.

SCP Example

The Athena Cluster provides multiple Quality of Service (QOS) options to optimize GPU allocation and job scheduling. The QOS you select determines job runtime, GPU limits, and job concurrency.

Each QOS name follows a structured format indicating its constraints:

<Partition>_<WallTime>_<Max GPUs>_<Max Running Jobs>

For example, mig_2H_2G_1J means:

• “mig” partition (for testing/evaluation).
• 2 Hours of maximum runtime (2H).
• 2 GPUs maximum (2G).
• 1 running job at a time (1J).
  
  

QOS Options for the mig Partition (Testing & Evaluation)

QOS Options for the work Partition (Main Workloads)

How to Choose the Right QOS

• How long will my job run? Short jobs → 1H QOS, Long jobs → 24H QOS.
• How many GPUs do I need? Small-scale → 1G or 2G, Large-scale → 8G or 16G.
• Do I need multiple jobs running at the same time? Need parallel jobs → QOS with higher job limits (e.g., 4J, 10J).
• Is this a test or production workload? Debugging/testing → mig QOS, Running actual jobs → work QOS.
  
  

A SLURM batch script is a Bash script that includes SLURM directives to define resource requests and execution parameters. These directives begin with #SBATCH and guide the job scheduler in allocating the necessary compute resources.

Essential SLURM Directives

A sample SLURM batch script for launching a deep learning training job:

#!/bin/bash
#SBATCH --job-name=my_training_job
#SBATCH --output=job_%j.out
#SBATCH --error=job_%j.err
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --gpus=4
#SBATCH --qos=work_24H_8G_4J
#SBATCH --partition=work
#SBATCH --time=12:00:00

# Load necessary modules
module load cuda/12.2
module load pytorch/2.1

# Run the job inside a container
srun --container-image=/home/user/container.sqsh --container-mounts=/home/user/data:/mnt/data python train.py
  

Explanation of Key Directives:

• Resources: Requests 4 GPUs and 32 CPU cores for computation.
• QOS & Partition: Uses the work_24H_8G_4J QOS in the work partition.
• Environment Setup: Loads CUDA and PyTorch modules before running the training script.
• Container Execution: Runs inside an Enroot container with external data mounted.

Once the batch script is ready, submit the job using:

sbatch my_job_script.sh

After submission, SLURM assigns a Job ID, which can be used to track the job’s status.

Example Output:

Submitted batch job 123456

To check the status of your submitted jobs:

1. List all your jobs:

squeue -u $USER

2. View job details:

scontrol show job <job_id>

3. Monitor real-time job logs:

tail -f job_<job_id>.out

If needed, cancel a job using:

scancel <job_id>

Example:

scancel 123456

This immediately stops the job and releases the allocated resources.

squeue -u $USER

Check SLURM output and error logs:

cat slurm-.out

Monitor GPU usage in real-time:

watch -n 1 nvidia-smi

Use efficient communication libraries like NCCL for multi-GPU jobs.

Always check resource requests to avoid jobs pending indefinitely.

du -sh ~

Most image repositories will advise you to copy-paste a line in the “docker” (Hub) form:
docker pull nvcr.io/nvidia/tensorflow:22.02-tf2-py3 but when we want to download the image using Enroot, as a squash file, to our filesystem, we need to convert the “pull” command to an “import” command according to the following instructions:

1. Replace the “/” character after “nvidia” with the “#” character.
2. Add “docker://” before “nvcr”.
3. Replace “docker pull” for “enroot import”.

enroot import docker://nvcr.io#nvidia/tensorflow:22.02-tf2-py3

• Efficient Resource Allocation: Different jobs have different needs, from quick debugging to long training runs.
• Preventing Job Starvation: Long-running jobs should not block smaller jobs from starting.
• Maximizing GPU Utilization: Fair allocation of compute power among users.
• Encouraging Testing Before Production Runs: Users can evaluate container images and test jobs in the mig partition before using full resources.
  

Enroot download containers layer-by-layer to the Enroot TMPDIR directory which is configured by default as “/tmp”.
The size of the “/tmp” directory is limited and might reach its capacity if multiple imports have been made and wasn’t yet deleted. If you encounter this scenario you may modify the TMPDIR environment variable:
export TMPDIR="/home//work/tmp"before importing an Enroot image to Login Node.

Check squeue and ensure sufficient resources are available.

Use rsync or WinSCP for efficient transfers.