HPC – Getting Started

To open HPC account, please fill and submit HPC Account request form.

1. For account on ATHENA/DGX GPU Cluster, please specify “GPU Cluster Users”   Affiliation.
2. For account on ZEUS CPU Cluster – please choose “CPU Cluster Users”  Affiliation.
3. “Technion – Attached Server Owner” Affiliation is for account on MAFAT cluster
   or any other private cluster (should be checked with your PI – Research Group leader)
4. “Academic Course Teaching” Affiliation is for students, who will use HPC resources
   to study  Academic Course (should be checked with your Teacher/PI). Please indicate
   course number as Research Title.
5. PI – Research Group leader (academic Staff member) should be specified as a Budget Owner.
6. Research group members should choose “Another person” as Budget Owner and introduce the PI’s details
7. Please fill all fields of the form in English.  Upon account opening notification letter with detailed
   instructions will be sent to your Technion e-mail address.

1. The username and password are identical to your Technion mail credentials.   
2. Your default disk quota for home directory is 300 GB and 7 daily snapshots are defined. You can check your personal quota with the command:  quota-u
3. You can access ZEUS via ssh from any Linux computer within the Technion network, using the command:
   ssh username@zeus.technion.ac.il    applying your Technion password.  
4. For Windows users it is recommended to download and install: 
   1. MobaXterm home edition of the X-windows emulator and use its terminal to establish an SSH connection. (license)
   2. WinSCP  for file transfer to/from the HPC server.  Note that text files, transferred from Windows, may contain unprintable characters. It is recommended to apply dos2unix <filename> on such files 
5. For WIFI connection in the Technion please use the TechSec network.   
6. Please note that there is no direct access to ZEUS outside the Technion network (e.g. from home), unless you are connected via VPN. To establish VPN connection, please follow the instructions. 
7. Running interactive jobs on the ZEUS login node and compute nodes is strictly prohibited. To use graphical software and/or graphical post-processing, you can login to ZEUS-POST server using command: ssh username@zeus-post.technion.ac.il    applying your Technion password.    
8. Portable batch System (PBS) must be used to run jobs on ZEUS. For help creating PBS scripts you can use the PBS script generator.  Please note, that It is strictly prohibited to use any applications and scripts that involve  launching  PBS jobs from compute nodes.
9. PBS CPU queues

   Queue Name	Unused CPUs	Allocated CPUs	Total CPUs	Unused Memory (GB)	Allocated Memory (GB)	Total Memory (GB)	Running Jobs	Queued Jobs	Avg Wait Time	Queue Policy
   zeus_all_q	1328	32	1360	6350	64	6414	9	0	1	Walltime: 24:00:00
   zeus_long_q	958	162	1120	3363	1920	5282	14	3	10	Walltime: 336:00:00
   zeus_short_q	1360	0	1360	6414	0	6414	0	1	0	Walltime: 03:00:00
   zeus_new_q	628	908	1536	7509	4577	12087	48	0	922	Walltime: 72:00:00
   zeus_combined_q	18232	0	18232	80057	0	80057	0	0	0	Walltime: 24:00:00
   zeus_comb_short	11000	0	11000	49089	0	49089	0	0	0	Walltime: 03:00:00

   Please note:

   in zeus_short_q there are 80  ncpus  on each node,  in zeus_all_q  there are 80  ncpus  on each node

   in zeus_long_q    there are  80  ncpus  on each node, in zeus_new_q    there are 128   ncpus  on each node

   There are 2 combined heterogeneous queues on Zeus, which contain public nodes and various private nodes:

   1. zeus_combined_q with priority 80, wall time limit of 24 hours.
   2. zeus_comb_short queue with priority 80, wall time limit of 3 hours.
   The limit on all public queues is 600 cores per user.
10. PBS GPU queues
    gpu_v100_q – 2  GPU nodes. Each GPU node contains 40 cores and 4 Tesla V100 graphical cards, 378 GB RAM.
11. PBS billing
    CPU usage – 0.015 NIS per core-thread per hour
    GPU usage – 0.18 NIS per GPU card per hour
    In the beginning of each month all HPC users receive detailed reports on their PBS jobs run during the previous month.
    The budget owners receive reports containing the CPU/GPU usage for each user in their group and the total fees for the previous month.
    They also receive a message with payment instructions.  The up-to-date Zeus usage can be checked by the command prj_usage.

1. The username and password are identical to your Technion mail credentials.   
2. Your default disk quota for home directory is 300 GB and 7 daily snapshots are defined. You can check your personal quota with the command:  quota-u
3. You can access ZEUS via ssh from any Linux computer within the Technion network, using the command:
   ssh username@mars.technion.ac.il   or   ssh username@zeus.technion.ac.il  applying your Technion password.
4. For Windows users it is recommended to download and install: 
   1. MobaXterm home edition of the X-windows emulator and use its terminal to establish an SSH connection. (license)
   2. WinSCP  for file transfer to/from the HPC server.  Note that text files, transferred from Windows, may contain unprintable characters. It is recommended to apply dos2unix <filename> on such files 
5. For WIFI connection in the Technion please use the TechSec network.   
6. Please note that there is no direct access to ZEUS outside the Technion network (e.g. from home), unless you are connected via VPN. To establish VPN connection, please follow the instructions.
7. Running interactive jobs on the ZEUS login node is prohibited. To use graphical software and/or graphical post-processing, you can login to ZEUS-POST server using command:
   ssh username@zeus-post.technion.ac.il    applying your Technion password.    
8. Portable batch System (PBS) must be used to run jobs on ZEUS. For help creating PBS scripts you can use the PBS script generator. Please note, that It is strictly prohibited to use any applications and scripts  that involve  launching  PBS jobs from compute nodes.
9. PBS CPU queues

Please note that as  MAFAT (MARS)  user you’re entitled to submit your  batch jobs from MARS server to  the PBS queues:

mafat_new_q – 18 compute nodes, each compute node contains 256  core-threads/1 TB RAM

mafat10_q     –  10 compute nodes, each compute node contains 384 core-threads/1.5 TB RAM

mafat14_512_q     –  14 compute nodes, each compute node contains 512 core-threads/1.5 TB RAM

PBS  jobs submitted to these queues are not charged for usage.

1. 1. 1. The username and password of your GPU-based HPC account are identical to your Technion mail credentials. 
      2. Two GPU clusters are available for users:

         1. DGX cluster, that currently consists of 7 GPU nodes, each equipped with 8 Nvidia A100 graphics cards, and running the Ubuntu 20.24 operating system.
            Remark: Older GPU cluster, provides limited options and flexibility

         2. Athena cluster, currently includes  six servers equipped with graphics processing units (GPUs) of various types and running the Rocky 9 operating system.More GPU-equipped servers will be added to the cluster in the future.
            Remark: New GPU cluster, utilizing modern hardware and software technologies

      3. Both clusters are managed by the SLURM workload manager, but the implementation of the software framework and resource management policies differs significantly between the two clusters.

      4. GPU clusters can be accessed  via ssh from any Linux computer within the Technion network, using the command:
         1.  ssh username@dgx-master.technion.ac.il  
         2. ssh username@athena.technion.ac.il             and applying your Technion password
      5. For Windows users it is recommended to download and install: 
         1. MobaXterm home edition of the X-windows emulator and use its terminal to establish an SSH connection. (license)
         2. WinSCP  for file transfer to/from the HPC server.  Note that text files, transferred from Windows, may contain unprintable characters.              It is recommended to apply dos2unix <filename> on such files 
      6. For WIFI connection in the Technion please use the TechSec network.   
      7. Please note that there is no direct access to DGX and Athena clusters outside the Technion network (e.g. from home),
         unless you are connected via VPN. To establish VPN connection, please follow the instructions.
      8. The disk quota for your home directory is 300 GB, 7 daily snapshots are defined.
         You can check your personal quota usage by the quota-u  command.
      9. It is recommended to place your work files under the folder $HOME/work,  a part of your common  group storage  /rg/groupname_prj space, which is limited by the group quota. The default quota is is 2TB .  You can check your group quota usage by the command:   quota-g Additional storage space is available for pay from the Technion CIS shop by increments of 1 TB.  The annual price is 450 NIS per 1 TB.
         Only Technion Academic Staff members, holding a research project group registered on one of the HPC clusters ,
         are eligible to purchase additional storage space.
      10. Group ownership for the $HOME/work directories has been set on the GPU Clusters. This means that each member of a project group
          can grant  read/access permissions to his/her /home/username/work directory for other group members with the following command:                  chmod -R g+rX  $HOME/work
      11. Your Technion  e-mail address has been added to mailing list  cis-gpu-users–l. Please follow messages to this list for updates on GPU clusters clusters.

   12.Please follow DGX User Guide   and  Athena User Guide   for the instructions of respective clusters usage.

Please see the following  HPC Billing Procedure

Please see the following Instructions for all types of support.