G2 FAQs – IT Support: CoE CIS Tech

Contents

1 Why is my job being killed?
2 My job uses /tmp (or /var/tmp) and fills it up.
3 How do I allocate memory (or other resources) for the job I am submitting to the cluster?
4 Why is my priority access to a compute node not working?
5 My job is dying without creating any log files
6 Can you help me understand SLURM preemption on G2?
7 How many GPUs can I ask for in my job?
8 Why won’t my job run on a node that I know has a free GPU?
9 How can I request a GPU based on the amount of memory it has?
10 How can I request a GPU based on the specific model of GPU?

Why is my job being killed?

Question: My jobs are being killed with the a message saying that it is out of memory but the server still has memory available. Why?
Question: My jobs gets killed after 4 hours of running. Why?
Question: My job fails when it cannot find a GPU. I know that the server that the job is assigned to has GPUs. Why can’t it use the GPUs?
Answer: By default, a job gets assigned 4 hours of runtime, 1 cpu, 1G of memory and no GPUs. If your jobs tries to use more cpu or memory than that, it will be killed automatically. You must request (when you submit the job) the maximum amount of resources that you job will need .
- Ex: If your job spends most of it’s runtime using 1G of memory, but occasionally spikes to 5G, you must request 5G of memory at job submission time.

My job uses /tmp (or /var/tmp) and fills it up.

Question: My job uses /tmp (or /var/tmp) and fills it up. This causes my job to fail and the job failing does not clean /tmp.
Answer: Create a tmpdir on your /share directory. Your program should use it instead of /tmp. You can define it in a couple of ways.
export TMPDIR=/home/NetID/test
OR
TMPDIR=/home/NetID/app-name/tmp start_my_app

To test to see if it is doing it correctly, use the command:

“mktemp -u”

EX:
********************************************
root@slurmctld:~# mktemp -u
/tmp/tmp.sT4cVqCbTb
root@slurmctld:~# TMPDIR=/home/maw349/app-name/tmp mktemp -u
/home/maw349/app-name/tmp/tmp.Kf0cHlM955
root@slurmctld:~# mktemp -u
/tmp/tmp.OoPLxTHWf0
root@slurmctld:~# export TMPDIR=/home/maw349/test
root@slurmctld:~# mktemp -u
/home/maw349/test/tmp.eci5c6xSZt

How do I allocate memory (or other resources) for the job I am submitting to the cluster?

Question: I’d like to know how to claim runtime, memory, cpus or GPUs on the G2 cluster. For example, if my program uses 16GB of memory at peak, should I claim –mem=16000?
Answer: When submitting a job, memory is defined in MBs, not GBs. Therefore, if you know that your jobs memory usage will top out at 16GB, then you should request “–mem=16384” (16 x 1024). To request more than one CPU, use “-m 5” (for 5 cpus) and if you need to use a GPU, you should use “–gres=gpu:1” (for one GPU). To increase the amount of time that your job can use, use “-t 8:0:0” for 8 hours.

Why is my priority access to a compute node not working?

Question: I know that I have priority access to my groups compute node. No one else in our group has any jobs running, but I cannot get my job to run on our compute node. Why?
Question: I know that I have priority access to my groups compute node and I am submitting my job to the priority partition. But my job is set to pending. Why?
Answer: To get priority access to your compute node, you must submit your job to your priority partition, not the “gpu” or “default_partition” partitions. This is done with the “-p” switch. If you are submitting to the appropriate priority partition and your job is still pending, it may be because others in your group are using the node or it may be that you are asking for more resources than are available. To find out what resources others in your group are using on your priority compute node, get in touch with us. We will set it up so that you can see what others in your group are using.
- EX: If you are in Prof. Garg’s group, you should use the switch “-p garg” in your job submission script.

My job is dying without creating any log files

Question: I submitted a job and received a job-id. However, the job never ran and created no log files, even though I configured the job submission script to create them
- Answer 1:
  - you must have write permission on the directory where the files will be created and the directory must exist.
    - EX: If you give a path of “output/%N.o%j” the directory “output” must exist and you must have permissions configured to allow you to create files in it.
- Answer 2:
  - The first line of the job submission script must start with a line containing only “#!/bin/bash” (or whatever shell you want to use). It must be followed by all lines beginning with “#SBATCH” with no line spaces until all of the “#SBATCH” lines have been completed.

Can you help me understand SLURM preemption on G2?

There are three priority levels of partitions (see below). There are the default_partition, gpu and high-priority partitions. Each compute node owner has a high-priority partition. The fewer resources your job needs, the more likely it is to be submitted to a node and the less likely it is to be preempted. Jobs submitted to Priority partitions cannot preempt other jobs already submitted to the priority partition. You can only submit jobs to the priority partition of the faculty that you are associated with.
- All jobs, including interactive jobs, must be submitted to a specific partition (or queue).
  - Preemption order. Please submit your job to the lowest priority that is needed.
    - Low-Priority: For batch and interactive jobs requiring CPUs and/or GPUs, use the “default_partition” partition.
    - Medium-priority, GPU required: For batch and interactive jobs requiring GPUs, use the “gpu” partition.
      - This partition will preempt any jobs running on a GPU server which were submitted to the Low-Priority partition.
    - High-priority: for batch and interactive jobs requiring CPUs and/or GPUS, use the priority queue that belongs to your group.
      - Only the servers owned by the faculty to whom this priority partition belongs will be available through these partitions.
      - This partition will preempt any jobs running on any server (owned by faculty to which it belongs) which were submitted to the Low/Medium-Priority partitions.

How many GPUs can I ask for in my job?

Unless you are using MPI, your job cannot ask for more GPUs that any one server in the cluster has (which is 8). When you submit a job asking for just one GPU, your job can run on any server that has enough resources (cpu and memory) plus one GPU. This gives you a potential pool of 60+ servers that your job can run on and frequently, many of those 60 servers have one GPU available. When your job asks for more resources, such as 4 GPUs. Then it can run on any server that has enough resources, plus 4 GPUs free. While this is still a potential pool of 50+ servers (not all servers have 4 physical GPUs), a very small percentage of the servers will actually have 4 GPUs unallocated so the chances of your job being run in a timely manner is much smaller than if you submitted 4 separate jobs, each asking for one GPU. If you ask for even more GPUs (say 6 or 8), then the pool of available servers is reduced to 25+, as only a small percentage of servers have enough GPUs to be able to service your request, and very few of them will have that many GPUs available in a timely manner. What I am trying to say is that the more resources you request per job, the less likely it is that your job will be able to run right away. If you ask for the max resources available on any one compute node, it may take weeks before your job is run. If you ask for more resources than is available on any compute node, it will not even submit the job.

Why won’t my job run on a node that I know has a free GPU?

For any job to be submitted to a compute node, the compute node must have at least as many GPUs, CPUs and GB of memory available for use as your job requests. The scheduler does not know how much of any of these resources are actually in use. It does it’s scheduling by calculating what each job asks for and look for compute nodes that have at least that much resource un-scheduled. If a compute node does not have any one of these resources unscheduled in sufficient quantity, the job will not be scheduled to that compute node. For Example: If you ask for a specific node, 6 CPUs, 8 GPUs and 240GB of memory and the compute node has 8 CPUs, 8 GPUs and 220GB of memory, the job will not be assigned to that node even though it has enough CPU and GPU, because it does not have enough memory.

How can I request a GPU based on the amount of memory it has?

G2 has 3 different categories (gpu classes) for GPU memory sizes. Insert these into the “srun/sbatch” commands as demonstrated below.
- Please utilize the gpu class with the lowest amount of GPU-memory that your job requires.
- If you don’t care which type of gpu that it runs on, don’t add this switch.
- Multiple gpu classes can be added with the “|” separator.
  - It is better to not use the constraint switch than to request all three gpu classes in the same job (same result, less overhead).
    - gpu-low # each GPU has less than 20G of memory.
      - These servers are automatically the first requested when submitting to the default partitions.
    - gpu-mid # each GPU has between 20G and 39G of memory
    - gpu-high # each GPU has 40G or more of memory
  - EX: Add the following switch to your:
    - srun command
      - –constraint=”gpu-low”
    - sbatch job submission script.
      - #SBATCH –constraint=”gpu-low|gpu-mid”

How can I request a GPU based on the specific model of GPU?

Each GPU has it’s own designation.
- Insert the following model name into the “srun/sbatch” commands (as demonstrated below) to request that your job be submitted to any node containing the requested GPU type and available in the partition requested.
- Multiple gpu models can be added with the “|” separator.
  - titanx # 12G memory
  - titanxp # 12G memory
  - 1080ti # 11G memory
  - 2080ti # 11G memory
  - titanrtx # 24G memory
  - r6000 # 22G memory
  - 3090 # 24G memory
  - a5000 # 24G memory
  - a40 # 48G memory
  - a6000 # 48G memory
  - a100-40 # 40G memory

- - EX: Add the following switch to your:
    - srun command
      - –constraint=”titanx”
    - sbatch job submission script.
      - #SBATCH –constraint=”titanx|titanxp|1080ti|2080ti”

Learn more about the ITSG

IT Maintenance and Alerts

Sitemap

IT@Cornell Services