Contents
- 1 Why is my job being killed?
- 2 How do I allocate memory (or other resources) for the job I am submitting to the cluster?
- 3 Why is my priority access to a compute node not working?
- 4 Can you help me understand SLURM preemption on G2?
- 5 How many GPUs can I ask for in my job?
- 6 Why won’t my job run on a node that I know has a free GPU?
Why is my job being killed?
- Question: My jobs are being killed with the a message saying that it is out of memory but the server still has memory available. Why?
- Question: My jobs gets killed after 4 hours of running. Why?
- Question: My job fails when it cannot find a GPU. I know that the server that the job is assigned to has GPUs. Why can’t it use the GPUs?
- Answer: By default, a job gets assigned 4 hours of runtime, 1 cpu, 1G of memory and no GPUs. If your jobs tries to use more cpu or memory than that, it will be killed automatically. You must request (when you submit the job) the maximum amount of resources that you job will need .
- Ex: If your job spends most of it’s runtime using 1G of memory, but occasionally spikes to 5G, you must request 5G of memory at job submission time.
How do I allocate memory (or other resources) for the job I am submitting to the cluster?
- Question: I’d like to know how to claim runtime, memory, cpus or GPUs on the G2 cluster. For example, if my program uses 16GB of memory at peak, should I claim –mem=16000?
- Answer: When submitting a job, memory is defined in MBs, not GBs. Therefore, if you know that your jobs memory usage will top out at 16GB, then you should request “–mem=16384” (16 x 1024). To request more than one CPU, use “-m 5” (for 5 cpus) and if you need to use a GPU, you should use “–gres=gpu:1” (for one GPU). To increase the amount of time that your job can use, use “-t 8:0:0” for 8 hours.
Why is my priority access to a compute node not working?
- Question: I know that I have priority access to my groups compute node. No one else in our group has any jobs running, but I cannot get my job to run on our compute node. Why?
- Question: I know that I have priority access to my groups compute node and I am submitting my job to the priority partition. But my job is set to pending. Why?
- Answer: To get priority access to your compute node, you must submit your job to your priority partition, not the “gpu” or “default_partition” partitions. This is done with the “-p” switch. If you are submitting to the appropriate priority partition and your job is still pending, it may be because others in your group are using the node or it may be that you are asking for more resources than are available. To find out what resources others in your group are using on your priority compute node, get in touch with us. We will set it up so that you can see what others in your group are using.
- EX: If you are in Prof. Garg’s group, you should use the switch “-p garg” in your job submission script.
Can you help me understand SLURM preemption on G2?
- There are three priority levels of partitions (see below). There are the default_partition, gpu and high-priority partitions. Each compute node owner has a high-priority partition. The fewer resources your job needs, the more likely it is to be submitted to a node and the less likely it is to be preempted. Jobs submitted to Priority partitions cannot preempt other jobs already submitted to the priority partition. You can only submit jobs to the priority partition of the faculty that you are associated with.
- All jobs, including interactive jobs, must be submitted to a specific partition (or queue).
- Preemption order. Please submit your job to the lowest priority that is needed.
- Low-Priority: For batch and interactive jobs requiring CPUs and/or GPUs, use the “default_partition” partition.
- Medium-priority, GPU required: For batch and interactive jobs requiring GPUs, use the “gpu” partition.
- This partition will preempt any jobs running on a GPU server which were submitted to the Low-Priority partition.
- High-priority: for batch and interactive jobs requiring CPUs and/or GPUS, use the priority queue that belongs to your group.
- Only the servers owned by the faculty to whom this priority partition belongs will be available through these partitions.
- This partition will preempt any jobs running on any server (owned by faculty to which it belongs) which were submitted to the Low/Medium-Priority partitions.
- Preemption order. Please submit your job to the lowest priority that is needed.
- All jobs, including interactive jobs, must be submitted to a specific partition (or queue).
How many GPUs can I ask for in my job?
- Unless you are using MPI, your job cannot ask for more GPUs that any one server in the cluster has (which is 8). When you submit a job asking for just one GPU, your job can run on any server that has enough resources (cpu and memory) plus one GPU. This gives you a potential pool of 60+ servers that your job can run on and frequently, many of those 60 servers have one GPU available. When your job asks for more resources, such as 4 GPUs. Then it can run on any server that has enough resources, plus 4 GPUs free. While this is still a potential pool of 50+ servers (not all servers have 4 physical GPUs), a very small percentage of the servers will actually have 4 GPUs unallocated so the chances of your job being run in a timely manner is much smaller than if you submitted 4 separate jobs, each asking for one GPU. If you ask for even more GPUs (say 6 or 8), then the pool of available servers is reduced to 25+, as only a small percentage of servers have enough GPUs to be able to service your request, and very few of them will have that many GPUs available in a timely manner. What I am trying to say is that the more resources you request per job, the less likely it is that your job will be able to run right away. If you ask for the max resources available on any one compute node, it may take weeks before your job is run. If you ask for more resources than is available on any compute node, it will not even submit the job.
Why won’t my job run on a node that I know has a free GPU?
- For any job to be submitted to a compute node, the compute node must have at least as many GPUs, CPUs and GB of memory available for use as your job requests. The scheduler does not know how much of any of these resources are actually in use. It does it’s scheduling by calculating what each job asks for and look for compute nodes that have at least that much resource un-scheduled. If a compute node does not have any one of these resources unscheduled in sufficient quantity, the job will not be scheduled to that compute node. For Example: If you ask for a specific node, 6 CPUs, 8 GPUs and 240GB of memory and the compute node has 8 CPUs, 8 GPUs and 220GB of memory, the job will not be assigned to that node even though it has enough CPU and GPU, because it does not have enough memory.