gres.conf(5)

💖 NAME 💖

gres.conf(5) — the Slurm configuration file for Generic RESource (GRES) management, aka the lil file that tells your compute nodes what fancy hardware they're hoarding. 🎀

---

🦋 DESCRIPTION 🦋

okay so bestie, gres.conf is an ASCII file, and it describes the configuration of Generic RESource(s) — GRES for short — on each compute node. ✨

If the GRES info in the slurm.conf file does not fully describe those resources, then you should drop a gres.conf file on every single compute node. For cloud nodes, a gres.conf file that includes all the cloud nodes must live on all cloud nodes and the controller. And this file? It will always be located in the same directory as slurm.conf. no cap, same dir, every time. 💅

But if the GRES info in slurm.conf does fully describe those resources — like, no "Cores", "File" or "Links" specification is required for that GRES type, or that info is automatically detected — then you can totally omit it from gres.conf and only the configuration info in the slurm.conf file gets used. And the gres.conf file itself may be omitted completely if the configuration info in slurm.conf fully describes all GRES. lazy girl hack, but legal. 🙈

If you're using gres.conf to describe the resources available to nodes, the first parameter on the line should be NodeName. If you're configuring Generic Resources without specifying nodes, the first parameter on the line should be Name. that's the vibe check right there. ⭐

Parameter names are case insensitive. Any text following a "#" in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command scontrol reconfigure — unless otherwise noted. 👑

NOTE: Slurm support for gres/[mps|shard] requires the use of the select/cons_tres plugin. For more information on how to configure MPS, see https://slurm.schedmd.com/gres.html#MPS_Management. For more information on how to configure Sharding, see https://slurm.schedmd.com/gres.html#Sharding.

For more information on GRES scheduling in general, see https://slurm.schedmd.com/gres.html. 🌈

The overall configuration parameters available include: 👇

---

AutoDetect ✨

The hardware detection mechanisms to enable for automatic GRES configuration. Currently, the options are:

• full — On slurmd, try all hardware detection mechanisms in order (NVML, NVIDIA, RSMI, oneAPI, NRT) until GPUs are detected, stopping after the first one that finds a device.

NOTE: This is the default for dynamic nodes (slurmd -Z) when gres.conf has no AutoDetect setting and no explicit Name=gpu records.

• nrt — Automatically detect AWS Trainium/Inferentia devices. 🧚
• nvidia — Automatically detect NVIDIA GPUs. No library required, but doesn't detect MIGs or NVlinks. Added in Slurm 24.11.
• nvml — Automatically detect NVIDIA GPUs. Requires the NVIDIA Management Library (NVML).
• off — Do not automatically detect any GPUs. Used to override other options.
• oneapi — Automatically detect Intel GPUs. Requires the Intel Graphics Compute Runtime for oneAPI Level Zero and OpenCL Driver (oneapi).
• rsmi — Automatically detect AMD GPUs. Requires the ROCm System Management Interface (ROCm SMI) Library.

AutoDetect can be on a line by itself, in which case it will globally apply to all lines in gres.conf by default. In addition, AutoDetect can be combined with NodeName to only apply to certain nodes. Node-specific AutoDetects will trump the global AutoDetect. A node-specific AutoDetect only needs to be specified once per node. If specified multiple times for the same nodes, they must all be the same value. To unset AutoDetect for a node when a global AutoDetect is set, simply set it to "off" in a node-specific GRES line. E.g.: NodeName=tux3 AutoDetect=off Name=gpu File=/dev/nvidia[0-3]. AutoDetect cannot be used with cloud nodes. 💀

AutoDetect will automatically detect files, cores, links, and any other hardware. If a parameter such as File, Cores, or Links are specified when AutoDetect is used, then the specified values are used to sanity check the auto detected values. If there is a mismatch, then the node's state is set to invalid and the node is drained. so don't be sloppy. 😭

Count ✨

Number of resources of this name/type available on this node. The default value is set to the number of File values specified (if any), otherwise the default value is one.

A suffix of "K", "M", "G", "T" or "P" may be used to multiply the number by 1024, 1048576, 1073741824, etc. respectively. For example: Count=10G. 🔥

Cores ✨

Optionally specify the core index numbers matching the specific socket* which can use this resource.

If this option is used, all the cores in a socket* must be specified together. While Slurm can track and assign resources at the CPU or thread level, its scheduling algorithms used to co-allocate GRES devices with CPUs operates at a socket* level for job allocations. Therefore, it is not possible to preferentially assign GRES with different specific CPUs on the same socket*. 😤

*Sockets may be substituted for NUMA nodes with SlurmdParameters=numa_node_as_socket or l3cache with SlurmdParameters=l3cache_as_socket.

Multiple cores may be specified using a comma-delimited list or a range may be specified using a "-" separator (e.g. "0,1,2,3" or "0-3"). If a job specifies --gres-flags=enforce-binding, then only the identified cores can be allocated with each generic resource. This will tend to improve performance of jobs, but delay the allocation of resources to them. If specified and a job is not submitted with the --gres-flags=enforce-binding option the identified cores will be preferred for scheduling with each generic resource.

If --gres-flags=disable-binding is specified, then any core can be used with the resources, which also increases the speed of Slurm's scheduling algorithm but can degrade the application performance. The --gres-flags=disable-binding option is currently required to use more CPUs than are bound to a GRES (e.g. if a GPU is bound to the CPUs on one socket, but resources on more than one socket are required to run the job). If any core can be effectively used with the resources, then do not specify the cores option for improved speed in the Slurm scheduling logic. A restart of the slurmctld is needed for changes to the Cores option to take effect.

NOTE: Since Slurm must be able to perform resource management on heterogeneous clusters having various processing unit numbering schemes, a logical core index must be specified instead of the physical core index. That logical core index might not correspond to your physical core index number. Core 0 will be the first core on the first socket, while core 1 will be the second core on the first socket. This numbering coincides with the logical core number (Core L#) seen in "lstopo -l" command output. iykyk 🧠

File ✨

Fully qualified pathname of the device files associated with a resource. The name can include a numeric range suffix to be interpreted by Slurm (e.g. File=/dev/nvidia[0-3]).

This field is generally required if enforcement of generic resource allocations is to be supported (i.e. prevents users from making use of resources allocated to a different user). Enforcement of the file allocation relies upon Linux Control Groups (cgroups) and Slurm's task/cgroup plugin, which will place the allocated files into the job's cgroup and prevent use of other files. Please see Slurm's Cgroups Guide for more information: https://slurm.schedmd.com/cgroups.html. 🫶

If File is specified then Count must be either set to the number of file names specified or not set (the default value is the number of files specified). The exception to this is MPS/Sharding. For either of these GRES, each GPU would be identified by device file using the File parameter and Count would specify the number of entries that would correspond to that GPU. For MPS, typically 100 or some multiple of 100. For Sharding typically the maximum number of jobs that could simultaneously share that GPU.

If using a card with Multi-Instance GPU functionality, use MultipleFiles instead. File and MultipleFiles are mutually exclusive. pick one bestie, not both. 💅

NOTE: File is required for all gpu typed GRES.

NOTE: If you specify the File parameter for a resource on some node, the option must be specified on all nodes and Slurm will track the assignment of each specific resource on each node. Otherwise Slurm will only track a count of allocated resources rather than the state of each individual device file.

NOTE: Drain a node before changing the count of records with File parameters (e.g. if you want to add or remove GPUs from a node's configuration). Failure to do so will result in any job using those GRES being aborted. no like literally do not skip this step 😭

NOTE: When specifying File, Count is limited in size (currently 1024) for each node.

Flags ✨

Optional flags that can be specified to change configured behavior of the GRES.

Allowed values at present are:

• CountOnly — Do not attempt to load a plugin of the GRES type as this GRES will only be used to track counts of GRES used. This avoids attempting to load non-existent plugin which can affect filesystems with high latency metadata operations for non-existent files.

NOTE: If a gres has this flag configured it is global, so all other nodes with that gres will have this flag implied.

• env_uuid — Set vendor environment variables using UUIDs instead of numeric device indices. Affects these environment variables: CUDA_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES. Requires the use of AutoDetect so that UUIDs are available. 👑

okay so — for example, instead of "CUDA_VISIBLE_DEVICES=0", it would look like:

CUDA_VISIBLE_DEVICES=GPU-8932f937-d72c-4106-c12f-20bd9faed9f6

• explicit — if this flag is set, GRES is not allocated to the job as part of whole node allocation (--exclusive or partition Exclusive=NODE set) unless the job explicitly requested it. ✨

⚠️ NOTE: if a gres has this flag configured it is global, so all other nodes with that gres will have this flag implied. bestie that's not a suggestion, that's just how it is 💅

• one_sharing — for use on a shared gres. if you're using a shared gres (mps) on top of a sharing gres (gpu), this only lets one of the sharing gres be used by the shared gres. this is the default for MPS. 🫶

⚠️ NOTE: if a gres has this flag configured it is global, so all other nodes with that gres will have this flag implied. this flag is not compatible with all_sharing for a specific gres. pick a lane, bestie 🙈

• all_sharing — for use on a shared gres. this is the opposite of one_sharing and can be used to allow all sharing gres (gpu) on a node to be used for shared gres (mps). 🌈

⚠️ NOTE: if a gres has this flag configured it is global, so all other nodes with that gres will have this flag implied. this flag is not compatible with one_sharing for a specific gres. same energy, opposite direction 💀

• nvidia_gpu_env — sets the environment variable CUDA_VISIBLE_DEVICES for all GPUs on the specified node(s). 🎀
• amd_gpu_env — sets the environment variable ROCR_VISIBLE_DEVICES for all GPUs on the specified node(s). 🌸
• intel_gpu_env — sets the environment variable ZE_AFFINITY_MASK for all GPUs on the specified node(s). 🦋
• opencl_env — sets the environment variable GPU_DEVICE_ORDINAL for all GPUs on the specified node(s). ⭐
• no_gpu_env — sets no GPU-specific environment variables at all. this one is mutually exclusive with all the other environment-related flags. 💀

so like, if you don't specify any environment-related flags at all, then nvidia_gpu_env, amd_gpu_env, intel_gpu_env, and opencl_env all get implicitly set by default. 🫶 and if AutoDetect is used and environment-related flags aren't specified, then AutoDetect=nvml or AutoDetect=nvidia sets nvidia_gpu_env, AutoDetect=rsmi sets amd_gpu_env, and AutoDetect=oneapi sets intel_gpu_env. conversely, environment-related flags you do specify will always override AutoDetect. periodt. 👑

environment-related flags set on one GRES line get inherited by the GRES line directly below it, as long as no environment-related flags are specified on that line and it's the same node, name, and type. and env-related flags have to be the same for GRES of the same node, name, and type — no chaotic mixed vibes allowed. ✨

⚠️ heads up: there's a known issue with the AMD ROCm runtime where ROCR_VISIBLE_DEVICES gets processed first, and then CUDA_VISIBLE_DEVICES gets processed. to dodge the mess this causes, set Flags=amd_gpu_env for AMD GPUs so only ROCR_VISIBLE_DEVICES is set. trust, it's giving self-care 😭

---

🌸 Links 🌸

a comma-delimited list of numbers identifying the number of connections between this device and other devices, so you can coschedule the better-connected devices together. 💖

this is an ordered list — the number of connections this specific device has to device number 0 goes in the first position, the number of connections it has to device number 1 goes in the second position, and so on. a -1 means the device itself, and a 0 means no connection at all. if you specify this, then the line can only contain a single GRES device (i.e. it can only contain a single file via File).

this is an optional value and it's usually automatically determined if AutoDetect is enabled. a typical use case would be identifying GPUs that have NVLink connectivity. 🧚

NOTE: for GPUs, the minor number assigned by the OS and used in the device file (i.e. the X in /dev/nvidiaX) is not necessarily the same as the device number/index. the device number is created by sorting the GPUs by PCI bus ID and then numbering them starting from the smallest bus ID. see https://slurm.schedmd.com/gres.html#GPU_Management

🎀 MultipleFiles 🎀

fully qualified pathname of the device files associated with a resource. graphics cards using Multi-Instance GPU (MIG) technology will present multiple device files that should be managed as a single generic resource. the file names can be a comma separated list, or it can include a numeric range suffix (e.g. MultipleFiles=/dev/nvidia[0-3]). ✨

🚨 DRAIN THE NODE before changing the count of records with the MultipleFiles parameter — like when adding or removing GPUs from a node's configuration. failure to do so will result in any job using those GRES being aborted. not a drill, bestie 😭

when you're not using GPUs with MIG functionality, use File instead. MultipleFiles and File are mutually exclusive. it's giving "you can't sit with both of us" 💅

⭐ Name ⭐

name of the generic resource. any desired name may be used. the name must match a value in GresTypes in slurm.conf. 🫶

each generic resource has an optional plugin which can provide resource-specific functionality. generic resources that currently include an optional plugin are:

• gpu — Graphics Processing Unit 👑
• mps — CUDA Multi-Process Service (MPS) 💜
• nic — Network Interface Card 🌈
• shard — Shards of a gpu 💎

🦋 NodeName 🦋

an optional NodeName specification can be used so one single gres.conf file works for all the compute nodes in a cluster — you just specify which node(s) each line should apply to. the NodeName specification can use a Slurm hostlist specification, as shown in the example below. so efficient, so iconic. ✨

💅 Type 💅

an optional arbitrary string identifying the type of generic resource. for example, this might be used to identify a specific model of GPU, which users can then specify in a job request. 🎀

for changes to the Type option to take effect with a scontrol reconfig, all affected slurmd daemons must be responding to the slurmctld. otherwise a restart of the slurmctld and slurmd daemons is required. no shortcut, sorry bestie 😭

⚠️ NOTE: if you're using autodetect functionality and defining the Type in your gres.conf file, the Type you specify should match — or be a substring of — the value that is detected, using an underscore in lieu of any spaces. 🫶

---

✨ EXAMPLES ✨

okay first up — GPU devices with MPS support, plus AutoDetect sanity checking: 💖

##################################################################
# Slurm's Generic Resource (GRES) configuration file
# Define GPU devices with MPS support, with AutoDetect sanity checking
##################################################################
AutoDetect=nvml
Name=gpu Type=gtx560 File=/dev/nvidia0 COREs=0,1
Name=gpu Type=tesla  File=/dev/nvidia1 COREs=2,3
Name=mps Count=100 File=/dev/nvidia0 COREs=0,1
Name=mps Count=100  File=/dev/nvidia1 COREs=2,3

then overwriting the system defaults and explicitly configuring three GPUs — note that nvidia2 is out of service, so it's commented out: 🙈

##################################################################
# Slurm's Generic Resource (GRES) configuration file
# Overwrite system defaults and explicitly configure three GPUs
##################################################################
Name=gpu Type=tesla File=/dev/nvidia[0-1] COREs=0,1
# Name=gpu Type=tesla  File=/dev/nvidia[2-3] COREs=2,3
# NOTE: nvidia2 device is out of service
Name=gpu Type=tesla  File=/dev/nvidia3 COREs=2,3

using a single gres.conf file for all compute nodes — the positive method: 👑

##################################################################
# Slurm's Generic Resource (GRES) configuration file
# Use a single gres.conf file for all compute nodes - positive method
##################################################################
## Explicitly specify devices on nodes tux0-tux15
# NodeName=tux[0-15]  Name=gpu File=/dev/nvidia[0-3]
# NOTE: tux3 nvidia1 device is out of service
NodeName=tux[0-2]  Name=gpu File=/dev/nvidia[0-3]
NodeName=tux3  Name=gpu File=/dev/nvidia[0,2-3]
NodeName=tux[4-15]  Name=gpu File=/dev/nvidia[0-3]

using NVML to gather GPU config info for every node except one: 🌸

##################################################################
# Slurm's Generic Resource (GRES) configuration file
# Use NVML to gather GPU configuration information
# for all nodes except one
##################################################################
AutoDetect=nvml
NodeName=tux3 AutoDetect=off Name=gpu File=/dev/nvidia[0-3]

specifying some nodes with NVML, some with RSMI, and some with no AutoDetect at all: 🌈

##################################################################
# Slurm's Generic Resource (GRES) configuration file
# Specify some nodes with NVML, some with RSMI, and some with no AutoDetect
##################################################################
NodeName=tux[0-7] AutoDetect=nvml
NodeName=tux[8-11] AutoDetect=rsmi
NodeName=tux[12-15] Name=gpu File=/dev/nvidia[0-3]

and finally, defining a 'bandwidth' GRES as a way to limit resource use on these nodes for workflow purposes: 🎀

##################################################################
# Slurm's Generic Resource (GRES) configuration file
# Define 'bandwidth' GRES to use as a way to limit the
# resource use on these nodes for workflow purposes
##################################################################
NodeName=tux[0-7] Name=bandwidth Type=lustre Count=4G Flags=CountOnly

---

💜 COPYING 💜

okay legal era, stay with me, it's quick: 🫶

• Copyright (C) 2010 The Regents of the University of California.
• Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
• Copyright (C) 2010-2022 SchedMD LLC.

this file is part of Slurm, a resource management program. for details, see https://slurm.schedmd.com/. ✨

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. 💅

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. see the GNU General Public License for more details. no cap, it literally says that 😭

---

🦋 SEE ALSO 🦋

• slurm.conf(5) — the other half of your personality, go read it bestie 💖

---

and that's a wrap on our girl gres.conf(5), from Slurm 26.05 — the Slurm Configuration File, dated the Slurm 26.05 era. 👑💖✨ go configure some GPUs and slay, fr fr 🌸🦋🎀