slurm.conf(5)

📦

version 26.05.5.1 · 11 versions

26.05.5.1 · arch rolling, epel 10, nixpkgs unstable (latest)
24.11.3 · opensuse-leap 16.0
25-11-8 · nixos 26.05
22.05.11 · epel 9
26.05.4 · ubuntu stonking
21.08.5 · ubuntu jammy
24.05.2 · fedora 43, fedora 44
23.11.4 · ubuntu noble
22.05.8 · debian bookworm
25.11.2 · ubuntu resolute
24.11.5 · debian trixie, ubuntu questing

, in arch rolling, epel 10, nixpkgs unstable

💖 NAME 💖

slurm.conf — the Slurm configuration file. Yep bestie, this is slurm.conf(5), the manual page for that exact file, and we’re about to read it together. 💅

✨ DESCRIPTION ✨

Okay so slurm.conf is an ASCII file that describes general Slurm configuration information, the nodes to be managed, information about how those nodes are grouped into partitions, and various scheduling parameters associated with those partitions. This file should be consistent across all nodes in the cluster. No cap, cluster-wide consistency is the vibe. 🌈

The file location can be modified at execution time by setting the SLURM_CONF environment variable. The Slurm daemons also allow you to override both the built-in and environment-provided location using the -f option on the command line. Iconic override behavior, honestly. ✨

The contents of the file are case insensitive except for the names of nodes and partitions. Any text following a # in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command scontrol reconfigure unless otherwise noted. Changes to TCP listening settings will require a daemon restart. Important little caveat, bestie. 👑

If a line begins with the word Include followed by whitespace and then a file name, that file will be included inline with the current configuration file. For large or complex systems, multiple configuration files may prove easier to manage and enable reuse of some files (See INCLUDE MODIFIERS for more details). File-splitting slay. 🎀

Note on file permissions: The slurm.conf file must be readable by all users of Slurm, since it is used by many of the Slurm commands. Other files that are defined in the slurm.conf file, such as log files and job accounting files, may need to be created/owned by the user SlurmUser to be successfully accessed. Use the chown and chmod commands to set the ownership and permissions appropriately. See the section FILE AND DIRECTORY PERMISSIONS for information about the various files and directories used by Slurm. 🫶

🌸 PARAMETERS 🌸

The overall configuration parameters available include:

AccountingStorageBackupHost — The name of the backup machine hosting the accounting storage database. If used with the accounting_storage/slurmdbd plugin, this is where the backup slurmdbd would be running. Only used with systems using SlurmDBD, ignored otherwise. Backup-host realness. 💖
AccountingStorageEnforce — This controls what level of association-based enforcement to impose on job submissions. Valid options are any comma-separated combination of the following, many of which will implicitly include other options:
- all — Implies all other available options except nojobs and nosteps.
- associations — No new job is allowed to run unless a corresponding association exists in the system.
- limits — Users can be limited by association to whatever job size or run time limits are defined. Implies associations.
- nojobs — Slurm will not account for any jobs or steps on the system. Implies nosteps.
- nosteps — Slurm will not account for any steps that have run.
- qos — Jobs will not be scheduled unless a valid qos is specified. Implies associations.
- safe — A job will only be launched against an association or qos that has a TRES-minutes limit set if the job will be able to run to completion. Without this option set, jobs will be launched as long as their usage hasn't reached the TRES-minutes limit. This can lead to jobs being launched but then killed when the limit is reached. With this option, a job won't be killed due to limits, even if the limits are changed after the job was started and the association or qos violates the updated limits. Implies limits and associations. Drama prevention mode: activated. 😭
- wckeys — Jobs will not be scheduled unless a valid workload characterization key is specified. Implies associations and TrackWCKey (a separate configuration option).
AccountingStorageExternalHost — A comma-separated list of external slurmdbds (<host/ip>[:port][,...]) to register with. If no port is given, the AccountingStoragePort will be used. This allows clusters registered with the external slurmdbd to communicate with each other using the --cluster/-M client command options. The cluster will add itself to the external slurmdbd if it doesn't exist. If a non-external cluster already exists on the external slurmdbd, the slurmctld will ignore registering to the external slurmdbd. Cluster networking, but make it cute. 🦋
AccountingStorageHost — The name of the machine hosting the accounting storage database. Only used with systems using SlurmDBD, ignored otherwise. 💜
AccountingStorageParameters — Comma-separated list of options.
- max_step_records= — The number of steps that are recorded in the database for each job -- excluding batch, extern, and interactive steps.
- The following comma-separated list of key-value options are used to establish a secure connection to the database:
  - SSL_CERT — The path name of the client public key certificate file.
  - SSL_CA — The path name of the Certificate Authority (CA) certificate file.
  - SSL_CAPATH — The path name of the directory that contains trusted SSL CA certificate files.
  - SSL_KEY — The path name of the client private key file.
  - SSL_CIPHER — The list of permissible ciphers for SSL encryption.
AccountingStoragePass — The password used to gain access to the database to store the accounting data. Only used for database type storage plugins, ignored otherwise. In the case of SlurmDBD (Database Daemon) with MUNGE authentication this can be configured to use a MUNGE daemon specifically configured to provide authentication between clusters while the default MUNGE daemon provides authentication within a cluster. In that case, AccountingStoragePass should specify the named port to be used for communications with the alternate MUNGE daemon (e.g. "/var/run/munge/global.socket.2"). The default value is NULL. MUNGE lore, lowkey fascinating. 🧚
AccountingStoragePort — The listening port of the accounting storage database server. Only used for database type storage plugins, ignored otherwise. The default value is SLURMDBD_PORT as established at system build time. If no value is explicitly specified, it will be set to 6819. This value must be equal to the DbdPort parameter in the slurmdbd.conf file. Port matching is a must, bestie. ⭐
AccountingStorageTRES — Comma-separated list of resources you wish to track on the cluster. These are the resources requested by the sbatch/srun job when it is submitted. Currently this consists of any GRES, BB (burst buffer) or license along with CPU, Memory, Node, Energy, FS/[Disk|Lustre], IC/OFED, Pages, and VMem. By default Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem are tracked. These default TRES cannot be disabled, but only appended to. AccountingStorageTRES=gres/craynetwork,license/iop1 will track billing, cpu, energy, memory, nodes, fs/disk, pages and vmem along with a gres called craynetwork as well as a license called iop1. Whenever these resources are used on the cluster they are recorded. The TRES are automatically set up in the database on the start of the slurmctld. Tracking, but make it thorough. 💅

If multiple GRES of different types are tracked (e.g. GPUs of different types), then job requests with matching type specifications will be recorded. Given a configuration of "AccountingStorageTRES=gres/gpu,gres/gpu:tesla,gres/gpu:volta" Then "gres/gpu:tesla" and "gres/gpu:volta" will track only jobs that explicitly request those two GPU types, while "gres/gpu" will track allocated GPUs of any type ("tesla", "volta" or any other GPU type).

Tracking only typed GRES (e.g. "gres/gpu:tesla,gres/gpu:volta") can lead to unintuitive behavior where the untyped counterpart (e.g. "gres/gpu") is inconsistently recognized as a valid resource name. In this case, if a job requests the untyped version, then its resource allocation will be recorded as a specific type, which may not match the actual type allocated to the job if multiple different types are allocated. Thus it is recommended to always track the untyped counterpart for any typed GRES that are tracked. In addition, you may wish to use a job_submit plugin to force jobs to explicitly specify some GRES type. Okay but seriously, track the untyped counterpart. 🙈

NOTE: Setting gres/gpu will also set gres/gpumem and gres/gpuutil. gres/gpumem and gres/gpuutil can be set individually when gres/gpu is not set.

AccountingStorageType — The accounting storage mechanism type. Unset by default, which indicates that accounting records are not maintained. Current options are:
- accounting_storage/slurmdbd — The accounting records will be written to the SlurmDBD, which manages an underlying MySQL database. See man slurmdbd for more information. SlurmDBD girlies rise. 👑
AccountingStoreFlags — Comma separated list used to modify which fields the slurmctld send to the accounting database. Current options are:
- job_comment — Include the job's comment field in the job complete message sent to the Accounting Storage database. Note the AdminComment and SystemComment are always recorded in the database.
- job_env — Include a batch job's environment variables used at job submission in the job start message sent to the Accounting Storage database.
- job_extra — Include the job's extra field in the job complete message sent to the Accounting Storage database.
- job_script — Include the job's batch script in the job start message sent to the Accounting Storage database.
- no_stdio — Exclude the stdio paths when recording data into the database on a job or step start. StdOut, StdErr and StdIn db fields for jobs and steps will be empty. No stdio, no problem. 🍓

`AcctGatherNodeFreq` 🌸

Okay so this one is the sampling interval for node accounting from the AcctGather plugins. If the AcctGather plugin value is none, this parameter is literally ignored, bestie. For all other values, this parameter is the number of seconds between node accounting samples. For the acct_gather_energy/rapl plugin, set a value less than 300 because the counters may overflow beyond this rate. The default value is zero, and this value disables accounting sampling for nodes. Note: the accounting sampling interval for jobs is determined by the value of JobAcctGatherFrequency. 🫶

`AcctGatherEnergyType` 💜

This identifies the plugin used for energy consumption accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect energy consumption data for jobs and nodes. The collection of energy consumption data takes place on the node level, hence only in case of exclusive job allocation the energy consumption measurements will reflect the job's real consumption. In case of node sharing between jobs, the reported consumed energy per job (through sstat or sacct) will not reflect the real energy consumed by the jobs. Default is nothing is collected. 🎀

Configurable values at present are:

acct_gather_energy/gpu — Energy consumption data is collected from the GPU management library (e.g. rsmi) for the corresponding type of GPU. Only available for rsmi at present. Note: slurmd will keep gpu plugin loaded after configuration when this is set.
acct_gather_energy/ipmi — Energy consumption data is collected from the Baseboard Management Controller (BMC) using the Intelligent Platform Management Interface (IPMI).
acct_gather_energy/pm_counters — Energy consumption data is collected from the Baseboard Management Controller (BMC) for HPE Cray systems.
acct_gather_energy/rapl — Energy consumption data is collected from hardware sensors using the Running Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the execution of the command sudo modprobe msr.
acct_gather_energy/xcc — Energy consumption data is collected from the Lenovo SD650 XClarity Controller (XCC) using IPMI OEM raw commands.

`AcctGatherInterconnectType` 🦋

This identifies the plugin used for interconnect network traffic accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect network traffic data for jobs and nodes. The collection of network traffic data takes place on the node level, hence only in case of exclusive job allocation the collected values will reflect the job's real traffic. In case of node sharing between jobs, the reported network traffic per job (through sstat or sacct) will not reflect the real network traffic by the jobs. 👑

Configurable values at present are:

acct_gather_interconnect/ofed — Infiniband network traffic data are collected from the hardware monitoring counters of Infiniband devices through the OFED library. In order to account for per job network traffic, add the ic/ofed TRES to AccountingStorageTRES.
acct_gather_interconnect/sysfs — Network traffic statistics are collected from the Linux sysfs pseudo-filesystem for specific interfaces defined in acct_gather.conf(5). In order to account for per job network traffic, add the ic/sysfs TRES to AccountingStorageTRES.

`AcctGatherFilesystemType` ✨

This identifies the plugin used for filesystem traffic accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect filesystem traffic data for jobs and nodes. The collection of filesystem traffic data takes place on the node level, hence only in case of exclusive job allocation the collected values will reflect the job's real traffic. In case of node sharing between jobs, the reported filesystem traffic per job (through sstat or sacct) will not reflect the real filesystem traffic by the jobs. 💖

Configurable values at present are:

acct_gather_filesystem/lustre — Lustre filesystem traffic data are collected from the counters found in /proc/fs/lustre/. In order to account for per job lustre traffic, add the fs/lustre TRES to AccountingStorageTRES.

`AcctGatherProfileType` 🌈

This identifies the plugin used for detailed job profiling. The jobacct_gather plugin and slurmd daemon call this plugin to collect detailed data such as I/O counts, memory usage, or energy consumption for jobs and nodes. There are interfaces in this plugin to collect data as step start and completion, task start and completion, and at the account gather frequency. The data collected at the node level is related to jobs only in case of exclusive job allocation. 🧚

Configurable values at present are:

acct_gather_profile/hdf5 — This enables the HDF5 plugin. The directory where the profile files are stored and which values are collected are configured in the acct_gather.conf file.
acct_gather_profile/influxdb — This enables the influxdb plugin. The influxdb instance host, port, database, retention policy and which values are collected are configured in the acct_gather.conf file.

`AllowSpecResourcesUsage` 🔥

If set to "YES", Slurm allows individual jobs to override node's configured CoreSpecCount value. For a job to take advantage of this feature, a command line option of --core-spec must be specified. The default value for this option is "YES" for Cray systems and "NO" for other system types. 💅

`AuthAltTypes` 🎀

Comma-separated list of alternative authentication plugins that the slurmctld will permit for communication. Acceptable values at present include auth/jwt. 👑

NOTE: If AuthAltParameters is not used to specify a path to the required jwt_hs256.key then slurmctld will default to looking for it in the StateSaveLocation. The jwt_hs256.key should only be visible to the SlurmUser and root. It is not suggested to place the jwt_hs256.key on any nodes other than the machine running slurmctld and the machine running slurmdbd. auth/jwt can be activated by the presence of the SLURM_JWT environment variable. When activated, it will override the default AuthType. 🫶

`AuthAltParameters` 💜

Used to define alternative authentication plugins options. Multiple options may be comma separated.

disable_token_creation — Disable scontrol token use by non-SlurmUser accounts.
max_token_lifespan=<seconds> — Set max lifespan (in seconds) for any token generated for user accounts. Limit applies to all users except SlurmUser. Sites wishing to have per user limits should generate tokens using JWT-compatible tools, andor an authenticating proxy, instead of using scontrol token.
jwks= — Absolute path to JWKS file. Key should be owned by SlurmUser or root, must be readable by SlurmUser, with suggested permissions of 0400. It must not be writable by 'other'. Only RS256 keys are supported, although other key types may be listed in the file. If set, no HS256 key will be loaded by default (and token generation is disabled), although the jwt_key setting may be used to explicitly re-enable HS256 key use (and token generation).
jwt_key= — Absolute path to JWT key file. Key must be HS256. Key should be owned by SlurmUser or root, must be readable by SlurmUser, with suggested permissions of 0400. It must not be accessible by 'other'. If not set, the default key file is jwt_hs256.key in StateSaveLocation.
use_jwt_client_ids — Allow JWT tokens to contain full identity information. When enabled, JWT tokens must contain uid and gid fields. They must also include a id claim with a JSON object that includes gecos, dir, shell, and either gids (array) or groups (dict) elements to provide complete user identity information. This bypasses slurmctld system lookups for user information when valid claims are present in the JWT. (For auth/jwt.)
use_jwt_client_ids_only — When this flag is set, auth/jwt authentication will fail if complete identity information is not available from JWT claims. This option implies use_jwt_client_ids. (For auth/jwt.)
userclaimfield= — use an alternative claim field for the Slurm UserName (sun) field. This option is designed to allow compatibility with tokens generated outside of Slurm. (This field may also be known as a grant.) Default: (disabled) 💅

🫶 `AuthInfo` — extra auth deets for daemon/client chatter

Additional information to be used for authentication of communications between the Slurm daemons (slurmctld and slurmd) and the Slurm clients. The interpretation of this option is specific to the configured AuthType — no cap. Multiple options may be specified in a comma-delimited list. If not specified, the default authentication information will be used.

cred_expire — default job step credential lifetime, in seconds (e.g. cred_expire=1200). It must be sufficiently long enough to load user environment, run prolog, deal with the slurmd getting paged out of memory, etc. This also controls how long a requeued job must wait before starting again. The default value is 120 seconds.
socket — path name to a MUNGE daemon socket to use (e.g. socket=/var/run/munge/munge.socket.2). The default value is /var/run/munge/munge.socket.2. Used by auth/munge and cred/munge.
ttl — credential lifetime, in seconds (e.g. ttl=300). The default value is dependent on the AuthType used. For auth/munge, the default value is dependent upon the MUNGE installation, but is typically 300 seconds. For auth/slurm, the default value is 60 seconds. For auth/jwt, the default value is 1800 seconds.
use_client_ids — allow the auth/slurm plugin to authenticate users without relying on the user information from LDAP or the operating system. When coupled with nss_slurm, the user information can be managed on the compute nodes by slurmstepd. This would allow the cluster to operate in an environment where only the login nodes have access to LDAP/OS user information. See https://slurm.schedmd.com/nss_slurm.html for more information. NOTE: Some Slurm capabilities require user authentication but are not covered by use_client_ids, including:
- Partition configuration with AllowGroups
- Reservation creation with Users
- The AllowUserBoot parameter in helpers.conf(5)

🔥 `AuthType` — the auth method for Slurm components

The authentication method for communications between Slurm components. All Slurm daemons and commands must be terminated prior to changing the value of AuthType and later restarted. Changes to this value will interrupt outstanding job steps and prevent them from completing, so no jobs should be running when this is changed. See https://slurm.schedmd.com/authentication.html for more information.

Acceptable values at present:

auth/munge — indicates that MUNGE is to be used (default). (See https://dun.github.io/munge/ for more information).
auth/slurm — use Slurm's internal authentication plugin.

⏰ `BatchStartTimeout` — the launch window, bestie

The maximum time (in seconds) that a batch job is permitted for launching before being considered missing and releasing the allocation. The default value is 10 (seconds). Larger values may be required if more time is required to execute the Prolog, load user environment variables, or if the slurmd daemon gets paged from memory.

NOTE: The test for a job being successfully launched is only performed when the Slurm daemon on the compute node registers state with the slurmctld daemon on the head node, which happens fairly rarely. Therefore a job will not necessarily be terminated if its start time exceeds BatchStartTimeout. This configuration parameter is also applied to launch tasks and avoid aborting srun commands due to long running Prolog scripts.

🦋 `BcastExclude` — dirs to skip when broadcasting

Comma-separated list of absolute directory paths to be excluded when autodetecting and broadcasting executable shared object dependencies through sbcast or srun --bcast. The keyword "none" can be used to indicate that no directory paths should be excluded. The default value is "/lib,/usr/lib,/lib64,/usr/lib64". This option can be overridden by sbcast --exclude and srun --bcast-exclude.

🎀 `BcastParameters` — `sbcast` / `srun --bcast` vibes

Controls sbcast and srun --bcast behavior. Multiple options can be specified in a comma separated list. Supported values include:

DestDir= — destination directory for file being broadcast to allocated compute nodes. Default value is current working directory, or --chdir for srun if set.
Compression= — specify default file compression library to be used. Supported values are "lz4" and "none". The default value with the sbcast --compress option is "lz4" and "none" otherwise. Some compression libraries may be unavailable on some systems.
send_libs — if set, attempt to autodetect and broadcast the executable's shared object dependencies to allocated compute nodes. The files are placed in a directory alongside the executable. For srun only, the LD_LIBRARY_PATH is automatically updated to include this cache directory as well. This can be overridden with either sbcast or srun --send-libs option. By default this is disabled.

💜 `BurstBufferType` — burst buffer plugin energy

The plugin used to manage burst buffers. Unset by default. Acceptable values at present are:

burst_buffer/datawarp — use Cray DataWarp API to provide burst buffer functionality.
burst_buffer/lua — this plugin provides hooks to an API that is defined by a Lua script. This plugin was developed to provide system administrators with a way to do any task (not only file staging) at different points in a job's life cycle.

🧚 `CertgenParameters` — certgen plugin options

Comma-separated options identifying certgen plugin options. Supported values include:

certgen_script= — absolute path to executable script to generate self-signed TLS certificate. The private key generated by keygen_script is passed in as stdin, and only the certificate PEM file should be printed to stdout. Must return 0 on success, and non-zero on error.
keygen_script= — absolute path to executable script to generate private key used later to generate a self-signed certificate. Only the private key PEM file should be printed to stdout, which will be later sent as stdin to certgen_script. Must return 0 on success, and non-zero on error.

⭐ `CertgenType` — which certgen plugin, exactly

Specify the certgen plugin that will be used. Acceptable values at present:

certgen/script — use built-in/configured scripts to generate certificate key pair.

🌈 `CertmgrParameters` — certmgr plugin params

Used to define parameters for certmgr plugin.

certificate_renewal_period= — slurmd/sackd will request a new signed certificate from slurmctld at this specified interval (in minutes). Default is 1440 minutes (once per day).
generate_csr_script= — path to script used to generate certificate signing requests. The nodename is passed in as an argument to the script. The script must print only the certificate signing request PEM file to stdout, and return 0 on success. Must return non-zero on error. Required with certmgr/script. Only run by daemons requesting certificates.
get_node_cert_key_script= — path to script used to get node's private key which was used to generate the CSR returned by generate_csr_script. The nodename is passed in as an argument to the script. The script must print the node's private key (PEM file) to stdout. Must return 0 on success, and non-zero on error. Required with certmgr/script. Only run by daemons requesting certificates.
get_node_token_script= — path to script used to get node's unique token which will be validated by slurmctld using the script set by validate_node_script=. The nodename is passed in as an argument to the script. The script must print the node's unique token to stdout, and return 0 on success. Must return non-zero on error. Required with certmgr/script. Only run by daemons requesting certificates.
sign_csr_script= — path to script used to sign incoming certificate signing requests. This script will only be called if validate_node_script= was already called on the accompanying unique node token and returned with a non-zero exit code. The certificate signing request (as given by generate_csr_script=) is passed as an argument to this script. The script must print the new signed certificate to stdout, and return 0 on success. Must return non-zero on error. Required with certmgr/script. Only run by slurmctld.
single_use_tokens — unique node tokens that are dynamically set (e.g. set via scontrol) will be consumed upon successful certificate signing.
validate_node_script= — the path to the script that validates a unique node token, bestie. The token gets passed in as an argument to this script. If the script says the token is valid, it returns 0; if the token is invalid, it returns non-zero. Required when you're running certmgr/script, and it's only run by slurmctld. 🎀

💖 `CertmgrType`

The plugin used to dynamically renew TLS certificates for slurmd/sackd. Right now the acceptable value is:

certmgr/script — uses script hooks to do the certificate management thing. Peep CertmgrParameters for the deets on how to set those scripts up. ✨

🌸 `CliFilterParameters`

Extra parameters for cli_filter plugins, babes. Multiple options can be comma-separated. Acceptable values at present are:

cli_filter_lua_path=<path> — the absolute path to the cli_filter.lua script used when cli_filter/lua is enabled. If you don't define this, the default path gets used instead (same path as slurm.conf).

NOTE: the configured directory holding cli_filter.lua should have 755 permissions, the script itself 644, and both should be owned by SlurmdUser. Not me reminding you twice, but like… also reminding you twice. 💅

🦋 `CliFilterPlugins`

A comma-delimited list of command line interface option filter/modification plugins, and they execute in the order you list them. No cli_filter plugins are used by default. Acceptable values at present are:

cli_filter/lua — lets you write your own implementation of a cli_filter using lua. Iconic, honestly.
cli_filter/syslog — enables logging of the job submission activities being performed. All the salloc/sbatch/srun options get logged to syslog along with environment variables in JSON format. ⚠️ If this plugin is not the last one in the list, it may log values different than what was actually sent to slurmctld.
cli_filter/user_defaults — looks for the file $HOME/.slurm/defaults and reads every line of it as a key=value pair, where key is any of the job submission options available to salloc/sbatch/srun and value is a default value defined by the user. For instance:

time=1:30
mem=2048

The above gives the user a default for each of their jobs of -t 1:30 and --mem=2048. Slay. 🔥

⭐ `ClusterName`

The name by which this Slurm managed cluster is known in the accounting database. You need this to distinguish accounting records when multiple clusters report to the same database. Because some databases have limitations, any upper case letters in the name will be silently mapped to lower case — so to avoid confusion, it's recommended the name be lower case. Also the cluster name must be 40 characters or less to comply with the limit on the maximum length for table names in MySQL/MariaDB. 💀

🫶 `CommunicationParameters`

Comma-separated options identifying communication options. Let's go through them:

block_null_hash — requires all Slurm authentication tokens to include a newer (20.11.9 and 21.08.8) payload that provides an additional layer of security against credential replay attacks. Only enable this once all Slurm daemons have been upgraded to 20.11.9/21.08.8 or newer, and all jobs that were started before the upgrade have been completed. No shortcuts, bestie.
disable_http — prevents slurmctld and slurmd from responding to incoming HTTP requests.
host_unreach_retry_count= — when a node tries to connect() to another node, connect() may return an error with EHOSTUNREACH if the host is unreachable. If this parameter is set, this is the number of times Slurm will retry making that connection. Slurm waits 500 milliseconds in between each try. The default for this parameter is zero (Slurm will not retry if EHOSTUNREACH is returned).
DisableIPv4 — disable IPv4 only operation for all slurm daemons (except slurmdbd). This should also be set in your slurmdbd.conf file.
EnableIPv6 — enable using IPv6 addresses for all slurm daemons (except slurmdbd). When using both IPv4 and IPv6, address family preferences are based on your /etc/gai.conf file. This should also be set in your slurmdbd.conf file.
getnameinfo_cache_timeout — when munge is used as AuthType, slurmctld makes use of getnameinfo to obtain the hostname from the IP address stored in the munge credential. This parameter controls the number of seconds slurmctld should keep that IP-to-hostname resolution cached. Set to 0 and the cache is disabled. The default value is 60. ✨
keepaliveinterval= — specifies the interval, in seconds, between keepalive probes on idle connections. This affects connections between srun and its slurmstepd process as well as all connections to the slurmdbd. The default is to use the system default settings.
keepaliveprobes= — specifies the number of unacknowledged keepalive probes sent before considering the connection broken. This affects connections between srun and its slurmstepd process as well as all connections to the slurmdbd. The default is to use the system default settings.
keepalivetime= — specifies how long, in seconds, before a connection is marked as needing a keepalive probe, as well as how long to delay closing a connection to process messages still in the queue. This affects connections between srun and its slurmstepd process as well as all connections to the slurmdbd. Longer values can be used to improve reliability of communications in the event of network failures. The default is for keepalive to be disabled.
NoCtldInAddrAny — used to directly bind to the address of what the node resolves to running the slurmctld, instead of binding messages to any address on the node, which is the default.
NoInAddrAny — used to directly bind to the address of what the node resolves to instead of binding messages to any address on the node, which is the default. This option is for all daemons/clients except for the slurmctld.

🧚 `CompleteWait`

The time to wait, in seconds, when any job in the cluster is in the COMPLETING state before any additional jobs are scheduled. This parameter is evaluated at the cluster scheduling level, not per node — it's trying to keep jobs on nodes that were recently in use, with the goal of preventing fragmentation. Set it to zero and pending jobs will be started as soon as possible. Since a COMPLETING job's resources are released for use by other jobs as soon as the Epilog completes on each individual node, you can end up with very fragmented resource allocations. 😭

To give jobs the minimum response time, a value of zero is recommended (no waiting). To minimize fragmentation of resources, a value equal to KillWait plus two is recommended — and in that case, setting KillWait to a small value may be beneficial. The default value of CompleteWait is zero seconds, and the value may not exceed 65533.

This parameter does not override the fact that nodes in the COMPLETING state are not eligible for job scheduling. Nodes will not accept new job placement until their Epilog completes and they leave the COMPLETING state, even if CompleteWait is set to zero. Periodt.

NOTE: setting reduce_completing_frag affects the behavior of CompleteWait.

🌈 `CpuFreqDef`

The default CPU governor to use when running a job step if it hasn't been explicitly set with the --cpu-freq option. Acceptable values at present include one of the following governors:

Conservative — attempts to use the Conservative CPU governor
OnDemand — attempts to use the OnDemand CPU governor
Performance — attempts to use the Performance CPU governor
PowerSave — attempts to use the PowerSave CPU governor

Default: use the system default. No attempt to set the governor is made if the --cpu-freq option has not been specified. 💅

🍓 `CpuFreqGovernors`

The list of CPU frequency governors allowed to be set with the salloc, sbatch, or srun option --cpu-freq. Acceptable values at present include:

Conservative — attempts to use the Conservative CPU governor
OnDemand — attempts to use the OnDemand CPU governor (a default value)
Performance — attempts to use the Performance CPU governor (a default value)
PowerSave — attempts to use the PowerSave CPU governor
SchedUtil — attempts to use the SchedUtil CPU governor
UserSpace — attempts to use the UserSpace CPU governor (a default value)

Default: OnDemand, Performance and UserSpace.

💜 `CredType`

The cryptographic signature tool used in the creation of job step credentials. Acceptable values at present are:

cred/munge — indicates that Munge is to be used (the default).
cred/slurm — use Slurm's internal credential format.

`DataParserParameters=<data_parser>`

Okay bestie, this sets the default value for data_parser plugin parameters 💅. Peep the --json or --yaml arguments in sacct(1), scontrol(1), sinfo(1), squeue(1), sacctmgr(1), sdiag(1), and sshare(1) ✨.

Default: Latest data_parser plugin version with no flags selected.

`DebugFlags`

This one defines specific subsystems that should give you more detailed event logging 🔥. Multiple subsystems can be specified with comma separators. Most DebugFlags will result in additional logging messages for the identified subsystems if SlurmctldDebug is at 'verbose' or higher. More logging may impact performance — so yeah, it can get loud 👀.

NOTE: You can also set debug flags by having the SLURM_DEBUG_FLAGS environment variable defined with the desired flags when the process (client command, daemon, etc.) is started. The environment variable takes precedence over the setting in the slurm.conf.

Valid subsystems available include:

Accrue — Accrue counters accounting details
Agent — RPC agents (outgoing RPCs from Slurm daemons)
AuditRPCs — For all inbound RPCs to slurmctld, print the originating address, authenticated user, and RPC type before the connection is processed.
AuditTLS — Print TLS certificates being used
Backfill — Backfill scheduler details
BackfillMap — Backfill scheduler to log a very verbose map of reserved resources through time. Combine with Backfill for a verbose and complete view of the backfill scheduler's work.
BurstBuffer — Burst Buffer plugin
Cgroup — Cgroup details
ConMgr — Connection manager details
CPU_Bind — CPU binding details for jobs and steps
CpuFrequency — Cpu frequency details for jobs and steps using the --cpu-freq option.
Data — Generic data structure details.
DBD_Agent — RPC agent (outgoing RPCs to the DBD)
Dependency — Job dependency debug info
Elasticsearch — Elasticsearch debug info (deprecated). Alias of JobComp.
Energy — AcctGatherEnergy debug info
Federation — Federation scheduling debug info
Gres — Generic resource details
Hetjob — Heterogeneous job details
Gang — Gang scheduling details
GLOB_SILENCE — Do not display error message of glob "*" symbols in conf files.
JobAccountGather — Common job account gathering details (not plugin specific).
JobComp — Job Completion plugin details
License — License management details
Namespace — Namespace plugin details
Metrics — Metrics plugin details
Network — Network details. Warning: activating this flag may cause logging of passwords, tokens or other authentication credentials.
NetworkRaw — Dump raw hex values of key Network communications. Warning: This flag will cause very verbose logs and may cause logging of passwords, tokens or other authentication credentials.
NodeFeatures — Node Features plugin debug info
NO_CONF_HASH — Do not log when the slurm.conf files differ between Slurm daemons
Power — Power management plugin and power save (suspend/resume programs) details
Priority — Job prioritization
Profile — AcctGatherProfile plugins details
Protocol — Communication protocol details
Reservation — Advanced reservations
Route — Message forwarding debug info
Script — Debug info regarding any script called by Slurm. This includes slurmctld executed scripts such as PrologSlurmctld and EpilogSlurmctld.
SelectType — Resource selection plugin
Steps — Slurmctld resource allocation for job steps
Switch — Switch plugin
thread — Thread and work management.
TLS — TLS plugin
TraceJobs — Trace jobs in slurmctld. It will print detailed job information including state, job ids and allocated nodes counter.
Triggers — Slurmctld triggers

`DefCpuPerGPU`

Default count of CPUs allocated per allocated GPU 🎀. This value is used only if the job didn't specify --cpus-per-task and --cpus-per-gpu.

`DefMemPerCPU`

Default real memory size available per usable allocated CPU in mebibytes 🧚. Used to avoid over-subscribing memory and causing paging. DefMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_tres). The default value is 0 (unlimited). Also see DefMemPerGPU, DefMemPerNode and MaxMemPerCPU. DefMemPerCPU, DefMemPerGPU and DefMemPerNode are mutually exclusive 💖.

NOTE: This applies to usable allocated CPUs in a job allocation. This is important when more than one thread per core is configured. If a job requests --threads-per-core with fewer threads on a core than exist on the core (or --hint=nomultithread which implies --threads-per-core=1), the job will be unable to use those extra threads on the core and those threads will not be included in the memory per CPU calculation. But if the job has access to all threads on the core, those threads will be included in the memory per CPU calculation even if the job did not explicitly request those threads.

In the following examples, each core has two threads.

In this first example, two tasks can run on separate hyperthreads in the same core because --threads-per-core is not used. The third task uses both threads of the second core. The allocated memory per cpu includes all threads:

$ salloc -n3 --mem-per-cpu=100
salloc: Granted job allocation 17199
$ sacct -j $SLURM_JOB_ID -X -o jobid%7,reqtres%35,alloctres%35
  JobID                             ReqTRES                           AllocTRES
------- ----------------------------------- -----------------------------------
  17199     billing=3,cpu=3,mem=300M,node=1     billing=4,cpu=4,mem=400M,node=1

In this second example, because of --threads-per-core=1, each task is allocated an entire core but is only able to use one thread per core. Allocated CPUs includes all threads on each core. However, allocated memory per cpu includes only the usable thread in each core.

$ salloc -n3 --mem-per-cpu=100 --threads-per-core=1
salloc: Granted job allocation 17200
$ sacct -j $SLURM_JOB_ID -X -o jobid%7,reqtres%35,alloctres%35
  JobID                             ReqTRES                           AllocTRES
------- ----------------------------------- -----------------------------------
  17200     billing=3,cpu=3,mem=300M,node=1     billing=6,cpu=6,mem=300M,node=1

`DefMemPerGPU`

Default real memory size available per allocated GPU in mebibytes 🌸. The default value is 0 (unlimited). Please note a best effort attempt is made to predict which GPUs on the system will be used, but this could change between job submission and start time, causing MaxMemPerNode to potentially not work as expected for heterogeneous jobs. Also see DefMemPerCPU and DefMemPerNode. DefMemPerCPU, DefMemPerGPU and DefMemPerNode are mutually exclusive.

`DefMemPerNode`

Default real memory size available per allocated node in mebibytes 🦋. Used to avoid over-subscribing memory and causing paging. DefMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). The default value is 0 (unlimited). Also see DefMemPerCPU, DefMemPerGPU and MaxMemPerCPU. DefMemPerCPU, DefMemPerGPU and DefMemPerNode are mutually exclusive.

`DependencyParameters`

Multiple options may be comma separated 👑

disable_remote_singleton — By default, when a federated job has a singleton dependency, each cluster in the federation must clear the singleton dependency before the job's singleton dependency is considered satisfied. Enabling this option means that only the origin cluster must clear the singleton dependency. This option must be set in every cluster in the federation.
kill_invalid_depend — If a job has an invalid dependency and it can never run terminate it and set its state to be JOB_CANCELLED. By default the job stays pending with reason DependencyNeverSatisfied.
max_depend_depth= — Maximum number of jobs to test for a circular job dependency. Stop testing after this number of job dependencies have been tested. The default value is 10 jobs.

`DisableRootJobs`

If set to "YES" then user root will be prevented from running any jobs 🙈. The default value is "NO", meaning user root will be able to execute jobs. DisableRootJobs may also be set by partition.

`EioTimeout`

The number of seconds srun waits for slurmstepd to close the TCP/IP connection used to relay data between the user application and srun when the user application terminates 💀. The default value is 60 seconds. May not exceed 65533.

`EnforcePartLimits`

Controls whether partition limits are enforced when a job is submitted to the cluster ⭐. The partition limits being considered by this option are its configured MaxMemPerCPU, MaxMemPerNode, MinNodes, MaxNodes, MaxTime, AllocNodes, AllowAccounts, AllowGroups, AllowQOS, and QOS usage threshold. It also considers if the job requests more nodes than exist in the partition. If set, then a job and job QOS cannot be submitted that exceed partition limits.

ALL — jobs that go over the node count in a partition and/or any of its configured limits get rejected right at submission time. And if a job is submitted to multiple partitions, it's gotta satisfy the limits on all the requested partitions. no partial credit 💅
ANY — jobs are accepted if they satisfy the limits on at least one of the requested partitions.
NO — partition limits aren't enforced at submit time, but they're still absolutely enforced during scheduling. This includes jobs that ask for more nodes than exist in any of the partitions, so yes, you can submit jobs to empty partitions (iconic chaos, we love it). A job that exceeds the limits on all requested partitions will just sit there queued until the partition limits get changed. This one is the default.

Epilog — the pathname of a program to execute as user root on every node when a user's job completes (think "/usr/local/slurm/epilog"). If it's not an absolute pathname (i.e. it doesn't start with a slash), it has to be relative to the directory that holds the slurm.conf file. You can also use a glob pattern (see glob(7)) to run more than one epilog script (e.g. "/etc/slurm/epilog.d/*"). When more than one epilog script is configured, they run in reverse alphabetical order (z-a -> Z-A -> 9-0). Your Epilog script(s) can purge files, disable user login, whatever you need. By default there's no epilog. See Prolog and Epilog Scripts for more info.

While the Epilog script is running on a node, that node gets placed in the COMPLETING state. Nodes in COMPLETING do not accept new job allocations, and no jobs will be scheduled on them until the Epilog finishes and the node returns to an available state. so don't be surprised when your node is temporarily busy being dramatic 🎭

NOTE: It is possible to configure multiple epilog scripts by including this option on multiple lines.

EpilogMsgTime — the number of microseconds that the slurmctld daemon needs to process an epilog completion message from the slurmd daemons. For jobs spanning more than 64 nodes, each host delays its message based on its index in the job's node list and this value — which prevents a burst of simultaneous RPCs that could result in lost messages (we're anti-chaos here ✨). The default value is 2000 microseconds. The maximum value is 1000000 (1 second). Values exceeding the maximum get reset to the default. For a 1000 node job, the default value spreads the epilog completion messages out over two seconds. math, but make it scheduling 🧮

EpilogSlurmctld — fully qualified pathname of a program for slurmctld to execute upon termination of a job allocation (e.g. "/usr/local/slurm/epilog_controller"). The program executes as SlurmUser, which gives it permission to drain nodes and requeue the job if a failure occurs (See scontrol(1)). Exactly what the program does and how it accomplishes it is completely up to the system administrator — main character energy, your call entirely 👑. Info about the job being initiated, its allocated nodes, etc. gets passed to the program using environment variables. See Prolog and Epilog Scripts for more info.

NOTE: It is possible to configure multiple epilog scripts by including this option on multiple lines.

EpilogTimeout — the interval in seconds Slurm waits for Epilog before terminating them. The default value is PrologEpilogTimeout. This interval applies to the Epilog run by the slurmd daemon after the job, the EpilogSlurmctld run by the slurmctld daemon, and the SPANK plugin epilog call: slurm_spank_job_epilog. If the Epilog or slurm_spank_job_epilog time out, the node is drained. In all cases, errors are logged. 😭

FairShareDampeningFactor — dampens the effect of exceeding a user's or group's fair share of allocated resources. Higher values give you a greater ability to differentiate between exceeding the fair share at high levels (e.g. a value of 1 results in almost no difference between overconsumption by a factor of 10 and 100, while a value of 5 will result in a significant difference in priority). The default value is 1. tune responsibly 🎀

FederationParameters — used to define federation options. Multiple options may be comma separated.

fed_display — if set, then the client status commands (e.g. squeue, sinfo, sprio, etc.) will display information in a federated view by default. This option is functionally equivalent to using the --federation options on each command. Use the client's --local option to override the federated view and get a local view of the given cluster.

Allow client commands to use the --cluster option even when the slurmdbd is down by retrieving cluster records from slurmctld instead.

FirstJobId — the job id to be used for the first job submitted to Slurm. Job id values generated will be incremented by 1 for each subsequent job. Value must be larger than 0. The default value is 1. Also see MaxJobId. counting, but make it ✨official✨

GresTypes — a comma-delimited list of generic resources to be managed (e.g. GresTypes=gpu,mps). These resources may have an associated GRES plugin of the same name providing additional functionality. No generic resources are managed by default. Ensure this parameter is consistent across all nodes in the cluster for proper operation — inconsistency here is giving broken vibes fr fr 💀

GroupUpdateForce — if set to a non-zero value, then info about which users are members of groups allowed to use a partition will be updated periodically, even when there have been no changes to the /etc/group file. If set to zero, group member info will be updated only after the /etc/group file is updated. The default value is 1. Also see the GroupUpdateTime parameter. 🫶

GroupUpdateTime — controls how frequently info about which users are members of groups allowed to use a partition gets updated, and how long user group membership lists are cached. The time interval is given in seconds with a default value of 600 seconds. A value of zero will prevent periodic updating of group membership info. Also see the GroupUpdateForce parameter.

GpuFreqDef=[<type]=value>[,<type=value>] — default GPU frequency to use when running a job step if it hasn't been explicitly set using the --gpu-freq option. This option can be used to independently configure the GPU and its memory frequencies. There is no default value. If unset, no attempt to change the GPU frequency is made if the --gpu-freq option has not been set. After the job is completed, the frequencies of all affected GPUs get reset to the highest possible values. In some cases, system power caps may override the requested values. The field type can be "memory". If type is not specified, the GPU frequency is implied. The value field can either be "low", "medium", "high", "highm1" or a numeric value in megahertz (MHz). If the specified numeric value isn't possible, a value as close as possible will be used. See below for the definition of the values. Examples of use include "GpuFreqDef=medium,memory=high" and "GpuFreqDef=450". 🦋

Supported value definitions:

low — the lowest available frequency.
medium — attempts to set a frequency in the middle of the available range.
high — the highest available frequency.
highm1 — (high minus one) will select the next highest available frequency.

HashPlugin — identifies the type of hash plugin to use for network communication. Acceptable values include:

hash/k12 — hashes are generated by the KangorooTwelve cryptographic hash function. This is the default.
hash/sha3 — hashes are generated by the SHA-3 cryptographic hash function.

NOTE: Make sure that HashPlugin has the same value both in slurm.conf and in slurmdbd.conf. mismatched hashes = bad vibes 😭

HealthCheckInterval — the interval in seconds between executions of HealthCheckProgram. The default value is zero, which disables execution. (zero = we're chilling 🧘)

HealthCheckNodeState — identifies what node states should execute the HealthCheckProgram. Multiple state values may be specified with a comma separator, except for the *_ONLY states. The default value is ANY to execute on nodes in any state.

ALLOC — run on nodes in the ALLOC state (all CPUs allocated).
ANY — run on nodes in any state.
CYCLE — rather than running the health check program on all nodes at the same time, cycle through running on all compute nodes through the course of the HealthCheckInterval. May be combined with the various node state options.
IDLE — run on nodes in the IDLE state.
NONDRAINED_IDLE — run on nodes that are in the IDLE state and not DRAINED.
MIXED — run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).
REBOOT_ONLY — run only after a node reboot. This state is mutually exclusive with any other state.
START_ONLY — run only at slurmd startup. This state is mutually exclusive with any other state. 🎬

`HealthCheckProgram` 💖

okay so this one is a fully qualified pathname to a script that gets run as user root, periodically, on every compute node that is not in the NOT_RESPONDING state. slay. what's it for? you can use it to check that a node is genuinely fully operational, and then DRAIN the node or send an email if a problem shows up.

but here's the vibe check: every action has to be explicitly performed by the program itself. slurm is not gonna do it for you, bestie. so like, if you want to drain a node, your script literally has to run:

scontrol update NodeName=foo State=drain Reason=tmp_file_system_full

how often does it run? that's the HealthCheckInterval parameter's whole job. and ngl the way this is timed is kind of iconic — HealthCheckProgram runs at the same time on all nodes so it doesn't mess with your parallel programs. considerate queen behavior 💅

it gets killed if it doesn't terminate normally within 60 seconds
it also runs when the slurmd daemon is first started, before it registers with the slurmctld daemon
if HealthCheckNodeState is START_ONLY or REBOOT_ONLY, it runs only when slurmd is first started, or when slurmd reports back after a reboot
default? no program will be executed. none. nada. 🫶

`HealthCheckTimeout` ⏱️

how many seconds is HealthCheckProgram allowed to run? that's it, that's the parameter. default value is 60.

two things you need to know:

HealthCheckTimeout should be less than HealthCheckInterval — otherwise an error gets logged 😭
a value of 0 is not allowed and gets reset to the default
the maximum value is 65535

so like, don't try to be cute with the zero. it's not gonna work.

`HttpParserType` 🌐

which http_parser implementation are we using, bestie? default is http_parser/libhttp_parser if it's found, otherwise http_parser/llhttp_parser. acceptable values at present:

http_parser/libhttp_parser — use the libhttp_parser based plugin
http_parser/llhttp_parser — use the llhttp based plugin

that's it! pick your parser. 🦋

`InactiveLimit` 💀

the interval, in seconds, after which a non-responsive job allocation command (think srun or salloc) results in the job being terminated. like, if the node where the command is running fails, or the command abnormally terminates, this will terminate its job allocation. no cap.

some important details so you don't get caught out:

this option has no effect upon batch jobs
when you're picking a value, remember that a debugger using srun to launch an application may leave the srun command in a stopped state for extended periods of time 🙈
this limit is ignored for jobs running in partitions with the RootOnly flag set (the scheduler running as root will be responsible for the job)
default value is unlimited (0) and may not exceed 65533 seconds

`InteractiveStepOptions` 🎀

when LaunchParameters=use_interactive_step is enabled, launching salloc will automatically start an srun process with InteractiveStepOptions to launch a terminal on a node in the job allocation. cute, right?

the default value is:

--interactive --preserve-env --pty $SHELL

okay so about that --interactive flag — it is intentionally not documented in the srun man page. it's meant only to be used in InteractiveStepOptions, to create an "interactive step" that won't consume resources, so that other steps may run in parallel with the interactive step. iykyk 🤫

`JobAcctGatherType` 📊

the JobAcctGather plugin collects memory, cpu, io, interconnect, energy and gpu usage info at the task level, depending on which plugins are configured in Slurm. this parameter controls how some of these metrics get collected. unset by default.

configurable values at present are:

jobacct_gather/cgroup (recommended) ✨ — collect cpu and memory statistics by reading the task's cgroup directory interfaces (e.g. memory.stat, cpu.stat) by issuing a call to the configured CgroupPlugin (see "man cgroup.conf"). this mechanism ignores JobAcctGatherParams=UsePSS or NoShared, since those are used only when reading memory usage from the proc filesystem.
jobacct_gather/linux 🐧 — collect cpu and memory statistics by reading procfs. the plugin takes all the pids of the task and, for each of them, reads /proc/<pid>/stats. if UsePSS is set it will also read /proc/<pid>/smaps, and if NoShare is set it will also read /proc/<pid>/statm (see JobAcctGatherParams for more info).

lowkey important: this plugin carries a performance penalty on jobs with a large number of spawned processes, since it needs to iterate over all the task pids and aggregate the stats into one single metric for the ppid, and then those values need to be aggregated to the task stats. it's giving a lot of work 😮‍💨

NOTE: changing the plugin type when jobs are running in the cluster is possible. the already running steps will keep using the previous plugin mechanism, while new steps will use the new mechanism. so no, nothing explodes. 💜

`JobAcctGatherFrequency` ⏳

the job accounting and profiling sampling intervals, specified for each data type. you can specify multiple comma-separated <datatype>=<interval> intervals. and if you provide an interval without a datatype, it gets assigned to the task datatype. iconic shortcut behavior.

supported datatypes are as follows:

Affects accounting and profiling:

task=<interval> — sampling interval in seconds for task usage by the jobacct_gather plugins, and for task profiling by the acct_gather_profile plugin. defaults to 30.

if this interval is 0 (disabled), accounting information is collected only at job termination, which reduces Slurm interference with the job — but it also means the statistics about a job are only derived from a single sample and don't reflect the average or maximum of several samples throughout the life of the job. 😭 so like, pick your fighter.

Affects profiling only:

energy=<interval> — sampling interval in seconds for energy profiling using the acct_gather_energy plugin. defaults to 0 (disabled).
network=<interval> — sampling interval in seconds for infiniband profiling using the acct_gather_interconnect plugin. defaults to 0 (disabled).
filesystem=<interval> — sampling interval in seconds for filesystem profiling using the acct_gather_filesystem plugin. defaults to 0 (disabled).

smaller (non-zero) values have a greater impact upon job performance, but a value of 30 seconds is not likely to be noticeable for applications having less than 10,000 tasks. and good news — users can independently override each interval on a per-job basis using the --acctg-freq option when submitting the job. 🫶

one more thing, and it's a real one: this value should be lower or equal to EnergyIPMIFreq when using acct_gather_energy/ipmi or xcc plugins, because otherwise it will unnecessarily get repeated values on successive polls. no waste, bestie. ⭐

`JobAcctGatherParams` 🎛️

arbitrary parameters for the job account gather plugin. acceptable values at present include:

DisableGPUAcct — do not do accounting of GPU usage, and skip any gpu driver library call. this parameter can help improve performance if the GPU driver response is slow. we love a speed-up 💅
no_file_cache — filesystem-backed memory (active_file and inactive_file) will be subtracted from the reported memory. this disables the use of the memory.peak interface, which can result in MaxRSS failing to record short memory spikes. only compatible with cgroup/v2 plugin.
NoShared — exclude shared memory from RSS. this option cannot be used with UsePSS. only compatible with jobacct_gather/linux plugin.
OverMemoryKill — kill processes that are being detected to be using more memory than requested by steps, every time accounting information is gathered by the JobAcctGather plugin. ⚠️ this parameter should be used with caution, because a job exceeding its memory allocation may affect other processes and/or machine health. fr fr, be careful.

NOTE: if available, it is recommended to limit memory by enabling task/cgroup as a TaskPlugin and making use of ConstrainRAMSpace=yes in the cgroup.conf instead of using this JobAcctGather mechanism for memory enforcement. using JobAcctGather is polling based and there is a delay before a job is killed, which could lead to system Out of Memory events. 💀

NOTE: when using OverMemoryKill, if the combined memory used by all the processes in a step exceeds the memory limit, the entire step will be killed/cancelled by the JobAcctGather plugin. this differs from the behavior when using ConstrainRAMSpace, where processes in the step will be killed but the step will be left active — possibly with other processes left running. so yeah, different vibes entirely. 🙈

💾 `UsePss`

Okay bestie, this one tells Slurm to use the PSS value instead of RSS when it's calculating the real usage of memory. The PSS value gets saved as RSS, iykyk. 🫶

🚩 Two things, and they're important: this option cannot be used with NoShared. And it's only compatible with the jobacct_gather/linux plugin. That's it, that's the whole vibe check.

🗄️ `JobCompHost`

The name of the machine hosting the job completion database. That's it, that's the girl.

Only used for database type storage plugins — ignored otherwise, no cap. It can also be a UNIX socket, like this: unix:/path/to/socket. Slay. ✨

📍 `JobCompLoc`

This option sets a string, and the string's whole personality changes depending on JobCompType. Let's break it down, one plugin at a time 💅

If jobcomp/elasticsearch:

Instructs this plugin to send the finished job records information to the Elasticsearch server URL endpoint (including the port number and the target index) configured in this option. This string should typically take the form of <host>:<port>/<target>/_doc. There is no default value for JobCompLoc when this plugin is enabled. 😭

NOTE: Refer to https://slurm.schedmd.com/elasticsearch.html for more information.

If jobcomp/filetxt:

Instructs this plugin to send the finished job records information to a file configured in this option. This string should represent an absolute path to a file. The default value for this plugin is /var/log/slurm_jobcomp.log. 🍓

If jobcomp/kafka:

When this plugin is configured, finished (and optionally start running) job records information is sent to a Kafka server. The plugin makes use of librdkafka. This string represents an absolute path to a file containing key=value pairs configuring the library behavior. For the plugin to work properly, this file needs to exist and at least the bootstrap.servers librdkafka property needs to be configured in it. There is no default value for JobCompLoc when this plugin is enabled. 🔥

NOTE: For a full list of librdkafka properties, please refer to the library documentation. You can also view the jobcomp_kafka page for more information: https://slurm.schedmd.com/jobcomp_kafka.html

NOTE: The target Kafka topic(s) and other plugin parameters can be configured via JobCompParams.

If jobcomp/lua:

This option is ignored in this plugin. The finished job record is processed by a hardcoded jobcomp.lua script expected to be located in the same location of slurm.conf. There is no default value for JobCompLoc when this plugin is enabled. 🙈

If jobcomp/mysql:

Instructs this plugin to send the finished job records information to a database name configured in this option. This string should represent a database name. The default value for this plugin is slurm_jobcomp_db. 💜

If jobcomp/script:

The finished job record information is made available via environment variables and processed by a script with name configured by this option. This string should represent a path to a script. There is no default value for JobCompLoc when this plugin is enabled. It needs to be explicitly configured or the plugin will fail to initialize. 💀

🎛️ `JobCompParams`

Pass an arbitrary text string to the job completion plugin. Also see JobCompType. Main character energy for a config parameter, honestly.

Optional comma-separated list for jobcomp/elasticsearch:

send_comment — Sends the job comment as part of jobcomp messages.
send_script — Sends the job script as part of jobcomp messages.

Optional comma-separated list for jobcomp/kafka:

enable_job_start — Instruct the jobcomp/kafka plugin to send a subset of the job record fields to the topic_job_start Kafka topic when a job first starts running.

NOTE: The writing when the job finishes (historical purpose of the plugin) is always enabled by default and can't be disabled. Periodt.

NOTE: The subset of fields for job start events is slightly smaller than those sent when the job finishes.

flush_timeout=<milliseconds> — Maximum time (in milliseconds) to wait for all outstanding produce requests, et.al, to be completed. This is passed as a timeout argument to the librdkafka flush API function, called on plugin termination. This is done prior to destroying the producer instance to make sure all queued and in-flight produce requests are completed before terminating. For non-blocking calls, set to 0. To wait indefinitely for an event, set to -1 (not recommended, since this is called on plugin fini and could block slurmctld graceful termination 😭). Accepted values are [-1,2147483647]. Defaults to 500 (milliseconds).
poll_interval=<seconds> — Seconds between calls to librdkafka API poll function, which polls the provided Kafka handle for events. The plugin spawns a separate thread to perform this call at the configured interval. Accepted values are [0,4294967295]. Defaults to 2 (seconds). ⭐
requeue_on_msg_timeout — Instruct the delivery report callback to requeue messages that failed delivery because their time waiting for successful delivery reached the librdkafka property message.timeout.ms. Defaults to not set (don't requeue and thus discard these messages). 🗑️
send_comment — Sends the job comment as part of jobcomp messages.
send_script — Sends the job script as part of jobcomp messages.
topic=<string> — Target Kafka topic to send messages to when a job finishes. Defaults to ClusterName.
topic_job_start=<string> — Target Kafka topic to send messages to when a job starts running. Defaults to <ClusterName>-job-start.

NOTE: It is advisable that job start running event records be sent to a different Kafka topic than the topic configured for job finish event records. Trust, separation is the moment. 💅

Optional comma-separated list for jobcomp/mysql:

token_duration — Duration in seconds to cache generated database passwords before requesting a new one from the StoragePassScript. Typically the token should refresh prior to actual expiration; upon token generation failure the cached token will continue to be used to avoid transient generation failures from causing connection failures. Default value is 300 seconds (5 minutes). 🕔

🔑 `JobCompPass`

The password used to gain access to the database to store the job completion data. Only used for database type storage plugins, ignored otherwise. Keep it secret, keep it safe bestie 🫶

🧚 `JobCompPassScript`

Absolute path to an executable script that generates ephemeral authentication tokens for database connections which are used instead of JobCompPass. The script must output the password/token to stdout and exit with status 0 on success. This allows dynamic password generation, instead of storing static credentials in configuration files. The script must be owned and executable by SlurmUser. 🔥

Environment variables provided to the script:

🌸 `SLURM_STORAGE_HOSTNAME`

Database hostname

🌸 `SLURM_STORAGE_PORT`

Database port number

🌸 `SLURM_STORAGE_USER`

Database username

Expected output format:

TOKEN=<authentication_token>

The script must exit with status 0 on success, non-zero on failure. Any output to stderr will be logged as an error. If there is a backup host specified, the script will still be provided the main hostname and the same token is used for both hosts. ✨

🔌 `JobCompPort`

The listening port of the job completion database server. Only used for database type storage plugins, ignored otherwise.

🏷️ `JobCompType`

The job completion logging mechanism type. Unset by default. Acceptable values at present include:

jobcomp/elasticsearch — Upon job completion, a record of the job should be written to an Elasticsearch server, specified by the JobCompLoc parameter.

NOTE: More information is available at the Slurm web site ( https://slurm.schedmd.com/elasticsearch.html ).

jobcomp/filetxt — Upon job completion, a record of the job should be written to a text file, specified by the JobCompLoc parameter.
jobcomp/kafka — Upon job completion (or optionally job start running), a record of the job should be sent to a Kafka server, specified by the file path referenced in JobCompLoc and/or using other JobCompParams.
jobcomp/lua — Upon job completion, a record of the job should be processed by the jobcomp.lua script, located in the default script directory (typically the subdirectory etc of the installation directory).
jobcomp/mysql — Upon job completion, a record of the job should be written to a MySQL or MariaDB database, specified by the JobCompLoc parameter.

jobcomp/script — when a job finishes, a script specified by the JobCompLoc parameter is executed, and it gets environment variables that provide the job information. it’s giving “completion with context” and honestly? iconic. 💖

`JobCompUser` 👑

The user account for accessing the job completion database. Only used for database type storage plugins, ignored otherwise. no cap, if it’s not a database plugin this one literally does nothing.

`JobFileAppend` 🎀

This option controls what to do if a job’s output or error file exist when the job is started. If JobFileAppend is set to a value of 1, then append to the existing file. By default, any existing file is truncated. so like… 1 = keep the vibes going, default = fresh start energy. ✨

`JobRequeue` 🔁

This option controls the default ability for batch jobs to be requeued. Jobs may be requeued explicitly by a system administrator, after node failure, or upon preemption by a higher priority job. If JobRequeue is set to a value of 1, then batch jobs may be requeued unless explicitly disabled by the user. If JobRequeue is set to a value of 0, then batch jobs will not be requeued unless explicitly enabled by the user. Use the sbatch --no-requeue or --requeue option to change the default behavior for individual jobs. The default value is 1. periodt.

`JobSubmitPlugins` 🧚

These are intended to be site-specific plugins which can be used to set default job parameters and/or logging events. Slurm can be configured to use multiple job_submit plugins if desired, which must be specified as a comma-delimited list and will be executed in the order listed.

e.g. for multiple job_submit plugin configuration:

JobSubmitPlugins=lua,require_timelimit

Take a look at <https://slurm.schedmd.com/job_submit_plugins.html> for further plugin implementation details. No job submission plugins are used by default. Currently available plugins are:

all_partitions — Set default partition to all partitions on the cluster.
defaults — Set default values for job submission or modify requests.
logging — Log select job submission and modification parameters.
lua — Execute a Lua script implementing site’s own job_submit logic. Only one Lua script will be executed. It must be named "job_submit.lua" and must be located in the default configuration directory (typically the subdirectory "etc" of the installation directory). Sample Lua scripts can be found with the Slurm distribution, in the directory contribs/lua. Slurmctld will fatal on startup if the configured lua script is invalid. Slurm will try to load the script for each job submission. If the script is broken or removed while slurmctld is running, Slurm will fallback to the previous working version of the script. ⚠️ Warning: slurmctld runs this script while holding internal locks, and only a single copy of this script can run at a time. This blocks most concurrency in slurmctld. Therefore, this script should run to completion as quickly as possible. like bestie please do not hold the lock and take a nap. 😭
partition — Set a job’s default partition based upon job submission parameters and available partitions.
pbs — Translate PBS job submission options to Slurm equivalent (if possible).
require_timelimit — Force job submissions to specify a timelimit.

NOTE: For examples of use see the Slurm code in "src/plugins/job_submit" and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs.

`KillOnBadExit` 💀

If set to 1, a step will be terminated immediately if any task is crashed or aborted, as indicated by a non-zero exit code. With the default value of 0, if one of the processes is crashed or aborted the other processes will continue to run while the crashed or aborted process waits. The user can override this configuration parameter by using srun’s -K, --kill-on-bad-exit. choose your fighter: instant cleanup or “we wait”. 🫶

`KillWait` ⏳

The interval, in seconds, given to a job’s processes between the SIGTERM and SIGKILL signals upon reaching its time limit. If the job fails to terminate gracefully in the interval specified, it will be forcibly terminated. The default value is 30 seconds. The value may not exceed 65533. so it’s basically the grace-period window before things get dramatic. ✨

`MaxBatchRequeue` 🔂

Maximum number of times a batch job may be automatically requeued before being marked as JobHeldAdmin. (Mainly useful when the SchedulerParameters option nohold_on_prolog_fail is enabled.) The default value is 5. lowkey the “how many comeback attempts before we hold you” setting. 💅

`NodeFeaturesPlugins` 🦋

Identifies the plugins to be used for support of node features which can change through time. For example, a node which might be booted with various BIOS setting. This is supported through the use of a node’s active_features and available_features information. Acceptable values at present include:

node_features/helpers — Used to report and modify features on nodes using arbitrary scripts or programs. See helpers.conf man page for more information: <https://slurm.schedmd.com/helpers.conf.html>.

`LaunchParameters` 🚀

Identifies options to the job launch plugin. Acceptable values include:

batch_step_set_cpu_freq — Set the cpu frequency for the batch step from given --cpu-freq, or slurm.conf CpuFreqDef, option. By default only steps started with srun will utilize the cpu freq setting options. NOTE: If you are using srun to launch your steps inside a batch script (advised) this option will create a situation where you may have multiple agents setting the cpu_freq as the batch step usually runs on the same resources one or more steps the sruns in the script will create. it’s giving “many cooks in the cpu-frequency kitchen” ngl. 👩‍🍳
interactive_step_set_cpu_freq — Set the cpu frequency for the interactive step from given --cpu-freq, or slurm.conf CpuFreqDef, options.
cray_net_exclusive — Allow jobs on a Cray XC cluster exclusive access to network resources. This should only be set on clusters providing exclusive access to each node to a single job at once, and not using parallel steps within the job, otherwise resources on the node can be oversubscribed. so like… only if the whole node is truly one job’s moment. 👑
enable_nss_slurm — Permits passwd and group resolution for a job to be serviced by slurmstepd rather than requiring a lookup from a network based service. See <https://slurm.schedmd.com/nss_slurm.html> for more information.
lustre_no_flush — If set on a Cray XC cluster, then do not flush the Lustre cache on job step completion. This setting will only take effect after reconfiguring, and will only take effect for newly launched jobs.
mpir_use_nodeaddr — When launching tasks Slurm creates entries in MPIR_proctable that are used by parallel debuggers, profilers, and related tools to attach to running process. By default the MPIR_proctable entries contain MPIR_procdesc structures where the host_name is set to NodeName by default. If this option is specified, NodeAddr will be used in this context instead.
disable_send_gids — By default, the slurmctld will look up and send the user_name and extended gids for a job, rather than independently on each node as part of each task launch. This helps mitigate issues around name service scalability when launching jobs involving many nodes. Using this option will disable this functionality. This option is ignored if enable_nss_slurm is specified.
slurmstepd_memlock — Lock the slurmstepd process’s current memory in RAM.
slurmstepd_memlock_all — Lock the slurmstepd process’s current and future memory in RAM.
srun_exclusive_allocation — Have srun --exclusive default to the --exclusive=allocation option. This disables the implicit --exact that exclusive adds for srun. Users can still specify srun --exclusive --exact to get the standard behavior.
test_exec — okay so this makes srun do a lil vibe check first 👀: it verifies that the executable program exists and that the user has execute permission on the node where srun was called before it even tries to launch it on nodes in the step. No cap, just preflight.
use_interactive_step — this tells salloc to use the Interactive Step to launch a shell on an allocated compute node instead of launching it locally wherever salloc was invoked 🫶. It pulls this off by launching the srun command with InteractiveStepOptions as options.

This does not affect salloc called with a command as an argument. Those jobs will continue to be executed as the calling user on the calling host. periodt.

ulimit_pam_adopt — when pam_slurm_adopt is used to join an external process into a job cgroup, RLIMIT_RSS is set, just like it is for tasks running in regular steps 💅.

LicenseParameters

Parameters to control the license behavior ✨. By default no parameters are set. Multiple parameters may be specified in a comma-separated list. Currently supported options are:

RemoteFuzzyMatch — allows jobs to be submitted with a shorthand for remote licenses that omits the server name 🦋. For example, "matlab@licenseServer" can be requested with "matlab". Requests including the server name will always be recognized. If a local (slurm.conf-defined) license exists, it will prefer to match that. If multiple servers exist with the same license name, the fuzzy match fails and an exact match (with the server name) must be used. iykyk.

Licenses

Specification of licenses (or other resources available on all nodes of the cluster) which can be allocated to jobs 👑. License names can optionally be followed by a colon and count with a default count of one. Multiple license names should be comma separated (e.g. "Licenses=foo:4,bar"). Note that Slurm prevents jobs from being scheduled if their required license specification is not available. Slurm does not prevent jobs from using licenses that are not explicitly listed in the job submission specification. bestie, that distinction matters 💀.

LogTimeFormat

Format of the timestamp in slurmctld and slurmd log files 🕰️. Accepted format values include "iso8601", "iso8601_ms", "rfc5424", "rfc5424_ms", "rfc3339", "clock", "short" and "thread_id". The values ending in "_ms" differ from the ones without in that fractional seconds with millisecond precision are printed. The default value is "iso8601_ms". The "rfc5424" formats are the same as the "iso8601" formats except that the timezone value is also shown. The "clock" format shows a timestamp in microseconds retrieved with the C standard clock() function. The "short" format is a short date and time format. The "thread_id" format shows the timestamp in the C standard ctime() function form without the year but including the microseconds, the daemon's process ID and the current thread name and ID. iconic logging behavior tbh 🌈.

MailDomain

Domain name to qualify usernames if email address is not explicitly given with the "--mail-user" option 📬. If unset, the local MTA will need to qualify the local address itself. Changes to MailDomain will only affect new jobs. no retroactive email glow-ups 💅.

MailProg

Fully qualified pathname to the program used to send email per user request 📧. The default value is "/bin/mail" (or "/usr/bin/mail" if "/bin/mail" does not exist but "/usr/bin/mail" does exist). The program is called with arguments suitable for the default mail command, however additional information about the job is passed in the form of environment variables.

Additional variables are the same as those passed to PrologSlurmctld and EpilogSlurmctld with additional variables in the following contexts:

ALL
- SLURM_JOB_STATE — the base state of the job when the MailProg is called.
- SLURM_JOB_MAIL_TYPE — the mail type triggering the mail.
BEGIN
- SLURM_JOB_QEUEUED_TIME — the amount of time the job was queued.
END, FAIL, REQUEUE, TIME_LIMIT_*
- SLURM_JOB_RUN_TIME — the amount of time the job ran for.
END, FAIL
- SLURM_JOB_EXIT_CODE_MAX — job's exit code or highest exit code for an array job.
- SLURM_JOB_EXIT_CODE_MIN — job's minimum exit code for an array job.
- SLURM_JOB_TERM_SIGNAL_MAX — job's highest signal for an array job.
STAGE_OUT
- SLURM_JOB_STAGE_OUT_TIME — job's staging out time.

MaxArraySize

The maximum job array task index value will be one less than MaxArraySize to allow for an index value of zero 🍓. Configure MaxArraySize to 0 in order to disable job array use. The value may not exceed 4000001. The value of MaxJobCount should be much larger than MaxArraySize, since each job array task still counts as a separate job (see MaxJobCount for further details). The default value is 1001. See also max_array_tasks in SchedulerParameters. the way arrays can sneak up on you 😭.

MaxDBDMsgs

When communication to the SlurmDBD is not possible the slurmctld will queue messages meant to processed when the SlurmDBD is available again 📨. In order to avoid running out of memory the slurmctld will only queue so many messages. The default value is 10000, or MaxJobCount * 2 + Node Count * 4, whichever is greater. The value can not be less than 10000. memory safety queen 👑.

MaxJobCount

The maximum number of jobs slurmctld can have in memory at one time 🧠. Combine with MinJobAge to ensure the slurmctld daemon does not exhaust its memory or other resources. Once this limit is reached, requests to submit additional jobs will fail. The default value is 10000 jobs.

NOTE: Each task of a job array counts as one job even though they will not occupy separate job records until modified or initiated. Performance can suffer with more than a few hundred thousand jobs. Setting MaxSubmitJobs per user is generally valuable to prevent a single user from filling the system with jobs. This is accomplished using Slurm's database and configuring enforcement of resource limits.

MaxJobId

The maximum job id to be used for jobs submitted to Slurm without a specific requested value 🔢. Job ids are unsigned 32bit integers with the first 26 bits reserved for local job ids and the remaining 6 bits reserved for a cluster id to identify a federated job's origin. The maximum allowed local job id is 67,108,863 (0x3FFFFFF). The default value is 67,043,328 (0x03ff0000). MaxJobId only applies to the local job id and not the federated job id. Job id values generated will be incremented by 1 for each subsequent job. Once MaxJobId is reached, the next job will be assigned FirstJobId. Federated jobs will always have a job ID of 67,108,865 or higher. Also see FirstJobId. the number lore is real 💀.

MaxMemPerCPU

Maximum real memory size available per allocated CPU in mebibytes 🧮. Used to avoid over-subscribing memory and causing paging. MaxMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_tres). The default value is 0 (unlimited). Also see DefMemPerCPU, DefMemPerGPU and MaxMemPerNode. MaxMemPerCPU and MaxMemPerNode are mutually exclusive. pick a lane, bestie ✨.

NOTE: If a job specifies a memory per CPU limit that exceeds this system limit, that job's count of CPUs per task will try to automatically increase. This may result in the job failing due to CPU count limits. This auto-adjustment feature is a best-effort one and optimal assignment is not guaranteed due to the possibility of having heterogeneous configurations and multi-partition/qos jobs. If this is a concern it is advised to use a job submit LUA plugin instead to enforce auto-adjustments to your specific needs. not me reading this twice 😭.

MaxMemPerNode

Maximum real memory size available per allocated node in a job allocation in mebibytes 💾. Used to avoid over-subscribing memory and causing paging. MaxMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). The default value is 0 (unlimited). Also see DefMemPerNode and MaxMemPerCPU. MaxMemPerCPU and MaxMemPerNode are mutually exclusive. again: mutually exclusive, choose your fighter 💅.

MaxNodeCount

Maximum count of nodes which may exist in the controller 🖥️. By default MaxNodeCount will be set to the number of nodes found in the slurm.conf. MaxNodeCount will be ignored if less than the number of nodes found in the slurm.conf. The total number of nodes in a system cannot exceed 65536. Increase MaxNodeCount to accommodate dynamically created nodes with dynamic node registrations and nodes created with scontrol. dynamic node main character energy 🦋.

MaxStepCount

The maximum number of steps that any job can initiate 🪜. This parameter is intended to limit the effect of bad batch scripts. The default value is 40000 steps. protect the cluster from chaos scripts, fr fr 🫶.

MaxTasksPerNode

Maximum number of tasks Slurm will allow a job step to spawn on a single node 🔥. The default MaxTasksPerNode is 512. May not exceed 65533. that upper bound is giving very specific, and we respect it 👑.

MCSParameters

MCS = Multi-Category Security MCS Plugin Parameters 🔐. The supported parameters are specific to the MCSPlugin. Changes to this value take effect when the Slurm daemons are reconfigured. More information about MCS is available here https://slurm.schedmd.com/mcs.html.

`MCSPlugin` 🌸

MCS stands for Multi-Category Security — the whole vibe is that you slap a security label on jobs and make sure nodes only get shared between jobs rockin' the same label. It's unset by default. The acceptable values are giving:

mcs/account — only users with the same account can share the nodes (heads up: this requires accounting to be enabled, no cap).
mcs/group — only users with the same group can share the nodes.
mcs/user — a node cannot be shared with other users. Like, at all. Exclusive bestie energy.
mcs/label — only jobs with the same arbitrary label string can share nodes.

`MessageTimeout` ⏱️

How long a round-trip communication is permitted to take, in seconds. Default is 10 seconds. On systems with shared nodes, the slurmd daemon could get paged out, so you might need to bump this value higher. Give it some room to breathe 💖.

`MetricsAuthUsers` 🔐

Controls which users are allowed to query metrics plugin endpoints. SlurmUser and root are always allowed — main character privileges, obviously. When set, it will ignore PrivateData and/or MetricsParameters=ignore_private_data, allowing users in the list to query the metrics endpoints. This option enables JWT authentication for querying metrics endpoints.

`MetricsParameters` 📊

Parameters related to metrics plugins. You can comma-separate multiple ones. Currently supported parameters include:

ignore_private_data — causes metrics plugins to ignore PrivateData when set, and allows all users global access to query metrics endpoints without authentication. This option will be ignored if MetricsAuthUsers is set. (So if both are on, MetricsAuthUsers wins — it's the alpha here 💅)

`MetricsType` ✨

Specifies the metrics plugin used to dump statistics from the slurmctld socket. Refer to the Metrics Guide for the full scoop: <https://slurm.schedmd.com/metrics.html>

Configurable values at present are:

metrics/openmetrics — use the OpenMetrics based plugin. Slay.

`MinJobAge` 🕰️

The minimum age of a completed job before its record is cleared from the list of jobs slurmctld keeps in memory. Pair it with MaxJobCount so the slurmctld daemon doesn't exhaust its memory or other resources — protect the queen's RAM, fr fr. Default value is 300 seconds. A value of zero prevents any job record purging. Jobs are not purged during a backfill cycle, so it can take longer than MinJobAge seconds to purge a job if you're using the backfill scheduling plugin. And to eliminate some possible race conditions, the minimum non-zero value for MinJobAge recommended is 2.

`MpiDefault` 🧚

Identifies the default type of MPI to be used. Unset by default — which lets Slurm work with versions of MPI other than the ones listed below. Srun may override this configuration parameter in any case, so don't stress. Currently supported versions include: pmi2 and pmix. More info about MPI use is available here: <https://slurm.schedmd.com/mpi_guide.html>

`MpiParams` 🌈

MPI-related parameters. Multiple parameters may be comma separated. Currently supported parameters include:

ports= — identifies a range of communication ports used by native Cray's PMI.
disable_slurm_hydra_bootstrap — disable environment variable injection in allocations for the following variables: I_MPI_HYDRA_BOOTSTRAP, I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS, HYDRA_BOOTSTRAP, HYDRA_LAUNCHER_EXTRA_ARGS.

Manually setting I_MPI_HYDRA_BOOTSTRAP or HYDRA_BOOTSTRAP to 'slurm' in the allocation will skip this parameter and injection of extra args will be performed as usual. iykyk 😌

`NamespaceType` 🦋

Identifies the plugin to be used for job isolation through namespaces. To use these plugins, PrologFlags=Contain must be set — no exceptions, bestie. Refer to the namespace page for further details: <https://slurm.schedmd.com/namespace.html>

NOTE: See ProctrackType for resource containment and usage tracking.

Acceptable values at present include:

namespace/linux — used to create a private filesystem namespace and optionally user and pid namespaces. The filesystem namespace houses temporary filesystems (/tmp and /dev/shm) for each job.

NOTE: This plugin requires cgroup/v2 to operate correctly. NOTE: When using user namespaces, bpf token support (added in kernel 6.9) is required to use ConstrainDevices in cgroup.conf.

namespace/tmpfs — used to create a private namespace on the filesystem for jobs, which houses temporary file systems (/tmp and /dev/shm) for each job.

`OverTimeLimit` ⏳

The number of minutes by which a job can exceed its time limit before being canceled. Normally a job's time limit is treated as a hard limit and the job gets killed upon reaching it. Configuring OverTimeLimit makes the job's time limit behave like a soft limit instead. Adding the OverTimeLimit value to the soft time limit gives you a hard time limit — and that's the point where the job is canceled. This is especially useful for backfill scheduling, which bases upon each job's soft time limit. Default value is zero. May not exceed 65533 minutes. A value of "UNLIMITED" is also supported.

`PluginDir` 📁

Identifies the places in which to look for Slurm plugins. This is a colon-separated list of directories, like the PATH environment variable. The default value is the prefix given at configure time + /lib/slurm.

`PlugStackConfig` 🎀

Location of the config file for Slurm stackable plugins that use the Stackable Plugin Architecture for Node job (K)control (SPANK). This provides support for a highly configurable set of plugins to be called before and/or after execution of each task spawned as part of a user's job step. Default location is "plugstack.conf" in the same directory as the system slurm.conf. For more information on SPANK plugins, see the spank(8) manual.

`PreemptMode` 👑

The mechanism used to preempt jobs or enable gang scheduling. When the PreemptType parameter is set to enable preemption, PreemptMode selects the default mechanism used to preempt the eligible jobs for the cluster.

PreemptMode may be specified on a per partition basis to override this default value if PreemptType=preempt/partition_prio. Alternatively, it can be specified on a per QOS basis if PreemptType=preempt/qos. In either case, a valid default PreemptMode value must be specified for the cluster as a whole when preemption is enabled.

The GANG option is used to enable gang scheduling independent of whether preemption is enabled (i.e. independent of the PreemptType setting). It can be specified in addition to a PreemptMode setting with the two options comma separated (e.g. PreemptMode=SUSPEND,GANG).

See <https://slurm.schedmd.com/preempt.html> and <https://slurm.schedmd.com/gang_scheduling.html> for more details.

NOTE: For performance reasons, the backfill scheduler reserves whole nodes for jobs, not partial nodes. If during backfill scheduling a job preempts one or more other jobs, the whole nodes for those preempted jobs are reserved for the preemptor job, even if the preemptor job requested fewer resources than that. These reserved nodes aren't available to other jobs during that backfill cycle, even if the other jobs could fit on the nodes. Therefore, jobs may preempt more resources during a single backfill iteration than they requested. (Yeah, it's giving overachiever 😭) NOTE: For a heterogeneous job to be considered for preemption all components must be eligible for preemption. When a heterogeneous job is to be preempted, the first identified component of the job with the highest order PreemptMode (SUSPEND (highest), REQUEUE, CANCEL (lowest)) will be used to set the PreemptMode for all components. The GraceTime and user warning signal for each component of the heterogeneous job remain unique. Heterogeneous jobs are excluded from GANG scheduling operations.

OFF — is the default value and disables job preemption and gang scheduling. It is only compatible with preemption being disabled at the global level. A common use case for this parameter is to set it on a partition to disable preemption for that partition.
CANCEL — the preempted job will be cancelled. Bye bestie 💀
GANG — enables gang scheduling (time slicing) of jobs in the same partition, and allows the resuming of suspended jobs. In order to use gang scheduling, the GANG option must be specified at the cluster level.

NOTE: If GANG scheduling is enabled with PreemptType=preempt/partition_prio, the controller will ignore the following PreemptParameters: reorder_count, strict_order, and youngest_first. NOTE: Gang scheduling is performed independently for each partition, so if you only want time-slicing by OverSubscribe, without any preemption, then configuring partitions with overlapping nodes is not recommended. On the other hand, if you want to use PreemptType=preempt/partition_prio to allow jobs from higher PriorityTier partitions to Suspend jobs from lower PriorityTier partitions you will need overlapping partitions, and PreemptMode=SUSPEND,GANG to use the Gang scheduler to resume the suspended job(s). You must configure the partition's OverSubscribe setting to FORCE for all partitions in which time-slicing is to take place. In any case, time-slicing won't happen between jobs on different partitions. NOTE: Heterogeneous jobs are excluded from GANG scheduling operations.

NOTE: this is the overlapping-partitions situation, bestie. If a node is allocated to a job that allows sharing of resources (Oversubscribe=FORCE, or Oversubscribe=YES and the job was submitted with -s/--oversubscribe) then that node can only be allocated by jobs from the same partition. Same partition only. No crossover. 💅

REQUEUE — preempts jobs by requeuing them (if possible) or canceling them. For jobs to actually get requeued they must have the --requeue sbatch option set, or the cluster-wide JobRequeue parameter in slurm.conf must be set to 1. No --requeue, no requeue, periodt.
SUSPEND — the preempted jobs get suspended, and later the Gang scheduler resumes them. So the SUSPEND preemption mode always needs the GANG option specified at the cluster level — non-negotiable. Also (and this is the lowkey stressful part) because suspended jobs still use memory on the allocated nodes, Slurm needs to be able to track memory resources to be able to suspend jobs at all. When suspending jobs, Slurm sends the SIGTSTP signal, waits the time specified by PreemptParameters=suspend_grace_time (default is 2 seconds), then sends the SIGSTOP signal. The SIGCONT signal is sent when resuming jobs. 😴✨ If PreemptType=preempt/qos is configured and if the preempted job(s) and the preemptor job are on the same partition, then they'll share resources with the Gang scheduler (time-slicing). If not (i.e. if the preemptees and preemptor are on different partitions) then the preempted jobs will remain suspended until the preemptor ends.

NOTE: because gang scheduling is performed independently for each partition, if using PreemptType=preempt/partition_prio then jobs in higher PriorityTier partitions will suspend jobs in lower PriorityTier partitions to run on the released resources. Only when the preemptor job ends will the suspended jobs be resumed by the Gang scheduler.

NOTE: suspended jobs will not release GRES. Higher priority jobs will not be able to preempt to gain access to GRES. Like, at all. Rip. 💀

PRIORITY — allow preemption only if the preemptor's job priority is higher than the preemptee's job priority. Simple as that, fr fr.
WITHIN — for PreemptType=preempt/qos, allow jobs within the same qos to preempt one another. While this can be set globally here, it is recommend that this only be set directly on a relevant subset of the system qos values instead. (Their typo, not mine — "recommend" stays iconic. 🎀)

🌸 `PreemptParameters`

Multiple options may be comma separated. Comma separated. Say it with me. 🫶

min_exempt_priority=Threshold — threshold value for the job's global priority. Only those jobs with priority lower than this value will be marked as preemptable.
reclaim_licenses — if set, jobs may be preempted to reclaim licenses. Otherwise jobs requesting busy licenses will have to wait even if they have preemption priority. The logic to support this option is only available in the select/cons_tres plugin. Jobs that use OR in the license request are not eligible to preempt other jobs to reclaim licenses, and preemption is never used to reclaim Hierarchical Resources (HRES). 🎟️
reorder_count= — specify how many attempts should be made in reordering preemptable jobs to minimize the total number of jobs that will be preempted. The default value is 1. High values may adversely impact performance (like, highkey watch out). Changes to the order of jobs on these attempts can be enabled with strict_order. The logic to support this option is only available in the select/cons_tres plugin.
send_user_signal — send the user signal (e.g. --signal=<sig_num>) at preemption time even if the signal time hasn't been reached. In the case of a gracetime preemption the user signal will be sent if the user signal has been specified and not sent, otherwise a SIGTERM will be sent to the tasks. 📣
strict_order — when reordering preemptable jobs, place the most recently tested job at the front of the list since we're certain that it actually added resources needed by the new job. This ensures that with enough reorder attempts, the minimum possible number of jobs will be preempted. See also reorder_count. The logic to support this option is only available in the select/cons_tres plugin. 👑
suspend_grace_time — specifies, in units of seconds, the preemption grace time when using PreemptMode=SUSPEND. When a job is suspended, the SIGTSTP signal will be sent, and then after waiting the specified suspend grace time, the SIGSTOP signal will be sent. The default value is 2 seconds.

NOTE: this parameter is only used when PreemptMode=SUSPEND is configured or when suspending jobs with scontrol suspend. For setting the preemption grace time when using other preemption modes, see GraceTime.

youngest_first — if set, then the preemption sorting algorithm will be changed to sort by the job start times to favor preempting younger jobs over older. (Requires preempt/partition_prio or preempt/qos plugins.) Sorry boomers, the youths are winning this one. 🧚

🦋 `PreemptType`

Specifies the plugin used to identify which jobs can be preempted in order to start a pending job. Unset by default.

preempt/partition_prio — job preemption is based upon partition PriorityTier. Jobs in higher PriorityTier partitions may preempt jobs from lower PriorityTier partitions. This is not compatible with PreemptMode=OFF.
preempt/qos — job preemption rules are specified by Quality Of Service (QOS) specifications in the Slurm database. In the case of PreemptMode=SUSPEND, a preempting job has to be submitted to a partition with a higher PriorityTier or to the same partition. Submission to the same partition is also supported, which results in the preemptor QoS to gang schedule the preemptee QoS. This option is not compatible with PreemptMode=OFF. A configuration of PreemptMode=SUSPEND is only supported by the SelectType=select/cons_tres plugin. See the sacctmgr man page to configure the options for preempt/qos. 📚

💜 `PreemptExemptTime`

Global option for minimum run time for all jobs before they can be considered for preemption. Any QOS PreemptExemptTime takes precedence over the global option. Hierarchy, bestie. 👑

A time of -1 disables the option, equivalent to 0. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes", and "days-hours:minutes:seconds".

🎀 `PrEpParameters`

Parameters to be passed to the PrEpPlugins. That's it. That's the whole thing. ✨

🌈 `PrEpPlugins`

A resource for programmers wishing to write their own plugins for the Prolog and Epilog (PrEp) scripts. The default, and currently the only implemented plugin, is prep/script. Additional plugins can be specified in a comma-separated list. For more information please see the PrEp Plugin API documentation page: https://slurm.schedmd.com/prep_plugins.html 🫶

⭐ `PriorityCalcPeriod`

The period of time in minutes in which the half-life decay will be re-calculated. Applicable only if PriorityType=priority/multifactor. The default value is 5 (minutes). Five. Iconic number. 💅

🍓 `PriorityDecayHalfLife`

This controls how long prior resource use is considered in determining how over- or under-serviced an association is (user, account and cluster) in determining job priority. The record of usage will be decayed over time, with half of the original value cleared at age PriorityDecayHalfLife. If set to 0 no decay will be applied — helpful if you want to enforce hard time limits per association. If set to 0, PriorityUsageResetPeriod must be set to some interval. Applicable only if PriorityType=priority/multifactor. The unit is a time string (i.e. min, hr:min:00, days-hr:min:00, or days-hr). The default value is 7-0 (7 days). ⏳

🧚 `PriorityFavorSmall`

Specifies that small jobs should be given preferential scheduling priority. Applicable only if PriorityType=priority/multifactor. Supported values are "YES" and "NO". The default value is "NO". Small jobs stay winning? Only if you say so. 🐣

🔥 `PriorityFlags`

Flags to modify priority behavior. Applicable only if PriorityType=priority/multifactor. The keywords below have no associated value (e.g. "PriorityFlags=ACCRUE_ALWAYS,SMALL_RELATIVE_TO_TIME"). No values, just vibes. 💫

ACCRUE_ALWAYS — if set, priority age factor will be increased despite job ineligibility due to either dependencies, holds or begin time in the future. Accrue limits are ignored.
CALCULATE_RUNNING — if set, priorities will be recalculated not only for pending jobs, but also running and suspended jobs.
DEPTH_OBLIVIOUS — if set, priority will be calculated based similar to the normal multifactor calculation, but depth of the associations in the tree does not adversely affect their priority. This option automatically enables NO_FAIR_TREE.
NO_FAIR_TREE — disables the "fair tree" algorithm, and reverts to "classic" fair share priority scheduling. Throwback era. 🕰️
INCR_ONLY — if set, priority values will only increase in value. Job priority will never decrease in value. Numbers go up, that's the whole plot. 📈
MAX_TRES — set this and the weighted TRES value (TRESBillingWeights) gets calculated as the sum of these lil' pieces:
- the MAX of the individual TRESs (the ones tied to a specific node, like cpus, mem, gres)
- the SUM of all the global TRESs (the ones available on any node, like licenses)

Want the full lore? Go read the TRES page: <https://slurm.schedmd.com/tres.html> ✨

MAX_TRES_GRES — okay so this one's lowkey the glow-up version. Set it and the weighted TRES value (TRESBillingWeights) becomes the sum of:
- the SUM of all billable GRESs (think GPUs 💅)
- the MAX of the other individual TRESs (the ones tied to a specific node, like cpus, mem)
- the SUM of all global TRESs (available on any node, like licenses)

Added in Slurm 25.05. TRES page for the details, bestie: <https://slurm.schedmd.com/tres.html>

NO_NORMAL_ALL — set it and boom, every single NO_NORMAL_* flag is set. all of them. at once. 🎀
NO_NORMAL_ASSOC — set it and the association factor does not get normalized against the highest association priority.
NO_NORMAL_PART — set it and the partition factor does not get normalized against the highest partition PriorityJobFactor.
NO_NORMAL_QOS — set it and the QOS factor does not get normalized against the highest qos priority.
NO_NORMAL_TRES — set it and the TRES factor does not get normalized against the job's partition TRES counts.
SMALL_RELATIVE_TO_TIME — set it and the job's size component isn't based on the job size alone, it's the job's size divided by its time limit. math girlie behavior honestly 🧮

`PriorityMaxAge`

This one's the job age that gets awarded the maximum age factor when priority is being computed. So like, say you set it to 30 minutes — then every job older than 30 minutes gets the exact same age-based priority. Very "everyone past this point is equally ancient" of it.

Only applies if PriorityType=priority/multifactor. The unit is a time string (i.e. min, hr:min:00, days-hr:min:00, or days-hr). Default value is 7-0 (7 days). ⭐

`PriorityParameters`

Arbitrary string used by the PriorityType plugin. it's a whole "you do you" situation.

`PrioritySiteFactorParameters`

Arbitrary string used by the PrioritySiteFactorPlugin plugin.

`PrioritySiteFactorPlugin`

Specifies an optional plugin to be used alongside "priority/multifactor" — its whole job is to initially set and then continuously update the SiteFactor priority factor. Unset by default. 🦋

`PriorityType`

This specifies the plugin used to establish a job's scheduling priority. Also peep PriorityFlags for configuration options. The default value is "priority/multifactor".

priority/basic — jobs are evaluated in a First In, First Out (FIFO) manner.
priority/multifactor — jobs get a priority based on a whole variety of factors, including size, age, Fairshare, etc.

When it's not FIFO scheduling, jobs are prioritized in this order, no cap:

Jobs that can preempt
Jobs with an advanced reservation
Partition PriorityTier
Job priority
Job submit time
Job ID

`PriorityUsageResetPeriod`

At this interval the usage of associations gets reset to 0. This is the move if you want to enforce hard limits of time usage per association. If PriorityDecayHalfLife is set to 0 then no decay happens at all, and this is the only way to reset the usage that running jobs have accumulated.

By default this is turned off, and honestly it's advised that you use the PriorityDecayHalfLife option so you don't end up with literally nothing running on your cluster 😭 — but if your schema is set up to only allow certain amounts of time on your system, this is how you do it. Only applies if PriorityType=priority/multifactor.

NONE — never clear historic usage. The default value.
NOW — clear the historic usage now. Executed at startup and reconfiguration time.
DAILY — cleared every day at midnight.
WEEKLY — cleared every week on Sunday at time 00:00.
MONTHLY — cleared on the first day of each month at time 00:00.
QUARTERLY — cleared on the first day of each quarter at time 00:00.
YEARLY — cleared on the first day of each year at time 00:00.

`PriorityWeightAge`

An integer value that sets the degree to which the queue wait time component contributes to the job's priority. Only applies if PriorityType=priority/multifactor. Requires AccountingStorageType=accounting_storage/slurmdbd. Default value is 0.

`PriorityWeightAssoc`

An integer value that sets the degree to which the association component contributes to the job's priority. Only applies if PriorityType=priority/multifactor. Default value is 0.

`PriorityWeightFairshare`

An integer value that sets the degree to which the fair-share component contributes to the job's priority. Only applies if PriorityType=priority/multifactor. Requires AccountingStorageType=accounting_storage/slurmdbd. Default value is 0.

`PriorityWeightJobSize`

An integer value that sets the degree to which the job size component contributes to the job's priority. Only applies if PriorityType=priority/multifactor. Default value is 0.

`PriorityWeightPartition`

Partition factor used by the priority/multifactor plugin when it's calculating job priority. Only applies if PriorityType=priority/multifactor. Default value is 0.

`PriorityWeightQOS`

An integer value that sets the degree to which the Quality Of Service component contributes to the job's priority. Only applies if PriorityType=priority/multifactor. Default value is 0. ✨

`PriorityWeightTRES`

A comma-separated list of TRES Types and weights that sets the degree to which each TRES Type contributes to the job's priority.

e.g.

PriorityWeightTRES=CPU=1000,Mem=2000,GRES/gpu=3000

Only applies if PriorityType=priority/multifactor and if AccountingStorageTRES is configured with each TRES Type. Negative values are allowed. Default values are 0.

`PrivateData`

This controls what type of information is hidden from regular users. By default, all information is visible to all users. User SlurmUser and root can always view all information, no matter what. When this option is set, it causes metrics plugins endpoints to be restricted (plus it enables JWT authentication for querying) and available only for SlurmUser and root, unless MetricsAuthUsers or MetricsParameters=ignore_private_data are configured. Multiple values may be specified with a comma separator. Acceptable values include:

accounts — (NON-SlurmDBD ACCOUNTING ONLY) stops users from viewing any account definitions unless they're coordinators of them.
events — prevents users from viewing event information unless they have operator status or above.
jobs — prevents users from viewing jobs or job steps belonging to other users. (NON-SlurmDBD ACCOUNTING ONLY) Prevents users from viewing job records belonging to other users unless they're coordinators of the association running the job when using sacct.
nodes — prevents users from viewing node state information.
partitions — prevents users from viewing partition state information.
reservations — prevents regular users from viewing reservations which they can't use.

NOTE: QOS and AllowedPartitions restrictions on the reservation do not apply to this option.

usage — prevents users from viewing usage of any other user, this applies to 'sshare' and Association Records through 'scontrol show assoc_mgr'. (NON-SlurmDBD ACCOUNTING ONLY) Prevents users from viewing usage of any other user, this applies to sreport.
users — (NON-SlurmDBD ACCOUNTING ONLY) prevents users from viewing information of any user other than themselves, and this also makes it so users can only see associations they deal with. Coordinators can see associations of all users in the account they're coordinator of, but can only see themselves when listing users.

`ProctrackType`

Identifies the plugin used for process tracking on a job step basis. The slurmd daemon uses this mechanism to identify all processes which are children of processes it spawns for a user job step.

NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to identify all processes associated with a job, because processes can become a child of the init process (when the parent process terminates) or change their process group. To reliably track all processes, "proctrack/cgroup" is highly recommended. 🔥

NOTE: The NamespaceType applies to a job namespace isolation, while ProctrackType applies to job resource limits and tracking. different vibes, different jobs.

Acceptable values at present include:

proctrack/cgroup — uses linux cgroups to constrain and track processes, and is the default for systems with cgroup support.

NOTE: See "man cgroup.conf" for configuration details.

proctrack/linuxproc — uses linux process tree using parent process IDs.
proctrack/pgid — uses Process Group IDs.

NOTE: This is the default for the BSD family.

`Prolog`

okay so bestie, Prolog is the pathname of a program that slurmd runs whenever it’s asked to run a job step from a new job allocation ✨ If it’s not an absolute pathname (i.e. it doesn’t start with a slash), it has to be relative to the directory containing the slurm.conf file. You can also use a glob pattern (see glob (7)) to specify more than one program to run — like "/etc/slurm/prolog.d/*" 💅 When more than one prolog script is configured, they run in reverse alphabetical order (z-a -> Z-A -> 9-0). The slurmd executes the prolog before starting the first job step. The prolog script(s) can purge files, enable user login, etc. By default there is no prolog. Any configured script is expected to complete execution quickly (in less time than MessageTimeout). If the prolog fails (returns a non-zero exit code), this results in the node being set to a DRAIN state and the job being requeued. The job will be placed in a held state, unless nohold_on_prolog_fail is configured in SchedulerParameters. See Prolog and Epilog Scripts for more information 🫶

NOTE: It is possible to configure multiple prolog scripts by including this option on multiple lines.

`PrologEpilogTimeout`

The interval in seconds Slurm waits for Prolog and Epilog before terminating them. The default behavior is to wait indefinitely 😭 This interval applies to the Prolog and Epilog run by the slurmd daemon before and after the job, the PrologSlurmctld and EpilogSlurmctld run by the slurmctld daemon, and the SPANK plugin prolog/epilog calls: slurm_spank_job_prolog and slurm_spank_job_epilog. If the PrologSlurmctld times out, the job is requeued if possible. If the Prolog or slurm_spank_job_prolog time out, the job is requeued if possible and the node is drained. If the Epilog or slurm_spank_job_epilog time out, the node is drained. In all cases, errors are logged 🔥

NOTE: This value is not used for prologs if PrologTimeout is configured. Likewise, this value is not used for epilogs if EpilogTimout is configured. (yes, that’s how it’s spelled in the original, bestie 💀)

`PrologTimeout`

The interval in seconds Slurm waits for Prolog before terminating them. The default value is PrologEpilogTimeout ✨ This interval applies to the Prolog run by slurmd daemon before the job, the PrologSlurmctld run by slurmctld daemon, and the SPANK plugin prolog call: slurm_spank_job_prolog. If the PrologSlurmctld times out, the job is requeued if possible. If the Prolog or slurm_spank_job_prolog time out, the job is requeued if possible and the node is drained. In all cases, errors are logged 🌸

`PrologFlags`

Flags to control the Prolog behavior. By default no flags are set. Multiple flags may be specified in a comma-separated list. Currently supported options are:

Alloc — If set, the Prolog script will always be executed at job allocation. Otherwise, the Prolog is executed just before the first task is launched, which includes batch and interactive steps (see LaunchParameters=use_interactive_step). Alloc can be useful for preparing things before a user starts to use any allocated resources. In particular, this flag is needed on a Cray system when cluster compatibility mode is enabled.

NOTE: Use of the Alloc flag will increase the time required to start jobs.

Contain — At job allocation time, use the ProcTrack plugin to create a job container on all allocated compute nodes. This container may be used for user processes not launched under Slurm control, for example pam_slurm_adopt may place processes launched through a direct user login into this container. If using pam_slurm_adopt, then ProcTrackType must be set to proctrack/cgroup. Setting the Contain flag implicitly sets the Alloc flag.
DeferBatch — If set, slurmctld will wait until the prolog completes on all allocated nodes before sending the batch job launch request. If just the Alloc flag is set, slurmctld will launch the batch step as soon as the job’s batch node completes the prolog. Setting the DeferBatch flag implicitly sets the Alloc flag.
NoHold — This will allow for salloc to not block until the prolog is finished on each node. The blocking will happen when steps reach the slurmd and before any execution has happened in the step. This is a much faster way to work and if using srun to launch your tasks you should use this flag. This flag cannot be combined with the Contain or X11 flags. Setting the NoHold flag implicitly sets the Alloc flag.
ForceRequeueOnFail — When a batch job fails to launch due to a Prolog failure, always requeue it automatically even if the job requested no requeues. Setting the ForceRequeueOnFail flag implicitly sets the Alloc flag.
RunInJob — Make the Prolog/Epilog run in the extern slurmstepd. This will contain it in one of the job’s processes. This will contain it in the cgroup if configured. Setting the RunInJob flag implicitly sets the Contain and Alloc flags.
Serial — By default, the Prolog and Epilog scripts run concurrently on each node. This flag forces those scripts to run serially within each node, but with a significant penalty to job throughput on each node.

NOTE: This is incompatible with RunInJob.

X11 — Enable Slurm’s built-in X11 forwarding capabilities. This is incompatible with ProctrackType=proctrack/linuxproc. Setting the X11 flag implicitly sets the Contain and Alloc flags.

`PrologSlurmctld`

Fully qualified pathname of a program for the slurmctld daemon to execute before granting a new job allocation (e.g. "/usr/local/slurm/prolog_controller") 💖 The program executes as SlurmUser on the same node where the slurmctld daemon executes, giving it permission to drain nodes and requeue the job if a failure occurs or cancel the job if appropriate. Exactly what the program does and how it accomplishes this is completely at the discretion of the system administrator. Information about the job being initiated, its allocated nodes, etc. are passed to the program using environment variables. While this program is running, the nodes associated with the job will have a POWER_UP/CONFIGURING flag set in their state, which can be readily viewed. The slurmctld daemon will wait indefinitely for this program to complete. Once the program completes with an exit code of zero, the nodes will be considered ready for use and the program will be started. If some node can not be made available for use, the program should drain the node (typically using the scontrol command) and terminate with a non-zero exit code. A non-zero exit code will result in the job being requeued (where possible) or killed. Note that only batch jobs can be requeued. See Prolog and Epilog Scripts for more information 🦋

NOTE: It is possible to configure multiple prolog scripts by including this option on multiple lines.

`PropagatePrioProcess`

Controls the scheduling priority (nice value) of user spawned tasks 👑

NOTE: This has limited functionality if used in combination with cgroups (proctrack/cgroup or task/cgroup).

0 — The tasks will inherit the scheduling priority from the slurm daemon. This is the default value.
1 — The tasks will inherit the scheduling priority of the command used to submit them (e.g. srun or sbatch). Unless the job is submitted by user root, the tasks will have a scheduling priority no higher than the slurm daemon spawning them.
2 — The tasks will inherit the scheduling priority of the command used to submit them (e.g. srun or sbatch) with the restriction that their nice value will always be one higher than the slurm daemon (i.e. the tasks scheduling priority will be lower than the slurm daemon).

`PropagateResourceLimits`

A comma-separated list of resource limit names. The slurmd daemon uses these names to obtain the associated (soft) limit values from the user’s process environment on the submit node. These limits are then propagated and applied to the jobs that will run on the compute nodes ✨ This parameter can be useful when system limits vary among nodes. Any resource limits that do not appear in the list are not propagated. However, the user can override this by specifying which resource limits to propagate with the sbatch or srun "--propagate" option. If neither PropagateResourceLimits or PropagateResourceLimitsExcept are configured and the "--propagate" option is not specified, then the default action is to propagate all limits. Only one of the parameters, either PropagateResourceLimits or PropagateResourceLimitsExcept, may be specified. The user limits can not exceed hard limits under which the slurmd daemon operates. If the user limits are not propagated, the limits from the slurmd daemon will be propagated to the user’s job. The limits used for the Slurm daemons can be set in the /etc/sysconf/slurm file. For more information, see: https://slurm.schedmd.com/faq.html#memlock The following limit names are supported by Slurm (although some options may not be supported on some systems):

ALL — All limits listed below (default)
NONE — No limits listed below
AS — The maximum address space (virtual memory) for a process.
CORE — The maximum size of core file
CPU — The maximum amount of CPU time
DATA — the max size of a process's data segment 🧠
FSIZE — the max size of files created. okay but real talk: if the user sets FSIZE to less than the current size of the slurmd.log, job launches will fail with a 'File size limit exceeded' error. 💀 like, the log is already bigger than the limit, so the limit says no. brutal but true.
MEMLOCK — the max size that may be locked into memory 🔒
NOFILE — the max number of open files 📂
NPROC — the max number of processes available 🧵
RSS — the max resident set size. heads up: this only has effect with Linux kernels 2.4.30 or older, or BSD. so on anything modern it's basically giving decoration ✨
STACK — the max stack size 📚

`PropagateResourceLimitsExcept`

A comma-separated list of resource limit names 🫶. By default, all resource limits get propagated (as described by the PropagateResourceLimits parameter), except for the limits that show up in this list. And yeah, the user can override the whole thing by specifying which resource limits to propagate with the sbatch or srun "--propagate" option. See PropagateResourceLimits above for the list of valid limit names 👀 (iykyk, it's literally the same list).

`RebootProgram`

The program that gets executed on each compute node to reboot it 🔄. It's invoked on each node once that node becomes idle, after the command scontrol reboot is run by an authorized user, or after a job is submitted with the "--reboot" option. Once the reboot is done, the node goes back to normal use 💅. This can alternatively be set to the name of a PowerAction. See ResumeTimeout to configure the time you expect a reboot to finish in — and bestie, a node will be marked DOWN if it doesn't reboot within ResumeTimeout. no cap.

`ReconfigFlags`

Flags to control various actions that may be taken when an "scontrol reconfig" command is issued 🎛️. Currently the options are:

KeepPartInfo — if set, an "scontrol reconfig" command will maintain the in-memory value of partition "state" and other parameters that may have been dynamically updated by "scontrol update". Partition information in the slurm.conf file will be merged with in-memory data. This flag supersedes the KeepPartState flag 🔥
KeepPartState — if set, an "scontrol reconfig" command will preserve only the current "state" value of in-memory partitions, and will reset all the other parameters of the partitions that may have been dynamically updated by "scontrol update" back to the values from the slurm.conf file. Partition information in the slurm.conf file will be merged with in-memory data.
KeepPowerSaveSettings — if set, an "scontrol reconfig" command will preserve the current state of SuspendExcNodes, SuspendExcParts and SuspendExcStates 🌙

The default for the above flags is not set, and "scontrol reconfig" will rebuild the partition information using only the definitions in the slurm.conf file. clean slate energy ✨

`RequeueExit`

Enables automatic requeue for batch jobs which exit with the specified values 🔁. Separate multiple exit codes with a comma, and/or specify numeric ranges using a - separator (e.g. "RequeueExit=1-9,18"). Jobs get put back into pending state and later scheduled again. Restarted jobs will have the environment variable SLURM_RESTART_COUNT set to the number of times the job has been restarted 👑

`RequeueExitHold`

Same idea, but with ✋ hold on — this enables automatic requeue for batch jobs which exit with the specified values, with these jobs being held until released manually by the user. Separate multiple exit codes with a comma, and/or specify numeric ranges using a - separator (e.g. "RequeueExitHold=10-12,16"). These jobs are put in the JOB_SPECIAL_EXIT exit state. Restarted jobs will have the environment variable SLURM_RESTART_COUNT set to the number of times the job has been restarted. same env var, same icon behavior 🫶

`ResumeFailProgram`

The program that will be executed when nodes fail to resume by ResumeTimeout 😭. The argument to the program will be the names of the failed nodes (using Slurm's hostlist expression format). It can alternatively be set to the name of a PowerAction. Programs will be killed if they run longer than the largest configured, global or partition, ResumeTimeout or SuspendTimeout. rude but fair 💀

`ResumeProgram`

Slurm supports a mechanism to reduce power consumption on nodes that remain idle for an extended period of time 🔋. This is typically accomplished by reducing voltage and frequency or powering the node down. ResumeProgram is the program that will be executed when a node in power save mode is assigned work to perform. It can alternatively be set to the name of a PowerAction.

For reasons of reliability, ResumeProgram may execute more than once for a node when the slurmctld daemon crashes and is restarted 🙈. If ResumeProgram is unable to restore a node to service with a responding slurmd and an updated BootTime, it should set the node state to DOWN, which will result in a requeue of any job associated with the node — and this will happen automatically if the node doesn't register within ResumeTimeout. SchedulerParameters=requeue_on_resume_failure can be used to always requeue batch jobs in this situation, even if the job requested no requeues. If the node isn't actually rebooted (i.e. when multiple-slurmd is configured) starting slurmd with "-b" option might be useful 👀

The program executes as SlurmUser. The argument to the program will be the names of nodes to be removed from power savings mode (using Slurm's hostlist expression format). A job-to-node mapping is available in JSON format by reading the temporary file specified by the SLURM_RESUME_FILE environment variable 📄. This file is closed once slurmctld shuts down. If ResumeProgram is running, slurmctld shutdown is delayed by up to ten seconds to give ResumeProgram time to read this file — therefore, this file should be read at the beginning of ResumeProgram. By default no program is run. Programs will be killed if they run longer than the largest configured, global or partition, ResumeTimeout or SuspendTimeout.

`ResumeRate`

The rate at which nodes in power save mode are returned to normal operation by ResumeProgram ⚡. The value is a number of nodes per minute, and it can be used to prevent power surges if a large number of nodes in power save mode are assigned work at the same time (e.g. a large job starts). A value of zero results in no limits being imposed. The default value is 300 nodes per minute. very much "don't all jump at once" energy 🎀

`ResumeTimeout`

Maximum time permitted (in seconds) between when a node resume request is issued and when the node is actually available for use ⏳. Nodes which fail to respond in this time frame will be marked DOWN and the jobs scheduled on the node requeued if possible. Nodes which reboot after this time frame will be marked DOWN with a reason of "Node unexpectedly rebooted." The default value is 60 seconds, and the maximum value is either 65533 or INFINITE. See also ReturnToService. 👑

`ResvEpilog`

Fully qualified pathname of a program for the slurmctld to execute when a reservation ends 🏁. It does not run when a running reservation is deleted. The program can be used to cancel jobs, modify partition configuration, etc. The reservation named will be passed as an argument to the program. By default there is no epilog.

`ResvOverRun`

Describes how long a job already running in a reservation should be permitted to execute after the end time of the reservation has been reached ⏱️. The time period is specified in minutes and the default value is 0 (kill the job immediately 🔪). The value may not exceed 65533 minutes, although a value of "UNLIMITED" is supported to permit a job to run indefinitely after its reservation is terminated. main character job energy, honestly.

`ResvProlog`

Fully qualified pathname of a program for the slurmctld to execute when a reservation begins 🎬. The program can be used to cancel jobs, modify partition configuration, etc. The reservation named will be passed as an argument to the program. By default there is no prolog.

`ReturnToService`

Controls when a DOWN node will be returned to service 🩹. The default value is 0. Supported values include:

0 — a node will remain in the DOWN state until a system administrator explicitly changes its state (even if the slurmd daemon registers and resumes communications). nope, it's staying there, hand on the wheel 🛑
1 — a DOWN node will become available for use upon registration with a valid configuration, only if it was set DOWN due to being non-responsive. If the node was set DOWN for any other reason (low memory, unexpected reboot, etc.), its state will not automatically be changed. A node registers with a valid configuration if its memory, GRES, CPU count, etc. are equal to or greater than the values configured in slurm.conf ✅
2 — a DOWN node will become available for use upon registration with a valid configuration. The node could have been set DOWN for any reason. A node registers with a valid configuration if its memory, GRES, CPU count, etc. are equal to or greater than the values configured in slurm.conf 🌈

`SchedulerParameters`

The interpretation of this parameter varies by SchedulerType 👀. Multiple options may be comma separated.

allow_zero_lic — if set, then job submissions requesting more than configured licenses won't be rejected. like, let them cook 🍳
assoc_limit_stop — if set and a job cannot start due to association limits, then do not attempt to initiate any lower priority jobs in that partition. Setting this can decrease system throughput and utilization, but avoid potentially starving larger jobs by preventing them from launching indefinitely ⚖️
batch_sched_delay — how long (in seconds) the scheduler is allowed to delay scheduling batch jobs. It's giving "let me breathe" energy for high-throughput setups where batch jobs get submitted at a truly unhinged rate (like, sbatch go brrr) and you want to cut down the overhead of trying to schedule every single job at submit time. Default is 3 seconds 🫶
bb_array_stage_cnt — how many tasks from a job array should be available for burst buffer resource allocation. Higher values = more system overhead, because every one of those tasks gets moved to its own job record in memory 😭 so honestly, keep it relatively small. Default is 10.
bf_allow_magnetic_slot — bestie, by default the backfill scheduler will not add a slot in the bf plan when a job tries to use a magnetic reservation. This option flips that: now the backfill scheduler will add slots in the bf plan when jobs are eligible to run in a magnetic reservation. With it enabled, jobs inside magnetic reservations respect priorities and also get counted against the backfill limits like bf_max_job_test 🔥

NOTE: backfill evaluates jobs inside reservations first, which means all the magnetic jobs get tested first. So when you turn this on, make sure to revise (increase if necessary) the backfill limits you've configured, so your backfill cycle actually gets to test the jobs you expect in the queue 👑

NOTE: if bf_one_resv_per_job is used alongside this option, the magnetic reservation slot is now the only slot in the bf plan. Otherwise the slot will be the first one outside the magnetic reservation.

bf_busy_nodes — when picking resources for pending jobs to reserve for future execution (i.e. that job can't start right now), preferentially select nodes that are already in use. The vibe: leave currently idle resources available to backfill longer-running jobs 💅 but ngl, this may result in allocations having less-than-optimal network topology. Currently only supported by the select/cons_tres plugin.
bf_continue — the backfill scheduler periodically releases locks so other operations can proceed instead of blocking literally all activity for what could be an extended period. Setting this makes the backfill scheduler keep processing pending jobs from its original job list after releasing locks, even if job or node state changes. No cap, it just continues ✨
bf_hetjob_immediate — tells the backfill scheduler to try starting a heterogeneous job as soon as all of its components are determined able to do so. Otherwise, backfill delays hetero job initiation attempts until after the rest of the queue has been processed. That delay can let lower priority jobs grab resources, which could then delay the heterogeneous job's start due to account and/or QOS limits being reached 😬 Disabled by default. If you enable it and bf_hetjob_prio=min isn't set, then it gets set automatically.
bf_hetjob_prio=[min|avg|max] — at the start of each backfill scheduling cycle, the list of pending-to-be-scheduled jobs gets sorted by the precedence order configured in PriorityType. This option tells the scheduler to alter that sorting so all components belonging to the same heterogeneous job are attempted consecutively (so they don't get fragmented in the resulting list) 🧚 More specifically: all components from the same heterogeneous job are treated as if they all have the same priority (minimum, average, or maximum, depending on this option's parameter) when compared against other jobs (or other heterogeneous job components). The original order is preserved within the same heterogeneous job. Note that this operation is calculated for the PriorityTier layer and for the Priority resulting from the priority/multifactor plugin calculations. When enabled, if any heterogeneous job requested an advanced reservation, then all of that job's components are treated as if they had requested an advanced reservation (and get preferential treatment in scheduling) 🌸

Note that this operation does not update the Priority values of the heterogeneous job components — only their order within the list — so the output of the sprio command won't be affected.

Heterogeneous jobs have special scheduling properties, iykyk: they're only scheduled by the backfill scheduling plugin, each of their components is considered separately when reserving resources (and might have different PriorityTier or different Priority values), and no heterogeneous job component actually gets allocated resources until all of its components can be initiated. This may imply potential scheduling deadlock scenarios 💀 because components from different heterogeneous jobs can start reserving resources in an interleaved fashion (not consecutively), but none of the jobs can reserve resources for all components and start. Enabling this option can help mitigate that problem. By default, it's disabled.

bf_interval — the number of seconds between backfill iterations. Higher values = less overhead and better responsiveness. Applies only to SchedulerType=sched/backfill. Default: 30, Min: 1, Max: 10800 (3h). A setting of -1 will disable the backfill scheduling loop entirely 🦋
bf_job_part_count_reserve — the backfill scheduling logic reserves resources for the specified count of highest priority jobs in each partition. For example, bf_job_part_count_reserve=10 makes backfill reserve resources for the ten highest priority jobs in each partition. Any lower priority job that can start using currently available resources and not adversely impact the expected start time of those higher priority jobs will get started by the backfill scheduler ✨ The default value is zero, which reserves resources for any pending job and delays initiation of lower priority jobs. Also see bf_min_age_reserve and bf_min_prio_reserve. Default: 0, Min: 0, Max: 100000.
bf_licenses — requires the backfill scheduling logic to track and plan for license availability. By default, any job blocked on license availability won't have resources reserved, which can lead to job starvation 😭 Heads up: this option implicitly enables bf_running_job_reserve 👑
bf_max_job_array_resv — the maximum number of tasks from a job array for which the backfill scheduler will reserve resources in the future. Since job arrays can potentially have millions of tasks, the overhead of reserving resources for all of them can be prohibitive. Plus, various limits may prevent all the jobs from starting at the expected times. This has no impact on the number of tasks from a job array that can start immediately — only those tasks expected to start at some future time. Default: 20, Min: 0, Max: 1000.

NOTE: jobs submitted to multiple partitions appear in the job queue once per partition. If different copies of a single job array record aren't consecutive in the job queue and another job array record is in between, then bf_max_job_array_resv tasks are considered per partition that the job is submitted to.

bf_max_job_assoc — the maximum number of jobs per user association to attempt starting with the backfill scheduler. This is similar to bf_max_job_user but comes in clutch when a user has multiple associations that basically equate to different users. You can set this limit to stop users from flooding the backfill queue with jobs that can't start and that block jobs from other users from starting 🙈 Applies only to SchedulerType=sched/backfill. Also see the bf_max_job_user, bf_max_job_part, bf_max_job_test, and bf_max_job_user_part options. Set bf_max_job_test to a value much higher than bf_max_job_assoc. Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
bf_max_job_part — the maximum number of jobs per partition to attempt starting with the backfill scheduler. Especially helpful for systems with large numbers of partitions and jobs 🌈 Applies only to SchedulerType=sched/backfill. Also see the partition_job_depth and bf_max_job_test options. Set bf_max_job_test to a value much higher than bf_max_job_part. Default: 0 (no limit), Min: 0, Max: bf_max_job_test.
bf_max_job_start — the maximum number of jobs that can be initiated in a single iteration of the backfill scheduler. Applies only to SchedulerType=sched/backfill. Default: 0 (no limit), Min: 0, Max: 10000 🔥
bf_max_job_test — the max number of jobs the backfill scheduler will attempt to schedule, aka the queue depth. Higher values = more overhead and less responsiveness, so like… pick your fighter. Important deets: until an attempt is actually made to backfill schedule a job, its expected initiation time value will not be set. In the case of large clusters, configuring a relatively small value may be desirable. This option applies only to SchedulerType=sched/backfill.
- Default: 500, Min: 1, Max: 1,000,000
bf_max_job_user — the max number of jobs per user that the backfill scheduler will attempt to start, and this counts across ALL partitions. This is the "please stop flooding the backfill queue with jobs that literally cannot start and are blocking everyone else" knob 😭. It's similar to the MAXIJOB limit in Maui. This option applies only to SchedulerType=sched/backfill. Also see the bf_max_job_part, bf_max_job_test, and bf_max_job_user_part options. Set bf_max_job_test to a value much higher than bf_max_job_user.
- Default: 0 (no limit), Min: 0, Max: bf_max_job_test
bf_max_job_user_part — the max number of jobs per user per partition the backfill scheduler will attempt to start for any single partition 🎀. This option applies only to SchedulerType=sched/backfill. Also see the bf_max_job_part, bf_max_job_test, and bf_max_job_user options.
- Default: 0 (no limit), Min: 0, Max: bf_max_job_test
bf_max_time — the maximum time in seconds the backfill scheduler can spend (and yes, that includes time spent sleeping when locks are released 🫶) before it discontinues, even if the maximum job counts haven't been reached. This option applies only to SchedulerType=sched/backfill. The default value is the value of bf_interval (which defaults to 30 seconds).
- Default: bf_interval value (def. 30 sec), Min: 1, Max: 3600 (1h)
- > 💀 NOTE: If bf_interval is short and bf_max_time is large, this may cause locks to be acquired too frequently and starve out other serviced RPCs. It's advisable if using this parameter to set max_rpc_cnt high enough that scheduling isn't always disabled, and low enough that the interactive workload can get through in a reasonable period of time. max_rpc_cnt needs to be below 256 (the default RPC thread limit). Running around the middle (150) may give you good results.
- > 💀 NOTE: When increasing the amount of time spent in the backfill scheduling cycle, Slurm can be prevented from responding to client requests in a timely manner. To address this you can use max_rpc_cnt to specify a number of queued RPCs before the scheduler stops to respond to these requests.
bf_min_age_reserve — the backfill and main scheduling logic will not reserve resources for pending jobs until they've been pending and runnable for at least this many seconds ⏳. Bonus behavior: jobs waiting for less than the specified number of seconds will not prevent a newly submitted job from starting immediately, even if the newly submitted job has a lower priority. This can be valuable if jobs lack time limits or all time limits have the same value. The default value is zero, which will reserve resources for any pending job and delay initiation of lower priority jobs. Also see bf_job_part_count_reserve and bf_min_prio_reserve.
- Default: 0, Min: 0, Max: 2592000 (30 days)
bf_min_prio_reserve — the backfill and main scheduling logic will not reserve resources for pending jobs unless they have a priority equal to or higher than the specified value 👑. Same bonus as above: jobs with a lower priority will not prevent a newly submitted job from starting immediately, even if the newly submitted job has a lower priority. This can be valuable if one wished to maximize system utilization without regard for job priority below a certain threshold. The default value is zero, which will reserve resources for any pending job and delay initiation of lower priority jobs. Also see bf_job_part_count_reserve and bf_min_age_reserve.
- Default: 0, Min: 0, Max: 2^63
bf_node_space_size — size of the backfill node_space table 🌸. Adding a single job to backfill reservations in the worst case can consume two node_space records. In the case of large clusters, configuring a relatively small value may be desirable. This option applies only to SchedulerType=sched/backfill. Also see bf_max_job_test and bf_running_job_reserve.
- Default: bf_max_job_test, Min: 2, Max: 2,000,000
bf_one_resv_per_job — disallows adding more than one backfill reservation per job 🚫. Okay so the tea here: the scheduling logic builds a sorted list of job-partition pairs. Jobs submitted to multiple partitions have as many entries in the list as requested partitions. By default, the backfill scheduler may evaluate all the job-partition entries for a single job, potentially reserving resources for each pair, but only starting the job in the reservation offering the earliest start time. Having a single job reserving resources for multiple partitions could impede other jobs (or hetjob components) from reserving resources already reserved for the partitions that don't offer the earliest start time. A single job that requests multiple partitions can also prevent itself from starting earlier in a lower priority partition if the partitions overlap nodes and a backfill reservation in the higher priority partition blocks nodes that are also in the lower priority partition 😵‍💫. This option makes it so that a job submitted to multiple partitions will stop reserving resources once the first job-partition pair has booked a backfill reservation. Subsequent pairs from the same job will only be tested to start now. This allows for other jobs to be able to book the other pairs' resources, at the cost of not guaranteeing that the multi partition job will start in the partition offering the earliest start time (unless it can start immediately). This option is disabled by default.
bf_resolution — the number of seconds in the resolution of data maintained about when jobs begin and end ⏱️. Higher values result in better responsiveness and quicker backfill cycles by using larger blocks of time to determine node eligibility. However, higher values lead to less efficient system planning, and may miss opportunities to improve system utilization. This option applies only to SchedulerType=sched/backfill.
- Default: 60, Min: 1, Max: 3600 (1 hour)
bf_running_job_reserve — adds an extra step to the backfill logic, which creates backfill reservations for jobs running on whole nodes ✨. This option is disabled by default.
bf_topopt_enable — enables the experimental hook to control whether to delay jobs in backfill for a better placement 🧪. Modify src/plugins/sched/backfill/oracle.c for testing.

It is recommended to disable the main scheduler so that all jobs are planned through backfill and utilize the oracle function(). This can be done by setting SchedulerParameters=sched_interval=-1.

It's also recommended to run with SchedulerParameters=bf_running_job_reserve for better planning.

bf_topopt_iterations — the number of successive backfill map slots that a job may be delayed 🦋. This option applies only when the bf_topopt_enable is set.
bf_window — the number of minutes into the future to look when considering jobs to schedule 🔮. Higher values result in more overhead and less responsiveness. A value at least as long as the highest allowed time limit is generally advisable to prevent job starvation. In order to limit the amount of data managed by the backfill scheduler, if the value of bf_window is increased, then it is generally advisable to also increase bf_resolution. This option applies only to SchedulerType=sched/backfill.
- Default: 1440 (1 day), Min: 1, Max: 43200 (30 days)
bf_window_linear — okay so for performance reasons, the backfill scheduler will decrease precision when it calculates job expected termination times. By default the precision starts at 30 seconds, and that time interval doubles with each evaluation of currently executing jobs when trying to figure out when a pending job can start. This algorithm can totally support an environment with many thousands of running jobs, but it can result in the expected start time of pending jobs being gradually deferred because of that lack of precision 😭 A value for bf_window_linear will cause the time interval to be increased by a constant amount on each iteration. The value is specified in units of seconds. For example, a value of 60 will cause the backfill scheduler on the first iteration to identify the job ending soonest and determine if the pending job can be started after that job plus all other jobs expected to end within 30 seconds (default initial value) of the first job. On the next iteration, the pending job will be evaluated for starting after the next job expected to end plus all jobs ending within 90 seconds of that time (30 second default, plus the 60 second option value). The third iteration will have a 150 second window and the fourth 210 seconds. Without this option, the time windows will double on each iteration and thus be 30, 60, 120, 240 seconds, etc. The use of bf_window_linear is not recommended with more than a few hundred simultaneously executing jobs 💅
bf_yield_interval — the backfill scheduler will periodically relinquish locks so other pending operations can take place. This specifies the times when the locks are relinquished, in microseconds. Smaller values may be helpful for high throughput computing when used in conjunction with the bf_continue option. Also see the bf_yield_sleep option. Default: 2,000,000 (2 sec), Min: 1, Max: 10,000,000 (10 sec).
bf_yield_rpc_cnt — if the number of active threads in the slurmctld daemon is lower than this value, continue scheduling of jobs. The scheduler will check this condition at certain points in code and release previously yielded locks if necessary. This is used to instruct Slurm when to stop processing requests and start scheduling new jobs, which is the opposite of max_rpc_cnt. In conjunction with max_rpc_cnt, it can improve Slurm's responsiveness to spikes of requests. Default: MAX((max_rpc_cnt / 10), 20) (option disabled), Min: 0, Max: 200.

NOTE: if a value is set, then a value lower than max_rpc_cnt is recommended. It may require some tuning for each system, but needs to be high enough that scheduling isn't always disabled, and low enough that requests can get through in a reasonable period of time. Avoid both values being close enough to cause continuous switching between request processing and scheduling 🫶

bf_yield_sleep — the backfill scheduler will periodically relinquish locks so other pending operations can take place. This specifies the length of time for which the locks are relinquished, in microseconds. Also see the bf_yield_interval option. Default: 500,000 (0.5 sec), Min: 1, Max: 10,000,000 (10 sec).
build_queue_timeout — defines the maximum time that can be devoted to building a queue of jobs to be tested for scheduling. If the system has a huge number of jobs with dependencies, just building the job queue can take so much time as to adversely impact overall system performance, and this parameter can be adjusted as needed. The default value is 2,000,000 microseconds (2 seconds). ⏳
correspond_after_task_cnt — defines the number of array tasks that get split for potential aftercorr dependency check. A low number may result in dependent task check failures when the job one depends on gets purged before the split 😬 Default: 10.
default_queue_depth — the default number of jobs to attempt scheduling (i.e. the queue depth) when a running job completes or other routine actions occur, however the frequency with which the scheduler is run may be limited by using the defer or sched_min_interval parameters described below. The main scheduling loop will run (ignoring this limit) on a less frequent basis as defined by the sched_interval option described below. The default value is 100. See the partition_job_depth option to limit depth by partition.
defer — setting this option will avoid attempting to schedule each job individually at job submit time, but defer it until a later time when scheduling multiple jobs simultaneously may be possible. This option may improve system responsiveness when large numbers of jobs (many hundreds) are submitted at the same time, but it will delay the initiation time of individual jobs. Also see default_queue_depth above.
defer_batch — like defer, but only will defer scheduling for batch jobs. Interactive allocations from salloc/srun will still attempt to schedule immediately upon submission ✨
delay_boot — do not reboot nodes in order to satisfy this job's feature specification if the job has been eligible to run for less than this time period. If the job has waited for less than the specified period, it will use only nodes which already have the specified features. The argument is in units of minutes. Individual jobs may override this default value with the --delay-boot option.
disable_job_shrink — deny user requests to shrink the size of running jobs. (However, running jobs may still shrink due to node failure if the --no-kill option was set.) 🚫
disable_hetjob_steps — disable job steps that span heterogeneous job allocations.
enable_hetjob_steps — enable job steps that span heterogeneous job allocations. The default value 👑
enable_user_top — enable use of the "scontrol top" command by non-privileged users.
extra_constraints — enable node filtering with the --extra option for salloc, sbatch, and srun and the node's Extra field.
ignore_constraint_validation — if set, and a job requests --constraint, any features in the request that would create an invalid request with the current system will not generate an error. This is helpful for dynamic systems where nodes with features come and go. Jobs will remain in the job queue until the requested feature is in the cluster and available. Please note using this option will not protect you from typos 🙈 See also ignore_prefer_validation.
Ignore_NUMA — some processors (e.g. AMD Opteron 6000 series) contain multiple NUMA nodes per socket. This is a configuration which does not map into the hardware entities that Slurm optimizes resource allocation for (PU/thread, core, socket, baseboard, node and network switch). In order to optimize resource allocations on such hardware, Slurm will consider each NUMA node within the socket as a separate socket by default. Use the Ignore_NUMA option to report the correct socket count, but not optimize resource allocations on the NUMA nodes.

NOTE: since hwloc 2.0 NUMA Nodes are not part of the main/CPU topology tree, because of that if Slurm is build with hwloc 2.0 or above Slurm will treat HWLOC_OBJ_PACKAGE as Socket, you can change this behavior using SlurmdParameters=l3cache_as_socket. 🦋

ignore_prefer_validation — if set, and a job requests --prefer, any features in the request that would create an invalid request with the current system will not generate an error. This is helpful for dynamic systems where nodes with features come and go. Please note using this option will not protect you from typos 🙈 See also ignore_constraint_validation.
max_array_tasks — specify the maximum number of tasks that can be included in a job array. The default limit is MaxArraySize, but this option can be used to set a lower limit. For example, max_array_tasks=1000 and MaxArraySize=100001 would permit a maximum task ID of 100000, but limit the number of tasks in any single job array to 1000. 💅
max_rpc_cnt — if the number of active threads in the slurmctld daemon is equal to or larger than this value, defer scheduling of jobs. The scheduler will check this condition at certain points in code and yield locks if necessary. This can improve Slurm's ability to process requests at a cost of initiating new jobs less frequently. Default: 0 (option disabled), Min: 0, Max: 1000.

okay bestie, quick heads-up on this one before we go: the max number of threads (MAX_SERVER_THREADS) is hard-set inside to 256, and that's literally how many RPCs can be served at any given moment. So if you set max_rpc_cnt higher than 256, the only thing it actually buys you is letting backfill keep scheduling stuff after locks get yielded (that's every 2 seconds, js) — and only if there are at most MAX(max_rpc_cnt/10, 20) RPCs sitting in the queue. Example time: max_rpc_cnt=1000 means the scheduler is only allowed to keep going after yielding locks when there are 100 or fewer pending RPCs. If you're setting a value at all, 10 or higher is the rec 💅. It's gonna need some tuning per system ngl — high enough that scheduling isn't permanently off, low enough that requests actually get through in a reasonable amount of time. No cap.

max_sched_time — how long, in seconds, the main scheduling loop runs before it dips out. Okay so real talk: every other Slurm operation, including responding to RPCs, gets deferred while the scheduling loop is doing its thing.

Default is 2 seconds in most cases, or 1 second if MessageTimeout is set to 3 or less. Maximum value is half of MessageTimeout. If you put in anything bigger, it just gets swapped for the applicable default value. 🙈

NOTE: a higher value lets the main scheduler evaluate more pending jobs per cycle. But if this timeout is reached, some low priority pending jobs won't get evaluated — which can leave their reason for pending as 'None' until a future scheduling cycle comes around to them. 😭

max_script_size — specify the maximum size of a batch script, in bytes. Default is 4 megabytes. Bigger values may adversely impact system performance. (translating: don't go feral, k?)
max_submit_line_size — specify the maximum size of a submit line, in bytes. Default is 1 megabtye. (yes that typo is canon, we're keeping her 🎀) This option cannot exceed 2 megabytes. Hard stop.
max_switch_wait — maximum number of seconds a job can delay execution while it's waiting for the specified desired switch count. Default value is 300 seconds.
no_backup_scheduling — if used, the backup controller will not schedule jobs when it takes over. It'll still let jobs be submitted, modified and cancelled, but it won't schedule new ones. This one's useful in Cray environments when the backup controller resides on an external Cray node. 🌈
nohold_on_prolog_fail — by default, if the Prolog exits with a non-zero value the job is requeued in a held state. Specify this parameter and the job will still be requeued, just not held — so the scheduler can dispatch it to another host. 🦋
pack_serial_at_end — if used with the select/cons_tres plugin, serial jobs get put at the end of the available nodes instead of using a best fit algorithm. This may reduce resource fragmentation for some workloads. she's tidy!
partition_job_depth — the default number of jobs to attempt scheduling (i.e. the queue depth) from each partition/queue in Slurm's main scheduling logic. This limit is enforced for all main scheduler cycles. Functionality is similar to what bf_max_job_part does for the backfill scheduling logic. Default value is 0 (no limit, go wild). Jobs excluded from attempted scheduling based on partition will not be counted against the default_queue_depth limit. Also see the bf_max_job_part option. 👑
reduce_completing_frag — controls how scheduling of resources happens when jobs are in the COMPLETING state, which influences potential fragmentation. If this option is not set, then no jobs will be started in any partition when any job is in the COMPLETING state for less than CompleteWait seconds. If this option is set, then no jobs will be started in any individual partition that has a job in COMPLETING state for less than CompleteWait seconds. On top of that, no jobs will be started in any partition with nodes that overlap with any nodes in the partition of the completing job. This option is meant to be used in conjunction with CompleteWait.

NOTE: CompleteWait must be set or this whole thing does nothing. If CompleteWait=0 then this option does nothing at all. 💀

NOTE: reduce_completing_frag only affects the main scheduler — not the backfill scheduler. periodt.

requeue_delay — delay before a non-Expedited Requeue job is eligible to run after being requeued. Defaults to the AuthInfo=cred_expire setting, which itself defaults to 120 seconds. iykyk.
requeue_on_resume_failure — in the event that nodes fail to resume by ResumeTimeout, all batch jobs will be requeued — even if the jobs requested not to be requeued. This is similar to PrologFlags=ForceRequeueOnFail. lowkey dramatic but necessary 😭
salloc_wait_nodes — if defined, the salloc command will wait until all allocated nodes are ready for use (i.e. booted) before the command returns. By default, salloc returns as soon as the resource allocation has been made. The salloc command can use the --wait-all-nodes option to override this configuration parameter. 🫶
sbatch_wait_nodes — if defined, the sbatch script will wait until all allocated nodes are ready for use (i.e. booted) before the initiation. By default, the sbatch script is initiated as soon as the first node in the job allocation is ready. The sbatch command can use the --wait-all-nodes option to override this configuration parameter. ✨
sched_interval — how frequently, in seconds, the main scheduling loop executes and tests all pending jobs, with only the partition_job_depth limit in place. Default value is 60 seconds. A setting of -1 disables the main scheduling loop entirely. 🧚
sched_max_job_start — the maximum number of jobs the main scheduling logic will start in any single execution. Default value is zero, which imposes no limit. main character energy fr.
sched_min_interval — how frequently, in microseconds, the main scheduling loop executes and tests any pending jobs. The scheduler runs in a limited fashion every time any event happens that could enable a job to start (e.g. job submit, job terminate, etc.). If those events happen at a high frequency, the scheduler can run very frequently and consume significant resources — unless it's throttled by this option. This option specifies the minimum time between the end of one scheduling cycle and the beginning of the next scheduling cycle. A value of zero disables throttling of the scheduling logic interval. Default value is 2 microseconds. tiny but mighty 🍓
spec_cores_first — specialized cores will be selected from the first cores of the first sockets, cycling through the sockets on a round robin basis. By default, specialized cores are selected from the last cores of the last sockets, cycling through the sockets on a round robin basis. flip the script!
step_retry_count — when a step completes and there are steps ending resource allocation, retry step allocations for at least this number of pending steps. Also see step_retry_time. Default value is 8 steps. 🌸
step_retry_time — when a step completes and there are steps ending resource allocation, retry step allocations for all steps which have been pending for at least this number of seconds. Also see step_retry_count. Default value is 60 seconds. ⭐
time_min_as_soft_limit — treat the --time-min limit as a soft time limit for the job. Scheduling will plan for the shorter duration, while permitting the job to continue running until the ("hard") --time limit. bestie gets to stay out past curfew 💅
whole_hetjob — requests to cancel, hold or release any component of a heterogeneous job will be applied to all components of the job.

NOTE: this option was previously named whole_pack and that name is still supported for backwards compatibility. we love a glow-up that doesn't break your config 💖

✨ `SchedulerTimeSlice` ✨

Number of seconds in each time slice when gang scheduling is enabled (PreemptMode=SUSPEND,GANG). The value must be between 5 seconds and 65533 seconds. Default value is 30 seconds. slay within bounds 🎀

✨ `SchedulerType` ✨

Identifies the type of scheduler to be used. The scontrol command can be used to manually change job priorities if that's your vibe. Acceptable values include:

sched/backfill — for a backfill scheduling module to augment the default FIFO scheduling. Backfill scheduling will initiate lower-priority jobs if doing so does not delay the expected initiation time of any higher priority job. Effectiveness of backfill scheduling is dependent upon users specifying job time limits — otherwise all jobs will have the same time limit and backfilling is impossible (😭 the whole thing falls apart). Note documentation for the SchedulerParameters option above. This is the default configuration. 👑
sched/builtin — the FIFO scheduler, bestie, and she is very literal: she starts jobs in priority order. Periodt. If any job in the partition can't be scheduled, then no lower priority job in that partition gets scheduled either. The one exception? Jobs that literally cannot run because of partition constraints (think: the time limit) or because nodes are down/drained — in that case, lower priority jobs can be initiated and they won't impact the higher priority job. Small mercies 💅. Setting this scheduler type will disable heterogeneous jobs, since those are handled by the backfill scheduler instead. Not her department, she doesn't do that.

🌸 ScronParameters 🌸

Multiple options may be comma separated.

enable — yes, enable the use of scrontab to submit and manage periodic repeating jobs ✨
explicit_scancel — when cancelling an scrontab job, the user has to explicitly ask to cancel it with the --cron flag in scancel. No accidental cancellations, no chaos, we love boundaries 🫶

👑 SelectType 👑

This one identifies the type of resource selection algorithm to be used. And listen — when this is changed, all job information (running and pending) will be lost, because the job state save format used by each plugin is different. 😭 The only exception to this is when changing from the legacy cons_res to cons_tres.

Acceptable values include:

select/cons_tres — the resources (cores, memory, GPUs and all other trackable resources) within a node are individually allocated as consumable resources. Note that whole nodes can be allocated to jobs for selected partitions by using the partition Exclusive parameter (e.g. Exclusive=NODE). See Exclusive and OverSubscribe for more info. This is the default value. 👑
select/linear — for allocation of entire nodes, assuming a one-dimensional array of nodes in which sequentially ordered nodes are preferable. For a heterogeneous cluster (e.g. different CPU counts on the various nodes), resource allocations will favor nodes with high CPU counts as needed based upon the job's node and CPU specification if TopologyPlugin=topology/flat is configured. Use of other topology plugins with select/linear and heterogeneous nodes is not recommended and may result in valid job allocation requests being rejected 🚩. The linear plugin is not designed to track generic resources on a node. In cases where generic resources (such as GPUs) need to be tracked, the cons_tres plugin should be used instead.

🦋 SelectTypeParameters 🦋

This controls how the SelectType algorithm will allocate nodes to jobs. Multiple comma-separated options may be provided as long as no conflicts described below are present. The permitted values depend upon the configured value of SelectType.

By default SelectType=select/linear allocates whole nodes to jobs without considering their memory consumption. The only supported options for select/linear are CR_ONE_TASK_PER_CORE (described below) and CR_Memory, which treats memory as a consumable resource and prevents memory oversubscription with job preemption or gang scheduling.

By default SelectType=select/cons_tres uses CR_Core_Memory, which allocates cores to jobs while considering their memory consumption.

Any one of the following options may be used with the select/cons_tres plugin to determine whether CPU resources are consumed by socket, core, or logical CPU, and whether memory is consumed:

CR_CPU — CPUs are consumable resources. Configure the number of CPUs on each node, which may be equal to the count of cores or hyper-threads on the node depending upon the desired minimum resource allocation. The node's Boards, Sockets, CoresPerSocket, and ThreadsPerCore may optionally be configured and result in job allocations which have improved locality; however doing so will prevent more than one job from being allocated on each core.
CR_CPU_Memory — CPUs and memory are consumable resources. Refer to CR_CPU for details of how CPUs are consumed. Setting a value for DefMemPerCPU is strongly recommended 💖
CR_Core — cores are consumable resources. On nodes with hyper-threads, each thread is counted as a CPU to satisfy a job's resource requirement, but multiple jobs are not allocated threads on the same core. The count of CPUs allocated to a job is rounded up to account for every CPU on an allocated core, which may also impact total allocated memory if it is derived from --mem-per-cpu.
CR_Core_Memory — cores and memory are consumable resources. Refer to CR_Core for details of how cores are consumed. Setting a value for DefMemPerCPU is strongly recommended 💖
CR_Socket — sockets are consumable resources. On nodes with multiple cores, each core or thread is counted as a CPU to satisfy a job's resource requirement, but multiple jobs are not allocated resources on the same socket.
CR_Socket_Memory — sockets and memory are consumable resources. Refer to CR_Socket for details of how sockets are consumed. Setting a value for DefMemPerCPU is strongly recommended 💖

Any number of the following options may be used with the select/cons_tres plugin to further adjust selection behavior:

CR_ONE_TASK_PER_CORE — allocate one task per core by default. Without this option, by default one task will be allocated per thread on nodes with more than one ThreadsPerCore configured.

NOTE: This option cannot be used with CR_CPU*.

CR_CORE_DEFAULT_DIST_BLOCK — allocate cores within a node using block distribution by default. This is a pseudo-best-fit algorithm that minimizes the number of boards and minimizes the number of sockets (within minimum boards) used for the allocation. This default behavior can be overridden by specifying a particular --distribution option at job submission. Without this option, cores will be allocated cyclically across the sockets.
CR_LLN — schedule resources to jobs on the least loaded nodes (based upon the number of idle CPUs). This is generally only recommended for an environment with serial jobs, since idle resources will tend to be highly fragmented, resulting in parallel jobs being distributed across many nodes 😬. Note that node Weight takes precedence over how many idle resources are on each node. Also see the partition configuration parameter LLN to use the least loaded nodes in selected partitions.
CR_Pack_Nodes — if a job allocation contains more resources than will be used for launching tasks (e.g. if whole nodes are allocated to a job), then rather than distributing a job's tasks evenly across its allocated nodes, pack them as tightly as possible on these nodes. For example, consider a job allocation containing two entire nodes with eight CPUs each. If the job starts ten tasks across those two nodes without this option, it will start five tasks on each of the two nodes. With this option, eight tasks will be started on the first node and two tasks on the second node. This only applies when the --distribution=block option is used. This can be superseded by setting the NoPack distribution option.
LL_SHARED_GRES — when allocating resources for a shared GRES (gres/mps, gres/shard), prefer the least loaded device (in terms of already allocated fraction). This way, jobs are spread across GRES devices on the node, instead of the default behavior where the first available device is used. Spread the love 🦋
MULTIPLE_SHARING_GRES_PJ — by default, only one sharing GRES per job is allowed on each node from shared GRES requests. This allows multiple sharing GRES to be used on a single node to satisfy shared GRES requirements per job. Example: if there are 10 shards to a GPU and 12 shards are requested, instead of being denied the job will be allocated with 2 GPUs: 1 using 10 shards and the other using 2 shards. Iconic behavior, honestly 👑
ENFORCE_BINDING_GRES — set --gres-flags=enforce-binding as the default in every job. This can be overridden with --gres-flags=disable-binding.
ONE_TASK_PER_SHARING_GRES — set --gres-flags=one-task-per-sharing as the default in every job. This can be overridden with --gres-flags=multiple-tasks-per-sharing.
CR_NO_DIST_TOPO_BLOCK — disable topology-aware task distribution for steps. By default, when a topology plugin is configured, tasks are distributed to nodes following topology order. This option reverts to the standard bitmap order.

NOTE: bestie, read this one carefully — if memory isn't configured as a consumable resource (CR_CPU, CR_Core, or CR_Socket without _Memory), memory can be oversubscribed and will not be constrained by task/cgroup even if it is configured in cgroup.conf. In this case the --mem option is only used to filter out nodes with lower configured memory and does not take running jobs into account. For instance, two jobs requesting all the memory of a node can run at the same time. 💀 Yes, really. That's not a drill.

💅 SlurmctldAddr

An optional address used for talking to whichever slurmctld is currently the active one — normally you'd use this with Virtual IP addressing of the active server, iykyk. If you don't set it, then each primary and backup server gets its own unique address for communications, exactly as spelled out in the SlurmctldHost parameter. And if you do set it? SlurmctldHost is still used for talking to specific slurmctld primary or backup servers — like, say, telling all of them to go read the current config files or shut down. 🫶

Also peep the SlurmctldPrimaryOffProg and SlurmctldPrimaryOnProg configuration parameters if you wanna set up programs to do virtual IP address manipulation. ✨

🦋 SlurmctldDebug

The level of detail you get in the slurmctld daemon's logs. Default value is info. If the slurmctld daemon got started with -v or --verbose, that debug level is preserved or restored upon reconfiguration (yes bestie, it remembers). 🎀

quiet — log literally nothing
fatal — log only fatal errors
error — log only errors
info — log errors and general informational messages
verbose — log errors and verbose informational messages
debug — log errors and verbose informational messages and debugging messages
debug2 — log errors and verbose informational messages and more debugging messages
debug3 — log errors and verbose informational messages and even more debugging messages
debug4 — log errors and verbose informational messages and even more debugging messages
debug5 — log errors and verbose informational messages and even more debugging messages

🌸 SlurmctldHost

The short, or long, hostname of the machine where the Slurm control daemon runs — as in, the name returned by hostname -s. This hostname is optionally followed by either the IP address or a name that identifies the address, all wrapped in parentheses. Like:

SlurmctldHost=slurmctl-primary(12.34.56.78)

Every host running an instance of slurmctld should have a SlurmctldHost= entry of its own, fr fr:

SlurmctldHost=slurmctl-primary1
SlurmctldHost=slurmctl-primary2
SlurmctldHost=slurmctl-primary3(12.34.56.78)

SlurmctldHost must be specified at least once. Specify it more than once and the first entry runs as the primary while all the others are standby backups — so first entry = main character energy. 👑 If the primary host fails, the first backup flips from standby to primary until the first host comes back online. And if the new primary fails? Same whole process again.

Slurm daemons need to be reconfigured (like, scontrol reconfig) for changes to this parameter to actually take effect. It's totally okay for jobs to be running while you make these changes — the running steps will get the updated SlurmctldHost info. 💖

Every slurmctld host controller needs access to the StateSaveLocation directory, and it must be readable and writable from the primary and all backup controllers at all times. No exceptions.

If you need to change this, go read the RELOCATING CONTROLLERS section. 📖

🧚 SlurmctldHttpAuthParameters

A comma-separated list of options that control how the slurmctld HTTP listener authenticates incoming requests. The recognized keys are the same JWT-related keys documented under AuthAltParameters — same keys, different party. When this is set, independent per-daemon authentication keys are loaded from these parameters, and tokens presented to the slurmctld HTTP endpoints get verified against these configured keys instead of against AuthAltParameters. 🔥

🍓 SlurmctldLogFile

Fully qualified pathname of the file the slurmctld daemon's logs get written into. Default value is none, which means it does its logging via syslog.

See the section LOGGING if you specify a pathname. ✨

🌈 SlurmctldParameters

Multiple options may be comma separated. Let's go, one by one:

💜 `allow_user_triggers`

Permit setting triggers from non-root/slurm_user users. Note that SlurmUser must also be set to root for these triggers to actually work. See the strigger man page for the extra details. 🎀

☁️ `cloud_dns`

By default, Slurm expects that the network address for a cloud node won't be known until the node gets created, and that Slurm will be notified of the node's address (e.g. scontrol update nodename=<name> nodeaddr=<addr>). Since Slurm communications rely on the node configuration found in the slurm.conf, Slurm will tell the client command each node's ip address after waiting for all nodes to boot. But in environments where the nodes are in DNS, you can skip that whole step by configuring this option. Slay. ✨

🔌 `conmgr_connect_timeout=<seconds>`

Wait <seconds> before considering an outbound connection attempt timed out. Defaults to the value of MessageTimeout.

🫧 `conmgr_max_connections=<connection_count>`

The maximum number of connections to be processed at any given time. Heads up: this does not influence the maximum number of pending connections, because the kernel runs that show. Increasing this value increases slurmctld's memory footprint, and sites are advised to monitor memory consumption when bumping it. Defaults to 512.

⏳ `conmgr_quiesce_timeout=<seconds>`

Wait <seconds> before considering quiesce timed out. Upon timeout, all (non-listening) active connections get closed so the quiesce can start. Defaults to two times the value of MessageTimeout.

📖 `conmgr_read_timeout=<seconds>`

Wait <seconds> before considering a read from a file descriptor timed out. Defaults to the value of MessageTimeout.

🧵 `conmgr_threads=<thread_count>`

The number of process threads in the thread pool used for receiving and processing connections on the listening sockets. In most cases you should decrease conmgr_threads to improve scheduling performance. Counterintuitive but iconic. 💅

👀 `conmgr_use_poll`

Use poll(2) instead of epoll(7) for monitoring file descriptors.

🐢 `conmgr_wait_write_delay=<seconds>`

When waiting for the kernel to flush the outgoing buffer, poll the kernel for changes every <seconds>. Defaults to the value of MessageTimeout.

✍️ `conmgr_write_timeout=<seconds>`

Wait <seconds> before considering a write from a file descriptor timed out. Defaults to the value of MessageTimeout.

🚫 `disable_triggers`

Disable the ability to register new triggers. That's it, that's the whole vibe.

⚡ `enable_async_reply`

Enable slurmctld to reply to incoming (supported) RPCs asynchronously, without blocking a thread in the conmgr thread pool. Multitasking queen. 👑

🪄 `enable_configless`

Permit "configless" operation by the slurmd, slurmstepd, and user commands. When enabled, the slurmd is permitted to retrieve config files plus Prolog, Epilog, TaskProlog, and TaskEpilog scripts from the slurmctld, and on any scontrol reconfigure command the new configs and scripts get automatically pushed out and applied to nodes running in this "configless" mode. More details at https://slurm.schedmd.com/configless_slurm.html.

NOTE: Included files with the Include directive will only be pushed if the filename has no path separators and is located adjacent to slurm.conf.

NOTE: Prolog and Epilog scripts will only be pushed if the filenames have no path separators and are located adjacent to slurm.conf. Glob patterns (see glob(7)) are not supported. 😭

🏃‍♀️ `enable_expedited_requeue`

Allow jobs to request an expedited requeue on certain events. An expedited requeue ensures the job is immediately eligible to run and gets placed at the top of the queue. Cutting the line, but make it legal.

🪜 `enable_stepmgr`

Enable the slurmstepd step management system wide. This enables job steps to be managed by a single extern slurmstepd associated with the job to manage steps — big benefit for jobs that submit many steps inside their allocations. PrologFlags=contain must be set. Required, bestie.

😴 `idle_on_node_suspend`

Mark nodes as idle, regardless of their current state, when suspending nodes with SuspendProgram — that way the nodes stay eligible to be resumed at a later time. Napping, but still bookable. 🦋

🗑️ `max_dbd_msg_action`

The action used once MaxDBDMsgs is reached. Options are 'discard' (the default) and 'exit'.

When 'discard' is specified and MaxDBDMsgs is reached, we start by purging pending messages of types Step start and complete. If it reaches MaxDBDMsgs again, Job start messages get purged. Job completes and node state changes keep consuming the empty space created from the purgings until MaxDBDMsgs is reached again — at which point no new message is tracked, creating data loss and potentially runaway jobs. 💀 Yeah. That.
When 'exit' is specified and MaxDBDMsgs is reached, the slurmctld exits instead of discarding any messages. With this option it will be impossible to start the slurmctld where the slurmdbd is down and the slurmctld is tracking more than MaxDBDMsgs.

🔋 `max_powered_nodes`

The max number of powered up nodes across the cluster. Once you hit it, jobs requesting additional nodes will not start, and scontrol power up <nodes> will fail. That's the ceiling, periodt.

node_reg_mem_percent — the percentage of memory a node is allowed to register with before it gets flagged as invalid with low memory. Default is 100. For State=CLOUD nodes the default is 90. Want to disable it for cloud nodes? Set it to 100. And heads up: config_overrides takes precedence over this option, always 💅

Also, gentle recommendation: configure task/cgroup with ConstrainRamSpace. A memory cgroup limit won't be set higher than the actual memory on the node, so if you need a buffer, configure AllowedRamSpace in cgroup.conf. 🫶

no_quick_restart — okay so by default, starting a new instance of slurmctld will kill the old one that's running before taking control. If this option is set, that will not happen without the -i option. That's it, that's the whole vibe.
power_save_interval — how often the power_save thread looks to resume and suspend nodes. The power_save thread will do work sooner if there are node state changes. Default is 10 seconds.
power_save_min_interval — how often the power_save thread looks to resume and suspend nodes, at a minimum. Default is 0.
reboot_from_controller — run the RebootProgram from the controller instead of on the slurmd's. The RebootProgram gets a comma-separated list of nodes to reboot as the first argument, and if applicable, the required features needed for reboot as the second argument. If RebootProgram is set to a PowerAction, this parameter is ignored in favor of the PowerAction Location. iykyk 🎀
reconfig_on_restart — every restart of slurmctld (and we mean a process restart, not one triggered by "scontrol reconfigure") will trigger a reconfiguration request to all slurmd and sackd daemons. For "configless" systems this ensures all processes are running the current configuration. ✨

NOTE: Use with caution when performing rolling upgrades — this could inadvertently trigger an upgrade of the slurmd or sackd daemons earlier than intended. 😭 seriously, be careful.

rl_bucket_size= — size of the token bucket. This permits a certain amount of RPC burst from a user before the steady-state rate limit takes effect. The default value is 30.
rl_enable — enable per-user RPC rate-limiting support. Client-commands will be told to back off and sleep for a second once the limit has been reached. It's implemented as a "token bucket" (yes, that's the real name 🪣) which permits a certain degree of "bursty" RPC load from an individual user before holding them to a steady-state RPC load established by the refill period and rate.
rl_log_freq= — the maximum frequency (in seconds) for which logs about the RPC limit being exceeded by an individual user are printed to the logs. Set to 0 to see every incidence. Set to -1 to disable the log message entirely. The default value is 0.
rl_refill_period= — how frequently, in seconds, additional tokens are added to each user bucket. The default value is 1.
rl_refill_rate= — how many tokens to add to the bucket on each period. The default value is 2.
rl_table_size= — number of entries in the user hash-table. The recommended value should be at least twice the number of active user accounts on the system. The default value is 8192.
user_resv_delete — allow any user who's able to run in a reservation to delete it. 🫡
validate_nodeaddr_threads= — during startup, slurmctld looks up the address for each compute node in the system. On large systems this can cause considerable delay, so this option lets slurmctld concurrently handle the lookup calls and can reduce system startup time considerably. The default value is 1. Maximum permitted value is 64.

SlurmctldPidFile — fully qualified pathname of a file into which the slurmctld daemon may write its process id. This may be used for automated signal processing. The default value is "/var/run/slurmctld.pid". 🎀
SlurmctldPort — the port number that the Slurm controller, slurmctld, listens to for work. The default value is SLURMCTLD_PORT as established at system build time. If none is explicitly specified, it will be set to 6817. SlurmctldPort may also be configured to support a range of port numbers for existing legacy configurations. 🔥

NOTE: Either slurmctld and slurmd daemons must not execute on the same nodes, or the values of SlurmctldPort and SlurmdPort must be different.

NOTE: On Cray systems, Realm-Specific IP Addressing (RSIP) will automatically try to interact with anything opened on ports 8192-60000. Configure SlurmctldPort to use a port outside of the configured SrunPortRange and RSIP's port range.

SlurmctldPrimaryOffProg — this program is executed when a slurmctld daemon running as the primary server becomes a backup server. The controller will wait for this script to end before fully shutting down. By default no program is executed. See also the related "SlurmctldPrimaryOnProg" parameter. 🦋
SlurmctldPrimaryOnProg — this program is executed when a slurmctld daemon running as a backup server becomes the primary server. The controller will wait for this script to end before fully starting up. By default no program is executed. When you're using virtual IP addresses to manage High Available Slurm services, this program can be used to add the IP address to an interface (and optionally try to kill the unresponsive slurmctld daemon and flush the ARP caches on nodes on the local Ethernet fabric 👑). See also the related "SlurmctldPrimaryOffProg" parameter.
SlurmctldSyslogDebug — the slurmctld daemon will log events to the syslog file at the specified level of detail. If not set, the slurmctld daemon will log to syslog at level fatal, unless there is no SlurmctldLogFile and it's running in the background — in which case it will log to syslog at the level specified by SlurmctldDebug (at fatal in the case that SlurmctldDebug is set to quiet) — or it's run in the foreground, when it will be set to quiet.
- quiet — log nothing
- fatal — log only fatal errors
- error — log only errors
- info — log errors and general informational messages
- verbose — log errors and verbose informational messages
- debug — log errors and verbose informational messages and debugging messages
- debug2 — log errors and verbose informational messages and more debugging messages
- debug3 — log errors and verbose informational messages and even more debugging messages
- debug4 — log errors and verbose informational messages and even more debugging messages
- debug5 — log errors and verbose informational messages and even more debugging messages

NOTE: By default, Slurm's systemd service file starts the slurmctld daemon in the foreground with the --systemd option. This means systemd will capture stdout/stderr output and print that to syslog, independent of Slurm printing to syslog directly. To prevent systemd from doing this, add "StandardOutput=null" and "StandardError=null" to the respective service files or override files. 💅

SlurmctldTimeout — the interval, in seconds, that the backup controller waits for the primary controller to respond before assuming control. The default value is 120 seconds. May not exceed 65533. ⏳
SlurmdDebug — the level of detail to provide slurmd daemon's logs. The default value is info.
- quiet — log nothing
- fatal — log only fatal errors
- error — log only errors
- info — log errors and general informational messages
- verbose — log errors and verbose informational messages
- debug — log errors and verbose informational messages and debugging messages
- debug2 — log errors and verbose informational messages and more debugging messages
- debug3 — log errors and verbose informational messages and even more debugging messages
- debug4 — log errors and verbose informational messages and even more debugging messages
- debug5 — log errors and verbose informational messages and even more debugging messages
SlurmdHttpAuthParameters — a comma-separated list of options controlling how the slurmd HTTP listener authenticates incoming requests 🛡️. The recognized keys are the same JWT-related keys documented under AuthAltParameters. When set, an independent per-daemon authentication keys are loaded from these parameters. Tokens presented to the slurmd HTTP endpoints are verified against this context rather than against AuthAltParameters.
SlurmdLogFile — fully qualified pathname of a file into which the slurmd daemon's logs are written. The default value is none (performs logging via syslog). The first "%h" within the name is replaced with the hostname on which the slurmd is running. The first "%n" within the name is replaced with the Slurm node name on which the slurmd is running. See the section LOGGING if a pathname is specified. 📝
SlurmdParameters — parameters specific to the Slurmd. Multiple options may be comma separated.

NOTE: Additional node-specific parameters can be specified using the Parameters option in individual node definitions.

allow_ecores — if set, and processors on your nodes have E-Cores, allows them to be used in for scheduling and task placement. (By default, E-Cores are ignored.) ⚡

config_overrides — if you set this, Slurm treats every node's config as exactly what the slurm.conf file says, and any node with less than the configured resources just... doesn't get marked INVAL/INVALID_REG. Basically it's for testing, bestie, that's it. It's the same thing as the now-deprecated FastSchedule=2 option.
conmgr_connect_timeout=<seconds> — how long to wait (<seconds>) before you're like "okay this outbound connection attempt is officially a flop" and call it timed out. Defaults to whatever MessageTimeout is set to. ✨
conmgr_max_connections=<connection_count> — caps how many connections get processed at any one time. Heads up: this does not change the max number of pending connections, because the kernel owns that number, not Slurm. Bumping this value will make slurmd's memory footprint bigger, so the docs literally advise sites to keep an eye on memory consumption when they raise it. Defaults to 512.
conmgr_quiesce_timeout=<seconds> — how long to wait (<seconds>) before quiesce counts as timed out. When that timeout hits, every active (non-listening) connection gets closed so the quiesce can actually start. Defaults to two times the value of MessageTimeout.
conmgr_read_timeout=<seconds> — how long to wait (<seconds>) before a read from a file descriptor is considered timed out. Defaults to the value of MessageTimeout.
conmgr_threads=<thread_count> — the number of process threads chilling in the thread pool that receive and process connections on the listening sockets. In most cases you should decrease conmgr_threads to improve scheduling performance. Defaults to 6 threads.
conmgr_use_poll — use poll(2) instead of epoll(7) for monitoring file descriptors. Old school, but valid. 💅
conmgr_wait_write_delay=<seconds> — while waiting for the kernel to flush the outgoing buffer, poll the kernel every <seconds> for changes. Defaults to the value of MessageTimeout.
conmgr_write_timeout=<seconds> — how long to wait (<seconds>) before a write to a file descriptor is considered timed out. Defaults to the value of MessageTimeout.
contain_spank — if this is set and a namespace plugin is specified, then the spank_user(), spank_task_post_fork(), and spank_task_exit() calls get run inside the job's namespace. 🫶
l3cache_as_socket — use the hwloc l3cache as the socket count. Genuinely useful on certain processors where the socket level is too coarse and the l3cache gives you better task distribution (e.g. along CCX boundaries instead of socket boundaries). Mutually exclusive with numa_node_as_socket. Requires hwloc v2.

NOTE: when this parameter is set, the node definitions in slurm.conf (Sockets or SocketsPerBoard, CoresPerSocket, and ThreadsPerCore) must be updated to reflect the resulting topology as reported by slurmd -C --parameters=l3cache_as_socket, not the physical hardware layout. If you don't update these values you'll get a mismatch between what slurmd detects and what the controller expects. 😭

numa_node_as_socket — use the hwloc NUMA Node to determine the main hierarchy object used as the socket. When this option is set, Slurm checks the parent object of the NUMA Node and uses that as the socket. Super useful for architectures like AMD Epyc, where the number of nodes per socket may be configured. Mutually exclusive with l3cache_as_socket. Requires hwloc v2.

NOTE: when this parameter is set, the node definitions in slurm.conf (Sockets or SocketsPerBoard, CoresPerSocket, and ThreadsPerCore) must be updated to reflect the resulting topology as reported by slurmd -C --parameters=numa_node_as_socket, not the physical hardware layout. Skip that and you get a mismatch between what slurmd detects and what the controller expects. Same drama, different flag. 💀

shutdown_on_reboot — if set, slurmd shuts itself down when a reboot request comes in. Like, it sees the request and exits stage left. 🚪

`SlurmdPidFile` 🎀

Fully qualified pathname of a file where the slurmd daemon may write its process id. This is the one you use for automated signal processing, iykyk.

The first "%h" in the name gets replaced with the hostname that slurmd is running on.
The first "%n" in the name gets replaced with the Slurm node name that slurmd is running on.

The default value is "/var/run/slurmd.pid".

`SlurmdPort` 🦋

The port number that the Slurm compute node daemon, slurmd, listens to for work. Defaults to SLURMD_PORT, as established at system build time. If nothing is explicitly specified, its value will be 6818.

NOTE: either slurmctld and slurmd must not run on the same nodes, or the values of SlurmctldPort and SlurmdPort must be different. Pick one, bestie.

NOTE: on Cray systems, Realm-Specific IP Addressing (RSIP) automatically tries to interact with anything opened on ports 8192-60000. So configure SlurmdPort to use a port outside of the configured SrunPortRange and RSIP's port range. ⚠️

`SlurmdSpoolDir` 🌸

Fully qualified pathname of a directory where the slurmd daemon's state info and batch job script info get written. This has to be a common pathname for all nodes, but it should point at a directory that's local to each node (like, reference a local file system, okay?). Default value is "/var/spool/slurmd".

The first "%h" in the name is replaced with the hostname that slurmd runs on.
The first "%n" in the name is replaced with the Slurm node name that slurmd runs on.

`SlurmdSyslogDebug` 💜

The slurmd daemon will log events to the syslog file at the level of detail you specify here. If you don't set it, slurmd logs to syslog at level fatal — unless there's no SlurmdLogFile and it's running in the background, in which case it logs to syslog at the level given by SlurmdDebug (or at fatal if SlurmdDebug is set to quiet). Or, if it's run in the foreground, it'll be set to quiet.

quiet — log nothing. silent treatment, iconic
fatal — log only fatal errors
error — log only errors
info — log errors and general informational messages
verbose — log errors and verbose informational messages
debug — log errors, verbose informational messages, and debugging messages
debug2 — log errors, verbose informational messages, and more debugging messages
debug3 — log errors, verbose informational messages, and even more debugging messages
debug4 — log errors, verbose informational messages, and even more debugging messages
debug5 — log errors, verbose informational messages, and even more debugging messages

NOTE: by default, Slurm's systemd service file starts the slurmd daemon in the foreground with the --systemd option. Which means systemd captures stdout/stderr output and prints that to syslog, independent of Slurm printing to syslog directly. Want systemd to stop doing that? Add "StandardOutput=null" and "StandardError=null" to the respective service files or override files. 🙈

`SlurmdTimeout` ⏱️

The interval, in seconds, that the Slurm controller waits for slurmd to respond before it sets that node's state to DOWN. A value of zero means the node will not be tested by slurmctld to confirm the state of slurmd — it won't be automatically set to DOWN for a non-responsive slurmd, and some other tool takes on the job of monitoring each compute node and its slurmd daemon. That's a you problem now, bestie.

Slurm's hierarchical communication mechanism is used to ping the slurmd daemons, keeping system noise and overhead to a minimum. Default value is 300 seconds. The value may not exceed 65533 seconds.

`SlurmdUser` 👑

The name of the user that the slurmd daemon executes as. This user has to exist on all nodes of the cluster, for authentication of communications between Slurm components. Default value is "root", and honestly keep it that way in almost all cases so slurmd can run jobs as the user who submitted them.

`SlurmSchedLogFile` 📝

Fully qualified pathname of the scheduling event logging file. The syntax of this parameter is the same as for SlurmctldLogFile. To configure scheduler logging you need to set both the SlurmSchedLogFile and SlurmSchedLogLevel parameters. Both or nothing, fr fr.

`SlurmSchedLogLevel` 🎚️

The initial level of scheduling event logging, similar to the SlurmctldDebug parameter that controls the initial level of slurmctld logging. Valid values are:

"0" — scheduler logging disabled
"1" — scheduler logging enabled

If you omit this parameter, the value defaults to "0" (disabled). To actually configure scheduler logging, set both the SlurmSchedLogFile and SlurmSchedLogLevel parameters. The scheduler logging level can be changed dynamically with scontrol, no restart needed. ✨

`SlurmUser` 🌈

The name of the user that the slurmctld daemon executes as. For security purposes, a user other than "root" is recommended. This user must exist on all nodes of the cluster for authentication of communications between Slurm components. The default value is "root".

`SrunEpilog`

Okay so this one is a fully qualified pathname of a program that srun runs after a job step finishes. 🎬 The command line arguments the executable gets are literally the command and arguments of the job step itself. You can override this with srun's --epilog parameter if you're feeling spicy.

Heads up bestie: all the other "Epilog" executables (like TaskEpilog) get run by slurmd on the compute nodes where the tasks actually do their thing — but SrunEpilog runs on the node where the srun itself is executing. Different zip code, same vibe. 📍

`SrunPortRange`

So srun opens up a little set of listening ports to talk to the controller, to slurmstepd, and to handle the application I/O. 💬 By default those ports are ephemeral, meaning the kernel just picks the numbers for you. This parameter lets a site configure a range of ports that srun will pick from instead — super useful if your network only allows certain port ranges through. 🔥

NOTE: on Cray systems, Realm-Specific IP Addressing (RSIP) will automatically try to interact with anything opened on ports 8192-60000. So configure SrunPortRange to use a range above the ones RSIP touches, ideally 1000 or more ports, for example "SrunPortRange=60001-63000". 🫶

NOTE: SrunPortRange has to be big enough to cover the expected number of srun ports created. A single srun opens 4 listening ports plus 2 more for every 48 hosts beyond the first 48. Using the --pty option means one additional port gets used. Ngl this math matters. 🧮

Example — look at the receipts:

srun -N 1        will use 4 listening ports.
srun --pty -N 1  will use 5 listening ports.
srun -N 48       will use 4 listening ports.
srun -N 50       will use 6 listening ports.
srun -N 200      will use 12 listening ports.

`SrunProlog`

The fully qualified pathname of a program for srun to execute prior to launching a job step. 🌅 The command line arguments for the executable will be the command and arguments of the job step. This config parameter can be overridden by srun's --prolog parameter. And again — while the other "Prolog" executables (e.g., TaskProlog) are run by slurmd on the compute nodes where the tasks run, SrunProlog runs on the node where the srun is executing. 💅

`StateSaveLocation`

Fully qualified pathname of a directory where the Slurm controller, slurmctld, saves its state (e.g. "/usr/local/slurm/checkpoint"). 💾 Slurm state gets saved here so it can recover from system failures — main character energy for sure. SlurmUser must be able to create files in this directory. If you've got a secondary SlurmctldHost configured, this location should be readable and writable by both systems. 👯

Since all running and pending job info lives here, a reliable file system (e.g. RAID) is recommended — fr fr, don't skimp. 🫶 The default value is "/var/spool". And if any slurm daemons terminate abnormally, their core files will also be written into this directory. 💀

`SuspendExcNodes`

Specifies the nodes that should not be placed in power save mode, even if the node stays idle for an extended period of time. 😴 Use Slurm's hostlist expression or NodeSets to identify nodes, with an optional ":" separator and count of nodes to exclude from the preceding range. For example "nid[10-20]:4" will prevent 4 powered up nodes in the set "nid[10-20]" from being powered down. Multiple sets of nodes can be specified with or without counts in a comma separated list (e.g "nid[10-20]:4,nid[80-90]:2"). By default no nodes are excluded. ✨

Nodes explicitly requested to power down — including nodes allocated from a partition configured with PowerDownOnIdle=YES — are not affected by this option. This value may be updated with scontrol. See ReconfigFlags=KeepPowerSaveSettings for setting persistence. 👑

`SuspendExcParts`

Specifies the partitions whose nodes are not to be placed in power save mode, even if the node stays idle for an extended period of time. Multiple partitions can be identified and separated by commas. By default no nodes are excluded. 🌸

Nodes explicitly requested to power down, including nodes allocated from a partition configured with PowerDownOnIdle=YES, are not affected by this option. This value may be updated with scontrol. See ReconfigFlags=KeepPowerSaveSettings for setting persistence. 💅

`SuspendExcStates`

Specifies node states that are not to be powered down automatically. Valid states include CLOUD, DOWN, DRAIN, DYNAMIC_FUTURE, DYNAMIC_NORM, FAIL, INVALID_REG, MAINTENANCE, NOT_RESPONDING, PLANNED, and RESERVED. 🌈

By default, any of these states, if idle for SuspendTime, would be powered down. Nodes explicitly requested to power down, including nodes allocated from a partition configured with PowerDownOnIdle=YES, are not affected by this option. This value may be updated with scontrol. See ReconfigFlags=KeepPowerSaveSettings for setting persistence. 🎀

`SuspendProgram`

SuspendProgram is the program that gets executed when a node remains idle for an extended period of time. ⏰ It can also be set to the name of a PowerAction. This program is expected to place the node into some power save mode — it can reduce the frequency and voltage of a node, or completely power the node off. 🔌

The argument to the program will be the names of nodes to be placed into power savings mode (using Slurm's hostlist expression format). By default, no program is run. Programs will be killed if they run longer than the largest configured, global or partition, ResumeTimeout or SuspendTimeout. 😭 Don't overstay your welcome.

`SuspendRate`

The rate at which nodes are placed into power save mode by SuspendProgram. The value is number of nodes per minute, and it can be used to prevent a large drop in power consumption (e.g. after a large job completes). 📉

A value of zero results in no limits being imposed. The default value is 60 nodes per minute. ⭐

`SuspendTime`

Nodes which remain idle or down for this number of seconds will be placed into power save mode by SuspendProgram. 😴 This value can be overridden on a per-partition or per-node basis. Setting SuspendTime to anything but INFINITE (or -1) will enable power save mode. INFINITE is the default. 💤

`SuspendTimeout`

Maximum time permitted (in seconds) between when a node suspend request is issued and when the node is shutdown. ⏳ At that time the node must be ready for a resume request to be issued as needed for new work. The default value is 30 seconds.

`SwitchParameters`

Optional parameters for the switch plugin. 🔀

On HPE Slingshot systems configured with SwitchType=switch/hpe_slingshot, the following parameters are supported (separate multiple parameters with a comma):

vnis=<min>-<max> — Range of VNIs to allocate for jobs and applications. The default value is 1024-65535. 🦋
destroy_retries=<retry attempts> — Configure the number of times destroying CXI services is retried at the end of the step. There is a one second pause between each retry. The default value is 5. 🔁
tcs=<class1>[:<class2>]... — Set of traffic classes to configure for applications. Supported traffic classes are DEDICATED_ACCESS, LOW_LATENCY, BULK_DATA, and BEST_EFFORT. The traffic classes may also be specified as TC_DEDICATED_ACCESS, TC_LOW_LATENCY, TC_BULK_DATA, and TC_BEST_EFFORT. 🚦
single_node_vni=<all|user|none> — If set to 'all', allocate a VNI for all job steps (by default, no VNI will be allocated for single-node job steps). If set to 'user', allocate a VNI for single-node job steps using the srun --network=single_node_vni option or SLURM_NETWORK=single_node_vni environment variable. If set to 'none' (or if single_node_vni is not set), do not allocate any VNI for single-node job steps. For backwards compatibility, setting single_node_vni with no argument is equivalent to 'all'. ✨
job_vni=<all|user|none> — If set to 'all', allocate an additional VNI for jobs, shared among all job steps. If set to 'user', allocate an additional VNI for any job using the srun --network=job_vni option or SLURM_NETWORK=job_vni environment variable. If set to 'none' (or if job_vni is not set), do not allocate any additional VNI for jobs. For backwards compatibility, setting job_vni with no argument is equivalent to 'all'. 💜
adjust_limits — If set, slurmd will set an upper bound on network resource reservations by taking the per-NIC maximum resource quantity and subtracting the reserved or used values (whichever is higher) for any system network services; this is the default. 🎯
no_adjust_limits — If set, slurmd will calculate network resource reservations based only upon the per-resource configuration default and number of tasks in the application; it will not set an upper bound on those reservation requests based on resource usage of already-existing system network services. Setting this will mean more application launches could fail based on network resource exhaustion, but if the application absolutely needs a certain amount of resources to function, this option will ensure that. ⚠️
hwcoll_addrs_per_job — The number of Slingshot hardware collectives multicast addresses to allocate per job. (That are larger than hwcoll_min_nodes nodes.) 👑
hwcoll_num_nodes — The minimum number of nodes for a job to be allocated Slingshot hardware collectives. Because the hardware collective engine is not expected to offer a meaningful performance boost for jobs spanning a small number of nodes. 🧚
fm_url — set this and Slurm will hit the configured URL to talk to the fabric manager, which is how you enable Slingshot hardware collectives. Heads up tho: enable_stepmgr also has to be set or hardware collectives won't run. both. not one. 💅
fm_auth — the HPE fabric manager REST API auth type. Your options are BASIC or OAUTH, and the default is OAUTH.
fm_authdir — the directory holding your authentication info files. Default is /etc/fmsim for BASIC authentication and /etc/wlm-client-auth for OAUTH authentication.
fm_authdir_ctld — the directory holding the authentication info files that slurmctld itself will use. Defaults to whatever you set for fm_authdir.
fm_mtls_url — an alternative URL to fm_url that the Slurm daemons will use to interface with the fabric manager, for enabling Slingshot hardware collectives when mTLS authentication is on. If you don't set it, fm_url gets used instead. To turn on mTLS authentication, see fm_mtls_ca, fm_mtls_cert, and fm_mtls_key.

Note: setting fm_url and enable_stepmgr is required to enable Slingshot hardware collectives. no shortcuts bestie 🫶

fm_mtls_ca — path to a Certificate Authority (CA) bundle file, or a directory containing a file signed by the fabric manager certificate. If you set it, the identity of the fabric manager server gets verified when Slingshot hardware collectives are enabled. See also fm_mtls_cert and fm_mtls_key.

Note: this option is not required to enable mTLS authentication with the fabric manager. But without it, the client (that's the slurmctld and stepmgr processes) won't be able to verify the server identity. so like… you can skip it, you just won't know who you're talking to 🙈

fm_mtls_cert — path to the client public certificate. This one is required to enable mTLS authentication with the fabric manager when Slingshot hardware collectives are enabled. See also fm_mtls_ca and fm_mtls_key.
fm_mtls_key — path to the client private key. Also required to enable mTLS authentication to the fabric manager when Slingshot hardware collectives are enabled. See also fm_mtls_ca and fm_mtls_cert.
nic_distribution_count=<val> — the default number of NICs that users will evenly distribute their tasks over. Users can override it with the --network=nic_distribution_count=<val> option or the SLURM_NETWORK=nic_distribution_count=<val> environment variable. Defaults to the number of NICs on each node.
def_<rsrc>=<val> — per-CPU reserved allocation for this resource.
res_<rsrc>=<val> — per-node reserved allocation for this resource. If it's set, it overrides the per-CPU allocation. it's giving "I'm the parent now" 💀
max_<rsrc>=<val> — maximum per-node application for this resource.

🎀 The resources that may be configured are:

txqs — transmit command queues. Default is 2 per-CPU, maximum 1024 per-node.
tgqs — target command queues. Default is 1 per-CPU, maximum 512 per-node.
eqs — event queues. Default is 2 per-CPU, maximum 2047 per-node.
cts — counters. Default is 1 per-CPU, maximum 2047 per-node.
tles — trigger list entries. Default is 1 per-CPU, maximum 2048 per-node.
ptes — portable table entries. Default is 6 per-CPU, maximum 2048 per-node.
les — list entries. Default is 16 per-CPU, maximum 16384 per-node.
acs — addressing contexts. Default is 2 per-CPU, maximum 1022 per-node.

On systems configured with SwitchType=switch/nvidia_imex, the following parameters are supported:

imex_channel_count — the number of channels that can be configured. Channels let nodes create a secure method of sharing memory. The default value is 2048.

By default, each job is allocated one IMEX channel that is accessible by the batch, interactive, and normal job steps on all nodes within the job. If you use --network=unique-channel-per-segment on job submission and topology/block is configured, then each segment gets allocated one IMEX channel that's accessible by the batch, interactive, and normal job steps on all nodes within that particular segment. per-segment girlies rise up ✨

🔀 `SwitchType`

Identifies the type of switch or interconnect used for application communications. Default value is no special plugin requiring special processing for job launch or termination (Ethernet, and InfiniBand).

okay this is the dramatic one: all Slurm daemons, commands and running jobs must be restarted or reconfigured for a change in SwitchType to take effect. And if running jobs exist at the time slurmctld is restarted with a new value of SwitchType, records of all jobs in any state may be lost. not "might get weird." lost. 😭

Acceptable values include:

switch/hpe_slingshot — for HPE Slingshot systems.
switch/nvidia_imex — for allocating unique channels within an NVIDIA IMEX domain.

`TaskEpilog`

Pathname of a program to be executed as the job's user after termination of each task. If it's not an absolute pathname (i.e. it doesn't start with a slash), it must be relative to the directory containing the slurm.conf file. It will run inside of the job's container if configured. Should not be used for policy enforcement. See TaskProlog for execution order details. order matters, bestie 🫶

`TaskPlugin`

Identifies the type of task launch plugin, typically used to provide resource management within a node (e.g. pinning tasks to specific processors). More than one task plugin can be specified in a comma-separated list. The prefix task/ is optional. Unset by default.

Acceptable values include:

task/affinity — binds processes to specified resources using sched_setaffinity(). This enables the --cpu-bind and/or --mem-bind srun options.
task/cgroup — enables process containment to specified resources using Cgroups cpuset interface. This enables the --cpu-bind and/or --mem-bind srun options. NOTE: see man cgroup.conf for configuration details.

NOTE: it's recommended to stack task/cgroup,task/affinity together when configuring TaskPlugin, and set ConstrainCores=yes in cgroup.conf. This setup uses the task/affinity plugin for setting the cpu mask for tasks and uses the task/cgroup plugin to fence tasks into the allocated cpus. they're a duo, it's a collab 👑

`TaskPluginParam`

Optional parameters for the task plugin. Multiple options should be comma separated. None, Sockets, Cores and Threads are mutually exclusive and treated as a last possible source of --cpu-bind default. See also Node and Partition CpuBind options.

Cores — bind tasks to cores by default. Overrides automatic binding.
None — perform no task binding by default. Overrides automatic binding.
Sockets — bind to sockets by default. Overrides automatic binding.
Threads — bind to threads by default. Overrides automatic binding.
SlurmdSpecOverride — if slurmd is started in a cgroup which has cpuset or memory constraints, then CpuSpecList and MemSpecLimit will be set and will override the configured values. This avoids scheduling resources from these constraints. In cgroup/v1, slurmd and slurmstepd daemons will now not be able to use any of these resources. While in normal behavior, cgroup/v1 constrains the daemons to CpuSpecList and MemSpecLimit.
SlurmdOffSpec — if specialized cores or CPUs are identified for the node (i.e. the CoreSpecCount or CpuSpecList are configured for the node), then Slurm daemons running on the compute node (i.e. slurmd and slurmstepd) should run outside of those resources (i.e. specialized resources are completely unavailable to Slurm daemons and jobs spawned by Slurm).
OOMKillStep — set this parameter to kill the whole step in all the nodes in case an OOM event is triggered in any task of the step.

This applies to entire allocations but does not apply to the external step. It can be overwritten by the user.

NOTE: this parameter requires the task/cgroup plugin, Cgroups v2, and a kernel newer than 4.19. old kernel? it's not gonna work, sorry babe 💀

Verbose — verbosely report binding before tasks run by default.
Autobind — set a default binding in the event that "auto binding" doesn't find a match. Set to Threads, Cores or Sockets (e.g. TaskPluginParam=autobind=threads).

`TaskProlog`

Pathname of a program to be executed as the job's user prior to initiation of each task. If it's not an absolute pathname (i.e. it doesn't start with a slash), it must be relative to the directory containing the slurm.conf file. Will run inside of the job's container if configured. Should not be used for policy enforcement. Besides the normal environment variables, this one has SLURM_TASK_PID available to identify the process ID of the task being started. Standard output from this program can be used to control the environment variables and output for the user program. main character prolog energy 🌟

export NAME=value — will set environment variables for the task being spawned. Everything after the equal sign to the end of the line will be used as the value for the environment variable. Exporting of functions is not currently supported. (functions? not yet bestie. someday. 🦋)
print ... — prints that line (minus the leading print ) straight to the job's standard output. Free real estate for debugging, bestie 💅
unset NAME — clears environment variables for the task being spawned. Bye bye, variable ✨

The order of task prolog/epilog execution is literally this, in order, no skipping:

pre_launch_priv() Function in TaskPlugin
pre_launch() Function in TaskPlugin
TaskProlog System-wide per task program defined in slurm.conf
User prolog Job-step-specific task program defined using srun's --task-prolog option or SLURM_TASK_PROLOG environment variable
Task Execute the job step's task
User epilog Job-step-specific task program defined using srun's --task-epilog option or SLURM_TASK_EPILOG environment variable
TaskEpilog System-wide per task program defined in slurm.conf
post_term() Function in TaskPlugin

🕐 TCPTimeout

Time permitted for TCP connection to be established. Default value is 2 seconds. Clock's ticking ⏳

🔐 TLSParameters

Comma-separated options for the TLS plugin configured by TLSType. Supported values include:

ca_cert_file= — path of certificate authority (CA) certificate. Must exist on all hosts and be accessible by all Slurm components. File permissions must be 644, and owned by SlurmUser/root. Default path is "ca_cert.pem" in the Slurm configuration directory.
ctld_cert_file= — path of certificate used by slurmctld. Must chain to ca_cert_file. Should only exist on host running slurmctld. File permissions must be 600, and owned by SlurmUser. Default path is "ctld_cert.pem" in the Slurm configuration directory.
ctld_cert_key_file= — path of private key that accompanies ctld_cert_file. Should only exist on host running slurmctld. File permissions must be 600, and owned by SlurmUser. Default path is "ctld_cert_key.pem" in the Slurm configuration directory.
restd_cert_file= — path of certificate used by slurmrestd. Must chain to ca_cert_file. Should only exist on host running slurmrestd. File permissions must be 600, and owned by the user that runs slurmrestd. Default path is "restd_cert.pem" in the Slurm configuration directory.
restd_cert_key_file= — path of private key that accompanies restd_cert_file. Should only exist on host running slurmrestd. File permissions must be 600, and owned by the user that runs slurmrestd. Default path is "restd_cert_key.pem" in the Slurm configuration directory.
sackd_cert_file= — path of certificate used by sackd. Must chain to ca_cert_file. Should only exist on host running sackd. File permissions must be 600, and owned by SlurmUser. Default path is "sackd_cert.pem" in the Slurm configuration directory.

NOTE: If not using the certmgr plugin, this file needs to exist. 👀

sackd_cert_key_file= — path of private key that accompanies sackd_cert_file. Should only exist on host running sackd. File permissions must be 600, and owned by SlurmUser. Default path is "sackd_cert_key.pem" in the Slurm configuration directory.

NOTE: If not using the certmgr plugin, this file needs to exist. 👀

slurmd_cert_file= — path of certificate used by slurmd. Must chain to ca_cert_file. Should only exist on host running slurmd. File permissions must be 600, and owned by SlurmUser. Default path is "slurmd_cert.pem" in the Slurm configuration directory.

NOTE: If not using the certmgr plugin, this file needs to exist. 👀

slurmd_cert_key_file= — path of private key that accompanies slurmd_cert_file. Should only exist on host running slurmd. File permissions must be 600, and owned by SlurmUser. Default path is "slurmd_cert_key.pem" in the Slurm configuration directory.

NOTE: If not using the certmgr plugin, this file needs to exist. 👀

load_system_certificates — load certificates found in default system locations (e.g. /etc/ssl) into trust store. Default is to not load system certificates, and to rely solely on ca_cert_file to establish trust.
security_policy_version= — security policy version used by s2n. See s2n documentation for more details. Default security policy is "20230317", which is FIPS compliant and includes TLS 1.3. Fancy 💅

🛡️ TLSType

Specify the TLS implementation that will be used. When set to a TLS plugin, RPC listening sockets reject connections that are not TLS wrapped. Refer to the TLS guide for more details: <https://slurm.schedmd.com/tls.html> Defaults to tls/none. Acceptable values at present:

tls/none — No TLS. This is the default. 🫶
tls/s2n — Use the s2n TLS plugin. Requires additional configuration and causes significant processing overhead, but allows all Slurm communication to be encrypted. 🔥

📁 TmpFS

Fully qualified pathname of the file system available to user jobs for temporary storage. This parameter is used in establishing a node's TmpDisk space. The default value is "/tmp".

🕸️ TopologyParam

Comma-separated options identifying network topology options.

Dragonfly — Optimize allocation for Dragonfly network. Valid when TopologyPlugin=topology/tree.
RoutePart — Instead of using the plugin's default route calculation, use partition node lists to route communications from the controller. Once on the compute node, communications will be routed using the requested plugin's normal algorithm, following TreeWidth if applicable. If a node is in multiple partitions, the first partition seen will be used. The controller will communicate directly with any nodes that aren't in a partition.
BlockAsNodeRank — Assign the same node rank to all nodes under one base block. This can be useful if the naming convention for the nodes does not match the network topology. Valid when topology/block is a cluster default topology.
SwitchAsNodeRank — Assign the same node rank to all nodes under one leaf switch. This can be useful if the naming convention for the nodes does not match the network topology. Valid when topology/tree is a cluster default topology.
RouteTree — Use the switch hierarchy defined in a topology.conf file for routing instead of just scheduling. Valid when TopologyPlugin=topology/tree. Incompatible with dynamic nodes. ⚠️
TopoMaxSizeUnroll= — Maximum number of individual job sizes automatically unrolled from min-max nodes job specification. Default: -1 (option disabled). Valid when TopologyPlugin=topology/block.
TopoOptional — Only optimize allocation for network topology if the job includes a switch option. Since optimizing resource allocation for topology involves much higher system overhead, this option can be used to impose the extra overhead only on jobs which can take advantage of it. If most job allocations are not optimized for network topology, they may fragment resources to the point that topology optimization for other jobs will be difficult to achieve.

NOTE: Jobs may span across nodes without common parent switches with this enabled. 😭

🌐 TopologyPlugin

Identifies the plugin to be used for determining the network topology and optimizing job allocations to minimize network contention. See NETWORK TOPOLOGY below for details. Additional plugins may be provided in the future which gather topology information directly from the network. Acceptable values include:

topology/block — used for a block network topology, as described in the topology.conf(5) man page
topology/flat — best-fit logic over one-dimensional topology. This is the default.
topology/tree — used for a hierarchical network with the select/cons_tres plugin, as described in the topology.conf(5) man page

NOTE: This option is ignored if topology.yaml exists. 🫠

🗂️ TrackWCKey

Boolean yes or no. Used to set display and track of the Workload Characterization Key. Must be set to track correct wckey usage.

NOTE: You must also set TrackWCKey in your slurmdbd.conf file to create historical usage reports. Bestie, don't forget this one, fr fr 🙈

TreeWidth — so the slurmd daemons talk to each other over a virtual tree network, right? This is the width of that tree (the fanout, iykyk). Default is 16, meaning each slurmd can chat with up to 16 other slurmds ✨. That number is a whole balance act between offloading slurmctld (max 16 threads running), communication time, and node fault tolerance — and yes, 4368 nodes can be reached in just three message hops with that setup. The default slay for most clusters 🌸, but on bigger systems you can bump it up to dodge long timeouts and retransmissions when nodes go unresponsive. The value may not exceed 65533.
UnkillableStepProgram — okay so if the processes in a job step are deemed unkillable for however long UnkillableStepTimeout says, then this program gets executed. By default no program is run 💀. Peep the UNKILLABLE STEP PROGRAM SCRIPT section for the deets.
UnkillableStepTimeout — how long, in seconds, Slurm waits before deciding processes in a job step are unkillable (after they've already been signaled with SIGKILL) and then runs UnkillableStepProgram. Default is 60 seconds or five times the value of MessageTimeout, whichever is greater 👑. If it's exceeded, the compute node gets drained so no future jobs get scheduled there.

NOTE: make sure UnkillableStepTimeout is at least 5 times larger than MessageTimeout, otherwise you can get unexpected draining of nodes 😭 fr fr.

UrlParserType — picks which url_parser implementation gets used. Default is url_parser/libhttp_parser if found, otherwise url_parser/internal. Current acceptable values:
- url_parser/libhttp_parser — uses the libhttp_parser based plugin.
- url_parser/internal — uses Slurm's internal url parser plugin. Note: required when HttpParserType=http_parser/llhttp_parser 🫶.
UsePAM — set it to 1 and PAM (Pluggable Authentication Modules for Linux) gets enabled 💅. PAM is what establishes the upper bounds for resource limits. With PAM support on, local system admins can dynamically configure system resource limits — but changing the upper bound of a resource limit will not alter the limits of already-running jobs, only jobs started after the change will pick up the new limits. Default is 0 (no PAM support). Remember PAM also needs to be configured to support Slurm as a service! For sites using PAM's directory based configuration option, create a config file named slurm with these module-type, control-flags, and module-path names:

auth        required      pam_localuser.so
auth        required      pam_shells.so
account     required      pam_unix.so
account     required      pam_access.so
session     required      pam_unix.so

For sites configuring PAM with a general configuration file, add the appropriate lines (same as above), where slurm is the service-name. See https://slurm.schedmd.com/pam_slurm_adopt.html for more details 🔥.

NOTE: the UsePAM option has nothing to do with the contribs/pam/pam_slurm and/or contribs/pam_slurm_adopt modules. Those two can work independently of whatever you set UsePAM to 🙈.

VSizeFactor — memory specs in job requests apply to real memory size (aka resident set size). You can enforce virtual memory limits for both jobs and job steps by limiting their virtual memory to some percentage of their real memory allocation. VSizeFactor sets that percentage. Example bestie: if a job's real memory limit is 500MiB and VSizeFactor is 101, the job gets killed if its real memory exceeds 500MiB or its virtual memory exceeds 505MiB (101 percent of the real memory limit) 💀. Default is 0, which disables enforcement of virtual memory limits. Value may not exceed 65533 percent.

NOTE: this parameter depends on OverMemoryKill being configured in JobAcctGatherParams. It's also possible to configure the TaskPlugin to use task/cgroup for memory enforcement — but VSizeFactor will not have an effect on memory enforcement done through cgroups 🧚.

WaitTime — how many seconds srun should by default wait after the first task terminates before terminating all the remaining tasks. The --wait option on the srun command line overrides this. Default is 0, which disables the feature. May not exceed 65533 seconds ⭐.
X11Parameters — for use with Slurm's built-in X11 forwarding implementation 🦋.
- home_xauthority — if set, xauth data on the compute node goes in ~/.Xauthority rather than a temporary file under TmpFS.

🌈 NODE CONFIGURATION 🌈

The configuration of nodes (machines) managed by Slurm also lives in /etc/slurm.conf 💖. Changes in node config (adding nodes, changing processor counts, etc.) require restarting or reconfiguring all slurmctld and slurmd daemons. Every slurmd daemon has to know every node in the system so it can forward messages for hierarchical communications, no cap.

Only NodeName must be supplied in the config file — literally everything else about a node is optional 😌. Still, it's advisable to establish baseline node configurations, especially if the cluster is heterogeneous. Nodes that register to the system with less than the configured resources (e.g. too little memory) get placed in the "DOWN" state to avoid scheduling jobs on them. Baseline configs also speed up Slurm's scheduling by letting it compare job requirements against these (relatively few) configuration parameters and possibly skip checking every individual node's config. The resources checked at node registration time are: CPUs, RealMemory and TmpDisk 👑

Default values can be specified with a record where NodeName is "DEFAULT". Heads up: default entry values apply only to lines following it in the config file, and defaults can be reset multiple times with multiple "NodeName=DEFAULT" entries. Each line where NodeName is "DEFAULT" replaces or adds to previous defaults and does not reinitialize them 🫶.

The "NodeName=" specification must be on every line describing node config. A single node name can't appear as a NodeName value in more than one line (duplicate node name records get ignored). Honestly it's generally possible and desirable to define all node configs in only a few lines — this convention permits significant optimization when scheduling larger clusters ✨. To support jobs requiring consecutive nodes on some architectures, put node specifications in this file in consecutive order. No single node name may be listed more than once. Use "DownNodes=" to record the state of nodes temporarily in a DOWN, DRAIN or FAILING state without altering permanent configuration info 🌸. A job step's tasks get allocated to nodes in the order the nodes appear in the config file — there is presently no capability within Slurm to arbitrarily order a job step's tasks 🙈.

Multiple node names may be comma separated (e.g. "alpha,beta,gamma") and/or a simple node range expression may optionally be used to specify numeric ranges of nodes, so you don't have to build a config file with a zillion entries 💅. The node range expression can contain one or more pairs of square brackets with a sequence of comma-separated numbers and/or ranges of numbers separated by a "-" (e.g., "linux[0-64,128]", "lx[15,18,32-33]", or "rack[0-63]_blade[0-41]"). Note that numeric ranges can include one or more leading zeros to indicate the numeric portion has a fixed number of digits (e.g. "linux[0000-1023]") 🔥.

The node configuration specifies the following information:

NodeName — the name Slurm uses to refer to a node. Typically this is the string /bin/hostname -s returns. It may also be the fully qualified domain name as returned by /bin/hostname -f (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings 🦋. Note that if the short form of the hostname isn't used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string). It may also be an arbitrary string if NodeHostname is specified. If the NodeName is "DEFAULT", the values specified with that record apply to subsequent node specifications unless explicitly set to other values in that node record or replaced with a different set of default values — and each line where NodeName is "DEFAULT" replaces or adds to previous defaults, it doesn't reinitialize them 👑. For architectures where node order is significant, nodes are considered consecutive in the order defined. Like, if the config for "NodeName=charlie" immediately follows the config for "NodeName=baker", they're considered adjacent in the computer ⭐.

NOTE: if NodeName is "ALL" the process parsing the configuration will exit immediately, because it's an internally reserved word 💀.

🖇️ NodeHostname

okay so this is the name the node actually answers to. Typically it's whatever /bin/hostname -s spits back at you. It can also be the fully qualified domain name that /bin/hostname -f returns (think foo1.bar.com), or literally any valid domain name tied to the host through the host database (/etc/hosts) or DNS — depends on your resolver settings, bestie. ⚠️ heads up though: if you don't use the short form of the hostname, you might break hostlist expressions (the numeric part in brackets has to be at the end of the string). No cap.

A node range expression can be used to specify a whole set of nodes. If you do use an expression, the number of nodes identified by NodeHostname on a line in the configuration file must be identical to the number of nodes identified by NodeName. Matching counts, always. 💅

By default, NodeHostname is identical in value to NodeName. slay.

📇 NodeAddr

The name a node should be referred to when establishing a communications path. This name gets used as an argument to the getaddrinfo() function for identification. If a node range expression is used to designate multiple nodes, they must exactly match the entries in the NodeName (e.g. NodeName=lx[0-7] NodeAddr=elx[0-7]). NodeAddr may also contain IP addresses, fyi. By default, NodeAddr is identical in value to NodeHostname. 🫶

📡 BcastAddr

An alternate network path used for sbcast network traffic to a given node. This name is used as an argument to the getaddrinfo() function. If a node range expression designates multiple nodes, they must exactly match the entries in the NodeName (e.g. NodeName=lx[0-7] BcastAddr=elx[0-7]). BcastAddr may also contain IP addresses. By default BcastAddr is unset, and sbcast traffic gets routed to the NodeAddr for that node. ✨

Note: cannot be used with CommunicationParameters=NoInAddrAny.

🧩 Boards

Number of baseboards in nodes that have a baseboard controller. Note that when Boards is specified, SocketsPerBoard, CoresPerSocket, and ThreadsPerCore should be specified too. Default value is 1. 💖

🛡️ CoreSpecCount

Number of cores reserved for system use. Depending on the TaskPluginParam option of SlurmdOffSpec, the Slurm daemon slurmd may either be confined to these resources (the default) or prevented from using these resources. If cgroup/v1 is used, the same thing applies to the slurmstepd processes. Isolating slurmd from user jobs may improve application performance, lowkey. 👑

A job can use these cores if AllowSpecResourcesUsage=yes and the user explicitly requests less than the configured CoreSpecCount. If this option and CpuSpecList are both designated for a node, an error is generated — pick one, bestie. For info on the algorithm Slurm uses to select the cores, go read the core specialization documentation ( https://slurm.schedmd.com/core_spec.html ).

🧠 CoresPerSocket

Number of cores in a single physical processor socket (e.g. "2"). CoresPerSocket describes physical cores, not the logical number of processors per socket. ⚠️ NOTE: if you have multi-core processors, you'll likely need to specify this parameter to optimize scheduling. Default value is 1.

📌 CpuBind

If a job step request doesn't specify an option to control how tasks are bound to allocated CPUs (via --cpu-bind) and all nodes allocated to the job have the same CpuBind option, then the node's CpuBind option controls how tasks are bound to allocated resources. Partition definitions get used next if the node definition(s) can't be used, followed by TaskPluginParam as a last resort, with the default being no binding. Supported values for CpuBind are none, socket, ldom (NUMA), core and thread. 🎀

💻 CPUs

Number of logical processors on the node (e.g. "2"). It can be set to the total number of sockets (supported only by select/linear), cores or threads. This is useful when you only want to schedule the cores on a hyper-threaded node. If CPUs is omitted, its default is set equal to the product of Boards, Sockets, CoresPerSocket, and ThreadsPerCore. math slay ✨

🎯 CpuSpecList

A comma-delimited list of Slurm abstract CPU IDs reserved for system use. These are indices in Slurm's per-node CPU numbering, which may not match the CPU IDs used by the operating system. The list gets expanded to include all other CPUs, if any, on the same cores. Depending on the TaskPluginParam option of SlurmdOffSpec, the Slurm daemon slurmd may either be confined to these resources (the default) or prevented from using them. If cgroup/v1 is used, the same applies to the slurmstepd processes. Isolating slurmd from user jobs may improve application performance.

A job can use these cores if AllowSpecResourcesUsage=yes and the user explicitly requests less than the number of CPUs in this list. If this option and CoreSpecCount are both designated for a node, an error is generated. 🚨

This option has no effect unless cgroup job confinement is also configured (i.e. the task/cgroup TaskPlugin is enabled and ConstrainCores=yes is set in cgroup.conf).

🏷️ Features

A comma-delimited list of arbitrary strings indicative of some characteristic associated with the node. There's no value or count associated with a feature at this time — a node either has a feature or it does not. Periodt. A desired feature may contain a numeric component indicating, like, processor speed or whatever, but that numeric component is considered part of the feature string. Features are meant to be used to filter nodes eligible to run jobs via the --constraint argument. By default a node has no features.

Also see Gres if you want more control over types and counts. Using features is faster than scheduling against GRES but is limited to Boolean operations. 🦋

NOTE: the hostlist function feature{myfeature} expands to all nodes with the specified feature. This may be used in place of or alongside regular hostlist expressions in commands or configuration files that interact with slurmctld. For example: scontrol update node=feature{myfeature} state=resume or PartitionName=p1 Nodes=feature{myfeature}.

🎮 Gres

A comma-delimited list of generic resources specifications for a node. The format is:

"<name>[:<type>][:no_consume]:<number>[K|M|G]"

The first field is the resource name, which matches the GresType configuration parameter name. The optional type field might be used to identify a model of that generic resource. It is forbidden to specify both an untyped GRES and a typed GRES with the same <name>. 💀

The optional no_consume field lets you specify that a generic resource doesn't have a finite number that gets consumed as it's requested. no_consume is a GRES specific setting and applies to the GRES regardless of the type specified. It should not be used with GRES that has a dedicated plugin — if you're looking for a way to overcommit GPUs to multiple processes at a time, you may be interested in using "shard" GRES instead.

The final field must specify a generic resources count. A suffix of "K", "M", "G", "T" or "P" may be used to multiply the number by 1024, 1048576, 1073741824, etc. respectively. (e.g. "Gres=gpu:tesla:1,gpu:kepler:1,bandwidth:lustre:no_consume:4G").

If Gres is configured on a default node entry (NodeName=DEFAULT), you can set an empty string (Gres="") to configure no Gres on a subsequent entry. By default a node has no generic resources and its maximum count is that of an unsigned 64bit integer. Also see Features for Boolean flags to filter nodes using job constraints. 🌈

NOTE: before changing a GRES's configured count, drain the node of all running jobs using that GRES if it has the File or MultipleFiles parameters set either manually or with AutoDetect in gres.conf(5). Similarly, drain the node before reducing the configured count of a shared GRES (gres/mps, gres/shard). Failure to do so will result in any running job using those GRES being aborted and, if eligible, requeued. 😭

🧮 MemSpecLimit

Amount of RealMemory, in mebibytes, reserved for system use and not available for user allocations. Must be less than the amount defined for RealMemory. If the task/cgroup plugin is configured and that plugin constrains memory allocations (i.e. the task/cgroup TaskPlugin is enabled and ConstrainRAMSpace=yes is set in cgroup.conf), then the slurmd will be allocated the specified memory limit. If cgroup/v1 is used, the slurmstepd will also be allocated the specified memory limit. If cgroup/v2 is used, the slurmstepd's consumption is completely dependent on the topology of the job. 🌀

Note that having the Memory set in SelectTypeParameters as any of the options that has it as a consumable resource is needed for this option to work. The daemons will not be killed if they exhaust the memory allocation (i.e. the Out-Of-Memory Killer is disabled for the daemon's memory cgroup). If the task/cgroup plugin is not configured, the specified memory will only be unavailable for user allocations. 🙈

⚙️ Parameters

Allows for node-specific additions to the global SlurmdParameters. Options are appended, and cannot override global options. iykyk 💅

🔌 Port

The port number that the Slurm compute node daemon, slurmd, listens to for work on this particular node. By default there's a single port number for all slurmd daemons on all compute nodes, as defined by the SlurmdPort configuration parameter. Use of this option is not generally recommended except for development or testing purposes. If multiple slurmd daemons execute on a node, this can specify a range of ports. 🧚

NOTE: on Cray systems, Realm-Specific IP Addressing (RSIP) will automatically try to interact with anything opened on ports 8192-60000. Configure Port to use a port outside of the configured SrunPortRange and RSIP's port range.

🪞 Procs

See CPUs. 💖

RealMemory — the size of real memory on the node, in mebibytes (like "2048"). Default is 1. Bestie, quick heads up: lowering RealMemory so you can set some aside for the OS and keep it out of job allocations will not work as intended unless Memory is set as a consumable resource in SelectTypeParameters. So one of the *_Memory options needs to be enabled for that goal to actually happen. Also see MemSpecLimit 💖
Reason — identifies the reason a node is in state "DOWN", "DRAINED", "DRAINING", "FAIL" or "FAILING". Got a reason with more than one word? Wrap it in quotes, no cap.
RestrictedCoresPerGPU — the number of cores per GPU restricted for only GPU use. If a job doesn't request a GPU, it will not have access to these cores. The node's GPUs must either be autodetected or have valid cores configured in gres.conf(5).

NOTE: configuring multiple GPU types on overlapping sockets can result in erroneous GPU type and restricted core pairings in allocations requesting gpus without specifying a type. NOTE: shared gpu gres (shards or mps) will have access to these cores, but there is no guarantee that reserved cores are used in proportion to the shared gres allocation. 😭

Sockets — number of physical processor sockets/chips on the node (like "2"). If Sockets is omitted, it gets inferred from CPUs, CoresPerSocket, and ThreadsPerCore. Got multi-core processors? You'll likely need to specify these parameters. Sockets and SocketsPerBoard are mutually exclusive. And if Sockets is specified when Boards is also used, Sockets is interpreted as SocketsPerBoard rather than total sockets. Default value is 1 ✨

NOTE: if SlurmdParameters=numa_node_as_socket or SlurmdParameters=l3cache_as_socket is configured, the values for Sockets, CoresPerSocket, and ThreadsPerCore must match the virtual topology reported by slurmd -C under that configuration, not the physical hardware layout. See the SlurmdParameters documentation for the deets 🫶

SocketsPerBoard — number of physical processor sockets/chips on a baseboard. Sockets and SocketsPerBoard are mutually exclusive. Default value is 1.
SuspendTime — nodes that stay idle or down for this number of seconds get placed into power save mode by SuspendProgram. This value takes precedence over the partition-level and global SuspendTime settings for the specified nodes. For nodes in multiple partitions with SuspendTime configured, the node-level value takes precedence. If it's not set on the node, the partition value and then the global value are used. Setting SuspendTime to INFINITE disables suspending of these nodes. Setting SuspendTime to anything but INFINITE (or -1) will enable power save mode ⚡
State — state of the node with respect to the initiation of user jobs. Acceptable values are CLOUD, DOWN, DRAIN, FAIL, FAILING, FUTURE and UNKNOWN. Node states of BUSY and IDLE should not be specified in the node configuration — set the node state to UNKNOWN instead, okay? Setting the node state to UNKNOWN will result in the node state being set to BUSY, IDLE or other appropriate state based upon recovered system state information. Default value is UNKNOWN. Also see the DownNodes parameter below 👀
- CLOUD — indicates the node exists in the cloud. Its initial state will be treated as powered down. The node will be available for use after its state is recovered from Slurm's state save file or the slurmd daemon starts on the compute node. ☁️
- DOWN — indicates the node failed and is unavailable to be allocated work.
- DRAIN — indicates the node is unavailable to be allocated work.
- FAIL — indicates the node is expected to fail soon, has no jobs allocated to it, and will not be allocated to any new jobs.
- FAILING — indicates the node is expected to fail soon, has one or more jobs allocated to it, but will not be allocated to any new jobs.
- FUTURE — indicates the node is defined for future use and need not exist when the Slurm daemons are started. These nodes can be made available for use simply by updating the node state using the scontrol command rather than restarting the slurmctld daemon. After these nodes are made available, change their State in the slurm.conf file. Until these nodes are made available, they will not be seen using any Slurm commands nor will any attempt be made to contact them. FUTURE nodes retain non-FUTURE state on restart. Use scontrol to put the node back into FUTURE state 🔮

🌱 Dynamic Future Nodes 🌱

A slurmd started with -F[<feature>] will be associated with a FUTURE node that matches the same configuration (sockets, cores, threads) as reported by slurmd -C. The node's NodeAddr and NodeHostname will automatically be retrieved from the slurmd and will be cleared when set back to the FUTURE state. It's giving self-sufficient, honestly 💅

UNKNOWN — indicates the node's state is undefined but will be established (set to BUSY or IDLE) when the slurmd daemon on that node registers. UNKNOWN is the default state.

ThreadsPerCore — number of logical threads in a single physical core (like "2"). Note that Slurm can allocate resources to jobs down to the resolution of a core. If your system is configured with more than one thread per core, execution of a different job on each thread is not supported unless you configure SelectTypeParameters=CR_CPU plus CPUs — and in that case, do not configure Sockets, CoresPerSocket or ThreadsPerCore. A job can execute a one task per thread from within one job step, or execute a distinct job step on each of the threads. Also note: if you're running with more than 1 thread per core and running the select/cons_tres plugin, then you'll want to set the SelectTypeParameters variable to something other than CR_CPU to avoid unexpected results 👀 Default value is 1.
TmpDisk — total size of temporary disk storage in TmpFS in mebibytes (like "16384"). TmpFS (for "Temporary File System") identifies the location which jobs should use for temporary storage. Note this does not indicate the amount of free space available to the user on the node, only the total file system size. The system administration should ensure this file system is purged as needed so that user jobs have access to most of this space. The Prolog and/or Epilog programs (specified in the configuration file) might be used to ensure the file system is kept clean. Default value is 0 🧹
Topology — comma-separated list of pairs in the format <topology_name>:<topology_unit>. Where <topology_unit> is the block name or the name of a leaf switch. Intermediate switch names — ':' delimited — can be provided and will be created if needed (e.g. Topology=topo-tree:sw_root:s1:s2). This setting overwrites the node topology affiliation configuration specified in topology.conf(5) and topology.yaml(5) 🌈
Weight — the priority of the node for scheduling purposes. All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements. For example, a heterogeneous collection of nodes might be placed into a single partition for greater system utilization, responsiveness and capability. It would be preferable to allocate smaller memory nodes rather than larger memory nodes if either will satisfy a job's requirements. The units of weight are arbitrary, but larger weights should be assigned to nodes with more processors, memory, disk space, higher processor speed, etc. Note that if a job allocation request can not be satisfied using the nodes with the lowest weight, the set of nodes with the next lowest weight is added to the set of nodes under consideration for use (repeat as needed for higher weight values). If you absolutely want to minimize the number of higher weight nodes allocated to a job (at a cost of higher scheduling overhead), give each node a distinct Weight value and they will be added to the pool of nodes being considered for scheduling individually. The default value is 1.

NOTE: node weights are first considered among currently available nodes. For example, POWERED_DOWN and POWERING_UP nodes with lower weights will not be evaluated before a powered up IDLE node. Main character energy only for the nodes that are actually awake, periodt 🔥

💀 DOWN NODE CONFIGURATION 💀

The DownNodes= parameter permits you to mark certain nodes as in a DOWN, DRAIN, FAIL, FAILING or FUTURE state without altering the permanent configuration information listed under a NodeName= specification. Lowkey so useful, honestly 🫶

DownNodes — any node name, or list of node names, from the NodeName= specifications.
Reason — identifies the reason for a node being in state DOWN, DRAIN, FAIL, FAILING or FUTURE. Use quotes to enclose a reason having more than one word ✨
State — state of the node with respect to the initiation of user jobs. Acceptable values are DOWN, DRAIN, FAIL, FAILING and FUTURE. For more information about these states see the descriptions under State in the NodeName= section above. The default value is DOWN.

🎀 NODESET CONFIGURATION 🎀

okay so the nodeset config lets you name a specific set of nodes, and that name can simplify your partition configuration section — especially for heterogeneous or condo-style systems. Each nodeset can be defined by an explicit list of nodes, and/or by filtering nodes by a particular configured feature. If both Feature= and Nodes= are used, the nodeset shall be the union of the two subsets. Note that nodesets are only used to simplify the partition definitions at present, and are not usable outside of the partition configuration. iykyk ✨

Feature 🌸

All nodes with this feature will be included as part of this nodeset. Only a single feature is allowed. slay, keep it simple 💅

Nodes 🦋

List of nodes in this set. literally the guest list, no cap 🎀

NodeSet 👑

Unique name for a set of nodes. Must not overlap with any NodeName definitions. bestie, do not let the names collide fr fr 💀

🌈 PARTITION CONFIGURATION 🌈

The partition configuration permits you to establish different job limits or access controls for various groups (or partitions) of nodes. Nodes may be in more than one partition, making partitions serve as general purpose queues. For example, one may put the same set of nodes into two different partitions, each with different constraints — time limit, job sizes, groups allowed to use the partition, etc. Jobs are allocated resources within a single partition. main character energy, honestly 👑

Default values can be specified with a record in which PartitionName is "DEFAULT". The default entry values will apply only to lines following it in the configuration file, and the default values can be reset multiple times in the configuration file with multiple entries where PartitionName="DEFAULT". The PartitionName= specification must be placed on every line describing the configuration of partitions. Each line where PartitionName is "DEFAULT" will replace or add to previous default values and not reinitialize the default values. A single partition name can not appear as a PartitionName value in more than one line — duplicate partition name records will be ignored. If a partition that is in use is deleted from the configuration and slurm is restarted or reconfigured (scontrol reconfigure), jobs using the partition are canceled. 😭

NOTE: Put all parameters for each partition on a single line. 📝

Each line of partition configuration information should represent a different partition. The partition configuration file contains the following information:

✨ `AllocNodes` ✨

Comma-separated list of nodes from which users can submit jobs in the partition. Node names may be specified using the node range expression syntax described above. The default value is "ALL". so basically the VIP list for submission access 💖

✨ `AllowAccounts` ✨

Comma-separated list of accounts which may execute jobs in the partition. The default value is "ALL". This list is hierarchical, meaning subaccounts are included automatically.

NOTE: If AllowAccounts is used then DenyAccounts will not be enforced. Also refer to DenyAccounts. 🫶

✨ `AllowGroups` ✨

Comma-separated list of group names which may execute jobs in this partition. A user will be permitted to submit a job to this partition if AllowGroups has at least one group associated with the user. Jobs executed as user root or as user SlurmUser will be allowed to use any partition, regardless of the value of AllowGroups. In addition, a Slurm Admin or Operator will be able to view any partition, regardless of the value of AllowGroups. If user root attempts to execute a job as another user — e.g. using srun's --uid option — then the job will be subject to AllowGroups as if it were submitted by that user. By default, AllowGroups is unset, meaning all groups are allowed to use this partition. The special value 'ALL' is equivalent to this.

Users who are not members of the specified group will not see information about this partition by default. However, this should not be treated as a security mechanism, since job information will be returned if a user requests details about the partition or a specific job. See the PrivateData parameter to restrict access to job information. 🔐

NOTE: For performance reasons, Slurm maintains a list of user IDs allowed to use each partition and this is checked at job submission time. This list of user IDs is updated when the slurmctld daemon is restarted, reconfigured (e.g. "scontrol reconfig") or the partition's AllowGroups value is reset, even if its value is unchanged (e.g. "scontrol update PartitionName=name AllowGroups=group"). For a user's access to a partition to change, both their group membership must change and Slurm's internal user ID list must change using one of the methods described above. the way that permissions need two things to update is lowkey iconic 😭💅

✨ `AllowQos` ✨

Comma-separated list of Qos which may execute jobs in the partition. Jobs executed as user root can use any partition without regard to the value of AllowQos. The default value is "ALL".

NOTE: If AllowQos is used then DenyQos will not be enforced. Also refer to DenyQos. 🌟

✨ `Alternate` ✨

Partition name of alternate partition to be used if the state of this partition is "DRAIN" or "INACTIVE." backup plan energy, obsessed 🎀

✨ `CpuBind` ✨

If a job step request does not specify an option to control how tasks are bound to allocated CPUs — by using --cpu-bind — and all nodes allocated to the job do not have the same CpuBind option for the node, then the partition's CpuBind option will control how tasks are bound to allocated resources. The TaskPluginParam will be used as a last resort, with the default being no binding. Supported values for CpuBind are none, socket, ldom (NUMA), core and thread. 🧵

✨ `Default` ✨

If this keyword is set, jobs submitted without a partition specification will utilize this partition. Possible values are "YES" and "NO". The default value is "NO". the default-default is no, bestie 🙈

✨ `DefaultTime` ✨

Run time limit used for jobs that don't specify a value. If not set, the default will be set to an applicable time limit, which may be from the partition (MaxTime) or from a QOS or association (MaxWall or GrpWall). Regardless, MaxTime will still be enforced. Format is the same as for MaxTime. ⏳

✨ `DefCpuPerGPU` ✨

Default count of CPUs allocated per allocated GPU. This value is used only if the job didn't specify --cpus-per-task and --cpus-per-gpu. 🍓

✨ `DefMemPerCPU` ✨

Default real memory size available per allocated CPU in mebibytes. Used to avoid over-subscribing memory and causing paging. DefMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_tres). If not set, the DefMemPerCPU value for the entire cluster will be used. Also see DefMemPerGPU, DefMemPerNode and MaxMemPerCPU. DefMemPerCPU, DefMemPerGPU and DefMemPerNode are mutually exclusive. pick one, bestie, they cannot all be main character 💀

✨ `DefMemPerGPU` ✨

Default real memory size available per allocated GPU in mebibytes. Please note a best effort attempt is made to predict which GPUs on the system will be used, but this could change between job submission and start time, causing MaxMemPerNode to potentially not work as expected for heterogeneous jobs. Also see DefMemPerCPU, DefMemPerNode and MaxMemPerCPU. DefMemPerCPU, DefMemPerGPU and DefMemPerNode are mutually exclusive. ngl the GPU prediction is giving best-effort, not prophecy 🔮

✨ `DefMemPerNode` ✨

Default real memory size available per allocated node in mebibytes. Used to avoid over-subscribing memory and causing paging. DefMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). If not set, the DefMemPerNode value for the entire cluster will be used. Also see DefMemPerCPU, DefMemPerGPU and MaxMemPerCPU. DefMemPerCPU, DefMemPerGPU and DefMemPerNode are mutually exclusive. whole-node girlies rise 🌈

✨ `DenyAccounts` ✨

Comma-separated list of accounts which may not execute jobs in the partition. By default, no accounts are denied access. This list is hierarchical, meaning subaccounts are included automatically.

NOTE: If AllowAccounts is used then DenyAccounts will not be enforced. Also refer to AllowAccounts. 🚫

✨ `DenyQos` ✨

Comma-separated list of Qos which may not execute jobs in the partition. By default, no QOS are denied access.

NOTE: If AllowQos is used then DenyQos will not be enforced. Also refer to AllowQos. 🙅‍♀️

✨ `DisableRootJobs` ✨

If set to "YES" then user root will be prevented from running any jobs on this partition. The default value will be the value of DisableRootJobs set outside of a partition specification — which is "NO", allowing user root to execute jobs. 🔥

✨ `Exclusive` ✨

Controls partition-level exclusivity together with OverSubscribe. Specify one of "NO", "NODE", "USER", or "TOPO". NODE allocates whole nodes and implies OverSubscribe=NO. USER allows only one user to have jobs on a node at a time — similar to --exclusive=user. TOPO allows only one job per topology segment — similar to --exclusive=topo — and implies OverSubscribe=NO. For node-exclusive partitions, jobs receive all CPUs and generic resources on allocated nodes, but only as much memory as requested; use --mem=0 at submit time to request all memory on the node, notably when using gang scheduling and suspended jobs still occupy memory. If unset, no partition-level exclusivity is added beyond OverSubscribe.

This replaces the deprecated options OverSubscribe=EXCLUSIVE, ExclusiveUser, and ExclusiveTopo. ExclusiveUser=YES and ExclusiveTopo=YES are mutually exclusive. If both deprecated options are set for the same partition, ExclusiveUser will be ignored. deprecated means don't keep using them, bestie — that's the whole vibe check 🫶✨

GraceTime — bestie this one is in seconds, and it's the preemption grace time handed to a job that's been picked for preemption 💅. It literally only does anything when PreemptType=partition_prio. Default is zero — meaning no grace time at all on this partition, ruthless. Once a job gets selected for preemption, its end time is set to the current time plus GraceTime, or left alone if it was already sooner than that. So a job whose time limit expires before GraceTime elapses only gets the remainder of its time limit, not the full GraceTime — no cap, it's giving "you get what's left" 😭.

Its tasks get SIGCONT and SIGTERM immediately, as a heads-up that termination is imminent 🔔. Then upon reaching the new end time you get the SIGCONT, SIGTERM and SIGKILL sequence — that second set goes to both the tasks and the containing batch script, if applicable. See also the global KillWait configuration parameter.

NOTE: this parameter does not apply to PreemptMode=SUSPEND. For setting the preemption grace time when using PreemptMode=SUSPEND, see PreemptParameters=suspend_grace_time.

NOTE: a job selected for preemption that exits before GraceTime expires will be handled according to PreemptMode regardless of why it exited (most visible if PreemptMode=REQUEUE). Slurm literally cannot tell the difference between an exit caused by the preemption signal and an unrelated exit that happens within GraceTime. So awkward 💀.

Hidden — specifies if the partition and its jobs are hidden by default. Hidden partitions will by default not be reported by the Slurm APIs or commands 🫣. Possible values are "YES" and "NO". The default value is "NO". Note that partitions a user lacks access to by virtue of the AllowGroups parameter will also be hidden by default — sneaky but iconic.
LLN — schedules resources to jobs on the least loaded nodes, based on the number of idle CPUs ✨. Generally only recommended for an environment with serial jobs, because idle resources will tend to be highly fragmented, which results in parallel jobs being spread across many nodes 😬. Note that node Weight takes precedence over how many idle resources are on each node. Also see the SelectTypeParameters configuration parameter CR_LLN to use the least loaded nodes in every partition. Slay.
MaxCPUsPerNode — maximum number of CPUs on any node available to all jobs from this partition. This can be especially useful to schedule GPUs 👑. For example a node can be associated with two Slurm partitions (e.g. "cpu" and "gpu") and the partition/queue "cpu" could be limited to only a subset of the node's CPUs, ensuring that one or more CPUs would be available to jobs in the "gpu" partition/queue. Also see MaxCPUsPerSocket.
MaxCPUsPerSocket — maximum number of CPUs on any node available on the all jobs from this partition. This can be especially useful to schedule GPUs 🔥. Also see MaxCPUsPerNode.
MaxMemPerCPU — maximum real memory size available per allocated CPU in mebibytes. Used to avoid over-subscribing memory and causing paging 🧚. MaxMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_tres). If not set, the MaxMemPerCPU value for the entire cluster will be used. Also see DefMemPerCPU and MaxMemPerNode. MaxMemPerCPU and MaxMemPerNode are mutually exclusive — pick one, bestie, they cannot both be your fave 🙈.
MaxMemPerNode — maximum real memory size available per allocated node in a job allocation in mebibytes. Used to avoid over-subscribing memory and causing paging 🌸. MaxMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). If not set, the MaxMemPerNode value for the entire cluster will be used. Also see DefMemPerNode and MaxMemPerCPU. MaxMemPerCPU and MaxMemPerNode are mutually exclusive — same deal, only one at a time 💅.
MaxNodes — maximum count of nodes which may be allocated to any single job. The default value is "UNLIMITED", which is represented internally as -1. Feral but valid ✨.
MaxTime — maximum run time limit for jobs ⏰. Format is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds or "UNLIMITED". Time resolution is one minute and second values are rounded up to the next minute. The job TimeLimit may be updated by root, SlurmUser or an Operator to a value higher than the configured MaxTime after job submission — so yes, some people can just vibe past the cap 👀.
MinNodes — minimum count of nodes which may be allocated to any single job. The default value is 0.
Nodes — comma-separated list of nodes or nodesets which are associated with this partition. Node names may be specified using the node range expression syntax described above 🎀. A blank list of nodes (i.e. Nodes="") can be used if one wants a partition to exist, but have no resources (possibly on a temporary basis) — an empty-shell partition, honestly relatable 😭. A value of "ALL" is mapped to all nodes configured in the cluster.
OverSubscribe — controls the ability of the partition to execute more than one job at a time on each resource (node, socket or core depending upon the value of SelectTypeParameters) 👑. If resources are to be over-subscribed, avoiding memory over-subscription is very important. SelectTypeParameters should be configured to treat memory as a consumable resource and the --mem option should be used for job allocations. Sharing of resources is typically useful only when using gang scheduling (PreemptMode=suspend,gang). Possible values for OverSubscribe are "FORCE", "YES", and "NO". Whole-node exclusive partitions can be configured with Exclusive=node (see above). Note that a value of "YES" or "FORCE" can negatively impact performance for systems with many thousands of running jobs 💀. The default value is "NO". For more information see the following web pages:

https://slurm.schedmd.com/cons_tres.html
https://slurm.schedmd.com/cons_tres_share.html
https://slurm.schedmd.com/gang_scheduling.html
https://slurm.schedmd.com/preempt.html

FORCE — makes all resources (except GRES) in the partition available for oversubscription without any means for users to disable it 😤. May be followed with a colon and maximum number of jobs in running or suspended state. For example OverSubscribe=FORCE:4 enables each node, socket or core to oversubscribe each resource four ways. Recommended only for systems using PreemptMode=suspend,gang.

NOTE: OverSubscribe=FORCE:1 is a special case that is not exactly equivalent to OverSubscribe=NO. OverSubscribe=FORCE:1 disables the regular oversubscription of resources in the same partition but it will still allow oversubscription due to preemption or on overlapping partitions with the same PriorityTier. Setting OverSubscribe=NO will prevent oversubscription from happening in all cases. Important distinction, bestie, do not skip it 💅.

NOTE: if using PreemptType=preempt/qos you can specify a value for FORCE that is greater than 1. For example, OverSubscribe=FORCE:2 will permit two jobs per resource normally, but a third job can be started only if done so through preemption based upon QOS. Sneaky third-job energy 🦋.

NOTE: if OverSubscribe is configured to FORCE or YES in your slurm.conf and the system is not configured to use preemption (PreemptMode=OFF) accounting can easily grow to values greater than the actual utilization. It may be common on such systems to get error messages in the slurmdbd log stating: "We have more allocated time than is possible." 😭 Yeah. That one.

YES — makes all resources (except GRES) in the partition available for sharing upon request by the job. Resources will only be over-subscribed when explicitly requested by the user using the "--oversubscribe" option on job submission 🙋. May be followed with a colon and maximum number of jobs in running or suspended state. For example "OverSubscribe=YES:4" enables each node, socket or core to execute up to four jobs at once. Recommended only for systems running with gang scheduling (PreemptMode=suspend,gang).
NO — selected resources are allocated to a single job. No resource will be allocated to more than one job. Clean, monogamous resource allocation 🎀.

NOTE: even if you are using PreemptMode=suspend,gang, setting OverSubscribe=NO will disable preemption on that partition. Use OverSubscribe=FORCE:1 if you want to disable normal oversubscription but still allow suspension due to preemption. Do not get caught out by this one fr fr 😳.

OverTimeLimit — number of minutes by which a job can exceed its time limit before being canceled ⏳. Normally a job's time limit is treated as a hard limit and the job will be killed upon reaching that limit. Configuring OverTimeLimit will result in the job's time limit being treated like a soft limit. Adding the OverTimeLimit value to the soft time limit provides a hard time limit, at which point the job is canceled. This is particularly useful for backfill scheduling, which bases upon each job's soft time limit ✨. If not set, the OverTimeLimit value for the entire cluster will be used. May not exceed 65533 minutes. A value of "UNLIMITED" is also supported. Go off, Overtime Queen 👑.
PartitionName — the name people use to reference this partition (like "Interactive"). This is the name users type when they submit jobs. Okay so here's the ✨special✨ part: if you set PartitionName to "DEFAULT", then whatever values you put on that line become the defaults for every partition spec that comes after it — unless those later records explicitly set something different or swap in a whole new set of defaults. Each line where PartitionName is "DEFAULT" will replace or add to the previous default values, and it does not reinitialize the defaults. 🎀
PowerDownOnIdle — if you set this to "YES" and power saving is enabled for the partition, then nodes allocated from this partition get a request to power down after they've had at least one job allocated on them. These nodes won't power down until they transition from COMPLETING to IDLE. If it's "NO", power saving just does its normal partition-configured thing. Default is "NO". More details at <https://slurm.schedmd.com/power_save.html> 💖

NOTE: these things will cause a transition from COMPLETING to IDLE: - Completing all running jobs without additional jobs being allocated. - Exclusive=USER and after all running jobs complete but before another user's job is allocated. - Exclusive=NODE and after the running job completes but before another job is allocated.

NOTE: nodes allocated from this partition will still get a power down request even if they're excluded from automatic power save mode by SuspendExcNodes, SuspendExcParts, or SuspendExcStates. 🙈

NOTE: nodes are still subject to powering down when being IDLE for SuspendTime when PowerDownOnIdle is "NO".

Also see SuspendTime.

PreemptMode — the mechanism used to preempt jobs running in this partition when PreemptType=preempt/partition_prio is configured. This partition-specific PreemptMode overrides the cluster-wide PreemptMode for jobs in this partition (main character energy, honestly 👑). You can set it to OFF to stop jobs in this partition from being preempted at all. See also PriorityTier and the cluster-wide PreemptMode description above for more. Details at <https://slurm.schedmd.com/preempt.html> ✨

NOTE: for performance reasons, the backfill scheduler reserves whole nodes for jobs, not partial nodes. If during backfill scheduling a job preempts one or more other jobs, the whole nodes for those preempted jobs are reserved for the preemptor job — even if the preemptor job requested fewer resources than that. Those reserved nodes aren't available to other jobs during that backfill cycle, even if the other jobs could fit on the nodes. So yeah, jobs may preempt more resources during a single backfill iteration than they requested. 💀

NOTE: for a heterogeneous job to be considered for preemption, all components must be eligible for preemption. When a heterogeneous job is to be preempted, the first identified component of the job with the highest order PreemptMode (SUSPEND (highest), REQUEUE, CANCEL (lowest)) will be used to set the PreemptMode for all components. The GraceTime and user warning signal for each component of the heterogeneous job remain unique. Heterogeneous jobs are excluded from GANG scheduling operations.

OFF — disables job preemption.
CANCEL — the preempted job will be cancelled.
REQUEUE — preempts jobs by requeuing them (if possible) or cancelling them. For jobs to be requeued they must have the --requeue sbatch option set, or the cluster-wide JobRequeue parameter in slurm.conf must be set to 1.
SUSPEND — the preempted jobs will be suspended, and later the Gang scheduler will resume them. Therefore the SUSPEND preemption mode always needs the GANG option specified at the cluster level. Also, because suspended jobs still use memory on the allocated nodes, Slurm needs to be able to track memory resources to be able to suspend jobs.

If the preemptees and preemptor are on different partitions, then the preempted jobs will remain suspended until the preemptor ends.

NOTE: because gang scheduling is performed independently for each partition, if using PreemptType=preempt/partition_prio then jobs in higher PriorityTier partitions will suspend jobs in lower PriorityTier partitions to run on the released resources. Only when the preemptor job ends will the suspended jobs be resumed by the Gang scheduler.

NOTE: suspended jobs will not release GRES. Higher priority jobs will not be able to preempt to gain access to GRES. 😭

PRIORITY — allow preemption only if the preemptor's job priority is higher than the preemptee's job priority.

PriorityJobFactor — partition factor used by the priority/multifactor plugin when it calculates job priority. Defaults to 1. A value of 0 prevents this partition from adding to the job's priority. The value may not exceed 65533. Also see PriorityTier. 🌸
PriorityTier — jobs submitted to a partition with a higher PriorityTier value get evaluated by the scheduler before pending jobs in a partition with a lower PriorityTier value. They're also considered for preemption of running jobs in partition(s) with lower PriorityTier values if PreemptType=preempt/partition_prio. The value may not exceed 65533. Also see PriorityJobFactor. 💅
QOS — used to extend the limits available to a QOS on a partition. Jobs won't be associated to this QOS outside of being associated to the partition. They'll still be associated to their requested QOS. By default, no QOS is used. Full details live in the QOS documentation at <https://slurm.schedmd.com/qos.html>, including the special conditions when a relative QOS is used for this parameter.

NOTE: if a limit is set in both the Partition's QOS and the Job's QOS, the Partition QOS limit will be honored — unless the Job's QOS has the OverPartQOS flag set, in which case the Job's QOS limit takes precedence. 🫶

ReqResv — specifies that users of this partition are required to designate a reservation when submitting a job. Handy for restricting usage of a partition that maybe has higher priority or extra resources, so it's only allowed within a reservation. Possible values are "YES" and "NO". Default is "NO". 🎀
ResumeTimeout — max time permitted (in seconds) between when a node resume request is issued and when the node is actually available for use. Nodes that fail to respond in this time frame get marked DOWN and the jobs scheduled on the node get requeued if possible. Nodes that reboot after this time frame get marked DOWN with a reason of "Node unexpectedly rebooted." For nodes that are in multiple partitions with this option set, the highest time takes effect. If it's not set on any partition, the node uses the ResumeTimeout value set for the entire cluster. The maximum value is either 65533 or INFINITE. ⭐
RootOnly — specifies if only user ID zero (i.e. user root) or SlurmUser may allocate resources in this partition. Possible values are "YES" and "NO". Default is "NO". 🔥
SelectTypeParameters — partition-specific resource allocation type. This option replaces the global SelectTypeParameters value. Supported values are CR_Core, CR_Core_Memory, CR_Socket and CR_Socket_Memory. Using it requires the system-wide SelectTypeParameters value be set to any of those four supported values previously listed; otherwise the partition-specific value is just ignored. 💜
State — state of the partition, or availability for use. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE". Default is "UP". See also the related "Alternate" keyword. 🌈
- UP — new jobs may be queued on the partition, and jobs may be allocated nodes and run from the partition.
- DOWN — new jobs may be queued on the partition, but queued jobs may not be allocated nodes and run from the partition. Jobs already running on the partition keep running. The jobs must be explicitly canceled to force their termination.
- DRAIN — no new jobs may be queued on the partition (job submission requests will be denied with an error message), but jobs already queued on the partition may be allocated nodes and run. See also the "Alternate" partition specification.
- INACTIVE — no new jobs may be queued on the partition, and jobs already queued may not be allocated nodes and run. See also the "Alternate" partition specification.
SuspendTime — nodes that remain idle or down for this number of seconds get placed into power save mode by SuspendProgram. For nodes in multiple partitions with this option set, the highest time takes effect. If it's not set on any partition, the node uses the SuspendTime value set for the entire cluster. A SuspendTime value set on a NodeName line takes precedence over this partition-level value. Setting SuspendTime to INFINITE disables suspending of nodes in this partition. Setting SuspendTime to anything but INFINITE (or -1) enables power save mode. ⚡
SuspendTimeout — the max time (in seconds) allowed between the moment a node suspend request goes out and the moment that node is actually shut down 🫶. By the end of that window the node has to be ready for a resume request whenever new work shows up. If a node lives in multiple partitions that all set this, the highest value wins, fr fr. And if literally no partition sets it, the node just uses whatever SuspendTimeout value is set for the whole cluster. easy.
Topology — the name of the topology (defined over in topology.yaml) that jobs in this partition should use ✨.
TRESBillingWeights — comma-separated list of <TRES Type>=<Numeric Weight> pairs 💅. This defines the billing weights for one or more tracked TRES types (peep AccountingStorageTRES) that get used when calculating a job's usage in this partition. That resulting usage amount is what gets used when calculating fairshare and when enforcing the TRES billing limit on jobs. Also see PriorityWeightTRES if you wanna adjust job priority directly based on TRES usage instead.

By default, jobs are billed against the total number of allocated CPUs — that's TRESBillingWeights="CPU=1" 👑.

More deets on the TRES page: <https://slurm.schedmd.com/tres.html>

⚡ POWERACTION CONFIGURATION ⚡

okay so this whole config block lets you create a named program that does a node reboot, a power up (Resume), or a power down (Suspend) 🔥. A PowerAction name can be dropped in wherever a program path would go for ResumeProgram, ResumeFailProgram, SuspendProgram, or RebootProgram — and you can also pick it for individual requests with the "action" option to scontrol power and scontrol reboot. iconic.

PowerAction=<name> Location=<slurmctld|slurmd> Program="<path> [arguments...]"

It includes the following parameters:

PowerAction — a unique name for the action 🎀. This is the name that ResumeProgram, ResumeFailProgram, SuspendProgram, RebootProgram, or the "action" option to scontrol will reference.
Location — where the program runs 🦋. Supported values are "slurmd" and "slurmctld". "slurmctld" runs the program on the controller as SlurmUser. "slurmd" runs the program on each affected compute node as SlurmdUser.

NOTE 💀: power up actions must use "slurmctld", because powered-down nodes literally cannot run a "slurmd" action. no cap.

Program — the program that does the task of power up, power down, or reboot ✨. Arguments you want passed to it can be included right in this field. Slurm passes the affected node name(s) as an argument after any user-specified arguments. With Location=slurmd, SLURM_NODE_NAME is also set in the environment 🌸.

🎬 PROLOG AND EPILOG SCRIPTS 🎬

there are a whole bunch of prolog and epilog program options, and they run with different permissions at different times — it's giving very "many hats" energy ngl 🙈. The four you're most likely to actually use are: Prolog and Epilog (executed once on each compute node, for each job) plus PrologSlurmctld and EpilogSlurmctld (executed once on the SlurmctldHost, for each job) 💖.

NOTE 💅: standard output and error messages are normally not preserved. If you want that info kept, explicitly write your output and error messages to an appropriate location. periodt.

NOTE 🫶: by default the Prolog script is ONLY run on any individual node when it first sees a job step from a new allocation. It does not run the Prolog immediately when an allocation is granted. If no job steps from an allocation ever run on a node, that node will never run the Prolog for that allocation. You can change this Prolog behavior with the PrologFlags parameter. The Epilog, on the other hand, always runs on every node of an allocation when the allocation is released. main character behavior honestly.

so what happens when things go wrong 😭:

If the Epilog fails (returns a non-zero exit code) → the node gets set to a DRAIN state 💀.
If the EpilogSlurmctld fails (returns a non-zero exit code) → this will only be logged.
If the Prolog fails (returns a non-zero exit code) → the node gets set to a DRAIN state and the job gets requeued. The job will be placed in a held state unless nohold_on_prolog_fail is configured in SchedulerParameters.
If the PrologSlurmctld fails (returns a non-zero exit code) → the job gets requeued to be executed on another node if possible. Only batch jobs can be requeued tho — interactive jobs (salloc and srun) will be cancelled if the PrologSlurmctld fails 😭.
If slurmctld is stopped while either PrologSlurmctld or EpilogSlurmctld is running, the script will be killed with SIGKILL. The script will restart when slurmctld restarts 🔁.

Info about the job gets passed to the script using environment variables 🌈. For the full list of env vars, see the Prolog and Epilog Guide <https://slurm.schedmd.com/prolog_epilog.html>.

💀 UNKILLABLE STEP PROGRAM SCRIPT 💀

this program exists so you can take special actions to clean up unkillable processes and/or notify your sysadmins 🎀. It runs as SlurmdUser (usually "root") on the compute node where UnkillableStepTimeout was triggered.

Info about the unkillable job step gets passed to the script using environment variables ✨:

SLURM_JOB_ID — job ID.
SLURM_JOB_SLUID — job SLUID.
SLURM_STEP_ID — job step ID. Heads up: the special steps "batch", "interactive", and "extern" are reported not by name but with integer Step IDs 4294967291, 4294967290, and 4294967292 respectively 🫶. iykyk.

🌐 NETWORK TOPOLOGY 🌐

Slurm is able to optimize job allocations to minimize network contention 👑. Special Slurm logic is used to optimize allocations on systems with a three-dimensional interconnect, and info about configuring those systems is available on web pages right here: <https://slurm.schedmd.com/>. For a hierarchical network, Slurm needs detailed information about how nodes are configured on the network switches.

The TopologyPlugin parameter controls which plugin is used to collect network topology information 🦋. The only values presently supported are "topology/flat" (best-fit logic over one-dimensional topology), "topology/tree", and "topology/block" (both of which determine the network topology based on information contained in a topology.conf or topology.yaml file — see "man topology.conf" and "man topology.yaml" for more info). Future plugins may gather topology information directly from the network ✨.

The topology information is optional. If you don't provide it, Slurm will perform a best-fit algorithm assuming the nodes are in a one-dimensional array as configured, and that the communications cost is related to the node distance in this array 💖.

🚚 RELOCATING CONTROLLERS 🚚

if the cluster's computers used for the primary or backup controller are gonna be out of service for an extended period of time, you might wanna relocate them 🫶. To do that, follow this procedure:

Stop the Slurm daemons on the old controller and nodes.
Modify the slurm.conf file appropriately.
Copy the files from the StateSaveLocation to the new controller, or make sure they're accessible to the new controller via a shared drive.
Distribute the updated slurm.conf file to all nodes.
Restart the Slurm daemons on the new controller and nodes.

There should be no loss of any pending jobs 🙌. Any running jobs will get the updated host info and finish normally. Just make sure any nodes added to the cluster have the current slurm.conf file installed.

CAUTION 🔥: if two nodes are simultaneously configured as the primary controller (two nodes on which SlurmctldHost specify the local host and the slurmctld daemon is executing on each), system behavior will be destructive 💀. If a compute node has an incorrect SlurmctldHost parameter, that node may be rendered unusable, but no other harm will result.

🌸 EXAMPLE 🌸

#
# Sample /etc/slurm.conf for dev[0-25].llnl.gov
# Author: John Doe
# Date: 11/06/2001
#
SlurmctldHost=dev0(12.34.56.78)  # Primary server
SlurmctldHost=dev1(12.34.56.79)  # Backup server
#
AuthType=auth/munge
Epilog=/usr/local/slurm/epilog
Prolog=/usr/local/slurm/prolog
FirstJobId=65536
InactiveLimit=120
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm/jobcomp
KillWait=30
MaxJobCount=10000
MinJobAge=300
PluginDir=/usr/local/lib:/usr/local/slurm/lib
ReturnToService=0
SchedulerType=sched/backfill
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmctldPort=7002
SlurmdPort=7003
SlurmdSpoolDir=/var/spool/slurmd.spool
StateSaveLocation=/var/spool/slurm.state
TmpFS=/tmp
WaitTime=30
#
# Node Configurations
#
NodeName=DEFAULT CPUs=2 RealMemory=2000 TmpDisk=64000
NodeName=DEFAULT State=UNKNOWN
NodeName=dev[0-25] NodeAddr=edev[0-25] Weight=16
# Update records for specific DOWN nodes
DownNodes=dev20 State=DOWN Reason="power,ETA=Dec25"
#
# Partition Configurations
#
PartitionName=DEFAULT MaxTime=30 MaxNodes=10 State=UP
PartitionName=debug Nodes=dev[0-8,18-25] Default=YES
PartitionName=batch Nodes=dev[9-17]  MinNodes=4
PartitionName=long Nodes=dev[9-17] MaxTime=120 AllowGroups=admin

🎀 INCLUDE MODIFIERS 🎀

the "include" key word can be used with modifiers inside the specified pathname ✨. These modifiers get replaced with the cluster name or other information depending on which modifier you've specified. If the included file isn't an absolute pathname (i.e. it doesn't start with a slash), it has to be relative to the directory containing the slurm.conf file 🫶.

%c — the cluster name specified in the slurm.conf will be used 💅.

✨ EXAMPLE ✨

ClusterName=linux
include /home/slurm/etc/%c_config
# Above line interpreted as
# "include /home/slurm/etc/linux_config"

💖 FILE AND DIRECTORY PERMISSIONS 💖

okay so real talk: there are three whole classes of files here and you gotta keep them straight or nothing works, bestie 🫶

Class 1 — files used by slurmctld need to be accessible by user SlurmUser and reachable from the primary and backup control machines.
Class 2 — files used by slurmd need to be accessible by user root and reachable from every single compute node.
Class 3 — a few files need to be accessible by normal users on all login and compute nodes.

and like, yes, there's a long list below, but ngl most of these files won't even be used with most configurations — so don't spiral 💅

Epilog — must be executable by user root. It's recommended that the file be readable by all users. And the file must exist on every compute node.
EpilogSlurmctld — must be executable by user SlurmUser. Recommended: readable by all users. The file must be accessible by the primary and backup control machines.
HealthCheckProgram — must be executable by user root. Recommended: readable by all users. Must exist on every compute node.
JobCompLoc — if this specifies a file, it must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.
MailProg — must be executable by user SlurmUser. Must not be writable by regular users (seriously, don't). The file must be accessible by the primary and backup control machines.
PowerAction — two whole modes, keep up 🦋
- If Location=slurmd, then the Program must be executable by user SlurmdUser and be accessible on all compute nodes.
- If Location=slurmctld, then the Program must be executable by user SlurmUser and be accessible by the primary and backup control machines.
Prolog — must be executable by user root. Recommended: readable by all users. Must exist on every compute node.
PrologSlurmctld — must be executable by user SlurmUser. Recommended: readable by all users. Must be accessible by the primary and backup control machines.
ResumeProgram — must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.
slurm.conf — readable to all users on all nodes. Must not be writable by regular users. periodt 🔒
SlurmctldLogFile — must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.
SlurmctldPidFile — must be writable by user root. Preferably writable and removable by SlurmUser. The file must be accessible by the primary and backup control machines.
SlurmdLogFile — must be writable by user root. A distinct file must exist on each compute node.
SlurmdPidFile — must be writable by user root. A distinct file must exist on each compute node.
SlurmdSpoolDir — must be writable by user root. Permissions must be set to 755 so that job scripts can be executed from this directory. A distinct file must exist on each compute node.
SrunEpilog — must be executable by all users. The file must exist on every login and compute node.
SrunProlog — must be executable by all users. The file must exist on every login and compute node.
StateSaveLocation — must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.
SuspendProgram — must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.
TaskEpilog — must be executable by all users. The file must exist on every compute node.
TaskProlog — must be executable by all users. The file must exist on every compute node.
UnkillableStepProgram — must be executable by user SlurmdUser. The file must be accessible by the primary and backup control machines.

📝 LOGGING 📝

okay so here's a thing that lowkey trips people up 😭 — the Slurm daemons do create log files and other files as needed, but they treat a missing parent directory as a fatal error. Like, absolutely fatal. no cap.

and honestly that's a feature, bestie 💖 — it stops the daemons from running at all if critical file systems aren't mounted, and it minimizes the risk of cold-starting (that's starting without preserving jobs, which, oof 💀).

Log files and job accounting files may need to be created/owned by the "SlurmUser" uid to be successfully accessed. Use the chown and chmod commands to set ownership and permissions appropriately. And go peek at the FILE AND DIRECTORY PERMISSIONS section above for the deets on all the various files and directories Slurm uses 🫶

It's recommended that you use the logrotate utility to make sure your log files don't get absolutely enormous. This also applies to the text files used for accounting, process tracking, and the slurmdbd log, if you're using those.

Here's a sample logrotate configuration — make your appropriate site modifications and save it as /etc/logrotate.d/slurm on all nodes. See the logrotate man page for more details ✨

##
# Slurm Logrotate Configuration
##
/var/log/slurm/*.log {
     compress
     missingok
     nocopytruncate
     nodelaycompress
     nomail
     notifempty
     noolddir
     rotate 5
     sharedscripts
     size=5M
     create 640 slurm root
     postrotate
          pkill -x --signal SIGUSR2 slurmctld
          pkill -x --signal SIGUSR2 slurmd
          pkill -x --signal SIGUSR2 slurmdbd
          exit 0
     endscript
}

©️ COPYING ©️

like, legal era, but read it, it matters 👑

Copyright (C) 2002-2007 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). Copyright (C) 2008-2010 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

📂 FILES 📂

/etc/slurm.conf — the whole vibe lives here 🎀

and that's the whole page, bestie!! you made it, I'm so proud of you 🥹💖 go forth and configure slay-urm. this was Slurm 26.05, Slurm Configuration File, slurm.conf(5) — catch you in the next man page, mwah 🫶✨🌸

📦 where this page comes from

arch rolling: slurm-llnl 26.05.5.1-1 · GPL-2.0-or-later
epel 10: slurm 26.05.5-1.el10_3 · GPL-2.0-or-later AND BSD-2-Clause AND BSD-1-Clause
nixpkgs unstable: slurm 26-05-5-1
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms

slurm.conf(5)

💖 NAME 💖

✨ DESCRIPTION ✨

🌸 PARAMETERS 🌸

AcctGatherNodeFreq 🌸

AcctGatherEnergyType 💜

AcctGatherInterconnectType 🦋

AcctGatherFilesystemType ✨

AcctGatherProfileType 🌈

AllowSpecResourcesUsage 🔥

AuthAltTypes 🎀

AuthAltParameters 💜

🫶 AuthInfo — extra auth deets for daemon/client chatter

🔥 AuthType — the auth method for Slurm components

⏰ BatchStartTimeout — the launch window, bestie

🦋 BcastExclude — dirs to skip when broadcasting

🎀 BcastParameters — sbcast / srun --bcast vibes

💜 BurstBufferType — burst buffer plugin energy

🧚 CertgenParameters — certgen plugin options

⭐ CertgenType — which certgen plugin, exactly

🌈 CertmgrParameters — certmgr plugin params

💖 CertmgrType

🌸 CliFilterParameters

🦋 CliFilterPlugins

⭐ ClusterName

🫶 CommunicationParameters

🧚 CompleteWait

🌈 CpuFreqDef

🍓 CpuFreqGovernors

💜 CredType

DataParserParameters=<data_parser>

DebugFlags

DefCpuPerGPU

DefMemPerCPU

DefMemPerGPU

DefMemPerNode

DependencyParameters

DisableRootJobs

EioTimeout

EnforcePartLimits

HealthCheckProgram 💖

HealthCheckTimeout ⏱️

HttpParserType 🌐

InactiveLimit 💀

InteractiveStepOptions 🎀

JobAcctGatherType 📊

JobAcctGatherFrequency ⏳

JobAcctGatherParams 🎛️

💾 UsePss

🗄️ JobCompHost

📍 JobCompLoc

🎛️ JobCompParams

🔑 JobCompPass

🧚 JobCompPassScript

🌸 SLURM_STORAGE_HOSTNAME

🌸 SLURM_STORAGE_PORT

🌸 SLURM_STORAGE_USER

🔌 JobCompPort

🏷️ JobCompType

JobCompUser 👑

JobFileAppend 🎀

JobRequeue 🔁

JobSubmitPlugins 🧚

KillOnBadExit 💀

KillWait ⏳

MaxBatchRequeue 🔂

NodeFeaturesPlugins 🦋

LaunchParameters 🚀

LicenseParameters

Licenses

LogTimeFormat

MailDomain

MailProg

MaxArraySize

MaxDBDMsgs

MaxJobCount

MaxJobId

MaxMemPerCPU

MaxMemPerNode

MaxNodeCount

`AcctGatherNodeFreq` 🌸

`AcctGatherEnergyType` 💜

`AcctGatherInterconnectType` 🦋

`AcctGatherFilesystemType` ✨

`AcctGatherProfileType` 🌈

`AllowSpecResourcesUsage` 🔥

`AuthAltTypes` 🎀

`AuthAltParameters` 💜

🫶 `AuthInfo` — extra auth deets for daemon/client chatter

🔥 `AuthType` — the auth method for Slurm components

⏰ `BatchStartTimeout` — the launch window, bestie

🦋 `BcastExclude` — dirs to skip when broadcasting

🎀 `BcastParameters` — `sbcast` / `srun --bcast` vibes

💜 `BurstBufferType` — burst buffer plugin energy

🧚 `CertgenParameters` — certgen plugin options

⭐ `CertgenType` — which certgen plugin, exactly

🌈 `CertmgrParameters` — certmgr plugin params

💖 `CertmgrType`

🌸 `CliFilterParameters`

🦋 `CliFilterPlugins`

⭐ `ClusterName`

🫶 `CommunicationParameters`

🧚 `CompleteWait`

🌈 `CpuFreqDef`

🍓 `CpuFreqGovernors`

💜 `CredType`

`DataParserParameters=<data_parser>`

`DebugFlags`

`DefCpuPerGPU`

`DefMemPerCPU`

`DefMemPerGPU`

`DefMemPerNode`

`DependencyParameters`

`DisableRootJobs`

`EioTimeout`

`EnforcePartLimits`

`HealthCheckProgram` 💖

`HealthCheckTimeout` ⏱️

`HttpParserType` 🌐

`InactiveLimit` 💀

`InteractiveStepOptions` 🎀

`JobAcctGatherType` 📊

`JobAcctGatherFrequency` ⏳

`JobAcctGatherParams` 🎛️

💾 `UsePss`

🗄️ `JobCompHost`

📍 `JobCompLoc`

🎛️ `JobCompParams`

🔑 `JobCompPass`

🧚 `JobCompPassScript`

🌸 `SLURM_STORAGE_HOSTNAME`

🌸 `SLURM_STORAGE_PORT`

🌸 `SLURM_STORAGE_USER`

🔌 `JobCompPort`

🏷️ `JobCompType`

`JobCompUser` 👑

`JobFileAppend` 🎀

`JobRequeue` 🔁

`JobSubmitPlugins` 🧚

`KillOnBadExit` 💀

`KillWait` ⏳

`MaxBatchRequeue` 🔂

`NodeFeaturesPlugins` 🦋

`LaunchParameters` 🚀

`MCSPlugin` 🌸

`MessageTimeout` ⏱️

`MetricsAuthUsers` 🔐

`MetricsParameters` 📊

`MetricsType` ✨

`MinJobAge` 🕰️

`MpiDefault` 🧚

`MpiParams` 🌈

`NamespaceType` 🦋

`OverTimeLimit` ⏳

`PluginDir` 📁

`PlugStackConfig` 🎀

`PreemptMode` 👑

🌸 `PreemptParameters`

🦋 `PreemptType`

💜 `PreemptExemptTime`