scontrol(1)

📦

version 26.05.5.1 · 10 versions

26.05.5.1 · arch rolling, epel 10, nixpkgs unstable (latest)
25-11-8 · nixos 26.05
22.05.11 · epel 9
26.05.4 · ubuntu stonking
21.08.5 · ubuntu jammy
24.05.2 · fedora 43, fedora 44
23.11.4 · ubuntu noble
22.05.8 · debian bookworm
25.11.2 · ubuntu resolute
24.11.5 · debian trixie, opensuse-leap 16.0, ubuntu questing

, in arch rolling, epel 10, nixpkgs unstable

okay bestie, welcome to scontrol(1), straight out of the Slurm Commands manual. This is the tool that lets you peek at or change Slurm's config and state, and yes, it's giving main-character energy for cluster admins. 💅

🌸 NAME 🌸

scontrol — view or modify Slurm configuration and state. That's the whole assignment. ✨

✨ SYNOPSIS ✨

scontrol [OPTIONS...] [COMMAND...]

💅 DESCRIPTION 💅

So scontrol is how you view or modify Slurm configuration, including job, job step, node, partition, reservation, and overall system configuration. Most commands need elevated permissions to run — also see the AUTHORIZATION section. If a user runs a command without the required permissions, an error message gets printed and the requested action will not occur. No sneaky workaround, bestie. 🙈

If no command is entered on the execute line, scontrol operates in interactive mode and prompts for input. It keeps prompting for input and executing commands until explicitly terminated. If a command is entered on the execute line, scontrol executes that command and terminates. Very "one and done" of it. ✨

All commands and options are case-insensitive, but node names, partition names, and reservation names are case-sensitive — node names LX and lx are distinct, so don't play yourself. All commands and options can be abbreviated to the extent that the specification is unique. A modified Slurm configuration can be written to a file using the scontrol write config command. The resulting file will be named using the convention slurm.conf.<datetime> and located in the same directory as the original slurm.conf file. The directory containing the original slurm.conf must be writable for this to occur. No writable directory, no config file — periodt. 💀

🎀 OPTIONS 🎀

Here's the whole option menu, bestie. No skipping. 🫶

-a, --all — When the show command is used, display all partitions, their jobs and jobs steps. This causes information to be displayed about partitions that are configured as hidden and partitions that are unavailable to user's group.
-M, --clusters=<string> — The cluster to issue commands to. Only one cluster name may be specified. Note that the slurmdbd must be up for this option to work properly, unless running in a federation with either FederationParameters=fed_display configured or the --federation option set. This option implicitly sets the --local option.
-d, --details — Causes the show command to provide additional details where available.
--federation — Report jobs from federation if a member of one.
-F, --future — Report nodes in FUTURE state.
-h, --help — Print a help message describing the usage of scontrol.
--hide — Do not display information about hidden partitions, their jobs and job steps. By default, neither partitions that are configured as hidden nor those partitions unavailable to user's group will be displayed (i.e. this is the default behavior).
--json, --json=list, --json=<data_parser> — Dump information as JSON using the default data_parser plugin or explicit data_parser with parameters. All information is dumped, even if it would normally not be. Sorting and formatting arguments passed to other options are ignored; however, most filtering arguments are still used. This option is not available for every command. This option implicitly sets the --details option.
--local — Show only information local to this cluster. Ignore other clusters in the federated if a member of one. Overrides --federation.
-o, --oneliner — Print information one line per record.
-Q, --quiet — Print no warning or informational messages, only fatal error messages.
--sibling — Show all sibling jobs on a federated cluster. Implies --federation.
-u, --uid=<uid> — Attempt to update a job as user <uid> instead of the invoking user id.
-v, --verbose — Print detailed event logging. Multiple -v's will further increase the verbosity of logging. By default only errors will be displayed.
-V, --version — Print version information and exit.
--yaml, --yaml=list, --yaml=<data_parser> — Dump information as YAML using the default data_parser plugin or explicit data_parser with parameters. All information is dumped, even if it would normally not be. Sorting and formatting arguments passed to other options are ignored; however, most filtering arguments are still used. This option is not available for every command. This option implicitly sets the --details option.

🦋 COMMANDS 🦋

Okay, command time. Signatures stay exact, descriptions get the vibe. 💜

cancel_reboot <NodeList> — Cancel pending reboots on nodes. The node will be undrain'ed and the reason cleared if the node was drained by an ASAP reboot.
create <SPECIFICATION> — Create a new node, partition, or reservation. See the full list of parameters below.
completing — Display all jobs in a COMPLETING state along with associated nodes in either a COMPLETING or DOWN state.
delete <SPECIFICATION> — Delete the entry with the specified SPECIFICATION. The following SPECIFICATION choices are available:
- NodeName=<nodelist> — Node name or list of node names to delete. Only dynamic nodes that have no running jobs and that are not part of a reservation can be deleted. Multiple node names may be specified using simple node range expressions (e.g., "lx[10-20]"). Nodesets can also be specified by themselves or mixed with node range expressions, using a comma as a list separator. If the keyword "ALL" is specified alone, then the delete will be attempted against all the nodes in the local cluster (and will fail on nodes ineligible for deletion).
- PartitionName=<name> — Name of partition to delete. The partition must have no associated jobs running or pending, otherwise the request is denied. It may be helpful to modify pending jobs or to change the partition state ahead of time. Note that if this partition is present in the slurm.conf file, it will be recreated the next time the controller is restarted or reconfigured.
- ReservationName=<name> — Name of reservation to delete. The reservation must have no associated jobs, otherwise the request is denied. It may be helpful to modify pending jobs to not use the reservation. You can also plan the reservation's deletion ahead of time by setting an EndTime or Duration. A privileged user can delete any reservation. An unprivileged user may be able to delete a reservation they have access to if SlurmctldParameters=user_resv_delete is set in slurm.conf or if Flags=USER_DELETE is set on the reservation.
errnumstr <ERRNO> — Given a Slurm error number, return a descriptive string.
fsdampeningfactor <FACTOR> — Set the FairShareDampeningFactor in slurmctld.
getaddrs <NODES> — Get IP addresses of <NODES> from slurmctld.
getent [<NodeName>] — Print entities (users and groups) associated with each step running on the host that invokes this command. This contacts any slurmstepd's running locally, and does not contact slurmctld. Use <NodeName> if using --enable-multiple-slurmd.

NOTE: This command will not work for any steps launched while LaunchParameters=disable_send_gids is set in slurm.conf.

help — Display a description of scontrol options and commands.
hold <job_list> — Prevent a pending job from being started (sets its priority to 0). The job_list argument is a space separated list of job IDs (numeric or SLUID) OR "jobname=" with the job's name, which will attempt to hold all jobs having that name. Only a privileged user, account coordinator, or job owner may hold jobs. If issued by a privileged user an admin-hold will be placed on the job, otherwise a user-hold will be placed on the job. The hold type determines which users may remove the hold with the release command (also see uhold). Additionally, attempting to hold a running job will not suspend or cancel it. But, it will set the job priority to 0 and update the job reason field, which would hold the job if it was requeued at a later time.
listjobs [<NodeName>] — Print jobs running on the host that runs this. This contacts any slurmstepd's running locally, and does not contact slurmctld.

Use <NodeName> if using --enable-multiple-slurmd.

listpids [<job_id>[.<step_id>]] [<NodeName>] — Print process IDs in a job step (if JOBID.STEPID is provided), or all of the job steps in a job (if job_id is provided), or all of the job steps in all of the jobs on the local node (if job_id is not provided or job_id is "*"). This will work only with processes on the node on which scontrol is run, and only for those processes spawned by Slurm and their descendants. Note that some Slurm configurations (ProctrackType value of pgid) are unable to identify all processes associated with a job or job step.

Note that the NodeName option is only really useful when you have multiple slurmd daemons running on the same host machine. Multiple slurmd daemons on one host are, in general, only used by Slurm developers.

liststeps [<NodeName>] — Print steps running on the host that runs this. This contacts any slurmstepd's running locally, and does not contact slurmctld.

Use <NodeName> if using --enable-multiple-slurmd.

notify <job_id> <message> — Send a message to standard error of the salloc or srun command or batch job associated with the specified job_id.
pidinfo <proc_id> — Print the Slurm job id and scheduled termination time corresponding to the supplied process id, proc_id, on the current node. This will work only with processes on node on which scontrol is run, and only for those processes spawned by Slurm and their descendants.
ping — Ping the primary and secondary slurmctld daemon and report if they are responding. If --clusters is specified it will only ping the primary slurmctld daemon of the specified cluster.

💅 `power` — the glow-up/glow-down button

power {up|down} [asap|force] {ALL|<NodeList>|<NodeSet>} [Action=<name>] [Reason=<reason>]

Okay so this one literally controls the power state of the node list/set you hand it, bestie ⚡

For power down, the optional ASAP flag gets added to the power down request — but it's highkey rejected for power up requests, so don't even try it 🙅
Every argument is processed insensitive of case... except for the node list/set. That one stays exactly how you typed it, periodt.
The optional reason can be specified only when powering down.
If "action" is set, the PowerAction with the specified name is used instead of the configured ResumeProgram or SuspendProgram for this request. Power up actions must be configured with Location=slurmctld, no cap.

This subcommand obsoletes the prior usage of scontrol's update command:

scontrol update NodeName=<nodes>
                State={POWER_UP|POWER_DOWN|POWER_DOWN_ASAP|POWER_DOWN_FORCE}

The actual commands 👑

`down`

Uses the configured SuspendProgram program to explicitly place node(s) into power saving mode. If a node is already in the process of being powered down, the command will only change the state of the node but won't have any effect until the configured SuspendTimeout is reached. This one's a lifesaver when a ResumeProgram — like capmc on Cray machines — is stalled and you want to restore the node to "IDLE" manually. In that case, rebooting the node and setting the state to "power down" will cancel the previous "power up" state and the node becomes "IDLE" ✨

`down asap`

Drains the node(s) and marks them for power down. Currently running jobs will complete first, and no additional jobs get allocated to the node(s). Very "finish your food before you leave the table" of it 🍽️

`down force`

Cancels all jobs on the node(s), powers them down, and resets their state to "IDLE". No survivors, bestie 💀

`up`

Uses the configured ResumeProgram program to explicitly move node(s) out of power saving mode. If a node is already in the process of being powered up, the command will only change the state of the node but won't have any effect until the configured ResumeTimeout is reached 🌸

🔄 `reboot` — the "turn it off and on again" of the cluster

reboot [ASAP] [FORCE] [nextstate={RESUME|DOWN}]
[reason=<reason>][action=<power action name>] {ALL|<NodeList>|<NodeSet>}

Reboots the nodes in the system when they become idle, using the RebootProgram as configured in Slurm's slurm.conf file. If "action" is set, the PowerAction by the specified name gets used instead, iykyk.

Each node gets the "REBOOT" flag added to its node state. After a node reboots and the slurmd daemon starts up again, the HealthCheckProgram will run once. Then slurmd registers itself with the slurmctld daemon and the "REBOOT" flag is cleared 🎀

The "ASAP" option adds the "DRAIN" flag to each node's state, preventing additional jobs from running on the node so it can be rebooted and returned to service "As Soon As Possible" (i.e. ASAP). "ASAP" will also set the node reason to "Reboot ASAP" if the "reason" option isn't specified, and will set nextstate=UNDRAIN if nextstate isn't specified.
The "FORCE" option will ignore the current state and any jobs running, resetting reboot and power states, requeueing jobs, and requesting immediate reboot. So dramatic 💅
If the "nextstate" option is "DOWN", the node stays in a down state after rebooting. If "nextstate" is "RESUME", the nodes resume as normal and the node's reason and "DRAIN" state get cleared. Resuming nodes are considered available in backfill future scheduling and won't be replaced by idle nodes in a reservation.
The "reason" option sets each node's reason to a user-defined message. A default reason of "reboot requested" is set if no other reason is set on the node.
The reason will be appended with: "reboot issued" when the reboot is issued; "reboot complete" when the node registers and has a "nextstate" of "DOWN"; or "reboot timed out" when the node fails to register within ResumeTimeout.

You must specify either a list of nodes or that ALL nodes are to be rebooted. That part is not optional, bestie 🫶

NOTE: The reboot request will be ignored for hosts in the following states: FUTURE, POWER_DOWN, POWERED_DOWN, POWERING_DOWN, REBOOT_ISSUED, REBOOT_REQUESTED

NOTE: By default, this command does not prevent additional jobs from being scheduled on any nodes before reboot. To do that, you can either use the "ASAP" option or explicitly drain the nodes beforehand. You can alternately create an advanced reservation to prevent additional jobs from being initiated on nodes to be rebooted. Pending reboots can be cancelled with scontrol cancel_reboot <node> or by setting the node state to "CANCEL_REBOOT". A node will be marked "DOWN" if it doesn't reboot within ResumeTimeout 😭

🧚 `reconfigure` — the "everyone re-read the group chat" command

Instruct all slurmctld and slurmd daemons to re-read the configuration file. This mechanism can be used to modify configuration parameters set in slurm.conf(5) without interrupting running jobs 🔥

Starting in 23.11, this command operates by creating new processes for the daemons, then passing control to the new processes when or if they start up successfully. This lets it gracefully catch configuration problems and keep running with the previous configuration if there's a problem. Iconic, honestly.

It will not be able to change the daemons' listening TCP port settings or authentication mechanism — so don't come for it about those, it simply won't 💅

🔓 `release` — free the jobs

release <job_list>

Release a previously held job to begin execution. The job_list argument is a space separated list of job IDs (numeric or SLUID) OR "jobname=" with the job's name, which will attempt to release all jobs having that name.

Note that a privileged user may release an admin-hold or a user-hold. An unprivileged job owner or coordinator may only release a user-hold. Also see hold 🎀

♻️ `requeue` — do it again, but pending this time

requeue  [<option>] <job_list>

Requeue a running, suspended or finished Slurm batch job into pending state. The job_list argument is a comma separated list of job IDs. The command accepts the following option:

Incomplete — Operate only on jobs (or tasks of a job array) which have not completed. Specifically, only jobs in the following states will be requeued: CONFIGURING, RUNNING, STOPPED or SUSPENDED.

🧸 `requeuehold` — requeue, but make it held

requeuehold [<option>] <job_list>

Requeue a running, suspended or finished Slurm batch job into pending state, and moreover the job is put in held state (priority zero). The job_list argument is a comma separated list of job IDs. A held job can be released using scontrol to reset its priority (e.g. "scontrol release <job_id>"). The command accepts the following options:

Incomplete — Operate only on jobs (or tasks of a job array) which have not completed. Specifically, only jobs in the following states will be requeued: CONFIGURING, RUNNING, STOPPED or SUSPENDED.
State=SpecialExit — The "SpecialExit" keyword specifies that the job has to be put in a special state JOB_SPECIAL_EXIT. The "scontrol show job" command will display the JobState as SPECIAL_EXIT, while the "squeue" command as SE. Two different names for the same vibe, we love consistency 😭

▶️ `resume` — un-pause the job

resume <job_list>

Resume a previously suspended job. The job_list argument is a comma separated list of job IDs. Also see suspend.

NOTE: A suspended job releases its CPUs for allocation to other jobs. Resuming a previously suspended job may result in multiple jobs being allocated the same CPUs, which could trigger gang scheduling with some configurations or severe degradation in performance with other configurations. Use of the scancel command to send SIGSTOP and SIGCONT signals would stop a job without releasing its CPUs for allocation to other jobs and would be a preferable mechanism in many cases. If you're doing system maintenance you may want to use suspend/resume in the following way: before suspending, set all nodes to draining or set all partitions to down so that no new jobs can be scheduled. Then suspend jobs. Once maintenance is done, resume jobs, then resume nodes and/or set all partitions back to up. Use with caution. Only a privileged user or an account coordinator can resume jobs 🙈

📝 `schedloglevel` — talk to the scheduler

schedloglevel <LEVEL>

Enable or disable scheduler logging. LEVEL may be "0", "1", "disable" or "enable". "0" has the same effect as "disable". "1" has the same effect as "enable".

This value is temporary and will be overwritten when the slurmctld daemon reads the slurm.conf configuration file (e.g. when the daemon is restarted or scontrol reconfigure is executed) if the SlurmSchedLogLevel parameter is present. So yeah, it's giving "fleeting" 💀

🐛 `setdebug` — turn the volume up

setdebug <LEVEL> [nodes=<NODES>]

Change the debug level of the slurmctld daemon for all active logging channels not originally configured off (quiet). LEVEL may be an integer value between zero and nine (using the same values as SlurmctldDebug in the slurm.conf file) or the name of the most detailed message type to be printed: "quiet", "fatal", "error", "info", "verbose", "debug", "debug2", "debug3", "debug4", or "debug5".

This value is temporary, and will be overwritten whenever the daemon reads the slurm.conf configuration file. (Such as when the daemon is restarted or scontrol reconfigure is executed.) Same temporary-energy as above, ngl.

nodes=<NodeName> — If set, the request to change debug level is sent to the slurmd processes on the nodes instead of to the slurmctld. A node range expression may be used for NodeName. A whole range, bestie, we're not typing 200 hostnames 🌈

🚩 `setdebugflags` — flip the flags

setdebugflags {+|-}<FLAG> [{+|-}<FLAG>] [nodes=<NODES>]

Add or remove DebugFlags of the slurmctld daemon. See "man slurm.conf" for a list of supported DebugFlags. Plus to add, minus to remove — the way that + and - just make sense here, obsessed 🫶

⚠️ heads up bestie: changing the value of some DebugFlags will have no effect unless you restart the slurmctld daemon — a restart is what makes it pick up DebugFlags from the slurm.conf config file or from the SLURM_DEBUG_FLAGS environment variable. And btw, the environment variable takes precedence over the setting in slurm.conf. iykyk 💅

nodes=<NodeName> — this one's optional, and if you use it the request to change the debug level gets sent to slurmd instead of slurmctld. You can use a node range expression for NodeName. 🎀

🌈 show `<ENTITY>[=<ID>]` or `<ENTITY> [<ID>]`

okay so this one is the moment — it displays the state of whatever entity you name, with the ID you give it. literally the main character of the whole command. let's go one by one ✨

🦋 `aliases`

Returns all NodeName values associated with a given NodeHostname — super useful for getting the list of virtual nodes associated with a real node in a configuration where multiple slurmd daemons execute on a single compute node. lowkey iconic for that use case tbh.

💜 `assoc_mgr`

Displays the current contents of the slurmctld's internal cache for users, associations and/or qos. you can filter the output by different record types:

users=<user1>[...,<userN>] — limits the User Records displayed to only those with the specified user name(s).

✨ NOTE: if PrivateData=users is set in slurm.conf, unprivileged users will be limited to their own records. Privileged users will be able to see all records.

accounts=<acct1>[...,<acctN>] — limits the Association Records displayed to those with the specified account name(s).

✨ NOTE: if PrivateData=usage is set in slurm.conf, unprivileged users will be limited to accounts where they are a member or coordinator. Privileged users will be able to see all records.

qos=<qos1>[...,<qosN>] — limits the QOS Records displayed to those with the specified QOS name(s).
flags={users|assoc|qos} — says which record type you actually want displayed. If you don't pass any flags, all record types get displayed. all of them. the whole buffet 🍓

🧚 `bbstat`

Displays output from the current burst buffer plugin's status tool (slurm_bb_get_status for lua, or dwstat for datawarp). Whatever options you put after bbstat get passed straight to the status tool by the slurmctld daemon, and the response comes right back to you. Equivalent to dwstat. 🫶

🌸 `burstbuffer`

Displays the current status of the BurstBuffer plugin. short and sweet, we love her.

⭐ `config`

Displays parameter names from the configuration files in mixed case (e.g. SlurmdPort=7003), while derived parameter names are shown in upper case only (e.g. SLURM_VERSION). yes the casing is Doing Something, it's not random 💅

💖 `daemons`

Reports which daemons should be running on this node.

🎀 `dwstat`

Displays output from the current burst buffer plugin's status tool (slurm_bb_get_status for lua, or dwstat for datawarp). Options following dwstat are passed directly to the status tool by the slurmctld daemon, and the response is returned to you. Equivalent to bbstat. (yes they're twins, no I will not be picking a favourite 🙈)

🌈 `federation`

Lists the federation name that the controller is part of, plus the sibling clusters that are part of the federation. found family energy fr fr.

🦋 `hostlist`

Takes a list of host names and prints the hostlist expression for them — it's the inverse of hostnames. hostlist can also take the absolute pathname of a file (must begin with the character /) containing a list of hostnames, or a single - to have it read from stdin directly. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]").

By default hostlist does not sort the node list or make it unique — so like tux2,tux1,tux2 = tux[2,1-2]. If you wanted a sorted list, use hostlistsorted instead — so tux2,tux1,tux2 = tux[1-2,2]. 📝

💜 `hostlistsorted`

Takes a list of host names and prints a sorted (but not unique) hostlist expression for them. See hostlist above. she's the organised sister.

🧚 `hostnames`

Takes an optional hostlist expression as input and writes a list of individual host names to standard output, one per line. If no hostlist expression is supplied, the contents of the SLURM_JOB_NODELIST environment variable get used instead.

For example, "tux[1-3]" is mapped to "tux1", "tux2" and "tux3" — one hostname per line. ✨

🌸 `job`

Displays statistics about all jobs by default. If you pass an optional jobid (numeric or SLUID), you'll get details for just that job.

Heads up: if the job doesn't specify socket-per-node, cores-per-socket or threads-per-core, then it'll display '*' in the ReqS:C:T=*:*:* field. the asterisk is literally saying "idk bestie" 😭

🎀 `licenses`

Displays statistics about all configured licenses (local and remote) by default. If you pass an optional license name, you'll get details for just that license.

⭐ `node`

Displays statistics about all nodes by default. If you pass an optional nodename, you'll get details for just that node.

💖 `partition`

Displays statistics about all partitions by default. If you pass an optional partition name, you'll get details for just that partition.

🌈 `reservation`

Displays statistics about all reservations by default. If you pass an optional reservation name, you'll get details for just that reservation.

🦋 `slurmd`

Displays statistics for the slurmd running on the current node. hi bestie, how are you doing, we're checking on you specifically 👑

💜 `step`

Displays statistics about all job steps by default.

If you pass an optional jobid (numeric or SLUID), you get details about the steps for just that job.
If you pass a jobid.stepid, you get details for just that step.
If you pass a container-id=<id>, then the first matching step with the given container-id gets displayed. 🔥

🧚 `topoconf`

Displays running configurations of multiple topologies, in the same format as topology.yaml. aesthetic consistency, we stan.

🌸 `topology [topology_name] [unit=NAME] [node=NAME]`

Displays information about the selected topology layout. If you give no arguments at all, the default topology gets returned. 🎀

If a unit is specified, information about that unit will be shown. The unit keyword can be replaced with switch, block, or ring (e.g. block=b2). Words are interchangeable, the vibe is the same.
If one node name is specified, all units connected to that node (and their parent switches) will be shown.
If more than one node name is specified, only units that connect to all the named nodes will be shown. selective queen behaviour 💅

🌙 shutdown `<OPTION>`

Instructs Slurm daemons to save current state and terminate. By default the Slurm controller (slurmctld) forwards the request to all the other daemons (the slurmd daemon on each compute node).

An OPTION of slurmctld or controller means only the slurmctld daemon gets shut down, and the slurmd daemons stay active. choose your fighter 🫶

😴 suspend `<job_list>`

Suspends a running job. The job_list argument is a comma separated list of job IDs. Use the resume command to resume its execution. 🔥

For this operation to actually be effective, user processes must stop on receipt of SIGSTOP and resume upon receipt of SIGCONT. Not all architectures and configurations support job suspension, so don't come for me if your setup says no.

Some very real caveats:

If a suspended job is requeued, it will be placed in a held state. ⚠️ The time a job is suspended will not count against a job's time limit. 🎀 Only a privileged user or an account coordinator can suspend jobs. no cap.

👑 takeover `[<INDEX>]`

Instructs one of Slurm's backup controllers (slurmctld) to take over system control. By default, the first backup controller (INDEX=1) requests control from the primary and waits for its termination. After that, it switches from backup mode to controller mode.

If the primary controller can't be contacted, it directly switches to controller mode — no waiting around, she's decisive. 💅

This can be used to speed up the Slurm controller fail-over mechanism when the primary node is down. It can also be used to minimize disruption if the computer executing the primary Slurm controller is scheduled down.

⚠️ NOTE: the primary controller daemon will shut down to allow the secondary to take control. It will take control back at startup. so it's a temporary glow-up, not a permanent promotion 😭

🔥 top `<job_list>`

Moves the specified job IDs to the top of the queue of jobs belonging to the identical user ID, partition name, account, and QOS. The job_list argument is a comma separated ordered list of job IDs.

The rules, so you know:

Any job not matching all of those fields will not be affected. 🫶
Only jobs submitted to a single partition will be affected.
This operation changes the order of jobs by adjusting job nice values.
The net effect on that user's throughput will be negligible to slightly negative. yeah. read that again. it's not free 💀
This operation is disabled by default for unprivileged users, but may be enabled by including SchedulerParameters=enable_user_top in slurm.conf.

🎟️ token `[lifespan=<lifespan>] [username=<username>]`

Returns an auth token which can be used to support JWT authentication, if AuthAltTypes=auth/jwt has been enabled on the system. 🔐

It supports two optional arguments:

lifespan= — specify the token's lifespan in seconds.
username= — request a token for a different username. this one's only available to SlurmUser/root, sorry bestie.

Lifespan can be set to "infinite" to have a token that will not expire. But — sites are suggested to use the smallest lifespan required and generate tokens more often, instead of using longer lived tokens. short-lived tokens are the moment ⭐

uhold <job_list> — like hold's softer sibling 💅 it places a user-hold on the job even if the uhold command was issued by a privileged user. The whole point? The job owner or a coordinator can release the job themselves, no admin intervention needed. autonomy icon behavior fr fr 🫶
update <SPECIFICATION> — updates job, step, node, partition, or reservation configuration per the spec you hand it. SPECIFICATION is in the same format as the Slurm configuration file and the output of the show command described above. Hot tip: run show on the exact entity you wanna update, then use cut-and-paste tools to enter your updated config values into the update. ⚠️ but bestie, listen — while most configuration values can be changed with this command, not all can be changed this way. In particular, the hardware configuration of a node, or the physical addition/removal of nodes from the cluster, may only be accomplished through editing the Slurm configuration file and executing the reconfigure command (described above). no shortcuts on that one 😭
update <SuspendExc*>[=|+=|-=]<LIST> — updates SuspendExcNodes, SuspendExcParts, or SuspendExcStates. <LIST> is either a NodeList, a list of partitions, or a list of node states, respectively. Use +=/-= to add/remove nodes, partitions, or states to/from the currently configured list. Use = to replace the current list. Heads up: SuspendExcNodes does not support "+="/"-=" when the ":" option is used — however, direct assignment "=" is always supported. Consider running scontrol show config | grep SuspendExc to see the current state of these settings. iykyk 💅
version — displays the version number of the scontrol being executed. that's it, that's the whole thing ✨
wait_job <job_id> — waits until a job and all of its nodes are ready for use, or the job has entered some termination state. Genuinely useful in the Slurm Prolog, or in the batch script itself, if nodes are powered down and restarted automatically as needed. 🧚

NOTE: don't use scontrol wait_job in PrologSlurmctld or in Prolog with PrologFlags=Alloc — that will result in a deadlock. we don't do deadlocks here 💀

NOTE: when using wait_job for an array job, use the SLURM_JOB_ID environment variable to reference the job rather than the SLURM_ARRAY_JOB_ID variable. important distinction, don't mix them up 🙈

write batch_script <job_id> [<optional_filename>] — writes the batch script for a given job_id to a file or to stdout. The file defaults to slurm-<job_id>.sh if you don't pass the optional filename argument. Pass - instead of a filename and the script goes to stdout instead. 🔒 the batch script can only be retrieved by a privileged user, or by the owner of the job.
write config <optional_filename> — writes the current configuration to a file with the naming convention "slurm.conf.<datetime>", in the same directory as the original slurm.conf file. If a filename is given, that file location gets created with a .<datetime> suffix. 🎀

🦋 INTERACTIVE COMMANDS 🦋

NOTE: all commands listed below can be used in interactive mode, but NOT on the initial command line. 🙈

all — shows all partitions, their jobs and job steps. This causes information to be displayed about partitions configured as hidden and partitions that are unavailable to the user's group. no gatekeeping today 💅
cluster <CLUSTER_NAME> — the cluster to issue commands to. Only one cluster name may be specified.
details — causes the show command to provide additional details where available. Job information will include CPUs and NUMA memory allocated on each node. Note that on computers with hyperthreading enabled and Slurm configured to allocate cores, each listed CPU represents one physical core. Each hyperthread on that core can be allocated a separate task, so a job's CPU count and task count may differ. See the --cpu-bind and --mem-bind option descriptions in the srun man pages for more info. The details option is currently only supported for the show job command.
exit — terminate the scontrol interactive session. bye bestie 🦋
hide — do not display partition, job or job step information for partitions configured as hidden or partitions unavailable to the user's group. This is the default behavior (so it's already doing this, she's lowkey always on) 🎀
oneliner — print information one line per record.
quiet — print no warning or informational messages, only fatal error messages. silence, but make it fatal 🤫
quit — terminate the execution of scontrol.
verbose — print detailed event logging. This includes time-stamps on data structures, record counts, etc. we love a data-rich queen 👑
!! — repeat the last command executed. yes, run it back 💅

🌸 JOBS - SPECIFICATIONS FOR UPDATE COMMAND 🌸

An unprivileged user may modify their own jobs subject to certain restrictions, as well as jobs and steps in any accounts where they have Coordinator permissions. Most of these fields can only be updated on pending jobs. Some may be updated on running or suspended jobs. A privileged user can modify any jobs without policy restrictions. For instance, if an administrator changes the QOS on a pending job, certain limits such as the TimeLimit will not be changed automatically, because changes made by privileged users are allowed to violate these restrictions. 💀

JobName=<name> — identify the name of jobs to be modified, or set the job's name to the specified value. When used to identify jobs to be modified, all jobs belonging to all users are modified unless the UserID option is used to identify a specific user. Either JobId or JobName is required.
Account=<account> — account name to be changed for this job's resource use. Value may be cleared with a blank data value, "Account=".
AdminComment=<spec> — arbitrary descriptive string. Can only be set by a Slurm administrator. 🔒
ArrayTaskThrottle=<count> — specify the maximum number of tasks in a job array that can execute at the same time. Set the count to zero to eliminate any limit. The task throttle count for a job array is reported as part of its ArrayTaskId field, preceded with a percent sign. Example: "ArrayTaskId=1-10%2" indicates the maximum number of running tasks is limited to 2. math slay 🍓
BurstBuffer=<spec> — burst buffer specification to be changed for this job's resource use. Value may be cleared with a blank data value, "BurstBuffer=". Format is burst buffer plugin specific. Only a privileged user can change this parameter.
Clusters=<spec> — specifies the clusters that the federated job can run on.
ClusterFeatures=<spec> — specifies features that a federated cluster must have to have a sibling job submitted to it. Slurm will attempt to submit a sibling job to a cluster if it has at least one of the specified features.
Comment=<spec> — arbitrary descriptive string. say what you need to say 💬
Contiguous={yes|no} — set the job's requirement for contiguous (consecutive) nodes to be allocated. Possible values are "YES" and "NO". Only a privileged user can change this parameter to "YES".

NOTE: whether this requirement is honored depends on the topology plugin. See the --contiguous option in sbatch(1).

CoreSpec=<count> — number of cores to reserve per node for system use. The job will be charged for these cores but be unable to use them. Will be reported as "*" if not constrained. Only a privileged user can change this parameter. paying rent for a room you can't enter 😭
CPUsPerTask=<count> — change the CPUsPerTask job's value.
Deadline=<time_spec> — remove the job if no ending is possible before this deadline (start > (deadline - time[-min])). Note that if neither DefaultTime nor MaxTime are configured on the partition the job is in, the job will need to specify some form of time limit (TimeMin/TimeLimit) if a deadline is to be used. The specified deadline must be later than the current time. Only a privileged user can change this parameter.

Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[+count[seconds(default)|minutes|hours|days|weeks]]
midnight, elevenses (11 AM), noon, fika (3 PM), teatime (4 PM),
or tomorrow

One or more time strings may be specified (e.g., 'tomorrow18:00'). If there is a conflict between them, the last one will silently take precedence. sneaky but that's the rule 🙈

DelayBoot=<time_spec> — change the time to decide whether to reboot nodes in order to satisfy the job's feature specification if the job has been eligible to run for less than this time period. See the salloc/sbatch man pages option --delay-boot. 🧚
Dependency=<dependency_list> — defer the job's initiation until the specified job dependency specification is satisfied. Once a dependency is satisfied, it is removed from the job. Cancel a dependency with an empty dependency_list (e.g. "Dependency="). <dependency_list> is of the form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs can share the same dependency, and these jobs may even belong to different users. sharing is caring 🫶
- after:job_id[:jobid...] — this job can begin execution after the specified jobs have begun execution or been canceled.
- afterany:job_id[:jobid...] — this job can begin execution after the specified jobs have terminated (any outcome, we don't judge).
- afternotok:job_id[:jobid...] — this job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc). This dependency must be added while the specified job is still active or within MinJobAge seconds after the specified job has ended.
- afterok:job_id[:jobid...] — this job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero). This dependency must be added while the specified job is still active or within MinJobAge seconds after the specified job has ended. the good-ending dependency, we love to see it 👑
singleton — a job with this set can only start after every previously launched job with the same job name and the same user has finished. Translation: exactly one job with that name, owned by that user, can be running or suspended at any given moment. One at a time, bestie. 🎀
EligibleTime=<time_spec> — go look at StartTime, it's the same vibe.
EndTime=<time_spec> — sets the job's expected end time, and the job's TimeLimit gets automatically recalculated to match the change. When the job actually ends, this field gets updated with the real end time. Heads up: only a privileged user is allowed to extend a job's EndTime. 👑
ExcNodeList=<nodes> — sets the job's list of excluded nodes. You can name multiple nodes using simple node range expressions (like "lx[10-20]"). Wanna clear it? Blank data value: "ExcNodeList=".
Extra=<spec> — just an arbitrary string, wrapped in single or double quotes if you're using spaces or some special characters. More deets here: https://slurm.schedmd.com/extra_constraints.html 🌈
Features=<features> — sets the job's required node features. The list can have multiple feature names separated by ampersand (AND) and/or vertical bar (OR) operators, like Features="opteron&video" or Features="fast|faster". In that first example, only nodes that have both the feature "opteron" AND the feature "video" get used. And no, there's legit no mechanism to say "I want one node with feature "opteron" and another node with feature "video"" in case no single node has both — it's giving all-or-nothing, sorry babe. 😭 If only one of a set of possible options should be used across all allocated nodes, use the OR operator and wrap the options in square brackets. Example: "Features=[rack1|rack2|rack3|rack4]" — that says all nodes must land on a single rack of the cluster, but any of those four racks is fine. A request can also specify how many nodes it needs with a given feature by appending an asterisk and a count after the feature name — "Features=graphics*4" means at least four allocated nodes must have the feature "graphics". Parentheses are supported too, for features that get ANDed together — "Features=[(knl&a2a&flat)*4&haswell*2]" means the allocation should include 4 nodes with ALL of the features "knl", "a2a", and "flat", plus 2 nodes with the feature "haswell". ⚠️ Constraints with node counts may only be combined with AND operators. Clear it with a blank data value, e.g. "Features=".
Gres=<list> — a comma-delimited list of generic consumable resources requested per node. Each entry is formatted "name[:count[*cpu]]": name is the consumable resource's name, count is how many of them (defaults to 1), and the specified resources get allocated to the job on each node allocated — unless you tack on "*cpu", in which case they're allocated on a per-CPU basis. What's available is configurable in slurm.conf. If the option argument is "help", it prints a list of available generic consumable resources and the command exits. Examples: "Gres=gpus:2*cpu,disk=40G" and "Gres=help". ✨
JobId=<job_list> — identifies the job(s) to update. job_list can be a comma separated list of job IDs. Either JobId or JobName is required (pick one, bestie). If the JobId is equal to the ArrayJobID, the update hits all of the individual jobs in the array. To update one specific individual job in that case, you gotta use the form <ArrayJobID>_<ArrayTaskId>. 🫶
Licenses=<name> — specify licenses (or other resources available on all nodes of the cluster) the way the salloc/sbatch/srun man pages describe. An unprivileged user can modify the licenses of pending jobs. A privileged user can modify the licenses of pending or running jobs. 💅
MailType=<types> — sets the mail event types. Valid values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), TIME_LIMIT_50 (reached 50 percent of time limit), and ARRAY_TASKS (send emails for each array task). Multiple type values go in a comma separated list. Unless ARRAY_TASKS is specified, mail notifications on job BEGIN, END and FAIL apply to the job array as a whole instead of generating a separate email for each task in the array. 📬
MailUser=<name> — sets the user who gets email notification of state changes. A blank string sets the mail user back to the default, which is the submitting user.
MCSLabel=<name> — sets the MCS label for the job, and it only takes effect if an MCS plugin is loaded in slurmctld. 🦋
MinCPUsNode=<count> — sets the job's minimum number of CPUs per node to the specified value.
MinMemoryCPU=<mebibytes> — sets the job's minimum real memory required per allocated CPU to the specified value. Either MinMemoryCPU or MinMemoryNode can be set, but not both.
MinMemoryNode=<mebibytes> — sets the job's minimum real memory required per node to the specified value. For running jobs, only reduction is allowed. If the job was submitted with --mem-per-cpu, the new per-node limit replaces the per-CPU limit (compared against the highest per-node allocation across all nodes). Either MinMemoryCPU or MinMemoryNode can be set, but not both (for pending jobs).
MinTmpDiskNode=<mebibytes> — sets the job's minimum temporary disk space required per node to the specified value.
Name[=<name>] — see JobName.
Nice[=<adjustment>] — updates the job with an adjusted scheduling priority within Slurm. No adjustment value? The scheduling priority decreases by 100. A negative nice value increases the priority, otherwise it decreases it. Adjustment range is +/- 2147483645. Only a privileged user can specify a negative adjustment. 👑
NodeList=<nodes> — changes the nodes allocated to a running job in order to shrink it. The list you give must be a subset of the nodes currently allocated to the job. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). After a job's allocation is reduced, subsequent srun commands must explicitly specify node and task counts that are valid for the new allocation. ⚠️ NOTE: the allocated nodes of jobs with arbitrary distribution can not be updated. Periodt.
NumCPUs=<min_count>[-<max_count>] — sets the job's minimum and, optionally, maximum count of CPUs to be allocated.
NumNodes=<min_count>[-<max_count>] — sets the job's minimum and, optionally, maximum count of nodes to be allocated. If the job is already running, use this to specify a node count less than what's currently allocated, and the resources previously allocated to the job get relinquished. After a job's allocation is reduced, subsequent srun commands must explicitly specify node and task counts that are valid for the new allocation. Also peep the NodeList parameter above. Same thing as ReqNodes. ⚠️ NOTE: the node count of jobs with arbitrary distribution can not be updated.
NumTasks=<count> — sets the job's count of requested tasks to the specified value. Lowkey important: the number of tasks started in a specific step inside the allocation may differ from this value — for instance when a different number of tasks is requested on step creation. This is the same as ReqProcs.
OverSubscribe={yes|no} — sets the job's ability to share compute resources (i.e. individual CPUs) with other jobs. Possible values are "YES" and "NO". This option can only be changed for pending jobs, and only a privileged user can change this parameter. 💅
Partition=<name> — sets the job's partition to the specified value.
Prefer=<features> — sets the job's preferred node features. This list is only preferred, not required the way Features is. It will override whatever is requested in Features. See the Features option above. 💖
Priority=<number> — sets the job's priority to the specified value. Big one: a job priority of zero stops the job from ever being scheduled at all — setting a job's priority to zero effectively holds it. Set the priority to a non-zero value to let it run, which also clears any previously set nice value. When a privileged user explicitly sets a job's priority, it becomes fixed and the priority plugin will no longer modify it. To restore the priority/multifactor plugin's ability to manage a job's priority, hold the job and then release it. An unprivileged user can only decrease job priority, and the priority plugin may still manage the value — so honestly, consider increasing Nice instead. 🙈
QOS=<name> — sets the job's QOS (Quality Of Service) to the specified value, or a comma separated list of QOS. If you request a list, it gets ordered based on the priority of the QOS given, with the first one being the highest priority. Clear it with a blank data value: "QOS=".
Reboot={yes|no} — sets the job's flag for whether to force the allocated nodes to reboot before starting the job. Only supported with some system configurations, so it could be silently ignored (rude, but okay 💀). Only a Slurm administrator can change this parameter.
ReqCores=<count> — changes the job's requested Cores count.
ReqNodeList=<nodes> — sets the job's list of required nodes. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). Clear it with a blank data value: "ReqNodeList=".
ReqNodes=<min_count>[-<max_count>] — see NumNodes. Already covered it, bestie, scroll up. 🧚
ReqProcs=<count> — literally just go look at NumTasks bestie, same energy 💫
ReqSockets=<count> — changes how many sockets the job is asking for 🔌
ReqThreads=<count> — changes the job's requested thread count 🧵
Requeue={0|1} — decides whether the job gets requeued after a node goes down: 0 = nope, 1 = yep 🔁
ReservationName=<name> — sets the job's reservation to whatever you pass in. You can wipe it back to blank with an empty value, like "ReservationName=" 🎀
ResetAccrueTime — sets the job's accrue time to now, meaning it loses all the priority time it had been stacking up. Super handy when you've got a huge queue of jobs already sitting there and you wanna start limiting how many jobs can accrue time without waiting for the whole queue to flush out 🫶
SiteFactor=<account> — sets the job's site priority factor, and it has to land in the range of +/-2147483645. Only a privileged user can touch this one 👑
StdErr=<filepath> — sets the batch job's stderr file path. You can reset it to the job default with a blank value, "StdErr=" 📝

NOTE: by default, StdErr gets merged into StdOut.

StdIn=<filepath> — sets the batch job's stdin file path. Blank value, "StdIn=", resets it to the job default 📥

NOTE: by default, StdIn will be /dev/null.

StdOut=<filepath> — sets the batch job's stdout file path. Blank value, "StdOut=", resets it to the job default 📤

NOTE: by default, StdOut will be based on the JobId.

StartTime=<time_spec> — sets the job's earliest start time, and the formats are honestly kind of a buffet 🍓:
- HH:MM:SS runs the job at a specific time of day — seconds are optional. If that time already passed today, the next day is assumed.
- You can also just say midnight, noon, elevenses (11 AM), fika (3 PM) or teatime (4 PM) — yes, for real, iconic behavior ✨
- You can slap AM or PM on a time-of-day to run it in the morning or the evening.
- Want to pick the day? Give a date as MMDDYY, MM/DD/YY or MM.DD.YY, or date-plus-time as YYYY-MM-DD[THH:MM[:SS]].
- You can also do now + count time-units, where the time-units are seconds (the default), minutes, hours, days, or weeks.
- The keywords today and tomorrow are also allowed 💜

Notes on date/time specifications:

even though the seconds field of the HH:MM:SS spec is allowed by the code, the Slurm scheduler's poll time is not precise enough to guarantee your job dispatches on the exact second 😭. The job just becomes eligible on the next poll after the specified time. The exact poll interval depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin).
if you don't give a time (HH:MM:SS), the default is 00:00:00.
if you give a date with no year (like MM/DD), the current year is assumed — unless MM/DD plus HH:MM:SS has already passed this year, in which case it rolls to next year.

Switches=<count>[@<max-time-to-wait>] — when a tree topology is in play, this sets the max count of switches you want for the job's allocation. If Slurm finds an allocation with more switches than you specified, the job just sits there pending until it finds one with the count you want or the time limit expires ⏳. By default there's no switch count limit and no time limit delay. Set the count to zero to clear any previously set count (which disables the limit). The job's max time delay can be capped by setting SchedulerParameters=max_switch_wait=# in slurm.conf. Also peep wait-for-switch below 👀
TasksPerNode=<count> — changes the job's requested TasksPerNode. Only a privileged user can change this parameter 🔐
ThreadSpec=<count> — how many threads to reserve per node for system use. The job gets charged for these threads but can't actually use them 💀. Gets reported as "*" if it's not constrained.
TimeLimit=<time> — the job's time limit. Output format is [days-]hours:minutes:seconds or "UNLIMITED". Input format (for the update command) can be minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, or days-hours:minutes:seconds. Time resolution is one minute, and second values get rounded up to the next minute ⌛

If you're changing a job's time limit, either give a whole new value, or put a + or - before the time and equal sign to increment or decrement the current limit (like "TimeLimit+=30"). To increment or decrement, the JobId specification must come before the TimeLimit specification. Heads up: incrementing or decrementing the time limit for a job array is only allowed before the job array has been split into more than one job record. And only a privileged user can increase a running or suspended job's TimeLimit 👑

TimeMin=<timespec> — changes the TimeMin value, which specifies the minimum time limit minutes of the job ⏱️
UserID=<UID or name> — used together with the JobName option to identify which jobs get modified. You can give a user name or a numeric ID (UID) 💖
wait-for-switch=<seconds> — changes the max time to wait for a switch, in <seconds> secs 🕰️
WCKey=<key> — sets the job's workload characterization key to whatever you give it 🔑
WorkDir=<directory_name> — sets the job's working directory to the specified value. Note that this can only be set for jobs in the PENDING state, and jobs may fail to launch if they rely on relative paths to the originally submitted WorkDir 🙈

🦋 JOBS - SPECIFICATIONS FOR SHOW COMMAND 🦋

So the "show" command, when you use it with the "job" or "job <jobid>" entity, displays detailed info about a job or jobs. A lot of that info can be modified with the "update job" command we just went through. But the following fields that show job displays are read-only and cannot be modified, periodt 💅:

AllocNode:Sid — the local node and system id that made the resource allocation 🏠
BatchFlag — jobs submitted with the sbatch command have BatchFlag set to 1. The BatchFlag gets incremented past 1 if the job is requeued due to a failure. Jobs submitted with other commands have BatchFlag set to 0 and won't be incremented 🔢
ExitCode=<exit>:<sig> — the exit status the wait() function reported for the job. The first number is the exit code, typically as set by the exit() function. The second number is the signal that killed the process, if it was terminated by a signal 💀
GroupId — the group the job was submitted under 👥
JobState — the current state of the job 📊
NodeListIndices — the NodeIndices expose the internal indices into the node table associated with the node(s) allocated to the job 🔎
NtasksPerN:B:S:C=<tasks_per_node>:<tasks_per_baseboard>:<tasks_per_socket>:<tasks_per_core> — specifies the number of tasks to start per hardware component (node, baseboard, socket and core). Unconstrained values may show as "0" or "*" 🧩
PreemptEligibleTime — the time the job becomes eligible for preemption. Modified by PreemptExemptTime, either from the global option in slurm.conf or the job QOS. This is hidden if the job hasn't started or if PreemptMode=OFF ⚡
PreemptTime — the time at which the job was signaled that it was selected for preemption. This value is only meaningful for PreemptMode=CANCEL and PreemptMode=REQUEUE, and for jobs in a partition or QOS that has a GraceTime value designated. Hidden if the job hasn't started or if PreemptMode=OFF 🚨
PreSusTime — the time the job ran prior to the last suspend ⏸️
Reason — the reason a job hasn't been started by the scheduler, e.g. waiting for "Resources". Details of job reason codes live here: https://slurm.schedmd.com/job_reason_codes.html 📋
ReqB:S:C:T=<baseboard_count>:<socket_per_baseboard_count>:<core_per_socket_count>:<thread_per_core_count> — specifies the count of various hardware components requested by the job. Unconstrained values may show as "0" or "*". The job submission flags --extra-node-info, --cores-per-socket, and --threads-per-core set the values of "C" and "T" 🛠️
SecsPreSuspend=<seconds> — if the job is suspended, this is the run time the job accumulated (in seconds) before being suspended ⏲️
SegmentSize=<size> — the requested segment size for the job 📦
Socks/Node=<count> — count of desired sockets per node 🔌
SubmitTime — the time and date stamp (in localtime) the job was submitted. The format of this output is identical to the EndTime field 🗓️

NOTE: if a job is requeued, the submit time is reset. To get the original submit time you have to use the "sacct -j <job_id[.<step_id>]" command, also designating the -D or --duplicate option so it displays all duplicate entries for a job.

SuspendTime — the time the job was last suspended or resumed ⏯️

NOTE on the info displayed for various job states: when you submit a request for the "show job" function, the scontrol process makes an RPC request call to slurmctld with a REQUEST_JOB_INFO message type. If the job's state is PENDING, it returns some detail info like min_nodes, min_procs, cpus_per_task, etc. If the state is anything other than PENDING, the code assumes it's in a further state like RUNNING, COMPLETE, etc. In those cases the code explicitly returns zero for those values. Those values are meaningless once the job's resources have been allocated and the job has started 🫠

✨ STEPS - SPECIFICATIONS FOR UPDATE COMMAND ✨

okay so if you're updating a step instead of a whole job, these are the knobs you get 💅 literally only two of them, we love a short list:

`StepId=<job_id>[.<step_id>]`

This is the one that tells slurm which step you're touching. Bestie, it's required — no cap, you cannot skip it. And here's the cute part: if you give the job_id but no step_id, then every step of that job gets modified. all of them. the whole squad 🫶

`TimeLimit=<time>`

The job's time limit. Output format is [days-]hours:minutes:seconds or "UNLIMITED". Input format (for the update command) set is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes or days-hours:minutes:seconds. Time resolution is one minute and second values are rounded up to the next minute. ⏰

Now the fun bit: if you're changing the time limit of a step, you either specify a brand new time limit value, or you precede the time with a "+" or "-" to increment or decrement the current time limit (e.g. "TimeLimit=+30"). it's giving math, it's giving flexible ✨

⚠️ okay but lowkey important: in order to increment or decrement the current time limit, the StepId specification must precede the TimeLimit specification. order matters, bestie. do not test me on this.

🦋 NODES - SPECIFICATIONS FOR CREATE COMMAND 🦋

Provide the same NodeName configuration as found in the slurm.conf. See slurm.conf man page for details. 👑

Only State=CLOUD, State=EXTERNAL and State=FUTURE nodes are allowed. that's the vibe check and those are the only three that pass it 💀

Creation is all-or-nothing. If any node in the request cannot be created, the command fails, no node from the request is created, and every node that already existed — whether created dynamically or defined in slurm.conf — is left unchanged. it's giving "we ride together or nobody rides at all" 🎀

🌈 NODES - SPECIFICATIONS FOR UPDATE COMMAND 🌈

This is the big one, the main character section, buckle up 🎢

`NodeName=<name>`

Identify the node(s) to be updated. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). Nodesets can also be specified by themselves or mixed with node range expressions, using a comma as a list separator. If the keyword "ALL" is specified alone, then the update will be attempted against all the nodes in the local cluster. 🌍

This specification is required — yeah, you gotta say who, obviously.

`ActiveFeatures=<features>`

Identify the feature(s) currently active on the specified node. Any previously active feature specification will be overwritten with the new value. Also see AvailableFeatures. 👀

Typically ActiveFeatures will be identical to AvailableFeatures; however ActiveFeatures may be configured as a subset of the AvailableFeatures. For example, a node may be booted in multiple configurations. In that case, all possible configurations may be identified as AvailableFeatures, while ActiveFeatures would identify the current node configuration. it's giving "everything you could be" vs "who you are today" 💖

When updating the ActiveFeatures with scontrol, the change is only made in slurmctld. When using a node_features plugin the state/features of the node must be updated on the node such that a new node start will report the updated state/features. ✨ (yes, that's on you to handle, bestie.)

`AvailableFeatures=<features>`

Identify the feature(s) available on the specified node. Any previously defined available feature specification will be overwritten with the new value. 📝

AvailableFeatures assigned via scontrol will only persist across the restart of the slurmctld daemon with the -R option and state files preserved or slurmctld's receipt of a SIGHUP. Update slurm.conf with any changes meant to be persistent across normal restarts of slurmctld or the execution of scontrol reconfig. 🔁

⚠️ NOTE: Available features being removed via scontrol must not be active (i.e. remove them from ActiveFeatures first). do it in the right order, please. the drama is not worth it 😭

`CertToken=<token>`

Unique token string used by certmgr plugin interface to validate node identity. 🔐 very identity-verification-core of you.

`Comment=<comment>`

Arbitrary descriptive string. Use quotes to enclose a comment having more than one word. 🎀 ("Comment=this node is a diva" — iconic)

`CpuBind=<node>`

Specify the task binding mode to be used by default for this node. Supported options include: "none", "socket", "ldom" (NUMA), "core", "thread" and "off" (remove previous binding mode). 🧵 pick your fighter.

`Extra=<comment>`

Arbitrary string on the node. Use quotes to enclose a string having more than one word. See https://slurm.schedmd.com/extra_constraints.html for more details. 📎

`Gres=<gres>`

Identify generic resources to be associated with the specified node. Any previously defined generic resources will be overwritten with the new value. Specifications for multiple generic resources should be comma separated. Each resource specification consists of a name followed by an optional colon with a numeric value (default value is one) (e.g. "Gres=bandwidth:10000"). 🍓

Modification of GRES count associated with specific files (e.g. GPUs) is not allowed other than to set their count on a node to zero. In order to change the GRES count to another value, modify your slurm.conf and gres.conf files and restart daemons. 🛑

If GRES are associated with specific sockets, that information will be reported. For example if all 4 GPUs on a node are all associated with socket zero, then "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then "Gres=gpu:4(S:0-1)". 🔥

Generic resources assigned via scontrol will only persist across the restart of the slurmctld daemon with the -R option and state files preserved or slurmctld's receipt of a SIGHUP. Update slurm.conf with any changes meant to be persistent across normal restarts of slurmctld or the execution of scontrol reconfig. 🔁 (yes bestie it's the same caveat again, we keep it real)

`InstanceId=<instance_id>`

Cloud instance ID. Multiple IDs may be specified using the same syntax as a node range expression (used in NodeName). If multiple IDs are specified, the total number must match the number of nodes being updated. ☁️✨

`InstanceType=<instance_type>`

Cloud instance type. Multiple types may be specified using the same syntax as a node range expression (used in NodeName). If multiple types are specified, the total number must match the number of nodes being updated. ☁️🦋 same energy, different field.

`NodeAddr=<node address>`

Name that a node should be referred to in establishing a communications path. This name will be used as an argument to the getaddrinfo() function for identification. If a node range expression is used to designate multiple nodes, they must exactly match the entries in the NodeName (e.g. "NodeName=lx[0-7] NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses. 🌐

`NodeHostname=<node hostname>`

Typically this would be the string that "/bin/hostname -s" returns. It may also be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings. 🏠

Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string). A node range expression can be used to specify a set of nodes. If an expression is used, the number of nodes identified by NodeHostname must be identical to the number of nodes identified by NodeName. 🎯 (count them. fr fr.)

`Reason=<reason>`

Identify the reason the node is in a "DOWN", "DRAINED", "DRAINING", "FAILING" or "FAIL" state. Use quotes to enclose a reason having more than one word. 💬 spill the tea, but in quotes.

`ResumeAfter=<seconds>`

Schedule a node state resume after this amount of seconds, when its state is updated to "DOWN" or "DRAIN". Upon state resume, the node's state will be changed from DRAIN, DRAINING, DOWN or REBOOT to IDLE and NoResp. slurmctld will then attempt to contact slurmd to request that the node register itself. 📞 the "hey bestie u up" arc.

⚠️ NOTE: A value of -1 will unschedule the node state resume. (the anti-reminder 🤡)

⚠️ NOTE: The same POWERED_DOWN and POWERING_DOWN behavior described under RESUME (below) applies. go read RESUME, i'll wait 💅

`State=<state>`

Assign one of the following states/actions to the node(s) specified by the update command. 🎛️ this is the dropdown menu of destiny:

CANCEL_REBOOT — Cancels a pending reboot on the node (same as scontrol cancel_reboot <node>). 🙅‍♀️ not today.
DOWN — Stop all running and suspended jobs and make the node unavailable for new jobs. 💀 brutal but honest.
DRAIN — Indicates that no new jobs may be started on this node. Existing jobs are allowed to run to completion, leaving the node in a DRAINED state once all the jobs have completed. 🚪 it's the "no new friends" setting.
FAIL — Similar to DRAIN except that some applications will seek to relinquish those nodes before the job completes. 😬 messier sibling of DRAIN.
FUTURE — Indicates the node is not fully configured, but is expected to be available at some point in the future. 🌱 still becoming, we respect the journey.
IDLE — Will clear DOWN, DRAIN and FAIL states and set the base state to IDLE. Unlike RESUME, this does not set NoResp and does not request that the node register itself. A node that already has the NoResp flag set does not become eligible to be allocated until it registers, unless it is POWERED_DOWN. POWERING_DOWN is not an exception here; such a node stays ineligible until it finishes powering down. 😴 (it's not a wake-up call, it's just a new base state, ok?)
NoResp — This will set the "Not Responding" flag for a node without changing its underlying state. 📵

⚠️ NOTE: On a node that is POWERED_DOWN or POWERING_DOWN this flag is cleared when slurmctld next reads its saved state on a reconfigure or a restart. Such a node cannot register while it is powered down, so a flag that persisted there would keep it out of the available nodes for good. 🫠 yeah. that would be bad. that's why we don't do that.

RESUME — Not an actual node state, but will change a node state from DRAIN, DRAINING, DOWN or REBOOT to IDLE and NoResp. slurmctld will then attempt to contact slurmd to request that the node register itself. Once registered, the node state will then remove the NoResp flag and will resume normal operations. It will also clear the POWERING_DOWN state of a node and make it eligible to be allocated. 🫶 it's giving emotional support action item. slay.

NOTE: This does not set NoResp on a node that is POWERED_DOWN or POWERING_DOWN. Such a node is neither pinged nor able to register while it is powered down, so the flag would persist and keep the node out of the available nodes. The node is still required to register when it powers up.

UNDRAIN — Clears the node from being drained (like RESUME), but will not change the node's base state (e.g. DOWN). For a node that is not POWERED_DOWN or POWERING_DOWN, UNDRAIN requires a valid node registration before new jobs can be scheduled on the node. Setting a node DOWN will cause all running and suspended jobs on that node to be terminated.

NOTE: As with RESUME, this does not set NoResp on a node that is POWERED_DOWN or POWERING_DOWN. A POWERED_DOWN node whose base state is IDLE is then eligible to be allocated; the scheduler will power it up and registration will follow. A POWERING_DOWN node is not eligible until it finishes powering down.

Okay so while all of the above states are valid, some of them are not valid new node states given their prior state. iykyk 💅

NOTE: The scontrol command should not be used to change node state on Cray systems. Use Cray tools such as xtprocadmin instead.

Topology=<topology_name>:<topology_unit>[,<topology_name>:<topology_unit>,...] — Where <topology_unit> is the name of the block, leaf switch, or ring. A ring name must be followed by its numeric position in the ring (colon-delimited, e.g., Topology=topo-ring:ring1:2). The ring position must be 0 when creating a new ring. A switch name may be followed by one or more intermediate switch names, which will be created if needed (colon-delimited, e.g., Topology=topo-tree:sw_root:s1:s2). This configuration overrides node topology affiliation settings from topology.conf and topology.yaml.
Weight=<weight> — Identify weight to be associated with specified nodes. This allows dynamic changes to weight associated with nodes, which will be used for the subsequent node allocation decisions. Weight assigned via scontrol will only persist across the restart of the slurmctld daemon with the -R option and state files preserved or slurmctld's receipt of a SIGHUP. Update slurm.conf with any changes meant to be persistent across normal restarts of slurmctld or the execution of scontrol reconfig.

🦋 NODES - SPECIFICATIONS FOR SHOW COMMAND 🦋

These are the fields you get to peek at when you show a node, bestie. Vibe check time ✨

AllocMem — The total memory, in MiB, currently allocated by jobs on the node.
CPULoad — CPU load of a node as reported by the OS.
CPUSpecList — The list of Slurm abstract CPU IDs on this node reserved for exclusive use by the Slurm compute node daemons (slurmd, slurmstepd). These are indices in Slurm's per-node CPU numbering, which may not match the CPU IDs used by the operating system.
FreeMem — The total memory, in MiB, currently free on the node as reported by the OS.
GresUsed — The generic resources on the node that are currently allocated to jobs. Only displayed with the -d, --details option. For shared GRES (shard or mps) usage syntax, see https://slurm.schedmd.com/gres.html#Shared_GRES_Usage.
LastBusyTime — The last time the node was busy (i.e. last time the node had jobs on it). This time is used in PowerSave to determine when to suspend nodes (e.g. now - LastBusy > SuspendTime).
MemSpecLimit — The combined memory limit, in mebibytes, on this node for the Slurm compute node daemons (slurmd, slurmstepd).
RealMemory — The total memory, in MiB, on the node.
State — Identify the state(s) assigned to the node with '+' delimited state flags.

States:

ALLOCATED — Indicates that the node has all CPUs allocated to job(s) running on the node.
DOWN — The node does not have any running jobs and is unavailable for new work.
ERROR — The node is in an error state. Consult the logs for more information about what caused this state.
FUTURE — The node is currently not fully configured, but expected to be available at some point in the indefinite future for use.
IDLE — Indicates that the node is available for work but does not currently have any jobs assigned to it.
MIXED — Indicates that the node is in multiple states. For instance if only part of the node is ALLOCATED and the rest of the node is IDLE the state will be MIXED.
UNKNOWN — The node has not yet registered with the controller and its state is not known.

Flags:

CLOUD — Indicates that the node is configured as a cloud node, to be brought up on demand, but not currently running.
COMPLETING — Indicates that the only job on the node or that all jobs on the node are in the process of completing.
DRAIN — The node is not accepting any new jobs and any currently running jobs will complete.
DYNAMIC — Slurm allows you to define multiple types of nodes in a FUTURE state. When starting slurmd on a node you can specify the -F flag to have the node match and use an existing definition in your slurm.conf file. The DYNAMIC state indicates that the node was started as a Dynamic Future node.
INVALID_REG — The node did not register correctly with the controller. This happens when a node registers with less resources than configured in the slurm.conf file. The node will clear from this state with a valid registration (i.e. a slurmd restart is required).
MAINTENANCE — The node is currently in a reservation that includes the maintenance flag.
NOT_RESPONDING — Node is not responding.
POWER_DOWN — Node is pending power down.
POWERED_DOWN — Node is currently powered down and not capable of running any jobs.
POWERING_DOWN — Node is in the process of powering down.
POWERING_UP — Node is in the process of powering up.
PLANNED — The node is earmarked for a job that will start in the future.
REBOOT_ISSUED — A reboot request has been sent to the agent configured to handle this request.
REBOOT_REQUESTED — A request to reboot this node has been made, but hasn't been handled yet.
RESERVED — Indicates the node is in an advanced reservation and not generally available.

The meaning of the energy information is as follows:

CurrentWatts — The instantaneous power consumption of the node at the time of the last node energy accounting sample, in watts.
LowestJoules — The energy consumed by the node between the last time it was powered on and the last time it was registered by slurmd, in joules.
ConsumedJoules — The energy consumed by the node between the last time it was registered by the slurmd daemon and the last node energy accounting sample, in joules.

If the reported value is "n/s" (not supported), the node does not support the configured AcctGatherEnergyType plugin. If the reported value is zero, energy accounting for nodes is disabled.

🌈 PARTITIONS - SPECIFICATIONS FOR CREATE AND UPDATE COMMANDS 🌈

PartitionName=<name> — Identify the partition to be updated. This specification is required. No cap, you literally cannot skip this one 🫶
AllocNodes=<name> — Comma separated list of nodes from which users can execute jobs in the partition. Node names may be specified using the node range expression syntax described above. The default value is "ALL".
AllowAccounts=<name> — Comma-separated list of accounts which may execute jobs in the partition. The default value is "ALL". This list is hierarchical, meaning subaccounts are included automatically. NOTE: If AllowAccounts is used then DenyAccounts will not be enforced. Also refer to DenyAccounts.
AllowGroups=<name> — Identify the user groups which may use this partition. Multiple groups may be specified in a comma separated list. To permit all groups to use the partition specify "AllowGroups=ALL".
AllowQOS=<name> — Identify the QOSs which may use this partition. Multiple QOSs may be specified in a comma separated list. To permit all QOSs to use the partition specify "AllowQOS=ALL".
Alternate=<partition name> — Alternate partition to be used if the state of this partition is "DRAIN" or "INACTIVE." The value "NONE" will clear a previously set alternate partition.
CpuBind=<node> — Specify the task binding mode to be used by default for this partition. Supported options include: "none", "socket", "ldom" (NUMA), "core", "thread" and "off" (remove previous binding mode).
Default={yes|no} — Specify if this partition is to be used by jobs which do not explicitly identify a partition to use. Possible output values are "YES" and "NO". In order to change the default partition of a running system, use the scontrol update command and set Default=yes for the partition that you want to become the new default.
DefaultTime=<time> — Run time limit used for jobs that don't specify a value. If not set then MaxTime will be used. Format is the same as for MaxTime.
DefMemPerCPU=<MB> — Set the default memory to be allocated per CPU for jobs in this partition. The memory size is specified in mebibytes.
DefMemPerNode=<MB> — Set the default memory to be allocated per node for jobs in this partition. The memory size is specified in mebibytes.
DenyAccounts=<name> — a comma-separated list of accounts that are not allowed to execute jobs in the partition. By default? nobody's denied, we're all friends here 💖. This list is hierarchical, meaning subaccounts get pulled in automatically, no extra effort required. > ⚠️ NOTE: if AllowAccounts is being used, then DenyAccounts will not be enforced. Also go peek at AllowAccounts.
DenyQOS=<name> — names the QOSs that should be denied access to this partition. You can stack multiple QOSs in one comma-separated list, slay.
DisableRootJobs={yes|no} — says whether jobs can be executed as user root or not. Possible values are "YES" and "NO".
Exclusive={no|none|node|user|topo} — sets partition-level exclusivity to a single mode. Pick your fighter:
- NO (or NONE): no partition-level exclusivity at all. This is the default, very chill.
- NODE: allocate whole nodes (similar to job --exclusive). This implies Oversubscribe=NO.
- USER: only one user's jobs may share a node (similar to --exclusive=user).
- TOPO: only one job per topology segment (similar to --exclusive=topo); also implies NODE. This implies Oversubscribe=NO too.

See the Exclusive entry in slurm.conf(5) for the full semantics, iykyk. The older ExclusiveUser=YES and ExclusiveTopo=YES are still accepted and are equivalent to Exclusive=USER and Exclusive=TOPO respectively. ExclusiveUser=YES and ExclusiveTopo=YES are mutually exclusive — they cannot both win. If both deprecated options are set for the same partition, ExclusiveUser will be ignored. 🙈

GraceTime=<seconds> — the preemption grace time, in seconds, extended to a job that's been selected for preemption. Default value is zero, meaning no preemption grace time is allowed on this partition or qos. (Meaningful only for PreemptMode=CANCEL and PreemptMode=REQUEUE.)
Hidden={yes|no} — specifies if the partition and its jobs should be hidden from view. Hidden partitions will by default not be reported by Slurm APIs or commands. Possible values are "YES" and "NO".
JobDefaults=<specs> — sets job default values using a comma-delimited list of "key=value" pairs. The supported keys include:
- DefCpuPerGPU — default number of CPUs per allocated GPU.
- DefMemPerGPU — default memory limit (in mebibytes) per allocated GPU.
LLN={yes|no} — schedule jobs on the least loaded nodes (based on the number of idle CPUs). Obsessed with balance ✨
MaxCPUsPerNode=<count> — the maximum number of CPUs that can be allocated per node to all jobs in this partition.
MaxCPUsPerSocket=<count> — the maximum number of CPUs that can be allocated per socket to all jobs in this partition.
MaxMemPerCPU=<MB> — the maximum memory to be allocated per CPU for jobs in this partition. Memory size is given in mebibytes.
MaxMemPerNode=<MB> — the maximum memory to be allocated per node for jobs in this partition. Memory size is given in mebibytes too.
MaxNodes=<count> — the maximum number of nodes that will be allocated to any single job in the partition. Give it a number, or "INFINITE", or "UNLIMITED". Changing the MaxNodes of a partition has no effect upon jobs that have already begun execution — those guys are grandfathered in, periodt.
MaxTime=<time> — the maximum run time for jobs. Output format is [days-]hours:minutes:seconds or "UNLIMITED". Input format (for the update command) is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes or days-hours:minutes:seconds. Time resolution is one minute, and second values are rounded up to the next minute. Changing the MaxTime of a partition has no effect upon jobs that have already begun execution.
MinNodes=<count> — the minimum number of nodes that will be allocated to any single job in the partition. Changing the MinNodes of a partition has no effect upon jobs that have already begun execution. Heads up: increasing this value may prevent pending jobs from starting, even if they were submitted without -N/--nodes. If you do get into that situation, updating the MinNodes value of a pending job using the scontrol command will let that job be scheduled. Crisis averted 🫶
Nodes=<name> — identifies the node(s) to be associated with this partition. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). A specification of "ALL" will associate all nodes. You can add or remove nodes by adding a '+' or '-' sign before the '=' sign. It also works to do nodes=+lx0,-lx[2-4] if you want to remove lx[2-4] and add lx0 at the same time. Note that jobs may only be associated with one partition at any time. Specify a blank data value to remove all nodes from a partition: "Nodes=". Changing the Nodes in a partition has no effect upon jobs that have already begun execution.
OverSubscribe={yes|no|force}[:<job_count>] — specifies if compute resources (i.e. individual CPUs) in this partition can be shared by multiple jobs. Possible values are "YES", "NO", and "FORCE". An optional job count says how many jobs can be allocated to use each resource. Whole-node-exclusive partitions can be configured with Exclusive=node (see above).
OverTimeLimit=<count> — the number of minutes by which a job can exceed its time limit before being canceled. The configured job time limit is treated as a soft limit, and adding OverTimeLimit to the soft limit gives you a hard limit, at which point the job gets canceled 💀. This is particularly useful for backfill scheduling, which bases itself on each job's soft time limit. A partition-specific OverTimeLimit will override any global OverTimeLimit value. If not specified, the global OverTimeLimit value takes precedence. May not exceed 65533 minutes. An input value of "UNLIMITED" will clear any previously configured partition-specific OverTimeLimit value.
PowerDownOnIdle — if set to "YES", then nodes allocated from this partition will immediately be requested to power down upon becoming IDLE. A power down request prevents further scheduling to the node until it has been put into power save mode by SuspendProgram. See slurm.conf(5) for more information. Also see SuspendTime.
PreemptMode=<mode> — resets the mechanism used to preempt jobs in this partition if PreemptType is configured to preempt/partition_prio. The default preemption mechanism is specified by the cluster-wide PreemptMode configuration parameter. Possible values are "OFF", "CANCEL", "REQUEUE" and "SUSPEND".
PriorityJobFactor=<count> — the partition factor used by the priority/multifactor plugin when calculating job priority. The value may not exceed 65533. Also see PriorityTier.
PriorityTier=<count> — jobs submitted to a partition with a higher priority tier value will be dispatched before pending jobs in a partition with a lower priority tier value and, if possible, they will preempt running jobs from partitions with lower priority tier values. Note that a partition's priority tier takes precedence over a job's priority — tier eats priority for breakfast 🔥. The value may not exceed 65533. Also see PriorityJobFactor.
QOS=<QOSname|blank to remove> — set the partition QOS with a QOS name, or leave the option blank to remove the Partition QOS. Blank = bye bye 👋
ReqResv={yes|no} — specifies if only allocation requests designating a reservation will be satisfied. Used to restrict partition usage so it's only allowed within a reservation. Possible values are "YES" and "NO".
RootOnly={yes|no} — specifies if only allocation requests initiated by user root will be satisfied. Can be used to restrict control of the partition to some meta-scheduler. Possible values are "YES" and "NO".
State={up|down|drain|inactive} — specifies if jobs can be allocated nodes or queued in this partition. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE". The lore:
- UP — designates that new jobs may be queued on the partition, and that jobs may be allocated nodes and run from the partition. Living the dream 🌈
- DOWN — designates that new jobs may be queued on the partition, but queued jobs may not be allocated nodes and run from the partition. Jobs already running on the partition continue to run. Those jobs must be explicitly canceled to force their termination — no auto-yeet.
- DRAIN — designates that no new jobs may be queued on the partition (job submission requests will be denied with an error message), but jobs already queued on the partition may be allocated nodes and run. See also the "Alternate" partition specification.
- INACTIVE — designates that no new jobs may be queued on the partition, and jobs already queued may not be allocated nodes and run. See also the "Alternate" partition specification.
Topology=<topology name> — the name of the topology, defined in topology.yaml, used by jobs in this partition.
TRESBillingWeights=<TRES Billing Weights> — TRESBillingWeights is used to define the billing weights of each TRES type that will be used in calculating the usage of a job. The calculated usage is used when calculating fairshare and when enforcing the TRES billing limit on jobs. Updates affect new jobs and not existing jobs. See the slurm.conf man page for more information.

🎀 RESERVATIONS - SPECIFICATIONS FOR CREATE AND UPDATE COMMANDS 🎀

A new reservation must specify at least one access control field: Users, Groups, Accounts, QOS, and AllowedPartitions. But plot twist: Users and Groups are mutually exclusive, you can't have both. If multiple access control fields are specified, a job must be approved by all of them to use the reservation. Strict gatekeeping, honestly iconic.

ReservationName=<name> — identifies the name of the reservation to be created or updated. This parameter is required for update. If you omit it when creating a reservation, a name will be created automatically (e.g., "scontrol create reservation ...") — the system names your baby for you 🍼
Accounts=<account list> — the list of accounts allowed to use the reserved nodes, like "Accounts=physcode1,physcode2". If a user is in any of the accounts or subaccounts you list, they can use the reserved nodes 💖. (Peep the start of this section for the general tea on how access control fields work.) You can also deny access instead of granting it — just put a - in front of every account name. Or, honestly iconic, put the - in front of the equal sign instead. So like "Accounts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2" will let literally any account except physcode1 and physcode2 use the reservation ✨. Wanna add or remove individual accounts from a reservation that already exists? Use the update command and slap a + or - before the = sign. Heads up tho: if any accounts are denied (name preceded by -), then all other accounts are implicitly allowed to use the reservation, and it is not possible to also explicitly specify allowed accounts. Pick a lane bestie 😭. And regardless of whatever accounts you set here, Root and the SlurmUser get access to all reservations, periodt 👑.
AllowedPartitions=<partition list> — the list of partitions permitted to use the reserved resources, for example "Partition=debug,low". Use any of the specified partitions and you can use the reserved resources 🌸. (Again, general access-control deets are at the start of this section.) You can deny partitions access too by preceding all the partition names with -, or by putting the - before the equal sign. For example, "AllowedPartitions=-debug,-low" or "AllowedPartitions-=debug,low" will permit any partition except debug and low to use the reservation 🎀. Add or remove individual partitions from an existing reservation with the update command and a + or - before the = sign. If any partitions are denied (partition name preceded by -), then all other partitions are implicitly allowed to use the reservation and it is not possible to also explicitly specify allowed partitions — same deal as above, no double-dipping 🙈. Root and the SlurmUser get access to all reservations regardless of the partition(s) set here 💅.
BurstBuffer=<buffer_spec>[,<buffer_spec>,...] — specs for the burst buffer resources you're reserving 🦋. A buffer_spec is made of four elements: [plugin:][type:]#[units]. plugin is the burst buffer plugin name — currently either "datawarp" or "generic". If you don't specify a plugin, the reservation applies to all configured burst buffer plugins. type specifies a Cray generic burst buffer resource, for example "nodes". If type isn't specified, the number is a measure of storage space. The units can be "N" (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (those are the powers of 1024 girlies) and "KB", "MB", "GB", "TB", "PB" (powers of 1000 💀). Default units are bytes for reservations of storage space. Examples: "BurstBuffer=datawarp:2TB" (reserve 2TB of storage plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (reserve 100 GB of storage from all configured burst buffer plugins). Jobs using this reservation are not restricted to these burst buffer resources — they may use these reserved resources plus any which are generally available ✨.

NOTE: Usually Slurm interprets KB, MB, GB, TB, PB units as powers of 1024, but for Burst Buffers size specifications Slurm supports both IEC/SI formats. Why? Because the CRAY API for managing DataWarps supports both formats 🤷‍♀️ so we all just vibe with it.

CoreCnt=<num> — how many cores to reserve. This option is only supported when select/cons_tres is used ⚠️. If NodeCnt or Nodelist is used, this is the total number of cores to reserve, where cores per node is CoreCnt/NodeCnt 🧮.
Duration=<time> — how long the reservation lasts ⏳. A new reservation must specify an end time or a duration (you gotta give one of them, okay?). Valid formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds, or UNLIMITED. Time resolution is one minute, and second values are rounded up to the next minute 🌙. Output format is always [days-]hours:minutes:seconds.
EndTime=<time_spec> — the end time for the reservation 🕐. A new reservation must specify an end time or a duration. Valid formats are the same as for StartTime 💖.
Flags=<flags> — flags associated with the reservation 🚩. You can add or remove individual flags from an existing reservation by adding a + or - sign before the = sign. For example: Flags-=DAILY (NOTE: this shortcut is not supported for all flags 🙃). Currently supported flags include:
- ANY_NODES — this is a reservation for burst buffers and/or licenses only, not compute nodes. If this flag is set, a job using this reservation may use the associated burst buffers and/or licenses plus any compute nodes. If this flag is not set, a job using this reservation may use only the nodes and licenses associated with the reservation ✨.
- DAILY — repeat the reservation at the same time every day 🔁.
- FLEX — permit jobs requesting the reservation to begin prior to the reservation's start time, end after the reservation's end time, and use any resources inside and/or outside of the reservation regardless of any constraints possibly set in the reservation 🌈. A typical use case is to prevent jobs not explicitly requesting the reservation from using those reserved resources, rather than forcing jobs requesting the reservation to use those resources in the time frame reserved. Another use case could be to always have a particular number of nodes with a specific feature reserved for a specific account, so users in this account may use these nodes plus possibly other nodes without this feature 🫶.

Unless this reservation also has the MAINT flag, resources held by other reservations are still off limits 🚫. A reservation that reserves whole nodes keeps those nodes excluded, unless this reservation also has the OVERLAP flag and the nodes belong to both reservations. When the other reservation reserves only part of each node, its reserved cores are excluded instead of its nodes, so a job may still be allocated the unreserved cores on those nodes 🧚. OVERLAP never grants access to another reservation that has the MAINT flag: if it reserves whole nodes those nodes stay excluded, and if it reserves only part of each node its reserved cores do 💀. A reservation that holds no nodes of its own (a license or burst buffer reservation created with ANY_NODES) and that also has the OVERLAP flag keeps the full OVERLAP grant: its jobs may use any node that is not in a MAINT reservation 👑.

FORCE_START — allow reoccurring reservations to have a start time in the past, so reoccurring reservations won't be required to be made a day in advance ⏰. This flag must be used with a reoccurring reservation flag.
HOURLY — repeat the reservation at the same time every hour 🕰️.
IGNORE_JOBS — ignore currently running jobs when creating the reservation 🏃‍♀️. This can be especially useful when reserving all nodes in the system for maintenance 🛠️.
LICENSE_ONLY — see ANY_NODES ⬆️.
MAGNETIC — this flag allows jobs to be considered for this reservation even if they didn't request it 🧲.
MAINT — maintenance mode, receives special accounting treatment 🧰. This reservation is permitted to use resources that are already in another reservation. The MAINT flag behaves similar to STATIC_ALLOC (see below) in that it will not replace nodes once they are set 🔒.
NO_HOLD_JOBS_AFTER — by default, when a reservation ends, the reservation request will be removed from any pending jobs submitted to the reservation and will be put into a held state 😴. Use this flag to let jobs run outside of the reservation after the reservation is gone. Flag removal with -= is not supported ⚠️.
OVERLAP — this reservation can be allocated resources that are already in another reservation 🔄. Resources held by a reservation with the MAINT flag are the exception: the whole nodes it reserves, or the cores it reserves on partial nodes, stay excluded with or without FLEX 🚫. When combined with the FLEX flag, jobs requesting this reservation may use another reservation's whole nodes only if those nodes belong to both reservations; see FLEX for reservations that reserve only part of each node 🌸. That last restriction does not apply when this reservation holds no nodes of its own (see ANY_NODES), in which case its jobs may use any node outside a MAINT reservation ✨. Flag removal with -= is not supported ⚠️.
PART_NODES — this flag can be used to reserve all nodes within the specified partition 👯‍♀️. PartitionName and Nodes=ALL must be specified with this flag.

PURGE_COMP[=<timespec>] — bestie, this one purges the reservation if it's ever idle for timespec, and "idle" means literally no jobs associated with it 💀. If you don't give a timespec, the default is 5 minutes. Valid timespec formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, and days-hours:minutes:seconds. Time resolution is one minute and second values are rounded up to the next minute, okurr. The output format is always [days-]hours:minutes:seconds.
REPLACE — nodes that are DOWN, DRAINED, or allocated to jobs get automatically replenished using idle resources ✨. You'd use this to keep a constant number of idle resources available for pending jobs (subject to availability of idle resources, no cap). This should be used with the NodeCnt reservation option — do not identify specific nodes to be included in the reservation, fr fr. Flag removal with -= is not supported.

NOTE: removing a node from the cluster while in a reservation with the REPLACE flag will not cause it to be replaced. Like, it's gone gone.

REPLACE_DOWN — nodes that are DOWN or DRAINED get automatically replenished using idle resources, and this is the default behavior 👑. Same idea as above: keep a constant sized pool of resources available for pending jobs (again, subject to availability of idle resources). Nodes in use by another reservation will not be considered as possible replacement nodes. This should be used with the NodeCnt reservation option — do not identify specific nodes to be included in the reservation. Flag removal with -= is not supported.

NOTE: removing a node from the cluster while in a reservation with the REPLACE_DOWN flag will not cause it to be replaced either, iykyk.

SPEC_NODES — reservation is for specific nodes (output only), so you can look but you can't touch 👀.
STATIC_ALLOC — makes it so that after the nodes are selected for a reservation, they don't change 🧊. Without this option, when nodes are selected for a reservation and one goes down, the reservation will select a new node to fill the spot. With it? Snapshot city. Immovable.
TIME_FLOAT — the reservation start time is relative to the current time and moves forward through time (e.g. a StartTime=now+10minutes will always be 10 minutes in the future) ⏳. Repeating (e.g. DAILY) floating reservations are not supported, sorry bestie. Flag cannot be added to or removed from an existing reservation.
USER_DELETE — allow any user able to run in the reservation to delete it 🫶.

NOTE: QOS and AllowedPartitions restrictions on the reservation do not apply to this option.

WEEKDAY — repeat the reservation at the same time on every weekday (Monday, Tuesday, Wednesday, Thursday and Friday) 🗓️.
WEEKEND — repeat the reservation at the same time on every weekend day (Saturday and Sunday). Self-care scheduling, honestly 🌸.
WEEKLY — repeat the reservation at the same time every week. Periodt 🔁.

Features=<features> — sets the reservation's required node features ✨. Multiple values may be & separated if all features are required (AND operation) or separated by | if any of the specified features are required (OR operation). Parenthesis are also supported for features to be ANDed together with counts of nodes having the specified features. For example, Features=[(knl&a2a&flat)*4&haswell*2] indicates the advanced reservation should include 4 nodes with ALL of the features knl, a2a, and flat plus 2 nodes with the feature haswell. It's giving very structured, very demure 🌈.

Value may be cleared with blank data value, Features=.

Groups=<group list> — list of groups permitted to use the reserved nodes, for example Group=bio,chem 🧪. Refer to the start of this section for general details about access control fields. Unlike users, you cannot deny access to reservations based on group membership. Root and the SlurmUser are given access to all reservations, regardless of the groups set here — main character exemption fr.

You can add or remove individual groups from an existing reservation by using the update command and adding a + or - sign before the = sign. You can also unset the Groups field by updating the reservation with an empty string: group=''

Prior to version 25.05, if a reservation used user- or group-based access controls and slurmctld ever failed to validate the user(s) or group(s), it would delete the reservation. Reservations are now resilient to temporary validation outages 💅.

Licenses=<license> — specification of licenses (or other resources available on all nodes of the cluster) which are to be reserved 🎟️. License names can be followed by a colon and count (the default count is one). Multiple license names should be comma separated (e.g. Licenses=foo:4,bar). A new reservation must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses. If a reservation includes Licenses, but no NodeCnt or Nodes, then the option Flags=LICENSE_ONLY must also be specified. Jobs using this reservation are not restricted to these licenses, but may use these reserved licenses plus any which are generally available — sharing is caring 🫶.
MaxStartDelay[=<timespec>] — change the MaxStartDelay value, which specifies the maximum time an eligible job not requesting this reservation can delay a job requesting it ⏰. Default is none. Valid formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds. Time resolution is one minute and second values are rounded up to the next minute. Output format is always [days-]hours:minutes:seconds.
NodeCnt=<num> — identify number of nodes to be reserved 🔢. The number can include a suffix of k or K, in which case the number specified is multiplied by 1024 (math slay). A new reservation must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses.
Nodes=<name> — identify the node(s) to be reserved 🖥️. Multiple node names may be specified using simple node range expressions (e.g. Nodes=lx[10-20]). When using Nodes to specify more or fewer nodes, NodeCnt will be updated to honor the new number of nodes. However, when setting an empty list (Nodes=), the nodelist will be filled with random nodes to fulfill the previous nodecnt and the SPEC_NODES flag will be removed 😳. A new reservation must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses. A specification of ALL will reserve all nodes. Set Flags=PART_NODES and PartitionName= in order for changes in the nodes associated with a partition to also be reflected in the nodes associated with a reservation. You can add or remove nodes from an existing reservation by adding a + or - sign before the = sign. It also works to do nodes=+lx0,-lx[2-4] if you want to remove lx[2-4] and add lx0 at the same time 💫.

NOTE: when updating a reservation, if Nodes and Nodecnt are set simultaneously, nodecnt will always be honored. The reservation will get a subset of nodes if nodes > nodecnt, or it will add extra nodes to the list when nodes < nodecnt. No negotiating.

PartitionName=<name> — partition used to reserve nodes from 🧩. This will attempt to allocate all nodes in the specified partition unless you request fewer resources than are available with CoreCnt, NodeCnt or TRES. If no partition is specified at submit time, this partition will override the job's default partition. Jobs explicitly requesting a different partition will still be allowed to use this reservation as long as there are enough overlapping nodes between both partitions to allocate the job. You cannot clear the reservation partition — that one's locked in 🔒.

NOTE: if AllowedPartitions is used and PartitionName is not given, the first partition in AllowedPartitions is what is used as PartitionName.

QOS=<qos list> — list of QOS permitted to use the reserved resources, for example QOS=normal,standby 👑. A user using any of the specified QOS may use the reserved resources. Refer to the start of this section for general details about access control fields. QOS can also be denied access to reservations by preceding all of the QOS names with -. Alternately precede the equal sign with -. For example, QOS=-normal,-standby or QOS-=normal,standby will permit any QOS except normal and standby to use the reservation. You can add or remove individual QOS from an existing reservation by using the update command and adding a + or - sign before the = sign. If QOS are denied access to a reservation (QOS name preceded by a -), then all other QOS are implicitly allowed to use the reservation and it is not possible to also explicitly specify allowed QOS — pick a lane bestie, you can't do both 🙈. Root and the SlurmUser are given access to all reservations, regardless of the QOS set here.
Skip — used on a reoccurring reservation, skip to the next reservation iteration ⏭️. Requires the same permissions as deleting a reservation.

NOTE: only available for update.

StartTime=<time_spec> — when the reservation kicks off. A brand-new reservation must have a start time, no skipping. Accepted forms: HH:MM:SS for a specific time of day (seconds are optional). If that time already passed today, Slurm assumes the next day — cute, it's forward-thinking. You can also just say midnight, noon, elevenses (11 AM), fika (3 PM) or teatime (4 PM) and honestly those are the only times that matter. You can tack AM or PM onto a time-of-day for morning vs evening vibes. Wanna pin down the actual day? Give a date as MMDDYY or MM/DD/YY or MM.DD.YY, or a date-and-time as YYYY-MM-DD[THH:MM[:SS]]. You can also do the relative thing: now + count time-units, where time-units is seconds (the default), minutes, hours, days, or weeks. The keywords today and tomorrow work too. And heads up: you cannot update the StartTime of a reservation that's in ACTIVE state. Periodt.

TRES=<tres_spec> — a comma-separated list of the TRES your reservation needs. The TRES types currently supported with reservations are: CPU, GRES, Node, License and BB. CPU and Node follow the same format as the CoreCnt and NodeCnt parameters respectively. License names can be followed by an equal = and a count:

License/<name1>=<count1>[,License/<name2>=<count2>,...]

BurstBuffer works basically the same way as the BurstBuffer parameter — the one difference is that the colon : gets swapped for an equal = so it plays nice with the TRES format.

Some examples of TRES valid specifications:

TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3

TRES=node=5k,license/iop1=2

TRES=gres/gpu:a100=2

Please note that CPU, Node, License and BB can override the CoreCnt, NodeCnt, Licenses and BurstBuffer parameters respectively. Also, CPU represents CoreCnt in a reservation and will be adjusted if you have threads per core on your nodes — so don't come for it, it's just doing math.

Note that a reservation containing nodes or cores is tied to one partition and can't span resources over multiple partitions. The one exception? When the reservation is created with explicitly requested nodes — then it can do its little multiverse thing.

Note that if GRES is requested, those GRES must be TRES listed in AccountingStorageTRES to be valid. No vibes-based GRES, sorry.

NOTE: Reservations will automatically attempt to enforce binding on GRES. Multiple GRES cannot be reserved across multiple sockets. That's the rule, bestie.

TRESPerNode=<tres_spec> — a comma-separated list of TRES required per node for the reservation. See TRES above for the supported types.

TRESPerNode=gres/gpu:a100=2

The above will allocate 2 gpu:a100 per node as specified in nodecnt. Slay, math.

Users=<user list> — the list of users permitted to use the reserved nodes, for example "User=jones1,smith2". Users can also be denied access to a reservation, indicated by having a - sign before their name, e.g. "User=-jones1,-smith2". Refer to the start of this section for general details about access control fields.

Root and the SlurmUser are given access to all reservations, regardless of the users set here. Main character privileges, honestly.

Each user in the system may be in one of three states in a reservation:

Explicitly allowed
Unspecified
Explicitly denied

If any users are explicitly allowed to use a reservation, all other users are implicitly denied access. If any users are explicitly denied access to a reservation, all other users are implicitly allowed access to the reservation. It's giving "if you're not on the list, you're not coming in" energy.

When updating existing reservations, you may use the += and -= operators to move users up (+) or down (-) on the list of states. However, you cannot have a mix of explicitly allowed and explicitly denied users on the same reservation, and an update command that would result in that state will result in an error. Pick a lane, babe. The + or - sign can also be placed directly before the username. For example, the following are equivalent:

User-=jones1,smith2
User=-jones1,-smith2

In these examples, if the users were explicitly allowed before, this assignment would remove their names from the allowed list and place them in the unspecified state. If the command was repeated again, the users would be explicitly denied from the reservation. Glow down of the century 😭.

You can also unset the Users field (reverting all to a neutral/unspecified state) by updating the reservation with an empty string:

user=''

Prior to version 25.05, if a reservation used user- or group-based access controls and slurmctld ever failed to validate the user(s) or group(s), it would delete the reservation. Reservations are now resilient to temporary validation outages. Character development fr fr 🫶.

🔥 PERFORMANCE 🔥

Executing scontrol sends a remote procedure call to slurmctld. If enough calls from scontrol or other Slurm client commands that send remote procedure calls to the slurmctld daemon come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service. Not the cute kind of viral moment.

Do not run scontrol or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Ma'am, that's a jump scare for the daemon. Ensure that programs limit calls to scontrol to the minimum necessary for the information you are trying to gather. Less is more, iykyk.

🌈 ENVIRONMENT VARIABLES 🌈

Some scontrol options may be set via environment variables. These environment variables, along with their corresponding options, are listed below. (Note: Command line options will always override these settings — CLI stays winning 👑.)

SCONTROL_ALL — -a, --all
SCONTROL_FEDERATION — --federation
SCONTROL_FUTURE — -F, --future
SCONTROL_LOCAL — --local
SCONTROL_SIBLING — --sibling

SLURM_BITSTR_LEN — Specifies the string length to be used for holding a job array's task ID expression. The default value is 64 bytes. A value of 0 will print the full expression with any length required. Larger values may adversely impact the application performance. So like… don't go feral with it.

SLURM_CLUSTERS — Same as --clusters.

SLURM_CONF — The location of the Slurm configuration file.

SLURM_CONF_OUT — When running 'write config', the location of the Slurm configuration file to be written.

SLURM_DEBUG_FLAGS — Specify debug flags for scontrol to use. See DebugFlags in the slurm.conf(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf.

SLURM_JSON — Control JSON serialization:

compact — Output JSON as compact as possible.
pretty — Output JSON in pretty format to make it more readable.

SLURM_TIME_FORMAT — Specify the format used to report time stamps. A value of standard, the default value, generates output in the form "year-month-dateThour:minute:second". A value of relative returns only "hour:minute:second" if the current day. For other dates in the current year it prints the "hour:minute" preceded by "Tomorr" (tomorrow), "Ystday" (yesterday), the name of the day for the coming week (e.g. "Mon", "Tue", etc.), otherwise the date (e.g. "25 Apr"). For other years it returns a date month and year without a time (e.g. "6 Jun 2012"). All of the time stamps use a 24 hour format.

A valid strftime() format can also be specified. For example, a value of "%a %T" will report the day of the week and a time stamp (e.g. "Mon 12:34:56"). Cute AND functional 💅.

SLURM_TOPO_LEN — Specify the maximum size of the line when printing Topology. If not set, the default value is unlimited.

SLURM_YAML — Control YAML serialization:

compact — Output YAML as compact as possible.
pretty — Output YAML in pretty format to make it more readable.

👑 AUTHORIZATION 👑

Slurm generally classifies users into three authorization levels:

Administrators include root, SlurmUser, and any users with "AdminLevel=Administrator"
Privileged Users include all Administrators and any users with "AdminLevel=Operator"
Effective Owners of a job or step include the direct owner and all Coordinators of the corresponding account.

Note that configuring AdminLevels or Coordinators requires the use of SlurmDBD. The authorization levels required to execute various commands are defined in the following table. (Table's gone, list's here — same facts, better energy ✨)

Job/step stuff:

scontrol update job: — Privileged or Effective Owner
scontrol requeue: — Privileged or Effective Owner
scontrol update step: — Privileged or Effective Owner
scontrol suspend: — Privileged or Coordinator
scontrol resume: — Privileged or Coordinator
scontrol notify: — Root, SlurmUser, or direct owner
scontrol write batch_script: — Privileged or direct owner

Node stuff:

scontrol create node: — Admin
scontrol update node: — Admin
scontrol delete node: — Admin

Partition stuff:

scontrol create partition: — Admin
scontrol update partition: — Admin
scontrol delete partition: — Admin

Reservation stuff:

scontrol create reservation: — Privileged
scontrol update reservation: — Privileged
scontrol delete reservation: — Privileged

Big boss stuff:

scontrol reconfig: — Admin
scontrol shutdown: — Admin
scontrol takeover: — Admin

The scontrol show commands are available to all users by default. If any PrivateData restrictions are defined in the slurm.conf file, the listed authorization levels will be required to access the corresponding data.

Show commands:

scontrol show job(s): — Privileged or Effective Owner
scontrol show step(s): — Privileged or Effective Owner
scontrol show node: — Privileged
scontrol show host*: — Privileged
scontrol show partition: — Privileged
scontrol show reservation: — Privileged or Access
scontrol show assoc_mgr: — See assoc_mgr description for details

✨ EXAMPLES ✨

okay bestie, buckle up — this is a whole lil' scontrol session start to finish, and it's honestly iconic. watch how she shows a partition, tweaks it, peeks at a job, updates the job, flexes on hostname expansion, then does a full reservation create/update/delete arc before logging off. no cap, it's giving "I run this cluster" energy 💅

$ scontrol
scontrol: show part debug
PartitionName=debug
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO Exclusive=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   NodeSets=ALL
   Nodes=snowflake[0-48]
   PriorityJobFactor=100 PriorityTier=100 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=REQUEUE
   State=UP TotalCPUs=588 TotalNodes=49 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=588,mem=1204224M,node=49,billing=49
scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
scontrol: show job 71701
JobId=71701 Name=hostname
   UserId=da(1000) GroupId=da(1000)
   Priority=66264 Account=none QOS=normal WCKey=*123
   JobState=COMPLETED Reason=None Dependency=(null)
   TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
   SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
   StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
   SuspendTime=None SecsPreSuspend=0
   Partition=debug AllocNode:Sid=snowflake:4702
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=snowflake0
   NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
   MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) Reservation=(null)
   OverSubscribe=OK Exclusive=NO Contiguous=0 Licenses=(null) Network=(null)
scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
scontrol: show hostnames tux[1-3]
tux1
tux2
tux3
scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
Reservation created: dbremer_1
scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
scontrol: delete Reservation=dbremer_1
scontrol: quit

like, the range expansion on tux[1-3]? she ate that. and the reservation went from created to maintained to deleted with zero drama. main character behavior, fr fr 🌈

🎀 COPYING 🎀

legal era, but make it cute. here's the exact receipt so nobody comes for us:

Copyright (C) 2002-2007 The Regents of the University of California.
Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
Copyright (C) 2008-2010 Lawrence Livermore National Security.
Copyright (C) 2010-2022 SchedMD LLC.

This file is part of Slurm, a resource management program.  For details,
see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your
option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
more details.

periodt. keep the copyright lines intact, bestie — they're not decorative 💜

🦋 FILES 🦋

/etc/slurm.conf

that's the one file listed. she's the blueprint, the moment, the whole config vibe.

👑 SEE ALSO 👑

the squad, in no particular order but very much all invited to the group chat:

scancel(1)
sinfo(1)
squeue(1)
slurm_create_partition (3)
slurm_delete_partition (3)
slurm_load_ctl_conf (3)
slurm_load_jobs (3)
slurm_load_node (3)
slurm_load_partitions (3)
slurm_reconfigure (3)
slurm_requeue (3)
slurm_resume (3)
slurm_shutdown (3)
slurm_suspend (3)
slurm_takeover (3)
slurm_update_job (3)
slurm_update_node (3)
slurm_update_partition (3)
slurm.conf(5)
slurmctld(8)

and that's the whole page, bestie 🫶 this edition is Slurm 26.05, hanging out in the Slurm Commands section, answering to scontrol(1) — footer receipt delivered, no extra lore invented, just vibes and facts. catch you in the next man page 💖✨

📦 where this page comes from

arch rolling: slurm-llnl 26.05.5.1-1 · GPL-2.0-or-later
epel 10: slurm 26.05.5-1.el10_3 · GPL-2.0-or-later AND BSD-2-Clause AND BSD-1-Clause
nixpkgs unstable: slurm 26-05-5-1
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms

scontrol(1)

🌸 NAME 🌸

✨ SYNOPSIS ✨

💅 DESCRIPTION 💅

🎀 OPTIONS 🎀

🦋 COMMANDS 🦋

💅 power — the glow-up/glow-down button

down

down asap

down force

up

🔄 reboot — the "turn it off and on again" of the cluster

🧚 reconfigure — the "everyone re-read the group chat" command

🔓 release — free the jobs

♻️ requeue — do it again, but pending this time

🧸 requeuehold — requeue, but make it held

▶️ resume — un-pause the job

📝 schedloglevel — talk to the scheduler

🐛 setdebug — turn the volume up

🚩 setdebugflags — flip the flags

🌈 show <ENTITY>[=<ID>] or <ENTITY> [<ID>]

🦋 aliases

💜 assoc_mgr

🧚 bbstat

🌸 burstbuffer

⭐ config

💖 daemons

🎀 dwstat

🌈 federation

🦋 hostlist

💜 hostlistsorted

🧚 hostnames

🌸 job

🎀 licenses

⭐ node

💖 partition

🌈 reservation

🦋 slurmd

💜 step

🧚 topoconf

🌸 topology [topology_name] [unit=NAME] [node=NAME]

🌙 shutdown <OPTION>

😴 suspend <job_list>

👑 takeover [<INDEX>]

🔥 top <job_list>

🎟️ token [lifespan=<lifespan>] [username=<username>]