sacctmgr(1)

📦

version 26.05.5.1 · 11 versions

26.05.5.1 · arch rolling, epel 10, nixpkgs unstable (latest)
24.11.3 · opensuse-leap 16.0
25-11-8 · nixos 26.05
22.05.11 · epel 9
26.05.4 · ubuntu stonking
21.08.5 · ubuntu jammy
24.05.2 · fedora 43, fedora 44
23.11.4 · ubuntu noble
22.05.8 · debian bookworm
25.11.2 · ubuntu resolute
24.11.5 · debian trixie, ubuntu questing

, in arch rolling, epel 10, nixpkgs unstable

🎀 sacctmgr(1) 🎀

okay so bestie, welcome to the Slurm Commands manual page for sacctmgr — literally the main character of Slurm accounting 💅✨

🌸 NAME 🌸

sacctmgr — used to view and modify Slurm account information. no cap, that's the whole vibe 🫶

🦋 SYNOPSIS 🦋

sacctmgr [OPTIONS...] [COMMAND...]

💖 DESCRIPTION 💖

sacctmgr is your girl for viewing or modifying Slurm account information. the account info lives in a database, and the interface to it is provided by slurmdbd (the Slurm Database daemon). this database can be a central storehouse of user and computer information for multiple computers at a single site. iconic centralization behavior 👑

Slurm account information is recorded based on four parameters that form what's called an association. those parameters are user, cluster, partition, and account. quick rundown:

user is the login name.
cluster is the name of a Slurm managed cluster as specified by the ClusterName parameter in the slurm.conf configuration file.
partition is the name of a Slurm partition on that cluster.
account is the account for a job.

the intended mode of operation is to initiate the sacctmgr command, add, delete, modify, and/or list association records, then commit the changes and exit. very "get in, do the task, leave" energy ✨

NOTE: the contents of Slurm's database are maintained in lower case. this may result in some sacctmgr output differing from that of other Slurm commands. lowercase era, iykyk 🙈

✨ OPTIONS ✨

-s, --associations — use with show or list to display associations with the entity. this is equivalent to the associations command. 🫶
-h, --help — print a help message describing the usage of sacctmgr. this is equivalent to the help command. the "bestie explain pls" button 💖
-i, --immediate — commit changes immediately without asking for confirmation. no vibe check, just send it 🔥
--json, --json=list, --json=<data_parser> — dump information as JSON using the default data_parser plugin or an explicit data_parser with parameters. sorting and formatting arguments will be ignored. this option is not available for every command. JSON girlies stay winning 🌈
-n, --noheader — no header will be added to the beginning of the output. just the data, periodt 💅
-p, --parsable — output will be '|' delimited with a '|' at the end. pipe-delimited realness 🎀
-P, --parsable2 — output will be '|' delimited without a '|' at the end. same pipe energy, less trailing drama ✨
-Q, --quiet — print no messages other than error messages. this is equivalent to the quiet command. shhh 🤫
-r, --readonly — makes it so the running sacctmgr cannot modify accounting information. the readonly option is for use within interactive mode. look, don't touch 🚫
--yaml, --yaml=list, --yaml=<data_parser> — dump information as YAML using the default data_parser plugin or an explicit data_parser with parameters. sorting and formatting arguments will be ignored. this option is not available for every command. YAML besties, this one's for you 🧚
-v, --verbose — enable detailed logging. this is equivalent to the verbose command. tell me everything 🔍
-V , --version — display version number. this is equivalent to the version command. gotta know the era 👑

💜 COMMANDS 💜

add <ENTITY> <SPECS> — add an entity. identical to the create command. new entity just dropped 🎀
archive {dump|load} <SPECS> — write database information to a flat file or load information that has previously been written to a file. archive icon behavior 🗂️
clear stats — clear the server statistics. clean slate energy ✨
create <ENTITY> <SPECS> — add an entity. identical to the add command. same slay, different command name 💅
delete <ENTITY> where <SPECS> — delete the specified entities. identical to the remove command. bye bestie 👋
dump <cluster> — dump cluster data to the specified file. if the filename is not specified it uses clustername.cfg filename by default. dump it and go 🫶
help — display a description of sacctmgr options and commands. the manual's built-in "hey bestie, over here" 💖
list <ENTITY> [<SPECS>] — display information about the specified entity. by default, all entries are displayed, you can narrow results by specifying SPECS in your query. identical to the show command. list it, filter it, love it 🌸
load <FILENAME> — load cluster data from the specified file. this is a configuration file generated by running the sacctmgr dump command. this command does not load archive data, see the sacctmgr archive load option instead. important distinction, no cap 🚨
modify <ENTITY> where <SPECS> set <SPECS> — modify an entity. glow-up time ✨
ping — ping slurmdbd. are you there bestie? 🫶
reconfigure — reconfigures the SlurmDBD if running with one. refresh the vibe 🔄
remove <ENTITY> where <SPECS> — delete the specified entities. identical to the delete command. same exit, different name 💅
show <ENTITY> [<SPECS>] — display information about the specified entity. by default, all entries are displayed, you can narrow results by specifying SPECS in your query. identical to the list command. show-stopping behavior 👑
shutdown — shutdown the server. turn it off, turn it down 💀
version — display the version number of sacctmgr. version check, periodt 🎀

🌈 INTERACTIVE COMMANDS 🌈

NOTE: all commands listed below can be used in the interactive mode, but NOT on the initial command line. interactive-only girlies, iykyk 💅

exit — terminate sacctmgr interactive mode. identical to the quit command. exit stage left 🚪
quiet — print no messages other than error messages. shush era 🤫
quit — terminate the execution of sacctmgr interactive mode. identical to the exit command. same exit, different spelling 💖
verbose — enable detailed logging. this includes time-stamps on data structures, record counts, etc. this is an independent command with no options meant for use in interactive mode. give me all the details 🔍
!! — repeat the last command. yes bestie, run that back 🔁

🦋 ENTITIES 🦋

account — an account, typically specified at job submit time using the --account= option. these may be arranged in a hierarchical fashion, for example accounts 'chemistry' and 'physics' may be children of the account 'science'. the hierarchy may have an arbitrary depth. family tree but make it accounting 🌳
association — the entity used to group information consisting of four parameters: account, cluster, partition (optional), and user. used only with the list or show command. add, modify, and delete should be done to a user, account or cluster entity, which will in turn update the underlying associations. modification of attributes like limits is allowed for an association but not a modification of the four core attributes of an association. you cannot change the partition setting (or set one if it has not been set) for an existing association. instead, you will need to create a new association with the partition included. you can either keep the previous association with no partition defined, or delete it. note that these newly added associations are unique entities and any existing usage information will not be carried over to the new association. read that again bestie, it's important 🚨
cluster — the ClusterName parameter in the slurm.conf configuration file, used to differentiate accounts on different machines. cluster girlies rise 🖥️
configuration — used only with the list or show command to report current system configuration. just the config, no edits 🎀
coordinator — a special privileged user, usually an account manager, that can add users or sub-accounts to the account they are coordinator over. this should be a trusted person since they can change limits on account and user associations, as well as cancel, requeue or reassign accounts of jobs inside their realm. trust is key, no cap 🔑
event — events like downed or drained nodes on clusters. note that this does not include transitory states like DRAINING. we're talking events, not the in-between era 💀
federation — a group of clusters that work together to schedule jobs. squad goals, but for clusters 🫶
job — used to modify specific fields of a job: Derived Exit Code, Comment, AdminComment, Extra, SystemComment, TRES, or WCKey. job fields, but make it editable ✨
problem — use with show or list to display entity problems. problems? exposed 🔍
qos — Quality of Service. bougie scheduling behavior 👑
reservation — a collection of resources set apart for use by a particular account, user or group of users for a given period of time. reserved for the girlies 🎀
resource — software resources for the system. those are software licenses shared among clusters. shared licenses, shared slay 💅
RunawayJobs — used only with the list or show command to report current jobs that have been orphaned on the local cluster and are now runaway. if there are jobs in this state it will also give you an option to "fix" them. this sets the end time for each job to the latest of the job's start, eligible, and submit times, and sets the state to completed by default. once corrected, this triggers the SlurmDBD to recalculate the usage from before the earliest submit time of all the runaway jobs. you must have an AdminLevel of at least Operator to perform this. this request is rejected while a usage rollup or archive is running; retry once it finishes. 🏃‍♀️💨

NOTE: this operation could take a long time and sreport may not return data until the recalculation is complete. it is best to run this when the system is not busy. don't do this during rush hour, bestie 😭

stats — used with list or show command to view server statistics. accepts optional argument of ave_time or total_time to sort on those fields. by default, sorts on increasing RPC count field. numbers girlies, this one's for you 📊
transaction — list of transactions that have occurred during a given time period. receipts, basically 🧾
tres — used with list or show command to view a list of Trackable RESources configured on the system. trackable resources, periodt 🎀
user — the login name. usernames are case-insensitive (forced to lowercase) unless the PreserveCaseUser option has been set in the SlurmDBD configuration file. lowercase unless you specifically said otherwise 💅
wckeys — Workload Characterization Key. an arbitrary string for grouping orthogonal accounts. grouping girlies unite 🫶

💖 GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES 💖

Okay bestie, buckle up, this is the part where we get into the actual knobs you can turn on associations. Every single one of these is a thing you can set, and they all come with the same energy: set it, or clear it with -1, no in-between. 🎀

⚠️ NOTE: the group limits (GrpJobs, GrpTRES, etc.) are tested at the moment a job is being considered for having resources allocated to it. So if starting a job would blow past any of its group limits, that job just does not get considered for scheduling — even if that job might have preempted other jobs and freed up enough group resources for the pending job to actually start. Yeah. That's the rule. No cap. 😭

⚠️ NOTE: TRES limit modifications of any kind using += or -= syntax will always produce a value somewhere between 0 and 18446744073709551600. So if you try to set a negative value using -=, that TRES gets set to 0 — even if there wasn't a TRES limit there beforehand. Wild, but that's the vibe check. 💀

DefaultQOS=<default_qos> — the QOS that this association and all its children use by default, as long as it's allowed in the QosLevel list below. It gets overridden if you set it directly on a user. To clear an existing value, set a new value of -1. ✨
Fairshare={<fairshare_number>|parent}, Share={<fairshare_number>|parent} — the allocated shares used for the fairshare calculation. It can also be the string parent, but that gets interpreted differently depending on where you set it:
- set it on a user → the parent association is used for fairshare
- set it on an account → that account's children get effectively re-parented for fairshare calculations to the first parent of their parent that isn't Fairshare=parent

Limits stay exactly the same — only the fairshare value is affected. To clear an existing value, set a new value of -1. 🦋

GrpJobs=<max_jobs> — the max number of running jobs, in aggregate, for this association and its children. To clear an existing value, set a new value of -1.
GrpJobsAccrue=<max_jobs> — the max number of pending jobs, in aggregate, that are able to accrue age priority for this association and its children. To clear an existing value, set a new value of -1.
GrpSubmit=<max_jobs>, GrpSubmitJobs=<max_jobs> — the max number of jobs in a pending or running state at any given time, in aggregate, for this association and its children. To clear an existing value, set a new value of -1.
GrpTRES=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...] — the max number of TRES that running jobs can be allocated, in aggregate, for this association and its children. 👑

Peep the TRES information section below for the deets, bestie.

GrpTRESMins=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...] — the max number of TRES minutes that can possibly be used by past, present and future jobs in this association and its children. Refer to the TRES information section below for more. 🌈

⚠️ NOTE: this limit is not enforced if it's set on the root association of a cluster. So even though it might show up in sacctmgr output, it will not be enforced. It's giving decorative. 🙈

⚠️ NOTE: this limit only applies when you're using the Priority Multifactor plugin. The time decays using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached, all associated jobs that are running get killed, and all future jobs submitted with associations in the group get delayed until they can run inside the limit. 😭

GrpTRESRunMins=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...] — the max number of TRES minutes that running jobs can be allocated in this association and its children. This takes the time limit of running jobs into consideration and consumes it. If the limit gets hit, no new jobs start until other jobs finish and free up some time. ⭐

Refer to the TRES information section below for further details.

GrpWall=<max_wall> — the max wall clock time that running jobs can be allocated, in aggregate, in this association and its children. The GrpWall format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. The value gets recorded in minutes, with rounding as needed. To clear an existing value, set a new value of -1.

⚠️ NOTE: although it may appear in sacctmgr output, this limit will not be enforced if it's set on the root association of a cluster.

⚠️ NOTE: this limit only applies when you're using the Priority Multifactor plugin. The time decays using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached, all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they can run inside the limit. 💀

MaxJobs=<max_jobs> — the max number of running jobs per user in this association. This gets overridden if you set it directly on a user. Default is the cluster's limit. To clear an existing value, set a new value of -1.
MaxJobsAccrue=<max_jobs> — the max number of pending jobs able to accrue age priority at any given time in this association. This gets overridden if you set it directly on a user. Default is the cluster's limit. To clear an existing value, set a new value of -1.
MaxSubmit=<max_jobs>, MaxSubmitJobs=<max_jobs> — the max number of jobs in a pending or running state at any given time in this association. Default is the cluster's limit. To clear an existing value, set a new value of -1. 💅
MaxTRES=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...], MaxTRESPJ=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...], MaxTRESPerJob=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...] — the max number of TRES that each job can use in this association. This gets overridden if you set it directly on a user. Default is the cluster's limit.

Refer to the TRES information section below for further details. 🫶

MaxTRESMins=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...], MaxTRESMinsPJ=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...], MaxTRESMinsPerJob=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...] — the max number of TRES minutes that each job can use in this association. This gets overridden if you set it directly on a user. Default is the cluster's limit.

Refer to the TRES information section below for further details.

MaxTRESPN=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...], MaxTRESPerNode=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...] — the max number of TRES that each node in a job allocation can use in this association. This gets overridden if you set it directly on a user. Default is the cluster's limit.

Refer to the TRES information section below for further details. 🌸

MaxWall=<max_wall>, MaxWallDurationPerJob=<max_wall> — the max wall clock time that each job can use in this association. This gets overridden if you set it directly on a user. Default is the cluster's limit. MaxWall format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. The value gets recorded in minutes, with rounding as needed. To clear an existing value, set a new value of -1. 🎀

⚠️ NOTE: changing this value has no effect on any running jobs, but it may update the default time limit for pending jobs. So like, don't expect retroactive magic. ✨

Priority — the association priority factor used by the priority/multifactor plugin. This gets overridden if you set it directly on a user. It's unset by default, which means no extra priority is granted. To clear an existing value, set a new value of -1. 👑
QosLevel<operator><comma_separated_list_of_qos_names> — the list of QOS names available to jobs running in this association. To get a list of valid QOSs, use sacctmgr list qos. This value overrides its parent's value and pushes down to its children as the new default. Setting a QosLevel to '' (two single quotes with nothing between them) restores its default setting. You can also use the += and -= operators to add or remove certain QOSs from the QOS list. 🦋

Valid <operator> values include:

= — set QosLevel to the specified value.

⚠️ NOTE: the QOS that can be used at a given account in the hierarchy are inherited by the children of that account. So by assigning QOS with the = sign, only the assigned QOS can be used by the account and its children. It's giving exclusive. 💅

+= — add the specified <qos> value to the current QosLevel. The account will have access to this QOS and any others previously assigned to it. 🌈
-= — remove the specified <qos> value from the current QosLevel. 🙈

Peep the EXAMPLES section below, bestie — fr fr, it's worth it. 🌟

🎀 SPECIFICATIONS FOR ACCOUNTS 🎀

Okay bestie, accounts can be created, modified, and deleted with sacctmgr, and these options let you set the matching attributes or filter on them when you're querying for Accounts. Slay.

Cluster=<cluster> — specific cluster to add the account to. Default is all in system. 💅
Description=<description> — an arbitrary string describing an account. Lowkey useful for context.
Flags=<flag>[,<flag>,...] — valid options are:
- NoUsersAreCoords — remove the privilege that UsersAreCoords sets.
- UsersAreCoords — if set, all users in this account will have the coordinator status here and of any sub-account in its hierarchy.
Name=<name> — the name of an account. Note the name must be unique and cannot represent different accounts at different points in the account hierarchy. Periodt.
Organization=<org> — organization to which the account belongs.
Parent=<parent> — parent account of this account. Default is the root account, a top level account.
RawUsage=<value> — this lets an administrator reset the raw usage accrued to an account. The only value currently supported is 0 (zero). This is a settable specification only — it cannot be used as a filter to list accounts.
WithAssoc — display all associations for this account.
WithCoord — display all coordinators for this account.
WithDeleted — display information with previously deleted data. Accounts that are deleted within 24 hours of being created and did not have a job run in the account during that time will be removed from the database. Otherwise, the account will be marked as deleted and will be viewable with the WithDeleted flag.

NOTE: If using the WithAssoc option you can also query against association specific information to view only certain associations this account may have. These extra options can be found in the SPECIFICATIONS FOR ASSOCIATIONS section. You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

✨ LIST/SHOW ACCOUNT FORMAT OPTIONS ✨

These are the fields you can display when viewing Account records by using the format= option. The default format is:

Account,Description,Organization

Account — the name of an account.
Description — an arbitrary string describing an account.
Flags — flags set on the account.
Organization — organization to which the account belongs.
Coordinators — list of users that are a coordinator of the account. (Only filled in when using the WithCoordinator option.)

NOTE: If using the WithAssoc option you can also view the information about the various associations the account may have on all the clusters in the system. The association information can be filtered. Note that all the accounts in the database will always be shown, since the filter only takes effect over the association data. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

🦋 SPECIFICATIONS FOR ASSOCIATIONS 🦋

Associations can be created, modified, and deleted with sacctmgr. These options let you set the corresponding attributes or filter on them when querying for Associations. No cap.

Clusters=<cluster_name>[,<cluster_name>,...] — list the associations of the cluster(s).
Accounts=<account_name>[,<account_name>,...] — list the associations of the account(s).
Users=<user_name>[,<user_name>,...] — list the associations of the user(s).
Partitions=<partition_name>[,<partition_name>,...] — list the associations of the partition(s).

NOTE: Use Partitions="" or Partitions='' with no other names listed when specifying the case where there is no partition. This can be useful when using a command with an entity that has associations with and without partitions. If given in a shell where the quotes will be consumed then they must be quoted themselves. For example: Partitions=\"\".

NOTE: You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

Other options unique for listing associations:

OnlyDefaults — display only associations that are default associations.
Tree — display account names in a hierarchical fashion.
WithDeleted — display information with previously deleted data. Associations that are deleted within 24 hours of being created and did not have a job run in the association during that time will be removed from the database. Otherwise, the association will be marked as deleted and will be viewable with the WithDeleted flag.
WithSubAccounts — display information with subaccounts. Only really valuable when used with the account= option. This will display all the subaccount associations along with the accounts listed in the option.
WOLimits — display information without limit information. This is for a smaller default format of "Cluster,Account,User,Partition".
WOPInfo — display information without parent information (i.e. parent id, and parent account name). This option also implicitly sets the WOPLimits option.
WOPLimits — display information without hierarchical parent limits (i.e. will only display limits where they are set instead of propagating them from the parent).

🌸 LIST/SHOW ASSOCIATION FORMAT OPTIONS 🌸

These are the fields you can display when viewing Association records by using the format= option. Iconic list incoming.

Account — the name of an account in the association.
Cluster — the name of a cluster in the association.
DefaultQOS — the QOS this association and its children will use by default if allowed in the QosLevel list mentioned below.
Fairshare, Share — allocated shares used for fairshare calculation. Can also be the string parent, which is interpreted differently if set on a user or on an account. If set on a user, the parent association is used for fairshare. If set on an account, that account's children will be effectively re-parented for fairshare calculations to the first parent of their parent that is not Fairshare=parent. Limits remain the same, only its fairshare value is affected.
Flags — flags set on the association.
GrpJobs — maximum number of running jobs in aggregate for this association and its children.
GrpJobsAccrue — maximum number of pending jobs in aggregate able to accrue age priority for this association and its children.
GrpSubmit, GrpSubmitJobs — maximum number of jobs in a pending or running state at any time in aggregate for this association and its children.
GrpTRES — maximum number of TRES able to be allocated by running jobs in aggregate for this association and its children.
GrpTRESMins — maximum number of TRES minutes that can possibly be used by past, present, and future jobs in this association and its children.
GrpTRESRunMins — maximum number of TRES minutes able to be allocated by running jobs in this association and its children. This takes into consideration time limit of running jobs and consumes it. If the limit is reached no new jobs are started until other jobs finish to allow time to free up.
GrpWall — maximum wall clock time able to be allocated by running jobs in aggregate in this association and its children.
ID — the id of the association.
Lineage — complete path up the hierarchy to the root association.
MaxJobs — maximum number of running jobs per user.
MaxJobsAccrue — maximum number of pending jobs able to accrue age priority at any given time.
MaxSubmit, MaxSubmitJobs — maximum number of jobs in a pending or running state at any time.
MaxTRES, MaxTRESPJ, MaxTRESPerJob — maximum number of TRES each job can use.
MaxTRESMins, MaxTRESMinsPJ, MaxTRESMinsPerJob — maximum number of TRES minutes each job can use.
MaxTRESPN, MaxTRESPerNode — maximum number of TRES each node in a job allocation can use.
MaxWall, MaxWallDurationPerJob — maximum wall clock time each job can use.
ParentID — the association id of the parent of this association.
ParentName — the account name of the parent of this association.
Partition — the name of a partition in the association.
Priority — association priority factor to be used by the priority/multifactor plugin.
Qos — valid QOSs for this association.
QosRaw — numeric IDs of valid QOSs for this association.
User — the name of a user in the association.
WithRawQOSLevel — display QosLevel in an unevaluated raw format, consisting of a comma-separated list of QOS names prepended with '' (nothing), '+' or '-' for the association. QOS names without +/- prepended were assigned (ie, sacctmgr modify ... set QosLevel=qos_name) for the entity listed or on one of its parents in the hierarchy. QOS names with +/- prepended indicate the QOS was added/filtered (ie, sacctmgr modify ... set QosLevel=[+-]qos_name) for the entity listed or on one of its parents in the hierarchy. Including WOPLimits will show exactly where each QOS was assigned, added or filtered in the hierarchy.

✨ SPECIFICATIONS FOR CLUSTERS ✨

Clusters can be created, modified, and deleted with sacctmgr. These options let you set the matching attributes or filter on them when you're querying for Clusters. Cluster girlies, this is your moment 💅

Classification=<classification> — Type of machine; current classifications are capability, capacity and capapacity.
Features[+|-]=<comma_separated_list_of_feature_names> — Features that are specific to the cluster. Federated jobs can be directed to clusters that contain the job requested features.

To add or remove individual features, use the += or -= operators. To clear all existing features, set a new value of '' (two single quotes with nothing between them). Yes, literally empty quotes, bestie — that's the move 🫶

Federation=<federation> — The federation that this cluster should be a member of. A cluster can only be a member of one federation at a time.
FedState=<state> — The state of the cluster in the federation. Valid states are:
- ACTIVE — Cluster will actively accept and schedule federated jobs.
- INACTIVE — Cluster will not schedule or accept any jobs.
- DRAIN — Cluster will not accept any new jobs and will let existing federated jobs complete.
- DRAIN+REMOVE — Cluster will not accept any new jobs and will remove itself from the federation once all federated jobs have completed. When removed from the federation, the cluster will accept jobs as a non-federated cluster.
Name=<name> — The name of a cluster. This should be equal to the ClusterName parameter in the slurm.conf configuration file for some Slurm-managed cluster.
RPC=<rpc_list> — Comma-separated list of numeric RPC values.
WithDeleted — Display information with previously deleted data. Clusters that are deleted within 24 hours of being created and did not have a job run in the cluster during that time will be removed from the database. Otherwise, the cluster will be marked as deleted and will be viewable with the WithDeleted flag.
WithFed — Appends federation related columns to default format options (e.g. Federation,ID,Features,FedState).
WOLimits — Display information without limit information. This is for a smaller default format of Cluster,ControlHost,ControlPort,RPC.

NOTE: You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section. Bestie, don't forget the general specs exist 💖

🔍 LIST/SHOW CLUSTER FORMAT OPTIONS 🔍

Fields you can display when viewing Cluster records by using the format= option. Pick your faves, it's giving build-your-own-cluster-view ✨

Classification — Type of machine, i.e. capability, capacity or capapacity.
Cluster — The name of the cluster.
ControlHost — When a slurmctld registers with the database the ip address of the controller is placed here.
ControlPort — When a slurmctld registers with the database the port the controller is listening on is placed here.
Features — The list of features on the cluster (if any).
Federation — The name of the federation this cluster is a member of (if any).
FedState — The state of the cluster in the federation (if a member of one).
FedStateRaw — Numeric value of the name of the FedState.
Flags — Attributes possessed by the cluster. Current flags include Cray, External and MultipleSlurmd.

External clusters are registration only clusters. A slurmctld can designate an external slurmdbd with the AccountingStorageExternalHost slurm.conf option. This allows a slurmctld to register to an external slurmdbd so that clusters attached to the external slurmdbd can communicate with the external cluster with Slurm commands. No cap, that's the whole external-cluster vibe 🦋

ID — The ID assigned to the cluster when a member of a federation. This ID uniquely identifies the cluster and its jobs in the federation.
NodeCount — The current count of nodes associated with the cluster.
NodeNames — The current Nodes associated with the cluster.
RPC — When a slurmctld registers with the database the rpc version the controller is running is placed here.
TRES — Trackable RESources (Billing, BB (Burst buffer), CPU, Energy, GRES, License, Memory, and Node) this cluster is accounting for.

NOTE: You can also view the information about the root association for the cluster. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section. Root association lore is real 💀

🎀 SPECIFICATIONS FOR COORDINATOR 🎀

Coordinators can be created, modified, and deleted with sacctmgr. These options let you set the corresponding attributes or filter on them when querying for Coordinators. Coordinator era 👑

Account=<account_name>[,<account_name>,...] — Account name to add this user as a coordinator to.
Names=<user_name>[,<user_name>,...] — Names of coordinators.

NOTE: To list coordinators use the WithCoordinator options with list account or list user. That's the little shortcut, iykyk 🌸

🔥 SPECIFICATIONS FOR EVENTS 🔥

Events are automatically generated and sent to slurmdbd to be stored. These are options you can specify to filter for specific types of events. Event filter besties, assemble 🫶

All_Clusters — Shortcut to get information on all clusters.
All_Time — Shortcut to get time period for all time.
Clusters=<cluster_name>[,<cluster_name>,...] — List the events of the cluster(s). Default is the cluster where the command was run.
CondFlags=<flag>[,<flag>,...] — Optional list of flags to filter events by. Valid options are:
- Open — If set, only open node events (currently down) will be returned.
End=<OPT> — Period ending of events. Default is now. Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

Event=<OPT> — Specific types of events to look for. Valid options are Cluster or Node. The default is both.
MaxCPUs=<OPT> — Max number of CPUs affected by an event.
MinCPUs=<OPT> — Min number of CPUs affected by an event.
Nodes=<node_name>[,<node_name>,...] — Node names affected by an event.
Reason=<reason>[,<reason>,...] — Reason associated with a node going down. A reason that contains a space should be surrounded by quotes.
Start=<OPT> — Period start of events. Default is 00:00:00 of previous day, unless states are given with the States=<spec> events. If this is the case the default behavior is to return events currently in the states specified. Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

States=<state>[,<state>,...] — State of a node in a node event. If this is set, the event type is set automatically to Node.
User=<user_name>[,<user_name>,...] — Query against users who set the event. If this is set, the event type is set automatically to Node since only the slurm user can perform a cluster event.

🌈 LIST/SHOW EVENT FORMAT OPTIONS 🌈

Fields you can display when viewing Event records by using the format= option. The default format is:

Cluster,NodeName,TimeStart,TimeEnd,State,Reason,User

Cluster — The name of the cluster event happened on.
ClusterNodes — The hostlist of nodes on a cluster in a cluster event.
Duration — Time period the event was around for.
End — Period when event ended.
Event — Name of the event.
EventRaw — Numeric value of the name of the event.
NodeName — The node affected by the event. In a cluster event, this is blank.
Reason — The reason an event happened.
Start — Period when event started.
State — On a node event this is the formatted state of the node during the event.
StateRaw — On a node event this is the numeric value of the state of the node during the event.
TRES — Number of TRES involved with the event.
User — On a node event this is the user who caused the event to happen.

🦋 SPECIFICATIONS FOR FEDERATION 🦋

Federations can be created, modified, and deleted with sacctmgr. These options let you set the corresponding attributes or filter on them when querying for Federations. Federation main character energy, periodt ✨

Clusters[+|-]=<cluster_name>[,<cluster_name>,...] — List of clusters to add/remove to a federation. A blank value (e.g. clusters=) will remove all federations for the federation.

NOTE: A cluster can only be a member of one federation. One federation only, bestie — no side quests 💅

Name=<name> — The name of the federation.
Tree — Display federations in a hierarchical fashion.
WithDeleted — Display information with previously deleted data. Federations that are deleted within 24 hours of being created will be removed from the database. Federations that were created more than 24 hours prior to the deletion request are just marked as deleted and will be viewable with the WithDeleted flag.

⭐ LIST/SHOW FEDERATION FORMAT OPTIONS ⭐

Fields you can display when viewing Federation records by using the format= option. The default format is:

Federation,Cluster,Features,FedState

Cluster — Name of the cluster that is a member of the federation.
Features — The list of features on the cluster.
Federation — The name of the federation.
FedState — The state of the cluster in the federation.
FedStateRaw — Numeric value of the name of the FedState.
Index — The index of the cluster in the federation.

✨ SPECIFICATIONS FOR INSTANCES ✨

Okay so bestie, cloud node instance info gets sent over to slurmdbd and stored there. These are the options you can throw at it to filter down to the specific instances you actually care about 💖

Clusters=<cluster_name>[,<cluster_name>,...] — the name of the cluster that the instance ran on. Default is literally just the cluster where you ran the command.
End=<OPT> — period ending of instances. Default is now.

Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

Extra=<OPT> — arbitrary string associated with the node during the life of the instance.
InstanceId=<OPT> — cloud instance ID.
InstanceType=<OPT> — cloud instance type.
Nodes=<node_name>[,<node_name>,...] — the node on which the instance ran.
Start=<OPT> — period start of instances. Default is 00:00:00 of the previous day.

Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

🦋 LIST/SHOW INSTANCE FORMAT OPTIONS 🦋

These are the fields you can display when you're viewing Instance records with the format= option. The default format is:

Cluster,NodeName,Start,End,InstanceID,InstanceType,Extra

Cluster — name of the cluster that the instance ran on.
End — time when the instance ended.
Extra — arbitrary string associated with the node during the life of the instance.
InstanceId — cloud instance ID.
InstanceType — cloud instance type.
NodeName — the node on which the instance ran.
Start — time when the instance started.

🎀 SPECIFICATIONS FOR JOB 🎀

Job info gets sent to slurmdbd automatically to be stored, no cap. These are the options you can use to filter for specific jobs. And hiii, there are also some attributes you can actually modify on a job record 👀

AdminComment=<admin_comment> — arbitrary descriptive string. Only a Slurm administrator can modify it. To clear an existing value, set a new value of '' (two single quotes with nothing in between them).
Comment=<comment> — the job's comment string, used when the AccountingStoreFlags parameter in the slurm.conf file contains job_comment. A user can only modify the comment string of their own job. To clear an existing value, set a new value of '' (two single quotes with nothing in between them).
Cluster=<cluster_list> — list of clusters to alter jobs on, defaults to the local cluster.
DerivedExitCode=<derived_exit_code> — the derived exit code can be modified after a job completes, based on the user's judgment of whether the job succeeded or failed. The user can only modify the derived exit code of their own job.
EndTime — jobs must end before this time to be modified. Format output is YYYY-MM-DDTHH:MM:SS, unless changed through the SLURM_TIME_FORMAT environment variable.
Extra=<extra> — the job's extra string, used when the AccountingStoreFlags parameter in the slurm.conf file contains job_extra. The user can only modify the extra string of their own job. To clear an existing value, set a new value of '' (two single quotes with nothing in between them).
JobID=<jobid_list> — the id of the job to change. Not needed if you're altering multiple jobs using the wckey specification.
NewWCKey=<new_wckey> — use this to rename a wckey on job(s) in the accounting database.
StartTime — jobs must start at or after this time to be modified, in the same format as EndTime.
SystemComment=<system_comment> — arbitrary descriptive string, usually managed by the BurstBufferPlugin. Only a Slurm administrator can modify it. To clear an existing value, set a new value of '' (two single quotes with nothing in between them).
TRES=<tres_name=value> — use to set or modify a TRES on job(s) in the accounting database that have already completed.

⚠️ WARNING: This is permanent, the original value will be lost afterwards. Like, gone gone. 💀

User=<user_list> — used to specify the users' jobs to alter.
WCKey=<wckey_list> — used to specify the wckeys to alter.

The AdminComment, Comment, DerivedExitCode, Extra, SystemComment, and WCKey fields are the only fields of a job record in the database that can be modified after job completion. Periodt. 🔥

🌈 LIST/SHOW JOB FORMAT OPTIONS 🌈

The sacct command is the exclusive command to display job records from the Slurm database. Yeah, sacctmgr is not the one for this bestie, go see sacct 💅

👑 SPECIFICATIONS FOR QOS 👑

A QOS can be created, modified, and deleted with sacctmgr. These options let you set the corresponding attributes, or filter on them when you're querying for a QOS.

📌 NOTE: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job is being considered for being allocated resources. If starting a job would cause any of its group limit to be exceeded, that job will not be considered for scheduling — even if that job might preempt other jobs which would release sufficient group resources for the pending job to be initiated. The math is mathing, sorry babe 😭

📌 NOTE: All TRES options (GrpTRES, MaxTRES, etc.) can also use the same [+|-]= syntax that's available for association-based entities. Slay ✨

Description — an arbitrary string describing a QOS. Only a Slurm administrator can modify it.
Flags — used by the slurmctld to override or enforce certain characteristics. To add or remove individual flags, use the += or -= operators. To clear all existing flags, set a new value of -1. Valid options are:
- DenyOnLimit — if set, jobs using this QOS will be rejected at submission time if they don't conform to the QOS Max or Min limits as stand-alone jobs. Jobs that exceed these limits when other jobs are considered, but conform to the limits when considered individually, will not be rejected — instead they'll pend until resources are available. Group limits (e.g. GrpTRES) will also be treated like Max limits (e.g. MaxTRESPerNode) and jobs will be denied if they'd violate the limit as stand-alone jobs. This currently only applies to QOS and Association limits.
- EnforceUsageThreshold — if set, and the QOS also has a UsageThreshold, any jobs submitted with this QOS that fall below the UsageThreshold will be held until their Fairshare Usage goes above the Threshold.
- NoDecay — if set, this QOS will not have its GrpTRESMins, GrpWall and UsageRaw decayed by the slurm.conf PriorityDecayHalfLife or PriorityUsageResetPeriod settings. This allows a QOS to provide aggregate limits that, once consumed, will not be replenished automatically. Such a QOS will act as a time-limited quota of resources for an association that has access to it. Account/user usage will still be decayed for associations using the QOS. The QOS GrpTRESMins and GrpWall limits can be increased, or the QOS RawUsage value reset to 0 (zero), to again allow jobs submitted with this QOS to be queued (if DenyOnLimit is set) or run (pending with QOSGrp{TRES}MinutesLimit or QOSGrpWallLimit reasons, where {TRES} is some type of trackable resource).
- NoReserve — if set and backfill scheduling is used, jobs using this QOS will not reserve resources in the backfill schedule's map of resources allocated through time. This flag is intended for use with a QOS that may be preempted by jobs associated with all other QOS (e.g. use with a "standby" QOS). If this flag is used with a QOS which can not be preempted by all other QOS, it could result in starvation of larger jobs. Not the vibe 🙈
- OverPartQOS — if set, jobs using this QOS will be able to override any limits used by the requested partition's QOS limits.
- PartitionMaxNodes — if set, jobs using this QOS will be able to override the requested partition's MaxNodes limit.
- PartitionMinNodes — if set, jobs using this QOS will be able to override the requested partition's MinNodes limit.
- PartitionTimeLimit — if set, jobs using this QOS will be able to override the requested partition's TimeLimit.
- Relative — if set, the QOS limits will be treated as percentages of the cluster or partition instead of absolute limits (numbers should be less than 100). The controller should be restarted or reconfigured after adding the Relative flag to the QOS.

If this is used as a partition QOS:

Limits will be calculated relative to the partition's resources.
Only one partition may have this QOS as its partition QOS.
Jobs will not be allowed to use it as a normal QOS.

Additional details are in the QOS documentation at https://slurm.schedmd.com/qos.html.

RequiresReservation — if set, jobs using this QOS must designate a reservation when submitting a job. This option can be useful in restricting usage of a QOS that may have greater preemptive capability or additional resources to be allowed only within a reservation.
UsageFactorSafe — if set and AccountingStorageEnforce includes Safe, jobs will only be able to run if the job can run to completion with the UsageFactor applied.

GraceTime — preemption grace time in seconds to be extended to a job which has been selected for preemption. The default value is zero, meaning no preemption grace time is allowed on this QOS. This value is only applicable for QOS PreemptMode=CANCEL and PreemptMode=REQUEUE. 🫶

⚠️ NOTE: if a job gets picked for preemption and then just… exits before GraceTime runs out, it's handled per PreemptMode no matter why it exited (you'll notice this most if PreemptMode=REQUEUE). Slurm literally cannot tell the difference between an exit caused by the preemption signal and some totally unrelated exit that happens inside GraceTime. No cap.

`GrpJobs`

Max number of running jobs, in aggregate, for this QOS. Wanna wipe out an existing value? Set a new one of -1.

`GrpJobsAccrue`

Max number of pending jobs in aggregate that are allowed to accrue age priority for this QOS. Heads up: this limit only applies to the job's QOS and not the partition's QOS. To clear an existing value, set a new value of -1.

`GrpSubmit`, `GrpSubmitJobs`

Max number of jobs in a pending or running state, at any given moment, in aggregate for this QOS. Clearing an existing value? Set a new value of -1. 💅

`GrpTRES`

Max number of TRES that running jobs can be allocated in aggregate for this QOS.

Refer to the TRES information section below for further details.

`GrpTRESMins`

Max number of TRES minutes that could possibly be used by past, present, and future jobs with this QOS.

Refer to the TRES information section below for further details.

⚠️ NOTE: this limit only does anything when you're using the Priority Multifactor plugin. The time decays using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in slurm.conf. When this limit is hit, every associated job that's running gets killed 😭 and every future job submitted with this QOS gets delayed until it can run inside the limit.

`GrpTRESRunMins`

Max number of TRES minutes that can be allocated by running jobs with this QOS. It takes the time limit of running jobs into consideration and consumes it. If the limit is reached, no new jobs start until other jobs finish so time can free up.

Refer to the TRES information section below for further details.

`GrpWall`

Max wall clock time that can be allocated by running jobs in aggregate for this QOS. If this limit is reached, job submissions get denied and the running jobs get killed. GrpWall format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded in minutes, with rounding as needed. To clear an existing value, set a new value of -1.

⚠️ NOTE: again, this limit only applies when you're using the Priority Multifactor plugin. The time decays using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in slurm.conf. When this limit is reached, all associated jobs running get killed and all future jobs submitted with this QOS get delayed until they can run inside the limit.

`LimitFactor`

A float that gets factored into an association's [Grp|Max]TRES limits. Like, if the LimitFactor is 2, then an association with a GrpTRES of 30 CPUs would be allowed to allocate 60 CPUs when running under this QOS. To clear an existing value, set a new value of -1.

NOTE: this factor is only applied to associations running in this QOS, and it is not applied to any limits in the QOS itself. Read that twice bestie.

`MaxJobsAccruePA`, `MaxJobsAccruePerAccount`

Max number of pending jobs an account (or subacct) can have accruing age priority at any given time. This limit only applies to the job's QOS and not the partition's QOS. To clear an existing value, set a new value of -1.

`MaxJobsAccruePU`, `MaxJobsAccruePerUser`

Max number of pending jobs a user can have accruing age priority at any given time. This limit only applies to the job's QOS and not the partition's QOS. To clear an existing value, set a new value of -1.

`MaxJobsPA`, `MaxJobsPerAccount`

Max number of running jobs per account. To clear an existing value, set a new value of -1.

`MaxJobsPU`, `MaxJobsPerUser`

Max number of running jobs per user. To clear an existing value, set a new value of -1.

`MaxSubmitJobsPA`, `MaxSubmitJobsPerAccount`

Max number of jobs in a pending or running state per account. To clear an existing value, set a new value of -1.

`MaxSubmitJobsPU`, `MaxSubmitJobsPerUser`

Max number of jobs in a pending or running state per user. To clear an existing value, set a new value of -1.

`MaxTRES`, `MaxTRESPJ`, `MaxTRESPerJob`

Max number of TRES each job can use.

Refer to the TRES information section below for further details.

`MaxTRESMins`, `MaxTRESMinsPJ`, `MaxTRESMinsPerJob`

Max number of TRES minutes each job can use.

Refer to the TRES information section below for further details.

`MaxTRESPA`, `MaxTRESPerAccount`

Max number of TRES each account can use.

Refer to the TRES information section below for further details.

`MaxTRESPN`, `MaxTRESPerNode`

Max number of TRES each node in a job allocation can use.

Refer to the TRES information section below for further details.

`MaxTRESPU`, `MaxTRESPerUser`

Max number of TRES each user can use.

Refer to the TRES information section below for further details.

`MaxTRESRunMinsPA`, `MaxTRESRunMinsPerAccount`

Max number of TRES minutes each account can use. This takes the time limit of running jobs into consideration. If the limit is reached, no new jobs are started until other jobs finish so time can free up.

Refer to the TRES information section below for further details.

`MaxTRESRunMinsPU`, `MaxTRESRunMinsPerUser`

Max number of TRES minutes each user can use. This takes the time limit of running jobs into consideration. If the limit is reached, no new jobs are started until other jobs finish so time can free up.

Refer to the TRES information section below for further details.

`MaxWall`, `MaxWallDurationPerJob`

Max wall clock time each job can use. MaxWall format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded in minutes, with rounding as needed. To clear an existing value, set a new value of -1.

`MinPrioThreshold`

Minimum priority required to reserve resources when scheduling. To clear an existing value, set a new value of -1.

`MinTRES`, `MinTRESPerJob`

Minimum number of TRES each job running under this QOS must request. Otherwise the job just pends until it gets modified. 🫠

Refer to the TRES information section below for further details.

`Name`

Name of the QOS. Needed for creation, so like… don't skip it.

`Preempt`

Other QOSs that this QOS can preempt. To clear an existing value, set a new value of '' (two single quotes with nothing between them).

⚠️ NOTE: the Priority of a QOS is NOT related to QOS preemption — only Preempt is used to define which QOS can preempt others. Don't mix them up, iykyk.

`PreemptExemptTime`

Specifies a minimum run time for jobs in this QOS before they're even considered for preemption. This QOS option takes precedence over the global PreemptExemptTime. Setting it to -1 disables the option, letting another QOS or the global option take effect. Setting it to 0 indicates no minimum run time and supersedes the lower priority QOS (see OverPartQOS) and/or the global option in slurm.conf.

`PreemptMode`

The mechanism used to preempt jobs running in this QOS when the cluster's PreemptType is set to preempt/qos. This QOS-specific PreemptMode overrides the cluster-wide PreemptMode for this QOS. See https://slurm.schedmd.com/preempt.html for more details. ✨

⚠️ NOTE: for performance reasons, the backfill scheduler reserves whole nodes for jobs, not partial nodes. So if during backfill scheduling a job preempts one or more other jobs, the whole nodes for those preempted jobs are reserved for the preemptor job — even if the preemptor job requested fewer resources than that. Those reserved nodes aren't available to other jobs during that backfill cycle, even if the other jobs would've fit on them. Which means jobs may preempt more resources during a single backfill iteration than they actually requested. Brutal, I know. 💀

⚠️ NOTE: for a heterogeneous job to be considered for preemption, all components must be eligible for preemption. When a heterogeneous job is going to be preempted, the first identified component of the job with the highest order PreemptMode (SUSPEND (highest), REQUEUE, CANCEL (lowest)) gets used to set the PreemptMode for all components. The GraceTime and user warning signal for each component of the heterogeneous job stay unique. Heterogeneous jobs are excluded from GANG scheduling operations.

CLUSTER — is the default value and uses the cluster-wide PreemptMode for job preemption. A value of "OFF" or "" (empty string) is equivalent. To keep jobs in this QOS from being preempted, make sure this QOS isn't listed in the Preempt field of any other QOS. 👑
CANCEL — the preempted job gets cancelled. periodt.
REQUEUE — preempts jobs by requeuing them (if possible) or cancelling them. For jobs to actually get requeued they must have the --requeue sbatch option set, or the cluster wide JobRequeue parameter in slurm.conf must be set to 1.
SUSPEND — the preempted jobs get suspended, and later the Gang scheduler resumes them 🫶. Which means: <purple>SUSPEND</purple> preemption mode always needs the GANG option set at the cluster level, no exceptions. Also, because suspended jobs are still holding onto memory on the allocated nodes, Slurm has to be able to track memory resources to be able to suspend jobs. If PreemptType=preempt/qos is configured and the preempted job(s) and the preemptor job are on the same partition, they'll share resources with the Gang scheduler (time-slicing ✨). If they're not (i.e. the preemptees and preemptor are on different partitions), then the preempted jobs stay suspended until the preemptor ends.

NOTE: suspended jobs will not release GRES 😭. Higher priority jobs will not be able to preempt to gain access to GRES. Lowkey tragic but that's the rule.

PRIORITY — allow preemption only if the preemptor's job priority is higher than the preemptee's job priority 👑.
WITHIN — allows for preemption between jobs sharing the same qos 🌀. By default, PreemptType=preempt/qos will only consider jobs eligible for preemption if they do not share the same qos value.

Priority

QOS priority factor used by the priority/multifactor plugin. Unset by default, which means no extra priority is granted, bestie, none, zero, nada 💅.

NOTE: the Priority of a QOS is NOT related to QOS preemption — for that, see Preempt instead. Don't mix them up, iykyk 🙈.

`RawUsage=<value>`

This one lets an administrator set the raw usage accrued to a QOS. Specifying a value of 0 (zero) will reset the raw usage 🧼. Heads up: this is a settable specification only — it cannot be used as a filter to list accounts. So don't try it, she will not answer.

UsageFactor

A float that gets factored into a job's TRES usage (e.g. RawUsage, TRESMins, TRESRunMins). So like, if the usagefactor was 2, for every TRESBillingUnit second a job ran it would count for 2 🔥. If the usagefactor was .5, every second would only count for half of the time. A setting of 0 would add no timed usage from the job.

The usage factor only applies to the job's QOS and not the partition QOS. Periodt.

If the UsageFactorSafe flag is set and AccountingStorageEnforce includes Safe, jobs will only be started if they can run to completion with the UsageFactor applied, and won't be killed due to limits ✅.
If the UsageFactorSafe flag is not set and AccountingStorageEnforce includes Safe, jobs will be started if they can run to completion without the UsageFactor applied, and won't be killed due to limits.
If the UsageFactorSafe flag is not set and AccountingStorageEnforce does not include Safe, jobs will be scheduled as long as the limits are not reached, but could be killed due to limits 💀.

See AccountingStorageEnforce in the slurm.conf man page for the full tea 🍵.

NOTE: a job's TRESRunMins usage is booked using the QOS UsageFactor in effect when the job starts, and is given back with that same factor when the job ends. Modifying a QOS's UsageFactor does not rescale the TRESRunMins usage of jobs that are already running; RawUsage and GrpWall continue to accrue with the current factor. However, when slurmctld is restarted or reconfigured, the usage of all running jobs is booked again with the UsageFactor in effect at that time. So yeah, restarts have consequences 🌸.

Default is 1. To clear an existing value, set a new value of -1.

UsageThreshold

A float representing the lowest fairshare of an association allowed to run a job. If an association falls below this threshold and has pending jobs or submits new jobs, those jobs will be held until the usage goes back above the threshold 🛑. Use sshare to see current shares on the system. To clear an existing value, set a new value of -1.

🦋 LIST/SHOW QOS FORMAT OPTIONS 🦋

Fields you can display when viewing QOS records by using the format= option. Pick your faves, build your little display, slay ✨.

Description

An arbitrary string describing a QOS.

Flags

Used by the slurmctld to override or enforce certain characteristics.

GraceTime

Preemption grace time extended to a job which has been selected for preemption, in the format of hh:mm:ss.

GrpJobs

Maximum number of running jobs in aggregate for this QOS.

GrpJobsAccrue

Maximum number of pending jobs in aggregate able to accrue age priority for this QOS. This limit only applies to the job's QOS and not the partition's QOS 💅.

GrpSubmit

GrpSubmitJobs

Maximum number of jobs in a pending or running state at any time in aggregate for this QOS.

GrpTRES

Maximum number of TRES able to be allocated by running jobs in aggregate for this QOS.

GrpTRESMins

Maximum number of TRES minutes that can possibly be used by past, present, and future jobs with this QOS ⏳.

GrpTRESRunMins

Maximum number of TRES minutes able to be allocated by running jobs with this QOS.

GrpWall

Maximum wall clock time able to be allocated by running jobs in aggregate for this QOS.

ID

The id of the QOS.

LimitFactor

A float that is factored into an association's [Grp|Max]TRES limits.

MaxJobsAccruePA

MaxJobsAccruePerAccount

Maximum number of pending jobs an account (or subacct) can have accruing age priority at any given time. This limit only applies to the job's QOS and not the partition's QOS.

MaxJobsAccruePU

MaxJobsAccruePerUser

Maximum number of pending jobs a user can have accruing age priority at any given time. This limit only applies to the job's QOS and not the partition's QOS.

MaxJobsPA

MaxJobsPerAccount

Maximum number of running jobs per account.

MaxJobsPU

MaxJobsPerUser

Maximum number of running jobs per user.

MaxSubmitJobsPA

MaxSubmitJobsPerAccount

Maximum number of jobs in a pending or running state per account.

MaxSubmitJobsPU

MaxSubmitJobsPerUser

Maximum number of jobs in a pending or running state per user.

MaxTRES

MaxTRESPJ

MaxTRESPerJob

Maximum number of TRES each job can use.

MaxTRESMins

MaxTRESMinsPJ

MaxTRESMinsPerJob

Maximum number of TRES minutes each job can use.

MaxTRESPA

MaxTRESPerAccount

Maximum number of TRES each account can use.

MaxTRESPN

MaxTRESPerNode

Maximum number of TRES each node in a job allocation can use.

MaxTRESPU

MaxTRESPerUser

Maximum number of TRES each user can use.

MaxTRESRunMinsPA

MaxTRESRunMinsPerAccount

Maximum number of TRES minutes each account can use.

MaxTRESRunMinsPU

MaxTRESRunMinsPerUser

Maximum number of TRES minutes each user can use.

MaxWall

MaxWallDurationPerJob

Maximum wall clock time each job can use.

MinPrioThreshold

Minimum priority required to reserve resources when scheduling.

MinTRES

Minimum number of TRES each job running under this QOS must request. Otherwise the job will pend until modified 😴.

Name

Name of the QOS.

Preempt

Other QOSs this QOS can preempt.

PreemptExemptTime

Specifies a minimum run time for jobs in this QOS before they are considered for preemption ⏱️.

PreemptMode

Mechanism used to preempt jobs or enable gang scheduling for this QOS when the cluster's PreemptType is set to preempt/qos. The default preemption mechanism is specified by the cluster-wide PreemptMode configuration parameter.

Priority

QOS priority factor to be used by the priority/multifactor plugin.

UsageFactor

A float that is factored into a job's TRES usage (e.g. RawUsage, TRESMins, TRESRunMins).

UsageThreshold

A float representing the lowest fairshare of an association allowed to run a job.

WithDeleted

Display information with previously deleted data. A QOS that is deleted within 24 hours of being created and did not have a job run in the QOS during that time will be removed from the database 🗑️. Otherwise, the QOS will be marked as deleted and will be viewable with the WithDeleted flag.

🎀 SPECIFICATIONS FOR RESERVATIONS 🎀

Reservations are created with the scontrol command, and info about the reservations gets sent to slurmdbd to be stored 🗄️. These are options you can specify to filter for specific reservations.

`Clusters=<cluster_name>[,<cluster_name>,...]`

List the reservations of the cluster(s). Default is the cluster where the command was run.

`End=<OPT>`

Period ending of reservations. Default is now.

Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

`ID=<OPT>`

Comma-separated list of reservation ids.

`Names=<OPT>`

Comma-separated list of reservation names.

`Nodes=<node_name>[,<node_name>,...]`

Node names where reservation ran.

`Start=<OPT>`

Period start of reservations. Default is 00:00:00 of previous day.

Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

✨ LIST/SHOW RESERVATION FORMAT OPTIONS ✨

okay so these are the fields you can display when you're viewing Reservation records and you use the format= option. the default format is literally:

Cluster,Name,TRES,Start,End,UnusedWall

Associations — the id's of the associations able to run in the reservation 💖
Cluster — name of cluster reservation was on.
End — end time of reservation.
Flags — flags set on the reservation.
ID — reservation ID.
Name — name of this reservation.
NodeNames — list of nodes in the reservation.
Start — start time of reservation.
TRES — list of TRES in the reservation.
UnusedWall — wall clock time in seconds unused by any job. A job's allocated usage is its run time multiplied by the ratio of its CPUs to the total number of CPUs in the reservation. For example, a job using all the CPUs in the reservation running for 1 minute would reduce unused_wall by 1 minute. 🫶

💅 SPECIFICATIONS FOR RESOURCE 💅

Resources can be created, modified, and deleted with sacctmgr. These options let you set the corresponding attributes or filter on them when you're querying for Resources. it's giving "full control of the resource shelf" energy ✨

LastConsumed=<OPT> — number of software resources of a specific name consumed out of Count on the system being controlled by a resource manager.
Clusters=<name_list> — comma-separated list of cluster names on which specified resources are to be available. If no names are designated then the clusters already allowed to use this resource will be altered.
Count=<OPT> — number of software resources of a specific name configured on the system being controlled by a resource manager.
Descriptions= — a brief description of the resource.
Flags[-|+]=<OPT> — flags that identify specific attributes of the system resource. Valid options are:
- Absolute — if set the resource will treat the counts for Allowed and Allocated as absolute counts instead of percentages.

NOTE: If removing this with flags-=absolute there is no effort to convert the numbers in the database back to percentages. This is required by the user. 😭 so yeah, that cleanup is on you, bestie.

SharedPool — allows each cluster to access the full resource pool without requiring a value to be exclusively allocated to each cluster. Requires each cluster's Allowed to be 0 or the full Count/100% (with/without Absolute).

Names=<OPT> — comma-separated list of the name of a resource configured on the system being controlled by a resource manager. If this resource is seen on the slurmctld its name will be name@server to distinguish it from local resources defined in a slurm.conf. Resource names are case-insensitive (forced to lowercase) unless the Parameters=PreserveCaseResource option has been set in slurmdbd.conf.
Allowed=<allowed> — percentage/count of a specific resource that can be used on specified cluster.
Server=<OPT> — arbitrary string indicating the name of the server serving up the resource. Default is 'slurmdb' indicating the licenses are being served by the database. This parameter is only for tagging purposes.
ServerType=<OPT> — arbitrary string used to tag the type of the software resource manager providing the licenses. For example FlexNext Publisher Flexlm license server or Reprise License Manager RLM. This does not imply any kind of integration with license managers.
Type=<OPT> — the type of the resource represented by this record. Currently the only valid type is License.
WithClusters — display the clusters percentage/count of resources. If a resource hasn't been given to a cluster the resource will not be displayed with this flag.
WithDeleted — display information with previously deleted data. Resources that are deleted within 24 hours of being created will be removed from the database. Resources that were created more than 24 hours prior to the deletion request are just marked as deleted and will be viewable with the WithDeleted flag.

NOTE: Resource is used to define each resource configured on a system available for usage by Slurm clusters. 🌈

🌸 LIST/SHOW RESOURCE FORMAT OPTIONS 🌸

these are the fields you can display when you're viewing Resource records and you use the format= option. the default format is:

Name,Server,Type,Count,LastConsumed,Allocated,ServerType,Flags

Allocated — the percent/count of licenses allocated to a cluster.
LastConsumed — the count of a specific resource consumed out of Count on the system globally.
Cluster — name of cluster resource is given to.
Count — the count of a specific resource configured on the system globally.
Description — description of the resource.
Name — name of this resource.
Server — server serving up the resource.
ServerType — the type of the server controlling the licenses.
Type — type of resource this record represents. 💖

🦋 SPECIFICATIONS FOR RUNAWAYJOB 🦋

under certain circumstances, jobs can complete without having that completion recorded by slurmdbd. This results in a "runaway job", where slurmdbd is not going to record a completion time for that job without intervention. This command will identify jobs that are in this state and offer to have slurmdbd clean up the job record(s). no cap, it's the "where did that completion go" detector 👀

This particular variant of the "show" command also permits the use of the "set" keyword to define the following specifications:

EndState=<state> — desired state to use as the end state for fixed jobs. Supported states are: Completed, Failed.

🎀 LIST/SHOW RUNAWAYJOB FORMAT OPTIONS 🎀

fields you can display when viewing runaway job records by using the format= option. The default format is:

ID,Name,Partition,Cluster,State,TimeSubmit,TimeStart,TimeEnd

Cluster — name of cluster job ran on.
ID — id of the job.
Name — name of the job.
Partition — partition job ran on.
State — current State of the job in the database.
TimeEnd — current recorded time of the end of the job.
TimeStart — time job started running.
TimeSubmit — time job was submitted. ✨

💜 SPECIFICATIONS FOR TRANSACTIONS 💜

information about changes to clusters, resources, accounts, associations, etc., are recorded as transactions by slurmdbd. These are options you can specify to filter for specific transactions. it's the receipts folder, basically 🧾

Accounts=<account_name>[,<account_name>,...] — only print out the transactions affecting specified accounts.
Action=<Specific_action_the_list_will_display> — only display transactions of the specified action type.
Actor=<Specific_name_the_list_will_display> — only display transactions done by a certain person.
Clusters=<cluster_name>[,<cluster_name>,...] — only print out the transactions affecting specified clusters.
End=<Date_and_time_of_last_transaction_to_return> — return all transactions before this Date and time. Default is now.
Start=<Date_and_time_of_first_transaction_to_return> — return all transactions after this Date and time. Default is epoch.

Valid time formats for End and Start are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

Users=<user_name>[,<user_name>,...] — only print out the transactions affecting specified users.
WithAssoc — get information about which associations were affected by the transactions.

🌟 LIST/SHOW TRANSACTIONS FORMAT OPTIONS 🌟

fields you can display when viewing Transaction records by using the format= option. The default format is:

Time,Action,Actor,Where,Info

Action — displays the type of Action that took place.
Actor — displays the Actor to generate a transaction.
Info — displays details of the transaction.
TimeStamp — displays when the transaction occurred.
Where — displays details of the constraints for the transaction.

NOTE: If using the WithAssoc option you can also view the information about the various associations the transaction affected. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section. 💅

💖 SPECIFICATIONS FOR USERS 💖

okay so bestie, sacctmgr can create, modify, and delete Users — iconic. These options let you set the matching attributes or filter on them when you're querying for Users. ✨

It is super important to recognize the difference between a User and an Association. There is a User entity that exists for each unique username. However, there can be multiple User Associations for the same User. The combination of a Cluster, Account, User, and optionally a Partition constitute a User Association. When adding an existing User to another Account, you are creating an additional User Association rather than modifying an existing User. 🫶

Account=<account> — Account name to add this user to. 💳
AdminLevel=<level> — Admin level of user. Valid levels are None, Operator, and Admin. 👑
Cluster=<cluster> — Specific cluster to add user to the account on. Default is all in system. 🌐
DefaultAccount=<account> — Identify the default account name to be used for a job if none is specified at submission time. 🎀
DefaultWCKey=<defaultwckey> — Identify the default Workload Characterization Key. 🔑
Name=<name> — Name of user. 🏷️
NewName=<newname> — Use to rename a user in the accounting database. ✏️
Partition=<name> — Partition name. 🧩

NOTE: See also Partitions listed in the SPECIFICATIONS FOR ASSOCIATIONS section.

RawUsage=<value> — This allows an administrator to reset the raw usage accrued to a user. The only value currently supported is 0 (zero). This is a settable specification only — it cannot be used as a filter to list users. ⚠️
WCKeys=<wckeys> — Workload Characterization Key values. 🔑
WithAssoc — Display all associations for this user. 🔗
WithCoord — Display all accounts a user is coordinator for. 👯
WithDeleted — Display information with previously deleted data. Users that are deleted within 24 hours of being created and did not have a job run by the user during that time will be removed from the database. Otherwise, the user will be marked as deleted and will be viewable with the WithDeleted flag. 💀

NOTE: If using the WithAssoc option you can also query against association specific information to view only certain associations this user may have. These extra options can be found in the SPECIFICATIONS FOR ASSOCIATIONS section. You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

🎀 LIST/SHOW USER FORMAT OPTIONS 🎀

Fields you can display when viewing User records by using the format= option. The default format is:

User,DefaultAccount,DefaultWCKey,AdminLevel

AdminLevel — Admin level of user. 👑
Coordinators — List of users that are a coordinator of the account. (Only filled in when using the WithCoordinator option.) 👯
DefaultAccount — The user's default account. 🎀
DefaultWCKey — The user's default wckey. 🔑
User — The name of a user. 🏷️

NOTE: If using the WithAssoc option you can also view the information about the various associations the user may have on all the clusters in the system. The association information can be filtered. Note that all the users in the database will always be shown as filter only takes effect over the association data. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

🌸 LIST/SHOW WCKey 🌸

Fields you can display when viewing WCKey records by using the format= option. The default format is:

WCKey,Cluster,User

Cluster — Specific cluster for the WCKey. 🌐
ID — The ID of the WCKey. 🆔
User — The name of a user for the WCKey. 🏷️
WCKey — Workload Characterization Key. 🔑
WithDeleted — Display information with previously deleted data. WCKeys that are deleted within 24 hours of being created and did not have a job run with the WCKey during that time will be removed from the database. Otherwise, the WCKey will be marked as deleted and will be viewable with the WithDeleted flag. 💀

🦋 LIST/SHOW TRES 🦋

Fields you can display when viewing TRES records by using the format= option. The default format is:

Type,Name,ID

ID — The identification number of the trackable resource as it appears in the database. 🆔
Name — The name of the trackable resource. This option is required for TRES types BB (Burst buffer), GRES, and License. Types CPU, Energy, Memory, and Node do not have Names. For example if GRES is the type then name is the denomination of the GRES itself e.g. GPU. 🏷️
Type — The type of the trackable resource. Current types are BB (Burst buffer), CPU, Energy, GRES, License, Memory, and Node. 🧬

✨ TRES information ✨

Trackable RESources (TRES) are used in many QOS or Association limits (GrpTRES* / MaxTRES*). When setting limits, one or more comma-separated "TRES=value" assignments can be specified. 🫶

Each TRES can be one of the Slurm defaults (e.g., cpu, mem, node), or any defined generic resource. You can see the list of available resources by running sacctmgr show tres. 👑

Each TRES has a different limit, e.g., "GrpTRES=cpu=16,mem=32G" would establish 2 different limits: one for 16 CPUs and another for 32 GiB of memory.

Setting a new value for a TRES type does not affect any other TRES types that were previously specified. For example, on the same entity as above, a subsequent assignment of "GrpTRES=gres/gpu=2" will result in "GrpTRES=cpu=16,mem=32G,gres/gpu=2" applying to that entity.

To remove a limit, assign -1 to the TRES type whose limit should be removed, e.g., "GrpTRES=cpu=-1" would remove only the cpu TRES limit, resulting in "GrpTRES=mem=32G,gres/gpu=2" on that entity.

Starting in Slurm 26.05, a TRES assignment can use a += or -= operator to add to or remove from the previously set value for that TRES, e.g., "GrpTRES=mem-=8G,gres/gpu+=2" would result in "GrpTRES=mem=24G,gres/gpu=4" on that entity. 🔥

NOTE: When dealing with Memory as a TRES the default units are MiB. A suffix of G, T, or P can be added to specify a limit in larger units.

NOTE: The Billing TRES is calculated from a partition's TRESBillingWeights. It is temporarily calculated during scheduling for each partition to enforce billing TRES limits. The final Billing TRES value is calculated after the job has been allocated resources and can be seen in scontrol show jobs and sacct output.

🌈 GLOBAL FORMAT OPTION 🌈

When using the format option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed. 🎀

e.g. format=name%30 will print 30 characters of field name right justified. A -30 will print 30 characters left justified.

📦 FLAT FILE DUMP AND LOAD 📦

sacctmgr has the capability to load and dump Slurm association data to and from a file. This method can easily add a new cluster or copy an existing cluster's associations into a new cluster with similar accounts. Each file contains Slurm association data for a single cluster. Beginning with version 25.05, QOS information is included in the dump file. Comments can be put into the file with the # character. Each line of information must begin with one of the five titles; QOS, Cluster, Parent, Account or User. Following the title is a space, dash, space, entity value, then specifications. Specifications are colon-separated. If any variable, such as an Organization name, has a space in it, surround the name with single or double quotes. 💅

sacctmgr dump/load must be run as a Slurm administrator or root. If using sacctmgr load on a database without any associations, it must be run as root (because there aren't any users in the database yet). 😭

🧚 dump 🧚

Dump cluster associations from the database into a file. If no file is given then one will be generated, using the cluster name for the file name. That file will be created in the current working directory.

To create a file with the association information you can run:

sacctmgr dump tux file=tux.cfg

Cluster= — Specify the cluster to dump the information for. 🌐
File= — Specify a file to save flat file data to. If the filename is not specified it uses clustername.cfg filename by default. 📄

🍓 load 🍓

Load cluster associations into the database. The imported associations will be reconciled with existing ones.

To load a previously created file you can run:

sacctmgr load file=tux.cfg

clean — Delete what was already there and start from scratch with this information. With no options this will only remove the cluster along with its associations. No accounts, users, or QOS will be removed. This also accepts a comma-separated list of other options to remove. Those include 'account', 'qos' and 'user'. If you would like to remove accounts, qos, and users along with the cluster and associations give the input of clean=account,qos,user. 🧹
Cluster= — Specify a different name for the cluster than that which is in the file. 🌐
File= — Specify a flat file to load from. 📄

🌸 SPECIFICATIONS FOR FLAT FILE 🌸

okay so bestie, since associations in the system follow a hierarchy, the flat file does too. Anything that is a parent needs to be defined before any children. The only exception is the understood root account — it’s always a default for any cluster and does not need to be defined. periodt 💅

To edit/create a file, start with a cluster line for the new cluster:

Cluster - cluster_name:MaxTRESPerJob=node=15

Anything included on this line will be the default for all associations on this cluster. The options for the cluster are:

FairShare= — Allocated shares used for fairshare calculation.
GrpJobs= — Maximum number of running jobs in aggregate for this association and its children.
GrpJobsAccrue= — Maximum number of pending jobs in aggregate able to accrue age priority for this association and its children.
GrpNodes= — This option has been deprecated in favor of the more versatile TRES. Equivalent limit definition is now GrpTRES=node=#.
GrpSubmitJobs= — Maximum number of jobs in a pending or running state at any time in aggregate for this association and its children.
GrpTRES= — Maximum number of TRES able to be allocated by running jobs in aggregate for this association and its children.
GrpTRESMins= — Maximum number of TRES minutes that can possibly be used by past, present and future jobs in this association and its children.
GrpTRESRunMins= — Maximum number of TRES minutes able to be allocated by running jobs in this association and its children. This takes into consideration time limit of running jobs and consumes it. If the limit is reached no new jobs are started until other jobs finish to allow time to free up.
GrpWall= — Maximum wall clock time able to be allocated by running jobs in aggregate in this association and its children.
MaxJobs= — Maximum number of running jobs per user in this association.
MaxTRESPerJob= — Maximum number of TRES each job can use in this association.
MaxWallDurationPerJob= — Maximum wall clock time each job can use in this association.
QOS= — Comma-separated list of Quality of Service names (Defined in sacctmgr).

After the entry for the root account you will have entries for the other accounts on the system. The entries will look similar to this example:

Parent - root
Account - cs:MaxTRESPerJob=node=5:MaxJobs=4:FairShare=399:MaxWallDurationPerJob=40:Description='Computer Science':Organization='LC'
Parent - cs
Account - test:MaxTRESPerJob=node=1:MaxJobs=1:FairShare=1:MaxWallDurationPerJob=1:Description='Test Account':Organization='Test'

Any of the options after a : can be left out, and they can be in any order. If you want to add any sub accounts, just list the Parent THAT HAS ALREADY BEEN CREATED before the account you are adding. no cap, order matters here 🫶

Account options are:

Description= — A brief description of the account.
FairShare= — Number used in conjunction with other associations to determine job priority.
GrpTRES= — Maximum number of TRES able to be allocated by running jobs in aggregate for this association and its children.
GrpTRESMins= — Maximum number of TRES minutes that can possibly be used by past, present, and future jobs in this association and its children.
GrpTRESRunMins= — Maximum number of TRES minutes able to be allocated by running jobs in this association and its children. This takes into consideration time limit of running jobs and consumes it. If the limit is reached no new jobs are started until other jobs finish to allow time to free up.
GrpJobs= — Maximum number of running jobs in aggregate for this association and its children.
GrpJobsAccrue= — Maximum number of pending jobs in aggregate able to accrue age priority for this association and its children.
GrpNodes= — This option has been deprecated in favor of the more versatile TRES. Equivalent limit definition is now GrpTRES=node=#.
GrpSubmitJobs= — Maximum number of jobs in a pending or running state at any time in aggregate for this association and its children.
GrpWall= — Maximum wall clock time able to be allocated by running jobs in aggregate in this association and its children.
MaxJobs= — Maximum number of running jobs per user in this association.
MaxNodesPerJob= — Maximum number of nodes per job in this association.
MaxWallDurationPerJob= — Maximum wall clock time each job can use in this association.
Organization= — Name of organization that owns this account.
QOS(=,+=,-=) — Comma-separated list of Quality of Service names (Defined in sacctmgr).

To add users to an account, add a line after the Parent line, similar to this:

Parent - test
User - adam:MaxTRESPerJob=node:2:MaxJobs=3:FairShare=1:MaxWallDurationPerJob=1:AdminLevel=Operator:Coordinator='test'

User options are:

AdminLevel= — Type of admin this user is (Administrator, Operator). Must be defined on the first occurrence of the user.
Coordinator= — Comma-separated list of accounts this user is coordinator over. Must be defined on the first occurrence of the user.
DefaultAccount= — System wide default account name. Must be defined on the first occurrence of the user.
FairShare= — Number used in conjunction with other associations to determine job priority.
MaxJobs= — Maximum number of running jobs from this user.
MaxTRESPerJob= — Maximum number of TRES each job from this user can use.
MaxWallDurationPerJob= — Maximum wall clock time each job from this user can use.
QOS(=,+=,-=) — Comma-separated list of Quality of Service names (Defined in sacctmgr).

🌸 ARCHIVE FUNCTIONALITY 🌸

Okay so sacctmgr can archive stuff to a flatfile and then load that data back later if you need it. But bestie, hear me: archiving is usually handled by slurmdbd, and it is highly recommended that you only do it through sacctmgr if you completely understand what you are doing. For the slurmdbd side of things, go see man slurmdbd, fr fr.

You can load data into the database from these files to either view old data or regenerate rolled-up data. For info about configuring an archive server, see https://slurm.schedmd.com/accounting.html#archive.

🗄️ archive dump

Dump accounting data to file. Data will not be archived unless the corresponding purge option is included in this command or in slurmdbd.conf. This operation cannot be rolled back once executed — like, no undo button, periodt. If one of the following options is not specified when sacctmgr is called, the value configured in slurmdbd.conf is used.

Directory= — Directory to store the archive data.
Events — When purging events, also archive them. Default is to permanently delete purged records.
Jobs — When purging jobs, also archive them, including scripts and environments. Default is to permanently delete purged records.
JobScript — When purging job scripts, also archive them. Default is to follow Jobs.
JobEnv — When purging job environments, also archive them. Default is to follow Jobs.
PurgeEventAfter= — Purge cluster event records older than the stated time. Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeJobAfter= — Purge individual job records older than the stated time (based on end time). Aggregated information will be preserved to "PurgeUsageAfter". Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.

NOTE: this includes job scripts and job environments if their specific purge options (PurgeJobScriptAfter and PurgeJobEnvAfter) are not set.

PurgeJobScriptAfter= — Purge job script records older than the stated time (based on end time). Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeJobEnvAfter= — Purge job environment records older than the stated time (based on end time). Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeResvAfter= — Purge reservation records older than the stated time (based on end time). Aggregated information will be preserved to "PurgeUsageAfter". Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeStepAfter= — Purge step records older than the stated time (based on end time). Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeSuspendAfter= — Purge job suspend records older than the stated time. Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeTXNAfter= — Purge transaction records older than the stated time. Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
PurgeUsageAfter= — Purge usage records (Cluster, Association, QOS and WCKey) older than the stated time (based on creation or last modified time). These tables are the source for reports generated by the sreport(1) command. Units are months (default), days, or hours. For example, a value of "12hours" would purge all such records older than 12 hours.
Reservations — When purging reservations, also archive them. Default is to permanently delete purged records.
Script= — Run this script instead of the generic form of archive to flat files.
Steps — When purging steps, also archive them. Default is to permanently delete purged records.
Suspend — When purging suspend records, also archive them. Default is to permanently delete purged records.
TXN — When purging transaction records, also archive them. Default is to permanently delete purged records.
Usage — When purging usage records, also archive them. Default is to permanently delete purged records.

🧚 archive load

Load previously archived data into the database. The archive file will not be loaded if the records already exist in the database — therefore, trying to load an archive file more than once will result in an error. When this data is again archived and purged from the database, if the old archive file is still in the directory ArchiveDir, a new archive file will be created (see ArchiveDir in the slurmdbd.conf man page), so the old file will not be overwritten and these files will have duplicate records. Iconic little chaos goblin behavior, ngl.

Archive files from the current or any prior Slurm release may be loaded through archive load.

File= — File to load into database. The specified file must exist on the slurmdbd host, which is not necessarily the machine running the command.
Insert= — SQL to insert directly into the database. This should be used very cautiously since this is writing your sql into the database.

🚀 PERFORMANCE

Executing sacctmgr sends a remote procedure call to slurmdbd. If enough calls from sacctmgr or other Slurm client commands that send remote procedure calls to the slurmdbd daemon come in at once, it can result in a degradation of performance of the slurmdbd daemon, possibly resulting in a denial of service. 💀

Do not run sacctmgr or other Slurm client commands that send remote procedure calls to slurmdbd from loops in shell scripts or other programs. Ensure that programs limit calls to sacctmgr to the minimum necessary for the information you are trying to gather. Like, be normal about it, bestie.

🌈 ENVIRONMENT VARIABLES

Some sacctmgr options may be set via environment variables. These environment variables, along with their corresponding options, are listed below. (Note: command line options will always override these settings.)

SLURM_CONF — The location of the Slurm configuration file.
SLURM_DEBUG_FLAGS — Specify debug flags for sacctmgr to use. See DebugFlags in the slurm.conf(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf.
SLURM_JSON — Control JSON serialization:
- compact — Output JSON as compact as possible.
- pretty — Output JSON in pretty format to make it more readable.
SLURM_YAML — Control YAML serialization:
- compact — Output YAML as compact as possible.
- pretty — Output YAML in pretty format to make it more readable.

💖 EXAMPLES 💖

Okay bestie, gather round — this section is giving tutorial era and it's lowkey the most important vibe check in the whole page. 🫶

NOTE: There is an order to set up accounting associations. You must define clusters before you add accounts and you must add accounts before you can add users.

Here's the sacred lineup, no cap:

$ sacctmgr create cluster tux
$ sacctmgr create account name=science fairshare=50
$ sacctmgr create account name=chemistry parent=science fairshare=30
$ sacctmgr create account name=physics parent=science fairshare=20
$ sacctmgr create user name=adam cluster=tux account=physics fairshare=10
$ sacctmgr delete user name=adam cluster=tux account=physics
$ sacctmgr delete user name=adam cluster=tux account=science partition=\"\"
$ sacctmgr delete account name=physics cluster=tux
$ sacctmgr modify user where name=adam cluster=tux account=physics set maxjobs=2 maxwall=30:00
$ sacctmgr add user brian account=chemistry
$ sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd
$ sacctmgr list transactions Action="Add Users" Start=11/03-10:30:00 format=Where,Time
$ sacctmgr dump cluster=tux file=tux_data_file
$ sacctmgr load tux_data_file

A user's account can not be changed directly. A new association needs to be created for the user with the new account. Then the association with the old account can be deleted.

When modifying an object placing the key words 'set' and the optional 'where' is critical to perform correctly below are examples to produce correct results. As a rule of thumb anything you put in front of the set will be used as a quantifier. If you want to put a quantifier after the key word 'set' you should use the key word 'where'.

The following is wrong:

$ sacctmgr modify user name=adam set fairshare=10 cluster=tux

This will produce an error as the above line reads modify user adam set fairshare=10 and cluster=tux. Either of the following is correct:

$ sacctmgr modify user name=adam cluster=tux set fairshare=10
$ sacctmgr modify user name=adam set fairshare=10 where cluster=tux

When changing qos for something only use the '=' operator when wanting to explicitly set the qos to something. In most cases you will want to use the '+=' or '-=' operator to either add to or remove from the existing qos already in place.

If a user already has qos of normal,standby for a parent or it was explicitly set you should use qos+=expedite to add this to the list in this fashion.

If you are looking to only add the qos expedite to only a certain account and or cluster you can do that by specifying them in the sacctmgr line.

$ sacctmgr modify user name=adam set qos+=expedite

$ sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite

Let's give an example how to add QOS to user accounts. List all available QOSs in the cluster.

$ sacctmgr show qos format=name
       Name
  ---------
     normal
   expedite

List all the associations in the cluster.

$ sacctmgr show assoc format=cluster,account,qos
   Cluster     Account                  QOS
  --------  ---------- --------------------
     zebra        root               normal
     zebra        root               normal
     zebra           g               normal
     zebra          g1               normal

Add the QOS expedite to account G1 and display the result. Using the operator += the QOS will be added together with the existing QOS to this account.

$ sacctmgr modify account name=g1 set qos+=expedite
$ sacctmgr show assoc format=cluster,account,qos
   Cluster     Account                  QOS
  --------  ---------- --------------------
     zebra        root               normal
     zebra        root               normal
     zebra           g               normal
     zebra          g1      expedite,normal

Now set the QOS expedite as the only QOS for the account G and display the result. Using the operator = that expedite is the only usable QOS by account G

$ sacctmgr modify account name=G set qos=expedite
$ sacctmgr show assoc format=cluster,account,qos
   Cluster     Account                  QOS
  --------  ---------- --------------------
     zebra        root               normal
     zebra        root               normal
     zebra           g             expedite
     zebra          g1      expedite,normal

If a new account is added under the account G it will inherit the QOS expedite and it will not have access to QOS normal.

$ sacctmgr add account banana parent=G
$ sacctmgr show assoc format=cluster,account,qos
   Cluster     Account                  QOS
  --------  ---------- --------------------
     zebra        root               normal
     zebra        root               normal
     zebra           g             expedite
     zebra      banana             expedite
     zebra          g1      expedite,normal

An example of listing trackable resources:

$ sacctmgr show tres
       Type              Name      ID
 ---------- ----------------- --------
        cpu                          1
        mem                          2
     energy                          3
       node                          4
    billing                          5
       gres         gpu:tesla     1001
    license               vcs     1002
         bb              cray     1003

📜 COPYING 📜

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

👀 SEE ALSO 👀

slurm.conf(5), slurmdbd(8)

Okay that's the whole page, bestie! Footer vibes, as requested: source/version Slurm 26.05, chilling under Slurm Commands, page sacctmgr(1) — mwah 💋✨

📦 where this page comes from

arch rolling: slurm-llnl 26.05.5.1-1 · GPL-2.0-or-later
epel 10: slurm 26.05.5-1.el10_3 · GPL-2.0-or-later AND BSD-2-Clause AND BSD-1-Clause
nixpkgs unstable: slurm 26-05-5-1
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-04: a language model's rewrite of the original, under its terms

sacctmgr(1)

🎀 sacctmgr(1) 🎀

🌸 NAME 🌸

🦋 SYNOPSIS 🦋

💖 DESCRIPTION 💖

✨ OPTIONS ✨

💜 COMMANDS 💜

🌈 INTERACTIVE COMMANDS 🌈

🦋 ENTITIES 🦋

💖 GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES 💖

🎀 SPECIFICATIONS FOR ACCOUNTS 🎀

✨ LIST/SHOW ACCOUNT FORMAT OPTIONS ✨

🦋 SPECIFICATIONS FOR ASSOCIATIONS 🦋

🌸 LIST/SHOW ASSOCIATION FORMAT OPTIONS 🌸

✨ SPECIFICATIONS FOR CLUSTERS ✨

🔍 LIST/SHOW CLUSTER FORMAT OPTIONS 🔍

🎀 SPECIFICATIONS FOR COORDINATOR 🎀

🔥 SPECIFICATIONS FOR EVENTS 🔥

🌈 LIST/SHOW EVENT FORMAT OPTIONS 🌈

🦋 SPECIFICATIONS FOR FEDERATION 🦋

⭐ LIST/SHOW FEDERATION FORMAT OPTIONS ⭐

✨ SPECIFICATIONS FOR INSTANCES ✨

🦋 LIST/SHOW INSTANCE FORMAT OPTIONS 🦋

🎀 SPECIFICATIONS FOR JOB 🎀

🌈 LIST/SHOW JOB FORMAT OPTIONS 🌈

👑 SPECIFICATIONS FOR QOS 👑

GrpJobs

GrpJobsAccrue

GrpSubmit, GrpSubmitJobs

GrpTRES

GrpTRESMins

GrpTRESRunMins

GrpWall

LimitFactor

MaxJobsAccruePA, MaxJobsAccruePerAccount

MaxJobsAccruePU, MaxJobsAccruePerUser

MaxJobsPA, MaxJobsPerAccount

MaxJobsPU, MaxJobsPerUser

MaxSubmitJobsPA, MaxSubmitJobsPerAccount

MaxSubmitJobsPU, MaxSubmitJobsPerUser

MaxTRES, MaxTRESPJ, MaxTRESPerJob

MaxTRESMins, MaxTRESMinsPJ, MaxTRESMinsPerJob

MaxTRESPA, MaxTRESPerAccount

MaxTRESPN, MaxTRESPerNode

MaxTRESPU, MaxTRESPerUser

MaxTRESRunMinsPA, MaxTRESRunMinsPerAccount

MaxTRESRunMinsPU, MaxTRESRunMinsPerUser

MaxWall, MaxWallDurationPerJob

MinPrioThreshold

MinTRES, MinTRESPerJob

Name

Preempt

PreemptExemptTime

PreemptMode

Priority

RawUsage=<value>

UsageFactor

UsageThreshold

🦋 LIST/SHOW QOS FORMAT OPTIONS 🦋

Description

Flags

GraceTime

GrpJobs

GrpJobsAccrue

GrpSubmit

GrpSubmitJobs

GrpTRES

GrpTRESMins

GrpTRESRunMins

GrpWall

ID

LimitFactor

MaxJobsAccruePA

MaxJobsAccruePerAccount

MaxJobsAccruePU

MaxJobsAccruePerUser

MaxJobsPA

MaxJobsPerAccount

MaxJobsPU

MaxJobsPerUser

`GrpJobs`

`GrpJobsAccrue`

`GrpSubmit`, `GrpSubmitJobs`

`GrpTRES`

`GrpTRESMins`

`GrpTRESRunMins`

`GrpWall`

`LimitFactor`

`MaxJobsAccruePA`, `MaxJobsAccruePerAccount`

`MaxJobsAccruePU`, `MaxJobsAccruePerUser`

`MaxJobsPA`, `MaxJobsPerAccount`

`MaxJobsPU`, `MaxJobsPerUser`

`MaxSubmitJobsPA`, `MaxSubmitJobsPerAccount`

`MaxSubmitJobsPU`, `MaxSubmitJobsPerUser`

`MaxTRES`, `MaxTRESPJ`, `MaxTRESPerJob`

`MaxTRESMins`, `MaxTRESMinsPJ`, `MaxTRESMinsPerJob`

`MaxTRESPA`, `MaxTRESPerAccount`

`MaxTRESPN`, `MaxTRESPerNode`

`MaxTRESPU`, `MaxTRESPerUser`

`MaxTRESRunMinsPA`, `MaxTRESRunMinsPerAccount`

`MaxTRESRunMinsPU`, `MaxTRESRunMinsPerUser`

`MaxWall`, `MaxWallDurationPerJob`

`MinPrioThreshold`

`MinTRES`, `MinTRESPerJob`

`Name`

`Preempt`

`PreemptExemptTime`

`PreemptMode`

`RawUsage=<value>`

`Clusters=<cluster_name>[,<cluster_name>,...]`

`End=<OPT>`

`ID=<OPT>`

`Names=<OPT>`

`Nodes=<node_name>[,<node_name>,...]`

`Start=<OPT>`