squeue(1)

📦
version 26.05.5.1 · 9 versions
, in arch rolling, epel 10, nixpkgs unstable

🎀 squeue(1) • Slurm Commands 🎀

✨ NAME ✨

squeue — view information about jobs located in the Slurm scheduling queue. 💖

🌸 SYNOPSIS 🌸

squeue [OPTIONS...]

🦋 DESCRIPTION 🦋

Okay bestie, squeue is the command you use to view job and job step information for jobs managed by Slurm. That’s the whole vibe. ✨

💅 OPTIONS 💅

  • -A, --account=<account_list> — Specify the accounts of the jobs to view. It accepts a comma separated list of account names. This has no effect when listing job steps.
  • -a, --all — Display information about jobs and job steps in all partitions. This causes information to be displayed about partitions that are configured as hidden, partitions that are unavailable to a user's group, and federated jobs that are in a "revoked" state.
  • -r, --array — Display one job array element per line. Without this option, the display will be optimized for use with job arrays (pending job array elements will be combined on one line of output with the array index values printed using a regular expression).
  • -M, --clusters=<cluster_name> — Clusters to issue commands to. Multiple cluster names may be comma separated. A value of 'all' will query to run on all clusters. Note that the slurmdbd must be up for this option to work properly, unless running in a federation with either FederationParameters=fed_display configured or the --federation option set. This option implicitly sets the --local option.
  • --expand-patterns — Expand any filename patterns from in StdOut, StdErr and StdIn. Fields that map to a range of values will use the first value of the range. For example "%t" for task id will be replaced by "0".
  • --federation — Show jobs from the federation if a member of one.
  • -o, --format=<output_format> — Specify the information to be displayed, its size and position (right or left justified). Also see the -O, --Format=<output_format> option described below (which supports less flexibility in formatting, but supports access to all fields). If the command is executed in a federated cluster environment and information about more than one cluster is to be displayed and the -h, --noheader option is used, then the cluster name will be displayed before the default output formats shown below.

The default formats with various options are:

  • default — "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
  • -l, --long — "%.18i %.9P %.8j %.8u %.8T %.10M %.9l %.6D %R"
  • -s, --steps — "%.15i %.8j %.9P %.8u %.9M %N"

The format of each field is "%[[.]size]type[suffix]".

  • size — Minimum field size. If no size is specified, whatever is needed to print the information will be used.
  • . — Indicates the output should be right justified and size must be specified. By default output is left justified.
  • suffix — Arbitrary string to append to the end of the field.

Note that many of these type specifications are valid only for jobs while others are valid only for job steps. Valid type specifications include:

  • %all — Print all fields available for this data type with a vertical bar separating each field.
  • %a — Account associated with the job. (Valid for jobs only)
  • %A — Number of tasks created by a job step. This reports the value of the srun --ntasks option. (Valid for job steps only)
  • %A — Job id. This will have a unique value for each element of job arrays. (Valid for jobs only)
  • %B — Executing (batch) host. For an allocated session, this is the host on which the session is executing (i.e. the node from which the srun or the salloc command was executed). For a batch job, this is the node executing the batch script. In the case of a typical Linux cluster, this would be the compute node zero of the allocation.
  • %c — Minimum number of CPUs (processors) per node requested by the job. This reports the value of the srun --mincpus option with a default value of zero. (Valid for jobs only)
  • %C — Number of CPUs (processors) requested by the job or allocated to it if already running. As a job is completing this number will reflect the current number of CPUs allocated. (Valid for jobs only)
  • %d — Minimum size of temporary disk space (in MiB) requested by the job. (Valid for jobs only)
  • %D — Number of nodes allocated to the job or the minimum number of nodes required by a pending job. The actual number of nodes allocated to a pending job may exceed this number if the job specified a node range count (e.g. minimum and maximum node counts) or the job specifies a processor count instead of a node count. As a job is completing this number will reflect the current number of nodes allocated. (Valid for jobs only)
  • %e — Time at which the job ended or is expected to end (based upon its time limit). (Valid for jobs only)
  • %E — Job dependencies remaining. This job will not begin execution until these dependent jobs complete. In the case of a job that can not run due to job dependencies never being satisfied, the full original job dependency specification will be reported. Once a dependency is satisfied, it is removed from the job. A value of NULL implies this job has no dependencies. (Valid for jobs only)
  • %f — Features required by the job. (Valid for jobs only)
  • %F — Job array's job ID. This is the base job ID. For non-array jobs, this is the job ID. (Valid for jobs only)
  • %g — Group name of the job. (Valid for jobs only)
  • %G — Group ID of the job. (Valid for jobs only)
  • %h — Oversubscribe disposition for the job (OVER_SUBSCRIBE column): "YES" if the job requested --oversubscribe; "NO" otherwise; "OK" if the partition allows oversubscription (without the job disabling it) when the job did not request --oversubscribe. This does not encode exclusivity; see the Exclusive field below. (Valid for jobs only)
  • %H — Number of sockets per node requested by the job. This reports the value of the srun --sockets-per-node option. When --sockets-per-node has not been set, "*" is displayed. (Valid for jobs only)
  • %i — Job or job step id. In the case of job arrays, the job ID format will be of the form "<base_job_id>_<index>". By default, the job array index field size will be limited to 64 bytes. Use the environment variable SLURM_BITSTR_LEN to specify larger field sizes. (Valid for jobs and job steps) In the case of heterogeneous job allocations, the job ID format will be of the form "#+#" where the first number is the "heterogeneous job leader" and the second number the zero origin offset for each component of the job.
  • %I — Number of cores per socket requested by the job. This reports the value of the srun --cores-per-socket option. When --cores-per-socket has not been set, "*" is displayed. (Valid for jobs only)
  • %j — Job or job step name. (Valid for jobs and job steps)
  • %J — Number of threads per core requested by the job. This reports the value of the srun --threads-per-core option. When --threads-per-core has not been set, "*" is displayed. (Valid for jobs only)
  • %k — Comment associated with the job. (Valid for jobs only)
  • %K — Job array index. By default, this field size will be limited to 64 bytes. Use the environment variable SLURM_BITSTR_LEN to specify larger field sizes. (Valid for jobs only)
  • %l — Time limit of the job or job step in days-hours:minutes:seconds. The value may be "NOT_SET" if not yet established or "UNLIMITED" for no limit. (Valid for jobs and job steps)
  • %L — Time left for the job to execute in days-hours:minutes:seconds. This value is calculated by subtracting the job's time used from its time limit. The value may be "NOT_SET" if not yet established or "UNLIMITED" for no limit. (Valid for jobs only)
  • %m — Minimum size of memory (in MiB) requested by the job. (Valid for jobs only) If memory was requested per CPU, or per GPU, the value is shown with the assumption that at least one CPU, GPU will be allocated respectively.
  • %M — Time used by the job or job step in days-hours:minutes:seconds. The days and hours are printed only as needed. For job steps this field shows the elapsed time since execution began and thus will be inaccurate for job steps which have been suspended. Clock skew between nodes in the cluster will cause the time to be inaccurate. If the time is obviously wrong (e.g. negative), it displays as "INVALID". (Valid for jobs and job steps)
  • %n — List of node names explicitly requested by the job. (Valid for jobs only)
  • %N — List of nodes allocated to the job or job step. In the case of a COMPLETING job, the list of nodes will comprise only those nodes that have not yet been returned to service. (Valid for jobs and job steps)
  • %o — The command to be executed.
  • %O — Are contiguous nodes requested by the job. (Valid for jobs only)
  • %p — Priority of the job (converted to a floating point number between 0.0 and 1.0). Also see %Q. (Valid for jobs only)
  • %P — Partition of the job or job step. (Valid for jobs and job steps)
  • %q — Quality of service associated with the job. (Valid for jobs only)
  • %Q — Priority of the job (generally a very large unsigned integer). Also see %p. (Valid for jobs only)
  • %r — the reason your job is sitting in its current state 🧐 hop down to JOB REASON CODES below for the tea. (jobs only, babes)
  • %R — okay this one's a triple threat 💅 for pending jobs it prints, inside parenthesis, why the scheduler hasn't started it yet; for terminated jobs that failed it prints, inside parenthesis, an explanation of why it flopped; and for every other job state it just prints the list of allocate nodes. See JOB REASON CODES below. (jobs only)
  • %s — the SLUID ✨ (jobs only)
  • %S — actual or expected start time of the job or job step 🕐 (jobs and job steps)
  • %t — job state, compact form 🎀 see JOB STATE CODES below for the full menu of possible states. (jobs only)
  • %T — job state, extended form 🌸 same deal, see JOB STATE CODES below. (jobs only)
  • %u — user name for a job or job step 🫶 (jobs and job steps)
  • %U — user ID for a job or job step 💜 (jobs and job steps)
  • %v — the reservation for the job 🎟️ (jobs only)
  • %V — the job's submission time ⏰
  • %w — the Workload Characterization Key (wckey) 👑 (jobs only)
  • %W — licenses requested by the job 📜 (jobs only)
  • %x — the list of node names the job explicitly excluded 🚫 (jobs only)
  • %X — count of cores reserved on each node for system use (that's core specialization, bestie) 🔒 (jobs only)
  • %y — nice value, aka the adjustment to a job's scheduling priority 😇 (jobs only)
  • %Y — for pending jobs, a list of the nodes expected to be used when the job is started 🔮
  • %z — number of requested sockets, cores, and threads (S:C:T) per node for the job. When (S:C:T) has not been set, * is displayed 🌟 (jobs only)
  • %Z — the job's working directory 📁
  • -O, --Format=<output_format> — tells squeue exactly what info to show you 🫡 also peep the -o, --format=<output_format> option described above (it's the flexible one, but lowkey it can't reach all the fields because, and I quote, "we ran out of letters" 😭). You hand it a comma separated list of job information to display.

Each field is formatted like this:

type[:[.][size][suffix]]
  • size — minimum field size. If you don't specify a size, 20 characters get allocated to print the information 📏
  • . — means the output should be right justified, and size must be specified. Default is left justified, no cap ⬅️
  • suffix — an arbitrary string to append to the end of the field 🎀

Okay so heads up: lots of these type specs are only valid for jobs, and others are only valid for job steps. The valid type specifications are:

  • Account — print the account associated with the job 💳 (jobs only)
  • AccrueTime — print the accrue time associated with the job ⏳ (jobs only)
  • admin_comment — the administrator comment associated with the job 📝 (jobs only)
  • AllocNodes — print the nodes allocated to the job 🖥️ (jobs only)
  • AllocSID — print the session ID used to submit the job 🔑 (jobs only)
  • ArrayJobID — prints the job ID of the job array 🎯 (jobs and job steps)
  • ArrayTaskID — prints the task ID of the job array 🧩 (jobs and job steps)
  • AssocID — prints the ID of the job association 🪪 (jobs only)
  • BatchFlag — prints whether the batch flag has been set 🚩 (jobs only)
  • BatchHost — the executing (batch) host. For an allocated session, this is the host on which the session is executing (i.e. the node from which the srun or the salloc command was executed). For a batch job, this is the node executing the batch script 🏠 in the case of a typical Linux cluster, this would be the compute node zero of the allocation. (jobs only)
  • BoardsPerNode — prints the number of boards per node allocated to the job 🧮 (jobs only)
  • BurstBuffer — Burst Buffer specification 💥 (jobs only)
  • BurstBufferState — Burst Buffer state 💥 (jobs only)
  • Cluster — name of the cluster that is running the job or job step 🌐
  • ClusterFeature — cluster features required by the job ✨ (jobs only)
  • Command — the command to be executed ⌨️ (jobs only)
  • Comment — comment associated with the job 💬 (jobs only)
  • Contiguous — are contiguous nodes requested by the job? 🧱 (jobs only)
  • Container — OCI container bundle path 📦
  • ContainerID — OCI container assigned ID 🆔
  • ContainerType — job container type for job 🏷️
  • Cores — number of cores per socket requested by the job. This reports the value of the srun --cores-per-socket option. When --cores-per-socket has not been set, * is displayed ⭐ (jobs only)
  • CoreSpec — count of cores reserved on each node for system use (core specialization) 🔒 (jobs only)
  • CPUFreq — prints the frequency of the allocated CPUs ⚡ (job steps only!)
  • cpus-per-task — prints the number of CPUs per tasks allocated to the job 🧠 (jobs only)
  • cpus-per-tres — print the memory required per trackable resources allocated to the job or job step 💾
  • CronJob — print Yes/No depending on whether the job has been generated by scrontab or not 🗓️ (jobs only)
  • Deadline — prints the deadline affected to the job ⌛ (jobs only)
  • DelayBoot — delay boot time 🥾 (jobs only)
  • Dependency — job dependencies remaining. This job will not begin execution until these dependent jobs complete 🚧 in the case of a job that can not run due to job dependencies never being satisfied, the full original job dependency specification will be reported. Once a dependency is satisfied, it is removed from the job. A value of NULL implies this job has no dependencies. (jobs only)
  • DerivedEC — the highest exit code returned by the job's job steps (srun invocations) 📉 following the colon is the signal that caused the process to terminate if it was terminated by a signal. (jobs only)
  • EligibleTime — time the job is eligible for running 🕒 (jobs only)
  • EndTime — the time of job termination, actual or expected 🔚 (jobs only)
  • Exclusive — exclusive disposition for the job (the EXCLUSIVE column) 🙅‍♀️ (jobs only). Possible values:
    • MCS: only jobs with the same MCS label may share a node (job submitted with --exclusive=mcs).
    • NO: no exclusivity.
    • NODE: whole-node allocation (job submitted with --exclusive, or partition Exclusive=NODE).
    • TOPO: only one job per topology segment (job submitted with --exclusive=topo, or partition Exclusive=TOPO). Same tokens as the Exclusive field in scontrol show job.
    • USER: only one user's jobs may share a node (job submitted with --exclusive=user, or partition Exclusive=USER).
  • ExcNodes — the nodes requested to be excluded when allocating this job 🚫 (jobs only)
  • exit_code — the exit code returned by the job, typically as set by the exit() function 🚪 following the colon is the signal that caused the process to terminate if it was terminated by a signal. (jobs only)
  • Feature — features required by the job 🌟 (jobs only)
  • GroupID — group ID of the job 👥 (jobs only)
  • GroupName — group name of the job 👥 (jobs only)
  • HetJobID — job ID of the heterogeneous job leader 🧬
  • HetJobIDSet — expression identifying all components job IDs within a heterogeneous job 🧬
  • HetJobOffset — zero origin offset within a collection of heterogeneous job components 🧬
  • JobArrayID — job array's job ID. This is the base job ID. For non-array jobs, this is the job ID 🔢 (jobs only)
  • JobID — job ID 🆔 this will have a unique value for each element of job arrays and each component of heterogeneous jobs. (jobs only)
  • LastSchedEval — prints the last time the job was evaluated for scheduling 🕵️‍♀️ (jobs only)
  • Licenses — licenses requested by the job 📜 (jobs only)
  • LicensesAlloc — licenses allocated to the job 📜 (jobs only)
  • MaxCPUs — prints the max number of CPUs allocated to the job 🧠 (jobs only)
  • MaxNodes — prints the max number of nodes allocated to the job 🖥️ (jobs only)
  • MCSLabel — prints the MCS_label of the job 💜 (valid for jobs only)
  • mem-per-tres — prints the memory (in MiB) required per trackable resource allocated to the job or job step. yes bestie, per TRES, it's giving very granular ✨
  • MinCpus — the minimum number of CPUs (processors) per node that the job asked for. this reports the value of the srun --mincpus option, and the default is zero. (valid for jobs only)
  • MinMemory — minimum size of memory (in MiB) requested by the job 🌸 (valid for jobs only)
  • MinTime — minimum time limit of the job ⏳ (valid for jobs only)
  • MinTmpDisk — minimum size of temporary disk space (in MiB) requested by the job 💾 (valid for jobs only)
  • Name — job or job step name 🏷️ (valid for jobs and job steps)
  • Network — the network the job is running on 🌐 (valid for jobs and job steps)
  • Nice — nice value, aka the adjustment to a job's scheduling priority 😇 (valid for jobs only)
  • NodeList — list of nodes allocated to the job or job step. plot twist: if the job is in a COMPLETING state, this list only has the nodes that haven't been returned to service yet 👀 (valid for jobs only)
  • Nodes — same thing, list of nodes allocated to the job or job step, and for a COMPLETING job it's only the nodes not yet returned to service. careful — this one is (valid job steps only), so don't mix them up 🎀
  • NTPerBoard — number of tasks per board allocated to the job 🧩 (valid for jobs only)
  • NTPerCore — number of tasks per core allocated to the job 💅 (valid for jobs only)
  • NTPerNode — number of tasks per node allocated to the job 🖥️ (valid for jobs only)
  • NTPerSocket — number of tasks per socket allocated to the job 🔌 (valid for jobs only)
  • NumCPUs — number of CPUs (processors) requested by the job, or allocated to it if it's already running. while a job is completing, this reflects the current number of CPUs allocated. (valid for jobs and job steps)
  • NumNodes — number of nodes allocated to the job, or the minimum number of nodes a pending job requires. lowkey important: the actual number allocated to a pending job can be more than this if the job gave a node range count (like min and max node counts) or if it specified a processor count instead of a node count. as the job completes, this reflects the current number of nodes allocated 💫 (valid for jobs only)
  • NumTasks — number of tasks requested by a job or job step. this reports the value of the --ntasks option 📋 (valid for jobs and job steps)
  • Origin — cluster name where the federated job originated from 🌍 (valid for federated jobs only)
  • OriginRaw — cluster ID where the federated job originated from 🆔 (valid for federated jobs only)
  • OverSubscribe — same values as %h / OVER_SUBSCRIBE: "YES", "NO", or "OK". see %h above ☝️ (valid for jobs only)
  • Partition — partition of the job or job step 🗂️ (valid for jobs and job steps)
  • PendingTime — the time (in seconds) between the job's start time and submit time. if the job hasn't started yet, it's the time (in seconds) between now and the submit time ⏱️ (valid for jobs only)
  • PreemptTime — the preempt time for the job ⚡ (valid for jobs only)
  • Prefer — the preferred features of a pending job 🎯 (valid for jobs only)
  • Priority — priority of the job, converted to a floating point number between 0.0 and 1.0. also see prioritylong 👑 (valid for jobs only)
  • PriorityLong — priority of the job (generally a very large unsigned integer). also see priority 🔢 (valid for jobs only)
  • Profile — profile of the job 📊 (valid for jobs only)
  • QOS — quality of service associated with the job 💎 (valid for jobs only)
  • Reason — the reason a job is in its current state. see the JOB REASON CODES section below for more 🕵️ (valid for jobs only)
  • ReasonList — depends on the state, bestie:
    • for pending jobs: the reason the job is waiting for execution is printed inside parenthesis
    • for terminated jobs with failure: an explanation of why the job failed is printed inside parenthesis
    • for all other job states: the list of allocated nodes see the JOB REASON CODES section below for more 📝 (valid for jobs only)
  • Reboot — indicates if the allocated nodes should be rebooted before the job starts 🔄 (valid on jobs only)
  • ReqNodes — list of node names explicitly requested by the job 📌 (valid for jobs only)
  • ReqSwitch — the max number of requested switches by for the job 🔀 (valid for jobs only)
  • Requeue — prints whether the job will be requeued on failure ♻️ (valid for jobs only)
  • Reservation — reservation for the job 🎟️ (valid for jobs only)
  • ResizeTime — the amount of time changed for the job to run 📐 (valid for jobs only)
  • RestartCnt — the number of restarts for the job 🔁 (valid for jobs only)
  • ResvPort — reserved ports of the job 🚪 (valid for job steps only)
  • SchedNodes — for pending jobs, a list of the nodes expected to be used when the job is started 🗺️ (valid for jobs only)
  • SCT — number of requested sockets, cores, and threads (S:C:T) per node for the job. when (S:C:T) hasn't been set, "*" is displayed ⭐ (valid for jobs only)
  • SegmentSize — segment size requested by the job 📏 (valid for jobs only)
  • SiblingsActive — cluster names of where federated sibling jobs exist 👯 (valid for federated jobs only)
  • SiblingsActiveRaw — cluster IDs of where federated sibling jobs exist 🆔 (valid for federated jobs only)
  • SiblingsViable — cluster names of where federated sibling jobs are viable to run 🌱 (valid for federated jobs only)
  • SiblingsViableRaw — cluster IDs of where federated sibling jobs are viable to run 🆔 (valid for federated jobs only)
  • Sluid — the Slurm Lexicographically-sortable Unique Identifier assigned to the job 🔑 this value stays constant during the lifetime of the job, even after a resize. but note: the accounting database tracks a separate SLUID per accounting record, and that one changes on resize or requeue. iykyk 💀
  • Sockets — number of sockets per node requested by the job. this reports the value of the srun --sockets-per-node option. when --sockets-per-node hasn't been set, "*" is displayed 🔌 (valid for jobs only)
  • SPerBoard — number of sockets per board allocated to the job 🧩 (valid for jobs only)
  • StartTime — actual or expected start time of the job or job step ⏰ (valid for jobs and job steps)
  • State — job state in extended form. see the JOB STATE CODES section below for the possible states 🚦 (valid for jobs only)
  • StateCompact — job state in compact form. see the JOB STATE CODES section below for the possible states 🚥 (valid for jobs only)
  • STDERR — the directory for standard error to output to 📕 (valid for jobs and steps)
  • STDIN — the directory for standard in 📗 (valid for jobs and steps)
  • STDOUT — the directory for standard out to output to 📘 (valid for jobs and steps)
  • StepID — job or job step ID. for job arrays, the job ID format is "<base_job_id>_<index>" 🏷️ (valid for job steps only)
  • StepName — job step name ✨ (valid for job steps only)
  • StepState — the state of the job step 🚦 (valid for job steps only)
  • SubmitTime — the time the job was submitted at 🕰️ (valid for jobs only)
  • system_comment — system comment associated with the job 💬 (valid for jobs only)
  • Threads — number of threads per core requested by the job. this reports the value of the srun --threads-per-core option. when --threads-per-core hasn't been set, "*" is displayed 🧵 (valid for jobs only)
  • TimeLeft — time left for the job to execute, in days-hours:minutes:seconds ⏳ it's calculated by subtracting the job's time used from its time limit. the value can be "NOT_SET" if it isn't established yet, or "UNLIMITED" for no limit 🫶 (valid for jobs only)
  • TimeLimit — timelimit for the job or job step ⏲️ (valid for jobs and job steps)
  • TimeUsed — time used by the job or job step, in days-hours:minutes:seconds. the days and hours only get printed as needed 📅 couple of caveats though: for job steps this field shows the elapsed time since execution began, so it'll be inaccurate for job steps that have been suspended. also, clock skew between nodes in the cluster will make the time inaccurate. and if the time is obviously wrong (like negative), it displays as "INVALID" 💀 (valid for jobs and job steps)
  • tres-alloc — Prints the trackable resources allocated to the job if it’s running. If it’s not running, then print the trackable resources requested by the job. literally knows what the job got or what it’s asking for 💅
  • tres-bind — Prints the trackable resources task binding requested by the job or job step. binding tea ☕✨
  • tres-freq — Prints the trackable resources frequencies requested by the job or job step. freq check 🔥
  • tres-per-job — Prints the trackable resources requested by the job. per-job, bestie 🎀
  • tres-per-node — Prints the trackable resources per node requested by the job or job step. node-by-node, no cap 🌈
  • tres-per-socket — Prints the trackable resources per socket requested by the job or job step. socket moment 🧚
  • tres-per-step — Prints the trackable resources requested by the job step. step-specific slay 💜
  • tres-per-task — Prints the trackable resources per task requested by the job or job step. task math, iconic 🍓
  • UserID — User ID for a job or job step. (Valid for jobs and job steps) iykyk 🦋
  • UserName — User name for a job or job step. (Valid for jobs and job steps) name drop 💖
  • Wait4Switch — The amount of time to wait for the desired number of switches. (Valid for jobs only) waiting era 😭
  • WCKey — Workload Characterization Key (wckey). (Valid for jobs only) wckey core 🎀
  • WorkDir — The job’s working directory. (Valid for jobs only) where the job lives 🏠✨
  • --help — Print a help message describing all options squeue. help bestie help 🫶
  • --hide — Do not display information about jobs and job steps in all partitions. By default, information about partitions that are configured as hidden or are not available to the user’s group will not be displayed (i.e. this is the default behavior). hidden means hidden, periodt 🙈
  • -i, --iterate=<seconds> — Repeatedly gather and report the requested information at the interval specified (in seconds). By default, prints a time stamp with the header. loop era 🔁✨
  • -j, --jobs[=<job_id_list>] — Specify a comma separated list of job IDs to display. Defaults to all jobs. The --jobs=<job_id_list> option may be used in conjunction with the --steps option to print step information about specific jobs. Note: if a list of job IDs is provided, the jobs are displayed even if they are on hidden partitions. Since this option’s argument is optional, for proper parsing the single letter option must be followed immediately with the value and not include a space between them. For example -j1008 and not -j 1008. The job ID format is job_id[_array_id]. A SLUID may also be used in place of a numeric job ID. Performance of the command can be measurably improved for systems with large numbers of jobs when a single job ID is specified. By default, this field size will be limited to 64 bytes. Use the environment variable SLURM_BITSTR_LEN to specify larger field sizes. job list main character energy 👑
Bestie PSA: for -j, no space between the flag and the value. -j1008 slays, -j 1008 does not parse right. do not do it 😭
  • --json, --json=list, --json=<data_parser> — Dump information as JSON using the default data_parser plugin or explicit data_parser with parameters. All information is dumped, even if it would normally not be. Sorting and formatting arguments passed to other options are ignored; however, most filtering arguments are still used. JSON dump, all the tea ☕💅
  • -L, --licenses=<license_list> — Request jobs requesting or using one or more of the named licenses. The license list consists of a comma separated list of license names. license check 🔥
  • --local — Show only jobs local to this cluster. Ignore other clusters in this federation (if any). Overrides --federation. stay local 🌸
  • -l, --long — Report more of the available information for the selected jobs or job steps, subject to any constraints specified. long mode = more lore 🎀
  • --me — Equivalent to --user=<my username>. it’s me, hi 💖
  • -n, --name=<name_list> — Request jobs or job steps having one of the specified names. The list consists of a comma separated list of job names. name filter, cute 🧚
  • --noconvert — Don’t convert units from their original type (e.g. 2048M won’t be converted to 2G). raw units, no glow-up 💜
  • -w, --nodelist=<hostlist> — Report only on jobs allocated to the specified node or list of nodes. This may either be the NodeName or NodeHostname as defined in slurm.conf(5) in the event that they differ. A node_name of localhost is mapped to the current host name. node list bestie 🍓
  • -h, --noheader — Do not print a header on the output. no header, just vibes 🦋
  • --notme — Opposite of --me; only display jobs that are not from the invoking user. not me, literally 🙈
  • --only-job-state — Only query for the job state. Query utilizes RPC that only retains JobID and State information, reducing work required by slurmctld to respond. state only, lowkey efficient ✨
  • -p, --partition=<part_list> — Specify the partitions of the jobs or steps to view. Accepts a comma separated list of partition names. partition picker 🌈
  • -P, --priority — For pending jobs submitted to multiple partitions, list the job once per partition. In addition, if jobs are sorted by priority, consider both the partition and job priority. This option can be used to produce a list of pending jobs in the same order considered for scheduling by Slurm with appropriate additional options (e.g. --sort=-p,i --states=PD). priority order, main character scheduling 👑
  • -q, --qos=<qos_list> — Specify the qos(s) of the jobs or steps to view. Accepts a comma separated list of qos’s. qos era 🎀
  • -R, --reservation=<reservation_list> — Specify the reservations of the jobs to view. Accepts a comma separated list of reservation names. Jobs matching any reservation will satisfy the filter (logic works like an OR). reservation filter, OR logic bestie 💜
  • --running-over=<time> — Display only jobs that have been running over the amount of specified time. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". over the limit 🔥
  • --running-under=<time> — Display only jobs that have been running under the amount of specified time. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds". under the limit 🌸
  • --sibling — Show all sibling jobs on a federated cluster. Implies --federation. sibling slay 🦋
  • -S, --sort=<sort_list> — Specification of the order in which records should be reported. This uses the same field specification as the <output_format>. The long format option "cluster" can also be used to sort jobs or job steps by cluster name (e.g. federated jobs). Multiple sorts may be performed by listing multiple sort fields separated by commas. The field specifications may be preceded by "+" or "-" for ascending (default) and descending order respectively. For example, a sort value of "P,U" will sort the records by partition name then by user id. The default value of sort for jobs is "P,t,-p" (increasing partition name then within a given partition by increasing job state and then decreasing priority). The default value of sort for job steps is "P,i" (increasing partition name then within a given partition by increasing step id). sorting is a whole personality 💅
  • --start — Report the expected start time and resources to be allocated for pending jobs in order of increasing start time. This is equivalent to the following options: --format="%.18i %.9P %.8j %.8u %.2t %.19S %.6D %20Y %R", --sort=S and --states=PENDING. Any of these options may be explicitly changed as desired by combining the --start option with other option values (e.g. to use a different output format). The expected start time of pending jobs is only available if the Slurm is configured to use the backfill scheduling plugin. pending jobs get a start-time forecast 🔮✨
  • -t, --states=<state_list> — Specify the states of jobs to view. Accepts a comma separated list of state names or "all". If "all" is specified then jobs of all states will be reported. If no state is specified then pending, running, and completing jobs are reported. See the JOB STATE CODES section below for a list of valid states. Both extended and compact forms are valid. Note the <state_list> supplied is case insensitive ("pd" and "PD" are equivalent). state check, case insensitive bestie 🌈
  • -s, --steps[=<step_list>] — Specify the job steps to view. This flag indicates that a comma separated list of job steps to view follows without an equal sign (see examples). The job step format is job_id[_array_id].step_id. Defaults to all job steps. Since this option’s argument is optional, for proper parsing the single letter option must be followed immediately with the value and not include a space between them. For example -s1008.0 and not -s 1008.0. step list, no space, no cap 💜
Another bestie PSA: -s also wants the value glued right on. -s1008.0 is the move; -s 1008.0 is a no. parsing said nope 😭
  • --usage — Print a brief help message listing the squeue options. quick usage recap 🫶
  • -u, --user=<user_list> — Request jobs or job steps from a comma separated list of users. The list can consist of user names or user id numbers. Performance of the command can be measurably improved for systems with large numbers of jobs when a single user is specified. user filter, single-user glow-up ✨
  • -v, --verbose — Report details of squeue’s actions. verbose mode = lore unlocked 🔥
  • -V, --version — Print version information and exit. version check, bye 👋💖
  • --yaml, --yaml=list, --yaml=<data_parser> — Dump information as YAML using the default data_parser plugin or explicit data_parser with parameters. All information is dumped, even if it would normally not be. Sorting and formatting arguments passed to other options are ignored; however, most filtering arguments are still used. YAML dump, all the info, iconic 🦋

💅 JOB REASON CODES 💅

Okay so these codes are literally the receipts for why your job hasn't started yet — the scheduler said no and here's the note it left 🫶

Bestie, hear me: there can be multiple reasons a job can't start right now, and in that case you only get shown the one reason that the attempted scheduling method actually ran into. So it's not the whole story, it's just the one that came up 💀

The Reasons listed below are some of the more common ones you might see. For a full list of Reason codes refer to this page: https://slurm.schedmd.com/job_reason_codes.html

  • AssocGrp*Limit — the job's association has hit an aggregate limit on some resource.
  • AssociationJobLimit — the job's association has reached its maximum job count.
  • AssocMax*Limit — the job is asking for a resource that violates a per-job limit on the requested association.
  • AssociationResourceLimit — the job's association has hit some resource limit.
  • AssociationTimeLimit — the job's association has reached its time limit.
  • BadConstraints — the job's constraints literally cannot be satisfied 😭
  • BeginTime — the job's earliest start time hasn't been reached yet. patience, queen.
  • Cleaning — the job is being requeued and is still cleaning up from its previous execution.
  • Dependency — this job depends on another job that hasn't been satisfied yet.
  • DependencyNeverSatisfied — this job depends on another job that will never be satisfied. yeah that one's not happening 💀
  • InactiveLimit — the job reached the system InactiveLimit.
  • InvalidAccount — the job's account is invalid.
  • InvalidQOS — the job's QOS is invalid.
  • JobHeldAdmin — the job is held by a system administrator.
  • JobHeldUser — the job is held by the user.
  • JobLaunchFailure — the job could not be launched. this may be due to a file system problem, invalid program name, etc.
  • Licenses — the job is waiting for a license.
  • NodeDown — a node required by the job is down.
  • NonZeroExitCode — the job terminated with a non-zero exit code.
  • PartitionDown — the partition required by this job is in a DOWN state.
  • PartitionInactive — the partition required by this job is in an Inactive state and not able to start jobs.
  • PartitionNodeLimit — the number of nodes required by this job is outside of its partition's current limits. Can also indicate that required nodes are DOWN or DRAINED.
  • PartitionTimeLimit — the job's time limit exceeds its partition's current time limit.
  • Priority — one or more higher priority jobs exist for this partition or advanced reservation. classic queue behavior 😤
  • Prolog — its Prolog program is still running.
  • QOSGrp*Limit — the job's QOS has reached an aggregate limit on some resource.
  • QOSJobLimit — the job's QOS has reached its maximum job count.
  • QOSMax*Limit — the job requests a resource that violates a per-job limit on the requested QOS.
  • QOSResourceLimit — the job's QOS has reached some resource limit.
  • QOSTimeLimit — the job's QOS has reached its time limit.
  • QOSUsageThreshold — required QOS threshold has been breached.
  • ReqNodeNotAvail — some node specifically required by the job is not currently available. The node may currently be in use, reserved for another job, in an advanced reservation, DOWN, DRAINED, or not responding. Nodes which are DOWN, DRAINED, or not responding will be identified as part of the job's "reason" field as "UnavailableNodes". Such nodes will typically require the intervention of a system administrator to make available. aka: go find your admin 🫶
  • Reservation — the job is waiting for its advanced reservation to become available.
  • Resources — the job is waiting for resources to become available.
  • SystemFailure — failure of the Slurm system, a file system, the network, etc. not your fault bestie.
  • TimeLimit — the job exhausted its time limit.
  • WaitingForScheduling — no reason has been set for this job yet. Waiting for the scheduler to determine the appropriate reason. it's giving "pending vibes only" ✨

🦋 JOB STATE CODES 🦋

Jobs typically pass through several states in the course of their execution. The typical states are PENDING, RUNNING, SUSPENDED, COMPLETING, and COMPLETED. The following states are recognized by squeue. A full list of possible states is available at https://slurm.schedmd.com/job_state_codes.html.

  • BF BOOT_FAIL — job terminated due to launch failure, typically due to a hardware failure (e.g. unable to boot the node or block and the job can not be requeued).
  • CA CANCELLED — job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
  • CD COMPLETED — job has terminated all processes on all nodes with an exit code of zero. we love a clean finish 👑
  • CF CONFIGURING — job has been allocated resources, but are waiting for them to become ready for use (e.g. booting).
  • CG COMPLETING — job is in the process of completing. Some processes on some nodes may still be active.
  • DL DEADLINE — job terminated on deadline.
  • F FAILED — job terminated with non-zero exit code or other failure condition. 😭
  • NF NODE_FAIL — job terminated due to failure of one or more allocated nodes.
  • OOM OUT_OF_MEMORY — job experienced out of memory error. the classic 💀
  • PD PENDING — job is awaiting resource allocation.
  • PR PREEMPTED — job terminated due to preemption.
  • R RUNNING — job currently has an allocation. main character energy, honestly ✨
  • RD RESV_DEL_HOLD — job is being held after requested reservation was deleted.
  • RF REQUEUE_FED — job is being requeued by a federation.
  • RH REQUEUE_HOLD — held job is being requeued.
  • RQ REQUEUED — completing job is being requeued.
  • RS RESIZING — job is about to change size.
  • RV REVOKED — sibling was removed from cluster due to other cluster starting the job.
  • SI SIGNALING — job is being signaled.
  • SE SPECIAL_EXIT — the job was requeued in a special state. This state can be set by users, typically in EpilogSlurmctld, if the job has terminated with a particular exit value.
  • SO STAGE_OUT — job is staging out files.
  • ST STOPPED — job has an allocation, but execution has been stopped with SIGSTOP signal. CPUS have been retained by this job.
  • S SUSPENDED — job has an allocation, but execution has been suspended and CPUs have been released for other jobs.
  • TO TIMEOUT — job terminated upon reaching its time limit. rip ⏰

🔥 PERFORMANCE 🔥

Okay so this part is the "be a considerate bestie" section, and honestly it matters 🫶

Executing squeue sends a remote procedure call to slurmctld. If enough calls from squeue or other Slurm client commands that send remote procedure calls to the slurmctld daemon come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service.

Do not run squeue or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Ensure that programs limit calls to squeue to the minimum necessary for the information you are trying to gather.

Like, no cap — spamming squeue in a while true loop is not the flex you think it is, it's lowkey a whole denial of service 💀 be gentle with the controller, she's doing her best 🌸

🌸 ENVIRONMENT VARIABLES 🌸

okay so bestie, plot twist: you don't have to type every flag every single time 💅 some squeue options can be set through environment variables, and they've each got a matching option, all listed below for you. One thing though, and this is important — command line options will ALWAYS override these settings. always. no cap. the env var is the vibe, the flag is the final word 👑

  • SLURM_BITSTR_LEN — sets the string length used to hold a job array's task ID expression. Default is 64 bytes. A value of 0 prints the full expression, however long that ends up being. Heads up though: larger values may adversely impact the application performance 😭
  • SLURM_CLUSTERS — same as --clusters
  • SLURM_CONF — where the Slurm configuration file lives
  • SLURM_DEBUG_FLAGS — specify debug flags for squeue to use. See DebugFlags in the slurm.conf(5) man page for the full list of flags. This env var takes precedence over the setting in the slurm.conf 🫶
  • SLURM_JSON — controls JSON serialization:
    • compact — output JSON as compact as possible
    • pretty — output JSON in pretty format to make it more readable
  • SLURM_TIME_FORMAT — specifies the format used to report time stamps ✨
    • standard — the default value, generates output in the form "year-month-dateThour:minute:second"
    • relative — returns only "hour:minute:second" if it's the current day. For other dates in the current year it prints the "hour:minute" preceded by "Tomorr" (tomorrow), "Ystday" (yesterday), or the name of the day for the coming week (e.g. "Mon", "Tue", etc.), otherwise the date (e.g. "25 Apr"). For other years it returns a date month and year without a time (e.g. "6 Jun 2012"). All of the time stamps use a 24 hour format 🕰️

a valid strftime() format can also be specified. For example, a value of "%a %T" will report the day of the week and a time stamp (e.g. "Mon 12:34:56") — iconic, honestly.

  • SLURM_YAML — controls YAML serialization:
    • compact — output YAML as compact as possible
    • pretty — output YAML in pretty format to make it more readable
  • SQUEUE_ACCOUNT — -A <account_list>, --account=<account_list>
  • SQUEUE_ALL — -a, --all
  • SQUEUE_ARRAY — -r, --array
  • SQUEUE_NAMES — --name=<name_list>
  • SQUEUE_FEDERATION — --federation
  • SQUEUE_FORMAT — -o <output_format>, --format=<output_format>
  • SQUEUE_FORMAT2 — -O <output_format>, --Format=<output_format>
  • SQUEUE_LICENSES — -p-l <license_list>, --license=<license_list>
  • SQUEUE_LOCAL — --local
  • SQUEUE_PARTITION — -p <part_list>, --partition=<part_list>
  • SQUEUE_PRIORITY — -P, --priority
  • SQUEUE_QOS — -p <qos_list>, --qos=<qos_list>
  • SQUEUE_SIBLING — --sibling
  • SQUEUE_SORT — -S <sort_list>, --sort=<sort_list>
  • SQUEUE_STATES — -t <state_list>, --states=<state_list>
  • SQUEUE_USERS — -u <user_list>, --users=<user_list>

🦋 EXAMPLES 🦋

okay these are my favorite part, look at all this slay in action 💖

Print the jobs scheduled in the debug partition and in the COMPLETED state, in the format with six right justified digits for the job id followed by the priority with an arbitrary fields size:

$ squeue -p debug -t COMPLETED -o "%.6i %p"
 JOBID PRIORITY
 65543 99993
 65544 99992
 65545 99991

Print the job steps in the debug partition sorted by user:

$ squeue -s -p debug -S u
  STEPID        NAME PARTITION     USER      TIME NODELIST
 65552.1       test1     debug    alice      0:23 dev[1-4]
 65562.2     big_run     debug      bob      0:18 dev22
 65550.1      param1     debug  candice   1:43:21 dev[6-12]

Print information only about jobs 12345, 12346 and 12348:

$ squeue --jobs 12345,12346,12348
 JOBID PARTITION NAME USER ST  TIME  NODES NODELIST(REASON)
 12345     debug job1 dave  R   0:21     4 dev[9-12]
 12346     debug job2 dave PD   0:00     8 (Resources)
 12348     debug job3 ed   PD   0:00     4 (Priority)

Print information only about job step 65552.1:

$ squeue --steps 65552.1
  STEPID     NAME PARTITION    USER    TIME  NODELIST
 65552.1    test2     debug   alice   12:49  dev[1-4]

see how easy that was? like, look at us go 😭🫶

🎀 COPYING 🎀

Legal time, bestie — I'm gonna keep it real and keep it exact, because this stuff is not the place to freestyle 🙈

Copyright (C) 2002-2007 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). Copyright (C) 2008-2010 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version 💜

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

🌈 SEE ALSO 🌈

go read these next, i'm literally obsessed with cross-referencing, iykyk 📚✨

and that's the whole page, bestie!! 🥳👑 you officially know squeue better than most people know their own phone passcode. this page came from the Slurm 26.05 release — that's the version stamp — living in the Slurm Commands section under squeue(1), with whatever date your build shipped it on sitting right there in the footer next to the version. go forth and queue responsibly 💅🎀🦋

📦 where this page comes from