sdiag(1)

🌸 sdiag — Slurm Commands, section 1 🌸

okay bestie, welcome to sdiag(1) from the Slurm Commands crew ✨ this one's a whole vibe, grab a snack 🍓

---

🌈 NAME 🌈

sdiag — the scheduling diagnostic tool for Slurm 💅

---

🦋 SYNOPSIS 🦋

sdiag

---

👑 DESCRIPTION 👑

So basically sdiag spills ALL the tea about slurmctld execution — we're talking threads, agents, jobs, and scheduling algorithms. The whole point? Get real data out of slurmctld's behavior so you can tweak your configuration parameters or your queue policies 🎀. The main character reason behind it is just... knowing how Slurm actually behaves on systems with a high throughput. no cap.

It has two execution modes, periodt:

• The default mode --all shows several counters and statistics, all explained later down the page 📊
• There's also another execution option --reset for resetting those values 🔄

Values get reset at midnight UTC time by default. Like clockwork, literally 🕛

The first block of info is all about global slurmctld execution 🧚:

✨ Server thread count

The number of current active slurmctld threads. A high number means high load processing events like job submissions, jobs dispatching, jobs completing, etc. If this is often close to MAX_SERVER_THREADS, it could be pointing at a potential bottleneck 😭

✨ Agent queue size

Slurm was designed with scalability in mind, and sending messages to thousands of nodes is not a trivial task 💀 The agent mechanism helps control communication between slurmctld and the slurmd daemons for a best effort. This value = the count of enqueued outgoing RPC requests in an internal retry list.

✨ Agent count

Number of agent threads. Each of these agent threads can create in turn a group of up to 2 + AGENT_THREAD_COUNT active threads at a time 👀

✨ Agent thread count

Total count of active threads created by all the agent threads.

✨ DBD Agent queue size

Slurm queues up the messages intended for the SlurmDBD and processes them in a separate thread. If the SlurmDBD, or the database, is down then this number will increase 📈

The max queue size is configured in the slurm.conf with MaxDBDMsgs. If this number begins to grow more than half of the max queue size, the slurmdbd and the database should be investigated immediately 🚨

✨ Jobs submitted

Number of jobs submitted since last reset.

✨ Jobs started

Number of jobs started since last reset. This includes backfilled jobs 🔥

✨ Jobs completed

Number of jobs completed since last reset.

✨ Jobs canceled

Number of jobs canceled since last reset.

✨ Jobs failed

Number of jobs failed due to slurmd or other internal issues since last reset 💔

✨ Job states ts:

Lists the timestamp of when the following job state counts were gathered.

✨ Jobs pending:

Number of jobs pending at the given time of the time stamp above.

✨ Jobs running:

Number of jobs running at the given time of the time stamp above.

---

The next block of information is related to the main scheduling algorithm based on jobs priorities 👑. A scheduling cycle implies to get the job_write_lock lock, then trying to get resources for jobs pending, starting from the most priority one and going in descending order. Once a job can not get the resources the loop keeps going but just for jobs requesting other partitions. Jobs with dependencies or affected by accounts limits are not processed. 🙅♀️

💜 Last cycle

Time in microseconds for last scheduling cycle.

💜 Max cycle

Maximum time in microseconds for any scheduling cycle since last reset.

💜 Total cycles

Total run time in microseconds for all scheduling cycles since last reset. Scheduling is performed periodically and (depending upon configuration) when a job is submitted or a job is completed.

💜 Mean cycle

Mean time in microseconds for all scheduling cycles since last reset.

💜 Mean depth cycle

Mean of cycle depth. Depth means number of jobs processed in a scheduling cycle.

💜 Cycles per minute

Counter of scheduling executions per minute ⏱️

💜 Last queue length

Length of jobs pending queue.

---

The next block is about the backfilling scheduling algorithm 🌈. A backfilling scheduling cycle implies to get locks for jobs, nodes and partitions objects then trying to get resources for jobs pending. Jobs are processed based on priorities. If a job can not get resources the algorithm calculates when it could get them, obtaining a future start time for the job. Then the next job is processed and the algorithm tries to get resources for that job but avoiding to affect the previous ones, and again it calculates the future start time if not current resources available. The backfilling algorithm takes more time for each new job to process since more priority jobs can not be affected 💅. The algorithm itself takes measures for avoiding a long execution cycle and for taking all the locks for too long.

🦋 Total backfilled jobs (since last slurm start)

Number of jobs started thanks to backfilling since last slurm start.

🦋 Total backfilled jobs (since last stats cycle start)

Number of jobs started thanks to backfilling since last time stats where reset. By default these values are reset at midnight UTC time 🕛

🦋 Total backfilled heterogeneous job components

Number of heterogeneous job components started thanks to backfilling since last Slurm start.

🦋 Total cycles

Number of backfill scheduling cycles since last reset.

🦋 Last cycle when

Time when last backfill scheduling cycle happened in the format "weekday Month MonthDay hour:minute.seconds year".

🦋 Last cycle

Time in microseconds of last backfill scheduling cycle. It counts only execution time, removing sleep time inside a scheduling cycle when it executes for an extended period time. Note that locks are released during the sleep time so that other work can proceed.

🦋 Max cycle

Time in microseconds of maximum backfill scheduling cycle execution since last reset. It counts only execution time, removing sleep time inside a scheduling cycle when it executes for an extended period time. Note that locks are released during the sleep time so that other work can proceed.

🦋 Mean cycle

Mean time in microseconds of backfilling scheduling cycles since last reset.

🦋 Last depth cycle

Number of processed jobs during last backfilling scheduling cycle. It counts every job even if that job can not be started due to dependencies or limits.

🦋 Last depth cycle (try sched)

Number of processed jobs during last backfilling scheduling cycle. It counts only jobs with a chance to start using available resources. These jobs consume more scheduling time than jobs which are found can not be started due to dependencies or limits.

🦋 Depth Mean

Mean count of jobs processed during all backfilling scheduling cycles since last reset. Jobs which are found to be ineligible to run when examined by the backfill scheduler are not counted (e.g. jobs submitted to multiple partitions and already started, jobs which have reached a QOS or account limit such as maximum running jobs for an account, etc).

🦋 Depth Mean (try sched)

The subset of Depth Mean that the backfill scheduler attempted to schedule.

🦋 Last queue length

Number of jobs pending to be processed by backfilling algorithm. A job is counted once for each partition it is queued to use. A pending job array will normally be counted as one job (tasks of a job array which have already been started/requeued or individually modified will already have individual job records and are each counted as a separate job).

🦋 Queue length Mean

Mean count of jobs pending to be processed by backfilling algorithm. A job is counted once for each partition it requested. A pending job array will normally be counted as one job (tasks of a job array which have already been started/requeued or individually modified will already have individual job records and are each counted as a separate job).

🦋 Last table size

Count of different time slots tested by the backfill scheduler in its last iteration.

🦋 Mean table size

Mean count of different time slots tested by the backfill scheduler. Larger counts increase the time required for the backfill operation. The table size is influenced by many scheduling parameters, including: bf_min_age_reserve, bf_min_prio_reserve, bf_resolution, and bf_window.

🦋 Latency for 1000 calls to gettimeofday()

Latency of 1000 calls to the gettimeofday() syscall in microseconds, as measured at controller startup.

---

The next blocks of information report the most frequently issued remote procedure calls (RPCs) — calls made for the Slurmctld daemon to perform some action 📞

The fourth block reports the RPCs issued by message type. You will need to look up those RPC codes in the Slurm source code by looking them up in the file src/common/slurm_protocol_defs.h 👀 The report includes the number of times each RPC is invoked, the total time consumed by all of those RPCs plus the average time consumed by each RPC in microseconds.

The fifth block reports the RPCs issued by user ID, the total number of RPCs they have issued, the total time consumed by all of those RPCs plus the average time consumed by each RPC in microseconds. RPCs statistics are collected for the life of the slurmctld process unless explicitly --reset 🔄

The sixth block of information, labeled Pending RPC Statistics, shows information about pending outgoing RPCs on the slurmctld agent queue. The first section of this block shows types of RPCs on the queue and the count of each. The second section shows up to the first 25 individual RPCs pending on the agent queue, including the type and the destination host list. This information is cached and only refreshed on 30 second intervals 🌸

---

okay that's the whole description bestie, but this page keeps going — part 2 coming right up 💖

✨ OPTIONS ✨

okay bestie, here's the whole menu of things you can hand to sdiag. pick your fighter 💅

• -a, --all — go get the info and report it back to you. this is the default mode, so if you don't pass anything, this is literally what you're already doing. no cap ✨
• -M, --cluster=<string> — which cluster you wanna send commands to. heads up: you can only name one cluster here, not a whole roster. and lowkey important — slurmdbd has to be up for this option to actually work properly, unless you're running in a federation with FederationParameters=fed_display configured in that case you're fine. 🫶
• -h, --help — prints a description of all the options and dips. that's it, that's the whole job.
• --json, --json=list, --json=<data_parser> — dump everything as JSON, using either the default data_parser plugin or an explicit one with its parameters. ⚠️ real talk: sorting and formatting arguments get straight up ignored when you do this. the vibes do not carry over. 💀
• --no-trunc — don't chop long lines down to 80 chars. let them live their full-length truth, main character energy, no truncation. 🌈
• -r, --reset — slam the scheduler and RPC counters back to 0. very satisfying. but ngl it's only supported for Slurm operators and administrators, so if that's not you, this button does not exist for you bestie. 🙈
• -i, --sort-by-id — sorts the Remote Procedure Call (RPC) data by message type ID and user ID.
• -t, --sort-by-time — sorts the RPC data by total run time.
• -T, --sort-by-time2 — sorts the RPC data by average run time. (yes it's a different letter, yes it's a different sort, stay with me 🎀)
• --usage — prints the list of options and bounces. shorter than --help, if you're in a rush.
• -V, --version — prints the current version number and exits. iconic, brief, slays. 👑
• --yaml, --yaml=list, --yaml=<data_parser> — same deal as JSON but make it YAML. dumps the info using the default data_parser plugin or an explicit one with parameters. again, sorting and formatting arguments? ignored. periodt.

---

🔥 PERFORMANCE 🔥

okay this part is the serious one, so listen up fr fr.

running sdiag fires off a remote procedure call to slurmctld. and if enough of those calls come flooding in at once — from sdiag or any other Slurm client command that pings the slurmctld daemon — it can degrade the performance of the slurmctld daemon, and it could even result in a denial of service. not a vibe. not cute. 🚨

Do not run sdiag (or any other Slurm client commands that send remote procedure calls to slurmctld) from loops in shell scripts or other programs. and if you are calling it from a program, make sure that program limits its calls to sdiag to the absolute minimum necessary for the information you're actually trying to gather. like, the bare minimum. pls. 🙏

---

🌸 ENVIRONMENT VARIABLES 🌸

some sdiag options can be set through environment variables. here they are, all matched up with their corresponding options:

Note: command line options will always override these settings. always. no exceptions. the CLI is the final boss. 💅

• SLURM_CLUSTERS — same as --cluster
• SLURM_CONF — the location of the Slurm configuration file.
• SLURM_JSON — controls JSON serialization:
  • compact — output JSON as compact as possible.
  • pretty — output JSON in pretty format to make it more readable.
• SLURM_YAML — controls YAML serialization:
  • compact — output YAML as compact as possible.
  • pretty — output YAML in pretty format to make it more readable.

---

🎀 COPYING 🎀

copyright notices, coming in hot:

Copyright (C) 2010-2011 Barcelona Supercomputing Center.
Copyright (C) 2010-2022 SchedMD LLC.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. 🫶

---

🦋 SEE ALSO 🦋

go bother these other pages when you're done here, iykyk:

• sinfo(1)
• squeue(1)
• scontrol(1)
• slurm.conf(5)

---

and that's the whole page bestie!! you made it to the end, which is genuinely iconic of you. 💖 this edition is from Slurm 26.05, sitting under the Slurm Commands section, and yes it's filed as sdiag(1). go forth and <rainbow>sdiag responsibly` 🌈✨