sinfo(1)

📦
version 26.05.5.1 · 8 versions
, in arch rolling, epel 10, nixpkgs unstable, ubuntu stonking

💖 sinfo(1) — Slurm Commands 💖

okay bestie, welcome to the sinfo(1) page from the Slurm Commands manual. this is literally the vibe check for your Slurm cluster's nodes and partitions. periodt. 🦋

✨ NAME ✨

sinfo — view information about Slurm nodes and partitions. 👑

🌸 SYNOPSIS 🌸

sinfo [OPTIONS...]

🎀 DESCRIPTION 🎀

sinfo is the tool you use to view partition and node information for a system running Slurm. it's giving "let me see what's going on in this cluster" and it ate. 💅

🔥 OPTIONS 🔥

  • -a, --all — display info about all partitions. this makes info show up for partitions configured as hidden and partitions that are unavailable to your user's group. no gatekeeping. ✨
  • -M, --clusters=<string> — clusters to issue commands to. multiple cluster names can be comma separated. a value of 'all' will query all clusters. note that the slurmdbd must be up for this option to work properly, unless running in a federation with either FederationParameters=fed_display configured or the --federation option set. this option implicitly sets the --local option. 🫶
  • -d, --dead — only show nodes that have unexpectedly stopped responding. this may not show nodes in a POWERED_DOWN state since they are expected to be down. 💀
  • -e, --exact — if set, do not group node information on multiple nodes unless their configurations to be reported are identical. otherwise cpu count, memory size, and disk space for nodes will be listed with the minimum value followed by a + for nodes with the same partition and state (e.g. "250+"). bestie, exact means exact. 🎯
  • --federation — show all partitions from the federation if a member of one. 🌐
  • -F, --future — report nodes in FUTURE state. 🔮
  • -o, --format=<output_format> — specify the information to be displayed using an sinfo format string. if the command is executed in a federated cluster environment and information about more than one cluster is to be displayed and the -h, --noheader option is used, then the cluster name will be displayed before the default output formats shown below. format strings transparently used by sinfo when running with various options are:
default        "%#P %.5a %.10l %.6D %.6t %N"

--summarize    "%#P %.5a %.10l %.16F  %N"

--long         "%#P %.5a %.10l %.10s %.4r %.8h %.10g %.6D %.11T %.11i %N"

--Node         "%#N %.6D %#P %6t"

--long --Node  "%#N %.6D %#P %.11T %.4c %.8z %.6m %.8d %.6w %.8f %20E"

--list-reasons "%20E %9u %19H %N"

--long --list-reasons
               "%20E %12U %19H %6t %N"

in the above format strings, the use of # represents the maximum length of any partition name or node list to be printed. a pass is made over the records to be printed to establish the size in order to align the sinfo output, then a second pass is made over the records to print them. note that the literal character # itself is not a valid field length specification, but is only used to document this behavior.

the format of each field is "%[[.]size]type[suffix]"

  • size — minimum field size. if no size is specified, whatever is needed to print the information will be used.
  • . — indicates the output should be right justified and size must be specified. by default output is left justified.
  • suffix — arbitrary string to append to the end of the field.

valid type specifications include:

  • %all — print all fields available for this data type with a vertical bar separating each field.
  • %a — state/availability of a partition.
  • %A — number of nodes by state in the format "allocated/idle". do not use this with a node state option ("%t" or "%T") or the different node states will be placed on separate lines.
  • %b — features currently active on the nodes, also see %f.
  • %B — the max number of CPUs per node available to jobs in the partition.
  • %c — number of CPUs per node.
  • %C — number of CPUs by state in the format "allocated/idle/other/total". do not use this with a node state option ("%t" or "%T") or the different node states will be placed on separate lines.
  • %d — size of temporary disk space per node in mebibytes.
  • %D — number of nodes.
  • %e — the total memory, in MiB, currently free on the node as reported by the OS. this value is for informational use only and is not used for scheduling.
  • %E — the reason a node is unavailable (down, drained, or draining states).
  • %f — features available the nodes, also see %b.
  • %F — number of nodes by state in the format "allocated/idle/other/total". note the use of this format option with a node state format option ("%t" or "%T") will result in the different node states being be reported on separate lines.
  • %g — groups which may use the nodes.
  • %G — generic resources (gres) associated with the nodes.
  • %h — print the OverSubscribe setting for the partition.
  • %H — print the timestamp of the reason a node is unavailable.
  • %i — if a node is in an advanced reservation print the name of that reservation.
  • %I — partition job priority weighting factor.
  • %l — maximum time for any job in the format "days-hours:minutes:seconds".
  • %L — default time for any job in the format "days-hours:minutes:seconds".
  • %m — size of memory per node in mebibytes.
  • %M — PreemptionMode.
  • %n — list of node hostnames.
  • %N — list of node names.
  • %o — list of node communication addresses.
  • %O — CPU load of a node as reported by the OS.
  • %p — partition scheduling tier priority.
  • %P — partition name followed by "*" for the default partition, also see %R.
  • %r — only user root may initiate jobs, "yes" or "no".
  • %R — partition name, also see %P.
  • %s — maximum job size in nodes.
  • %S — allowed allocating nodes.
  • %t — state of nodes, compact form.
  • %T — state of nodes, extended form.
  • %u — print the user name of who set the reason a node is unavailable.
  • %U — print the user name and uid of who set the reason a node is unavailable.
  • %v — print the running slurmd version. if reporting on a node list, print the version of the first node in the list.
  • %V — print the cluster name if running in a federation.
  • %w — scheduling weight of the nodes.
  • %X — number of sockets per node.
  • %Y — number of cores per socket.
  • %Z — number of threads per core.
  • %z — extended processor information: number of sockets, cores, threads (S:C:T) per node.
  • -O, --Format=<output_format> — specify the information to be displayed. also see the -o <output_format>, --format=<output_format> option (which supports greater flexibility in formatting, but does not support access to all fields because we ran out of letters). requests a comma separated list of job information to be displayed.

the format of each field is "type[:[.][size][suffix]]"

  • size — the maximum field size. if no size is specified, 20 characters will be allocated to print the information.
  • . — indicates the output should be right justified and size must be specified. by default, output is left justified.
  • suffix — arbitrary string to append to the end of the field.

valid type specifications include:

  • All — print all fields available in the -o format for this data type with a vertical bar separating each field.
  • AllocMem — prints the amount of allocated memory on a node.
  • AllocNodes — allowed allocating nodes.
  • Available — state/availability of a partition.
  • Cluster — print the cluster name if running in a federation.
  • Comment — comment. (arbitrary descriptive string)
  • Cores — number of cores per socket.
  • CPUs — number of CPUs per node.
  • CPUsLoad — CPU load of a node as reported by the OS.
  • CPUsState — number of CPUs by state in the format "allocated/idle/other/total". do not use this with a node state option ("%t" or "%T") or the different node states will be placed on separate lines.
  • DefaultTime — default time for any job in the format "days-hours:minutes:seconds".
  • Disk — size of temporary disk space per node in mebibytes.
  • Extra — arbitrary string on the node.
  • Features — features available on the nodes. also see features_act.
  • features_act — features currently active on the nodes. also see features.
  • FreeMem — the total memory, in MiB, currently free on the node as reported by the OS. this value is for informational use only and is not used for scheduling.
  • Gres — generic resources (gres) associated with the nodes.
  • GresUsed — generic resources (gres) currently in use on the nodes. for shared GRES usage details, see https://slurm.schedmd.com/gres.html#Shared_GRES_Usage.
  • Groups — groups which may use the nodes.
  • MaxCPUsPerNode — the max number of CPUs per node available to jobs in the partition.
  • Memory — size of memory per node in mebibytes.
  • NodeAddr — list of node communication addresses.
  • NodeAI — number of nodes by state in the format "allocated/idle". do not use this with a node state option ("%t" or "%T") or the different node states will be placed on separate lines.
  • NodeAIOT — number of nodes by state in the format "allocated/idle/other/total". do not use this with a node state option ("%t" or "%T") or the different node states will be placed on separate lines.
  • NodeHost — list of node hostnames.
  • NodeList — list of node names.
  • Nodes — number of nodes.
  • OverSubscribe — can jobs oversubscribe compute resources (like CPUs)? yes/no energy ✨
  • Partition — the partition name, with a * slapped on at the end if it's the default partition. also peep %R 👀
  • PartitionName — just the plain partition name. also see %P.
  • Port — node TCP port 🌐
  • PreemptMode — the preemption mode, duh
  • PriorityJobFactor — the partition factor the priority/multifactor plugin uses when it's calculating job priority 🧮
  • PriorityTier or Priority — partition scheduling tier priority, the hierarchy vibes 👑
  • Reason — why a node is unavailable (aka it's in a down, drained, or draining state). it's giving "we have our reasons" 💀
  • Size — max job size, counted in nodes
  • SocketCoreThread — extended processor tea ☕: number of sockets, cores, threads (S:C:T) per node
  • Sockets — how many sockets per node
  • StateCompact — node state, but make it compact
  • StateLong — node state, extended form, the deluxe edition
  • StateComplete — node state with every single node state flag included, e.g. "idle+cloud+power" — it's showing off fr fr
  • Threads — number of threads per core 🧵
  • Time — max time for any job, formatted "days-hours:minutes:seconds"
  • TimeStamp — print the timestamp of the reason a node is unavailable ⏰
  • User — print the user name of the person who set that reason a node is unavailable
  • UserLong — print the user name and uid of whoever set the reason a node is unavailable
  • Version — print the running slurmd version. if you're reporting on a node list, it prints the version of the first node in the list 🔢
  • Weight — scheduling weight of the nodes ⚖️

okay the option buffet continues, stay with me bestie 💅

  • --help — prints a message describing all the sinfo options. literally all of them, it's a lot, we love that for it 🫶
  • --hide — don't display info about hidden partitions. partitions configured as hidden, or ones that just aren't available to your group, won't show up. heads up: this is the default behavior already, so it's giving "quiet luxury" 🎀
  • -i, --iterate=<seconds> — prints the state on a periodic basis, sleeping for the number of seconds you gave it between reports. by default it prints a time stamp along with the header 🔁
  • --json, --json=list, --json=<data_parser> — dumps info as JSON using the default data_parser plugin, or an explicit data_parser with parameters. all information gets dumped, even the stuff that normally wouldn't be shown ✨ note that sorting and formatting arguments you passed to other options get ignored — however, most filtering arguments are still used
  • -R, --list-reasons — lists the reasons nodes are in the down, drained, fail or failing state. when nodes are in these states, Slurm lets an administrator add a "reason" string. this option shows the first 20 characters of the reason field plus the list of nodes with that reason, for all nodes that are — by default — down, drained, draining or failing. you can pair it with other node filtering options (like -r, -d, -t, -n), but combinations that end up producing a list of nodes that aren't down, drained or failing will produce zero output, bestie, just an empty void 🕳️ when you use it with -l, the output also includes the current node state
  • --local — show only jobs local to this cluster. ignore the other clusters in this federation (if there even are any). this overrides --federation 🏠
  • -l, --long — prints more detailed information. this one gets ignored if you specified the --format option, so don't be surprised 🙈
  • --noconvert — don't convert units from their original type (e.g. 2048M won't get converted to 2G). the units stay exactly as they were, no glow-up 🔒
  • -N, --Node — prints info in a node-oriented format, with one line per node and partition. so if a node belongs to more than one partition, you get one line for each node-partition pair 📋 if --partition is also specified, then you get only one line per node in this partition. the default is to print in a partition-oriented format. this one is also ignored if the --format option is specified
  • -n, --nodes=<nodes> — prints info about the specified node(s). multiple nodes can be comma separated or given as a node range expression (e.g. "linux[00-17]"). ngl, limiting the query to just the relevant nodes can measurably improve the performance of the command on large clusters, so do that 💨
  • -h, --noheader — don't print a header on the output. minimalist era 🎀
  • -p, --partition=<partition> — prints info about the node(s) in the specified partition(s). multiple partitions get separated by commas
  • -T, --reservation — only display info about Slurm reservations
NOTE: this option causes sinfo to ignore most other options, which are focused on partition and node information. it said "my reservations only" 😭
  • -r, --responding — excludes any nodes that have unexpectedly stopped responding. this is the exact opposite of --dead 💀
  • -S, --sort=<sort_list> — specifies the order in which records should be reported. it uses the same field specification as the <output_format>. you can do multiple sorts by listing multiple sort fields separated by commas. field specs can be preceded by "+" or "-" for ascending (the default) and descending order respectively. the partition field specification, "P", may be preceded by a "#" to report partitions in the same order they appear in Slurm's configuration file, slurm.conf. example: a sort value of "+P,-m" asks for records printed in order of increasing partition name, and within a partition by decreasing memory size. the default value of sort is "#P,-t" (partitions ordered as configured, then decreasing node state). if the --Node option is selected, the default sort value is "N" (increasing node name) 📊
  • -t, --states=<states> — lists nodes only having the given state(s). multiple states can be comma separated and the comparison is case insensitive. if the states are separated by '+', then the nodes must be in all of those states. the state can be prefixed with '~' which will invert the result of the match 🔄 possible values include (case insensitive): ALLOC, ALLOCATED, BLOCKED, CLOUD, COMP, COMPLETING, DOWN, DRAIN (for a node in DRAINING or DRAINED states), DRAINED, DRAINING, FAIL, FUTURE, FUTR, IDLE, MAINT, MIX, MIXED, NO_RESPOND, PLANNED, POWER_DOWN, POWERING_DOWN, POWERED_DOWN, POWERING_UP, REBOOT_ISSUED, REBOOT_REQUESTED, RESV, RESERVED, UNK, and UNKNOWN. by default, nodes in the specified state are reported whether they're responding or not. the --dead and --responding options can be used to filter nodes by the corresponding flag
  • -s, --summarize — lists only a partition state summary, with no node state details. this gets ignored if the --format option is specified 🧾
  • --usage — prints a brief message listing the sinfo options ⚡
  • -v, --verbose — provides detailed event logging through program execution. it's giving narrator voice 🎙️
  • -V, --version — prints version information and exits 👋
  • --yaml, --yaml=list, --yaml=<data_parser> — dumps information as YAML using the default data_parser plugin, or an explicit data_parser with parameters. all information is dumped, even if it normally wouldn't be 💖 sorting and formatting arguments passed to other options are ignored; however, most filtering arguments are still used

✨ OUTPUT FIELD DESCRIPTIONS ✨

Okay bestie, here's the ✨tea✨ on every single thing sinfo can print at you. Each field, decoded, no gatekeeping 💖

  • AVAIL — the partition's state, babe. It can be up, down, drain, or inact (that's the one for INACTIVE). Wanna know more? Check the partition definition's State parameter in the slurm.conf(5) man page 📖
  • CPUS — how many CPUs (processors, same thing) live on these nodes.
  • S:C:T — the count of sockets (S), cores (C), and threads (T) on these nodes. It's giving inventory 🎀
  • SOCKETS — just the socket count on these nodes.
  • CORES — just the core count on these nodes.
  • THREADS — just the thread count on these nodes.
  • GROUPS — resource allocations in this partition are restricted to the named groups. And all means, literally, all groups may use this partition. everyone's invited 🫶
  • JOB_SIZE — the minimum and maximum node count that can be handed to any user job. If you see a single number, that means the min and max are the same. And infinite is how we say "no maximum node count, sky's the limit" ☁️
  • TIMELIMIT — the maximum time limit for any user job, written as days-hours:minutes:seconds. And infinite = partitions with no job time limit at all. queen behavior, no curfew 👑
  • MEMORY — size of real memory on these nodes, in mebibytes.
  • NODELIST — the names of the nodes chillin' in this particular configuration.
  • NODES — the count of nodes with this particular configuration.
  • NODES(A/I) — count of nodes in this particular configuration, broken down by node state as "allocated/idle".
  • NODES(A/I/O/T) — same vibe, more detail: count by node state as "allocated/idle/other/total".
  • PARTITION — the name of a partition. And heads up: that little "*" suffix? That's how you spot the default partition 👀
  • PORT — the local TCP port that slurmd uses on the node.
  • ROOT — is the ability to allocate resources in this partition restricted to user root, yes or no. Simple as that 💅
  • OVERSUBSCRIBE — whether the partition allows more than one job at a time on the same resource (that's node, socket, or core per SelectTypeParameters), plus the maximum share when that applies. The options go like this:
    • NO: no sharing at all, only one job per resource. strict era 🙅
    • YES:n: sharing is allowed when the job asks for --oversubscribe, up to n jobs per resource.
    • FORCE:n: sharing is always allowed, up to n jobs per resource (and jobs literally cannot opt out). no takebacks ✋
  • STATE — the state of the nodes. Possible states include: allocated, blocked, completing, down, drained, draining, fail, failing, future, idle, maint, mixed, planned, power_down, power_up, reserved, and unknown. Their abbreviated forms are: alloc, block, comp, down, drain, drng, fail, failg, futr, idle, maint, mix, plnd, pow_dn, pow_up, resv, and unk respectively.
NOTE: that "*" suffix means those nodes are presently not responding. not me ghosting the controller 😭
  • TMP_DISK — the size of temporary disk space on these nodes, in mebibytes.

💀 NODE STATE CODES 💀

So the node state codes get shortened as needed to fit the field size — they're flexible like that. And these node states may be followed by a special character to identify state flags associated with the node. Here's the full roster of suffixes and states, bestie:

  • * — the node is presently not responding and will not be allocated any new work. If it stays non-responsive, it'll get placed in the DOWN state (except for COMPLETING, DRAINED, DRAINING, FAIL, FAILING nodes — they're the exception squad).
  • ~ — the node is presently in powered off. sleep mode fr 😴
  • # — the node is presently being powered up or configured.
  • ! — the node is pending power down.
  • % — the node is presently being powered down.
  • $ — the node is currently in a reservation with a flag value of "maintenance".
  • @ — the node is pending reboot.
  • ^ — the node reboot was issued.
  • - — the node is planned by the backfill scheduler for a higher priority job. prioritizing, iconic ⭐
  • ALLOCATED — the node has been allocated to one or more jobs.
  • ALLOCATED+ — the node is allocated to one or more active jobs, plus one or more jobs are in the process of COMPLETING.
  • BLOCKED — the node has been blocked by exclusive topo job.
  • COMPLETING — all jobs associated with this node are in the process of COMPLETING. This node state gets removed when all of the job's processes have terminated and the Slurm epilog program (if any) has terminated too. And while a node is in the COMPLETING state, it is not eligible for new job placement, and no jobs will be scheduled on it. See the Epilog parameter description in the slurm.conf(5) man page for more info.
  • DOWN — the node is unavailable for use. Slurm can automatically drop nodes into this state if some failure occurs, and system administrators may also explicitly place nodes in this state. But if a node resumes normal operation, Slurm can automatically return it to service 🫶 See the ReturnToService and SlurmdTimeout parameter descriptions in the slurm.conf(5) man page.
  • DRAINED — the node is unavailable for use per system administrator request. See the update node command in the scontrol(1) man page or the slurm.conf(5) man page.
  • DRAINING — the node is currently allocated a job, but will not be allocated additional jobs. The node state changes to DRAINED once the last job on it completes. Nodes enter this state per system administrator request. See the update node command in the scontrol(1) man page or the slurm.conf(5) man page.
  • FAIL — the node is expected to fail soon and is unavailable for use per system administrator request. See the update node command in the scontrol(1) man page or the slurm.conf(5) man page.
  • FAILING — the node is currently executing a job, but it's expected to fail soon and is unavailable for use per system administrator request. See the update node command in the scontrol(1) man page or the slurm.conf(5) man page.
  • FUTURE — the node is currently not fully configured, but it's expected to be available at some point in the indefinite future for use. manifesting ✨
  • IDLE — the node is not allocated to any jobs and is available for use. living its best life 🌸
  • INVAL — the node did not register correctly with the controller. This happens when a node registers with less resources than what's configured in the slurm.conf file. The node will clear from this state with a valid registration (i.e. a slurmd restart is required).
  • MAINT — the node is currently in a reservation with a flag value of "maintenance".
  • REBOOT_ISSUED — a reboot request has been sent to the agent configured to handle this request.
  • REBOOT_REQUESTED — a request to reboot this node has been made, but it hasn't been handled yet. pending, bestie ⏳
  • MIXED — the node has some of its CPUs ALLOCATED while others are IDLE. Or, the node has a suspended job allocated to some of its TRES (e.g. memory). both things at once, multitalented 💅
  • PLANNED — the node is planned by the backfill scheduler for a higher priority job.
  • POWER_DOWN — the node is pending power down.
  • POWERED_DOWN — the node is currently powered down and not capable of running any jobs.
  • POWERING_DOWN — the node is in the process of powering down and not capable of running any jobs.
  • POWERING_UP — the node is in the process of being powered up.
  • RESERVED — the node is in an advanced reservation and not generally available.
  • UNKNOWN — the Slurm controller has just started and the node's state has not yet been determined. mystery era 🔮

🔥 PERFORMANCE 🔥

Okay so real talk, this part matters. Executing sinfo sends a remote procedure call to slurmctld. And if enough calls from sinfo — or other Slurm client commands that send remote procedure calls to the slurmctld daemon — come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service. not the vibe, we do not want that 💀

Do not run sinfo or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Seriously. Don't. 🙈

And make sure your programs limit calls to sinfo to the minimum necessary for the information you're actually trying to gather. be mindful, bestie 🫶

🫶 ENVIRONMENT VARIABLES 🫶

okay so bestie, some of these sinfo options can be set through environment variables too, which is giving lazy-girl-enablement in the best way 💅. Here's every single one of them with the option they're basically cosplaying as. NOTE (and this is the important part ✨): command line options will always override these settings, no cap. Env vars are the vibe, but the command line is the main character. 👑

  • SINFO_ALL — same as -a, --all 🌈
  • SINFO_FEDERATION — same as --federation 🦋
  • SCONTROL_FUTURE — same as -F, --future ⭐
  • SINFO_FORMAT — same as -o <output_format>, --format=<output_format> 🎀
  • SINFO_LOCAL — same as --local 💜
  • SINFO_PARTITION — same as -p <partition>, --partition=<partition> 🌸
  • SINFO_SORT — same as -S <sort>, --sort=<sort> 💖
  • SLURM_CLUSTERS — same as --clusters 🧚
  • SLURM_CONF — the location of the Slurm configuration file. 📁
  • SLURM_DEBUG_FLAGS — specify debug flags for sinfo to use. Peep DebugFlags in the slurm.conf(5) man page for the full list of flags. This environment variable takes precedence over the setting in the slurm.conf. 👑
  • SLURM_JSON — controls JSON serialization:
    • compact — output JSON as compact as possible.
    • pretty — output JSON in pretty format to make it more readable. ✨
  • SLURM_TIME_FORMAT — specifies the format used to report time stamps. A value of standard (which is the default value, btw) generates output in the form "year-month-dateThour:minute:second". A value of relative returns only "hour:minute:second" if it's the current day. For other dates in the current year it prints the "hour:minute" preceded by "Tomorr" (tomorrow), "Ystday" (yesterday), or the name of the day for the coming week (like "Mon", "Tue", etc.), otherwise it's the date (like "25 Apr"). For years that aren't the current one, it returns a date, month and year with no time at all (like "6 Jun 2012"). All of the time stamps use a 24 hour format. ⏰

A valid strftime() format can also be specified, bestie. For example, a value of "%a %T" will report the day of the week and a time stamp (like "Mon 12:34:56"). iconic behavior fr fr 💖

  • SLURM_YAML — controls YAML serialization:
    • compact — output YAML as compact as possible.
    • pretty — output YAML in pretty format to make it more readable. 🌸

🌈 EXAMPLES 🌈

here's where it gets fun, bestie — let's actually run this thing 🫶

Report basic node and partition configurations:

$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE  NODELIST
batch     up     infinite     2 alloc  adev[8-9]
batch     up     infinite     6 idle   adev[10-15]
debug*    up        30:00     8 idle   adev[0-7]

Report partition summary information:

$ sinfo -s
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
batch     up     infinite 2/6/0/8        adev[8-15]
debug*    up        30:00 0/8/0/8        adev[0-7]

Report more complete information about the partition debug:

$ sinfo --long --partition=debug
PARTITION AVAIL TIMELIMIT JOB_SIZE ROOT OVERSUBS GROUPS NODES STATE NODELIST
debug*    up        30:00        8 no   no       all        8 idle  dev[0-7]

Report only those nodes that are in state DRAINED:

$ sinfo --states=drained
PARTITION AVAIL NODES TIMELIMIT STATE  NODELIST
debug*    up        2     30:00 drain  adev[6-7]

Report node-oriented information with details and exact matches:

$ sinfo -Nel
NODELIST    NODES PARTITION STATE  CPUS MEMORY TMP_DISK WEIGHT FEATURES REASON
adev[0-1]       2 debug*    idle      2   3448    38536     16 (null)   (null)
adev[2,4-7]     5 debug*    idle      2   3384    38536     16 (null)   (null)
adev3           1 debug*    idle      2   3394    38536     16 (null)   (null)
adev[8-9]       2 batch     allocated 2    246    82306     16 (null)   (null)
adev[10-15]     6 batch     idle      2    246    82306     16 (null)   (null)

Report only down, drained and draining nodes and their reason field:

$ sinfo -R
REASON                              NODELIST
Memory errors                       dev[0,5]
Not Responding                      dev8

💜 COPYING 💜

Copyright (C) 2002-2007 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). Copyright (C) 2008-2009 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. 🫶

🦋 SEE ALSO 🦋

the extended slay-verse of friends and neighbors, go give them a read bestie:

  • scontrol(1)
  • squeue(1)
  • slurm_load_ctl_conf (3)
  • slurm_load_jobs (3)
  • slurm_load_node (3)
  • slurm_load_partitions (3)
  • slurm_reconfigure (3)
  • slurm_shutdown (3)
  • slurm_update_job (3)
  • slurm_update_node (3)
  • slurm_update_partition (3)
  • slurm.conf(5)

and that's the whole page, bestie!! 🎀 you just read the entire sinfo(1) manual from top to bottom, which is lowkey iconic of you. this masterpiece is brought to you by Slurm 26.05, filed under the Slurm Commands girlies, and it ate, no crumbs. go check your nodes, stay drained-free, and remember: the command line always overrides the env var. 💅✨👑

📦 where this page comes from