slurmctld(8)

📦
version 26.05.5.1 · 11 versions
, in arch rolling, epel 10, nixpkgs unstable

💖 slurmctld(8) — the Slurm Daemon girlie herself 💖

okay so bestie, welcome to section 8 of the manual, the one and only slurmctld page. this is the main-character daemon of Slurm and she's about to explain herself. buckle up 🎀

✨ NAME ✨

slurmctld — the central management daemon of Slurm. yes. the central one. the queen bee 👑

🦋 SYNOPSIS 🦋

slurmctld [OPTIONS...]

🌸 DESCRIPTION 🌸

okay so slurmctld is literally the central management daemon of Slurm. she monitors all the other Slurm daemons and resources, she accepts work (that's jobs, bestie), and she allocates resources to those jobs. it's giving head girl energy 👑

and because her functionality is so critical, you can have a backup server ready to assume all of these functions in the event that the primary server fails. a literal understudy. slay 💅

✨ OPTIONS ✨

  • -c — Discard previous slurmctld state from its last checkpoint. and bestie, this is brutal: all running and queued jobs will be killed and all node states reset, including the State and Reason fields of DOWN, DRAINED, and DRAINING nodes. The last job ID, reservation state, and association/QOS usage are unaffected by this option. Dynamically created nodes are not recreated, and reservations naming them lose those nodes and may be deleted. If the state is unreadable due to a version mismatch, slurmctld will still refuse to start unless -i is also given.
⚠️ NOTE: It is rare you would ever want to use this in production as all jobs will be killed. like, for real. don't. 😭
  • -D — Run slurmctld in the foreground with logging copied to stderr. This limits the resilience of scontrol reconfigure and should be avoided in production. cute for debugging, not cute for prod 🫶
  • -f <file> — Read configuration from the specified file. See NOTES below. 👀
  • -h — Help; print a brief summary of command options. classic.
  • -i — Ignore errors found while reading in state files on startup.
⚠️ Warning: Use of this option will mean losing the data that wasn't recovered from the state files. gone. lost. poof. 💀
  • -L <file> — Write log messages to the specified file.
  • -n <value> — Set the daemon's nice value to the specified value, typically a negative number. be nice. or don't 🎀
  • -r — Recover partial state from last checkpoint: jobs and node DOWN/DRAIN state and reason information state. No partition state is recovered. This is the default action. (so if you do nothing, this is what you get ✨)
  • -R — Recover full state from last checkpoint: jobs, node, partition state, and power save settings. Without this option, previously running jobs will be preserved along with node State of DOWN, DRAINED and DRAINING nodes and the associated Reason field for those nodes. No other node or partition state will be preserved.
  • -s — Change working directory of slurmctld to SlurmctldLogFile path if possible, or to Slurm's StateSaveLocation otherwise. If both of them fail it will fallback to /var/tmp. it's giving "wherever i end up, i end up" 💅
  • --systemd — Use when starting the daemon with systemd. This will allow slurmctld to notify systemd of the new PID when using scontrol reconfigure.
⚠️ NOTE: The User and Group options in the slurmctld's systemd unit file need to both specify the SlurmUser. both of them. yes. 🫶
  • -v — Verbose operation. Multiple v's can be specified, with each 'v' beyond the first increasing verbosity, up to 6 times (i.e. -vvvvvv). six v's max, bestie, that's the whole ladder 🌈
  • -V — Print version information and exit.

🧚 ENVIRONMENT VARIABLES 🧚

The following environment variables can be used to override settings compiled into slurmctld. yes, you can vibe-check her via env vars 💜

  • ABORT_ON_FATAL — When a fatal error is detected, use abort() instead of exit() to terminate the process. This allows backtraces to be captured without recompiling Slurm. very forensic of you 🔥
  • SLURM_CONF — The location of the Slurm configuration file. This is overridden by explicitly naming a configuration file on the command line. (command line stays winning 👑)
  • SLURM_DEBUG_FLAGS — Specify debug flags for the scheduler to use. See DebugFlags in the slurm.conf(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf.

🦋 HTTP server 🦋

Unless disabled via CommunicationParameters=disable_http in slurm.conf, slurmctld will accept incoming HTTP/1.1 compliant requests to any socket listening as configured by SlurmctldPort in slurm.conf. Authentication of HTTP requests is not supported. TLS wrapping optionally supported without requiring TLSType in slurm.conf. very chill of her, honestly 🙈

The following endpoints are currently supported:

  • GET / — Get list of endpoints.
  • GET /healthz — Test if slurmctld loaded successfully.
  • GET /livez — Test if slurmctld loaded successfully.
  • GET /readyz[?verbose] — Test if slurmctld is ready to accept incoming RPCs. Adding the verbose query parameter will result in a verbose output of each probe in the response.
  • GET /metrics/jobs — Get job metrics.
  • GET /metrics/nodes — Get node metrics.
  • GET /metrics/partitions — Get partition metrics.
  • GET /metrics/scheduler — Return scheduler metrics.
  • GET /metrics/jobs-users-accts — Return user and account job metrics.

When multiple SlurmctldHost entries are configured, a daemon in standby mode will answer GET /metrics* with HTTP 303 "See Other" with the Location header pointing to the same path and query on the first listed controller. If no such address is available, the response will be HTTP 503 "Service Unavailable" instead of a redirect. See SlurmctldHost in slurm.conf(5). she's literally forwarding your call, so polite 💅

🎀 CORE FILE LOCATION 🎀

If slurmctld is started with the -D option then the core file will be written to the current working directory.

Otherwise, if SlurmctldLogFile is a fully qualified path name (starting with a slash), the core file will be written to the same directory as the log file, provided SlurmUser has write permission on the directory.

Otherwise the core file will be written to the StateSaveLocation, or "/var/tmp/" as a last resort.

If none of the above directories have write permission for SlurmUser, no core file will be produced. and honestly? she tried. 😭

🔥 SIGNALS 🔥

  • SIGTERM SIGINT SIGQUIT — slurmctld will shutdown cleanly, saving its current state to the state save directory. graceful queen behavior ✨
  • SIGABRT — slurmctld will shutdown cleanly, saving its current state, and perform a core dump. dramatic exit, still tidy 💀
  • SIGHUP — Reloads the slurm configuration files, similar to scontrol reconfigure.
  • SIGTSTP — Stop the process from a terminal. This also stops slurmscriptd. (yes, she takes slurmscriptd with her 🫶)
  • SIGUSR2 — Reread the log level from the configs, and then reopen the log file. This should be used when setting up logrotate(8).
  • SIGPROF — Logs connection manager state when debug level is at least info.
  • SIGCHLD SIGUSR1 SIGXCPU SIGPIPE SIGALRM — These signals are explicitly ignored. not her problem, blocked, unbothered 💅

💜 NOTES 💜

It may be useful to experiment with different slurmctld specific configuration parameters using a distinct configuration file (e.g. timeouts). However, this special configuration file will not be used by the slurmd daemon or the Slurm programs, unless you specifically tell each of them to use it. If you desire changing communication ports, the location of the temporary file system, or other parameters used by other Slurm components, change the common configuration file, slurm.conf. iykyk — one config for the whole friend group 🌈

👑 COPYING 👑

Copyright (C) 2002-2007 The Regents of the University of California. Copyright (C) 2008-2010 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). CODE-OCEC-09-009. All rights reserved.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

⭐ SEE ALSO ⭐

slurm.conf(5), slurmd(8) — go read those too bestie, they're icons in their own right 🫶

and that's the whole page! this was slurmctld(8) from the Slurm Daemon manual, version Slurm 26.05 — same version, same manual section, all the facts intact, zero notes lost. mwah 💋✨🎀

📦 where this page comes from