slurmctld(8)

📦
version 26.05.4 · 6 versions
, in ubuntu stonking

💖 slurmctld — the central management daemon of Slurm, bestie edition 💖

okay so. this page? it's slurmctld(8), from the Slurm Daemon manual fam, and it's literally the main character of your whole cluster. let's get into it 🫶

✨ NAME ✨

slurmctld — The central management daemon of Slurm.

🌸 SYNOPSIS 🌸

slurmctld [OPTIONS...]

👑 DESCRIPTION 👑

slurmctld is THE central management daemon of Slurm. no cap. she monitors all the other Slurm daemons and resources, accepts work (aka jobs 💅), and allocates resources to those jobs. iconic behaviour honestly.

and because her functionality is so critical, there may be a backup server ready to assume these functions if the primary server fails. it's giving understudy energy but make it high availability ✨

🎀 OPTIONS 🎀

  • -c — Clear all previous slurmctld state from its last checkpoint. With this option, all jobs (both running AND queued) and all node states get deleted. Without it, previously running jobs will be preserved, along with node State of DOWN, DRAINED and DRAINING nodes and the associated Reason field for those nodes.
⚠️ NOTE: it is rare you would ever want to use this in production, because all jobs will be killed. bestie. ALL of them. 😭
  • -D — Run slurmctld in the foreground with logging copied to stderr. This limits the resilience of scontrol reconfigure and should be avoided in production. cute for debugging, not for the real world 🙈
  • -f <file> — Read configuration from the specified file. See NOTES below.
  • -h — Help; print a brief summary of command options. the little "what do i do again" button 🥰
  • -i — Ignore errors found while reading in state files on startup.
⚠️ Warning: using this option means losing the data that wasn't recovered from the state files. that data is GONE gone, fr fr.
  • -L <file> — Write log messages to the specified file.
  • -n <value> — Set the daemon's nice value to the specified value, typically a negative number.
  • -r — Recover partial state from last checkpoint: jobs and node DOWN/DRAIN state and reason information state. No partition state is recovered. This is the default action. the no-effort option, we love her 🦋
  • -R — Recover full state from last checkpoint: jobs, node, partition state, and power save settings. Without this option, previously running jobs will be preserved, along with node State of DOWN, DRAINED and DRAINING nodes and the associated Reason field for those nodes. No other node or partition state will be preserved. so like, the full-course meal version 🍓
  • -s — Change working directory of slurmctld to SlurmctldLogFile path if possible, or to Slurm's StateSaveLocation otherwise. If both of them fail, it will fallback to /var/tmp. she's trying her best 💀
  • --systemd — Use when starting the daemon with systemd. This will allow slurmctld to notify systemd of the new PID when using scontrol reconfigure.
⚠️ NOTE: The User and Group options in the slurmctld's systemd unit file need to both specify the SlurmUser. both of them. do not skip one. ✋
  • -v — Verbose operation. Multiple v's can be specified, with each v beyond the first increasing verbosity, up to 6 times (i.e. -vvvvvv). more v's, more tea ☕
  • -V — Print version information and exit.

🌈 ENVIRONMENT VARIABLES 🌈

The following environment variables can be used to override settings compiled into slurmctld.

  • ABORT_ON_FATAL — When a fatal error is detected, use abort() instead of exit() to terminate the process. This allows backtraces to be captured without recompiling Slurm. forensic girlies rise 🕵️
  • SLURM_CONF — The location of the Slurm configuration file. This is overridden by explicitly naming a configuration file on the command line. (so -f wins, iykyk)
  • SLURM_DEBUG_FLAGS — Specify debug flags for the scheduler to use. See DebugFlags in the slurm.conf(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf.

🧚 HTTP server 🧚

Unless disabled via CommunicationParameters=disable_http in slurm.conf, slurmctld will accept incoming HTTP/1.1 compliant requests to any socket listening as configured by SlurmctldPort in slurm.conf. Authentication of HTTP requests is not supported. TLS wrapping is optionally supported without requiring TLSType in slurm.conf.

The following endpoints are currently supported:

  • GET / — Get list of endpoints.
  • GET /healthz — Test if slurmctld loaded successfully.
  • GET /livez — Test if slurmctld loaded successfully.
  • GET /readyz[?verbose] — Test if slurmctld is ready to accept incoming RPCs. Adding the verbose query parameter will result in a verbose output of each probe in the response.
  • GET /metrics/jobs — Get job metrics.
  • GET /metrics/nodes — Get node metrics.
  • GET /metrics/partitions — Get partition metrics.
  • GET /metrics/scheduler — Return scheduler metrics.
  • GET /metrics/jobs-users-accts — Return user and account job metrics.

When multiple SlurmctldHost entries are configured, a daemon in standby mode will answer GET /metrics* with HTTP 303 "See Other" with the Location header pointing to the same path and query on the first listed controller. If no such address is available, the response will be HTTP 503 "Service Unavailable" instead of a redirect. See SlurmctldHost in slurm.conf(5). she's redirecting you to her sister, respectfully 💅

🦋 CORE FILE LOCATION 🦋

  • If slurmctld is started with the -D option, the core file will be written to the current working directory.
  • Otherwise, if SlurmctldLogFile is a fully qualified path name (starting with a slash), the core file will be written to the same directory as the log file, provided SlurmUser has write permission on the directory.
  • Otherwise, the core file will be written to the StateSaveLocation, or "/var/tmp/" as a last resort.
  • If none of the above directories have write permission for SlurmUser, no core file will be produced. 💀 nothing. nada. bye.

⭐ SIGNALS ⭐

  • SIGTERM SIGINT SIGQUIT — slurmctld will shutdown cleanly, saving its current state to the state save directory. a graceful queen 👑
  • SIGABRT — slurmctld will shutdown cleanly, saving its current state, and perform a core dump.
  • SIGHUP — Reloads the slurm configuration files, similar to scontrol reconfigure.
  • SIGTSTP — Stop the process from a terminal. This also stops slurmscriptd.
  • SIGUSR2 — Reread the log level from the configs, and then reopen the log file. This should be used when setting up logrotate(8).
  • SIGPROF — Logs connection manager state when debug level is at least info.
  • SIGCHLD SIGUSR1 SIGXCPU SIGPIPE SIGALRM — These signals are explicitly ignored. not her problem, not her business 🙈

💜 NOTES 💜

It may be useful to experiment with different slurmctld-specific configuration parameters using a distinct configuration file (e.g. timeouts). However, this special configuration file will not be used by the slurmd daemon or the Slurm programs, unless you specifically tell each of them to use it. you have to invite them individually, they don't just show up 🫶

If you desire changing communication ports, the location of the temporary file system, or other parameters used by other Slurm components, change the common configuration file, slurm.conf.

💖 COPYING 💖

Copyright (C) 2002-2007 The Regents of the University of California. Copyright (C) 2008-2010 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). CODE-OCEC-09-009. All rights reserved.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

🌟 SEE ALSO 🌟

and that's the whole page bestie, no skips, no crumbs left behind 💅✨ this was Slurm 26.05, from the Slurm Daemon manual set, and you can find slurmctld(8) right there in the same corner of the man pages. go run your cluster, you've got this 🫶🎀

📦 where this page comes from