slurmd(8)

📦
version 26.05.5.1 · 10 versions
, in arch rolling, epel 10, nixpkgs unstable

🌸 slurmd(8) — the Slurm Daemon, but make it make sense 💅

okay bestie, so this is slurmd, and it's literally the compute node daemon of Slurm. Like, this is the girl on the node doing the actual work. She monitors all the tasks running on the compute node, accepts work (tasks), launches tasks, and — when somebody asks nicely (or not nicely, whatever) — kills running tasks upon request. main character energy on every single node, no cap 👑

✨ SYNOPSIS ✨

slurmd [OPTIONS...]

That's it. That's the whole vibe. Options go here, technically optional, practically you'll want some.

🦋 DESCRIPTION 🦋

slurmd is the compute node daemon of Slurm. Think of her as the node's resident bestie: she monitors all tasks running on the compute node, accepts work (tasks), launches tasks, and kills running tasks upon request. She's not the boss (that's slurmctld, we don't talk about her rn) — she's the one who actually does the thing 💪

🎀 OPTIONS 🎀

so many flags, but we're getting through all of them, no skipping, periodt.

  • --authinfo — used with configless to set an alternate AuthInfo parameter, which gets used to establish communication with slurmctld before the configuration file has been retrieved. (Like, e.g., to specify an alternate MUNGE socket location.) 🫶
  • -b — report node rebooted when daemon restarted. Used for testing purposes.
  • -c — discard the saved slurmd state and kill all jobs and job steps running on this node.

fr fr though: restarting slurmd without -c is normally sufficient after an abnormal termination. The independent slurmstepd(8) processes handle the execution of job steps and are picked up again on restart. Use -c when the jobs on this node should actually be killed. This is the "nuke it" option, not the "fix it" option 😭

  • -C — print the actual hardware configuration (not the configuration from the slurm.conf file) and exit. The format of output is the same as used in slurm.conf to describe a node's configuration plus its uptime. You may use --parameters to specify options such as 'l3cache_as_socket' which may alter the reported node topology.
  • --ca-cert-file <file> — absolute path to CA certificate used for fetching configuration when running configless in a TLS enabled cluster. 🔐
  • --conf <node parameters> — used in conjunction with the -Z option. Used to override or define additional parameters of a dynamic node using the same syntax and parameters used to define nodes in the slurm.conf. Specifying any of CPUs, Boards, SocketsPerBoard, CoresPerSocket or ThreadsPerCore will override the defaults defined by the -C option. NodeName and Port are not supported. This can also be used to specify a cloud node's topology, in which case the -Z flag should not be used (e.g., --conf="topology=topo-switch:s1").

For example if slurmd -C reports

NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848

the following --conf specifications will generate the corresponding node definitions:

--conf "Gres=gpu:2"
NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848 Gres=gpu:2
--conf "RealMemory=30000"
NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=30000
--conf "CPUs=16"
NodeName=node1 CPUs=16 RealMemory=331848
--conf "CPUs=16 RealMemory=30000 Gres=gpu:2"
NodeName=node1 CPUs=16 RealMemory=30000 Gres=gpu:2"

GPU detection occurs automatically when:

  • --conf does not specify any GPU GRES
  • gres.conf does not contain any explicit GPU GRES records for the node
  • gres.conf does not contain AutoDetect=off

If AutoDetect in gres.conf specifies a particular detection mechanism (e.g., NVML, NVIDIA, RSMI, oneAPI, NRT), it will use that mechanism to try to detect a GPU. If no AutoDetect is specified, the slurmd defaults to the full GPU detection method.

The following shows GPU autodetection behavior for a node with 2 local A100 GPUs 🌈:

no --conf, no AutoDetect in gres.conf:

NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848 Gres=gpu:a100:2

--conf "Gres=tmpfs:1000", no AutoDetect in gres.conf:

NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848 Gres=tmpfs:1000,gpu:a100:2

--conf "Gres=tmpfs:1000", AutoDetect=nvml in gres.conf:

NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848 Gres=tmpfs:1000,gpu:a100:2

--conf "Gres=tmpfs:1000", AutoDetect=off in gres.conf:

NodeName=node1 CPUs=16 Boards=1 SocketsPerBoard=1 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31848 Gres=tmpfs:1000
  • --conf-server <host|ip address>[:<port>] — comma-separated list of controllers, the first being the primary slurmctld. A port can (optionally) be specified for each controller. These hosts are where the slurmd will fetch the configuration from when running in "configless" mode.
⚠️ NOTE: If specifying an IPv6 address, wrap the <ip address> in [] to distinguish the address from the port. This is required even if no port is specified. Don't skip the brackets, bestie, she will not parse it 💀
  • -d <file> — specify the fully qualified pathname to the slurmstepd program to be used for shepherding user job steps. This can be useful for testing purposes.
  • -D — run slurmd in the foreground with logging copied to stderr. (This is the "I want to watch it happen live" mode ✨)
  • --extra <arbitrary string> — set "extra" data on node startup. If this is a json string and SchedulerParameters=extra_constraints is set in slurm.conf, then jobs may use the --extra option to filter based on this "extra" data.
  • -f <file> — read configuration from the specified file. See NOTES below. (Part 2, babes 🫶)
  • -F[feature] — start this node as a Dynamic Future node. It will try to match a node definition with a state of FUTURE, optionally using the specified feature to match the node definition.
  • -G — print Generic RESource (GRES) configuration (based upon slurm.conf GRES merged with gres.conf contents for this node) and exit.
  • -h — help; print a brief summary of command options. we love a helpful icon 💖
  • --instance-id <cloud instance id> — set cloud instance ID on node startup. 🩵
  • --instance-type <cloud instance type> — set cloud instance type on node startup.
  • -L <file> — write log messages to the specified file.
  • -M — lock slurmd pages into system memory using mlockall (2) to disable paging of the slurmd process. This may help in cases where nodes are marked DOWN during periods of heavy swap activity. If the mlockall (2) system call is not available, an error will be printed to the log and slurmd will continue as normal.

It is suggested to set LaunchParameters=slurmstepd_memlock in slurm.conf(5) when setting -M.

  • -n <value> — set the daemon's nice value to the specified value, typically a negative number. Also note the PropagatePrioProcess configuration parameter.
  • -N <nodename> — run the daemon with the given nodename. Used to emulate a larger system with more than one slurmd daemon per node. Requires that Slurm be built using the --enable-multiple-slurmd configure option. (only for the girls who compile 👑)
  • --parameters <slurmd parameters> — allows for temporary node-specific changes to SlurmdParameters. Can be used alongside -C to specify options such as 'l3cache_as_socket' which may alter the reported node topology. These parameters are appended to any node-specific Parameters defined in slurm.conf.
  • -s — change working directory of slurmd to SlurmdLogFile path if possible, or to SlurmdSpoolDir otherwise. If both of them fail it will fallback to /var/tmp. she tries her best, okay 😭
  • --systemd — use when starting the daemon with systemd. This will allow slurmd to notify systemd of the new PID when using 'scontrol reconfigure'.
  • -v — verbose operation. Multiple -v's increase verbosity. stack 'em, babe 🍓
  • -V, --version — print version information and exit.
  • -Z — start this node as a Dynamic Normal node. If no --conf is specified, then the slurmd will register with the same hardware configuration (except GRES) as defined by the -C option.

🌿 ENVIRONMENT VARIABLES 🌿

The following environment variables can be used to override settings compiled into slurmd. yes she reads the vibes of your shell, it's giving observant 💅

  • ABORT_ON_FATAL — When a fatal error is detected, use abort() instead of exit() to terminate the process. This allows backtraces to be captured without recompiling Slurm.
  • SLURM_CONF — The location of the Slurm configuration file. This is overridden by explicitly naming a configuration file on the command line.
  • SLURM_DEBUG_FLAGS — Specify debug flags for slurmd to use. See DebugFlags in the slurm.conf(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf.

🌈 HTTP server 🌈

Unless disabled via CommunicationParameters=disable_http in slurm.conf, slurmd will accept incoming HTTP/1.1 compliant requests to any socket listening as configured by SlurmdPort in slurm.conf. Authentication of HTTP requests is not supported. TLS wrapping optionally supported without requiring TLSType in slurm.conf. The following endpoints are currently supported:

  • GET / — Get list of endpoints.
  • GET /healthz — Test if slurmd loaded successfully.
  • GET /livez — Test if slurmd loaded successfully.
  • GET /readyz[?verbose] — Test if slurmd is ready to accept incoming RPCs. Adding the verbose query parameter will result in a verbose output of each probe in the response.
⚠️ real talk: authentication of HTTP requests is not supported. don't @ me.

🌸 SIGNALS 🌸

  • SIGTERM SIGINT SIGQUIT — slurmd will shutdown cleanly. goodbye bestie, gracefully 🫶
  • SIGHUP — Reloads the slurm configuration files, similar to 'scontrol reconfigure'.
  • SIGUSR2 — Reread the log level from the configs, and then reopen the log file. This should be used when setting up logrotate(8).
  • SIGPIPE — This signal is explicitly ignored. she said "not today" 💀

🌟 CORE FILE LOCATION 🌟

If slurmd is started with the -D option then the core file will be written to the current working directory. Otherwise if SlurmdLogFile is a fully qualified path name (starting with a slash), the core file will be written to the same directory as the log file. Otherwise the core file will be written to the SlurmdSpoolDir directory, or "/var/tmp/" as a last resort. If none of the above directories can be written, no core file will be produced.

like, she has a whole priority list for where to dump her feelings. respect the order 🙈

(okay that's part 1, the rest is coming — don't go anywhere bestie 💜)

🌸 NOTES 🌸

ok so bestie, real talk — if you wanna mess around with slurmd-specific config parameters (think stuff like timeouts), it's genuinely useful to experiment using a distinct configuration file. slay, go off, test things in your little sandbox. 🧪

But — and this is the plot twist 😭 — that special config file will not be used by the slurmctld daemon or the Slurm programs, unless you specifically tell each of them to use it. Like, they're not psychic, bestie. You have to tell them. iykyk.

So if what you actually want to change is the communication ports, the location of the temporary file system, or any other parameters that other Slurm components use too? That's a group-chat decision, not a solo vibe. Change the common configuration file, slurm.conf. 💅

Also, quick heads up for the configless-mode girlies: if you've got a login node that runs a LOT of client commands, lowkey consider running slurmd on that machine too, so it can manage a cached version of the configuration files. 🎀

Otherwise? Every single client command has to use the DNS record to contact the controller and get the configuration information — which could place additional load on the controller. Not the main character behavior we want for your controller. 🙈

🫶 COPYING 🫶

Copyright (C) 2002-2007 The Regents of the University of California. Copyright (C) 2008-2010 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. 👑

fr fr read the license, it's iconic and it's literally what makes this whole thing free.

🦋 FILES 🦋

  • /etc/slurm.conf — that's the one, bestie. 📁

⭐ SEE ALSO ⭐

like, go read those next, they're the sibling pages and they ate. 💖

✨ and that's the whole page, bestie ✨ this gorgeous little number is from the Slurm 26.05 manual, filed under Slurm Daemon, and it lives in section slurmd(8) — go forth and daemon responsibly, ily, mwah 💅💜🎀

📦 where this page comes from