nonstop.conf(5)

✨ nonstop.conf(5) — Slurm Configuration File ✨

okay bestie, welcome to the fault-tolerant era of Slurm 💖 this is the manual page for nonstop.conf, and we are gonna get you all the way together. no gatekeeping here, periodt.

---

🎀 NAME 🎀

nonstop.conf — the Slurm configuration file for fault-tolerant computing, aka the glow-up your cluster deserves when nodes start acting up 🦋

---

🌸 DESCRIPTION 🌸

So. nonstop.conf is an ASCII file, and it describes the configuration used for fault-tolerant computing with Slurm using the optional slurmctld/nonstop plugin. That plugin is literally the moment: it gives users a way to notify Slurm about nodes they believe are suspect 👀, to replace the job's failing or failed nodes, and to extend a job's — yes, the original sentence just says "extend a job's" and then moves on, we are not fixing that, we are respecting it 😭 — in response to failures.

The file will always be located in the same directory as the slurm.conf. like, always always. same folder, they're roommates, they're besties.

Parameter names are case insensitive. Any text following a # in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command scontrol reconfigure — unless otherwise noted. ✨

The configuration parameters available include:

• BackupAddr — the communications address used for the slurmctld daemon. Can be a hostname or an IP address. Typically this is the same as the secondary SlurmctldHost in the slurm.conf file, when applicable.
• ControlAddr — the communications address used for the slurmctld daemon. Can be a hostname or an IP address. Typically this is the same as the SlurmctldHost in the slurm.conf file.
• Debug — a number indicating the level of additional logging desired for the plugin. Default is zero, which generates no additional logging. (iconic behavior tbh)
• HotSpareCount — this identifies how many nodes in each partition should be maintained as spare resources 🫶. When a job fails, this pool of resources will be depleted and then replenished when possible using idle resources. The value should be a comma-delimited list of partition and node count pairs separated by a colon.
• MaxSpareNodeCount — this identifies the maximum number of nodes any single job may replace through the job's entire lifetime. It could prevent a single job from causing all of the nodes in a cluster to fail 😭. By default, there is no maximum node count.
• Port — port used for communications. The default value is 6820.
• TimeLimitDelay — if a job requires replacement resources and none are immediately available, then permit a job to extend its time limit by the length of time required to secure replacement resources, up to the number of minutes specified by TimeLimitDelay. This option will only take effect if no hot spare resources are available at the time replacement resources are requested. This time limit extension is in addition to the value calculated using the TimeLimitExtend. The default value is zero (no time limit extension). The value may not exceed 65533 seconds.
• TimeLimitDrop — specifies the number of minutes that a job can extend its time limit for each failed or failing node removed from the job's allocation. The default value is zero (no time limit extension). The value may not exceed 65533 seconds.
• TimeLimitExtend — specifies the number of minutes that a job can extend its time limit for each replaced node. The default value is zero (no time limit extension). The value may not exceed 65533 seconds.
• UserDrainAllow — this identifies a comma-delimited list of user names or user IDs of users who are authorized to drain nodes they believe are failing. Specify a value of "ALL" to permit any user to drain nodes 💅. By default, no users may drain nodes using this interface.
• UserDrainDeny — this identifies a comma-delimited list of user names or user IDs of users who are NOT authorized to drain nodes they believe are failing. Specifying a value for UserDrainDeny implicitly allows all other users to drain nodes (sets the value of UserDrainAllow to "ALL").

---

🦋 EXAMPLE 🦋

here's the sample file, verbatim, exactly as the manual shows it — yes, including the MaxSpareNodesCount and the double TimeLimitExtend, we do not touch the sacred text 🙈:

#
# Sample nonstop.conf file
# Date: 12 Feb 2013
#
ControlAddr=12.34.56.78
BackupAddr=12.34.56.79
Port=1234
#
HotSpareCount=batch:6,interactive:0
MaxSpareNodesCount=4
TimeLimitDelay=30
TimeLimitExtend=20
TimeLimitExtend=10
UserDrainAllow=adam,brenda

---

👑 COPYING 👑

Copyright (C) 2013-2022 SchedMD LLC. All rights reserved.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

fr fr, read the GPL. it's giving legally binding and also kind of a bop. 💜

---

💜 SEE ALSO 💜

• slurm.conf(5)

---

okay that's the whole page bestie, you ate that 🍓 this was nonstop.conf(5), part of the Slurm Configuration File family, brought to you by the fine folks at SchedMD, dated January 2022. go configure something fault-tolerant, ily, bye 💅✨🎀