helpers.conf(5)

💖 NAME 💖

helpers.conf — the Slurm configuration file for the helpers plugin. That's it, that's the name. short, iconic, no notes. ✨

---

🦋 DESCRIPTION 🦋

okay so. helpers.conf is an ASCII file that defines the parameters used by Slurm's "helpers" node feature plugin. And here's the thing bestie — this file will always live in the same directory as your slurm.conf. not "usually", not "if you feel like it". always. 💅

---

✨ PARAMETERS ✨

Parameter names are case insensitive, fr fr. Any text following a # in the configuration file is treated as a comment through the end of that line. The size of each line in the file is limited to 1024 characters — do not test this, she will not budge. 😭

Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command scontrol reconfigure — unless otherwise noted. so like, you gotta do something, it's not gonna read your mind. 🫶

• AllowUserBoot=<user1>[,<user2>...] — controls which users are allowed to change the features with this plugin. Default is to allow ALL users. everyone's invited to the party, apparently. 🙈
• BootTime=<time> — controls how much time a node has to reboot before a timeout occurs and a failure is assumed. Default value is 300 seconds.
• ExecTime=<time> — controls how much time the Helper program can run before a timeout occurs and a failure is assumed. Default value is 10 seconds. ten seconds. blink and you'll miss it. 💀
• Feature=<string> Helper=<file> <Flags=<flags> — defines Feature(s) and a corresponding Helper program that reports and modifies the status of the feature(s). Multiple Feature entries are allowed, one for each feature and corresponding program/script. A comma separated list of features can also be defined for one Helper.

Features can be defined per node by creating a unique file for each node. The controller must have a helpers.conf that lists all possible helper features. (the controller is the main character here, she needs the whole guest list 👑)

Currently supported flags are:

• rebootless — indicate that the feature doesn't require a node reboot. If set, feature activation won't execute the RebootProgram, the node will just register that the node rebooted to slurmctld without an actual reboot. fake it till you make it, honestly. 💅

A single helpers.conf can be created that defines features for specific nodes by prepending NodeName=<nodelist> to the front of the Feature line. A Feature not prepended with NodeName will apply to all nodes.

# helpers.conf
NodeName=n1_[1-10] Feature=a1,a2 Helper=/path/helper.sh
NodeName=n2_[1-10] Feature=b1,b2 Helper=/path/helper.sh
Feature=c1,c2 Helper=/path/helper.sh

If a feature is defined in the helpers.conf and is not defined on a specific node in the helpers.conf but is defined for that node in the slurm.conf, that feature is treated as a changeable/rebootable feature by the controller. For example, if feature fa is defined on node node1 in the slurm.conf but is only listed on node2 in the helpers.conf, the feature will still trigger the node to be rebooted if not active. plot twist energy, truly. 😳

The Helper is an arbitrary program or script that reports and modifies the feature set on a given node. The helpers are site-specific and are not included with Slurm — you're on your own for those, bestie, bring your own. Features modified by the helpers require a reboot of the node using the RebootProgram. The Helper program/script must be executable by the SlurmdUser. The same program/script can be used to control multiple features. slurmd will execute the Helper in one of two ways:

1. Execute with no arguments to query the status of node features. It must return an exit code of 0 and either print a superset of the features expected by Slurm, or it can print nothing. Otherwise, the node will be drained. 😭 (put some respect on that exit code.)
2. Execute with a single argument of the feature to be activated on node reboot. In the case of multiple features the script is called multiple times. she's consistent like that. ✨

• MutuallyExclusive=<feature_list> — prevents certain features from being specified for the same job. There can be multiple MutuallyExclusive entries, each with their own list of features that are mutually exclusive among themselves (i.e. features on one line are only mutually exclusive with other features on the same line, but not mutually exclusive with features on other lines). so it's giving "you can't sit with us" but ONLY within your own little clique. 🎀

---

🌈 EXAMPLE 🌈

/etc/slurm/slurm.conf:

To enable the helpers plugin, the slurm.conf needs to have the following entry:

NodeFeaturesPlugins=node_features/helpers

/etc/slurm/helpers.conf:

The following example helpers.conf demonstrates that multiple features can use the same Helper script and that there can be multiple lists of features that are mutually exclusive. For example, with the following configuration a job cannot request both "nps1" and "nps2", nor can it request both "mig=on" and "mig=off". However, it could request "nps1" and "mig=on" at the same time. see? flexible. we love range. 💅

# helpers.conf
Feature=nps1,nps2,nps4 Helper=/usr/local/bin/nps
Feature=mig=on Helper=/usr/local/bin/mig
Feature=mig=off Helper=/usr/local/bin/mig
MutuallyExclusive=nps1,nps2,nps4
MutuallyExclusive=mig=on,mig=off
ExecTime=60
BootTime=60
AllowUserBoot=user1,user2

Example Helper script:

Helper scripts need to have the executable bit set and must exist on the nodes, where they will be executed by slurmd. When the helper script is called with no arguments it should return the feature(s) that are currently active for the node, with multiple features being new-line delimited. When the helper script is called with a feature to be enabled for the node, it should configure the node in a way that the specified feature will be enabled when the node reboots. This example script just writes the active feature name to a file but production scripts will probably be more complex. (read: please write a real one, this is the demo version 🫶)

#!/bin/bash

if [ "$1" = "nps1" ]; then
    echo "$1" > /etc/slurm/feature
elif [ "$1" = "nps2" ]; then
    echo "$1" > /etc/slurm/feature
elif [ "$1" = "nps4" ]; then
    echo "$1" > /etc/slurm/feature
else
    cat /etc/slurm/feature
fi

---

📜 COPYING 📜

okay legal besties, this part is the real deal, no skipping:

Copyright (C) 2021 NVIDIA CORPORATION. All rights reserved. Copyright (C) 2021 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

yes the ALL CAPS warranty disclaimer is shouting on purpose. that's the legal vibe, respect it. 💀

---

👀 SEE ALSO 👀

• slurm.conf(5)

---

and that's the whole page, bestie!! this was Slurm 26.05, living under the Slurm Configuration File umbrella, written up all cute for you. go forth and configure your features, you absolute queen. 👑💖✨🦋🎀