SPANK(7)

💖 SPANK 💖

Slurm Plug-in Architecture for Node and job (K)control — that's the whole vibe, bestie. This is the SPANK(7) manual page, and it's here to walk you through the Slurm Plug-in Architecture for Node and job Kontrol. Yes, the acronym has a K in it for no reason. We love that for it. 🎀

Welcome to the SPANK arc of the Slurm cinematic universe. Grab a strawberry drink, this is part one of five. 🍓

---

✨ NAME ✨

SPANK — Slurm Plug-in Architecture for Node and job (K)control. That's it, that's the name, she's iconic and she knows it. 👑

---

🌸 DESCRIPTION 🌸

okay so. This manual is your little intro tour of what the Slurm Plug-in Architecture for Node and job Kontrol — SPANK — can actually do, plus everything you need to know about the SPANK config file, which by default lives at plugstack.conf. 📁

The whole point of SPANK is that it hands you a very generic interface for stackable plug-ins. Stackable!! Like little stacking rings but for job launch code. You can use them to dynamically modify the job launch code in Slurm, no source diving required. ✨

Here's the part that makes everyone obsessed: SPANK plugins can be built without any access to the Slurm source code at all. Nope, none, zero, not needed. All a plugin needs is:

1. Be compiled against Slurm's spank.h header file.
2. Get added to the SPANK config file plugstack.conf.
3. That's it. That's the list. 💅

Once those boxes are checked, the plugin gets loaded at runtime during the next job launch. Automatic. No restart ceremony, no drama. Which means the SPANK infrastructure gives administrators and other developers a low cost, low effort way to dynamically change how Slurm behaves at job launch time. Cheaper than therapy, honestly. 😭

NOTE: Every single SPANK plugin should be recompiled whenever you upgrade Slurm to a new major release. The SPANK API is not guaranteed to be ABI compatible between major releases. And any SPANK plugin that links against any of the Slurm libraries needs to be carefully checked, because the Slurm APIs and headers can change between major releases too. Do not skip this. The vibe check does not apply here, the recompile does. 💀

---

🦋 SPANK PLUGINS 🦋

okay bestie so the plugins get loaded in up to five different contexts during a Slurm job. five!! and each one hits different. let's break them down, no cap:

• local — in this context the plugin gets loaded by srun. (it's literally the "local" part of a parallel job, she's just doing her bit 💅)
• remote — here the plugin is loaded by slurmstepd. (the "remote" half of a parallel job, doing the heavy lifting)
• allocator — in this context the plugin is loaded inside one of the job allocation utilities salloc, sbatch or scrontab.
• slurmd — here the plugin is loaded inside the slurmd daemon itself. NOTE: plugins loaded in slurmd context stick around for the entire time slurmd is running, so if you change config or update your plugins, you gotta restart slurmd for it to actually take effect. no shortcut, sorry 🫠
• job_script — in this context plugins are loaded up in the context of the job prolog or epilog. NOTE: plugins get loaded in job_script context on every single run of the job prolog or epilog, in a separate address space from the plugins in slurmd context. which means there is zero state shared between this context and the other contexts — not even between one call to slurm_spank_job_prolog or slurm_spank_job_epilog and the calls that come after it. it's giving fresh start every time ✨

so like, in local context, only the init, exit, init_post_opt, and local_user_init functions get called. in allocator context, only init, exit, and init_post_opt get called. same energy in slurmd context — only the init and slurmd_exit callbacks are active. and in the job_script context? only the job_prolog and job_epilog callbacks get used. plugins can figure out which context they're living in by asking spank_context and spank_remote, both defined in spank.h. iykyk 🎀

---

SPANK plugins can get called from a whole bunch of points during the Slurm job launch, and a plugin may define these functions:

slurm_spank_init

Called right after the plugins are loaded. In remote context this is right after the job step is initialized. This function runs before any plugin option processing, so keep that in mind bestie.

slurm_spank_job_prolog

Called at the exact same time as the job prolog. if this function returns a non-zero value and the SPANK plugin containing it is required in the plugstack.conf, the node it's running on gets drained. yeah. consequences.

slurm_spank_init_post_opt

Called at the same point as slurm_spank_init, except it's after all the user options to the plugin have been processed. the reason the init and init_post_opt callbacks are split like this is so plugins can process system-wide options from plugstack.conf in the init callback, then handle user options, and finally take action in slurm_spank_init_post_opt if they need to. side note: in the case of a heterogeneous job, slurm_spank_init gets invoked once per job component. 🌈

slurm_spank_local_user_init

Called in local (srun) context only, after all options have been processed. this is after the job ID and step IDs are available. so it happens in srun after the allocation is made, but before tasks are launched. ⭐

slurm_spank_user_init

Called after privileges are temporarily dropped. (remote context only, queen)

slurm_spank_task_init_privileged

Called for each task right after fork, but before all the elevated privileges are dropped. this one can run in parallel with slurm_spank_task_post_fork. (remote context only)

slurm_spank_task_init

Called for each task right before execve (2). quick heads up: if you're restricting memory with cgroups, memory allocated here will land in the job's cgroup. (remote context only)

slurm_spank_task_post_fork

Called for each task from the parent process after fork (2) is complete. and because slurmd doesn't exec any tasks until all tasks have completed fork (2), this call is guaranteed to run before the user task is executed. can run in parallel with slurm_spank_task_init_privileged. (remote context only)

slurm_spank_task_exit

Called for each task as its exit status is collected by Slurm. (remote context only)

slurm_spank_exit

Called once, right before slurmstepd exits in remote context. and in local context? called before srun exits. bye bestie 👋

slurm_spank_job_epilog

Called at the same time as the job epilog. if this function returns a non-zero value and the SPANK plugin containing it is required in plugstack.conf, the node it's running on gets drained. same deal as the prolog one.

slurm_spank_slurmd_exit

Called in slurmd when the daemon is shutting down.

---

All of these functions have the same prototype, like this for example:

int slurm_spank_init (spank_t spank, int ac, char *argv[])

where spank is the SPANK handle, and it must be passed back to Slurm whenever the plugin calls functions like spank_get_item and spank_getenv. the configured arguments (peep CONFIGURATION below) come through in the argument vector argv, with argument count ac. 🫶

---

A plugin may also define these variables, which Slurm will use:

slurm_spank_init_failure_mode

when a slurm_spank_init call fails, this changes how Slurm handles that failure. the recognized values are:

• ESPANK_NODE_FAILURE — Slurm considers the node to be at fault and marks it drained. the job may be requeued. this is the default.
• ESPANK_JOB_FAILURE — Slurm considers the job to be at fault and marks it failed, not to be requeued. the node will not be drained.

---

SPANK plugins can also query the current list of supported slurm_spank symbols to find out whether the current version supports a given plugin hook. this is lowkey useful because the list of plugin symbols may grow in the future. the query uses the spank_symbol_supported function, which has this prototype:

int spank_symbol_supported (const char *sym);

the return value is 1 if the symbol is supported, 0 if not. simple as that 💅

---

SPANK plugins do not have direct access to internally defined Slurm data structures. instead, info about the currently executing job comes from the spank_get_item function call:

spank_err_t spank_get_item (spank_t spank, spank_item_t item, ...);

the spank_get_item call has to be passed the current SPANK handle, plus the item you're requesting, defined by the spank_item_t you pass. a variable number of pointer arguments also get passed, depending on which item the plugin asked for. the full list of valid values for item lives in the spank.h header file. some iconic examples:

S_JOB_UID

User id for running job. (uid_t *) is third arg of spank_get_item

S_JOB_STEPID

Job step id for running job. (uint32_t *) is third arg of spank_get_item.

S_TASK_EXIT_STATUS

Exit status for exited task. Only valid from slurm_spank_task_exit. (int *) is third arg of spank_get_item.

S_JOB_ARGV

Complete job command line. Third and fourth args to spank_get_item are (int *, char ***).

See spank.h for more details. 👑

---

SPANK functions in the local and allocator environment should use the getenv, setenv, and unsetenv functions to view and modify the job's environment. SPANK functions in the remote environment should use spank_getenv, spank_setenv, and spank_unsetenv to view and modify the job's environment.

so spank_getenv searches the job's environment for the environment variable var and copies the current value into a buffer buf of length len. spank_setenv lets a SPANK plugin set or overwrite a variable in the job's environment, and spank_unsetenv unsets an environment variable in the job's environment. the prototypes are:

 spank_err_t spank_getenv (spank_t spank, const char *var,
                     char *buf, int len);
 spank_err_t spank_setenv (spank_t spank, const char *var,
                     const char *val, int overwrite);
 spank_err_t spank_unsetenv (spank_t spank, const char *var);

these are only necessary in remote context, since modifying the standard process environment with setenv (3), getenv (3), and unsetenv (3) is totally fine in local context.

---

functions are also available from inside the SPANK plugins to set up environment variables that get exported to the Slurm PrologSlurmctld, Prolog, Epilog and EpilogSlurmctld programs (the so-called job control environment). the names of environment variables created by these calls get prepended with the string SPANK_ to dodge any security implications of letting just anyone control arbitrary environment variables. (i mean, obviously — those job control scripts run as root or as the Slurm user 😭)

these functions are available from local context only:

 spank_err_t spank_job_control_getenv(spank_t spank, const char *var,
                      char *buf, int len);
 spank_err_t spank_job_control_setenv(spank_t spank, const char *var,
                      const char *val, int overwrite);
 spank_err_t spank_job_control_unsetenv(spank_t spank, const char *var);

See spank.h for more information.

---

so many of the SPANK functions available to plugins return errors via the spank_err_t error type. on success, the return value gets set to ESPANK_SUCCESS. on failure, it gets set to one of the many error values defined in spank.h. the SPANK interface hands you a cute little function:

 const char * spank_strerror(spank_err_t err);

which you can use to translate a spank_err_t value into its string representation. obsessed, honestly. ✨

and the slurm_spank_log function can be used to print messages back to the user at an error level. this exists so users don't have to rely on the slurm_error function, which gets confusing because it prepends "error:" to every single message. yeah. every one. 🙈

✨ SPANK OPTIONS ✨

okay bestie so here's the tea: SPANK plugins get their own lil interface to define and implement extra job options. 💅 and these options show up for the user through Slurm commands like srun(1), salloc(1), and sbatch(1). so when the user specifies one of these options, the value gets forwarded and registered with the plugin in slurmd when the job runs. and that's literally how SPANK plugins can dynamically hand new options and functionality to Slurm. iconic behaviour, honestly.

Every option a plugin registers with Slurm takes the form of a struct spank_option, which is declared in spank.h like this:

struct spank_option {
   char *         name;
   char *         arginfo;
   char *         usage;
   int            has_arg;
   int            val;
   spank_opt_cb_f cb;
};

Where:

• name — the name of the option, bestie. its length is limited to SPANK_OPTION_MAXLEN, which is defined in spank.h. don't go over, she has limits.
• arginfo — a description of the argument to the option, if the option does take an argument.
• usage — a short description of the option, suitable for --help output.
• has_arg — 0 if the option takes no argument, 1 if the option takes an argument, and 2 if the option takes an optional argument. (see getopt_long (3), she's the blueprint.)
• val — a plugin-local value to return to the option callback function.
• cb — a callback function that gets invoked when the plugin option is registered with Slurm. spank_opt_cb_f is typedef'd in spank.h as:

typedef int (*spank_opt_cb_f) (int val, const char *optarg,
                         int remote);

Where val is the value of the val field in the spank_option struct, optarg is the supplied argument if applicable, and remote is 0 if the function is being called from the "local" host (like, the host where srun or sbatch/salloc are invoked) or 1 from the "remote" host (the host where slurmd/slurmstepd run) — but it's only actually executed by slurmstepd (remote context) if the option was registered for that context. okay? okay. 🫶

Plugin options may be registered with Slurm using the spank_option_register function. and listen — this function is only valid when called from the plugin's slurm_spank_init handler. one option at a time, bestie, no batching. the prototype is:

spank_err_t spank_option_register (spank_t sp,
          struct spank_option *opt);

This function returns ESPANK_SUCCESS on successful registration of an option, or ESPANK_BAD_ARG for errors — including an invalid spank_t handle, or when the function isn't called from the slurm_spank_init function. 📋

All options need to be registered from all contexts in which they will be used. like, if an option is only used in local (srun) and remote (slurmd) contexts, then spank_option_register should only be called from within those contexts. for example:

if (spank_context() != S_CTX_ALLOCATOR)
   spank_option_register (sp, opt);

If, however, the option is used in all contexts, then spank_option_register needs to be called everywhere. no skipping, no shortcuts. ✨

In addition to spank_option_register, plugins may also export options to Slurm by defining a table of struct spank_option with the symbol name spank_options. But — and this is lowkey important — this method is not supported for use with sbatch and salloc (allocator context), so using spank_option_register is preferred. when you use the spank_options table, the final element in the array must be filled with zeros. a SPANK_OPTIONS_TABLE_END macro is provided in spank.h for exactly this purpose. use it, she's there for you. 🎀

When an option is provided by the user on the local side — either by command line options or by environment variables — Slurm will immediately invoke the option's callback with remote=0. this is meant for the plugin to do local sanity checking of the option before the value gets sent to the remote side during job launch. if the argument the user specified is invalid, the plugin should issue an error and issue a non-zero return code from the callback. and the plugin should be able to handle cases where the spank option is set multiple times through environment variables and command line options. environment variables are processed before command line options. so heads up on the ordering. 🧚

On the remote side, options and their arguments are registered just after SPANK plugins are loaded and before the spank_init handler is called. this lets plugins modify the behavior of all plugin functionality based on the value of user-provided options. so much power, honestly. 👑

As an alternative to using an option callback and a global variable, plugins can use the spank_option_getopt option to check for supplied options after option processing. this function has the prototype:

spank_err_t spank_option_getopt(spank_t sp,
    struct spank_option *opt, char **optargp);

This function returns ESPANK_SUCCESS if the option defined in the struct spank_option opt has been used by the user. if optargp is non-NULL then it's set to any option argument passed (if the option takes an argument). and note this: using this method is required to process options in job_script context (slurm_spank_job_prolog and slurm_spank_job_epilog). 🚨

This function is valid in the following contexts:

• slurm_spank_job_prolog
• slurm_spank_local_user_init
• slurm_spank_user_init
• slurm_spank_task_init_privileged
• slurm_spank_task_init
• slurm_spank_task_exit
• slurm_spank_job_epilog

---

🌸 CONFIGURATION 🌸

The default SPANK plug-in stack configuration file is plugstack.conf, sitting in the same directory as slurm.conf(5) — though this may be changed via the Slurm config parameter PlugStackConfig. normally the plugstack.conf file should be identical on all nodes of the cluster, bestie. consistency is the vibe check. 💖

The config file lists SPANK plugins, one per line, along with whether the plugin is required or optional, and any global arguments that are to be passed to the plugin for runtime configuration. comments are preceded with # and extend to the end of the line. and if the configuration file is missing or empty? it will simply be ignored. no drama. 🙈

NOTE: The SPANK plugins need to be installed on the machines that execute slurmd (compute nodes) as well as on the machines that execute job allocation utilities such as salloc, sbatch, etc (login nodes). both, bestie. both. 🫶

The format of each non-comment line in the configuration file is:

required/optional   plugin   arguments

For example:

optional /usr/lib/slurm/test.so

Tells slurmd to load the plugin test.so passing no arguments. if a SPANK plugin is required, then failure of any of the plugin's functions will cause slurmd, or the job allocator command, to terminate the job — while optional plugins only cause a warning. 👀 pick wisely.

If a fully-qualified path is not specified for a plugin, then the currently configured PluginDir in slurm.conf(5) is searched.

SPANK plugins are stackable, meaning more than one plugin may be placed into the config file. the plugins will simply be called in order, one after the other, and appropriate action taken on failure given that state of the plugin's optional flag. it's giving stack energy. 🥞

Additional config files or directories of config files may be included in plugstack.conf with the include keyword. the include keyword must appear on its own line, and takes a glob as its parameter, so multiple files may be included from one include line. for example, the following syntax will load all config files in the /etc/slurm/plugstack.conf.d directory, in local collation order:

include /etc/slurm/plugstack.conf.d/*

which might be considered a more flexible method for building up a spank plugin stack. honestly? iconic. ✨

The SPANK config file is re-read on each job launch, so editing the config file will not affect running jobs. However — care should be taken so that a partially edited config file is not read by a launching job. 😭 seriously, don't get caught mid-edit.

---

💀 Errors 💀

When SPANK plugin results in a non-zero result, the following changes will result:

okay so this part is literally just THE TABLE and bestie it is a lot of rows, but we're gonna get through it together because every single cell matters. 🫶

The columns are: Command, Function, Context, Exit code, Drains Node, Fails job.

lowkey obsessed with how this table has an empty row at the top and a doubled separator line. sloppy? yes. iconic? also yes. we're not judging, we're just transcribing. 💅

---

🏃 srun — the little go-getter

Nine-ish callbacks, all cute, let's go:

• slurm_spank_init — context local, exit code 1, drains node: no, fails job: yes
• slurm_spank_init_post_opt — context local, exit code 1, drains node: no, fails job: yes
• slurm_spank_local_user_init — context local, exit code 1, drains node: no, fails job: yes
• slurm_spank_init — context remote, exit code 1, drains node: no, fails job: no
• slurm_spank_user_init — context remote, exit code 0, drains node: no, fails job: no
• slurm_spank_task_init_privileged — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_post_fork — context remote, exit code 0, drains node: no, fails job: no
• slurm_spank_task_init — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_exit — context remote, exit code 0, drains node: no, fails job: no
• slurm_spank_exit — context local, exit code 0, drains node: no, fails job: yes

notice how slurm_spank_init shows up twice — once local, once remote — and they do NOT behave the same. bestie read the context column. it's giving "same name, different girl". 💀

---

🎀 salloc — allocation queen

• slurm_spank_init — context allocator, exit code 1, drains node: no, fails job: yes
• slurm_spank_init_post_opt — context allocator, exit code 1, drains node: no, fails job: yes
• slurm_spank_init — context remote, exit code 1, drains node: no, fails job: no
• slurm_spank_user_init — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_init_privileged — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_post_fork — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_init — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_exit — context remote, exit code 0, drains node: no, fails job: no
• slurm_spank_exit — context allocator, exit code 0, drains node: no, fails job: yes

so real talk: salloc and srun agree that slurm_spank_user_init in the remote context... wait no, they do NOT. srun says exit code 0 and fails job no; salloc says exit code 1 and fails job yes. bestie. they are different. do not assume. periodt. ✨

---

🦋 sbatch — the one that drains nodes apparently

• slurm_spank_init — context allocator, exit code 1, drains node: no, fails job: yes
• slurm_spank_init_post_opt — context allocator, exit code 1, drains node: no, fails job: yes
• slurm_spank_init — context remote, exit code 1, drains node: yes, fails job: no
• slurm_spank_user_init — context remote, exit code 1, drains node: yes, fails job: yes
• slurm_spank_task_init_privileged — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_post_fork — context remote, exit code 1, drains node: yes, fails job: yes
• slurm_spank_task_init — context remote, exit code 1, drains node: no, fails job: yes
• slurm_spank_task_exit — context remote, exit code 0, drains node: no, fails job: no
• slurm_spank_exit — context allocator, exit code 0, drains node: no, fails job: no

look at all those yes in the Drains Node column for sbatch. 🫣 that's the drama. the remote callbacks under sbatch are the only ones anywhere in this whole table that can drain a node, and slurm_spank_init, slurm_spank_user_init and slurm_spank_task_post_fork are all carrying that weight. slurm_spank_exit in the allocator context is the one lone sbatch row that does not fail the job — everyone else there does.

---

and then, right at the bottom, there's another duplicated separator line, like the table was just gonna keep going and then... didn't. 🙈 cliffhanger energy fr fr.

• scrontab · slurm_spank_init · allocator · 1 · no · no
• scrontab · slurm_spank_exit · allocator · 0 · no · no

And boom, that's the whole table, bestie, we made it to the end of it 💅

NOTE: the behavior for ProctrackType=proctrack/pgid may result in timeouts for slurm_spank_task_post_fork with remote context on failure. Not a vibe, but forewarned is forearmed 🫶

---

🎀 COPYING 🎀

Portions copyright (C) 2010-2022 SchedMD LLC. Copyright (C) 2006 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). CODE-OCEC-09-009. All rights reserved.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

okay so the ✨legal girlies✨ really said it first: free software, GPL v2 or later, no warranty, and the paperwork is right here. Respect the license, it's the law, periodt 👑

---

🗂️ FILES 🗂️

• /etc/slurm/slurm.conf — the Slurm configuration file. your main character config 💖
• /etc/slurm/plugstack.conf — the SPANK configuration file. this is where the plugins get their glow-up ✨
• /usr/include/slurm/spank.h — the SPANK header file. the sacred text for anyone writing their own plugin 🧚

---

👀 SEE ALSO 👀

• srun(1)
• slurm.conf(5)

go read those next, i promise they're worth it 🦋

---

and that's the whole page, babe 🌈 This was SPANK(7) from Slurm 26.05, straight out of the Slurm Component chapter — so if you ever need to look it up again on some other machine, that's the version and the section you're hunting for, iykyk 💜

ily, go spawn some tasks, and remember: read the caveats before you run anything in prod 😭💖