SPANK(8)

🌸 SPANK(8) 🌸

part of the Slurm Component fam — welcome to the plug-in party, bestie 💅

---

✨ NAME ✨

SPANK — the Slurm Plug-in Architecture for Node and job (K)control 🎀

---

💖 DESCRIPTION 💖

Okay so this page is your little tour of what SPANK can actually do, plus the whole deal with the SPANK config file — which, btw, lives at plugstack.conf by default 📁 (key words: by default, so don't come for me if yours is somewhere else).

SPANK hands you a genuinely generic interface for stackable plug-ins — think of it like layering friendship bracelets on your wrist, except each bracelet can dynamically modify the job launch code in Slurm 🦋 Stackable literally means stackable, iykyk.

And here's the slay part: SPANK plugins can be built without access to Slurm source code 😭👑 No secret handshake, no repo clone, none of that. All you need to do is:

1. compile your plugin against Slurm's spank.h header file
2. add it to the SPANK config file plugstack.conf
3. that's it, bestie — it gets loaded at runtime during the next job launch ✨

So basically the SPANK infrastructure gives administrators and other developers a low cost, low effort way to dynamically modify the runtime behavior of Slurm job launch. It's giving "accessible" and honestly? Iconic. 🫶

⚠️ NOTE, and this one matters, fr fr: all SPANK plugins should be recompiled whenever you upgrade Slurm to a new major release. The SPANK API is not guaranteed to be ABI compatible between major releases �� And any SPANK plugin linking to any of the Slurm libraries should be carefully checked, because the Slurm APIs and headers can change between major releases. No cap — read that twice if you have to. 💀

🎀 SPANK PLUGINS 🎀

okay bestie, so here's the whole deal with these plugins — they get loaded in up to five separate contexts during a Slurm job. Let's break those down, like, briefly:

• local — in local context, the plugin is loaded by srun. (aka the "local" part of a parallel job, duh 💅)
• remote — in remote context, the plugin is loaded by slurmstepd. (the "remote" part of a parallel job, iykyk)
• allocator — in allocator context, the plugin is loaded in one of the job allocation utilities salloc, sbatch or scrontab.
• slurmd — in slurmd context, the plugin is loaded in the slurmd daemon itself. > ⚠️ NOTE: plugins loaded in slurmd context persist for the entire time slurmd is running, so if configuration is changed or plugins are updated, slurmd must be restarted for the changes to take effect. no soft-launching these updates, bestie 😭
• job_script — in the job_script context, plugins are loaded in the context of the job prolog or epilog. > ⚠️ NOTE: plugins are loaded in job_script context on each run on the job prolog or epilog, in a separate address space from plugins in slurmd context. That means there is no state shared between this context and other contexts — or even between one call to slurm_spank_job_prolog or slurm_spank_job_epilog and subsequent calls. literally zero carryover, it's giving amnesia 🌸

---

💫 which functions run where

• In local context, only the init, exit, init_post_opt, and local_user_init functions are called.
• In allocator context, only the init, exit, and init_post_opt functions are called.
• Similarly, in slurmd context, only the init and slurmd_exit callbacks are active.
• And in the job_script context, only the job_prolog and job_epilog callbacks are used.

Plugins can totally query which context they're running in with the spank_context and spank_remote functions defined in spank.h — go look, they're right there 🦋

---

✨ the plugin callbacks ✨

SPANK plugins may be called from multiple points during the Slurm job launch. A plugin may define the following functions:

slurm_spank_init

Called just after plugins are loaded. In remote context, this is just after the job step is initialized. This function is called before any plugin option processing.

slurm_spank_job_prolog

Called at the same time as the job prolog. If this function returns a non-zero value and the SPANK plugin that contains it is required in the plugstack.conf, the node that this is run on will be drained. 😬

slurm_spank_init_post_opt

Called at the same point as slurm_spank_init, but after all user options to the plugin have been processed. The reason that the init and init_post_opt callbacks are separated is so that plugins can process system-wide options specified in plugstack.conf in the init callback, then process user options, and finally take some action in slurm_spank_init_post_opt if necessary. In the case of a heterogeneous job, slurm_spank_init is invoked once per job component. multitasking queen 👑

slurm_spank_local_user_init

Called in local (srun) context only after all options have been processed. This is called after the job ID and step IDs are available. This happens in srun after the allocation is made, but before tasks are launched.

slurm_spank_user_init

Called after privileges are temporarily dropped. (remote context only)

slurm_spank_task_init_privileged

Called for each task just after fork, but before all elevated privileges are dropped. This can run in parallel with slurm_spank_task_post_fork. (remote context only)

slurm_spank_task_init

Called for each task just before execve (2). If you are restricting memory with cgroups, memory allocated here will be in the job's cgroup. (remote context only)

slurm_spank_task_post_fork

Called for each task from the parent process after fork (2) is complete. Due to the fact that slurmd does not exec any tasks until all tasks have completed fork (2), this call is guaranteed to run before the user task is executed. This can run in parallel with slurm_spank_task_init_privileged. (remote context only)

slurm_spank_task_exit

Called for each task as its exit status is collected by Slurm. (remote context only)

slurm_spank_exit

Called once just before slurmstepd exits in remote context. In local context, called before srun exits. the goodbye kiss 💜

slurm_spank_job_epilog

Called at the same time as the job epilog. If this function returns a non-zero value and the SPANK plugin that contains it is required in the plugstack.conf, the node that this is run on will be drained. same drama as the prolog, fr 💀

slurm_spank_slurmd_exit

Called in slurmd when the daemon is shut down.

---

All of these functions have the same prototype, for example:

int slurm_spank_init (spank_t spank, int ac, char *argv[])

Where spank is the SPANK handle which must be passed back to Slurm when the plugin calls functions like spank_get_item and spank_getenv — don't lose it, it's your hall pass 🫶. Configured arguments (see CONFIGURATION below) are passed in the argument vector argv with argument count ac.

---

🌈 the plugin variables

A plugin may also define the following variables that will be used by Slurm:

• slurm_spank_init_failure_mode — when a slurm_spank_init call fails, this changes how that failure is handled by Slurm. Recognized values are:
  • ESPANK_NODE_FAILURE — Slurm considers the node to be at fault and marks it as drained. The job may be requeued. This is the default.
  • ESPANK_JOB_FAILURE — Slurm considers the job to be at fault and marks it as failed (not to be requeued). The node will not be drained.

---

🔍 checking if a symbol is supported

SPANK plugins can query the current list of supported slurm_spank symbols to determine if the current version supports a given plugin hook. This may be useful because the list of plugin symbols may grow in the future. The query is done using the spank_symbol_supported function, which has the following prototype:

int spank_symbol_supported (const char *sym);

The return value is 1 if the symbol is supported, 0 if not. simple as that, no cap ✨

---

🍓 getting info about the job

SPANK plugins do not have direct access to internally defined Slurm data structures. Instead, information about the currently executing job is obtained via the spank_get_item function call:

spank_err_t spank_get_item (spank_t spank, spank_item_t item, ...);

The spank_get_item call must be passed the current SPANK handle as well as the item requested, which is defined by the passed spank_item_t. A variable number of pointer arguments are also passed, depending on which item was requested by the plugin. A list of the valid values for item is kept in the spank.h header file. Some examples are:

• S_JOB_UID — user id for running job. (uid_t *) is the third arg of spank_get_item.
• S_JOB_STEPID — job step id for running job. (uint32_t *) is the third arg of spank_get_item.
• S_TASK_EXIT_STATUS — exit status for exited task. Only valid from slurm_spank_task_exit. (int *) is the third arg of spank_get_item.
• S_JOB_ARGV — complete job command line. Third and fourth args to spank_get_item are (int *, char ***).

See spank.h for more details — it's the whole mood board 💅

---

🧚 environment things (local vs remote)

SPANK functions in the local and allocator environment should use the getenv, setenv, and unsetenv functions to view and modify the job's environment. SPANK functions in the remote environment should use the spank_getenv, spank_setenv, and spank_unsetenv functions to view and modify the job's environment.

spank_getenv searches the job's environment for the environment variable var and copies the current value into a buffer buf of length len. spank_setenv allows a SPANK plugin to set or overwrite a variable in the job's environment, and spank_unsetenv unsets an environment variable in the job's environment. The prototypes are:

spank_err_t spank_getenv (spank_t spank, const char *var,
                    char *buf, int len);
spank_err_t spank_setenv (spank_t spank, const char *var,
                    const char *val, int overwrite);
spank_err_t spank_unsetenv (spank_t spank, const char *var);

These are only necessary in remote context since modifications of the standard process environment using setenv (3), getenv (3), and unsetenv (3) may be used in local context. so don't do the extra work for no reason, bestie 🙈

---

👑 the job control environment

Functions are also available from within the SPANK plugins to establish environment variables to be exported to the Slurm PrologSlurmctld, Prolog, Epilog and EpilogSlurmctld programs (the so-called job control environment). The name of environment variables established by these calls will be prepended with the string SPANK_ in order to avoid any security implications of arbitrary environment variable control. (After all, the job control scripts do run as root or the Slurm user.) 🔥 security said no thank you

These functions are available from local context only:

spank_err_t spank_job_control_getenv(spank_t spank, const char *var,
                       char *buf, int len);
spank_err_t spank_job_control_setenv(spank_t spank, const char *var,
                       const char *val, int overwrite);
spank_err_t spank_job_control_unsetenv(spank_t spank, const char *var);

See spank.h for more information.

---

💀 errors, slayed

Many of the described SPANK functions available to plugins return errors via the spank_err_t error type. On success, the return value will be set to ESPANK_SUCCESS, while on failure, the return value will be set to one of many error values defined in spank.h. The SPANK interface provides a simple function:

const char * spank_strerror(spank_err_t err);

which may be used to translate a spank_err_t value into its string representation. it's giving translator app for your error codes 🦋

The slurm_spank_log function can be used to print messages back to the user at an error level. This is to keep users from having to rely on the slurm_error function, which can be confusing because it prepends "error:" to every message. like okay we get it, it's an error 😭

✨ SPANK OPTIONS ✨

okay so this is the fun part, bestie 💅 — SPANK plugins get their own whole interface where they can define and implement extra job options. Like, brand new options that didn't exist before. And they get surfaced to you through the regular Slurm commands: srun(1), salloc(1), and sbatch(1). So if you specify one of these options, its value gets forwarded and registered with the plugin in slurmd when the job actually runs. Which is literally how SPANK plugins can dynamically hand Slurm new options and new functionality. iconic behavior, honestly 👑

Every option a plugin registers with Slurm is in the form of a struct spank_option, and it's declared in spank.h like this:

struct spank_option {
   char *         name;
   char *         arginfo;
   char *         usage;
   int            has_arg;
   int            val;
   spank_opt_cb_f cb;
};

And here's what each field is giving 🫶

• name — the name of the option. Heads up: its length is limited to SPANK_OPTION_MAXLEN, which is defined in spank.h.
• arginfo — a description of the argument to the option, if the option actually takes an argument.
• usage — a short description of the option, suitable for --help output.
• has_arg — 0 if the option takes no argument, 1 if it takes an argument, and 2 if the option takes an optional argument. (See getopt_long(3) if you wanna go deeper on that.)
• val — a plugin-local value that gets returned to the option callback function.
• cb — a callback function that gets invoked when the plugin option is registered with Slurm. spank_opt_cb_f is typedef'd in spank.h as:

typedef int (*spank_opt_cb_f) (int val, const char *optarg,
                         int remote);

Where val is the value of the val field in the spank_option struct, optarg is the supplied argument if applicable, and remote is 0 if the function is being called from the "local" host (like, the host where srun or sbatch/salloc are invoked) or 1 from the "remote" host (the host where slurmd/slurmstepd run) — but it's only executed by slurmstepd (remote context) if the option was registered for that context. no cap 🔥

---

Plugins can register options with Slurm using the spank_option_register function. Little caveat though: this function is only valid when it's called from the plugin's slurm_spank_init handler, and it registers one option at a time. The prototype is:

spank_err_t spank_option_register (spank_t sp,
          struct spank_option *opt);

This function returns ESPANK_SUCCESS on successful registration of an option, or ESPANK_BAD_ARG for errors — including an invalid spank_t handle, or when the function isn't called from the slurm_spank_init function. And listen: all options need to be registered from all contexts in which they'll be used. So for example, if an option is only used in local (srun) and remote (slurmd) contexts, then spank_option_register should only be called from within those contexts. Like:

if (spank_context() != S_CTX_ALLOCATOR)
   spank_option_register (sp, opt);

If, however, the option is used in all contexts, then spank_option_register needs to be called everywhere. periodt 💯

---

In addition to spank_option_register, plugins can also export options to Slurm by defining a table of struct spank_option with the symbol name spank_options. But — and this is a big but — this method is not supported for use with sbatch and salloc (allocator context), so using spank_option_register is preferred. When you're using the spank_options table, the final element in the array has to be filled with zeros. There's a SPANK_OPTIONS_TABLE_END macro provided in spank.h for exactly this purpose, so use it 🙈

---

When a user provides an option on the local side — either by command line options or by environment variables — Slurm will immediately invoke that option's callback with remote=0. The whole point of this is for the plugin to do local sanity checking of the option before the value gets sent over to the remote side during job launch. And if the argument the user specified is invalid? The plugin should issue an error and issue a non-zero return code from the callback. Also, the plugin should be able to handle the case where the spank option gets set multiple times through environment variables and command line options. And fun fact: environment variables are processed before command line options. 🌸

On the remote side, options and their arguments are registered just after the SPANK plugins are loaded and before the spank_init handler is called. This is what lets plugins modify the behavior of all plugin functionality based on the value of user-provided options. so much power honestly ⭐

---

As an alternative to using an option callback and a global variable, plugins can use spank_option_getopt to check for supplied options after option processing. This function has the prototype:

spank_err_t spank_option_getopt(spank_t sp,
    struct spank_option *opt, char **optargp);

This function returns ESPANK_SUCCESS if the option defined in the struct spank_option opt has been used by the user. And if optargp is non-NULL, then it gets set to any option argument passed (if the option takes an argument, obviously). You have to use this method to process options in the job_script context (slurm_spank_job_prolog and slurm_spank_job_epilog). mandatory, bestie 💅

This function is valid in the following contexts:

• slurm_spank_job_prolog
• slurm_spank_local_user_init
• slurm_spank_user_init
• slurm_spank_task_init_privileged
• slurm_spank_task_init
• slurm_spank_task_exit
• slurm_spank_job_epilog

---

🎀 CONFIGURATION 🎀

The default SPANK plug-in stack configuration file is plugstack.conf, sitting in the same directory as slurm.conf(5) — though you can change that via the Slurm config parameter PlugStackConfig. Normally the plugstack.conf file should be identical on all nodes of the cluster, like, no mixing and matching bestie 🧚

The config file lists SPANK plugins, one per line, along with whether the plugin is required or optional, and any global arguments that are to be passed to the plugin for runtime configuration. Comments are preceded with # and extend to the end of the line. And if the configuration file is missing or empty? It'll simply be ignored. no stress 💜

NOTE: The SPANK plugins need to be installed on the machines that execute slurmd (compute nodes) as well as on the machines that execute job allocation utilities such as salloc, sbatch, etc (login nodes). Both. not one or the other 😭

---

The format of each non-comment line in the configuration file is:

required/optional   plugin   arguments

For example:

optional /usr/lib/slurm/test.so

That tells slurmd to load the plugin test.so passing no arguments. And if a SPANK plugin is required, then failure of any of the plugin's functions will cause slurmd, or the job allocator command, to terminate the job — while optional plugins only cause a warning. so yeah, that first word is doing a lot of heavy lifting 💀

If a fully-qualified path isn't specified for a plugin, then the currently configured PluginDir in slurm.conf(5) is searched.

SPANK plugins are stackable, meaning more than one plugin may be placed into the config file. The plugins will simply be called in order, one after the other, and appropriate action taken on failure given the state of the plugin's optional flag. stack 'em up 🥞

---

Extra config files, or directories of config files, may be included in plugstack.conf with the include keyword. The include keyword must appear on its own line, and takes a glob as its parameter, so multiple files may be included from one include line. For example, the following syntax will load all config files in the /etc/slurm/plugstack.conf.d directory, in local collation order:

include /etc/slurm/plugstack.conf.d/*

which might be considered a more flexible method for building up a spank plugin stack. cute, modular, we love her 🌈

The SPANK config file is re-read on each job launch, so editing the config file will not affect running jobs. However, care should be taken so that a partially edited config file is not read by a launching job. like please don't save halfway through, it's giving chaos 🙈

---

Errors

When a SPANK plugin results in a non-zero result, the following changes will result:

okay so this is the big grid, the one everyone screenshots, and bestie it is a lot — but it's also literally just four columns of facts per row, so breathe. 💖

Here's how you read it: every row is one Command + one Function, and then what happens for that combo — its Context, its Exitcode, whether it Drains Node, and whether it Fails job. That's it. That's the tea. No hidden lore, the table says exactly what it says. 🫶

Let me walk you through it command by command, no skips, no "you get the idea" — because you do deserve the whole thing. ✨

---

🏃 srun — the whole lineup, in order:

• slurm_spank_init — context: local · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_init_post_opt — context: local · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_local_user_init — context: local · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_init — context: remote · exit code: 1 · drains node: no · fails job: no
• slurm_spank_user_init — context: remote · exit code: 0 · drains node: no · fails job: no
• slurm_spank_task_init_privileged — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_post_fork — context: remote · exit code: 0 · drains node: no · fails job: no
• slurm_spank_task_init — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_exit — context: remote · exit code: 0 · drains node: no · fails job: no
• slurm_spank_exit — context: local · exit code: 0 · drains node: no · fails job: yes

---

🎀 salloc — yes it gets its own set, they're not copy-paste, look closer:

• slurm_spank_init — context: allocator · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_init_post_opt — context: allocator · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_init — context: remote · exit code: 1 · drains node: no · fails job: no
• slurm_spank_user_init — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_init_privileged — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_post_fork — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_init — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_exit — context: remote · exit code: 0 · drains node: no · fails job: no
• slurm_spank_exit — context: allocator · exit code: 0 · drains node: no · fails job: yes

---

📦 sbatch — the one with the actually interesting "drains node" column, istg:

• slurm_spank_init — context: allocator · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_init_post_opt — context: allocator · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_init — context: remote · exit code: 1 · drains node: yes · fails job: no
• slurm_spank_user_init — context: remote · exit code: 1 · drains node: yes · fails job: yes
• slurm_spank_task_init_privileged — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_post_fork — context: remote · exit code: 1 · drains node: yes · fails job: yes
• slurm_spank_task_init — context: remote · exit code: 1 · drains node: no · fails job: yes
• slurm_spank_task_exit — context: remote · exit code: 0 · drains node: no · fails job: no
• slurm_spank_exit — context: allocator · exit code: 0 · drains node: no · fails job: no

---

🚨 okay genuine heads up before you skim past it: this table is the reference, not a summary. Don't go merging rows in your head because two of them look similar — same function name shows up in more than one context with different exit codes and a different drains/fails story, and that difference is the entire point. Read the row. The whole row. Periodt. 💅

And notice srun starts with three local rows before it goes remote, while salloc and sbatch both open on allocator and then head remote. Also like… slurm_spank_exit ends up in a different context per command (local for srun, allocator for the other two) and the fails-job verdict flips between them — yes, yes, no. Not me double-checking that three times, but I did. 😭

The rows are grouped exactly like this in the source, with the srun block, the salloc block, and the sbatch block separated out — so if you're cross-referencing against a plugin's callback implementations, go block by block and you'll never lose your place. 🦋

The table wraps up with these last two rows, in column order:

• scrontab, slurm_spank_init, allocator, 1, no, no
• scrontab, slurm_spank_exit, allocator, 0, no, no

Yes, bestie, those are two separate no cells at the end — don't merge them, that's not the vibe. 💅

⚠️ NOTE: with ProctrackType=proctrack/pgid, the behavior may result in timeouts for slurm_spank_task_post_fork with remote context on failure. That's a real warning, so don't gloss over it. 😭

---

📜 COPYING 📜

Okay, legal time. I'm giving you the license text verbatim, because copyright and license lines are not a freestyle situation, bestie. 🫶

Portions copyright (C) 2010-2022 SchedMD LLC.  Copyright (C) 2006 The
Regents of the University of California.  Produced at Lawrence Livermore
National Laboratory (cf, DISCLAIMER).  CODE-OCEC-09-009. All rights
reserved.

This file is part of Slurm, a resource management program.  For details,
see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your
option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
more details.

---

📂 FILES 📂

These are the paths in the group chat, no cap:

• /etc/slurm/slurm.conf — Slurm configuration file.
• /etc/slurm/plugstack.conf — SPANK configuration file.
• /usr/include/slurm/spank.h — SPANK header file.

---

👀 SEE ALSO 👀

Go touch grass in these other pages too:

• srun(1)
• slurm.conf(5)

---

💖 that's a wrap on SPANK(8), bestie. The footer says source/version Slurm 26.05, component Slurm Component, page SPANK(8) — no date on this one, so I'm not inventing one. periodt. 🎀✨