job_container.conf(5)

📦
version 26.05.5.1 · 7 versions
, in arch rolling, epel 10, nixpkgs unstable, ubuntu stonking

✨ NAME ✨

job_container.conf — it's giving Slurm configuration file, but make it namespace plugins 💅 (that's the Slurm Configuration File manual, section 5, bestie — say hi to job_container.conf(5)).

🌸 DESCRIPTION 🌸

So. job_container.conf is an ASCII file, and it's where you define the parameters that Slurm's namespace/tmpfs plugin runs on. Based on whatever you put in there, the plugin goes and creates the appropriate job-specifc namespace(s) for you. No cap. ✨

The namespace/tmpfs plugin creates a filesystem namespace and will construct a private (or optionally shared, if you're feeling generous) filesystem namespace, and then mount a list of directories (defaults to /tmp and /dev/shm) inside it — which gives the job a private view of those directories. These paths get mounted inside the location specified by BasePath in the job_container.conf file. Iconic behavior, honestly. 👑

When the job completes, the namespace is unmounted and all files therein are automatically removed. Cleanup queen. 🧹

To make use of this plugin, PrologFlags=Contain must also be present in your slurm.conf file, as shown:

NamespaceType=namespace/tmpfs
PrologFlags=Contain

The file will always be located in the same directory as the slurm.conf. Same house, same address, they're roommates. 🏠

If using the job_container.conf file to define a namespace available to nodes, the first parameter on the line should be NodeName. If configuring a namespace without specifying nodes, the first parameter on the line should be BasePath.

Parameter names are case insensitive. Any text following a # in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons. So like… restart, bestie. 😌

The following job_container.conf parameters are defined to control the behavior of the namespace/tmpfs plugin. 💖

💫 the parameters 💫

  • AutoBasePath — this decides if the plugin should create the BasePath directory or not. Set it to true if the directory is not pre-created before slurm startup. If set to true, the directory is created with permission 0755. The directory is not deleted during slurm shutdown, so don't come for it later. If set to false or not specified, the plugin would expect the directory to exist. You can use this option on a global or per-line basis. This parameter is optional. 🎀
  • BasePath — specify the PATH that the tmpfs plugin should use as a base to mount the private directories. This path must be readable and writable by the plugin. The plugin constructs a directory for each job inside this path, which is then used for mounting. The BasePath gets mounted as private during slurmd start and remains mounted until shutdown. The first %h within the name is replaced with the hostname on which the slurmd is running. The first %n within the name is replaced with the Slurm node name on which the slurmd is running. Set PATH to none to disable the tmpfs plugin on node subsets when there is a global setting.
NOTE: The BasePath must be unique to each node. If BasePath is on a shared filesystem, you can use %h or %n to create node-unique directories. 😭
NOTE: The BasePath parameter cannot be set to any of the paths specified by Dirs. Using these directories will cause conflicts when trying to mount and unmount the private directories for the job. 💀
  • CloneNSScript — specify fully qualified pathname of an optional initialization script. This script is run after the namespace construction of a job. The script will be provided the SLURM_NS environment variable containing the path to the namespace that can be used by the nsenter command. That variable lets the script join the newly created namespace and do further setup work. This parameter is optional. 🦋
  • CloneNSScript_Wait — the number of seconds to wait for the CloneNSScript to complete before considering the script failed. The default value is 10 seconds. ⏳
  • CloneNSEpilog — specify fully qualified pathname of an optional epilog script. This script runs just before the namespace is torn down. The script will be provided the SLURM_NS environment variable containing the path to the namespace that can be used by the nsenter command. That variable lets the script join the soon to be removed namespace and do any cleanup work. This parameter is optional. 🌈
  • CloneNSEpilog_Wait — the number of seconds to wait for the CloneNSEpilog to complete before considering the script failed. The default value is 10 seconds. ⏳
  • EntireStepInNS — specifying EntireStepInNS=true will pivot all slurmstepd processes (excluding the external step, which is tasked with creating the namespace) into the constructed namespace. Heads up though — this will cause issues if certain paths such as SlurmdSpoolDir are inaccessible. This parameter is optional. 🙈
  • InitScript — specify fully qualified pathname of an optional initialization script. This script is run before the namespace construction of a job. It can be used to make the job join additional namespaces prior to the construction of /tmp namespace, or it can be used for any site-specific setup. This parameter is optional. 💜
  • NodeName — a NodeName specification can be used to permit one job_container.conf file to be used for all compute nodes in a cluster, by specifying the node(s) that each line should apply to. The NodeName specification can use a Slurm hostlist specification as shown in the example below. This parameter is optional. 🔥
  • Shared — specifying Shared=true will propagate new mounts between the job specific filesystem namespace and the root filesystem namespace, enable using autofs on the node. This parameter is optional. ✨

💀 NOTES 💀

If any parameters in job_container.conf are changed while slurm is running, then slurmd on the respective nodes will need to be restarted for changes to take effect (scontrol reconfigure is not sufficient). Additionally this can be disruptive to jobs already running on the node. So care must be taken to make sure no jobs are running if any changes to job_container.conf are deployed. Like, please. Read that part twice. 🫶

Restarting slurmd is safe and non-disruptive to running jobs, as long as job_container.conf is not changed between restarts — in which case the above point applies. So the vibe check is: restart = fine, edit-then-restart = chaos. 😭

🌟 EXAMPLE 🌟

/etc/slurm/slurm.conf

These are the entries required in slurm.conf to activate the namespace/tmpfs plugin.

NamespaceType=namespace/tmpfs
PrologFlags=Contain

/etc/slurm/job_container.conf

The first sample file will define 1 basepath for all nodes and it will be automatically created.

AutoBasePath=true
BasePath=/var/nvme/storage

The second sample file will define 2 basepaths. The first will only be on largemem[1-2] and it will be automatically created. The second will only be on gpu[1-10], will be expected to exist and will run an initscript before each job.

NodeName=largemem[1-2] AutoBasePath=true BasePath=/var/nvme/storage_a
NodeName=gpu[1-10] BasePath=/var/nvme/storage_b InitScript=/etc/slurm/init.sh

The third sample file will define 1 basepath that will be on all nodes, automatically created, with /tmp and /var/tmp as private mounts.

AutoBasePath=true
BasePath=/var/nvme/storage Dirs=/tmp,/var/tmp

👑 COPYING 👑

Copyright (C) 2021 Regents of the University of California. Produced at Lawrence Berkeley National Laboratory. Copyright (C) 2021-2022 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. 🫡

🦋 SEE ALSO 🦋

okay that's the whole page, you ate that fr fr 💅 this was job_container.conf(5) from Slurm 26.05, filed under Slurm Configuration File, and that's the tea — go configure something gorgeous bestie 🎀✨

📦 where this page comes from