cgroup.conf(5)

📦
version 26.05.5.1 · 7 versions
, in arch rolling, epel 10, nixpkgs unstable, ubuntu stonking

💖 cgroup.conf(5) — Slurm Configuration File 💖

okay so bestie, pull up a chair 🫶 this is cgroup.conf, the Slurm config file that handles all the cgroup support, and it's genuinely lowkey iconic. let's get into it ✨

🎀 NAME 🎀

cgroup.conf — the Slurm configuration file for the cgroup support 💅

🦋 DESCRIPTION 🦋

cgroup.conf is an ASCII file that defines the parameters used by Slurm's Linux cgroup related plugins. and the way that it always lives in the same directory as the slurm.conf?? yeah, they're roommates, no cap 💖

Parameter names are case insensitive, bestie, so don't stress about the caps. Any text following a # in the configuration file is treated as a comment through the end of that line — so vent in the comments all you want, it's fine 🙈

Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command scontrol reconfigure — unless otherwise noted. so you've got options, pick your fave ⭐

For general Slurm cgroups information, go check out the Cgroups Guide at https://slurm.schedmd.com/cgroups.html 🧚

The following cgroup.conf parameters are defined to control the general behavior of Slurm cgroup plugins. periodt 💅

✨ THE PARAMETERS ✨

  • CgroupMountpoint=PATH — okay so this one is only intended for development and testing, just so you know 🌸 It specifies the PATH under which cgroup controllers should be mounted. The default PATH is /sys/fs/cgroup.
  • CgroupPlugin=<cgroup/v1|cgroup/v2|autodetect|disabled> — this is the main character right here 👑 it specifies the plugin to be used when interacting with the cgroup subsystem. The supported values are:
    • "disabled" — to completely disable any interaction with the cgroups 🙈
    • "cgroup/v1" — which supports the legacy interface of cgroup v1
    • "cgroup/v2" — for the unified cgroup2 architecture 🔥
    • "autodetect" — which tries to determine which cgroup version does your system provide

"autodetect" is useful to have a single configuration in clusters where nodes can have different cgroup versions. The default value is "autodetect" on linux, otherwise the default is "disabled".

NOTE: the cgroup/v1 plugin is deprecated and no new features will be added to it 😭 It is recommended that you switch to cgroup/v2 at your earliest convenience, fr fr.
  • CgroupSlice=SLICE_NAME — start slurmd and the slurmstepd scope in the specified slice. If you're starting slurmd with systemd, you should also add the Slice= property under the [Service] section of the slurmd.service file — don't skip that step bestie, both places or it doesn't count 💅

Changing this parameter requires you to stop all the slurmstepd scopes in the node and restart slurmd. Applies only to cgroup v2. The default is system.slice.

  • CgroupJobIdPaths=<yes|no> — if configured to "yes" then job-level cgroup directories are named with the numeric job ID instead of with the SLUID, e.g. /sys/fs/cgroup/system.slice/slurmstepd.scope/job_123/ instead of /sys/fs/cgroup/system.slice/slurmstepd.scope/sEKNKTV3WPV500/ 🌈 This restores the pre-26.05 cgroup/v2 path layout and may be useful for sites with external tooling keyed on job IDs. Only applies to cgroup/v2. The default value is no.
  • SystemdTimeout=<number> — okay so on slow systems like virtual machines, or when systemd is busy, it can take a lot of time to initialize and prepare the scope for slurmd during startup 😭 Slurm will wait a maximum of this amount of time (in milliseconds) for the scope to be ready before failing. Only applies to cgroup/v2. The default is 1000 ms.
  • IgnoreSystemd=<yes|no> — only for cgroup/v2, and only for development and testing 🧪 It will avoid any call to dbus and contact with systemd, and instead will prepare all the cgroup hierarchy manually.
⚠️ This option is dangerous in systems with systemd, since the cgroup can be modified by systemd and cause issues to jobs. like, be careful bestie 💀
  • IgnoreSystemdOnFailure=<yes|no> — only for cgroup/v2, and only for development and testing. It has similar functionality to IgnoreSystemd but only in the case that a dbus call does not succeed 🙈
  • EnableControllers=<yes|no> — only for cgroup/v2, and generally for development, testing, when running on old kernels and/or old systemd versions (e.g. RHEL8, systemd < 244) where not all the controllers are enabled in the cgroup tree 🌸 With this parameter, slurmd gets the available controllers from root's cgroup.controllers file (CgroupMountPoint, by default /sys/fs/cgroup/cgroup.controllers) and makes them available in all levels of the cgroup tree until it reaches Slurm's cgroup leaf. she's thorough, she said every level 💅
  • EnableExtraControllers=<all|controller1[,controller2,...]> — this will enable extra controllers not natively used by Slurm in the job cgroup subtree ✨ If combined with EnableControllers it will enable them in all the levels of the cgroup tree from the OS root cgroup. The controllers that can be enabled are io, pids, rdma, hugetlb and misc or all. If they do not exist in the system an error is logged. Only for cgroup/v2.

🌸 hold that thought bestie, we're only halfway through this file… 💖

💅 TASK/CGROUP PLUGIN 💅

okay bestie, these are the cgroup.conf parameters that run the whole vibe for this specific plugin. like, this is the plugin's personal rules, no cap:

  • AllowedRAMSpace=<number> — caps the job/step cgroup RAM at this percentage of the memory that was actually allocated. you can totally write the percentage as a float, like 101.5, she's flexible 💅. what it does: sets the cgroup soft memory limit at the allocated memory size, and then sets the job/step hard memory limit at (AllowedRAMSpace/100) * allocated memory. if the job/step blows past the hard limit, it might trigger Out Of Memory (OOM) events — including oom-kill — and those get logged to the kernel log ring buffer (dmesg on Linux). setting AllowedRAMSpace above 100 may cause system OOM events, because it lets a job/step allocate more memory than the nodes were configured with. the suggested move is to reduce the configured node available memory to avoid system OOM events. setting AllowedRAMSpace below 100 means jobs get less memory than allocated, and the soft memory limit gets set to the same value as the hard limit. also see ConstrainRAMSpace. the default value is 100. 🫶
  • AllowedSwapSpace=<number> — constrains the job cgroup swap space to this percentage of the allocated memory. default is 0, which means RAM+Swap gets limited to AllowedRAMSpace. float percentages are fine here too, e.g. 50.5. if the limit gets exceeded, the job steps get killed and a warning message gets written to standard error. 😭 also see ConstrainSwapSpace.
NOTE bestie: setting AllowedSwapSpace to 0 does not stop the Linux kernel from using swap space. to control how the kernel uses swap space, see MemorySwappiness.
  • ConstrainCores=<yes|no> — if it's "yes", it constrains the allowed cores to the subset of allocated resources. this one is powered by the cpuset subsystem ✨. heads up tho: because of a bug that got fixed in version 1.11.5 of HWLOC, the task/affinity plugin may be required in addition to task/cgroup for this to work properly. default value is "no".
  • ConstrainDevices=<yes|no> — if it's "yes", it constrains the job's allowed devices based on GRES allocated resources. it uses the devices subsystem for that. default value is "no".
  • ConstrainRAMSpace=<yes|no> — if it's "yes", it constrains the job's RAM usage by setting the memory soft limit to the allocated memory and the hard limit to the allocated memory * AllowedRAMSpace. default value is "no", and in that case the job's RAM limit gets set to its swap space limit if ConstrainSwapSpace is set to "yes". CR_*_Memory must be set in slurm.conf for this parameter to do literally anything. also see AllowedSwapSpace, AllowedRAMSpace and ConstrainSwapSpace.
NOTE: when you're using ConstrainRAMSpace and the combined memory used by all processes in a step goes over the limit, the kernel will trigger an OOM event, killing one or more of the processes in that step. the step state gets marked as OOM, but here's the wild part — the step itself keeps running, and the other processes in the step may keep running too. this is different from the behavior of OverMemoryKill, where the whole step gets killed/cancelled. 💀
NOTE: when enabled, ConstrainRAMSpace can lead to a noticeable decline in per-node job throughout. sites with high-throughput requirements should carefully weigh the tradeoff between per-node throughput versus the potential problems that can arise from unconstrained memory usage on the node. see <https://slurm.schedmd.com/high_throughput.html> for further discussion.
  • ConstrainSwapSpace=<yes|no> — if it's "yes", it constrains the job's swap space usage. default is "no". now here's the tea 🍵: when this is set to "yes" and ConstrainRAMSpace is set to "no", AllowedRAMSpace is automatically set to 100% — that's so the RAM+Swap amount gets limited to 100% of the job's requirement plus the percent of allowed swap space. this amount gets set to both the RAM and RAM+Swap limits. which means in that particular case, ConstrainRAMSpace is automatically enabled with the same limit as the one used to constrain swap space. CR_*_Memory must be set in slurm.conf for this parameter to take any effect. also see AllowedSwapSpace.
  • MaxRAMPercent=PERCENT — sets an upper bound, in percent of total RAM (the configured RealMemory of the node), on the RAM constraint for a job. this is the memory constraint that gets applied to jobs that were not explicitly allocated memory by Slurm (i.e. Slurm's select plugin isn't configured to manage memory allocations). PERCENT can be any arbitrary floating point number. default value is 100.
  • MaxSwapPercent=PERCENT — sets an upper bound (in percent of total RAM, the configured RealMemory of the node) on the amount of RAM+Swap that may be used for a job. this is the swap limit applied to jobs on systems where memory isn't being explicitly allocated to the job. PERCENT can be any arbitrary floating point number between 0 and 100. default value is 100.
  • MemorySwappiness=<number> — only for cgroup/v1. configures the kernel's priority for swapping out anonymous pages (like, program data) verses file cache pages for the job cgroup. valid values are between 0 and 100, inclusive. a value of 0 prevents the kernel from swapping out program data. a value of 100 gives equal priority to swapping out file cache or anonymous pages. if it's not set, the kernel's default swappiness value gets used. ConstrainSwapSpace must be set to yes for this parameter to be applied. ✨
  • MinRAMSpace=<number> — sets a lower bound (in MiB) on the memory limits defined by AllowedRAMSpace and AllowedSwapSpace. this one is the safety net 🫶 — it prevents accidentally creating a memory cgroup with such a low limit that slurmstepd gets immediately killed due to lack of RAM. and if that happens? cleanup will not be performed, and temporary files, sockets and directories can remain in the node. the default limit is 30M.

🧚 PROCTRACK/CGROUP PLUGIN 🧚

these are the cgroup.conf parameters that control the behavior of this particular plugin, bestie:

  • SignalChildrenProcesses=<yes|no> — if it's "yes", then signals get sent (for cancelling, suspending, resuming, etc.) to all children processes in a job/step. otherwise, signals only go to the parent process of a job/step. the default setting is "no".

🌈 DISTRIBUTION-SPECIFIC NOTES 🌈

Debian and its derivatives (e.g. Ubuntu) usually exclude the memory and memsw (swap) cgroups by default. to include them, add the following parameters to the kernel command line:

cgroup_enable=memory swapaccount=1

this can usually be placed in /etc/default/grub inside the GRUB_CMDLINE_LINUX variable. a command such as update-grub must be run after updating the file. 📝

🎀 EXAMPLE 🎀

/etc/slurm/cgroup.conf

this example cgroup.conf file shows a configuration that enables the more commonly used cgroup enforcement mechanisms. iconic behavior, honestly:

###
# Slurm cgroup support configuration file.
###
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes

/etc/slurm/slurm.conf

these are the entries required in slurm.conf to activate the cgroup enforcement mechanisms. make sure the node definitions in your slurm.conf closely match the configuration as shown by slurmd -C. either MemSpecLimit should be set, or RealMemory should be defined with less than the actual amount of memory for a node — that's to ensure all system/non-job processes will have sufficient memory at all times. sites should also configure pam_slurm_adopt to ensure users can't escape the cgroups via ssh. 👀

###
# Slurm configuration entries for cgroups
###
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
JobAcctGatherType=jobacct_gather/cgroup #optional for gathering metrics
PrologFlags=Contain                     #X11 flag is also suggested

💖 COPYING 💖

Copyright (C) 2010-2012 Lawrence Livermore National Security. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).

Copyright (C) 2010-2022 SchedMD LLC.

this file is part of Slurm, a resource management program. for details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. see the GNU General Public License for more details.

⭐ SEE ALSO ⭐

and that's the whole page, bestie, you ate it up 👑 this was Slurm 26.05, the Slurm Configuration File, filed under cgroup.conf(5) — go forth and configure responsibly 💜

📦 where this page comes from