CPUSET(4)

📦 version 1.0, in debian bookworm, debian trixie, opensuse-leap 16.0, ubuntu jammy, ubuntu noble, ubuntu questing

👑 cpuset(4) — Linux Programmer's Manual 👑

okay bestie, sit down, hydrate, because we're about to talk about cpuset and it's genuinely one of the most underrated little corners of the Linux kernel. this is section 4 of the manual, so it's a file-formats / device page, and it's giving big "here's how you actually pin your processes to the CPUs you love" energy. ✨

🎀 NAME 🎀

cpuset — confine tasks to processor and memory node subsets

like, literally, that's the whole vibe. you're telling the kernel "these tasks live here and nowhere else." 💅

🦋 DESCRIPTION 🦋

so the cpuset file system is a pseudo-filesystem interface to the kernel's cpuset mechanism, which is the thing that controls the processor and memory placement of tasks. it's commonly mounted at /dev/cpuset. periodt.

a cpuset defines a list of CPUs and memory nodes. cpusets are represented as directories in a hierarchical virtual file system, where the top directory in the hierarchy (/dev/cpuset) represents the entire system — like, all online CPUs and memory nodes, the whole squad 🌈 — and any cpuset that is the child (descendant) of another parent cpuset contains a subset of that parent's CPUs and memory nodes. and yes, the directories and files representing cpusets have normal file system permissions. nothing weird, nothing extra.

every task in the system belongs to exactly one cpuset. not two, not zero, exactly one. a task is confined to only run on the CPUs in the cpuset it belongs to, and to allocate memory only on the memory nodes in that cpuset. when a task forks, the child task is placed in the same cpuset as its parent — like mother like daughter fr fr 👯. and with sufficient privilege, a task may be moved from one cpuset to another, and the allowed CPUs and memory nodes of an existing cpuset may be changed. glow-up enabled.

when the system begins booting, only the top cpuset is defined and all tasks are in that cpuset. during the boot process or later during normal system operation, other cpusets may be created, as sub-directories of the top cpuset, under the control of the system administrator, and tasks may be placed in these other cpusets. so yes, the sysadmin is the main character here, as always. 😌

cpusets are integrated with the sched_setaffinity(2) scheduling affinity mechanism and the mbind(2) and set_mempolicy(2) memory placement mechanisms in the kernel. neither of these mechanisms let a task make use of a CPU or memory node that is not allowed by cpusets. and if changes to a task's cpuset placement conflict with these other mechanisms, then cpuset placement is enforced even if it means overriding these other mechanisms. cpuset said "i'm the parent, i decide." 🔥

typically, a cpuset is used to manage the CPU and memory node confinement for the entire set of tasks in a job, and these other mechanisms are used to manage the placement of individual tasks or memory regions within a job. so like — cpuset is the group chat, and the affinity/mbind stuff is the DMs. different tools, different jobs. 🫶

💖 FILES 💖

each directory below /dev/cpuset represents a cpuset and contains several files describing the state of that cpuset.

new cpusets are created using the mkdir system call or shell command. the properties of a cpuset, such as its flags, allowed CPUs and memory nodes, and attached tasks, are queried and modified by reading or writing to the appropriate file in that cpuset's directory, as listed below. it's giving very "just echo into the file, bestie." 🌸

the files in each cpuset directory are automatically created when the cpuset is created, as a result of the mkdir invocation. it is not allowed to add or remove files from a cpuset directory. no, you cannot decorate. 🙈

the files in each cpuset directory are small text files that may be read and written using traditional shell utilities such as cat(1) and echo(1), or using ordinary file access routines from programmatic languages, such as open(2), read(2), write(2), and close(2) from the 'C' library. these files represent internal kernel state and do not have any persistent image on disk. so if you were looking for them in /var somewhere… no ma'am. they're vibes, not files. 💫

each of these per-cpuset files is listed and described below.

  • tasks — list of the process IDs (PIDs) of the tasks in that cpuset. the list is formatted as a series of ASCII decimal numbers, each followed by a newline. a task may be added to a cpuset (removing it from the cpuset previously containing it) by writing its PID to that cpuset's tasks file (with or without a trailing newline.)
beware bestie ⚠️ — only one PID may be written to the tasks file at a time. if a string is written that contains more than one PID, only the first one will be considered. the rest just get ghosted. 😭
  • notify_on_release — flag (0 or 1). if set (1), that cpuset will receive special handling whenever its last using task and last child cpuset goes away. see the Notify On Release section, below.
  • cpus — list of CPUs on which tasks in that cpuset are allowed to execute. see List Format below for a description of the format of cpus.

the CPUs allowed to a cpuset may be changed by writing a new list to its cpus file. note however, such a change does not take affect until the PIDs of the tasks in the cpuset are rewritten to the cpuset's tasks file. see the WARNINGS section, below. yes, this is a jump-scare. yes, it's real.

  • cpu_exclusive — flag (0 or 1). if set (1), the cpuset has exclusive use of its CPUs (no sibling or cousin cpuset may overlap CPUs). by default this is off (0). newly created cpusets also initially default this to off (0). no gatekeeping, everyone's welcome — unless you flip it on, and then it's very much "this is MY CPU." 👑
  • mems — list of memory nodes on which tasks in that cpuset are allowed to allocate memory. see List Format below for a description of the format of mems.
  • mem_exclusive — flag (0 or 1). if set (1), the cpuset has exclusive use of its memory nodes (no sibling or cousin may overlap). by default this is off (0). newly created cpusets also initially default this to off (0).
  • memory_migrate — flag (0 or 1). if set (1), then memory migration is enabled. see the Memory Migration section, below.
  • memory_pressure — a measure of how much memory pressure the tasks in this cpuset are causing. see the Memory Pressure section, below. unless memory_pressure_enabled is enabled, always has value zero (0). this file is read-only. see the WARNINGS section, below.
  • memory_pressure_enabled — flag (0 or 1). this file is only present in the root cpuset, normally /dev/cpuset. if set (1), the memory_pressure calculations are enabled for all cpusets in the system. see the Memory Pressure section, below. it's the one file that lives in the penthouse suite. 🏙️
  • memory_spread_page — flag (0 or 1). if set (1), the kernel page cache (file system buffers) are uniformly spread across the cpuset. see the Memory Spread section, below.
  • memory_spread_slab — flag (0 or 1). if set (1), the kernel slab caches for file I/O (directory and inode structures) are uniformly spread across the cpuset. see the Memory Spread section, below.

in addition to the above special files in each directory below /dev/cpuset, each task under /proc has an added file named cpuset, displaying the cpuset name, as the path relative to the root of the cpuset file system. cute little breadcrumb trail. 🍓

also the /proc/<pid>/status file for each task has two added lines, displaying the task's cpus_allowed (on which CPUs it may be scheduled) and mems_allowed (on which memory nodes it may obtain memory), in the Mask Format (see below) as shown in the following example:

Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
Mems_allowed:   ffffffff,ffffffff

like okay hex mask realness. that's the flex. ✨

✨ EXTENDED CAPABILITIES ✨

Okay bestie, so cpusets don't just do the "which cpus and mems can this task touch" thing. They have extended capabilities, and they are lowkey iconic. Let's go 💅

👑 Exclusive Cpusets

If a cpuset is marked cpu_exclusive or mem_exclusive, then no other cpuset — other than a direct ancestor or descendant — may share any of the same CPUs or memory nodes. That's it. That's the rule. It's giving "this is my node, back off" 🚫

A cpuset that is cpu_exclusive has a scheduler (sched) domain associated with it. That sched domain = all the CPUs in the current cpuset that are not part of any exclusive child cpusets. Why do we care? Because it means the scheduler load balancing code only balances against the CPUs that are in the sched domain (as defined above), and not all the CPUs in the system. This removes any overhead from load balancing code trying to pull tasks outside of the cpu_exclusive cpuset, only to be blocked by the tasks' cpus_allowed mask. Basically: no wasted effort, no drama 🎀

A cpuset that is mem_exclusive restricts kernel allocations for page, buffer and other data commonly shared by the kernel across multiple users. And heads up: all cpusets, whether mem_exclusive or not, restrict allocations of memory for user space. So what's the point? It lets you configure a system so several independent jobs can share common kernel data (like file system pages) while isolating each job's user allocation in its own cpuset. How? Construct a large mem_exclusive cpuset to hold all the jobs, then construct child, non-mem_exclusive cpusets for each individual job. And ngl, only a small amount of typical kernel memory — like requests from interrupt handlers — is allowed to be taken outside even a mem_exclusive cpuset. So it's almost airtight, not fully airtight, and that's the design 🌸

🔔 Notify On Release

If the notify_on_release flag is enabled (1) in a cpuset, then whenever the last task in the cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of that cpuset is removed, the kernel will run the command /sbin/cpuset_release_agent, supplying the pathname (relative to the mount point of the cpuset file system) of the abandoned cpuset. This enables automatic removal of abandoned cpusets. Automatic cleanup?? We love a self-maintaining queen ✨

The default value of notify_on_release in the root cpuset at system boot is disabled (0). The default value of other cpusets at creation is the current value of their parent's notify_on_release setting. So it's hereditary, iykyk 🧬

The command /sbin/cpuset_release_agent is invoked with the name (/dev/cpuset relative path) of that cpuset in argv[1]. This supports automatic cleanup of abandoned cpusets.

The usual contents of the command /sbin/cpuset_release_agent is simply the shell script:

#!/bin/sh
rmdir /dev/cpuset/$1

By default, notify_on_release is off (0). Newly created cpusets inherit their notify_on_release setting from their parent cpuset.

As with other flag values below, this flag can be changed by writing an ASCII number 0 or 1 (with optional trailing newline) into the file, to clear or set the flag, respectively. Simple as that, periodt 💅

🍓 Memory Pressure

The memory_pressure of a cpuset provides a simple per-cpuset metric of the rate that the tasks in a cpuset are attempting to free up in-use memory on the nodes of the cpuset to satisfy additional memory requests. It's basically a stress-o-meter for a cpuset 📈

This enables batch managers monitoring jobs running in dedicated cpusets to efficiently detect what level of memory pressure that job is causing. Super useful, fr fr.

It's useful in two very different flavors of situation:

  • On tightly managed systems running a wide mix of submitted jobs, which may choose to terminate or re-prioritize jobs that are trying to use more memory than allowed on the nodes assigned them.
  • With tightly coupled, long running, massively parallel scientific computing jobs that will dramatically fail to meet required performance goals if they start to use more memory than allowed to them. dramatically fail is doing a lot of work in that sentence 😭

This mechanism provides a very economical way for the batch manager to monitor a cpuset for signs of memory pressure. It's up to the batch manager or other user code to decide what to do about it and take action. The kernel is just here to report, not to parent you 🫶

⚠️ Unless memory pressure calculation is enabled by setting the special file /dev/cpuset/memory_pressure_enabled, it is not computed for any cpuset, and always reads a value of zero. See the WARNINGS section, below. So if you're staring at a 0 and wondering why your job is clearly struggling — yeah, check that file first bestie.

Why a per-cpuset, running average: (because you know someone asked, and the answer is genuinely good)

  • Because this meter is per-cpuset rather than per-task or mm, the system load imposed by a batch scheduler monitoring this metric is sharply reduced on large systems, because a scan of the tasklist can be avoided on each set of queries.
  • Because this meter is a running average rather than an accumulating counter, a batch scheduler can detect memory pressure with a single read, instead of having to read and accumulate results for a period of time.
  • Because this meter is per-cpuset rather than per-task or mm, the batch scheduler can obtain the key information — memory pressure in a cpuset — with a single read, rather than having to query and accumulate results over all the (dynamically changing) set of tasks in the cpuset.

A per-cpuset simple digital filter is kept within the kernel, and updated by any task attached to that cpuset, if it enters the synchronous (direct) page reclaim code.

A per-cpuset file provides an integer number representing the recent (half-life of 10 seconds) rate of direct page reclaims caused by the tasks in the cpuset, in units of reclaims attempted per second, times 1000. Numbers, bestie. Beautiful, precise numbers 💜

🦋 Memory Spread

There are two Boolean flag files per cpuset that control where the kernel allocates pages for the file system buffers and related in-kernel data structures. They are called memory_spread_page and memory_spread_slab. Two flags, two vibes.

  • If the per-cpuset Boolean flag file memory_spread_page is set, then the kernel will spread the file system buffers (page cache) evenly over all the nodes that the faulting task is allowed to use, instead of preferring to put those pages on the node where the task is running.
  • If the per-cpuset Boolean flag file memory_spread_slab is set, then the kernel will spread some file system related slab caches — such as for inodes and directory entries — evenly over all the nodes that the faulting task is allowed to use, instead of preferring to put those pages on the node where the task is running.

The setting of these flags does not affect anonymous data segment or stack segment pages of a task. So not everything gets spread, okay? Know the scope 💅

By default, both kinds of memory spreading are off and the kernel prefers to allocate memory pages on the node local to where the requesting task is running. If that node is not allowed by the task's NUMA mempolicy or cpuset configuration, or if there are insufficient free memory pages on that node, then the kernel looks for the nearest node that is allowed and does have sufficient free memory. Nearest-neighbor energy 🌈

When new cpusets are created, they inherit the memory spread settings of their parent. (Again with the inheritance! It's a whole thing.)

Setting memory spreading causes allocations for the affected page or slab caches to ignore the task's NUMA mempolicy and be spread instead. Tasks using mbind() or set_mempolicy() calls to set NUMA mempolicies will not notice any change in these calls as a result of their containing task's memory spread settings. If memory spreading is turned off, the currently specified NUMA mempolicy once again applies to memory page allocations.

Both memory_spread_page and memory_spread_slab are Boolean flag files. By default they contain "0", meaning that the feature is off for that cpuset. If a "1" is written to that file, that turns the named feature on. Classic 0/1 situation 🎀

This memory placement policy is also known (in other contexts) as round-robin or interleave.

This policy can provide substantial improvements for jobs that need to place thread local data on the corresponding node, but that need to access large file system data sets that need to be spread across the several nodes in the job's cpuset in order to fit. Without this policy — especially for jobs that might have one thread reading in the data set — the memory allocation across the nodes in the job's cpuset can become very uneven. Uneven memory allocation is not the vibe. Spread it out 🧚

🧚 Memory Migration

Normally, under the default setting (disabled) of memory_migrate, once a page is allocated (given a physical page of main memory) then that page stays on whatever node it was allocated, so long as it remains allocated — even if the cpuset's memory placement policy mems subsequently changes. It's committed. It's not moving. Loyalty 💀

When memory migration is enabled in a cpuset, if the mems setting of the cpuset is changed, then any memory page in use by any task in the cpuset that is on a memory node no longer allowed will be migrated to a memory node that is allowed. Now that's adaptive ✨

Also, if a task is moved into a cpuset with memory_migrate enabled, any memory pages it uses that were on memory nodes allowed in its previous cpuset, but which are not allowed in its new cpuset, will be migrated to a memory node allowed in the new cpuset. Moving house and taking your pages with you, iconic 🏠

The relative placement of a migrated page within the cpuset is preserved during these migration operations if possible. For example, if the page was on the second valid node of the prior cpuset, then the page will be placed on the second valid node of the new cpuset, if possible. So it keeps its spot in the lineup. Very demure, very mindful 🫶

📐 FORMATS 📐

The following formats are used to represent sets of CPUs and memory nodes. Two formats, two jobs, let's meet them 💖

🎀 Mask Format

The Mask Format is used to represent CPU and memory node bitmasks in the /proc/<pid>/status file.

It is hexadecimal, using ASCII characters "0" - "9" and "a" - "f". This format displays each 32-bit word in hex (zero filled) and, for masks longer than one word, uses a comma separator between words. Words are displayed in big-endian order, most significant first. And hex digits within a word are also in big-endian order. Big-endian twice, because why not 🎀

The number of 32-bit words displayed is the minimum number needed to display all bits of the bitmask, based on the size of the bitmask. No padding for the sake of padding, we're efficient here 💅

Examples of the Mask Format:

00000001                        # just bit 0 set
80000000,00000000,00000000      # just bit 95 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32-39 set
00000000,000E3862               # 1,5,6,11-13,17-19 set

A mask with bits 0, 1, 2, 4, 8, 16, 32 and 64 set displays as "00000001,00000001,00010117". The first "1" is for bit 64, the second for bit 32, the third for bit 16, the fourth for bit 8, the fifth for bit 4, and the "7" is for bits 2, 1 and 0. See? Every digit is pulling its weight. Main character digits only 👑

🌈 List Format

The List Format for cpus and mems is a comma separated list of CPU or memory node numbers and ranges of numbers, in ASCII decimal.

Examples of the List Format:

0-4,9           # bits 0, 1, 2, 3, 4 and 9 set
0-2,7,12-14     # bits 0, 1, 2, 7, 12, 13 and 14 set

Ranges and singles, comma separated, decimal, done. Short, sweet, readable, obsessed 🦋

✨ RULES ✨

okay bestie, every single cpuset out here has to follow these rules, no exceptions, fr fr:

  • Its CPUs and memory nodes have to be a (possibly equal) subset of its parent's. like, you can't just vibe outside your parent's lane 💅
  • It can only be marked cpu_exclusive if its parent is.
  • It can only be marked mem_exclusive if its parent is.
  • If it's cpu_exclusive, its CPUs may not overlap any sibling. periodt.
  • If it's memory_exclusive, its memory nodes may not overlap any sibling.

🔐 PERMISSIONS 🔐

The permissions of a cpuset are decided by the permissions of the special files and directories in the cpuset file system, which normally lives mounted at /dev/cpuset.

So for instance, a task can put itself into some other cpuset (like, not the one it's currently in) if it can write the tasks file for that cpuset — and that needs execute permission on the encompassing directories plus write permission on that tasks file. iykyk.

There's also an extra lil constraint that kicks in when you're trying to place some other task in a cpuset: one task may not attach another to a cpuset unless it would have permission to send that task a signal. yes, really, that's the rule 🫶

A task can create a child cpuset if it can access and write the parent cpuset directory. And it can modify the CPUs or memory nodes in a cpuset if it can access that cpuset's directory (execute permissions on the encompassing directories) and write the corresponding cpus or mems file.

but lowkey note this: since changes to the CPUs of a cpuset don't apply to any task in that cpuset until said task is reattached to that cpuset, it would normally not be a good idea to arrange the permissions on a cpuset so that some task could write the cpus file unless it could also write the tasks file to reattach the tasks in there. like, that's a setup for confusion, bestie 😭

There's one tiny difference between how these permissions get evaluated vs. how normal file system operation permissions get evaluated. The kernel evaluates relative pathnames starting at a task's current working directory. Even if you're operating on a cpuset file, relative pathnames are evaluated relative to the current working directory, not relative to a task's current cpuset. The only ways cpuset paths relative to a task's current cpuset can actually be used are:

  1. if the task's current working directory is its cpuset (meaning it first did a cd or chdir to its cpuset directory beneath /dev/cpuset, which is a bit unusual), or
  2. if some user code converts the relative cpuset path to a full file system path.

🚨 WARNINGS 🚨

⚠️ Updating a cpusets cpus

Changes to a cpuset's cpus file do not take affect for any task in that cpuset until that task's process ID (PID) is rewritten to the cpuset's tasks file. This unusual requirement is needed to optimize a critical code path in the Linux kernel. Beware that only one PID can be written at a time to a cpuset's tasks file. Additional PIDs on a single write(2) system call are ignored. One (unobvious) way to satisfy this requirement — rewrite the tasks file after updating the cpus file — is to use the -u unbuffered option to the sed(1) command, like this:

cd /dev/cpuset/foo              # /foo is an existing cpuset
/bin/echo 3 > cpus              # change /foo's cpus
sed -un p < tasks > tasks       # rewrite /foo's tasks file

If you go peek at the Cpus_allowed value in the /proc/<pid>/status file for one of the tasks in cpuset /foo in that scenario above, you'll notice the value does not change when the cpus file is written (the echo command) — it only changes later, after the tasks file is rewritten (the sed command). the way that works is so specific, honestly 💀

🧠 Enabling memory_pressure

By default, the per-cpuset file memory_pressure always contains zero (0). Unless this feature is enabled by writing "1" to the special file /dev/cpuset/memory_pressure_enabled, the kernel does not compute per-cpuset memory_pressure.

🐚 Using the echo command

When you're using the echo command at the shell prompt to change the values of cpuset files, beware that most shell built-in echo commands do not display an error message if the write(2) system call fails. For example, if this command:

echo 19 > mems

failed because memory node 19 was not allowed (maybe the current system doesn't even have a memory node 19), then that echo command would not display any error. It's better to use the /bin/echo external command to change cpuset file settings, because this command will display write(2) errors, like:

/bin/echo 19 > mems
/bin/echo: write error: No space left on device

🌈 EXCEPTIONS 🌈

Not every single allocation of system memory is constrained by cpusets, and here's why:

  • If hot-plug functionality is used to remove all the CPUs currently assigned to a cpuset, then the kernel will automatically update the cpus_allowed of all tasks attached to CPUs in that cpuset to allow all CPUs. When memory hot-plug functionality for removing memory nodes is available, a similar exception is expected to apply there as well. In general, the kernel prefers to violate cpuset placement over starving a task that has had all its allowed CPUs or memory nodes taken offline. User code should reconfigure cpusets to only refer to online CPUs and memory nodes when using hot-plug to add or remove such resources.
  • A few kernel critical internal memory allocation requests, marked GFP_ATOMIC, must be satisfied, immediately. The kernel may drop some request or malfunction if one of these allocations fail. If such a request cannot be satisfied within the current task's cpuset, then we relax the cpuset, and look for memory anywhere we can find it. It's better to violate the cpuset than stress the kernel.
  • Allocations of memory requested by kernel drivers while processing an interrupt lack any relevant task context, and are not confined by cpusets.

🫠 LIMITATIONS 🫠

🧩 Kernel limitations updating cpusets

In order to minimize the impact of cpusets on critical kernel code, such as the scheduler, and due to the fact that the kernel does not support one task updating the memory placement of another task directly, the impact on a task of changing its cpuset CPU or memory node placement — or of changing to which cpuset a task is attached — is subtle. like, subtle subtle.

If a cpuset has its memory nodes modified, then for each task attached to that cpuset, the next time that the kernel attempts to allocate a page of memory for that task, the kernel will notice the change in the task's cpuset, and update its per-task memory placement to stay within the new cpuset's memory placement. If the task was using mempolicy MPOL_BIND, and the nodes to which it was bound overlap with its new cpuset, then the task will continue to use whatever subset of MPOL_BIND nodes are still allowed in the new cpuset. If the task was using MPOL_BIND and now none of its MPOL_BIND nodes are allowed in the new cpuset, then the task will be essentially treated as if it was MPOL_BIND bound to the new cpuset (even though its NUMA placement, as queried by get_mempolicy(), doesn't change). If a task is moved from one cpuset to another, then the kernel will adjust the task's memory placement, as above, the next time that the kernel attempts to allocate a page of memory for that task.

If a cpuset has its CPUs modified, each task using that cpuset does _not_ change its behavior automatically. In order to minimize the impact on the critical scheduling code in the kernel, tasks will continue to use their prior CPU placement until they are rebound to their cpuset, by rewriting their PID to the tasks file of their cpuset. If a task had been bound to some subset of its cpuset using the sched_setaffinity() call, and if any of that subset is still allowed in its new cpuset settings, then the task will be restricted to the intersection of the CPUs it was allowed on before, and its new cpuset CPU placement. If, on the other hand, there is no overlap between a task's prior placement and its new cpuset CPU placement, then the task will be allowed to run on any CPU allowed in its new cpuset. If a task is moved from one cpuset to another, its CPU placement is updated in the same way as if the task's PID is rewritten to the tasks file of its current cpuset.

In summary: the memory placement of a task whose cpuset is changed is updated by the kernel, on the next allocation of a page for that task, but the processor placement is not updated, until that task's PID is rewritten to the tasks file of its cpuset. This is done to avoid impacting the scheduler code in the kernel with a check for changes in a task's processor placement. main character energy for the scheduler honestly 🔥

🏷️ Rename limitations

You can use the rename(2) system call to rename cpusets. Only simple renaming is supported — changing the name of a cpuset directory while keeping its same parent. that's it, that's the whole vibe 🎀

📝 NOTES 📝

Despite its name, the pid parameter is actually a thread id, and each thread in a threaded group can be attached to a different cpuset. The value returned from a call to gettid(2) can be passed in the argument pid. sneaky naming ngl, but there you go 💜

🧪 EXAMPLES (yes bestie, we're doing hands-on) 🧪

The following examples demonstrate querying and setting cpuset options using shell commands.

🌸 Creating and attaching to a cpuset

To create a new cpuset and attach the current command shell to it, the steps are:

  1. mkdir /dev/cpuset (if not already done)
  2. mount -t cpuset none /dev/cpuset (if not already done)
  3. Create the new cpuset using mkdir(1).
  4. Assign CPUs and memory nodes to the new cpuset.
  5. Attach the shell to the new cpuset.

For example, the following sequence of commands will setup a cpuset named "Charlie", containing just CPUs 2 and 3, and memory node 1, and then attach the current shell to that cpuset.

mkdir /dev/cpuset
mount -t cpuset cpuset /dev/cpuset
cd /dev/cpuset
mkdir Charlie
cd Charlie
/bin/echo 2-3 > cpus
/bin/echo 1 > mems
/bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
cat /proc/self/cpuset
💅 Charlie is literally living his main character moment. Not a single CPU more, not a single node less.

🦋 Migrating a job to different memory nodes

To migrate a job (the set of tasks attached to a cpuset) to different CPUs and memory nodes in the system, including moving the memory pages currently allocated to that job, perform the following steps.

  1. Lets say we want to move the job in cpuset alpha (CPUs 4-7 and memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes 8-9).
  2. First create the new cpuset beta.
  3. Then allow CPUs 16-19 and memory nodes 8-9 in beta.
  4. Then enable memory_migration in beta.
  5. Then move each task from alpha to beta.

The following sequence of commands accomplishes this.

cd /dev/cpuset
mkdir beta
cd beta
/bin/echo 16-19 > cpus
/bin/echo 8-9 > mems
/bin/echo 1 > memory_migrate
while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

The above should move any tasks in alpha to beta, and any memory held by these tasks on memory nodes 2-3 to memory nodes 8-9, respectively.

Notice that the last step of the above sequence did not do:

cp ../alpha/tasks tasks

The while loop, rather than the seemingly easier use of the cp(1) command, was necessary because only one task PID at a time may be written to the tasks file.

😭 bestie do NOT try to be cute and cp the tasks file. One PID at a time. That's the law.

The same affect (writing one pid at a time) as the while loop can be accomplished more efficiently, in fewer keystrokes and in syntax that works on any shell, but alas more obscurely, by using the sed -u [unbuffered] option:

sed -un p < ../alpha/tasks > tasks

💀 ERRORS 💀

The Linux kernel implementation of cpusets sets errno to specify the reason for a failed system call affecting cpusets.

The possible errno settings and their meaning when set on a failed cpuset call are as listed below.

  • ENOMEM — Insufficient memory is available.
  • EBUSY — Attempted to remove a cpuset with attached tasks.
  • EBUSY — Attempted to remove a cpuset with child cpusets.
  • ENOENT — Attempted to create a cpuset in a parent cpuset that doesn't exist.
  • ENOENT — Attempted to access a non-existent file in a cpuset directory.
  • EEXIST — Attempted to create a cpuset that already exists.
  • EEXIST — Attempted to rename(2) a cpuset to a name that already exists.
  • ENOTDIR — Attempted to rename(2) a non-existent cpuset.
  • E2BIG — Attempted a write(2) system call on a special cpuset file with a length larger than some kernel determined upper limit on the length of such writes.
  • ESRCH — Attempted to write the process ID (PID) of a non-existent task to a cpuset tasks file.
  • EACCES — Attempted to write the process ID (PID) of a task to a cpuset tasks file when one lacks permission to move that task.
  • EACCESS — Attempted to write(2) a memory_pressure file.
  • ENOSPC — Attempted to write the process ID (PID) of a task to a cpuset tasks file when the cpuset had an empty cpus or empty mems setting.
  • EINVAL — Attempted to change a cpuset in a way that would violate a cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.
  • EINVAL — Attempted to write(2) an empty cpus or mems list to the kernel. The kernel creates new cpusets (via mkdir(2)) with empty cpus and mems. But the kernel will not allow an empty list to be written to the special cpus or mems files of a cpuset.
  • EIO — Attempted to write(2) a string to a cpuset tasks file that does not begin with an ASCII decimal integer.
  • EIO — Attempted to rename(2) a cpuset outside of its current directory.
  • ENOSPC — Attempted to write(2) a list to a cpus file that did not include any online CPUs.
  • ENOSPC — Attempted to write(2) a list to a mems file that did not include any online memory nodes.
  • ENODEV — The cpuset was removed by another task at the same time as a write(2) was attempted on one of the special files in the cpuset directory.
  • EACCES — Attempted to add a CPU or memory node to a cpuset that is not already in its parent.
  • EACCES — Attempted to set cpu_exclusive or mem_exclusive on a cpuset whose parent lacks the same setting.
  • EBUSY — Attempted to remove a CPU or memory node from a cpuset that is also in a child of that cpuset.
  • EFAULT — Attempted to read(2) or write(2) a cpuset file using a buffer that is outside your accessible address space.
  • ENAMETOOLONG — Attempted to read a /proc/<pid>/cpuset file for a cpuset path that is longer than the kernel page size.
  • ENAMETOOLONG — Attempted to create a cpuset whose base directory name is longer than 255 characters.
  • ENAMETOOLONG — Attempted to create a cpuset whose full pathname including the "/dev/cpuset/" prefix is longer than 4095 characters.
  • EINVAL — Specified a cpus or mems list to the kernel which included a range with the second number smaller than the first number.
  • EINVAL — Specified a cpus or mems list to the kernel which included an invalid character in the string.
  • ERANGE — Specified a cpus or mems list to the kernel which included a number too large for the kernel to set in its bitmasks.
ngl the errno table is giving "character development arc for your shell script". screenshot it, keep it, it WILL save you at 2am.

👀 SEE ALSO 👀

📜 HISTORY 📜

Cpusets appeared in version 2.6.13 of the Linux kernel.

🐛 BUGS 🐛

memory_pressure cpuset files can be opened for writing, creation or truncation, but then the write(2) fails with errno == EACCESS, and the creation and truncation options on open(2) have no affect.

fr fr, opening it works, writing to it does not. don't let it gaslight you. 🙈

✍️ AUTHOR ✍️

This man page was written by Paul Jackson.

and that's a wrap on the whole cpuset saga, bestie 🎀 you read all of it, that's genuinely iconic of you. go forth and pin your processes to exactly the CPUs they deserve. 🌈💜

💖 sourced from <strong>Linux 2.6</strong>, dated <strong>2006-05-25</strong>, page <strong>CPUSET(4)</strong> — periodt. 💖

📦 where this page comes from