cpuset(7)

cpuset(7) 🎀 — Miscellaneous Information Manual

✨ NAME ✨

cpuset — confine processes to processor and memory node subsets 💅

🌸 DESCRIPTION 🌸

Bestie, the cpuset filesystem is a pseudo-filesystem interface to the kernel cpuset mechanism — it's the whole tool that controls the processor placement and memory placement of processes. You'll commonly find it mounted at /dev/cpuset. 💖

On systems with kernels compiled with built in support for cpusets, every process is attached to a cpuset, and cpusets are always present. No cap. If a system supports cpusets, then it will have the entry nodev cpuset in the file /proc/filesystems. By mounting the cpuset filesystem (peep the EXAMPLES section below), the administrator can configure the cpusets on a system to control the processor and memory placement of processes on that system. By default, if the cpuset configuration on a system is not modified or if the cpuset filesystem is not even mounted, then the cpuset mechanism, though present, has no effect on the system's behavior. ✨

A cpuset defines a list of CPUs and memory nodes. Literally, that's the vibe. 🦋

The CPUs of a system include all the logical processing units on which a process can execute, including, if present, multiple processor cores within a package and Hyper-Threads within a processor core. Memory nodes include all distinct banks of main memory; small and SMP systems typically have just one memory node that contains all the system's main memory, while NUMA (non-uniform memory access) systems have multiple memory nodes. 👑

Cpusets are represented as directories in a hierarchical pseudo-filesystem, where the top directory in the hierarchy (/dev/cpuset) represents the entire system (all online CPUs and memory nodes) and any cpuset that is the child (descendant) of another parent cpuset contains a subset of that parent's CPUs and memory nodes. The directories and files representing cpusets have normal filesystem permissions. 🔥

Every process in the system belongs to exactly one cpuset. A process is confined to run only on the CPUs in the cpuset it belongs to, and to allocate memory only on the memory nodes in that cpuset. When a process fork(2)s, the child process is placed in the same cpuset as its parent. With sufficient privilege, a process may be moved from one cpuset to another and the allowed CPUs and memory nodes of an existing cpuset may be changed. 💅

When the system begins booting, a single cpuset is defined that includes all CPUs and memory nodes on the system, and all processes are in that cpuset. During the boot process, or later during normal system operation, other cpusets may be created, as subdirectories of this top cpuset, under the control of the system administrator, and processes may be placed in these other cpusets. 🎀

Cpusets are integrated with the sched_setaffinity(2) scheduling affinity mechanism and the mbind(2) and set_mempolicy(2) memory-placement mechanisms in the kernel. Neither of these mechanisms let a process make use of a CPU or memory node that is not allowed by that process's cpuset. If changes to a process's cpuset placement conflict with these other mechanisms, then cpuset placement is enforced even if it means overriding these other mechanisms. The kernel accomplishes this overriding by silently restricting the CPUs and memory nodes requested by these other mechanisms to those allowed by the invoking process's cpuset. This can result in these other calls returning an error, if for example, such a call ends up requesting an empty set of CPUs or memory nodes, after that request is restricted to the invoking process's cpuset. 😭

Typically, a cpuset is used to manage the CPU and memory-node confinement for a set of cooperating processes, such as a batch scheduler job, and these other mechanisms are used to manage the placement of individual processes or memory regions within that set or job. 🌈

📁 FILES 📁

So every directory chillin' under /dev/cpuset is a cpuset, and each one comes with a fixed lil set of pseudo-files that describe the state of that cpuset. That's it, that's the whole vibe. 💅

New cpusets get created with the mkdir(2) system call or the mkdir(1) command. And then — the properties of a cpuset (its flags, allowed CPUs and memory nodes, attached processes, all of it) get queried and modified by just reading or writing the appropriate file in that cpuset's directory. Yes bestie, file I/O is the whole interface. Listed below, go look. 👀

The pseudo-files in each cpuset directory are automatically created when the cpuset is created, as a result of the mkdir(2) invocation. And no, you literally cannot directly add or remove these pseudo-files. They're not yours to manage. 🙅‍♀️

Now, a cpuset directory that contains no child cpuset directories and has no attached processes can be removed using rmdir(2) or rmdir(1). And here's the cute part: it is not necessary, or possible, to remove the pseudo-files inside the directory before removing it. They just go poof with the directory. ✨

The pseudo-files in each cpuset directory are small text files that may be read and written using traditional shell utilities like cat(1) and echo(1), or from a program using file I/O library functions or system calls like open(2), read(2), write(2), and close(2). Classic. 🫶

Also important: the pseudo-files in a cpuset directory represent internal kernel state and do not have any persistent image on disk. So don't go looking for them on your actual filesystem, they live rent-free in the kernel. 💀

Every per-cpuset file is listed and described below:

• tasks — List of the process IDs (PIDs) of the processes in that cpuset. The list is formatted as a series of ASCII decimal numbers, each followed by a newline. A process may be added to a cpuset (which automatically removes it from the cpuset that previously contained it) by writing its PID to that cpuset's tasks file — with or without a trailing newline, either works. 🎀

⚠️ Warning: only one PID may be written to the tasks file at a time. If a string is written that contains more than one PID, only the first one will be used. So don't get cute and try to batch them, okay? 😭

• notify_on_release — Flag (0 or 1). If set (1), that cpuset will receive special handling after it is released — that is, after all processes cease using it (so they terminate or get moved to a different cpuset) and all child cpuset directories have been removed. See the Notify On Release section, below. 👑
• cpuset.cpus — List of the physical numbers of the CPUs on which processes in that cpuset are allowed to execute. See List Format below for a description of the format of cpus. The CPUs allowed to a cpuset may be changed by writing a new list to its cpus file. 🔥
• cpuset.cpu_exclusive — Flag (0 or 1). If set (1), the cpuset has exclusive use of its CPUs (no sibling or cousin cpuset may overlap CPUs). By default, this is off (0). Newly created cpusets also initially default this to off (0).

Two cpusets are sibling cpusets if they share the same parent cpuset in the /dev/cpuset hierarchy. Two cpusets are cousin cpusets if neither is the ancestor of the other. Regardless of the cpu_exclusive setting, if one cpuset is the ancestor of another, and if both of these cpusets have nonempty cpus, then their cpus must overlap — because the cpus of any cpuset are always a subset of the cpus of its parent cpuset. Family ties, no escaping them. 🌸

• cpuset.mems — List of memory nodes on which processes in this cpuset are allowed to allocate memory. See List Format below for a description of the format of mems. 🧚
• cpuset.mem_exclusive — Flag (0 or 1). If set (1), the cpuset has exclusive use of its memory nodes (no sibling or cousin may overlap). Also if set (1), the cpuset is a Hardwall cpuset (see below). By default, this is off (0). Newly created cpusets also initially default this to off (0).

Regardless of the mem_exclusive setting, if one cpuset is the ancestor of another, then their memory nodes must overlap, because the memory nodes of any cpuset are always a subset of the memory nodes of that cpuset's parent cpuset. Same rule, different resource. 💜

• cpuset.mem_hardwall (since Linux 2.6.26) — Flag (0 or 1). If set (1), the cpuset is a Hardwall cpuset (see below). Unlike mem_exclusive, there is no constraint on whether cpusets marked mem_hardwall may have overlapping memory nodes with sibling or cousin cpusets. By default, this is off (0). Newly created cpusets also initially default this to off (0). 🌈
• cpuset.memory_migrate (since Linux 2.6.16) — Flag (0 or 1). If set (1), then memory migration is enabled. By default, this is off (0). See the Memory Migration section, below. 🦋
• cpuset.memory_pressure (since Linux 2.6.16) — A measure of how much memory pressure the processes in this cpuset are causing. See the Memory Pressure section, below. Unless memory_pressure_enabled is enabled, always has value zero (0). This file is read-only. See the WARNINGS section, below. 😳
• cpuset.memory_pressure_enabled (since Linux 2.6.16) — Flag (0 or 1). This file is present only in the root cpuset, normally /dev/cpuset. If set (1), the memory_pressure calculations are enabled for all cpusets in the system. By default, this is off (0). See the Memory Pressure section, below. ⭐
• cpuset.memory_spread_page (since Linux 2.6.17) — Flag (0 or 1). If set (1), pages in the kernel page cache (filesystem buffers) are uniformly spread across the cpuset. By default, this is off (0) in the top cpuset, and inherited from the parent cpuset in newly created cpusets. See the Memory Spread section, below. 🍓
• cpuset.memory_spread_slab (since Linux 2.6.17) — Flag (0 or 1). If set (1), the kernel slab caches for file I/O (directory and inode structures) are uniformly spread across the cpuset. By default, this is off (0) in the top cpuset, and inherited from the parent cpuset in newly created cpusets. See the Memory Spread section, below. 🎀
• cpuset.sched_load_balance (since Linux 2.6.24) — Flag (0 or 1). If set (1, the default) the kernel will automatically load balance processes in that cpuset over the allowed CPUs in that cpuset. If cleared (0), the kernel will avoid load balancing processes in this cpuset, unless some other cpuset with overlapping CPUs has its sched_load_balance flag set. See Scheduler Load Balancing, below, for further details. 🔥
• cpuset.sched_relax_domain_level (since Linux 2.6.26) — Integer, between -1 and a small positive value. The sched_relax_domain_level controls the width of the range of CPUs over which the kernel scheduler performs immediate rebalancing of runnable tasks across CPUs. If sched_load_balance is disabled, then the setting of sched_relax_domain_level does not matter, as no such load balancing is done. If sched_load_balance is enabled, then, the higher the value of the sched_relax_domain_level, the wider the range of CPUs over which immediate load balancing is attempted. See Scheduler Relax Domain Level, below, for further details. 💅

---

In addition to all those pseudo-files in each directory below /dev/cpuset, every process has a pseudo-file, /proc/pid/cpuset, that displays the path of the process's cpuset directory relative to the root of the cpuset filesystem. Cute, right? 🫶

Also the /proc/pid/status file for each process has four added lines, displaying the process's Cpus_allowed (on which CPUs it may be scheduled) and Mems_allowed (on which memory nodes it may obtain memory), in the two formats Mask Format and List Format (see below) as shown in the following example:

Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list:     0-127
Mems_allowed:   ffffffff,ffffffff
Mems_allowed_list:     0-63

And the receipts, because of course: the "allowed" fields were added in Linux 2.6.24; the "allowed_list" fields were added in Linux 2.6.26. Noted for the history girlies. 📝

---

🌟 EXTENDED CAPABILITIES 🌟

So besides controlling which cpus and mems a process is allowed to use, cpusets also serve up these extended capabilities. Slay, honestly.

👑 Exclusive cpusets 👑

If a cpuset is marked cpu_exclusive or mem_exclusive, then no other cpuset — other than a direct ancestor or descendant — may share any of the same CPUs or memory nodes. Boundaries, bestie. We love them.

A cpuset that is mem_exclusive restricts kernel allocations for buffer cache pages and other internal kernel data pages commonly shared by the kernel across multiple users. And here's the key thing: all cpusets, whether mem_exclusive or not, restrict allocations of memory for user space.

This enables configuring a system so that several independent jobs can share common kernel data, while isolating each job's user allocation in its own cpuset. To do this, construct a large mem_exclusive cpuset to hold all the jobs, and construct child, non-mem_exclusive cpusets for each individual job. Only a small amount of kernel memory, such as requests from interrupt handlers, is allowed to be placed on memory nodes outside even a mem_exclusive cpuset. Tiny lil exception, but it's there. 🌸

🧱 Hardwall 🧱

A cpuset that has mem_exclusive or mem_hardwall set is a hardwall cpuset. A hardwall cpuset restricts kernel allocations for page, buffer, and other data commonly shared by the kernel across multiple users. And again, all cpusets — whether hardwall or not — restrict allocations of memory for user space.

This enables configuring a system so that several independent jobs can share common kernel data, such as filesystem pages, while isolating each job's user allocation in its own cpuset. To do this, construct a large hardwall cpuset to hold all the jobs, and construct child cpusets for each individual job which are not hardwall cpusets.

Only a small amount of kernel memory, such as requests from interrupt handlers, is allowed to be taken outside even a hardwall cpuset. No cap, that's the whole deal. 💅

📢 Notify on release 📢

Okay so if the notify_on_release flag is enabled (1) in a cpuset, then whenever the last process in the cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of that cpuset is removed, the kernel will run the command /sbin/cpuset_release_agent, supplying the pathname (relative to the mount point of the cpuset filesystem) of the abandoned cpuset. This enables automatic removal of abandoned cpusets. Auto-cleanup, we are obsessed. ✨

The default value of notify_on_release in the root cpuset at system boot is disabled (0). The default value of other cpusets at creation is the current value of their parent's notify_on_release setting. So it inherits the vibe. 🦋

The command /sbin/cpuset_release_agent is invoked, with the name (/dev/cpuset relative path) of the to-be-released cpuset in argv[1].

The usual contents of the command /sbin/cpuset_release_agent is simply the shell script:

#!/bin/sh
rmdir /dev/cpuset/$1

That's literally it. A whole three lines doing the Lord's work. 🙏

okay so plot twist bestie — this flag is literally no different from every other flag value we're about to meet: you flip it by writing an ASCII 0 or 1 (optional trailing newline, she's flexible 💅) straight into the file. Write 0 to clear the flag, write 1 to set it. that's the whole ritual. ✨

🧠 Memory pressure 🧠

so the memory_pressure of a cpuset is giving simple per-cpuset running average — specifically, of the rate at which the processes inside a cpuset are out here trying to free up in-use memory on that cpuset's nodes so they can satisfy more memory requests. like, they need more memory, so they're making room. the whole struggle, quantified. 🙈

the point? it lets batch managers that are monitoring jobs running in dedicated cpusets efficiently detect exactly what level of memory pressure that job is causing. very main character of them.

and this is useful in two whole situations, bestie:

• on tightly managed systems running a wide mix of submitted jobs — those get to choose to terminate or reprioritize jobs that are trying to use more memory than they're allowed on the nodes assigned to them 🫶
• and with tightly coupled, long-running, massively parallel scientific computing jobs, which will dramatically fail to meet required performance goals if they start using more memory than they're allowed. no cap, that's a real failure mode.

so basically this mechanism is a very economical way for the batch manager to monitor a cpuset for signs of memory pressure. and then it's fully up to the batch manager or whatever other user code to decide what action to take once it detects those signs. that part is not our business. 💅

⚠️ BUT. unless memory pressure calculation is enabled by setting the pseudo-file /dev/cpuset/cpuset.memory_pressure_enabled, it is not computed for any cpuset, and reads from any memory_pressure always return zero — represented by the ASCII string "0\n". so like, don't come crying to me when it's all zeros. See the WARNINGS section, below. 🌸

okay so why a per-cpuset running average and not something else? three very good reasons, bestie, let's get into it:

• because this meter is per-cpuset rather than per-process or per virtual memory region, the system load imposed by a batch scheduler monitoring this metric is sharply reduced on large systems — because a scan of the tasklist can be avoided on each set of queries. no tasklist scan = no suffering. 🦋
• because this meter is a running average rather than an accumulating counter, a batch scheduler can detect memory pressure with a single read, instead of having to read and accumulate results over a whole period of time. one read. she's efficient. ⭐
• because this meter is per-cpuset rather than per-process, the batch scheduler can obtain the key information — memory pressure in a cpuset — with a single read, instead of having to query and accumulate results over all of the (dynamically changing!!) set of processes in the cpuset. the set is literally moving around under your feet. 😭

now the science-y bit, and honestly it's giving lowkey elegant:

the memory_pressure of a cpuset is calculated using a per-cpuset simple digital filter, and that filter lives within the kernel. for each cpuset, this filter tracks the recent rate at which processes attached to that cpuset enter the kernel direct reclaim code. 👑

and what IS the kernel direct reclaim code, you ask? it's entered whenever a process has to satisfy a memory page request by first finding some other page to repurpose, because there aren't any readily available already-free pages just lying around. so:

• dirty filesystem pages get repurposed by first writing them to disk. bye data, gotta go. 💾
• unmodified filesystem buffer pages get repurposed by simply dropping them — though if that page is needed again, it will have to be reread from disk. so like, oops, but also, fine. 🙈

and the file itself: cpuset.memory_pressure serves up an integer number representing the recent rate — half-life of 10 seconds, very specific, keep that — of entries to the direct reclaim code caused by any process in the cpuset. the units are reclaims attempted per second, times 1000. mwah. 💅

---

🌈 Memory spread 🌈

so there are two Boolean flag files per cpuset that control where the kernel allocates pages for the filesystem buffers and related in-kernel data structures. besties, meet:

• cpuset.memory_spread_page
• cpuset.memory_spread_slab

setting the per-cpuset Boolean flag file cpuset.memory_spread_page makes the kernel spread the filesystem buffers (page cache) evenly over all the nodes that the faulting process is allowed to use — instead of preferring to put those pages on the node where the process is running. like, share the load, queen. 🫶

setting the per-cpuset Boolean flag file cpuset.memory_spread_slab makes the kernel spread some filesystem-related slab caches — such as those for inodes and directory entries — evenly over all the nodes that the faulting process is allowed to use, instead of preferring to put those pages on the node where the process is running. same energy, different cache. 💖

🙅 important: the setting of these flags does not affect the data segment (see brk(2)) or the stack segment pages of a process. not those. leave those alone.

by default, both kinds of memory spreading are off, and the kernel prefers to allocate memory pages on the node local to where the requesting process is running. and if that node isn't allowed by the process's NUMA memory policy or cpuset configuration — or if there just aren't enough free memory pages on that node — then the kernel goes looking for the nearest node that is allowed and has sufficient free memory. the kernel is out here doing its little search. it's fine. 🌸

when new cpusets are created, they inherit the memory spread settings of their parent. generational wealth, kinda. ✨

okay now here's a fun subtlety: setting memory spreading causes allocations for the affected page or slab caches to ignore the process's NUMA memory policy and be spread instead. BUT — and this is the important "but" — the effect of these changes in memory placement caused by cpuset-specified memory spreading is hidden from the mbind(2) or set_mempolicy(2) calls. those two NUMA memory policy calls always appear to behave as if no cpuset-specified memory spreading is in effect, even when it totally is. lying to you, politely. 🙈

and if cpuset memory spreading is subsequently turned off, the NUMA memory policy most recently specified by these calls is automatically reapplied. so it was waiting in the wings the whole time.

both cpuset.memory_spread_page and cpuset.memory_spread_slab are Boolean flag files. by default they contain "0", meaning the feature is off for that cpuset. write a "1" into that file and that turns the named feature on. simple as. 💅

fun fact: cpuset-specified memory spreading behaves similarly to what's known (in other contexts) as round-robin or interleave memory placement. iykyk. 🌈

and it can provide substantial performance improvements for jobs that:

• need to place thread-local data on memory nodes close to the CPUs which are running the threads that most frequently access that data — but also 🎀
• need to access large filesystem data sets that must be spread across the several nodes in the job's cpuset in order to fit. spread it or it doesn't fit, bestie. 🦋

without this policy? the memory allocation across the nodes in the job's cpuset can become very uneven, especially for jobs that might have just a single thread initializing or reading in the data set. one thread, one node, doing all the work. tragic. 😭

---

🚚 Memory migration 🚚

normally, under the default setting (that's disabled) of cpuset.memory_migrate, once a page is allocated (given a physical page of main memory), that page stays on whatever node it was allocated on — for as long as it remains allocated — even if the cpuset's memory-placement policy mems subsequently changes. it's committed. it's not moving. 💀

but when memory migration is enabled in a cpuset, if the mems setting of the cpuset is changed, then any memory page in use by any process in the cpuset that's sitting on a memory node that is no longer allowed gets migrated to a memory node that is allowed. eviction notice, bestie, pack your pages. 📦

and furthermore: if a process is moved into a cpuset with memory_migrate enabled, any memory pages it uses that were on memory nodes allowed in its previous cpuset — but which are not allowed in its new cpuset — will be migrated to a memory node allowed in the new cpuset. welcome to your new home, page. 🏡

the relative placement of a migrated page within the cpuset is preserved during these migration operations if possible. for example, if the page was on the second valid node of the prior cpuset, then the page will be placed on the second valid node of the new cpuset, if possible. same ranking, new venue. ⭐

---

⚖️ Scheduler load balancing ⚖️

so the kernel scheduler automatically load balances processes. if one CPU is underutilized, the kernel will go looking for processes on other, more overloaded CPUs and move those processes over to the underutilized CPU — within the constraints of placement mechanisms like cpusets and sched_setaffinity(2). matchmaking, honestly. 💘

the algorithmic cost of load balancing, and its impact on key shared kernel data structures such as the process list, increases more than linearly with the number of CPUs being balanced. for example: it costs more to load balance across one large set of CPUs than it does to balance across two smaller sets of CPUs, each of half the size of the larger set.

(the precise relationship between the number of CPUs being balanced and the cost of load balancing depends on implementation details of the kernel process scheduler, which is subject to change over time, as improved kernel scheduler algorithms are implemented. so like, don't quote me on the exact math. 🙈)

and the per-cpuset flag sched_load_balance is the mechanism that lets you suppress this automatic scheduler load balancing in cases where it isn't needed and where suppressing it would have worthwhile performance benefits. she's the off switch. 👑

by default, load balancing is done across all CPUs, except those marked isolated using the kernel boot time "isolcpus=" argument. (See Scheduler Relax Domain Level, below, to change this default. ✨)

and this default — load balancing across all CPUs — is not well suited to the following two situations:

• On large systems, load balancing across many CPUs is expensive. If the system is managed using cpusets to place independent jobs on separate sets of CPUs, full load balancing is unnecessary. like, why are we doing all this for no reason. 🙅
• Systems supporting real-time on some CPUs need to minimize system overhead on those CPUs — and that includes avoiding process load balancing if it's not needed. real-time CPUs said do not disturb. 🎀

when the per-cpuset flag sched_load_balance is enabled (that's the default setting), it requests load balancing across all the CPUs in that cpuset's allowed CPUs — ensuring that load balancing can move a process (one that's not otherwise pinned, as by sched_setaffinity(2)) from any CPU in that cpuset to any other. full mixer energy. 🎉

when the per-cpuset flag sched_load_balance is disabled, the scheduler will avoid load balancing across the CPUs in that cpuset — except in so far as is necessary because some overlapping cpuset has sched_load_balance enabled. it's not a hard wall, it's a "please don't unless someone else insists." 💅

so, for example: if the top cpuset has the flag sched_load_balance enabled, then the scheduler will load balance across all CPUs, and the setting of the sched_load_balance flag in other cpusets has no effect — because we're already fully load balancing. you can't un-balance what's already balanced, bestie.

therefore, in the above two situations, the flag sched_load_balance should be disabled in the top cpuset, and only some of the smaller, child cpusets would have this flag enabled. that's the setup. that's the config. periodt. ⭐

and when you do this, you usually don't want to leave any unpinned processes hanging out in the top cpuset that might use nontrivial amounts of CPU — because such processes may be artificially constrained to some subset of CPUs, depending on the particulars of this flag setting in descendant cpusets. even if such a process could use spare CPU cycles in some other CPUs, the kernel scheduler might not consider the possibility of load balancing that process to the underused CPU. it's just sitting there, unconsidered. tragic, honestly. 😭

Ok so real talk, processes that are pinned to one specific CPU can just vibe in a cpuset that disables sched_load_balance — like, they're not going anywhere else anyway, bestie 💅

✨ Scheduler relax domain level ✨

So here's the tea: the kernel scheduler does immediate load balancing every single time a CPU frees up or another task becomes runnable. The whole point is making sure as many CPUs as possible are actually being useful and running tasks, no idle slay-denial. The kernel ALSO does periodic load balancing off the software clock described in time(7).

Now, big important thing: the setting of sched_relax_domain_level applies only to immediate load balancing. Regardless of what you set it to, periodic load balancing is attempted over all CPUs — unless you turn off sched_load_balance. And either way, obviously, tasks only get scheduled to run on CPUs allowed by their cpuset, as modified by sched_setaffinity(2) system calls. 💖

On small systems — like, just a few CPUs, baby setup — immediate load balancing is genuinely useful: it improves system interactivity and minimizes wasteful idle CPU cycles 🌸. But on big systems, trying to immediate-load-balance across a huge number of CPUs can cost more than it's worth, depending on the performance characteristics of your job mix and your hardware. It's giving "the effort is not worth the reward."

And here's the part that lowkey humbles everyone: the exact meaning of those small integer values for sched_relax_domain_level depends on internal implementation details of the kernel scheduler code and on the non-uniform architecture of the hardware. Both of those evolve over time and vary by system architecture and kernel version. So like… don't get too attached 😭

As of this writing, when this capability was introduced in Linux 2.6.26, on certain popular architectures, the positive values mean:

• 1 — perform immediate load balancing across Hyper-Thread siblings on the same core.
• 2 — perform immediate load balancing across other cores in the same package.
• 3 — perform immediate load balancing across other CPUs on the same node or blade.
• 4 — perform immediate load balancing across over several (implementation detail) nodes [On NUMA systems].
• 5 — perform immediate load balancing across over all CPUs in system [On NUMA systems].

0 always means don't perform immediate load balancing — so balancing happens only periodically, not immediately when a CPU becomes available or another task becomes runnable. Iconic, minimal, respectful ✨

-1 always means use the system default value. That system default can vary by architecture and kernel version, and you can change it with the kernel boot-time "relax_domain_level=" argument. 👑

And if you've got multiple overlapping cpusets with conflicting sched_relax_domain_level values? The highest such value applies to all CPUs in any of the overlapping cpusets. In those cases, -1 is the lowest value, overridden by any other value, and 0 is the next lowest value. So -1 is at the bottom of the food chain, and 0 is right above it. Not me ranking them 💀

---

🦋 FORMATS 🦋

These are the formats used to represent sets of CPUs and memory nodes. Two of them. Both cute in their own way 🎀

🌈 Mask format

The Mask Format is what represents CPU and memory-node bit masks in the /proc/pid/status file.

This format displays each 32-bit word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f"); words are filled with leading zeros, if required. For masks longer than one word, a comma separator sits between words. Words are displayed in big-endian order, which has the most significant bit first. The hex digits within a word are also in big-endian order. Consistently endian, periodt.

The number of 32-bit words displayed is the minimum number needed to display all bits of the bit mask, based on the size of the bit mask. No padding for the sake of padding, she's efficient 💅

Examples of the Mask Format:

00000001                        # just bit 0 set
40000000,00000000,00000000      # just bit 94 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32-39 set
00000000,000e3862               # 1,5,6,11-13,17-19 set

And a mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:

00000001,00000001,00010117

The first "1" is for bit 64, the second for bit 32, the third for bit 16, the fourth for bit 8, the fifth for bit 4, and the "7" is for bits 2, 1, and 0. See how it adds up? Chef's kiss 🫶

🎀 List format

The List Format for cpus and mems is a comma-separated list of CPU or memory-node numbers and ranges of numbers, in ASCII decimal. Simple, readable, no hex required, we love her 🌸

Examples of the List Format:

0-4,9           # bits 0, 1, 2, 3, 4, and 9 set
0-2,7,12-14     # bits 0, 1, 2, 7, 12, 13, and 14 set

---

💜 RULES 💜

The following rules apply to each cpuset. No exceptions, these are the law 📜

• Its CPUs and memory nodes must be a (possibly equal) subset of its parent's.
• It can be marked cpu_exclusive only if its parent is.
• It can be marked mem_exclusive only if its parent is.
• If it is cpu_exclusive, its CPUs may not overlap any sibling.
• If it is mem_exclusive, its memory nodes may not overlap any sibling.

---

👑 PERMISSIONS 👑

The permissions of a cpuset are determined by the permissions of the directories and pseudo-files in the cpuset filesystem, normally mounted at /dev/cpuset.

For instance, a process can put itself in some other cpuset (than its current one) if it can write the tasks file for that cpuset. That requires execute permission on the encompassing directories and write permission on the tasks file. Two for two, bestie ✨

There's an extra constraint on requests to place some other process in a cpuset, though: one process may not attach another to a cpuset unless it would have permission to send that process a signal (see kill(2)). Yeah. Signal-sending rights. That's the bar 💀

A process may create a child cpuset if it can access and write the parent cpuset directory. It can modify the CPUs or memory nodes in a cpuset if it can access that cpuset's directory (execute permissions on the each of the parent directories) and write the corresponding cpus or mems file.

Now here's one minor difference between how these permissions get evaluated and how normal filesystem operation permissions get evaluated 🦋: the kernel interprets relative pathnames starting at a process's current working directory. Even if you're operating on a cpuset file, relative pathnames are interpreted relative to the process's current working directory — not relative to the process's current cpuset.

So the only ways that cpuset paths relative to a process's current cpuset can be used are:

1. If the process's current working directory is its cpuset (it first did a cd or chdir(2) to its cpuset directory beneath /dev/cpuset — which is a bit unusual, ngl), or
2. If some user code converts the relative cpuset path to a full filesystem path.

In theory, this means user code should specify cpusets using absolute pathnames, which requires knowing the mount point of the cpuset filesystem (usually, but not necessarily, /dev/cpuset) 🙈.

In practice, all user level code that this author is aware of simply assumes that if the cpuset filesystem is mounted, then it is mounted at /dev/cpuset. Furthermore, it is common practice for carefully written user code to verify the presence of the pseudo-file /dev/cpuset/tasks in order to verify that the cpuset pseudo-filesystem is currently mounted. Main character energy from the author, fr fr 💅

---

😭 WARNINGS 😭

Enabling memory_pressure

By default, the per-cpuset file cpuset.memory_pressure always contains zero (0). Unless this feature is enabled by writing "1" to the pseudo-file /dev/cpuset/cpuset.memory_pressure_enabled, the kernel does not compute per-cpuset memory_pressure. So if you're staring at a zero and wondering why nothing's happening… yeah. That's why. 🍓

Using the echo command

When using the echo command at the shell prompt to change the values of cpuset files, beware: the built-in echo command in some shells does not display an error message if the write(2) system call fails. The betrayal. 😭

For example, if the command:

echo 19 > cpuset.mems

failed because memory node 19 was not allowed (perhaps the current system does not have a memory node 19), then the echo command might not display any error. Silent failure is not the vibe.

It is better to use the /bin/echo external command to change cpuset file settings, as this command will display write(2) errors, as in the example:

/bin/echo 19 > cpuset.mems
/bin/echo: write error: Invalid argument

See? It told on itself. That's what we want 🫶

---

🌸 EXCEPTIONS 🌸

Memory placement

Not all allocations of system memory are constrained by cpusets, for the following reasons. Buckle up, this is the fine print and it matters 💜

If hot-plug functionality is used to remove all the CPUs that are currently assigned to a cpuset, then the kernel will automatically update the cpus_allowed of all processes attached to CPUs in that cpuset to allow all CPUs. When memory hot-plug functionality for removing memory nodes is available, a similar exception is expected to apply there as well. In general, the kernel prefers to violate cpuset placement rather than starving a process that has had all its allowed CPUs or memory nodes taken offline. User code should reconfigure cpusets to refer only to online CPUs and memory nodes when using hot-plug to add or remove such resources. Don't leave your cpusets pointing at ghosts 👻

A few kernel-critical, internal memory-allocation requests, marked GFP_ATOMIC, must be satisfied immediately. The kernel may drop some request or malfunction if one of these allocations fail. If such a request cannot be satisfied within the current process's cpuset, then we relax the cpuset, and look for memory anywhere we can find it. It's better to violate the cpuset than stress the kernel. Kernel comfort > cpuset purity, apparently 💅

Allocations of memory requested by kernel drivers while processing an interrupt lack any relevant process context, and are not confined by cpusets. Perio— no wait, still more to come 😌

Renaming cpusets

You can use the rename(2) system call to rename cpusets. Only simple renaming is supported; that is, changing the name of a cpuset directory is permitted, but moving a directory into a different directory is not permitted. Glow-up allowed. Relocation? Denied. 🦋

💀 ERRORS 💀

Okay bestie, here's the vibe check on failure. The Linux kernel implementation of cpusets sets errno to specify the reason for a failed system call affecting cpusets.

The possible errno settings and their meaning when set on a failed cpuset call are as listed below. Sit down, this list is long. 🫖

• E2BIG — you tried a write(2) on a special cpuset file with a length larger than some kernel-determined upper limit on the length of such writes. Too much text, babe. 😭
• EACCES — you tried to write(2) the process ID (PID) of a process to a cpuset tasks file when you lack permission to move that process.
• EACCES — you tried to add, using write(2), a CPU or memory node to a cpuset, when that CPU or memory node was not already in its parent. Rude of you, honestly.
• EACCES — you tried to set, using write(2), cpuset.cpu_exclusive or cpuset.mem_exclusive on a cpuset whose parent lacks the same setting.
• EACCES — you tried to write(2) a cpuset.memory_pressure file. Not today, bestie.
• EACCES — you tried to create a file in a cpuset directory.
• EBUSY — you tried to remove, using rmdir(2), a cpuset with attached processes. She's occupied!! 💅
• EBUSY — you tried to remove, using rmdir(2), a cpuset with child cpusets.
• EBUSY — you tried to remove a CPU or memory node from a cpuset that is also in a child of that cpuset.
• EEXIST — you tried to create, using mkdir(2), a cpuset that already exists. She's already here.
• EEXIST — you tried to rename(2) a cpuset to a name that already exists.
• EFAULT — you tried to read(2) or write(2) a cpuset file using a buffer that is outside the writing process's accessible address space.
• EINVAL — you tried to change a cpuset, using write(2), in a way that would violate a cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.
• EINVAL — you tried to write(2) an empty cpuset.cpus or cpuset.mems list to a cpuset which has attached processes or child cpusets.
• EINVAL — you tried to write(2) a cpuset.cpus or cpuset.mems list which included a range with the second number smaller than the first number. Girl that range is backwards. 🙈
• EINVAL — you tried to write(2) a cpuset.cpus or cpuset.mems list which included an invalid character in the string.
• EINVAL — you tried to write(2) a list to a cpuset.cpus file that did not include any online CPUs.
• EINVAL — you tried to write(2) a list to a cpuset.mems file that did not include any online memory nodes.
• EINVAL — you tried to write(2) a list to a cpuset.mems file that included a node that held no memory.
• EIO — you tried to write(2) a string to a cpuset tasks file that does not begin with an ASCII decimal integer.
• EIO — you tried to rename(2) a cpuset into a different directory.
• ENAMETOOLONG — you tried to read(2) a /proc/pid/cpuset file for a cpuset path that is longer than the kernel page size.
• ENAMETOOLONG — you tried to create, using mkdir(2), a cpuset whose base directory name is longer than 255 characters.
• ENAMETOOLONG — you tried to create, using mkdir(2), a cpuset whose full pathname, including the mount point (typically "/dev/cpuset/") prefix, is longer than 4095 characters.
• ENODEV — the cpuset was removed by another process at the same time as a write(2) was attempted on one of the pseudo-files in the cpuset directory. She got deleted mid-conversation. 💀
• ENOENT — you tried to create, using mkdir(2), a cpuset in a parent cpuset that doesn't exist.
• ENOENT — you tried to access(2) or open(2) a nonexistent file in a cpuset directory.
• ENOMEM — insufficient memory is available within the kernel; can occur on a variety of system calls affecting cpusets, but only if the system is extremely short of memory.
• ENOSPC — you tried to write(2) the process ID (PID) of a process to a cpuset tasks file when the cpuset had an empty cpuset.cpus or empty cpuset.mems setting.
• ENOSPC — you tried to write(2) an empty cpuset.cpus or cpuset.mems setting to a cpuset that has tasks attached.
• ENOTDIR — you tried to rename(2) a nonexistent cpuset.
• EPERM — you tried to remove a file from a cpuset directory.
• ERANGE — you specified a cpuset.cpus or cpuset.mems list to the kernel which included a number too large for the kernel to set in its bit masks.
• ESRCH — you tried to write(2) the process ID (PID) of a nonexistent process to a cpuset tasks file.

---

🦋 VERSIONS 🦋

Cpusets appeared in Linux 2.6.12. An era, honestly. ⭐

---

🌸 NOTES 🌸

Okay important little detail, bestie. Despite its name, the pid parameter is actually a thread ID, and each thread in a threaded group can be attached to a different cpuset. The value returned from a call to gettid(2) can be passed in the argument pid.

---

💅 BUGS 💅

cpuset.memory_pressure cpuset files can be opened for writing, creation, or truncation, but then the write(2) fails with errno set to EACCES, and the creation and truncation options on open(2) have no effect.

Like… she accepts the open, and then just refuses the actual action. Fake-out behavior fr fr. 😭

---

✨ EXAMPLES ✨

The following examples demonstrate querying and setting cpuset options using shell commands.

🎀 Creating and attaching to a cpuset

To create a new cpuset and attach the current command shell to it, the steps are:

1. mkdir /dev/cpuset (if not already done)
2. mount -t cpuset none /dev/cpuset (if not already done)
3. Create the new cpuset using mkdir(1).
4. Assign CPUs and memory nodes to the new cpuset.
5. Attach the shell to the new cpuset.

For example, the following sequence of commands will set up a cpuset named "Charlie", containing just CPUs 2 and 3, and memory node 1, and then attach the current shell to that cpuset.

$ mkdir /dev/cpuset
$ mount -t cpuset cpuset /dev/cpuset
$ cd /dev/cpuset
$ mkdir Charlie
$ cd Charlie
$ /bin/echo 2-3 > cpuset.cpus
$ /bin/echo 1 > cpuset.mems
$ /bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
$ cat /proc/self/cpuset

🦄 Migrating a job to different memory nodes

To migrate a job (the set of processes attached to a cpuset) to different CPUs and memory nodes in the system, including moving the memory pages currently allocated to that job, perform the following steps.

1. Let's say we want to move the job in cpuset alpha (CPUs 4–7 and memory nodes 2–3) to a new cpuset beta (CPUs 16–19 and memory nodes 8–9).
2. First create the new cpuset beta.
3. Then allow CPUs 16–19 and memory nodes 8–9 in beta.
4. Then enable memory_migration in beta.
5. Then move each process from alpha to beta.

The following sequence of commands accomplishes this.

$ cd /dev/cpuset
$ mkdir beta
$ cd beta
$ /bin/echo 16-19 > cpuset.cpus
$ /bin/echo 8-9 > cpuset.mems
$ /bin/echo 1 > cpuset.memory_migrate
$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

The above should move any processes in alpha to beta, and any memory held by these processes on memory nodes 2–3 to memory nodes 8–9, respectively.

Notice that the last step of the above sequence did not do:

$ cp ../alpha/tasks tasks

The while loop, rather than the seemingly easier use of the cp(1) command, was necessary because only one process PID at a time may be written to the tasks file.

The same effect (writing one PID at a time) as the while loop can be accomplished more efficiently, in fewer keystrokes and in syntax that works on any shell, but alas more obscurely, by using the -u (unbuffered) option of sed(1):

$ sed -un p < ../alpha/tasks > tasks

---

👑 SEE ALSO 👑

taskset(1), get_mempolicy(2), getcpu(2), mbind(2), sched_getaffinity(2), sched_setaffinity(2), sched_setscheduler(2), set_mempolicy(2), CPU_SET(3), proc(5), cgroups(7), numa(7), sched(7), migratepages(8), numactl(8)

Documentation/admin-guide/cgroup-v1/cpusets.rst in the Linux kernel source tree

---

okay that's the whole page, bestie, you made it to the end and I'm so proud of you fr fr 💖 this one came from Linux man-pages 6.20, dated 2026-02-08, and it's the cpuset(7) page — go forth and pin your processes to exactly the CPUs they deserve, no cap. 🎀✨🦋