LIBCPUSET(3)

LIBCPUSET(3) 💅✨

okay bestie, welcome to LIBCPUSET(3) — part 1 of 2. This is the library for managing CPU and Memory placement, and it's lowkey iconic. We're starting with NAME and SYNOPSIS because that's the order, periodt. 💖

---

NAME 🎀

so the whole squad is:

cpuset, cpuset_version, cpuset_pin, cpuset_size, cpuset_where, cpuset_unpin, cpuset_alloc, cpuset_free, cpuset_cpus_nbits, cpuset_mems_nbits, cpuset_setcpus, cpuset_setmems, cpuset_set_iopt, cpuset_set_sopt, cpuset_getcpus, cpuset_getmems, cpuset_cpus_weight, cpuset_mems_weight, cpuset_get_iopt, cpuset_get_sopt, cpuset_localcpus, cpuset_localmems, cpuset_cpumemdist, cpuset_cpu2node, cpuset_addr2node, cpuset_create, cpuset_delete, cpuset_query, cpuset_modify, cpuset_getcpusetpath, cpuset_cpusetofpid, cpuset_mountpoint, cpuset_collides_exclusive, cpuset_nuke, cpuset_init_pidlist, cpuset_pidlist_length, cpuset_get_pidlist, cpuset_freepidlist, cpuset_move, cpuset_move_all, cpuset_move_cpuset_tasks, cpuset_migrate, cpuset_migrate_all, cpuset_reattach, cpuset_open_memory_pressure, cpuset_read_memory_pressure, cpuset_close_memory_pressure, cpuset_c_rel_to_sys_cpu, cpuset_c_sys_to_rel_cpu, cpuset_c_rel_to_sys_mem, cpuset_c_sys_to_rel_mem, cpuset_p_rel_to_sys_cpu, cpuset_p_sys_to_rel_cpu, cpuset_p_rel_to_sys_mem, cpuset_p_sys_to_rel_mem, cpuset_get_placement, cpuset_equal_placement, cpuset_free_placement, cpuset_fts_open, cpuset_fts_read, cpuset_fts_reverse, cpuset_fts_rewind, cpuset_fts_get_path, cpuset_fts_get_stat, cpuset_fts_get_cpuset, cpuset_fts_get_errno, cpuset_fts_get_info, cpuset_fts_close, cpuset_cpubind, cpuset_latestcpu, cpuset_membind, cpuset_get_cpu_affinity, cpuset_set_cpu_affinity, cpuset_get_mem_affinity, cpuset_export, cpuset_import, cpuset_function — and the vibe is Library for managing CPU and Memory placement. No cap, that's the whole point. 🦋

---

SYNOPSIS ✨

first, the includes and the compile line, bestie:

#include <bitmask.h>
#include <cpuset.h>

cc ... -lcpuset

then we get into the function groups. buckle up. 💅

Basic cpuset routines 🧸

these are the basics, fr fr:

int cpuset_pin(int relcpu);
int cpuset_size();
int cpuset_where();
int cpuset_unpin();

Basic cpuset routines (Fortran interface) 🧚

Fortran besties, this one's for you:

include 'cpusetf.h'
integer relcpu

irtn = cpuset_pin(relcpu)
irtn = cpuset_size()
irtn = cpuset_unpin()
irtn = cpuset_where()

Cpuset library version. 👑

you can check the version, slay:

int cpuset_version();

Allocate and free struct cpuset *: 🎀

struct cpuset *cpuset_alloc();
void cpuset_free(struct cpuset *cp);

Lengths of CPUs and Memory Nodes bitmasks - use to alloc them: 🌸

int cpuset_cpus_nbits();
int cpuset_mems_nbits();

Set various attributes of a cpuset: 💖

int cpuset_setcpus(struct cpuset *cp, const struct bitmask *cpus);
void cpuset_setmems(struct cpuset *cp, const struct bitmask *mems);
int cpuset_set_iopt(struct cpuset *cp, const char *optname, int val);
int cpuset_set_sopt(struct cpuset *cp, const char *optname, const char *val);

Query various attributes of a cpuset: 🔍

int cpuset_getcpus(const struct cpuset *cp, struct bitmask *cpus);
int cpuset_getmems(const struct cpuset *cp, struct bitmask *mems);
int cpuset_cpus_weight(const struct cpuset *cp);
int cpuset_mems_weight(const struct cpuset *cp);
int cpuset_get_iopt(const struct cpuset *cp, const char *optname);
const char *cpuset_get_sopt(const struct cpuset *cp, const char *optname);

Local CPUs and Memory Nodes: 🗺️

int cpuset_localcpus(const struct bitmask *mems, struct bitmask *cpus);
int cpuset_localmems(const struct bitmask *cpus, struct bitmask *mems);
unsigned int cpuset_cpumemdist(int cpu, int mem);
int cpuset_cpu2node(int cpu);
int cpuset_addr2node(void *addr);

Create, delete, query, modify, list and examine cpusets: 🛠️

int cpuset_create(const char *path, const struct *cp);
int cpuset_delete(const char *path);
int cpuset_query(struct cpuset *cp, const char *path);
int cpuset_modify(const char *path, const struct *cp);
char *cpuset_getcpusetpath(pid_t pid, char *buf, size_t size);
int cpuset_cpusetofpid(struct cpuset *cp, int pid);
const char *cpuset_mountpoint();
int cpuset_collides_exclusive(const char *cpusetpath, const struct cpuset *cp);
int cpuset_nuke(const char *path, unsigned int seconds);

List tasks (pids) currently attached to a cpuset: 📋

struct cpuset_pidlist *cpuset_init_pidlist(const char *path, int recurse);
int cpuset_pidlist_length(const struct cpuset_pidlist *pl);
pid_t cpuset_get_pidlist(const struct cpuset_pidlist *pl, int i);
void cpuset_freepidlist(struct cpuset_pidlist *pl);

Attach tasks to cpusets: 📎

int cpuset_move(pid_t p, const char *path);
int cpuset_move_all(struct cpuset_pid_list *pl, const char *path);
int cpuset_move_cpuset_tasks(const char *fromrelpath, const char *torelpath);
int cpuset_migrate(pid_t pid, const char *path);
int cpuset_migrate_all(struct cpuset_pid_list *pl, const char *path);
int cpuset_reattach(const char *path);

Monitor memory pressure caused by tasks in a cpuset: 😮‍💨

int cpuset_open_memory_pressure(const char *cpusetpath);
int cpuset_read_memory_pressure(int han);
void cpuset_close_memory_pressure(int han);

Converting relative and system CPU and Memory Node numbers: 🔄

int cpuset_c_rel_to_sys_cpu(const struct cpuset *cp, int relcpu);
int cpuset_c_sys_to_rel_cpu(const struct cpuset *cp, int cpu);
int cpuset_c_rel_to_sys_mem(const struct cpuset *cp, int relmem);
int cpuset_c_sys_to_rel_mem(const struct cpuset *cp, int mem);
int cpuset_p_rel_to_sys_cpu(pid_t pid, int relcpu);
int cpuset_p_sys_to_rel_cpu(pid_t pid, int cpu);
int cpuset_p_rel_to_sys_mem(pid_t pid, int relmem);
int cpuset_p_sys_to_rel_mem(pid_t pid, int mem);

Placement operations for detecting cpuset migration: 🧭

struct cpuset_placement *cpuset_get_placement(pid_t pid);
int cpuset_equal_placement(const struct cpuset_placement *plc1,
         const struct cpuset_placement *plc2);
void cpuset_free_placement(struct cpuset_placement *plc);

Traverse a cpuset hierarchy: 🌳

struct cpuset_fts_tree *cpuset_fts_open(const char *cpusetpath);
const struct cpuset_fts_entry *cpuset_fts_read(
         struct cpuset_fts_tree *cs_tree);
void cpuset_fts_reverse(struct cpuset_fts_tree *cs_tree);
void cpuset_fts_rewind(struct cpuset_fts_tree *cs_tree);
const char *cpuset_fts_get_path(
         const struct cpuset_fts_entry *cs_entry);
const struct stat *cpuset_fts_get_stat(
         const struct cpuset_fts_entry *cs_entry);
const struct cpuset *cpuset_fts_get_cpuset(
         const struct cpuset_fts_entry *cs_entry);
int cpuset_fts_get_errno(
         const struct cpuset_fts_entry *cs_entry);
int cpuset_fts_get_info(
         const struct cpuset_fts_entry *cs_entry);
void cpuset_fts_close(struct cpuset_fts_tree *cs_tree);

Bind to a CPU or Memory Node within the current cpuset: 📌

int cpuset_cpubind(int cpu);
int cpuset_latestcpu(pid_t pid);
int cpuset_membind(int mem);

Export settings to and import from Text Representation: 📤📥

int cpuset_export(const struct cpuset *cp, char *buf, int buflen);
int cpuset_import(struct cpuset *cp, const char *file,
         int *errlinenum_ptr, char *errmsg_bufptr, int errmsg_buflen);

Calling [optional] cpuset_* API routines: 🧚

cpuset_function(const char *function_name);

💖 DESCRIPTION 💖

okay bestie, so cpusets are literally named subsets of a system's CPUs and Memory Nodes. On big SMP and NUMA systems, you lowkey need to control which processes and threads can run on which CPUs and Memory Nodes. That placement is how you highkey improve overall system performance and manage system utilization in line with the appropriate policies. 💅

This cpuset library hands you the mechanisms you need to create, destroy, and manage cpusets, plus control where processes and threads get placed inside them. 🫶

Even though this library has a ton of functions, most applications using it will only need the few functions in the Basic Cpuset API. Those basic functions let an app manage CPU and Memory placement on SMP and NUMA systems within cpusets that already exist. ✨

The Advanced Cpuset API provides support for creating, destroying, and managing cpusets, and for more complex placement within cpusets. Workload managers, batch processors, and other system services can use the Advanced Cpuset API to manage cpusets across an entire system, or a dedicated portion of a system. 👑

All functions are part of the same library, and fully interoperable. A system imposed permission model ensures that one application will not be able to make changes to the cpusets or placement of other applications, outside of its currently allowed cpuset. no cap. 🔒

✨ Basic Cpuset API ✨

The Basic Cpuset API provides functions usable from C for processor and memory placement within a cpuset.

The basic functions enable an application to place various threads of its execution on specific CPUs within its current cpuset, and perform related functions such as asking how large the current cpuset is, and on which CPU within the current cpuset a thread is currently executing. it's giving "where am I and how big is my world" energy. 🦋

The basic cpuset_pin(int relcpu) function uses cpuset relative numbering. In a cpuset of N CPUs, relcpu ranges from zero to N-1. All functions in the Advanced Cpuset API use system wide CPU and Memory Node numbering, to provide a consistent numbering regardless of cpuset configuration. 🌈

Memory placement is done automatically by the basic functions, local to the requested CPU. Threads may only be pinned on a single CPU, or unpinned and allowed the run of the entire current cpuset. This avoids the need to allocate and free the bitmasks required to specify a set of several CPUs. The basic functions do not support creating or removing cpusets, only the placement of threads within an existing cpuset. This avoids the need to explicitly allocate and free cpuset structures. Operations only apply to the current thread, avoiding the need to pass the process id of the thread to be affected. so convenient fr fr. 🎀

🎀 Basic Functions 🎀

The Basic Cpuset API supports the following functions.

int cpuset_pin(int relcpu);

Pin the current task to execute only on the CPU relcpu, which is a relative CPU number within the current cpuset of that task. Also automatically pin the memory allowed to be used by the current task to the memory on that same node (as determined by the advanced cpuset_cpu2node() function). one CPU, one memory node, locked in. 🔥

int cpuset_size();

Return the number of CPUs in the current tasks cpuset. The relative CPU numbers that are passed to the cpuset_pin() function and that are returned by the cpuset_where() function, must be between 0 and N - 1 inclusive, where N is the value returned by cpuset_size(). 💖

int cpuset_where();

Return the CPU number, relative to the current tasks cpuset, of the CPU on which the current task most recently executed. If a task is allowed to execute on more than one CPU, then there is no guarantee that the task is still executing on the CPU returned by cpuset_where, by the time that the user code obtains the return value. so yes, it's a little "where was I?" snapshot, not a live tracker. 🙈

int cpuset_unpin();

Remove the CPU and Memory pinning affects of any previous cpuset_pin call, allowing the current task to execute on any CPU in its current cpuset and to allocate memory on any Memory Node in its current cpuset. free bird era. 🦋

💀 ERRORS 💀

here's the error code lore, bestie. all the ways things can go sideways, in order:

• ENOSYS — Invoked on an operating system kernel that does not support cpusets.
• ENODEV — Invoked on a system that supports cpusets, but when the cpuset file system is not currently mounted at /dev/cpuset.
• ENOMEM — Insufficient memory is available.
• EBUSY — Attempted cpuset_delete() on a cpuset with attached tasks.
• EBUSY — Attempted cpuset_delete() on a cpuset with child cpusets.
• ENOENT — Attempted cpuset_create() in a parent cpuset that doesn't exist.
• EEXIST — Attempted cpuset_create() for a cpuset that already exists.
• EEXIST — Attempted to rename() a cpuset to a name that already exists.
• ENOTDIR — Attempted to rename() a non-existant cpuset.
• E2BIG — Attempted a write(2) system call on a special cpuset file with a length larger than some kernel determined upper limit on the length of such writes.
• ESRCH — Attempted to cpuset_move() a non-existance task.
• EACCES — Attempted to cpuset_move() a task which one lacks permission to move.
• EACCES — Attempted to write(2) a memory_pressure file.
• ENOSPC — Attempted to cpuset_move() a task to an empty cpuset.
• EINVAL — The relcpu argument to cpuset_pin() is out of range (not between zero and cpuset_size() - 1).
• EINVAL — Attempted to change a cpuset in a way that would violate a cpu_exclusive or mem_exclusive attribute of that cpuset or any of its siblings.
• EINVAL — Attempted to write an empty cpus or mems bitmask to the kernel. The kernel creates new cpusets (via mkdir) with empty cpus and mems, and the user level cpuset and bitmask code works with empty masks. But the kernel will not allow an empty bitmask (no bits set) to be written to the special cpus and mems files of a cpuset.
• EIO — Attempted to write(2) a string to a cpuset tasks file that does not begin with an ASCII decimal integer.
• EIO — Attempted to rename(2) a cpuset outside of its current directory.
• ENOSPC — Attempted to write(2) a list to a cpus file that did not include any online cpus.
• ENOSPC — Attempted to write(2) a list to a mems file that did not include any online memory nodes.
• EACCES — Attempted to add a cpu or mem to a cpuset that is not already in its parent.
• EACCES — Attempted to set cpu_exclusive or mem_exclusive on a cpuset whose parent lacks the same setting.
• ENODEV — The cpuset was removed by another task at the same time as a write(2) was attempted on one of the special files in the cpuset directory.
• EBUSY — Attempted to remove a cpu or mem from a cpuset that is also in a child of that cpuset.
• EFAULT — Attempted to read or write a cpuset file using a buffer that was outside your accessible address space.
• ENAMETOOLONG — Attempted to read a /proc/pid/cpuset file for a cpuset path that was longer than the kernel page size.
• ENAMETOOLONG — Attempted to create a cpuset whose base directory name was longer than 255 characters.
• ENAMETOOLONG — Attempted to create a cpuset whose full pathname including "/dev/cpuset" was longer than 4095 characters.
• ENXIO — Attempted to create a cpu_exclusive cpuset whose cpus covered just part of one or more physical processor packages, such as including just one of the two Cores on a package. For Linux kernel version 2.6.16 on i386 and x86_64, this operation is rejected with this error to avoid a fatal kernel bug. Otherwise, this is a normal and supported operation.
• EINVAL — Specified a cpus or mems list to the kernel which included a range with the second number smaller than the first number.
• EINVAL — Specified a cpus or mems list to the kernel which included an invalid character in the string.
• ERANGE — Specified a cpus or mems list to the kernel which included a number too large for the kernel to set in its bitmasks.
• ETIME — Time limit for cpuset_nuke() operation reached without successful completion of operation.
• ENOTEMPTY — Tasks remain after multiple attempts by cpuset_move_cpuset_tasks() to move them to a different cpuset.
• EPERM — Lacked permission to kill (send a signal to) a task.
• EPERM — Lacked permission to read a cpuset or its files.
• EPERM — Attempted to unlink a per-cpuset file. Such files can not be unlinked. They can only be removed by removing (rmdir) the directory representing the cpuset that contains these files.

🦋 Advanced Cpuset API 🦋

For additional documentation on cpusets, and for details of the all the other, advanced, routines, see /usr/share/doc/packages/libcpuset/libcpuset.html and /usr/share/doc/packages/libbitmask/libbitmask.html. These same documents are available in plain text format as /usr/share/doc/packages/libcpuset/libcpuset.txt and /usr/share/doc/packages/libbitmask/libbitmask.txt. go read, bestie, the lore is deep. 📚✨

👀 SEE ALSO 👀

libbitmask(3)

👑 AUTHOR 👑

pj@sgi.com (Paul Jackson)

---

💖 this iconic page is from Silicon Graphics, Inc, dated 2006-05-25, and it's LIBCPUSET(3) 💖