DOMAINSET(9)

💖 domainset(9) — the domainset functions and operation manual 💖

okay bestie, sit down, hydrate, and let's talk about domainset(9) — that's the kernel developer's manual page for the domainset API, and honestly? it's giving NUMA queen. 👑

---

✨ NAME ✨

domainset — domainset functions and operation 🦋

---

🌸 SYNOPSIS 🌸

first things first, you gotta invite these headers to the party:

#include <sys/_domainset.h>
#include <sys/domainset.h>

and here's the struct itself, the whole vibe, the moment:

struct domainset {
        domainset_t     ds_mask;
        uint16_t        ds_policy;
        domainid_t      ds_prefer;
        ...
};

now the actual cast of characters:

struct domainset *
DOMAINSET_FIXED(domain);

struct domainset *
DOMAINSET_RR();

struct domainset *
DOMAINSET_PREF(domain);

struct domainset *
domainset_create(const struct domainset *key);

int
sysctl_handle_domainset(SYSCTL_HANDLER_ARGS);

---

💅 DESCRIPTION 💅

so the domainset(9) API is literally the memory domain allocation policy for NUMA machines. that's its whole thing. every domainset holds a bitmask of allowed domains, an integer policy, and an optional preferred domain. and together? together they spec the search order for memory allocations, plus the ability to restrict threads and objects to just a subset of the available memory domains — which is what you want for system partitioning and resource management. slay.

every single thread in the system has a reference to a struct domainset. and optionally, so does every vm_object_t — which is the thing used to represent files and other memory sources. when an allocation happens, the domainset associated with the object gets consulted first, and if there isn't one, the system falls back to the thread policy. so it's like object policy is the main character and thread policy is the understudy. ✨

the allocation policy can be one of these values, and bestie, let's go through all four:

• DOMAINSET_POLICY_ROUNDROBIN — memory gets allocated from each domain in the mask in a round-robin fashion. this spreads the bandwidth evenly across the available domains. and yes, this policy can also specify a single domain for a fixed allocation. iconic.
• DOMAINSET_POLICY_FIRSTTOUCH — memory is allocated from the node it's first accessed on. if the current domain isn't in the allowed set, or it's out of memory, the allocation falls back to round-robin. this policy optimizes for locality, but ngl it can give pessimal results if the memory gets accessed from a bunch of CPUs that aren't in the local domain. 💀
• DOMAINSET_POLICY_PREFER — memory is allocated from the node sitting in the prefer member. the preferred node must be set in the allowed mask, okay? no sneaking around that. and if the preferred node runs out of memory, the allocation falls back to round-robin among the allowed sets. 🎀
• DOMAINSET_POLICY_INTERLEAVE — memory is allocated in a striped fashion, with multiple pages allocated to each domain in the set according to the offset within the object. the strip width is object dependent and may be as large as a super-page (2MB on amd64). this gives good distribution among memory domains while keeping system efficiency higher — and it's preferential to round-robin for general use. highkey the better default, fr fr. 🫶

so the macros DOMAINSET_FIXED(), DOMAINSET_RR() and DOMAINSET_PREF() hand you pointers to global pre-defined policies — perfect for when you already know which policy you want at compile time. DOMAINSET_FIXED() is the policy that only permits allocations from the specified domain. DOMAINSET_RR() gives you round-robin selection among all domains in the system. and the DOMAINSET_PREF() policies try to allocate from the specified domain, but — and this is the important part — unlike DOMAINSET_FIXED(), they'll fall back to other domains to actually satisfy the request.

⚠️ heads up bestie: these policies should be used in preference to DOMAINSET_FIXED() so you don't end up blocking indefinitely on a M_WAITOK request. that's a real "we're stuck here forever" moment and we do not want that. 😭

the domainset_create() function takes a partially filled in domainset as a key and returns a valid domainset, or NULL. and listen — it is critical that consumers don't use domainsets that haven't been returned by this function. domainset is an immutable type, shared among all matching keys, and it must not be modified after return. no editing, no touching, no "just one little change". periodt. 💅

and finally, the sysctl_handle_domainset() function is there as a convenience for modifying or viewing domainsets that aren't accessible via cpuset(2). it's intended for use with sysctl(9). 🌈

---

🦋 SEE ALSO 🦋

• cpuset(1)
• cpuset(2)
• cpuset_setdomain(2)
• bitset(9)

---

👑 HISTORY 👑

<sys/domainset.h> first appeared in FreeBSD 12.0. a whole era. ⭐

---

and that's the whole page, bestie — domainset girlies stay winning. this was the Debian build of the page, dated October 30, 2018, and the manual is DOMAINSET(9). mwah. 💖✨🎀