ATOMIC(9)

📦 version 15.1, in freebsd 14.5, freebsd 15.1

💅 ATOMIC(9) — the atomic operations page, explained by your favorite kernel bestie

okay so. this is ATOMIC(9), the Kernel Developer's Manual page for atomic operations in FreeBSD, and honestly? it's iconic. it's the whole "do a thing to memory and NOBODY can catch you mid-thing" page. let's get into it 🫶

✨ NAME ✨

so the whole squad of functions on this page is:

atomic_add, atomic_clear, atomic_cmpset, atomic_fcmpset, atomic_fetchadd, atomic_interrupt_fence, atomic_load, atomic_readandclear, atomic_set, atomic_subtract, atomic_store, atomic_thread_fence

…and collectively they're just: atomic operations. that's the vibe. 🌈

🌸 SYNOPSIS 🌸

first thing you do, literally always, is pull in the header:

#include <machine/atomic.h>

then all these signatures, exactly as written, no cap:

void
atomic_add_[acq_|rel_]<type>(volatile <type> *p, <type> v);
void
atomic_clear_[acq_|rel_]<type>(volatile <type> *p, <type> v);
int
atomic_cmpset_[acq_|rel_]<type>(volatile <type> *dst, <type> old,
    <type> new);
int
atomic_fcmpset_[acq_|rel_]<type>(volatile <type> *dst, <type> *old,
    <type> new);
<type>
atomic_fetchadd_<type>(volatile <type> *p, <type> v);
void
atomic_interrupt_fence(void);
<type>
atomic_load_[acq_]<type>(const volatile <type> *p);
<type>
atomic_readandclear_<type>(volatile <type> *p);
void
atomic_set_[acq_|rel_]<type>(volatile <type> *p, <type> v);
void
atomic_subtract_[acq_|rel_]<type>(volatile <type> *p, <type> v);
void
atomic_store_[rel_]<type>(volatile <type> *p, <type> v);
<type>
atomic_swap_<type>(volatile <type> *p, <type> v);
int
atomic_testandclear_<type>(volatile <type> *p, u_int v);
int
atomic_testandset_<type>(volatile <type> *p, u_int v);
void
atomic_thread_fence_[acq|acq_rel|rel|seq_cst](void);

like, look at those little suffixes. _acq, _rel, _seq_cst. they're doing SO much. 💅

🔥 DESCRIPTION 🔥

okay so what are atomic operations for? two main-character jobs, bestie:

  • they're the go-to for implementing reference counts, and
  • they're the building blocks for synchronization primitives, like mutexes.

and the whole point: all of these operations are performed atomically across multiple threads and in the presence of interrupts. read that again. it means from the POV of any other thread running at the same time, or any interrupt handler, the operation happens in an indivisible way. no half-measures. no "oops you caught me mid-write." it's giving main character energy, honestly. 👑

now here's a fact that trips people up so listen up 🧚:

on all architectures supported by FreeBSD, ordinary loads and stores of integers in cache-coherent memory are inherently atomic if the integer is naturally aligned and its size does not exceed the processor's word size. cool cool cool. BUT — and this is the big BUT — those loads and stores may be elided from the program by the compiler, whereas atomic operations are always performed. so like. the compiler can just… delete your normal load. it cannot delete an atomic. 💀

when atomic operations are performed on cache-coherent memory, all operations on the same location are totally ordered. everything knows its place. slay.

when an atomic load happens on a location in cache-coherent memory, it reads the entire value that was defined by the last atomic store to each byte of the location. and an atomic load will never return a value out of thin air. nope. not on our watch. when an atomic store happens on a location, no other thread or interrupt handler will observe a torn write, or partial modification of the location. no crumbs left behind. ✨

and except as noted below (yes, there's an "except as noted below," keep reading), the semantics of these operations are almost identical to the semantics of similarly named C11 atomic operations. almost. we do a couple things our own way. 🦋

💖 Types 💖

most atomic operations act upon a specific type, and that type is indicated in the function name. and here's the difference from C11: C11 atomic ops are on atomic types, but FreeBSD's atomic operations are performed on ordinary integer types. so the available types are:

  • int — unsigned integer
  • long — unsigned long integer
  • ptr — unsigned integer the size of a pointer
  • 32 — unsigned 32-bit integer
  • 64 — unsigned 64-bit integer

for example, the function to atomically add two integers is literally called atomic_add_int(). the name tells you everything. iykyk. 🎀

certain architectures also provide operations for types smaller than "int":

  • char — unsigned character
  • short — unsigned short integer
  • 8 — unsigned 8-bit integer
  • 16 — unsigned 16-bit integer
⚠️ these smaller types must not be used in machine-independent code. periodt. they're architecture-specific, so keep them out of your portable code or you're gonna have a bad time. 🙈

🌈 Acquire and Release Operations 🌈

okay this is the part where things get juicy, so buckle up.

by default, a thread's accesses to different memory locations might not be performed in program order — meaning, the order they appear in your source code. and why? because both the compiler and the processor might reorder the thread's accesses, all in the name of optimizing your program's execution.

now, both of them do guarantee that their reordering is not visible to the thread itself — otherwise the traditional memory model that single-threaded programs expect would be violated, and everything would fall apart. BUT. other threads in a multithreaded program — like, oh, idk, the FreeBSD kernel — might observe the reordering. 😭 and in some cases, like when you're implementing synchronization between threads, arbitrary reordering might result in the incorrect execution of the program. so. yeah. we need to constrain that reordering, and that's exactly what acquire and release semantics are for.

atomic operations on memory have up to three variants:

  1. the first, the relaxed variant, performs the operation without imposing any ordering constraints on accesses to other memory locations. this is the default.
  2. the second variant has acquire semantics.
  3. the third variant has release semantics.

acquire semantics, bestie: an atomic operation can only have acquire semantics if it performs a load from memory. when an atomic operation has acquire semantics, a load performed as part of the operation must have completed before any subsequent load or store (by program order) is performed. conversely — and this matters — acquire semantics do not require that prior loads or stores have completed before a load from the atomic operation is performed. to denote acquire semantics, the suffix _acq is inserted into the function name immediately prior to the _⟨type⟩ suffix. for example, to subtract two integers ensuring that the load of the value from memory is completed before any subsequent loads and stores are performed, use atomic_subtract_acq_int(). 🫶

release semantics, same energy but backwards: an atomic operation can only have release semantics if it performs a store to memory. when an atomic operation has release semantics, all prior loads or stores (by program order) must have completed before a store executed as part of the operation is performed. conversely, release semantics do not require that a store from the atomic operation must have completed before any subsequent load or store is performed. to denote release semantics, the suffix _rel is inserted into the function name immediately prior to the _⟨type⟩ suffix. for example, to add two long integers ensuring that all prior loads and stores are completed before the store of the result is performed, use atomic_add_rel_long(). ⭐

now, the synchronizing bit — this is where it all comes together:

when a release operation by one thread synchronizes with an acquire operation by another thread — which usually means the acquire operation reads the value written by the release operation — then:

  • the effects of all prior stores by the releasing thread must become visible to subsequent loads by the acquiring thread, and
  • moreover, the effects of all stores (by other threads) that were visible to the releasing thread must also become visible to the acquiring thread.

and the fine print: these rules only apply to the synchronizing threads. other threads might observe these stores in a different order. so don't go assuming everyone sees the same thing. 💀

in effect, atomic operations with acquire and release semantics establish one-way barriers to reordering — and that's what lets synchronization primitives express their ordering requirements without also imposing unnecessary ordering. for example: for a critical section guarded by a mutex, an acquire operation when the mutex is locked and a release operation when the mutex is unlocked will prevent any loads or stores from moving outside of the critical section. however, they will not prevent the compiler or processor from moving loads or stores into the critical section — which, funnily enough, does not violate the semantics of a mutex. so it's fine. it's fine!! 🧚

🦋 Architecture-dependent caveats for compare-and-swap 🦋

okay, caveat time. pay attention, this one's per-architecture and it's highkey important.

the atomic_[f]cmpset_<type>() operations — specifically those without explicitly specified memory ordering — are defined as relaxed. consequently, a thread's accesses to memory locations different from that of the atomic operation can be reordered in relation to the atomic operation. 😬

however, the implementation on the amd64 and i386 architectures provides sequentially consistent semantics. in particular, the reordering mentioned above cannot occur there. so on those, you get more than the contract promised. cute. ✨

on the arm64/aarch64 architecture, the operation may include either acquire semantics on the constituent load or release semantics on the constituent store. what does that mean in practice?

  • accesses to other locations in program order before the atomic might be observed as executed after the load that is part of the atomic operation (but not after the store from the operation, because of release).
  • similarly, accesses after the atomic might be observed as executed before the store.

so like. it's not nothing, but it's not everything either. go in knowing that. 💅

🌸 Thread Fence Operations 🌸

alternatively — and honestly this is such a serve — a programmer can use atomic thread fence operations to constrain the reordering of accesses. and in contrast to the other atomic operations, fences do not, themselves, access memory. they're just vibes and ordering. no memory touched. 🎀

acquire fence: when a fence has acquire semantics, all prior loads (by program order) must have completed before any subsequent load or store is performed. thus, an acquire fence is a two-way barrier for load operations. to denote acquire semantics, the suffix _acq is appended to the function name — for example, atomic_thread_fence_acq(). ⭐

release fence: when a fence has release semantics, all prior loads or stores (by program order) must have completed before any subsequent store operation is performed. thus, a release fence is a two-way barrier for store operations. to denote release semantics, the suffix _rel is appended to the function name — for example, atomic_thread_fence_rel(). ⭐

now this next bit is the part people get wrong, so listen:

although atomic_thread_fence_acq_rel() implements both acquire and release semantics, it is not a full barrier. for example, a store prior to the fence (in program order) may be completed after a load subsequent to the fence. 😭 in contrast, atomic_thread_fence_seq_cst() implements a full barrier — neither loads nor stores may cross this barrier in either direction. it's the wall. the actual wall. 🧱

and the C11 comparison, because we love a "well actually" moment:

in C11, a release fence by one thread synchronizes with an acquire fence by another thread when an atomic load that is prior to the acquire fence (by program order) reads the value written by an atomic store that is subsequent to the release fence.

in contrast, in FreeBSD, because of the atomicity of ordinary, naturally aligned loads and stores, fences can also be synchronized by ordinary loads and stores. and that's not a bug, that's a feature — it simplifies the implementation and use of some synchronization primitives in FreeBSD. slay. 🫶

finally, the why behind the two-way barrier thing: since neither a compiler nor a processor can foresee which (atomic) load will read the value written by an (atomic) store, the ordering constraints imposed by fences must be more restrictive than acquire loads and release stores. essentially, that's literally why fences are two-way barriers. and that's the whole reasoning. beautiful. iconic. no notes. 👑

so anyway, fences are extra — like they impose stricter ordering than acquire loads and release stores do, but the way they split access from ordering means they can sometimes lowkey unlock more efficient implementations of synchronization primitives. For example: you can use them to dodge executing a memory barrier until a memory access actually shows that some condition is satisfied. iykyk 💅

🌸 Interrupt Fence Operations 🌸

The atomic_interrupt_fence() function sets up ordering between the spot where you call it and any interrupt handler running on the same CPU. It's modeled after the similar C11 atomic_signal_fence() function, just adapted for the kernel environment. cute, functional, iconic. ✨

🦋 Multiple Processors 🦋

On multiprocessor systems, whether your atomic ops on memory are actually atomic depends on the underlying architecture supporting cache coherence. In general, cache coherence on the default memory type, VM_MEMATTR_DEFAULT, is guaranteed by all architectures FreeBSD supports. For example, cache coherence is guaranteed on write-back memory by the amd64 and i386 architectures. 🫶

But heads up bestie — on some architectures, cache coherence might not be enabled on all memory types. To figure out if it's enabled for a non-default memory type, go consult that architecture's documentation. no cap, that's the move. 📖

💜 Semantics 💜

This section breaks down the semantics of each operation using C-like notation. slay, let's go. 🔥

atomic_add(p, v)

*p += v;

atomic_clear(p, v)

*p &= ~v;

atomic_cmpset(dst, old, new)

if (*dst == old) {
        *dst = new;
        return (1);
} else
        return (0);
Some architectures do not implement the atomic_cmpset() functions for the types "char", "short", "8", and "16". 😭

atomic_fcmpset(dst, *old, new)

On architectures implementing the Compare And Swap operation in hardware, the functionality can be described as:

if (*dst == *old) {
        *dst = new;
        return (1);
} else {
        *old = *dst;
        return (0);
}

On architectures that provide the Load Linked/Store Conditional primitive, the write to *dst might also fail for several reasons — the most important one being a parallel write to *dst cache line by another CPU. In that case atomic_fcmpset() also returns false, despite:

*old == *dst.
And yeah — some architectures do not implement the atomic_fcmpset() functions for the types "char", "short", "8", and "16". 💀

atomic_fetchadd(p, v)

tmp = *p;
*p += v;
return (tmp);
The atomic_fetchadd() functions are only implemented for the types "int", "long" and "32" and do not have any variants with memory barriers at this time. okay noted. 🙈

atomic_load(p)

return (*p);

atomic_readandclear(p)

tmp = *p;
*p = 0;
return (tmp);
The atomic_readandclear() functions are not implemented for the types "char", "short", "ptr", "8", and "16" and do not have any variants with memory barriers at this time.

atomic_set(p, v)

*p |= v;

atomic_subtract(p, v)

*p -= v;

atomic_store(p, v)

*p = v;

atomic_swap(p, v)

tmp = *p;
*p = v;
return (tmp);
The atomic_swap() functions are not implemented for the types "char", "short", "ptr", "8", and "16" and do not have any variants with memory barriers at this time.

atomic_testandclear(p, v)

bit = 1 << (v % (sizeof(*p) * NBBY));
tmp = (*p & bit) != 0;
*p &= ~bit;
return (tmp);

atomic_testandset(p, v)

bit = 1 << (v % (sizeof(*p) * NBBY));
tmp = (*p & bit) != 0;
*p |= bit;
return (tmp);
The atomic_testandset() and atomic_testandclear() functions are only implemented for the types "int", "long", "ptr", "32", and "64" and generally do not have any variants with memory barriers at this time — except for atomic_testandset_acq_long(). 🎀
And one more thing: the type "64" is currently not implemented for some of the atomic operations on the arm, i386, and powerpc architectures. 😔

⭐ RETURN VALUES ⭐

okay so who returns what, let's break it down bestie:

  • The atomic_cmpset() function returns the result of the compare operation. 🫡
  • The atomic_fcmpset() function returns true if the operation succeeded. Otherwise it returns false and sets *old to the found value.
  • The atomic_fetchadd(), atomic_load(), atomic_readandclear(), and atomic_swap() functions return the value at the specified address. ✨
  • The atomic_testandset() and atomic_testandclear() function returns the result of the test operation.

🧚 EXAMPLES 🧚

This example uses the atomic_cmpset_acq_ptr() and atomic_set_ptr() functions to obtain a sleep mutex and handle recursion. Since the mtx_lock member of a struct mtx is a pointer, the "ptr" type is used. main character energy fr fr 👑

/* Try to obtain mtx_lock once. */
#define _obtain_lock(mp, tid)                                           \
        atomic_cmpset_acq_ptr(&(mp)->mtx_lock, MTX_UNOWNED, (tid))

/* Get a sleep lock, deal with recursion inline. */
#define _get_sleep_lock(mp, tid, opts, file, line) do {                 \
        uintptr_t _tid = (uintptr_t)(tid);                              \
                                                                        \
        if (!_obtain_lock(mp, tid)) {                                   \
                if (((mp)->mtx_lock & MTX_FLAGMASK) != _tid)            \
                        _mtx_lock_sleep((mp), _tid, (opts), (file), (line));\
                else {                                                  \
                        atomic_set_ptr(&(mp)->mtx_lock, MTX_RECURSE);   \
                        (mp)->mtx_recurse++;                            \
                }                                                       \
        }                                                               \
} while (0)

🌈 HISTORY 🌈

a lil timeline moment, we love a documented glow-up 💖

  • The atomic_add(), atomic_clear(), atomic_set(), and atomic_subtract() operations were introduced in FreeBSD 3.0. Initially, these operations were defined on the types "char", "short", "int", and "long".
  • The atomic_cmpset(), atomic_load_acq(), atomic_readandclear(), and atomic_store_rel() operations were added in FreeBSD 5.0. Simultaneously, the acquire and release variants were introduced, and support was added for operation on the types "8", "16", "32", "64", and "ptr".
  • The atomic_fetchadd() operation was added in FreeBSD 6.0.
  • The atomic_swap() and atomic_testandset() operations were added in FreeBSD 10.0.
  • The atomic_testandclear() and atomic_thread_fence() operations were added in FreeBSD 11.0.
  • The relaxed variants of atomic_load() and atomic_store() were added in FreeBSD 12.0.
  • The atomic_interrupt_fence() operation was added in FreeBSD 13.0.

okay that's the whole page bestie, you're officially an atomic girlie now 🎀 this one's from the Debian gang, dated December 16, 2024, and it's the ATOMIC(9) page — go forth and synchronize responsibly 💅✨🦋

📦 where this page comes from