COUNTERS_ALLOC(9)

🎀 counters_alloc(9) — per CPU counters 🎀

okay so real talk bestie: this page is from the Kernel Developer's Manual (section 9, kernel-devs-only territory, iykyk) and it's literally the whole rundown on per CPU counters. if you've ever needed a pile of uint64_t counters that live on every CPU and get summed up later without tanking your performance, this is your page. let's get into it 💅

---

🌸 NAME 🌸

counters_alloc, counters_free, COUNTERS_BOOT_MEMORY, COUNTERS_BOOT_INITIALIZER, counters_alloc_ncpus, counters_enter, counters_leave, counters_inc, counters_add, counters_pkt, counters_read, counters_zero — per CPU counters ✨

---

🦋 SYNOPSIS 🦋

#include <sys/percpu.h>

struct cpumem *
counters_alloc(unsigned int ncounters);

void
counters_free(struct cpumem *cm, unsigned int ncounters);

COUNTERS_BOOT_MEMORY(NAME, unsigned int ncounters);

COUNTERS_BOOT_INITIALIZER(NAME);

struct cpumemt *
counters_alloc_ncpus(struct cpumem *cm, unsigned int ncounters);

uint64_t *
counters_enter(struct counters_ref *ref, struct cpumem *cm);

void
counters_leave(struct counters_ref *ref, struct cpumem *cm);

void
counters_inc(struct cpumem *cm, unsigned int counter);

void
counters_add(struct cpumem *cm, unsigned int counter, uint64_t v);

void
counters_pkt(struct cpumem *cm, unsigned int pcounter,
    unsigned int bcounter, uint64_t bytes);

void
counters_read(struct cpumem *cm, uint64_t *counters,
    unsigned int ncounters, uint64_t *scratch);

void
counters_zero(struct cpumem *cm, unsigned int ncounters);

heads up bestie: yes, that really does say struct cpumemt * in the counters_alloc_ncpus() declaration. that's exactly how it's written, so we're keeping it character for character. 🙂 no notes, moving on.

---

💖 DESCRIPTION 💖

the per CPU counter API builds on the per CPU memory API and gives you access to a whole series of uint64_t counter values on each CPU. it's giving "counters, but distributed." 🫶

the API does versioning of counter updates without using interlocked CPU instructions, so they can all be read in a consistent state. huge. and updates to counters should be limited to addition or subtraction of uint64_t values — that's the whole vibe, don't get creative with it.

there's also an alternate implementation of the API on uni-processor systems (i.e. when the kernel is not built with MULTIPROCESSOR defined) that provides no overhead compared to direct access to a data structure. so you can use the API without affecting the performance of uni-processor systems — literally free, we love that for you ✨

• counters_alloc() — allocates memory for a series of uint64_t values on each CPU. ncounters specifies the number of counters to be allocated. the counters will be zeroed on allocation. clean slate, iconic.
• counters_free() — deallocates each CPU's counters. the same ncounters argument originally provided to counters_alloc() must be passed to counters_free(). no mixing up your numbers, bestie, they have to match 💅
• COUNTERS_BOOT_MEMORY() — statically allocates a set of counter for use on the boot CPU before the other CPUs in the system have been attached. the allocation is identified by NAME and provides memory for the number of counters specified by ncounters. the counters will be initialised to zero. early-boot main character energy 👑
• COUNTERS_BOOT_INITIALIZER() — used to initialise a cpumem pointer with the memory that was previously allocated using COUNTERS_BOOT_MEMORY() and identified by NAME. it's the matching set, the duo, the friendship bracelet 🤝
• counters_alloc_ncpus() — allocates additional sets of counters for the CPUs that were attached during boot. the cpumem structure cm must have been initialised with COUNTERS_BOOT_INITIALIZER(). the same number of counters originally passed to COUNTERS_BOOT_MEMORY must be specified by ncounters. the counters on the boot CPU will be preserved, while the counters for the additional CPUs will be zeroed on allocation. glow-up for the new CPUs, no reset for the OG 🔥
• counters_enter() — provides access to the current CPU's set of counters referenced by cm. the caller's reference to the counters is held by ref. yes, you need ref, it's your little ticket stub 🎟️
• counters_leave() — indicates the end of access to the current CPU's set of counters referenced by cm. the reference held by ref is released by this call. bye bestie, come back soon 🌸
• counters_inc() — increments the counter at the index specified by counter in the array of counters referenced by cm. +1, we stan.
• counters_add() — adds the value of v to the counter at the index specified by counter in the array of counters referenced by cm. bulk vibes only.
• counters_pkt() — increments the value at the index specified by pcounter and adds the value of bytes to the counter at the index specified by bcounter in the array of counters referenced by cm. two-for-one, packet style 🎀
• counters_read() — iterates over each CPU's set of counters referenced by cm, takes a consistent snapshot of the counters, and then sums them together. the sum of the counters is written to the counters array. the number of counters is specified with ncounters. scratch may point to a buffer used to temporarily hold counters. if NULL, one will be dynamically allocated and freed. it's the "collect them all and add them up" moment 🧚
• counters_zero() — iterates over each CPU's set of counters referenced by cm and zeroes them. the number of counters is specified with ncounters. but listen: counters_zero() itself does not prevent concurrent updates of the counters; it is up to the caller to serialise this call with other actions. not me warning you twice about this, but here we are 😭

✨ the extra important bits from above, in one place: counters_alloc() may only be used after all the CPUs in the system have been attached. if a set of CPU counters needs to be available during early boot, a cpumem pointer and counters for the boot CPU may be statically allocated. and — say it with me — counters that have been allocated with COUNTERS_BOOT_MEMORY() and counters_alloc_ncpus() cannot be deallocated with counters_free. do not try it. nobody wins. 💀

---

🌈 CONTEXT 🌈

• counters_alloc(), counters_free(), counters_alloc_ncpus(), and counters_read() may be called during autoconf, or from process context.
• counters_enter(), counters_leave(), counters_inc(), counters_add(), counters_pkt(), and counters_zero() may be called during autoconf, from process context, or from interrupt context.

⚠️ okay this is the serious bit, bestie: the per CPU counters API does not provide any locking or serialisation of access to each CPU's set of counters beyond isolating each CPU's update. it is up to the caller to provide appropriate locking or serialisation around calls to these functions to prevent concurrent access to the relevant data structures. no cap, this one matters. 🔒

---

⭐ RETURN VALUES ⭐

• counters_alloc() and counters_alloc_ncpus() will return an opaque cpumem pointer that references each CPU's set of counters.
• counters_enter() returns a reference to the current CPU's set of counters.

---

💜 EXAMPLES 💜

the following is an example of providing per CPU counters at boot time based on the mbuf(9) statistics code in sys/kern/uipc_mbuf.c. presenting: the whole thing, verbatim, because it ate and left no crumbs 🍓

/* mbuf stats */
COUNTERS_BOOT_MEMORY(mbstat_boot, MBSTAT_COUNT);
struct cpumem *mbstat = COUNTERS_BOOT_INITIALIZER(mbstat_boot);

/*
 * this function is called from init_main.c after devices
 * (including additional CPUs) have been attached
 */
void
mbcpuinit()
{
        mbstat = counters_alloc_ncpus(mbstat, MBSTAT_COUNT);
}

struct mbuf *
m_get(int nowait, int type)
{
        ...

        struct counters_ref cr;
        uint64_t *counters;
        int s;

        ...

        s = splnet();
        counters = counters_enter(&cr, mbstat);
        counters[type]++;
        counters_leave(&cr, mbstat);
        splx(s);

        ...
}

struct mbuf *
m_free(struct mbuf *m)
{
        ...

        struct counters_ref cr;
        uint64_t *counters;
        int s;

        ...

        s = splnet();
        counters = counters_enter(&cr, mbstat);
        counters[m->m_type]--;
        counters_leave(&cr, mbstat);
        splx(s);

        ...
}

see how it's ++ in one and -- in the other? addition or subtraction of uint64_t values, exactly like we said. consistency queen 💅

---

🧚 SEE ALSO 🧚

cpumem_get(9), malloc(9)

---

🔥 HISTORY 🔥

the per CPU counter API first appeared in OpenBSD 6.1. vintage but make it fresh ✨

---

👑 AUTHORS 👑

the per CPU counter API was written by David Gwynne <dlg@openbsd.org>. absolute legend behaviour, we bow 🫡

---

and that's the page!! this Debian build is dated September 16, 2023, and it's filed under COUNTERS_ALLOC(9) — go forth and count your per-CPU little hearts out, bestie 💖🎀🦋