XCALL(9)

XCALL(9) — cross-call interface (Kernel Developer's Manual) 💅✨

Ok bestie, this is the XCALL(9) page — the one about making one CPU politely ask another CPU to run a function. Cross-call interface, but make it group-chat. 🫶

✨ NAME ✨

xcall, xc_broadcast, xc_unicast, xc_wait, xc_barrier — cross-call interface. That's the whole lineup, no cap. 💖

🦋 SYNOPSIS 🦋

#include <sys/xcall.h>

typedef void (*xcfunc_t)(void *, void *);

uint64_t
xc_broadcast(u_int flags, xcfunc_t func, void *arg1, void *arg2);

uint64_t
xc_unicast(u_int flags, xcfunc_t func, void *arg1, void *arg2,
    struct cpu_info *ci);

void
xc_wait(uint64_t where);

void
xc_barrier(u_int flags);

💖 DESCRIPTION 💖

The machine-independent xcall interface lets any CPU in the system request that an arbitrary function be executed on any other CPU. Literally remote-control for functions. 🎮

Sometimes you gotta modify hardware state that is tied directly to individual CPUs — think a CPU's local timer — and those updates can't be done remotely by another CPU. The LWP requesting the update may be unable to guarantee that it will be running on the CPU where the update must occur, when the update occurs. So it's giving “I need this done over there, but I can't be over there.” 😭

Also: sometimes it is necessary to modify per-CPU software state from a remote CPU. Where these update operations are so rare or the access to the per-CPU data so frequent that the cost of using locking or atomic operations to provide coherency is prohibitive, another way must be found. Enter cross calls, stage left. 🦋

Cross calls help to solve these types of problem. However, since this facility is heavyweight, it is expected that it will not be used often. It's an occasional main-character entrance, not an every-episode thing. 👑

xcall provides a mechanism for making “low priority” cross calls. The function to be executed runs on the remote CPU within a thread context, and not from a software interrupt, so it can ensure that it is not interrupting other code running on the CPU, and so has exclusive access to the CPU. Keep in mind that unless disabled, it may cause a kernel preemption. Lowkey important. 🫶

xcall also provides a mechanism for making “high priority” cross calls. The function to be executed runs on the remote CPU within a software interrupt context, possibly interrupting other lower-priority code running on the CPU. High-priority means it does not wait its turn. 💅

⭐ NOTES ⭐

Functions being called should be relatively lightweight. They may block on locks, but carefully and minimally, to not interfere with other cross calls in the system. No hogging the group chat. 🍓

🔥 FUNCTIONS 🔥

• xc_broadcast(flags, func, arg1, arg2) — Call (*func)(arg1, arg2) on all CPUs in the system. Return a uint64_t “ticket” to xc_wait() on for the cross-call to complete. flags should be XC_HIGHPRI or XC_HIGHPRI_IPL(ipl) for a "high priority" call, and 0 for a "low priority" call. XC_HIGHPRI uses an IPL_SOFTSERIAL software interrupt while XC_HIGHPRI_IPL uses a software interrupt with an IPL specified by ipl. xc_broadcast() should not be called from interrupt context.

🚨 Important vibe check: xc_broadcast() should not be called from interrupt context.

• xc_unicast(flags, func, arg1, arg2, ci) — Like xc_broadcast(), but call func on only the CPU indicated by ci. xc_unicast() also returns a “ticket”. It's the DM version of a broadcast. 💌
• xc_wait(where) — Wait on the “ticket” returned by a prior xc_broadcast() or xc_unicast() for the corresponding cross-call to complete. xc_wait() should be called from a thread context.
• xc_barrier(flags) — Issue a broadcast cross-call that does nothing, and wait for it to complete.

This functions like a memory barrier that forces all prior operations in program order to globally happen before all subsequent operations in program order, as witnessed by every CPU. ✨

This additionally waits for all higher-priority activity on the CPU to complete, according to flags:

• xc_barrier(0 ) waits for any pending kpreempt_disable(9) sections or activity at interrupt priority level above IPL_NONE to finish on all CPUs.
• xc_barrier(XC_HIGHPRI_IPL(ipl) ) waits for any pending activity at the software interrupt priority level ipl or higher to finish on all CPUs.

xc_barrier() is much more expensive than membar_ops(3), so it should be used sparingly, only to publish information infrequently — for example, during module load and unload — when the cost of a memory barrier on the consumer side would be prohibitive. Use it like the fancy perfume, not the daily body spray. 🎀

🔍 CODE REFERENCES 🔍

The xcall interface is implemented within the file sys/kern/subr_xcall.c. Go peek if you want the source tea. ☕

👀 SEE ALSO 👀

membar_ops(3), kpreempt(9), percpu(9), softint(9). The whole friend group. 🫶

📜 HISTORY 📜

The xcall interface first appeared in NetBSD 5.0. Vintage but still iconic. 💖

👑 AUTHORS 👑

Andrew Doran <ad@NetBSD.org> — thank you for the cross-call lore. 👑

---

💌 That's the whole vibe: this page is from Debian, dated February 1, 2020, and it's the XCALL(9) manual page. Go make those cross-CPU calls behave, bestie. 🫶✨