TIMEOUT(9)

🌸 NAME 🌸

okay bestie, gather round, this is TIMEOUT(9) from the Kernel Developer's Manual, and it's giving "execute a function after a specified length of time" ✨

The whole squad of functions in this page:

callout_active, callout_deactivate, callout_async_drain, callout_drain, callout_handle_init, callout_init, callout_init_mtx, callout_init_rm, callout_init_rw, callout_pending, callout_reset, callout_reset_curcpu, callout_reset_on, callout_reset_sbt, callout_reset_sbt_curcpu, callout_reset_sbt_on, callout_schedule, callout_schedule_curcpu, callout_schedule_on, callout_schedule_sbt, callout_schedule_sbt_curcpu, callout_schedule_sbt_on, callout_stop, callout_when, timeout, untimeout 💅

---

🎀 SYNOPSIS 🎀

no cap, here's the whole cast list, verbatim, don't skip a single one:

#include <sys/types.h>
#include <sys/callout.h>
#include <sys/systm.h>

typedef void callout_func_t (void *);
typedef void timeout_t (void *);

int
callout_active(struct callout *c);

void
callout_deactivate(struct callout *c);

int
callout_async_drain(struct callout *c, callout_func_t *drain);

int
callout_drain(struct callout *c);

void
callout_handle_init(struct callout_handle *handle);

struct callout_handle handle = CALLOUT_HANDLE_INITIALIZER(&handle);

void
callout_init(struct callout *c, int mpsafe);

void
callout_init_mtx(struct callout *c, struct mtx *mtx, int flags);

void
callout_init_rm(struct callout *c, struct rmlock *rm, int flags);

void
callout_init_rw(struct callout *c, struct rwlock *rw, int flags);

int
callout_pending(struct callout *c);

int
callout_reset(struct callout *c, int ticks, callout_func_t *func,
    void *arg);

int
callout_reset_curcpu(struct callout *c, int ticks, callout_func_t *func,
    void *arg);

int
callout_reset_on(struct callout *c, int ticks, callout_func_t *func,
    void *arg, int cpu);

int
callout_reset_sbt(struct callout *c, sbintime_t sbt, sbintime_t pr,
    callout_func_t *func, void *arg, int flags);

int
callout_reset_sbt_curcpu(struct callout *c, sbintime_t sbt,
    sbintime_t pr, callout_func_t *func, void *arg, int flags);

int
callout_reset_sbt_on(struct callout *c, sbintime_t sbt, sbintime_t pr,
    callout_func_t *func, void *arg, int cpu, int flags);

int
callout_schedule(struct callout *c, int ticks);

int
callout_schedule_curcpu(struct callout *c, int ticks);

int
callout_schedule_on(struct callout *c, int ticks, int cpu);

int
callout_schedule_sbt(struct callout *c, sbintime_t sbt, sbintime_t pr,
    int flags);

int
callout_schedule_sbt_curcpu(struct callout *c, sbintime_t sbt,
    sbintime_t pr, int flags);

int
callout_schedule_sbt_on(struct callout *c, sbintime_t sbt,
    sbintime_t pr, int cpu, int flags);

int
callout_stop(struct callout *c);

sbintime_t
callout_when(sbintime_t sbt, sbintime_t precision, int flags,
    sbintime_t *sbt_res, sbintime_t *precision_res);

struct callout_handle
timeout(timeout_t *func, void *arg, int ticks);

void
untimeout(timeout_t *func, void *arg, struct callout_handle handle);

---

🦋 DESCRIPTION 🦋

okay so, the callout API is literally just "hey kernel, run this random function for me at some specific time in the future" ✨ Simple concept, extremely main character energy.

Consumers of this API are required to allocate a callout structure (struct callout) for each pending function invocation. No sharing, bestie, one per pending call. This structure is where all the state lives — the function to be called, and the time at which it should be invoked. You can cancel pending function calls, or reschedule them to a different time. And here's the cute part: once a scheduled call is done, you can reuse that same callout structure to schedule a brand new function call. ♻️

⏰ Single-shot only, no exceptions

Callouts only provide a single-shot mode, periodt. If you want a periodic timer, you have to explicitly reschedule each function call yourself. The way this is normally done is by rescheduling the subsequent call from inside the called function. iykyk 🫶

🚫 Callout functions must not sleep

not me having to say this but: callout functions must not sleep. Like, at all. They may not acquire sleepable locks, they may not wait on condition variables, they may not perform blocking allocation requests, and they may not invoke any other action that might sleep. 💀 Don't do it.

🔐 Initialization, because vibes alone won't cut it

Every single callout structure must be initialized by callout_init(), callout_init_mtx(), callout_init_rm(), or callout_init_rw() before it gets passed to any of the other callout functions. No shortcuts.

• callout_init() initializes a callout structure in c that is not associated with a specific lock. Now, if the mpsafe argument is zero, that callout structure is not considered to be “multi-processor safe” — and the Giant lock will be acquired before calling the callout function, and released when the callout function returns. 🌈
• callout_init_mtx(), callout_init_rm(), callout_init_rw() initialize a callout structure in c that is associated with a specific lock. That lock gets specified by the mtx, rm, or rw parameter. The associated lock must be held while stopping or rescheduling the callout. The callout subsystem acquires the associated lock before calling the callout function, and releases it after the function returns. And if the callout was cancelled while the callout subsystem was waiting for the associated lock? Then the callout function is not called, and the associated lock is released. This is what ensures that stopping or rescheduling the callout will abort any previously scheduled invocation. ✨

🫶 Important lock lore: only regular mutexes may be used with callout_init_mtx() — spin mutexes are not supported. A sleepable read-mostly lock (one initialized with the RM_SLEEPABLE flag) may not be used with callout_init_rm(). Similarly, other sleepable lock types such as sx(9) and lockmgr(9) cannot be used with callouts, because sleeping is not permitted in the callout subsystem. 💅

🎀 The init flags 🎀

These flags may be specified for callout_init_mtx(), callout_init_rm(), or callout_init_rw():

• CALLOUT_RETURNUNLOCKED — the callout function will release the associated lock itself, so the callout subsystem should not attempt to unlock it after the callout function returns. She's independent, she handles her own business. 👑
• CALLOUT_SHAREDLOCK — the lock is only acquired in read mode when running the callout handler. This flag is ignored by callout_init_mtx(). 🙈

🛑 Stopping things: callout_stop()

The function callout_stop() cancels a callout c if it is currently pending. Here's the return-value tea ☕:

• If the callout is pending and successfully stopped, callout_stop() returns a value of one.
• If the callout is not set, or has already been serviced, then negative one is returned.
• If the callout is currently being serviced and cannot be stopped, then zero will be returned.
• If the callout is currently being serviced and cannot be stopped, and at the same time a next invocation of the same callout is also scheduled, then callout_stop() unschedules the next run and returns zero.

And if the callout has an associated lock, then that lock must be held when this function is called. 🔥

🌊 callout_async_drain()

The function callout_async_drain() is identical to callout_stop() with one difference. When callout_async_drain() returns zero, it will arrange for the function drain to be called using the same argument given to the callout_reset() function. If the callout has an associated lock, then that lock must be held when this function is called.

😭 Note that when stopping multiple callouts that use the same lock, it is possible to get multiple return's of zero and multiple calls to the drain function, depending upon which CPU's the callouts are running.

The drain function itself is called from the context of the completing callout — i.e. softclock or hardclock — just like a callout itself.

🕳️ callout_drain()

The function callout_drain() is identical to callout_stop() except that it will wait for the callout c to complete if it is already in progress.

💀 This function MUST NOT be called while holding any locks on which the callout might block, or deadlock will result. I am so serious, bestie.

Note that if the callout subsystem has already begun processing this callout, then the callout function may be invoked before callout_drain() returns. However, the callout subsystem does guarantee that the callout will be fully stopped before callout_drain() returns. 🫶

🔁 Scheduling things: the callout_reset() and callout_schedule() families

The callout_reset() and callout_schedule() function families schedule a future function invocation for callout c. If c already has a pending callout, it is cancelled before the new invocation is scheduled.

These functions return a value of one if a pending callout was cancelled, and zero if there was no pending callout.

And yes, if the callout has an associated lock, then that lock must be held when any of these functions are called. 🔐

⏱️ How the time gets picked

The time at which the callout function will be invoked is determined by either the ticks argument, or the sbt, pr, and flags arguments.

When ticks is used, the callout is scheduled to execute after ticks/hz seconds. Non-positive values of ticks are silently converted to the value ‘1’. (not her quietly fixing your bad input 💅)

The sbt, pr, and flags arguments provide more control over the scheduled time, including support for higher resolution times, specifying the precision of the scheduled time, and setting an absolute deadline instead of a relative timeout.

The callout is scheduled to execute in a time window which begins at the time specified in sbt and extends for the amount of time specified in pr. If sbt specifies a time in the past, the window is adjusted to start at the current time. A non-zero value for pr allows the callout subsystem to coalesce callouts scheduled close to each other into fewer timer interrupts, which reduces processing overhead and power consumption. 🌱✨

🚩 The sbt/pr flags

These flags may be specified to adjust the interpretation of sbt and pr:

• C_ABSOLUTE — handle the sbt argument as an absolute time since boot. By default, sbt is treated as a relative amount of time, similar to ticks.
• C_DIRECT_EXEC — run the handler directly from hardware interrupt context instead of from the softclock thread. This reduces latency and overhead, but puts more constraints on the callout function. Callout functions run in this context may use only spin mutexes for locking, and should be as small as possible, because they run with absolute priority. ⚡
• C_PREL() — specifies relative event time precision as the binary logarithm of time interval divided by acceptable time deviation: 1 -- 1/2, 2 -- 1/4, etc. Note that the larger of pr or this value is used as the length of the time window. Smaller values (which result in larger time intervals) allow the callout subsystem to aggregate more events in one timer interrupt. 🧮
• C_PRECALC — okay so when you pass this, the sbt argument is literally the absolute time the callout should run at, and pr is the precision you're asking for, and bestie it will not be adjusted during the scheduling process, no cap. Both the sbt and pr values should come from an earlier call to callout_when(), which takes your user-supplied sbt, pr and flags and does the math.
• C_HARDCLOCK — align the timeouts to hardclock() calls if possible. Slay. ✨

The callout_reset() functions take a func argument — that's the function that gets called when the time expires, and it has to be a pointer to a function that takes a single void * argument. When it's invoked, func receives arg as its only argument. The callout_schedule() functions are the low-effort girlies here: they reuse the func and arg from the previous callout. But heads up 🫶 — you must call one of the callout_reset() functions to initialize func and arg before you can use one of the callout_schedule() functions. No init, no schedule, periodt.

The callout subsystem gives every CPU in the system its own softclock thread. Callouts get assigned to a single CPU and are executed by the softclock thread for that CPU. Out of the box, callouts are assigned to CPU 0. The callout_reset_on(), callout_reset_sbt_on(), callout_schedule_on() and callout_schedule_sbt_on() functions assign the callout to CPU cpu. The callout_reset_curcpu(), callout_reset_sbt_curpu(), callout_schedule_curcpu() and callout_schedule_sbt_curcpu() functions assign the callout to the current CPU. And the plain callout_reset(), callout_reset_sbt(), callout_schedule() and callout_schedule_sbt() functions schedule the callout to execute in the softclock thread of the CPU it's currently assigned to.

Softclock threads are not pinned to their respective CPUs by default. Want CPU 0's softclock thread glued to CPU 0? Set the kern.pin_default_swi loader tunable to a non-zero value. Want the softclock threads for CPUs other than zero pinned to their own CPUs? Set the kern.pin_pcpu_swi loader tunable to a non-zero value. There you go 💅

The macros callout_pending(), callout_active() and callout_deactivate() give you access to the current state of the callout. callout_pending() checks whether a callout is pending — a callout counts as pending when a timeout has been set but the time hasn't arrived yet. Not me warning you though: once the timeout time arrives and the callout subsystem starts processing that callout, callout_pending() will return FALSE even though the callout function may not have finished (or even begun) executing. callout_active() checks whether a callout is marked as active, and callout_deactivate() clears the callout's active flag. The callout subsystem marks a callout as active when a timeout is set, and it clears the active flag in callout_stop() and callout_drain() — but it does not clear it when a callout expires normally via the execution of the callout function. That little detail is the whole plot twist, bestie.

The callout_when() function is your girl for pre-calculating the absolute time at which the timeout should be run and the precision of the scheduled run time, according to the required time sbt, precision precision, and additional adjustments requested by the flags argument. The flags it accepts are the same as the flags for the callout_reset() function. The resulting time is assigned to the variable pointed to by the sbt_res argument, and the resulting precision is assigned to *precision_res. When you pass the results to callout_reset, add the C_PRECALC flag to flags, to avoid incorrect re-adjustment. This function is intended for situations where the precise time of the callout run should be known in advance, since trying to read this time from the callout structure itself after a callout_reset() call is racy. 🙈

---

🦋 Avoiding Race Conditions 🦋

The callout subsystem invokes callout functions from its own thread context. Without some kind of synchronization, it is possible that a callout function will be invoked concurrently with an attempt to stop or reset the callout by another thread. In particular — since callout functions typically acquire a lock as their first action — the callout function may have already been invoked, but be blocked waiting for that lock at the time that another thread tries to reset or stop the callout. It's giving chaos. 😭

There are three main techniques for addressing these synchronization concerns. The first approach is preferred because it's the simplest:

1. Callouts can be associated with a specific lock when they are initialized by callout_init_mtx(), callout_init_rm(), or callout_init_rw(). When a callout is associated with a lock, the callout subsystem acquires the lock before the callout function is invoked. This lets the callout subsystem transparently handle races between callout cancellation, scheduling, and execution. Note that the associated lock must be acquired before calling callout_stop() or one of the callout_reset() or callout_schedule() functions to provide this safety.

A callout initialized via callout_init() with mpsafe set to zero is implicitly associated with the Giant mutex. If Giant is held when cancelling or rescheduling the callout, then its use will prevent races with the callout function.

2. The return value from callout_stop() (or the callout_reset() and callout_schedule() function families) indicates whether or not the callout was removed. If it is known that the callout was set and the callout function has not yet executed, then a return value of FALSE indicates that the callout function is about to be called. For example:

     if (sc->sc_flags & SCFLG_CALLOUT_RUNNING) {
             if (callout_stop(&sc->sc_callout)) {
                     sc->sc_flags &= ~SCFLG_CALLOUT_RUNNING;
                     /* successfully stopped */
             } else {
                     /*
                      * callout has expired and callout
                      * function is about to be executed
                      */
             }
     }

3. The callout_pending(), callout_active() and callout_deactivate() macros can be used together to work around the race conditions. When a callout's timeout is set, the callout subsystem marks the callout as both active and pending. When the timeout time arrives, the callout subsystem begins processing the callout by first clearing the pending flag. It then invokes the callout function without changing the active flag, and does not clear the active flag even after the callout function returns. The mechanism described here requires the callout function itself to clear the active flag using the callout_deactivate() macro. The callout_stop() and callout_drain() functions always clear both the active and pending flags before returning.

The callout function should first check the pending flag and return without action if callout_pending() returns TRUE. This indicates that the callout was rescheduled using callout_reset() just before the callout function was invoked. If callout_active() returns FALSE then the callout function should also return without action — this indicates that the callout has been stopped. Finally, the callout function should call callout_deactivate() to clear the active flag. For example:

     mtx_lock(&sc->sc_mtx);
     if (callout_pending(&sc->sc_callout)) {
             /* callout was reset */
             mtx_unlock(&sc->sc_mtx);
             return;
     }
     if (!callout_active(&sc->sc_callout)) {
             /* callout was stopped */
             mtx_unlock(&sc->sc_mtx);
             return;
     }
     callout_deactivate(&sc->sc_callout);
     /* rest of callout function */

Together with appropriate synchronization, such as the mutex used above, this approach permits the callout_stop() and callout_reset() functions to be used at any time without races. For example:

     mtx_lock(&sc->sc_mtx);
     callout_stop(&sc->sc_callout);
     /* The callout is effectively stopped now. */

If the callout is still pending then these functions operate normally, but if processing of the callout has already begun then the tests in the callout function cause it to return without further action. Synchronization between the callout function and other code ensures that stopping or resetting the callout will never be attempted while the callout function is past the callout_deactivate() call.

The above technique additionally ensures that the active flag always reflects whether the callout is effectively enabled or disabled. If callout_active() returns false, then the callout is effectively disabled, since even if the callout subsystem is actually just about to invoke the callout function, the callout function will return without action.

There is one final race condition that must be considered when a callout is being stopped for the last time. In this case it may not be safe to let the callout function itself detect that the callout was stopped, since it may need to access data objects that have already been destroyed or recycled. To ensure that the callout is completely finished, a call to callout_drain() should be used. In particular, a callout should always be drained prior to destroying its associated lock or releasing the storage for the callout structure.

This is the part where you do not skip the drain. It's giving "I forgot callout_drain() and now I'm debugging at 3am" energy, and nobody wants that for you. 💀

✨ LEGACY API ✨ (the ancient section, handle with care bestie)

okay so real talk 💅 these functions down here are a legacy API that's getting removed in a future release. brand new code should NOT be using these routines, periodt. it's giving "museum exhibit", not "build your app on this". 🫶

• the function timeout() schedules a call to the function given by the argument func to take place after ticks/hz seconds. Non-positive values of ticks are silently converted to the value 1 🙈. func should be a pointer to a function that takes a void * argument. Upon invocation, func will receive arg as its only argument. The return value from timeout() is a struct callout_handle which can be used together with the untimeout() function to request that a scheduled timeout be canceled. it's giving "set a little alarm clock for later" ⏰
• the function callout_handle_init() can be used to initialize a handle to a state which will cause any calls to untimeout() with that handle to return with no side effects. so it's like the "nothing to see here" starter pack 🎀
• assigning a callout handle the value of CALLOUT_HANDLE_INITIALIZER() performs the same function as callout_handle_init() and is provided for use on statically declared or global callout handles. same energy, different spelling ✨
• the function untimeout() cancels the timeout associated with handle using the func and arg arguments to validate the handle. If the handle does not correspond to a timeout with the function func taking the argument arg, no action is taken. handle must be initialized by a previous call to timeout(), callout_handle_init(), or assigned the value of CALLOUT_HANDLE_INITIALIZER(&handle) before being passed to untimeout().

the behavior of calling untimeout() with an uninitialized handle is undefined. 💀 like, not "probably fine", not "yolo", just undefined. don't do it bestie, i'm begging 😭

• as handles are recycled by the system, it is possible (although unlikely) that a handle from one invocation of timeout() may match the handle of another invocation of timeout() if both calls used the same function pointer and argument, and the first timeout is expired or canceled before the second call. a little glitch in the matrix moment, no cap 🌈
• the timeout facility offers O(1) running time for timeout() and untimeout(). Timeouts are executed from softclock() with the Giant lock held. Thus they are protected from re-entrancy. locked and loaded fr fr 🔥

---

💖 RETURN VALUES 💖

okay so here's what everything hands back to you:

• the callout_active() macro returns the state of a callout's active flag. 👑
• the callout_pending() macro returns the state of a callout's pending flag. ⏳
• the callout_reset() and callout_schedule() function families return a value of one if the callout was pending before the new function invocation was scheduled. little receipt for your trouble 🧾
• the callout_stop() and callout_drain() functions return a value of one if the callout was still pending when it was called, a zero if the callout could not be stopped and a negative one if it was either not running or has already completed.
• the timeout() function returns a struct callout_handle that can be passed to untimeout(). 🎀

heads up: those one / zero / negative one returns are doing a LOT of emotional labor, so read them carefully before you assume anything ✨

---

📜 HISTORY 📜

story time bestie, because this lore is actually iconic 🫶

• The current timeout and untimeout routines are based on the work of Adam M. Costello and George Varghese, published in a technical report entitled Redesigning the BSD Callout and Timer Facilities and modified slightly for inclusion in FreeBSD by Justin T. Gibbs.
• The original work on the data structures used in this implementation was published by G. Varghese and A. Lauck in the paper Hashed and Hierarchical Timing Wheels: Data Structures for the Efficient Implementation of a Timer Facility in the Proceedings of the 11th ACM Annual Symposium on Operating Systems Principles.
• The current implementation replaces the long standing BSD linked list callout mechanism which offered O(n) insertion and removal running time but did not generate or require handles for untimeout operations. glow up of the century fr fr 💅

---

and that's the whole page bestie, you ate it UP 👑💖 shoutout to the Debian crew for this one, dated December 10, 2019, living in section TIMEOUT(9) of the manual. go forth and schedule responsibly 🦋✨