FAIL(9)

💖 FAIL(9) — fail points 💖

okay bestie, sit down, hydrate, because today we're talking about fail points — the section 9 kernel developer magic from the FAIL(9) manual page, and honestly? it's giving chaos-in-a-good-way. 🦋

the whole cast of characters is: KFAIL_POINT_CODE, KFAIL_POINT_CODE_FLAGS, KFAIL_POINT_CODE_COND, KFAIL_POINT_RETURN, KFAIL_POINT_RETURN_VOID, KFAIL_POINT_ERROR, KFAIL_POINT_GOTO, KFAIL_POINT_SLEEP_CALLBACKS, fail_point, and DEBUG_FP. yes queen, that's a lot of names, we'll get to all of them, no cap. 💅

---

✨ SYNOPSIS ✨

first things first, you gotta include the header, obviously:

#include <sys/fail.h>

and then here's the whole girl group of macros, verbatim, don't @ me:

KFAIL_POINT_CODE(parent, name, code);

KFAIL_POINT_CODE_FLAGS(parent, name, flags, code);

KFAIL_POINT_CODE_COND(parent, name, cond, flags, code);

KFAIL_POINT_RETURN(parent, name);

KFAIL_POINT_RETURN_VOID(parent, name);

KFAIL_POINT_ERROR(parent, name, error_var);

KFAIL_POINT_GOTO(parent, name, error_var, label);

KFAIL_POINT_SLEEP_CALLBACKS(parent, name, pre_func, pre_arg, post_func,
    post_arg, code);

---

🌸 DESCRIPTION 🌸

so what even are fail points? they're little code points you sprinkle into your kernel where errors may be injected, in a user-controlled fashion. 🎀 they're a convenient lil wrapper around your user-provided error injection code, and they hand you two gifts: a sysctl(9) MIB, plus a parser for that MIB that describes how the error injection code should fire. iconic behaviour honestly.

the base fail point macro is KFAIL_POINT_CODE(), where:

• parent — a sysctl tree (frequently DEBUG_FP for kernel fail points, but various subsystems may wish to provide their own fail point trees, go off)
• name — the name of the MIB in that tree
• code — the error injection code

the code argument does not require braces, but bestie, it is considered good style to use braces for any multi-line code arguments. do it for the aesthetic. inside the code argument, the evaluation of RETURN_VALUE is derived from the return() value set in the sysctl MIB. 🔥

---

and then we have KFAIL_POINT_CODE_FLAGS(), which gives you a flags argument that controls the fail point's behaviour. you can use this to, e.g., mark the fail point's context as non-sleepable, which causes the sleep action to be coerced to a busy wait. the supported flags are:

• FAIL_POINT_USE_TIMEOUT_PATH — rather than sleeping on a sleep() call, just fire the post-sleep function after a timeout fires. lowkey efficient.
• FAIL_POINT_NONSLEEPABLE — mark the fail point as being in a non-sleepable context, which coerces sleep() calls to delay() calls. the way that it just ✨ redirects ✨.

---

likewise, KFAIL_POINT_CODE_COND() supplies a cond argument, which lets you set the condition under which the fail point's code may fire. it's equivalent to this whole situation:

if (cond)
        KFAIL_POINT_CODE_FLAGS(...);

see “SYSCTL VARIABLES” below, bestie. 👇

---

the remaining KFAIL_POINT_*() macros are wrappers around common error injection paths:

• KFAIL_POINT_RETURN(parent, name) is the equivalent of KFAIL_POINT_CODE(..., return RETURN_VALUE)
• KFAIL_POINT_RETURN_VOID(parent, name) is the equivalent of KFAIL_POINT_CODE(..., return)
• KFAIL_POINT_ERROR(parent, name, error_var) is the equivalent of KFAIL_POINT_CODE(..., error_var = RETURN_VALUE)
• KFAIL_POINT_GOTO(parent, name, error_var, label) is the equivalent of KFAIL_POINT_CODE(..., { error_var = RETURN_VALUE; goto label;})

---

🎀 SYSCTL VARIABLES 🎀

the KFAIL_POINT_*() macros add sysctl MIBs where specified. many base kernel MIBs can be found in the debug.fail_point tree (referenced in code by DEBUG_FP). 👑

the sysctl variable may be set in a number of ways:

[<pct>%][<cnt>*]<type>[(args...)][-><more terms>]

the <type> argument specifies which action to take; it can be one of:

• off — take no action (does not trigger fail point code). the "not me, I'm just vibing" option. 🙈
• return — trigger fail point code with specified argument
• sleep — sleep the specified number of milliseconds
• panic — panic 💀
• break — break into the debugger, or trap if there is no debugger support
• print — print that the fail point executed
• pause — threads sleep at the fail point until the fail point is set to off
• yield — thread yields the cpu when the fail point is evaluated
• delay — similar to sleep, but busy waits the cpu. (useful in non-sleepable contexts.)

---

the <pct>% and <cnt>* modifiers prior to <type> control when <type> is executed. fr fr:

• the <pct>% form (e.g. "1.2%") can be used to specify a probability that <type> will execute. this is a decimal in the range (0, 100] which can specify up to 1/10,000% precision. ✨
• the <cnt>* form (e.g. "5*") can be used to specify the number of times <type> should be executed before this <term> is disabled.
• only the last probability and the last count are used if multiple are specified, i.e. "1.2%2%" is the same as "2%".
• when both a probability and a count are specified, the probability is evaluated before the count, i.e. "2%5*" means "2% of the time, but only 5 times total". 🫶

---

the operator -> can be used to express cascading terms. if you specify <term1>-><term2>, it means that if <term1> does not 'execute', <term2> is evaluated. for the purpose of this operator, the return() and print() operators are the only types that cascade. a return() term only cascades if the code executes, and a print() term only cascades when passed a non-zero argument.

a pid can optionally be specified. the fail point term is only executed when invoked by a process with a matching p_pid. 🎯

---

🌈 EXAMPLES 🌈

here's where it gets fun, bestie. look at these:

sysctl debug.fail_point.foobar="2.1%return(5)"

21/1000ths of the time, execute code with RETURN_VALUE set to 5.

sysctl debug.fail_point.foobar="2%return(5)->5%return(22)"

2/100ths of the time, execute code with RETURN_VALUE set to 5. if that does not happen, 5% of the time execute code with RETURN_VALUE set to 22.

sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)"

for 5 times, return 5. after that, 1/1000th of the time, return 22.

sysctl debug.fail_point.foobar="0.1%5*return(5)"

return 5 for 1 in 1000 executions, but only 5 times total.

sysctl debug.fail_point.foobar="1%*sleep(50)"

1/100th of the time, sleep 50ms.

sysctl debug.fail_point.foobar="1*return(5)[pid 1234]"

return 5 once, when pid 1234 executes the fail point. 🧚

---

💜 AUTHORS 💜

this manual page was written by

Matthew Bryan <matthew.bryan@isilon.com> and

Zach Loafman <zml@FreeBSD.org>.

give them their flowers, they ate. 👑

---

⭐ CAVEATS ⭐

okay so now the serious part, bestie, listen up:

it is easy to shoot yourself in the foot by setting fail points too aggressively or setting too many in combination. for example, forcing malloc() to fail consistently is potentially harmful to uptime. 😭

the sleep() sysctl setting may not be appropriate in all situations. currently, fail_point_eval() does not verify whether the context is appropriate for calling msleep(). you can force it to evaluate a sleep action as a delay action by specifying the FAIL_POINT_NONSLEEPABLE flag at the point the fail point is declared.

---

and that's the whole page, diva. this came to you from Debian, dated March 15, 2016, and it's the FAIL(9) manual page — stay safe out there and don't set your fail points too aggressively. mwah. 💖💅✨