PMFAULT(3)

💖 PMFAULT(3) — the fault injection page, but make it cute 💖

okay bestie, welcome to the Library Functions Manual entry for PMFAULT ✨ this is the whole "Fault Injection Infrastructure for QA" situation and yes it is as iconic as it sounds 🫶

✨ NAME ✨

__pmFaultInject, __pmFaultSummary, PM_FAULT_POINT, PM_FAULT_RETURN, PM_FAULT_CHECK, PM_FAULT_CLEAR — Fault Injection Infrastructure for QA. that's the whole squad, no notes 💅

---

🌸 C SYNOPSIS 🌸

#include <pcp/pmapi.h>
#include <pcp/fault.h>

void __pmFaultInject(const char *ident, int class);
void __pmFaultSummary(FILE *f);

PM_FAULT_POINT(ident, class);
PM_FAULT_RETURN(retvalue);
PM_FAULT_CHECK;
PM_FAULT_CLEAR;

cc -DPM_FAULT_INJECTION=1 ... -lpcp_fault

like. that last line is the whole vibe honestly. -lpcp_fault or nothing 💀

---

🦋 DESCRIPTION 🦋

so back in the PCP 3.6 era, as part of the coverage-driven glow-up of QA, it became very obvious that we needed some way to exercise the ``uncommon'' code paths — you know, the error detection and recovery girlies. the ones nobody ever tests because nothing ever goes wrong. 🙈

the facilities below give you a basic fault injection infrastructure. libpcp only at this stage, but the mechanism is far more general and could easily be extended. it's giving "starter pack but the potential is unlimited" ✨

a special build is required to create libpcp_fault and the associated <pcp/fault.h> header file. once that's done:

• new QA applications may be built with -DPM_FAULT_INJECTION=1, and/or
• existing applications can be exercised in the presence of fault injection by forcing libpcp_fault to be used in preference to libpcp, as described below.

in the code you want to test, __pmFaultInject defines a fault point where a fault of type class may be injected. ident is a string that uniquely identifies the fault point across all of the PCP source code, so something like "libpcp/" __FILE__ ":<number>" works just fine 🎀 the ident string also decides whether a fault gets injected at run-time or not — go read the RUN-TIME CONTROL section below, bestie. class picks the failure type, using one of these defined values (and this list may well grow over time, so don't get too attached):

• PM_FAULT_ALLOC — makes the next call to malloc(3), realloc(3) or strdup(3) fail, returning NULL and setting errno to ENOMEM. we could extend coverage to all the malloc-related routines, but these three are enough to cover the vast majority of uses within libpcp. lowkey efficient 💅
• PM_FAULT_CALL — makes the next call to an instrumented routine fail by returning an error code (possibly the new PM_ERR_FAULT code). the actual error code is defined in the PM_FAULT_RETURN macro at the head of an instrumented routine. initially, only __pmRegisterAnon(3) (returns PM_ERR_FAULT), __pmGetPDU(3) (returns PM_ERR_TIMEOUT) and __pmAllocResult(3) (returns NULL) were instrumented as a proof of concept for this part of the facility — however other routines may have this fault injection capability added over time. so it's a growing collection, periodt 🌱
• PM_FAULT_MISC — the ``other'' class. currently used with PM_FAULT_CHECK as described below. it's giving mysterious, it's giving unspecified, we love her 💜

to let fault injection co-exist inside the production source code, PM_FAULT_POINT is a macro that emits no code by default, but when PM_FAULT_INJECTION is defined it becomes a call to __pmFaultInject. throughout libpcp we use PM_FAULT_POINT and not __pmFaultInject so that both libpcp and libpcp_fault can be built from the same source code. iconic behaviour honestly ✨

similarly, the macro PM_FAULT_RETURN emits no code unless PM_FAULT_INJECTION is defined — in which case, if a fault of type PM_FAULT_CALL has been armed with __pmFaultInject, then the enclosing routine returns with the function value retvalue. 🫶

the PM_FAULT_CHECK macro returns a value that may be 0 or 1. if PM_FAULT_INJECTION is defined, then if a fault of type PM_FAULT_MISC has been armed with __pmFaultInject the value is 1, else it is 0.

PM_FAULT_CHECK is most often used in concert with the PM_FAULT_POINT macro with the PM_FAULT_MISC class, to potentially arm a trigger, then test PM_FAULT_CHECK — and if this has the value 1, then the PM_FAULT_CLEAR macro is used to clear any armed faults, and the fault injection code is executed. it's a whole little choreography, ngl 💃

this is illustrated in the example below from src/libpcp/src/exec.c:

pid = fork();

/* begin fault-injection block */
PM_FAULT_POINT("libpcp/" __FILE__ ":4", PM_FAULT_MISC);
if (PM_FAULT_CHECK) {
 PM_FAULT_CLEAR;
 if (pid > (pid_t)0)
     kill(pid, SIGKILL);
 setoserror(EAGAIN);
 pid = -1;
}
/* end fault-injection block */

a summary of fault points seen and faults injected is produced on stdio stream f by __pmFaultSummary. receipts! 📝

additional tracing (via -Dfault or pmDebugOptions.fault) and a new PMAPI error code (PM_ERR_FAULT) are also defined, although these will only ever be seen or used in libpcp_fault. if pmDebugOptions.fault is set the first time __pmFaultInject is called, then __pmFaultSummary will be called automatically to report on stderr when the application exits (via atexit(3)). she reports herself, self-sufficient queen 👑

⚠️ fault injection cannot be nested. each call to __pmFaultInject clears any previous fault injection that has been armed but not yet executed. one at a time, bestie.

⚠️ the fault injection infrastructure is not thread-safe and should only be used with applications that are known to be single-threaded. do not bring her to a multithreaded party 💀

---

🌈 RUN-TIME CONTROL 🌈

by default, no fault injection is enabled at run-time, even when __pmFaultInject is called. like, you called her but she's not answering.

faults are selectively enabled using a control file, identified by the environment variable $PM_FAULT_CONTROL — if this is not set, no faults are enabled. simple as that ✨

the control file (if it exists) is read the first time __pmFaultInject is called, and contains lines of the form:

ident op number

that define fault injection guards. 🛡️

ident is a fault point string (as defined by a call to __pmFaultInject, or more usually the PM_FAULT_POINT macro). so yes, you need access to the libpcp source code to figure out the available ident strings and their semantics. iykyk 🙈

op is one of the C-style operators >=, >, ==, <, <=, != or %, and number is an unsigned integer. op number is optional and the default is >0.

the semantics of the fault injection guards are that each time __pmFaultInject is called for a particular ident, a trip count is incremented (the first trip is 1); if the C-style expression tripcount op number has the value 1 (so true for most ops, or the remainder equals 1 for the % op), then a fault of the class defined for the fault point associated with ident will be armed, and executed as soon as possible. math girlies rise 🧮✨

within the control file, blank lines are ignored and lines beginning with # are treated as comments.

for an existing application linked with libpcp, fault injection may still be used by forcing libpcp_fault to be used in the place of libpcp. the following example shows how this might be done. 💅

$ PM_FAULT_CONTROL=/tmp/control; export PM_FAULT_CONTROL
$ cat $PM_FAULT_CONTROL
# ok for 2 trips, then inject errors
libpcp/events.c:1  >2

$ LD_PRELOAD=/usr/lib/libpcp_fault.so; export LD_PRELOAD
$ pmevent -Dfault -s 3 sample.event.records
host:      localhost
samples:   3
interval:  1.00 sec
sample.event.records[fungus]: 0 event records
__pmFaultInject(libpcp/events.c:1) ntrip=1 SKIP
sample.event.records[bogus]: 2 event records
  10:46:12.413 --- event record [0] flags 0x1 (point) ---
    sample.event.param_string "fetch #0"
  10:46:12.413 --- event record [1] flags 0x1 (point) ---
    sample.event.param_string "bingo!"
__pmFaultInject(libpcp/events.c:1) ntrip=2 SKIP
sample.event.records[fungus]: 1 event records
  10:46:03.416 --- event record [0] flags 0x1 (point) ---
__pmFaultInject(libpcp/events.c:1) ntrip=3 INJECT
sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
__pmFaultInject(libpcp/events.c:1) ntrip=4 INJECT
sample.event.records[fungus]: pmUnpackEventRecords: Cannot allocate memory
__pmFaultInject(libpcp/events.c:1) ntrip=5 INJECT
sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory
=== Fault Injection Summary Report ===
libpcp/events.c:1: guard trip>2, 5 trips, 3 faults

see how it skipped twice and then started injecting? that's the >2 guard doing her thing. ate. 🔥

---

🎀 EXAMPLES 🎀

refer to the PCP and PCP QA source code. the lore is in the source, bestie:

• the macro definitions are in src/include/pcp/fault.h.
• src/libpcp/src/fault.c contains all of the underlying implementation.
• src/libpcp_fault and src/libpcp_fault/src contain the recipe and Makefiles for creating and installing libpcp_fault.so and <pcp/fault.h>.
• PM_FAULT_RETURN was initially used in the following libpcp source files: derive_parser.y.in, pdu.c and result.c.
• PM_FAULT_POINT was initially used in the following libpcp source files: derive_parser.y.in, desc.c, e_indom.c, e_labels.c, err.c, events.c, exec.c, fetch.c, help.c, instance.c, interp.c, labels.c, logmeta.c, pmns.c, p_profile.c and store.c.

the ``fault'' group of QA tests show examples of control file use. to see which tests are involved:

$ cd qa
$ check -n -g fault

go look, it's a whole mood 🌸

---

💀 DIAGNOSTICS 💀

some non-recoverable errors are reported on stderr. that's it, that's the diagnostic. she's not dramatic about it.

---

💜 ENVIRONMENT 💜

• PM_FAULT_CONTROL — full path to the fault injection control file.
• LD_PRELOAD — force libpcp_fault to be used in preference to libpcp.

---

👑 SEE ALSO 👑

PMAPI(3)

---

and that's the whole page, bestie! 🦋 this one came out of the Performance Co-Pilot collection — it's PMFAULT(3) and you're welcome 💖