STAPPROBES(3stap)

📦
version 5.6 · 5 versions
, in arch rolling, centos-stream 10, centos-stream 9, fedora 43, fedora 44

💅 stapprobes — systemtap probe points 💅

okay so you found the 3stap manual section and this one is stapprobes, and bestie, this is literally THE map of every probe point systemtap knows about. no cap, it's a big one. grab a snack. 🍓✨

🎀 NAME 🎀

stapprobes — systemtap probe points. that's the whole name, short and iconic. 👑

🦋 DESCRIPTION 🦋

The following sections enumerate the variety of probe points supported by the systemtap translator, plus some extra aliases that get defined by standard tapset scripts. A whole bunch of them have their own dedicated manual page over in the 3stap section, with the probe:: prefix. iykyk. 💖

🌸 SYNTAX 🌸

probe PROBEPOINT [, PROBEPOINT] { [STMT ...] }

A probe declaration may list multiple comma-separated probe points, which is the move when you want one handler attached to all of the named events. Normally the handler statements run whenever any of those events occur. Depending on the type of probe point, the handler statements may refer to context variables — those are the ones with a dollar-sign prefix like $foo — to read or write state. That may include function parameters for function probes, or local variables for statement probes. 🫶

The syntax of a single probe point is a general dotted-symbol sequence. This lets the event namespace get broken down into parts, kinda like the Domain Name System does on the Internet. Each component identifier may be parametrized by a string or number literal, with a syntax like a function call. A component may include a "*" character, to expand to a set of matching probe points. It may also include "**" to match multiple sequential components at once. Probe aliases likewise expand to other probe points. 🌈

Probe aliases can be given on their own, or with a suffix. The suffix attaches to the underlying probe point that the alias is expanded to. For example,

syscall.read.return.maxactive(10)

expands to

kernel.function("sys_read").return.maxactive(10)

with the component maxactive(10) being recognized as a suffix. ✨

Normally, each and every probe point resulting from wildcard- and alias-expansion must be resolved to some low-level system instrumentation facility (e.g., a kprobe address, marker, or a timer configuration), otherwise the elaboration phase will fail. that's a hard fail, bestie. 💀

However — plot twist — a probe point may be followed by a "?" character, to indicate that it is optional, and that no error should result if it fails to resolve. Optionalness passes down through all levels of alias/wildcard expansion. Alternately, a probe point may be followed by a "!" character, to indicate that it is both optional and sufficient. (Think vaguely of the Prolog cut operator.) If it does resolve, then no further probe points in the same comma-separated list will be resolved. Therefore, the "!" sufficiency mark only makes sense in a list of probe point alternatives. 🎯

Additionally, a probe point may be followed by a "if (expr)" statement, in order to enable/disable the probe point on-the-fly. With the "if" statement, if the "expr" is false when the probe point is hit, the whole probe body including alias's body is skipped. The condition is stacked up through all levels of alias/wildcard expansion. So the final condition becomes the logical-and of conditions of all expanded alias/wildcard. The expressions are necessarily restricted to global variables. 🙈

These are all syntactically valid probe points. (They are generally semantically invalid, depending on the contents of the tapsets, and the versions of kernel/user software installed. so like, syntax slay, semantics maybe not. 💀)

kernel.function("foo").return
process("/bin/vi").statement(0x2222)
end
syscall.*
syscall.*.return.maxactive(10)
syscall.{open,close}
sys**open
kernel.function("no_such_function") ?
module("awol").function("no_such_function") !
signal.*? if (switch)
kprobe.function("foo")

Probes may be broadly classified into "synchronous" and "asynchronous". A "synchronous" event is deemed to occur when any processor executes an instruction matched by the specification. This gives these probes a reference point (instruction address) from which more contextual data may be available. Other families of probe points refer to "asynchronous" events such as timers/counters rolling over, where there is no fixed reference point that is related. Each probe point specification may match multiple locations (for example, using wildcards or aliases), and all them are then probed. A probe declaration may also contain several comma-separated specifications, all of which are probed. 🔥

Brace expansion is a mechanism which allows a list of probe points to be generated. It is very similar to shell expansion. A component may be surrounded by a pair of curly braces to indicate that the comma-separated sequence of one or more subcomponents will each constitute a new probe point. The braces may be arbitrarily nested. The ordering of expanded results is based on product order. 🧚

The question mark (?), exclamation mark (!) indicators and probe point conditions may not be placed in any expansions that are before the last component. (so put them at the end, bestie, or it's not valid.)

The following is an example of brace expansion. 👇

syscall.{write,read}
# Expands to
syscall.write, syscall.read

{kernel,module("nfs")}.function("nfs*")!
# Expands to
kernel.function("nfs*")!, module("nfs").function("nfs*")!

🌈 DWARF DEBUGINFO 🌈

Resolving some probe points requires DWARF debuginfo or "debug symbols" for the specific program being instrumented. For some others, DWARF is automatically synthesized on the fly from source code header files. For others, it is not needed at all. Since a systemtap script may use any mixture of probe points together, the union of their DWARF requirements has to be met on the computer where script compilation occurs. (See the --use-server option and the stap-server(8) man page for information about the remote compilation facility, which allows these requirements to be met on a different machine.) 🖥️✨

The following point lists many of the available probe point families, to classify them with respect to their need for DWARF debuginfo for the specific program for that probe point.

DWARF 🐛

  • kernel.function, .statement
  • module.function, .statement, .tracepoint
  • process.function, .statement
  • process.mark*
  • .function.callee
  • python3
  • debuginfod

AUTO-GENERATED-DWARF ✨

  • kernel.trace
  • kernel.tracepoint
  • tp_syscall
  • kernel.lsm
  • kernel.xdp

NON-DWARF 🙅

  • kernel.mark
  • process.mark, process.plt
  • begin, end, error, never
  • timer
  • perf
  • procfs
  • kernel.statement.absolute
  • kernel.data
  • kprobe.function
  • process.statement.absolute
  • process.begin, .end
  • netfilter
  • java

SYMBOL-TABLE 📖

  • kernel.function*
  • module.function*
  • process.function*

The probe types marked with * asterisks mark fallbacks, where systemtap can sometimes infer subset or substitute information. In general, the more symbolic / debugging information available, the higher quality probing will be available. (more info = better vibes, it's giving quality. 👑)

🎀 ON-THE-FLY ARMING 🎀

The following types of probe points may be armed/disarmed on-the-fly to save overheads during uninteresting times. Arming conditions may also be added to other types of probes, but will be treated as a wrapping conditional and won't benefit from overhead savings. (so you get the behaviour but not the speed-up, ngl.) ⚡

DISARMABLE 🎯

  • kernel.function, kernel.statement
  • module.function, module.statement
  • process.*.function, process.*.statement
  • process.*.plt, process.*.mark
  • timer.
  • java

exceptions 🚫

  • timer.profile

👑 PROBE POINT FAMILIES 👑

🌟 BEGIN/END/ERROR 🌟

The probe points begin and end are defined by the translator to refer to the time of session startup and shutdown. All "begin" probe handlers are run, in some sequence, during the startup of the session. All global variables will have been initialized prior to this point. All "end" probes are run, in some sequence, during the normal shutdown of a session, such as in the aftermath of an exit () function call, or an interruption from the user. In the case of an error-triggered shutdown, "end" probes are not run. There are no target variables available in either context. 😭

If the order of execution among "begin" or "end" probes is significant, then an optional sequence number may be provided:

begin(N)
end(N)

The number N may be positive or negative. The probe handlers are run in increasing order, and the order between handlers with the same sequence number is unspecified. When "begin" or "end" are given without a sequence, they are effectively sequence zero. 🔢

The error probe point is similar to the end probe, except that each such probe handler run when the session ends after errors have occurred. In such cases, "end" probes are skipped, but each "error" probe is still attempted. This kind of probe can be used to clean up or emit a "final gasp". It may also be numerically parametrized to set a sequence. 💅

🌙 NEVER 🌙

The probe point never is specially defined by the translator to mean "never". Its probe handler is never run, though its statements are analyzed for symbol / type correctness as usual. This probe point may be useful in conjunction with optional probes. (it's literally the "do nothing but still get type-checked" icon. 😌)

🐚 SYSCALL and ND_SYSCALL 🐚

The syscall.* and nd_syscall.* aliases define several hundred probes, too many to detail here. They are of the general form:

syscall.NAME
nd_syscall.NAME
syscall.NAME.return
nd_syscall.NAME.return

Generally, a pair of probes are defined for each normal system call as listed in the syscalls(2) manual page, one for entry and one for return. Those system calls that never return do not have a corresponding .return probe. The nd_* family of probes are about the same, except it uses non-DWARF based searching mechanisms, which may result in a lower quality of symbolic context data (parameters), and may miss some system calls. You may want to try them first, in case kernel debugging information is not immediately available. 🔍

Each probe alias provides a variety of variables. Looking at the tapset source code is the most reliable way. Generally, each variable listed in the standard manual page is made available as a script-level variable, so syscall.open exposes filename, flags, and mode. In addition, a standard suite of variables is available at most aliases: 🎁

  • argstr — A pretty-printed form of the entire argument list, without parentheses.
  • name — The name of the system call.
  • retval — for return probes, this is the raw numeric system-call result. just the number bestie, no glow-up 💅
  • retstr — for return probes, a pretty-printed string version of the system-call result. the number but it got dressed up ✨

okay so important lore about probe aliases in general: these variables all get initialized once from the underlying $context variables. so if you go change $context stuff later, your variables are NOT gonna automatically update. they're locked in, periodt. 🫶

and also — not every single probe alias follows all these general guidelines. like, some of them are doing their own thing 🙈 so if you run into one that's being bothersome, please report it as a bug. we wanna know!!

heads up bestie: on some kernel/userspace architecture combos (thinking of like 32-bit userspace on a 64-bit kernel 👀), the underlying $context variables might need explicit sign extension / masking. when that's the vibe, consider using the tapset-provided variables instead of raw $context variables. they already did the work for you 💖

and if debuginfo availability is being a whole problem 😭 you can try the non-DWARF syscall probe aliases instead. just use the nd_syscall. prefix instead of syscall. — same context variables are there, as far as possible. iconic behavior honestly 🌈

nd_syscall probes on kernels that use syscall wrappers to pass arguments via pt_regs (currently 4.17+ on x86_64 and 4.19+ on aarch64) do support syscall argument writing when guru mode is enabled. the way that works: if you modify a probe syscall parameter in the probe body, then immediately before the probe exits, the parameter's current value gets written to pt_regs. this overwrites the previous value. no take-backs 💀

nd_syscall probes also include two parameters for each of the syscall's string parameters:

  1. one holds a quoted version of the string passed to the syscall
  2. the other holds an unquoted version of the string — this is the one you're meant to use when modifying the parameter

so if the probe modifies that unquoted string variable, then as the probe is about to exit, the contents of this variable get written to the user space buffer that was passed to the syscall. 🦋

⚠️ it is the user's responsibility to make sure that buffer is big enough to hold the modified string AND that it lives in a writable memory segment. do not skip this. fr fr.

⏰ TIMERS ⏰

okay so there are two main types of timer probes:

  • "jiffies" timer probes
  • time interval timer probes

jiffies timers

intervals defined by the standard kernel "jiffies" timer can trigger probe handlers asynchronously ✨ the translator supports two probe point variants:

timer.jiffies(N)
timer.jiffies(N).randomize(M)

the probe handler runs every N jiffies (a kernel-defined unit of time, typically somewhere between 1 and 60 ms).

if the "randomize" component is given, a linearly distributed random value in the range [-M..+M] gets added to N every single time the handler runs. 🎲

  • N is restricted to a reasonable range (1 to around a million)
  • M is restricted to be smaller than N

there are no target variables provided in either context. and yes, it's possible for these probes to run concurrently on a multi-processor computer. chaos but make it scheduled 💅

time interval timers

alternatively you can specify intervals in units of time ⏳ there are two probe point variants, similar to the jiffies timer:

timer.ms(N)
timer.ms(N).randomize(M)

here, N and M are specified in milliseconds, but the full options for units are:

  • seconds (s/sec)
  • milliseconds (ms/msec)
  • microseconds (us/usec)
  • nanoseconds (ns/nsec)
  • hertz (hz)
🚫 randomization is not supported for hertz timers. nope. not happening.

the resolution tea ☕

the actual resolution of the timers depends on the target kernel:

  • for kernels prior to 2.6.17, timers are limited to jiffies resolution, so intervals get rounded up to the nearest jiffies interval.
  • after 2.6.17, the implementation uses hrtimers for tighter precision, though the actual resolution will be arch-dependent.

either way, if the "randomize" component is given, then the random value gets added to the interval before any rounding occurs. order matters bestie 🫶

profiling timers 📊

profiling timers are also available, giving you probes that execute on all CPUs at the rate of the system tick (CONFIG_HZ) or at a given frequency (hz).

💀 on some kernels, this is a one-concurrent-user-only or disabled facility, which results in error -16 (EBUSY) during probe registration. annoying but that's the tea.
timer.profile.tick
timer.profile.freq.hz(N)

full context information of the interrupted process is available, which makes this probe suitable for a time-based sampling profiler. that's the whole point of it. main character energy honestly 👑

it is recommended to use the tapset probe timer.profile rather than timer.profile.tick. this probe point behaves identically to timer.profile.tick when the underlying functionality is available, and falls back to using perf.sw.cpu_clock on some recent kernels which lack the corresponding profile timer facility. flexible ✨

profiling timers with specified frequencies are only accurate up to around 100 hz. you may need to provide a larger value to achieve the desired rate. lowkey annoying but that's just how it is 🌸

📢 note: if a timer probe is set to fire at a very high rate and the probe body is complex, succeeding timer probes can get skipped — because the time for them to run has already passed. normally systemtap reports missed probes, but it will not report these skipped probes. so like... they just ghost you 🙈

🧚 DWARF 🧚

this family of probe points uses symbolic debugging information for the target kernel/module/program — the kind you'd find in unstripped executables, or in the separate debuginfo packages. they let you place probes logically into the execution path of the target program, by specifying a set of points in the source or object code. when a matching statement executes on any processor, the probe handler runs in that context. slay. 💅

DWARF family probe points can be identified by:

  • the target kernel module (or user process)
  • source file
  • line number
  • function name
  • or some combination of these ✨

here's the list of DWARF probe points currently supported 👇

kernel.function(PATTERN)
kernel.function(PATTERN).call
kernel.function(PATTERN).callee(PATTERN)
kernel.function(PATTERN).callee(PATTERN).return
kernel.function(PATTERN).callee(PATTERN).call
kernel.function(PATTERN).callees(DEPTH)
kernel.function(PATTERN).return
kernel.function(PATTERN).inline
kernel.function(PATTERN).label(LPATTERN)
module(MPATTERN).function(PATTERN)
module(MPATTERN).function(PATTERN).call
module(MPATTERN).function(PATTERN).callee(PATTERN)
module(MPATTERN).function(PATTERN).callee(PATTERN).return
module(MPATTERN).function(PATTERN).callee(PATTERN).call
module(MPATTERN).function(PATTERN).callees(DEPTH)
module(MPATTERN).function(PATTERN).return
module(MPATTERN).function(PATTERN).inline
module(MPATTERN).function(PATTERN).label(LPATTERN)
kernel.statement(PATTERN)
kernel.statement(PATTERN).nearest
kernel.statement(ADDRESS).absolute
module(MPATTERN).statement(PATTERN)
process("PATH").function("NAME")
process("PATH").statement("*@FILE.c:123")
process("PATH").library("PATH").function("NAME")
process("PATH").library("PATH").statement("*@FILE.c:123")
process("PATH").library("PATH").statement("*@FILE.c:123").nearest
process("PATH").function("*").return
process("PATH").function("myfun").label("foo")
process("PATH").function("foo").callee("bar")
process("PATH").function("foo").callee("bar").return
process("PATH").function("foo").callee("bar").call
process("PATH").function("foo").callees(DEPTH)
process(PID).function("NAME")
process(PID).function("myfun").label("foo")
process(PID).plt("NAME")
process(PID).plt("NAME").return
process(PID).statement("*@FILE.c:123")
process(PID).statement("*@FILE.c:123").nearest
process(PID).statement(ADDRESS).absolute
debuginfod.process("PATH").**

(peep the USER-SPACE section below for more info on the process probes, bestie 🫶)

that list above has a bunch of variants and modifiers that give you extra functionality or act as filters. here they all are:

  • .function — places a probe near the beginning of the named function, so that parameters are available as context variables. ✨
  • .return — places a probe at the moment after the return from the named function, so the return value is available as the "$return" context variable.
  • .inline — filters the results to include only instances of inlined functions. note that inlined functions don't have an identifiable return point, so .return is not supported on .inline probes. 🚫
  • .call — filters the results to include only non-inlined functions (aka the opposite set of .inline).
  • .exported — filters the results to include only exported functions.
  • .statement — places a probe at the exact spot, exposing those local variables that are visible there. 👀
  • .statement.nearest — places a probe at the nearest available line number for each line number given in the statement.
  • .callee — places a probe on the callee function given in the .callee modifier, where the callee must be a function called by the target function given in .function. the advantage of doing this over directly probing the callee function: this probe point only runs when the callee is called from the target function. (add the -DSTAP_CALLEE_MATCHALL directive to override this when calling stap(1).)
📝 note: only callees that can be statically determined are available. for example, calls through function pointers are not available. additionally, calls to functions located in other objects (e.g. libraries) are not available — instead use another probe point. and this feature will only work for code compiled with GCC 4.7+.
  • .callees — shortcut for .callee("*"), which places a probe on all callees of the function. 💖
  • .callees(DEPTH) — recursively places probes on callees. for example, .callees(2) will probe both the callees of the target function and the callees of those callees. and .callees(3) goes one level deeper, etc... 🌈 a callee probe at depth N is only triggered when the N callers in the callstack match those that were statically determined during analysis (this also may be overridden using -DSTAP_CALLEE_MATCHALL).

MPATTERN is a string literal that's out here trying to identify the loaded kernel module you're obsessed with. For in-tree kernel modules, the name alone is enough (like "btrfs"). You can also throw "*", "[]", and "?" wildcards in there to match a whole bunch of in-tree modules at once. Out-of-tree modules? Also supported, but you gotta hand over the full path to the .ko file — and wildcards are not supported there, sorry bestie. The file also has to follow the convention of being named <module_name>.ko, where the characters ',' and '-' get swapped out for '_'.

LPATTERN is a source program label. It can hold "*", "[]", and "?" wildcards too.

PATTERN is a string literal aiming to identify a point in the program, and it's built from three parts:

  • The first part is the name of a function, exactly like it'd show up in the nm program's output. This part can use the "*" and "?" wildcarding operators to match multiple names. Also, heads up: kernel builds may expose linker stubs named with a __pfx_ prefix (like __pfx_sys_read sitting right next to sys_read). Those stubs get left out of wildcard matches unless your PATTERN explicitly asks for them — either an exact __pfx_… name, or a wildcard that begins with __pfx or otherwise contains __pfx_, such as __pfx_* or *__pfx_*.
  • The second part is optional and begins with the "@" character. After it comes the path to the source file containing the function, which may include a wildcard pattern like mm/slab*. If it doesn't match as-is, an implicit "*/" is optionally added before the pattern, so your script only needs to name the last few components of a possibly super long source directory path.
  • Finally, the third part is optional if the file name part was given, and it identifies the line number in the source file, preceded by a ":" or a "+". The line number counts as an absolute line number if it's got a ":" in front, or relative to the declaration line of the function if it's got a "+". All the lines in the function match with ":*". A range of lines x through y matches with ":x-y". And you can mix ranges and specific lines with commas, e.g. ":x,y-z".

As an alternative, PATTERN may be a numeric constant, which means it's an address. You can find such an address from the symbol tables of the appropriate kernel / module object file. It gets verified against known statement code boundaries, and it'll be relocated for use at run time.

Guru mode only: absolute kernel-space addresses may be specified with the ".absolute" suffix. That address is considered already relocated, as if it came straight from /proc/kallsyms, so it cannot be checked against statement/instruction boundaries. 😭

🫶 CONTEXT VARIABLES 🫶

A whole bunch of the source-level context variables — function parameters, locals, globals visible in the compilation unit — may be visible to probe handlers. You refer to them by prefixing their name with "$" inside your scripts. On top of that, a special syntax allows limited traversal of structures, pointers, and arrays, and more syntax allows pretty-printing of individual variables or their groups. See also @cast. Real talk though: variables may be inaccessible because they got paged out, or for a few other reasons. See also man error::fault(7stap).

Functions called from DWARF class probe points and from process.mark probes may also refer to context variables.

  • $var — refers to an in-scope variable or thread local storage variable "var". If it's an integer-like type, it gets cast to a 64-bit int for systemtap script use. String-like pointers (char *) may be copied to systemtap string values using the kernel_string or user_string functions.
  • @var("varname") — an alternative syntax for $varname.
  • @var("varname","module") — the global variable or global thread local storage variable in scope of the given module already loaded into the current probed process. Useful for getting an exported variable in a shared library loaded into the process being probed, or a global variable in a process while a shared library probe is being executed. For user-space modules only. For example: @var("_r_debug","/lib/ld-linux.so.2")
  • @var("varname@src/file.c") — refers to the global (either file local or external) variable varname defined when the file src/file.c was compiled. The CU in which the variable is resolved is the first CU in the module of the probe point which matches the given file name at the end and has the shortest file name path (e.g. given @var("foo@bar/baz.c") and CUs with file name paths src/sub/module/bar/baz.c and src/bar/baz.c, the second CU will be chosen to resolve the (file) global variable foo)
  • @var("varname@src/file.c","module") — the global variable in scope of the given CU, defined in the given module, even if the variable is static (so the name isn't unique without the CU name).
  • $var->field — traversal via a structure's or a pointer's field. This generalized indirection operator may be repeated to follow more levels. Note that the . operator is not used for plain structure members — only -> for both purposes. (That's because "." is reserved for string concatenation.) Also note that for direct dereferencing of a $var pointer, {kernel,user}_{char,int,...}($var) should be used. (Refer to stapfuncs(5) for more details.)
  • $return — available in return probes only, for functions that are declared with a return value, which you can figure out using @defined($return).
  • $var[N] — indexes into an array. The index is given with a literal number or even an arbitrary numeric expression.

There's a whole set of operators for these basic context variable expressions:

  • $$vars — expands to a character string that is equivalent to
sprintf("parm1=%x ... parmN=%x var1=%x ... varN=%x",
        parm1, ..., parmN, var1, ..., varN)

for each variable in scope at the probe point. Some values may be printed as =? if their run-time location cannot be found.

  • $$locals — expands to a subset of $$vars for only local variables.
  • $$parms — expands to a subset of $$vars for only function parameters.
  • $$return — available in return probes only. It expands to a string equivalent to sprintf("return=%x", $return) if the probed function has a return value, or else an empty string.
  • & $EXPR — expands to the address of the given context variable expression, if it's addressable.
  • @defined($EXPR) — expands to 1 or 0 iff the given context variable expression is resolvable, for use in conditionals such as
@defined($foo->bar) ? $foo->bar : 0

Use full $context variables or @cast() expressions in @define parameters for unambiguous processing. The TYPECAST script-level variable facility is incompletely supported. Simplify the parameters by reducing nesting or indirection if you run into "Collapsing unresolved @define to ..." warnings.

  • @probewrite($VAR) — see the PROBES section of stap(1).
  • $EXPR$ — expands to a string with all of $EXPR's members, equivalent to
sprintf("{.a=%i, .b=%u, .c={...}, .d=[...]}",
         $EXPR->a, $EXPR->b)
  • $EXPR$$ — expands to a string with all of $var's members and submembers, equivalent to
sprintf("{.a=%i, .b=%u, .c={.x=%p, .y=%c}, .d=[%i, ...]}",
        $EXPR->a, $EXPR->b, $EXPR->c->x, $EXPR->c->y, $EXPR->d[0])
  • @errno — expands to the last value the C library global variable errno was set to.

✨ MORE ON RETURN PROBES ✨

For the kernel ".return" probes, only a certain fixed number of returns may be outstanding at once. The default is a relatively small number — on the order of a few times the number of physical CPUs. So if a bunch of different threads concurrently call the same blocking function, like futex(2) or read(2), that limit could get exceeded, and skipped "kretprobes" would be reported by "stap -t". To work around it, specify a

probe FOO.return.maxactive(NNN)

suffix, with a big enough NNN to cover all the expected concurrently blocked threads. Or, alternately, use the

stap -DKRETACTIVE=NNNN

stap command line macro setting to override the default for all ".return" probes. 💅

For ".return" probes, context variables other than the "$return" may be accessible, as a little convenience for a script programmer who wants to access function parameters. These values are snapshots taken at the time of function entry. (Local variables within the function are not generally accessible, since those variables didn't exist in allocated/initialized form at the snapshot moment.) These entry-snapshot variables should be accessed via @entry($var).

Also, arbitrary entry-time expressions can be saved for ".return" probes using the @entry(expr) operator. For example, you can compute the elapsed time of a function:

probe kernel.function("do_filp_open").return {
    println( get_timeofday_us() - @entry(get_timeofday_us()) )
}

Here's the rundown of how values related to a function parameter context variable — a pointer named addr — may be accessed from a .return probe:

  • at-entry $addr → past-exit value: not available
  • at-entry $addr->x->y → past-exit value: @cast(@entry($addr),"struct zz")->x->y
  • at-entry $addr[0] → past-exit value: {kernel,user}_{char,int,...}(& $addr[0])

💀 DWARFLESS 💀

When there's no debugging information around, entry & exit points of kernel & module functions can still be probed using the "kprobe" family of probes. The catch? These do not let you look up the arguments / local variables of the function. The following constructs are supported:

kprobe.function(FUNCTION)
kprobe.function(FUNCTION).call
kprobe.function(FUNCTION).return
kprobe.module(NAME).function(FUNCTION)
kprobe.module(NAME).function(FUNCTION).call
kprobe.module(NAME).function(FUNCTION).return
kprobe.statement(ADDRESS).absolute

Probes of type function are the recommended ones for kernel functions, whereas probes of type module are recommended for probing functions of the specified module. And in case you know the absolute address of a kernel or module function, statement probes can be utilized.

Okay so real talk: FUNCTION and MODULE names must not contain wildcards, or the probe will not be registered. Also, statement probes must be run under guru-mode only. 🧚

🫶 USER-SPACE 🫶

okay bestie, user-space probing is a whole thing and it's only available if your kernel is configured with the utrace extensions, or if it's got the uprobes facility in linux 3.5. also, various kernel build configuration options need to be enabled — and don't stress, systemtap will literally tell you if any of those are missing. it's giving helpful. 💅

so there are several forms, and we're going through all of them, no skipping.

First up, the non-symbolic probe point:

process(PID).statement(ADDRESS).absolute

this one is analogous to kernel.statement(ADDRESS).absolute — because both of them use raw (unverified) virtual addresses and give you no $variables. the target PID parameter has to identify a running process, and ADDRESS should identify a valid instruction address. all threads of that process will be probed. periodt. ✨

Second, the non-symbolic user-kernel interface events handled by utrace can be probed:

process(PID).begin
process("FULLPATH").begin
process.begin
process(PID).thread.begin
process("FULLPATH").thread.begin
process.thread.begin
process(PID).end
process("FULLPATH").end
process.end
process(PID).thread.end
process("FULLPATH").thread.end
process.thread.end
process(PID).syscall
process("FULLPATH").syscall
process.syscall
process(PID).syscall.return
process("FULLPATH").syscall.return
process.syscall.return

let me break these down for you, because there's a lot going on:

  • a process.begin probe gets called when a new process described by PID or FULLPATH gets created. in addition, it's called once from the context of each preexisting process, at systemtap script startup. this is useful to track live processes 👀
  • a process.thread.begin probe gets called when a new thread described by PID or FULLPATH gets created.
  • a process.end probe gets called when the process described by PID or FULLPATH dies. rip 🪦
  • a process.thread.end probe gets called when a thread described by PID or FULLPATH dies.
  • a process.syscall probe gets called when a thread described by PID or FULLPATH makes a system call. the system call number is available in the $syscall context variable, and the first 6 arguments of the system call are available in the $argN (ex. $arg1, $arg2, ...) context variable.
  • a process.syscall.return probe gets called when a thread described by PID or FULLPATH returns from a system call. the system call number is available in the $syscall context variable, and the return value of the system call is available in the $return context variable. A
ngl bestie, if a process probe is specified without a PID or FULLPATH, all user threads will be probed. but if systemtap was invoked with the -c or -x options, then process probes are restricted to the process hierarchy associated with the target process.

and okay, this next bit is lowkey important: if a process probe is unspecified (i.e. without a PID or FULLPATH), but with the -c option, the PATH of the -c cmd will be heuristically filled into the process PATH. in that case, only command parameters are allowed in the -c command (i.e. no command substitution allowed and no occurrences of any of these characters: '|&;<>(){}'). so watch out for that fr fr 🙈

Third, symbolic static instrumentation compiled into programs and shared libraries may be probed:

process("PATH").mark("LABEL")
process("PATH").provider("PROVIDER").mark("LABEL")
process(PID).mark("LABEL")
process(PID).provider("PROVIDER").mark("LABEL")

a .mark probe gets called via a static probe which is defined in the application by STAP_PROBE1(PROVIDER,LABEL,arg1) — which are macros defined in sys/sdt.h. the PROVIDER is an arbitrary application identifier, LABEL is the marker site identifier, and arg1 is the integer-typed argument. STAP_PROBE1 is used for probes with 1 argument, STAP_PROBE2 is used for probes with 2 arguments, and so on forever, iykyk 💫

the arguments of the probe are available in the context variables $arg1, $arg2, ... and an alternative to using the STAP_PROBE macros is to use the dtrace script to create custom macros. additionally, the variables $$name and $$provider are available as parts of the probe point name. and the sys/sdt.h macro names DTRACE_PROBE* are available as aliases for STAP_PROBE*. iconic. 🎀

Finally, full symbolic source-level probes in user-space programs and shared libraries are supported. these are exactly analogous to the symbolic DWARF-based kernel/module probes described above, and they expose the same sorts of context $variables for function parameters, local variables, and so on. main character energy honestly 👑

process("PATH").function("NAME")
process("PATH").statement("*@FILE.c:123")
process("PATH").plt("NAME")
process("PATH").library("PATH").plt("NAME")
process("PATH").library("PATH").function("NAME")
process("PATH").library("PATH").statement("*@FILE.c:123")
process("PATH").function("*").return
process("PATH").function("myfun").label("foo")
process("PATH").function("foo").callee("bar")
process("PATH").plt("NAME").return
debuginfod.process("PATH").**
process(PID).function("NAME")
process(PID).statement("*@FILE.c:123")
process(PID).plt("NAME")
🚨 note that for all process probes, PATH names refer to executables that are searched the same way shells do: relative to the working directory if they contain a "/" character, otherwise in $PATH. and if PATH names refer to scripts, the actual interpreters (specified in the script in the first line after the #! characters) are probed. in the debuginfod probe family PATH names likewise refer to executables, but are searched for in the currently defined $DEBUGINFOD_URLS.

tapset process probes placed in the special directory $prefix/share/systemtap/tapset/PATH/ with relative paths will have their process parameter prefixed with the location of the tapset. for example,

process("foo").function("NAME")

expands to

process("/usr/bin/foo").function("NAME")

when placed in $prefix/share/systemtap/tapset/PATH/usr/bin/. it's giving ✨auto-magic✨

if PATH is a process component parameter referring to shared libraries then all processes that map it at runtime would be selected for probing. if PATH is a library component parameter referring to shared libraries then the process specified by the process component would be selected. note that the PATH pattern in a library component will always apply to libraries statically determined to be in use by the process. however, you may also specify the full path to any library file even if not statically needed by the process. slay 💅

a .plt probe will probe functions in the program linkage table corresponding to the rest of the probe point. .plt can be specified as a shorthand for .plt("*"). the symbol name is available as a $$name context variable; function arguments are not available, since PLTs are processed without debuginfo. a .plt.return probe places a probe at the moment after the return from the named function. 🦋

if the PATH string contains wildcards as in the MPATTERN case, then standard globbing is performed to find all matching paths. in this case, the $PATH environment variable is not used. (sorry bestie, them's the rules 🌸)

and just to say it again: if systemtap was invoked with the -c or -x options, then process probes are restricted to the process hierarchy associated with the target process.

🌈 DEBUGINFOD 🌈

these probes take the form

debuginfod.process("PATH").**

they are very similar to the process("PATH").** probe family. the key difference is that the process probes search for PATH in the host filesystem, while debuginfod probes search the current federation of debuginfod servers, using the currently defined $DEBUGINFOD_URLS (see debuginfod(8)). 💖

in order to probe the contents of one or more elf/archive files and/or elf/archive containing directories, the below will create a debuginfod server which will scan and process the elf files within and prepare them for systemtap:

$ debuginfod [options] [-F -R -Z etc.] /path1 /path2
$ env DEBUGINFOD_URLS=http://localhost:8002/ stap ...

👑 JAVA 👑

support for probing Java methods is available using Byteman as a backend. Byteman is an instrumentation tool from the JBoss project which systemtap can use to monitor invocations for a specific method or line in a Java program. 🍓

systemtap does so by generating a Byteman script listing the probes to instrument and then invoking the Byteman bminstall utility.

⚠️ okay so listen: this Java instrumentation support is currently a prototype feature with major limitations. moreover, Java probing currently does not work across users — the stap script must run (with appropriate permissions) under the same user as the Java process being probed. (thus a stap script under root currently cannot probe Java methods in a non-root-user Java process.) 💀

the first probe type refers to Java processes by the name of the Java process:

java("PNAME").class("CLASSNAME").method("PATTERN")
java("PNAME").class("CLASSNAME").method("PATTERN").return

the PNAME argument must be a pre-existing jvm pid, and be identifiable via a jps listing. 🔥

the PATTERN parameter specifies the signature of the Java method to probe. the signature must consist of the exact name of the method, followed by a bracketed list of the types of the arguments, for instance "myMethod(int,double,Foo)". wildcards are not supported. 😭

the probe can be set to trigger at a specific line within the method by appending a line number with colon, just as in other types of probes: "myMethod(int,double,Foo):245". 📍

the CLASSNAME parameter identifies the Java class the method belongs to, either with or without the package qualification. by default, the probe only triggers on descendants of the class that do not override the method definition of the original class. however, CLASSNAME can take an optional caret prefix, as in ^org.my.MyClass, which specifies that the probe should also trigger on all descendants of MyClass that override the original method. for instance, every method with signature foo(int) in program org.my.MyApp can be probed at once using

java("org.my.MyApp").class("^java.lang.Object").method("foo(int)")

ate. 💅

the second probe type works analogously, but refers to Java processes by PID:

java(PID).class("CLASSNAME").method("PATTERN")
java(PID).class("CLASSNAME").method("PATTERN").return

(PIDs for an already running process can be obtained using the jps(1) utility.) 🫶

context variables defined within java probes include $arg1 through $arg10 (for up to the first 10 arguments of a method), represented as character-pointers for the toString() form of each actual argument. the arg1 through arg10 script variables provide access to these as ordinary strings, fetched via user_string_warn(). ⭐

okay so quick lil flashback moment before we move on bestie — back before systemtap 3.1, $arg1 through $arg10 could hold either integers or character pointers, depending on what types of objects were getting passed into each particular java method. It was giving chaos, honestly. If you want that old-school behaviour back, you can summon it with the stap --compatible=3.0 flag. ✨

PROCFS 🫶

These probe points are the moment, fr — they let you create procfs "files" in /proc/systemtap/MODNAME that you can read and write, using a permission that you can tweak with the proper umask value. 💅

Default permissions? Bestie, memorize this:

  • 0400 for read probes
  • 0200 for write probes
  • if you've got both a read and a write probe on the same file, the default permission is 0600

So if you went procfs.umask(0040).read, you'd end up with a 0404 permission set on the file. Cute, right? (MODNAME is the name of the systemtap module, iykyk.)

The proc filesystem is a pseudo-filesystem used as an interface to kernel data structures. The translator supports several probe point variants:

procfs("PATH").read
procfs("PATH").umask(UMASK).read
procfs("PATH").read.maxsize(MAXSIZE)
procfs("PATH").umask(UMASK).maxsize(MAXSIZE)
procfs("PATH").write
procfs("PATH").umask(UMASK).write
procfs.read
procfs.umask(UMASK).read
procfs.read.maxsize(MAXSIZE)
procfs.umask(UMASK).read.maxsize(MAXSIZE)
procfs.write
procfs.umask(UMASK).write
heads up bestie 💀 there are a few differences when procfs probes are used in the stapbpf runtime. FIFO special files get used instead of proc filesystem files. These files are created in /var/tmp/systemtap-USER/MODNAME. (USER is the name of the user.) And on top of that, users cannot create both read and write probes on the same file. No cap.

PATH is the file name (relative to /proc/systemtap/MODNAME or /var/tmp/systemtap-USER/MODNAME) that's gonna be created. If you don't specify a PATH (like in the last two variants above), PATH defaults to "command". 🌸

Also: the file name "__stdin" is used internally by systemtap for input probes, so do NOT use it as a PATH for procfs probes — check the input probe section below for the tea. 🙈

When a user reads /proc/systemtap/MODNAME/PATH (normal runtime) or /var/tmp/systemtap-USER/MODNAME (stapbpf runtime), the corresponding procfs read probe gets triggered. The string data to be read should be assigned to a variable named $value, like this:

procfs("PATH").read { $value = "100\n" }

When a user writes into /proc/systemtap/MODNAME/PATH (normal runtime) or /var/tmp/systemtap-USER/MODNAME (stapbpf runtime), the corresponding procfs write probe gets triggered. The data the user wrote is available in the string variable named $value, like this:

procfs("PATH").write { printf("user wrote: %s", $value) }

MAXSIZE is the size of the procfs read buffer. Specifying MAXSIZE lets you get larger procfs output — obsessed. If no MAXSIZE is specified, the procfs read buffer defaults to STP_PROCFS_BUFSIZE (which defaults to MAXSTRINGLEN, the maximum length of a string). And if you need to set the procfs read buffers for more than one file, it may be easiest to just override the STP_PROCFS_BUFSIZE definition. Here's an example of using MAXSIZE:

procfs.read.maxsize(1024) {
    $value = "long string..."
    $value .= "another long string..."
    $value .= "another long string..."
    $value .= "another long string..."
}

INPUT 🎀

These probe points make input from stdin available to the script during runtime. The translator currently supports two variants of this family:

input.char
input.line

input.char is triggered each time a character is read from stdin. The current character is available in the string variable named char. There is no newline buffering — the next character is read from stdin as soon as it becomes available. ⚡

input.line buffers all characters read from stdin until a newline is read, at which point the probe triggers. The current line of characters (including the newline) is made available in a string variable named line. Note that no more than MAXSTRINGLEN characters will be buffered, and any additional characters will not be included in line. 😭

Input probes are aliases for procfs("__stdin").write. Systemtap reconfigures stdin if the presence of this procfs probe is detected, which is exactly why "__stdin" should not be used as a path argument for procfs probes. And additionally? Input probes will not work with the -F and --remote options. periodt. 💅

NETFILTER HOOKS 🌈

These probe points let you observe network packets using the netfilter mechanism. A netfilter probe in systemtap corresponds to a netfilter hook function in the original netfilter probes API. Honestly it's probably more convenient to use tapset::netfilter(3stap), which wraps the primitive netfilter hooks and does all the work of extracting useful info from the context variables for you. 🫶

There are several probe point variants supported by the translator:

netfilter.hook("HOOKNAME").pf("PROTOCOL_F")
netfilter.pf("PROTOCOL_F").hook("HOOKNAME")
netfilter.hook("HOOKNAME").pf("PROTOCOL_F").priority("PRIORITY")
netfilter.pf("PROTOCOL_F").hook("HOOKNAME").priority("PRIORITY")

PROTOCOL_F is the protocol family to listen for — currently one of NFPROTO_IPV4, NFPROTO_IPV6, NFPROTO_ARP, or NFPROTO_BRIDGE. 👑

HOOKNAME is the point, or 'hook', in the protocol stack at which to intercept the packet. The available hook names for each protocol family are taken from the kernel header files <linux/netfilter_ipv4.h>, <linux/netfilter_ipv6.h>, <linux/netfilter_arp.h> and <linux/netfilter_bridge.h>. For instance, allowable hook names for NFPROTO_IPV4 are NF_INET_PRE_ROUTING, NF_INET_LOCAL_IN, NF_INET_FORWARD, NF_INET_LOCAL_OUT, and NF_INET_POST_ROUTING. ✨

PRIORITY is an integer priority giving the order in which the probe point should be triggered relative to any other netfilter hook functions that trigger on the same packet. Hook functions execute on each packet in order from smallest priority number to largest priority number. If no PRIORITY is specified (like in the first two probe point variants above), PRIORITY defaults to "0".

There are a number of predefined priority names of the form NF_IP_PRI_* and NF_IP6_PRI_* which are defined in the kernel header files <linux/netfilter_ipv4.h> and <linux/netfilter_ipv6.h> respectively. The script is permitted to use these instead of specifying an integer priority. (The probe points for NFPROTO_ARP and NFPROTO_BRIDGE currently do not expose any named hook priorities to the script writer.) So the allowable ways to specify the priority include:

priority("255")
priority("NF_IP_PRI_SELINUX_LAST")
guru mode warning bestie 😭 a script using guru mode is permitted to specify any identifier or number as the parameter for hook, pf, and priority. This feature should be used with caution, because the parameter is inserted verbatim into the C code generated by systemtap. Like, no filter, no vibes check. Be careful.

The netfilter probe points define the following context variables:

  • $hooknum — The hook number.
  • $skb — The address of the sk_buff struct representing the packet. See <linux/skbuff.h> for details on how to use this struct, or alternatively use the tapset tapset::netfilter(3stap) for easy access to key information.
  • $in — The address of the net_device struct representing the network device on which the packet was received (if any). May be 0 if the device is unknown or undefined at that stage in the protocol stack.
  • $out — The address of the net_device struct representing the network device on which the packet will be sent (if any). May be 0 if the device is unknown or undefined at that stage in the protocol stack.
  • $verdict — (Guru mode only.) Assigning one of the verdict values defined in <linux/netfilter.h> to this variable alters the further progress of the packet through the protocol stack. For instance, the following guru mode script forces all ipv6 network packets to be dropped:
probe netfilter.pf("NFPROTO_IPV6").hook("NF_IP6_PRE_ROUTING") {
  $verdict = 0 /* nf_drop */
}

For convenience, unlike the primitive probe points discussed here, the probes defined in tapset::netfilter(3stap) export the lowercase names of the verdict constants (e.g. NF_DROP becomes nf_drop) as local variables. 🦋

KERNEL TRACEPOINTS ⭐

This family of probe points hooks up to static probing tracepoints inserted into the kernel or modules. As with markers, these tracepoints are special macro calls inserted by kernel developers to make probing faster and more reliable than with DWARF-based probes — and DWARF debugging information is not required to probe tracepoints. 🎉

Tracepoints also have an extra advantage: more strongly-typed parameters than markers. slay.

Tracepoint probes look like: kernel.trace("name"). The tracepoint name string, which may contain the usual wildcard characters, is matched against the names defined by the kernel developers in the tracepoint header files. To restrict the search to specific subsystems (e.g. sched, ext3, etc...), you can use this syntax: kernel.trace("system:name"). The tracepoint system string may also contain the usual wildcard characters. ✨

The handler associated with a tracepoint-based probe may read the optional parameters specified at the macro call site. These are named according to the declaration by the tracepoint author. For example, the tracepoint probe kernel.trace("sched:sched_switch") provides the parameters $prev and $next. If the parameter is a complex type, as in a struct pointer, then a script can access fields with the same syntax as DWARF $target variables. Also — tracepoint parameters cannot be modified, but in guru-mode a script may modify fields of parameters. 👑

The subsystem and name of the tracepoint are available in $$system and $$name, and a string of name=value pairs for all parameters of the tracepoint is available in $$vars or $$parms. 💖

BTF TRACEPOINTS 💜

This family of probe points also hooks up to kernel and module tracepoints, but it discovers them from btf_trace_* callback typedefs rather than from trace event header files. The kernel catalog comes from vmlinux.h in the kernel build tree (requires CONFIG_DEBUG_INFO_BTF). Module tracepoints are discovered from the module's debuginfo when it contains matching btf_trace_* typedefs (typical when CONFIG_DEBUG_INFO_BTF_MODULES is enabled). 🌸

Okay so kernel probes are like kernel.tracepoint("name") 💖 and module probes are module("modname").tracepoint("name"). The name string can hold the usual wildcard characters, so go wild bestie. But here's the thing — unlike kernel.trace(), these probes do not use the system:name syntax. You specify only the tracepoint registration name, like sched_switch, not sched:sched_switch. Don't do the extra thing, it's giving extra 💅

Handler arguments are positional and numbered from one: $arg1, $arg2, and so on, derived from the callback typedef in vmlinux.h or module debuginfo. The leading void * cookie parameter is omitted — we don't talk about the cookie. And named parameters from TP_PROTO? Not available on this probe type, sorry 😭 use kernel.trace("system:name") if you need those names, that's the move.

The tracepoint name lives in $$name. And $$system is always empty for BTF tracepoint probes (yes, always, it's just vibes in there). A string of name=value pairs for usable arguments is available in $$vars or $$parms — pick your fighter.

Example uses:

# Kernel tracepoint by registration name
probe kernel.tracepoint("sched_switch") {
  printf("preempt=%d cpu=%d\n", $arg1, $arg4)
}

# Module tracepoint (module debuginfo required)
probe module("nfs").tracepoint("nfs_getattr_enter") {
  printf("inode=%d\n", $arg1)
}

# List available kernel tracepoints
$ stap -L 'kernel.tracepoint("sched_switch")'
$ stap -l 'kernel.tracepoint("*")'

🌀 SYSCALL TRACEPOINT DISPATCH (tp_syscall) 🌀

tp_syscall("name") and tp_syscall("name").return attach to the raw sys_enter / sys_exit tracepoints (the BTF kernel.tracepoint() catalog, which requires vmlinux.h) but dispatch in generated C with a switch on __NR_name. Unmarked names are looked up in the native syscall table, no cap.

Okay so unfortunately — and this is the messy part 😭 — the full treatment of duplication, aliases, overlap, and backward-compatibility in kernel syscall numbering sometimes requires addition of special markup to make the decoder routine's work straightforward (and require no hard-coded heuristics in the translator). Iconic of the kernel tbh.

A ~ prefix selects the compat-task table instead — tp_syscall("~compat_read"), tp_syscall("~ia32_truncate64"), or tp_syscall("~fadvise64") when that table reuses the native number. There is no automatic compat_ or ia32_ rewrite of the identifier. Aliases that share an NR token share that case; a probe that lists several numbers has its handler call replicated on each label. Native and compat ABIs use separate switches (_stp_is_compat_task()) because they reuse the same small integers — the audacity fr fr.

Several names may be listed in one string — tp_syscall("chown,chown32,~compat_chown32") — to match extra NR tokens; names listed together on one probe share a case if their NRs coincide. Use = to declare a merge-only alias that may have the same number as another probe's token, without matching that other name: tp_syscall("umount2=umount,~compat_umount2=compat_umount"). The left-hand name is preferred as the case label. 💅

Several tp_syscall() probes in one script share a single tracepoint registration instead of N isolated next filters concatenated by probe combining — efficiency ate. And tp_syscall("name").return supports @entry() the same way as kretprobes: a synthetic tp_syscall("name") sibling records the operand at sys_enter. Obsessed.

Tapset aliases such as tp_syscall.read expand to tp_syscall("read,~compat_read") with a kernel.trace("sys_enter") fallback. Handler arguments are the same positional $arg1 ... as kernel.tracepoint("sys_enter") (typically pt_regs then syscall id). The usual tapset name / argstr locals are filled after dispatch.

Example:

probe tp_syscall("read"), tp_syscall("write") {
  printf("syscall nr dispatch hit\n")
}

Use the tp_syscall.read tapset alias when you need name / argstr. 🫶

🔐 LSM (LINUX SECURITY MODULE) HOOKS 🔐

This family of probe points, available with the BPF runtime (--runtime=bpf), allows attaching to Linux Security Module hooks in the kernel. LSM hooks are called during security-sensitive operations and can be used to monitor or control access to files, sockets, processes, and other kernel resources. Requires Linux kernel 5.7 or later with CONFIG_BPF_LSM=y and kernel headers for type information.

LSM hook probes look like kernel.lsm("hook_name"), or you can use the tapset aliases: kernel.lsm.hook_name. The hook name says which security hook to attach to — like bprm_check_security (program execution), file_open (file opening), socket_create (socket creation), etc. Pick your hook, bestie.

The tapset library provides typed pointer variables for each hook, accessible via @cast() in guru mode. For example, lsm.bprm_check_security provides bprm (pointer to struct linux_binprm), while lsm.file_open provides file (pointer to struct file). All probes also provide $ctx for raw context access. Slay.

LSM probes can return values to allow or deny operations. In guru mode (-g), set $return to 0 to allow the operation (default), or set it to a negative error code like -EPERM or -EACCES to deny it. The kernel validates return values and uses them to enforce security policy — so yeah, this is real power, use it responsibly 👑

Example uses:

# Monitor program execution
probe lsm.bprm_check_security {
  filename = kernel_string(bprm->filename)
  printf("Executing: %s\n", filename)
}

# Monitor file opening
probe lsm.file_open {
  pathname = d_path(&file->f_path)
  printf("Opening: %s\n", pathname)
}

# Block socket bind operations (guru mode)
probe lsm.socket_bind {
  $return = -1  # EPERM - deny all binds
}

Supported hook categories include:

  • Binary execution: bprm_check_security, bprm_creds_for_exec
  • File operations: file_open, file_permission, file_ioctl, file_lock
  • Inode operations: inode_permission, inode_create, inode_unlink, inode_mkdir, inode_rmdir
  • Socket operations: socket_create, socket_connect, socket_bind, socket_listen
  • Path operations: path_unlink, path_mkdir, path_chmod, path_chown
  • Process/task: task_alloc, task_free
  • Memory mapping: mmap_file, file_mprotect

See tapset/bpf/lsm.stp for a complete list of available hooks and their typed pointer variables. 📜

📡 XDP PACKET-PROCESSING PROBES 📡

These probe points attach eXpress Data Path (XDP) programs to network interfaces, so a script can inspect each incoming packet very early and pass, drop, or bounce it. Requires CONFIG_BPF_SYSCALL=y and an interface whose driver supports XDP.

XDP probes look like kernel.xdp("eth0") or the shorter bare form xdp("eth0"). Several interfaces may be listed, separated by commas: xdp("eth0,eth1"). With no argument, the bare form attaches to every interface that is up — all of them, bestie, the whole squad.

The probe context provides $ctx, a pointer to struct xdp_md (access its fields with @cast() in guru mode), and $return, the packet verdict written back to the driver. A script sets the verdict by assigning to $return:

  • XDP_PASS passes the packet (the default)
  • XDP_DROP drops it 💀
  • XDP_TX bounces it back out the receiving port
  • XDP_REDIRECT hands it to a redirect map

These helper functions and their numeric counterparts (2, 1, 3, 4) are defined in tapset/bpf/xdp.stp.

Example uses:

# Count incoming packets per ethernet protocol on one NIC
global count, total
probe xdp("eth0") {
  eth = @cast($ctx, "xdp_md", "kernel")->data
  proto = @cast(eth, "ethhdr", "kernel")->h_proto
  proto = ((proto & 0xff) << 8) | (proto >> 8 & 0xff)
  count[proto]++
  total++
  $return = XDP_PASS
}

# Report a few known protocols and quit
probe timer.ms(5000) {
  printf("%d total: ipv4 %d arp %d\n", total, count[2048], count[2054])
  exit()
}

👵 KERNEL MARKERS (OBSOLETE) 👵

Heads up bestie — this family is obsolete. It's the vintage stuff. Cute, but vintage. 💀

This family of probe points hooks up to an older style of static probing markers inserted into older kernels or modules. These markers are special STAP_MARK macro calls inserted by kernel developers to make probing faster and more reliable than with DWARF-based probes. Further, DWARF debugging information is not required to probe markers — that part ate, ngl.

Marker probe points begin with kernel. The next part names the marker itself: mark("name"). The marker name string, which may contain the usual wildcard characters, is matched against the names given to the marker macros when the kernel and/or module was compiled. Optionally, you can specify format("format"). Specifying the marker format string allows differentiation between two markers with the same name but different marker format strings — so you can tell the twins apart 💅

The handler associated with a marker-based probe may read the optional parameters specified at the macro call site. These are named $arg1 through $argNN, where NN is the number of parameters supplied by the macro. Number and string parameters are passed in a type-safe manner. Fancy.

The marker format string associated with a marker is available in $format. And also the marker name string is available in $name. Two little locals, living their best lives. 🎀

🛠️ KERNEL HARDWARE BREAKPOINTS 🛠️

This family of probes is used to set hardware watchpoints for a given (global) kernel symbol. The probes take three components as inputs:

  1. The virtual address / name of the kernel symbol to be traced is supplied as argument to this class of probes. (Probes for only data segment variables are supported. Probing local variables of a function cannot be done — sorry, that door is locked 🔒)
  2. Nature of access to be probed:
    • .write probe gets triggered when a write happens at the specified address/symbol name.
    • rw probe is triggered when either a read or write happens.
  3. .length (optional) — users have the option of specifying the address interval to be probed using length constructs. The user-specified length gets approximated to the closest possible address length that the architecture can support. If the specified length exceeds the limits imposed by architecture, an error message is flagged and probe registration fails. Wherever length is not specified, the translator requests a hardware breakpoint probe of length 1. It should be noted that the length construct is not valid with symbol names. Not valid! Don't try it!

Following constructs are supported:

probe kernel.data(ADDRESS).write
probe kernel.data(ADDRESS).rw
probe kernel.data(ADDRESS).length(LEN).write
probe kernel.data(ADDRESS).length(LEN).rw
probe kernel.data("SYMBOL_NAME").write
probe kernel.data("SYMBOL_NAME").rw

This set of probes make use of the debug registers of the processor, which is a scarce resource. (4 on x86, 1 on powerpc.) The script translation flags a warning if a user requests more hardware breakpoint probes than the limits set by architecture. For example, a pass-2 warning is flashed when an input script requests 5 hardware breakpoint probes on an x86 system while x86 architecture supports a maximum of 4 breakpoints. Users are cautioned to set probes judiciously — like, budget your breakpoints bestie, they are not infinite 🙈

Okay so like — yes, you can hand this family of probes some userspace virtual memory addresses, and the handlers will totally fire on the matching memory read/write events in those processes 💅. But here's the tea: you can't easily control which processes get monitored. Using if (pid() == target()) is a workaround, yeah, but it's inefficient bestie. Do yourself a favor and use the userland hardware breakpoint probes below instead. ✨

🦋 USERLAND HARDWARE BREAKPOINTS 🦋

This family is the userland-only sister of its kernel-space counterpart — same vibe, but it targets userland processes and only userland processes.

Here's what's supported rn:

probe process.data(ADDRESS).write
probe process.data(ADDRESS).rw
probe process.data(ADDRESS).length(LEN).write
probe process.data(ADDRESS).length(LEN).rw
probe process.data("SYMBOL").write
probe process.data("SYMBOL").rw
probe process.data("SYMBOL").length(LEN).write
probe process.data("SYMBOL").length(LEN).rw

Currently, only the target process you named with -x PID or -c CMD gets the watchpoints registered. And the ADDRESS has to be a legit virtual memory address inside that process's address space, no cap.

Numeric ADDRESS forms work with the default kernel runtime and with --runtime=dyninst. String SYMBOL forms (i.e. a global variable name) require --runtime=dyninst; stapdyn resolves the name in the target via Dyninst at startup. If the symbol is missing, a warning gets printed and that watchpoint is skipped — rude, but okay. When .length is omitted for a symbol probe, the variable's size from Dyninst is used, and then rounded to a hardware breakpoint width (1, 2, 4, or 8 bytes), with a warning if the size wasn't already one of those values. Also: kernel.data probes are not available with --runtime=dyninst. And HW watchpoints depend on Dyninst/host support (Linux/x86 with debug registers today, iykyk); everywhere else, registration fails at startup if none are available. 😭

With the kernel runtime, a process.data probe may also fire once during the target's execve(2) while the new address space is getting initialized (BSS clearing, for example), before main. The dyninst runtime attaches after that phase, so it only reports later userspace accesses. 🫶

✨ PERF ✨

This family of probe points interfaces to the kernel "perf event" infrastructure for controlling hardware performance counters. The events being attached to are described by the "type", "config" fields of the perf_event_attr structure, and are sampled at an interval governed by the "sample_period" and "sample_freq" fields.

These fields get handed to systemtap scripts using this syntax:

probe perf.type(NN).config(MM).sample(XX)
probe perf.type(NN).config(MM).hz(XX)
probe perf.type(NN).config(MM)
probe perf.type(NN).config(MM).process("PROC")
probe perf.type(NN).config(MM).counter("COUNTER")
probe perf.type(NN).config(MM).process("PROC").counter("NAME")

The systemtap probe handler gets called once per XX increments of the underlying performance counter when you use the .sample field, or at a frequency in hertz when you use the .hz field. When you don't specify either, the default behavior is to sample at a count of 1000000. The range of valid type/config is described by the perf_event_open(2) system call, and/or the linux/perf_event.h file. Invalid combinations or exhausted hardware counter resources result in errors during systemtap script startup. Systemtap does not sanity-check the values: it merely passes them through to the kernel for error- and safety-checking. By default the perf event probe is systemwide unless .process is specified, which binds the probe to a specific task. If the name is omitted, it's inferred from the stap -c argument. A perf event can be read on demand using .counter. The body of the perf probe handler will not be invoked for a .counter probe; instead, the counter is read in a user space probe via:

process("PROC").statement("func@file") {stat <<< @perf("NAME")}

🐍 PYTHON 🐍

Support for probing python 3 functions is available with the help of an extra python support module. Heads up though: the debuginfo for the version of python being probed is required. To run a python script with the extra python support module, you'd add the '-m HelperSDT' option to your python command, like this:

stap foo.stp -c "python -m HelperSDT foo.py"

Python probes look like the following:

python3.module("MPATTERN").function("PATTERN")
python3.module("MPATTERN").function("PATTERN").call
python3.module("MPATTERN").function("PATTERN").return

The list above includes multiple variants and modifiers that provide extra functionality or filters. They are:

  • .function — places a probe at the beginning of the named function by default, unless modified by PATTERN. Parameters are available as context variables.
  • .call — places a probe at the beginning of the named function. Parameters are available as context variables.
  • .return — places a probe at the moment before the return from the named function. Parameters and local/global python variables are available as context variables.

PATTERN stands for a string literal that aims to identify a point in the python program. It's made up of three parts:

  • The first part is the name of a function (e.g. "foo") or class method (e.g. "bar.baz"). This part may use the "*" and "?" wildcarding operators to match multiple names.
  • The second part is optional and begins with the "@" character. It's followed by the path to the source file containing the function, which may include a wildcard pattern. The python path is searched for a matching filename.
  • Finally, the third part is optional if the file name part was given, and identifies the line number in the source file preceded by a ":" or a "+". The line number is assumed to be an absolute line number if preceded by a ":", or relative to the declaration line of the function if preceded by a "+". All the lines in the function can be matched with ":*". A range of lines x through y can be matched with ":x-y". Ranges and specific lines can be mixed using commas, e.g. ":x,y-z".

In the above list of probe points, MPATTERN stands for a python module or script name that names the python module of interest. This part may use the "*" and "?" wildcarding operators to match multiple names. The python path is searched for a matching filename.

🌈 EXAMPLES 🌈

Here are some example probe points, defining the associated events. Buckle up, this list eats 💅

  • begin, end, end — refers to the startup and normal shutdown of the session. In this case, the handler would run once during startup and twice during shutdown.
  • timer.jiffies(1000).randomize(200) — refers to a periodic interrupt, every 1000 +/- 200 jiffies.
  • kernel.function("*init*"), kernel.function("*exit*") — refers to all kernel functions with "init" or "exit" in the name.
  • kernel.function("*@kernel/time.c:240") — refers to any functions within the "kernel/time.c" file that span line 240. Note that this is not a probe at the statement at that line number. Use the kernel.statement probe instead.
  • kernel.trace("sched_*") — refers to all scheduler-related (really, prefixed) tracepoints in the kernel.
  • kernel.tracepoint("sched_switch") — refers to the sched_switch tracepoint using the BTF/vmlinux.h catalog and $arg1, $arg2, etc. arguments.
  • module("nfs").tracepoint("nfs_*") — refers to NFS module tracepoints discovered from module debuginfo.
  • kernel.mark("getuid") — refers to an obsolete STAP_MARK(getuid, ...) macro call in the kernel.
  • module("usb*").function("*sync*").return — refers to the moment of return from all functions with "sync" in the name in any of the USB drivers.
  • kernel.statement(0xc0044852) — refers to the first byte of the statement whose compiled instructions include the given address in the kernel.
  • kernel.statement("*@kernel/time.c:296") — refers to the statement of line 296 within "kernel/time.c".
  • kernel.statement("bio_init@fs/bio.c+3") — refers to the statement at line bio_init+3 within "fs/bio.c".
  • kernel.data("pid_max").write — refers to a hardware breakpoint of type "write" set on pid_max
  • syscall.*.return — refers to the group of probe aliases with any name in the third position

👑 SEE ALSO 👑

Go read these next, bestie, they're the whole friend group:

  • stap(1),
  • probe::*(3stap),
  • tapset::*(3stap)

And that's the whole page, babes 🎀 you made it to the end of STAPPROBES(3stap) — yes, that's the source/version tag this whole thing lives under, straight from the systemtap crew. Now go arm some probes and be the main character of your own trace session. Slay, and probe responsibly 💖✨🦋

📦 where this page comes from