BPFTRACE(8)

📦
version 0.27.0 · 17 versions
, in arch rolling

💅 BPFTRACE(8) — the bpftrace(8) manual, but make it slay

okay so. welcome to bpftrace(8), bestie. this is the page for bpftrace, a high-level tracing language. grab a boba, we're gonna get into it. 🧚✨

✨ NAME ✨

bpftrace — a high-level tracing language. that's it. that's the whole vibe. it traces things. iconically. 👑

🌸 SYNOPSIS 🌸

bpftrace [OPTIONS] FILENAME
bpftrace [OPTIONS] -e 'program code'

so when FILENAME is -, bpftrace will read program code from stdin. cute, right? 🎀

and a program just keeps running until you hit Ctrl+C, or until an exit function is called. when a program exits, all populated maps are printed (more deets below, keep reading 💖).

🦋 DESCRIPTION 🦋

bpftrace is a high-level tracing language for Linux. it uses LLVM as a backend to compile scripts to eBPF-bytecode, and it makes use of libbpf and bcc for interacting with the Linux BPF subsystem, plus existing Linux tracing capabilities. literally the whole squad working together. no cap. 🔥

🌈 EXAMPLES 🌈

bestie, examples are the best part. here we go:

Trace processes calling sleep 😴

# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'

Trace processes calling sleep while spawning sleep 5 as a child process

# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }' -c 'sleep 5'

List all probes with "sleep" in their name

# bpftrace -l '*sleep*'

List all the probes attached in the program

# bpftrace -l -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'

see? told you. we ate. 💅

💜 OPTIONS 💜

here's where it gets juicy. every single flag, coming right up. 🔥

🫶 -B MODE

Set the output buffer mode (this applies to terminal output and file output via -o). ok so:

  • none — No buffering. Each I/O is written as soon as possible. zero patience energy, honestly.
  • line — Data is written on the first newline or when the buffer is full. This is the default mode. the main character of buffer modes ✨
  • full — Data is written once the buffer is full.

💖 -c COMMAND

Run COMMAND as a child process. when the child terminates, bpftrace will also terminate, as if exit() had been called. and if bpftrace terminates before the child process does, the child process will be terminated with a SIGTERM 😭. if used, USDT probes will only be attached to the child process.

to avoid a race condition when using USDTs, the child is stopped after execve using ptrace(2) and continued when all USDT probes are attached. also: the child process runs with the same privileges as bpftrace itself (usually root). 👑

unless otherwise specified, bpftrace does not perform any implicit filtering. so if you're only interested in events in COMMAND, you may want to filter based on the child PID. the child PID is available to programs as the cpid builtin. for example, you could add the predicate /pid == cpid/ to probes with userspace context. iykyk 🧠

🎀 -d STAGE

Enable debug mode. for more details see the Debug Output section. (debug girlies rise 🙈)

🦋 --debuginfo DIR[:DIR]

Add the directory DIR to the search path for DWARF debug information. paths may be absolute or relative to the traced binary's location. specify multiple paths either by separating them with a colon (:) or by repeating the option — e.g. --debuginfo=/bin/debug:./lib/debug:.. and --debuginfo=/bin/debug --debuginfo=./lib/debug, respectively. debug files in these paths are matched by build ID when available; prefix a path with + to enable CRC32 validation when needed. files that do not match are skipped. by default, bpftrace searches under standard debug paths, including .debug relative to the traced binary and /usr/lib/debug; it may also query available debuginfod servers. 🗂️

heads up bestie: split debuginfo format (.dwo, .dwp) is not yet supported for this option, but continues to work with the default probe binary path. 😭

💀 --dry-run

Terminate execution right after attaching all the probes. useful for testing that the script can be parsed, loaded, and attached, without actually running it. it's giving "rehearsal dinner, not the wedding" 🥂

⭐ -e PROGRAM

Execute PROGRAM instead of reading the program from a file or stdin. the shortcut queen 👑

🌸 -f FORMAT

Set the output format. valid values are:

  • json
  • text

the JSON output is compatible with NDJSON and JSON Lines, meaning each line of the streamed output is a single blob of valid JSON. gorgeous. valid. both. ✨

🧚 --fmt FILENAME

Output standard format for the bpftrace file FILENAME.

🫶 -h, --help

Print the help summary. when in doubt, ask for help. self care 💖

🍓 -I DIR

Add the directory DIR to the search path for C headers. this option can be used multiple times. for more details see the Preprocessor Options section.

💜 --include FILENAME

Add FILENAME as an include for the pre-processor. this is equal to adding #include FILENAME at the top of the program. this option can be used multiple times. for more details see the Preprocessor Options section. ✨

🔥 --info

Print detailed information about features supported by the kernel and the bpftrace build. it's giving receipts. 📋

👑 -k

This flag enables runtime warnings for errors from probe_read_* and map lookup BPF helpers. when errors occur, bpftrace will log an error containing the source location and the error code:

stdin:48-57: WARNING: Failed to probe_read_user_str: Bad address (-14)
u:lib.so:"fn(char const*)" { printf("arg0:%s\n", str(arg0));}
                                                 ~~~~~~~~~

see how it points right at the problem? iconic behavior, honestly. 💅

🦋 -l [SEARCH|FILENAME]

List all probes that match the SEARCH pattern. if the pattern is omitted, all probes will be listed. this pattern supports wildcards in the same way that probes do. e.g. -l kprobe:*file* to list all kprobes with file in the name. this can be used with a program, which will list all probes in that program. for more details see the Listing Probes section. 🔍

🌈 --no-feature feature,feature,...

Disable use of detected features. valid values are:

  • uprobe_multi — to disable uprobe_multi link
  • kprobe_multi — to disable kprobe_multi link
  • kprobe_session — to disable automatic collapse of kprobe/kretprobe into kprobe session

like turning off read receipts but for kernel features 🙈

💀 --no-warnings

Suppress all warning messages created by bpftrace. ignorance is bliss, bestie.

🎀 --probe-filter REGEX

Only attach and run probes whose name matches the regular expression REGEX. probes that do not match are not loaded into the kernel. special probes (BEGIN and END) are not affected by this filter. if no probes match, bpftrace exits with an error. 😭 rude but fair.

⭐ --traceable-functions FILENAME

Specify the file containing the list of traceable kernel functions. if not set, bpftrace uses /sys/kernel/tracing/available_filter_functions, which depends on dynamic ftrace. the format is the same as in available_filter_functions: each line is either "symbol" or "symbol [module]". 📝

🍓 -o FILENAME

Write bpftrace tracing output to FILENAME instead of stdout. this doesn't include child process (-c option) output. errors are still written to stderr. (errors always find their way home 😤)

💜 -p PID

Attach to the process with or filter actions by PID. if the process terminates, bpftrace will also terminate. when using USDT, uprobes, uretprobes, hardware, software, profile, interval, or watchpoint probes, they will be attached to only this process. for all other probes, except begin/end, the pid will act like a predicate to filter out events not from that pid. for listing uprobes/uretprobes, set the target to '*' and the process's address space will be searched for the symbols. 🎯

🧚 -q

Keep messages quiet. 🤫

🔥 --unsafe

Some calls, like system, are marked as unsafe as they can have dangerous side effects (system("rm -rf")) and are disabled by default. this flag allows their use. 😬

bestie. please be careful with this one. it is literally named --unsafe. that is not a vibe check you want to fail.

in addition, it makes kprobe/kretprobe checks less restrictive. it can be used to probe functions that bpftrace reports as not traceable, but are supported if the kernel is configured to allow probing notrace functions.

🌸 --usdt-file-activation

Activate usdt semaphores based on file path.

💅 -V, --version

Print bpftrace version information. know your queen's lore. 👑

🦋 -v

Enable verbose messages. for more details see the Verbose Output section. it's giving commentary track 🎙️

🫶 Program Options

okay this part is lowkey the coolest. you can also pass custom options to a bpftrace program/script itself via positional or named parameters. positional parameters can be placed before or after a double dash, but named parameters can ONLY come after a double dash. e.g.

# bpftrace -e 'begin { print(($1, $2, getopt("aa", 1), getopt("bb"))); }' p1 -- --aa=20 --bb p2

// (p1, p2, 20, true) is printed

or

# bpftrace myscript.bt -- p1 --aa=20 --bb p2

in these examples there are two positional parameters (p1, p2) and two named parameters (aa, which is set to 20, and bb, which is set to true). named program parameters require the = to set their value unless they are boolean parameters (like bb above). read about how to access positional and named parameters here: https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#command-line-parameters. 🧚✨

💖 THE LANGUAGE 💖

Syntax, types, and concepts for bpftrace are available here: https://github.com/bpftrace/bpftrace/blob/master/docs/language.md. go read it, it's giving required reading for the book club 📚

⭐ PROBES ⭐

bpftrace supports various probe types which allow the user to attach BPF programs to different types of events. each probe starts with a provider (e.g. kprobe) followed by a colon (:) separated list of options. the amount of options and their meaning depend on the provider. optionally, the attachpoint specified can be prefixed by a user-provided name followed by an equal sign (e.g. myname=kprobe:...); this is used for certain advanced features and is typically omitted. full list of probe types: https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#probes. 🎯

🌸 STANDARD LIBRARY 🌸

the standard library of all builtins (https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#builtins), functions (https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#functions), and map functions (https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#map-functions) is available here: https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md. literally your toolbox, bestie 🧰💅

🎀 CONFIGURATION 🎀

Okay bestie, this is the little control room for config stuff 💅

🧚 Config Variables 🧚

So some behavior can only be controlled through config variables, and they live here: https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#config-variables. Iconic.

You can set them via the Config Block directly in a script (before any probes), or via their environment variable equivalent. The env var version is upper case and includes the BPFTRACE_ prefix — for example, stack_mode's environment variable would be BPFTRACE_STACK_MODE. Slay.

🌈 Environment Variables 🌈

These ones are not part of the standard set of Config Variables above, and they can only be set as environment variables. No Config Block shortcut for these, sorry bestie 😭

🌸 BPFTRACE_BTF 🌸

Default: None

The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. See src/btf.cpp for the details.

🦋 BPFTRACE_KERNEL_BUILD 🦋

Default: /lib/modules/$(uname -r)

Only used with BPFTRACE_KERNEL_SOURCE if it is out-of-tree Linux kernel build.

🎀 BPFTRACE_KERNEL_SOURCE 🎀

Default: /lib/modules/$(uname -r)

bpftrace requires kernel headers for certain features, which are searched for in this directory.

💜 BPFTRACE_VMLINUX 💜

Default: None

This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. If this value is not given, bpftrace searches vmlinux from pre defined locations. See src/attached_probe.cpp:find_vmlinux() for details.

🍓 BPFTRACE_COLOR 🍓

Default: auto

Colorize the bpftrace log output message. Valid values are auto, always and never.

🔥 OPTIONS EXPANDED 🔥

🐛 Debug Output 🐛

The -d STAGE option produces debug output. It prints the output of the bpftrace execution stage given by the STAGE argument. The option can be used multiple times (with different stage names), and the special value all prints the output of all the supported stages. The option also takes multiple stages in one invocation as comma separated values. Very multitasking icon behavior ✨

Note: This is primarily used for bpftrace developers.

The supported options are:

  • ast — Prints the Abstract Syntax Tree (AST) after every pass.
  • codegen — Prints the unoptimized LLVM IR as produced by CodegenLLVM.
  • codegen-opt — Prints the optimized LLVM IR, i.e. the code which will be compiled into BPF bytecode.
  • dis — Disassembles and prints out the generated bytecode that libbpf will see. Only available in debug builds.
  • libbpf — Captures and prints libbpf log for all libbpf operations that bpftrace uses.
  • verifier — Captures and prints the BPF verifier log.
  • all — Prints the output of all of the above stages.

🔍 Listing Probes 🔍

Probe listing is the method to discover which probes are supported by the current system. Listing supports the same syntax as normal attachment does, and alternatively can be combined with -e or filename args to see all the probes that a program would attach to. It's giving detective energy 🕵️‍♀️

# bpftrace -l 'kprobe:*'
# bpftrace -l 't:syscalls:*openat*
# bpftrace -l 'kprobe:tcp*,trace
# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
# bpftrace -l -e 'tracepoint:xdp:mem_* { exit(); }'
# bpftrace -l my_script.bt
# bpftrace -lv 'enum cpu_usage_stat'

The verbose flag (-v) can be specified to inspect arguments (args) for providers that support it:

# bpftrace -l 'fexit:tcp_reset,tracepoint:syscalls:sys_enter_openat' -v
fexit:tcp_reset
    struct sock * sk
    struct sk_buff * skb
tracepoint:syscalls:sys_enter_openat
    int __syscall_nr
    int dfd
    const char * filename
    int flags
    umode_t mode

# bpftrace -l 'uprobe:/bin/bash:rl_set_prompt' -v    # works only if /bin/bash has DWARF
uprobe:/bin/bash:rl_set_prompt
    const char *prompt

# bpftrace -lv 'struct css_task_iter'
vmlinux: struct css_task_iter {
        struct cgroup_subsys *ss;
        unsigned int flags;
        struct list_head *cset_pos;
        struct list_head *cset_head;
        struct list_head *tcset_pos;
        struct list_head *tcset_head;
        struct list_head *task_pos;
        struct list_head *cur_tasks_head;
        struct css_set *cur_cset;
        struct css_set *cur_dcset;
        struct task_struct *cur_task;
        struct list_head iters_node;
};

🧵 Preprocessor Options 🧵

The -I option can be used to add directories to the list of directories that bpftrace uses to look for headers. Can be defined multiple times. Lowkey handy.

# cat program.bt
#include <foo.h>

begin { @ = FOO }

# bpftrace program.bt

definitions.h:1:10: fatal error: 'foo.h' file not found

# /tmp/include
foo.h

# bpftrace -I /tmp/include program.bt

Attached 1 probe

The --include option can be used to include headers by default. Can be defined multiple times. Headers are included in the order they are defined, and they are included before any other include in the program being executed. Periodt.

# bpftrace --include linux/path.h --include linux/dcache.h \
    -e 'kprobe:vfs_open { printf("open path: %s\n", str(((struct path *)arg0).dentry.d_name.name)); }'

Attached 1 probe
open path: .com.google.Chrome.ASsbu2
open path: .com.google.Chrome.gimc10
open path: .com.google.Chrome.R1234s

🔊 Verbose Output 🔊

The -v option prints more information about the program as it is run:

# bpftrace -v -e 'tracepoint:syscalls:sys_enter_nanosleep { printf("%s is sleeping.\n", comm); }'
AST node count: 7
Attached 1 probe

load tracepoint:syscalls:sys_enter_nanosleep, with BTF, with func_infos: Success

Program ID: 111
Attaching tracepoint:syscalls:sys_enter_nanosleep
iscsid is sleeping.
iscsid is sleeping.
[...]

🦋 TERMINOLOGY 🦋

okay bestie, vocab check time 💅 before we go any further, here's the group chat glossary so you're not lost when the docs start throwing these words around:

  • BPF — Berkeley Packet Filter: a kernel technology originally developed for optimizing the processing of packet filters (eg, tcpdump expressions). it's giving "old but iconic" 👑
  • BPF map — A BPF memory object, which is used by bpftrace to create many higher-level objects.
  • BTF — BPF Type Format: the metadata format which encodes the debug info related to BPF program/map.
  • dynamic tracing — Also known as dynamic instrumentation, this is a technology that can instrument any software event, such as function calls and returns, by live modification of instruction text. Target software usually does not need special capabilities to support dynamic tracing, other than a symbol table that bpftrace can read. Since this instruments all software text, it is not considered a stable API, and the target functions may not be documented outside of their source code. ⚠️ so like, it works on everything but stability is not guaranteed — proceed with main character energy but keep your expectations in check fr fr
  • eBPF — Enhanced BPF: a kernel technology that extends BPF so that it can execute more generic programs on any events, such as the bpftrace programs listed below. It makes use of the BPF sandboxed virtual machine environment. Also note that eBPF is often just referred to as BPF. 🫶 (yes the naming is chaos, we live with it)
  • kprobes — A Linux kernel technology for providing dynamic tracing of kernel functions.
  • probe — An instrumentation point in software or hardware, that generates events that can execute bpftrace programs.
  • static tracing — Hard-coded instrumentation points in code. Since these are fixed, they may be provided as part of a stable API, and documented. ✨ the reliable one
  • tracepoints — A Linux kernel technology for providing static tracing.
  • uprobes — A Linux kernel technology for providing dynamic tracing of user-level functions.
  • USDT — User Statically-Defined Tracing: static tracing points for user-level software. Some applications support USDT.

🎀 PROGRAM FILES 🎀

bestie, here's a thing that lowkey slays: you can save your programs as files 🍓 — they're often called scripts, and you can just run them by specifying their file name. The convention is to use the .bt file extension but it is not required (the .bt is a vibe, not a rule 💅).

For example, listing the sleepers.bt file using cat:

# cat sleepers.bt

tracepoint:syscalls:sys_enter_nanosleep {
  printf("%s is sleeping.\n", comm);
}

And then calling it:

# bpftrace sleepers.bt

Attached 1 probe
iscsid is sleeping.
iscsid is sleeping.

And it can also be made executable to run stand-alone 👑 ngl this is the fun part. Start by adding an interpreter line at the top (#!) with either the path to your installed bpftrace (/usr/local/bin is the default) or the path to env (usually just /usr/bin/env) followed by bpftrace (so it will find bpftrace in your $PATH):

#!/usr/local/bin/bpftrace

tracepoint:syscalls:sys_enter_nanosleep {
  printf("%s is sleeping.\n", comm);
}

Then make it executable:

# chmod 755 sleepers.bt
# ./sleepers.bt

Attached 1 probe
iscsid is sleeping.
iscsid is sleeping.
🧚 friendly reminder: that # prompt in the examples is just showing you the shell — the #! line inside the file is the actual interpreter magic. two very different #s, iykyk 😭

and that's the whole page, bestie!! you've officially read the entire BPFTRACE(8) manual and honestly? proud of you 🫶💖 this edition is dated 2026-09-24 and it's filed under BPFTRACE(8) — so if anyone asks, you got receipts. go forth and trace dynamically, angel 🌈⭐💜

📦 where this page comes from