kprobe(8)

📦 version 2017-12, in nixos 26.05, nixpkgs unstable

💖 kprobe — like, the manual page, bestie 💖

okay so you found the kprobe(8) page — welcome to the System Manager's Manual section 8, my love. this is the one about tracing a kprobe definition, aka kernel dynamic tracing, powered by Linux ftrace. put your hair up, we're going in 🔥

✨ NAME ✨

kprobe — trace a given kprobe definition. kernel dynamic tracing. uses Linux ftrace. that's the whole vibe, no cap.

🌸 SYNOPSIS 🌸

kprobe [-FhHsv] [-d secs] [-p PID] [-L TID] kprobe_definition [filter]

she's short, she's iconic, she's got options 💅

🦋 DESCRIPTION 🦋

so basically: this tool creates a kprobe definition, traces it, and then destroys it. like a little pop-up shop for kernel probes. 🎀

for the syntax of a kprobe definition, go read Documentation/trace/kprobetrace.txt in the Linux kernel source. and run kprobe -h for examples, bestie, it's right there.

fun lil thing: with this tool the probe alias is optional. if you don't specify one, it just becomes kprobe:<funcname>. she's flexible like that ✨

⚠️ WARNING, and I mean it: this uses dynamic tracing of kernel functions, and it could cause kernel panics or freezes, depending on the function traced. test in a lab environment. know what you are doing. before use. that's the whole sentence, no notes. 💀

also beware of feedback loops, and this is so important:

  • tracing tcp functions over an ssh session
  • writing ext4 functions to an ext4 file system

for the tcp one, tcp trace data could be redirected to a file (as in the usage message). for the ext4 one, trace to the screen or a different file system. learn from the vibes of people who didn't. 😭

SEE ALSO: functrace(8), which can perform basic tracing (event only) of multiple kernel functions using wildcards. she's the low-stakes bestie version.

since this uses ftrace, only the root user can use this tool. periodt. 👑

🌈 REQUIREMENTS 🌈

FTRACE and KPROBES CONFIG. which you may already have enabled and available on recent kernels — you might literally already have them, check before you stress 🫶

💅 OPTIONS 💅

  • -F — Force. Trace despite warnings. By default the specified kernel function must exist in the available_filter_functions file. this option overrides this check. it might expose you to more unsafe functions, which could cause kernel panics or freezes when traced. so like… iconic but also terrifying 🙈
  • -d seconds — Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.

the ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). if you think events are missing, try increasing that size. ⭐

  • -h — Print usage message. the classic.
  • -H — Print column headers.
  • -s — Print kernel stack traces after each event. the receipts, basically 🧚
  • -v — Show the kprobe format file only (do not trace), identifying possible variables for use in a custom filter. window shopping, no buying 💖
  • -p PID — Only trace kernel functions when this process ID is on-CPU.
  • -L TID — Only trace kernel functions when this thread ID is on-CPU.
  • kprobe_definition — A full kprobe definition, as documented by Documentation/trace/kprobetrace.txt in the Linux kernel source. note that the probe alias name is optional with kprobe(8), and if not specified, the tracepoint will become kprobe:<funcname>. see the EXAMPLES section. 📖
  • filter — An ftrace filter definition.

🎀 EXAMPLES 🎀

these examples may need modification to match your kernel version's function names and your platform's register usage. if using platform specific registers becomes too painful in practice, consider a kernel debuginfo-based tracer, which can trace variables names instead. for example, perf_events. lowkey a whole mood shift, but okay. 💜

Trace do_sys_open() entry:

# kprobe p:do_sys_open

Trace do_sys_open() return:

# kprobe r:do_sys_open

Trace do_sys_open() return value:

# kprobe 'r:do_sys_open $retval'

Trace do_sys_open() return value, with a custom probe alias myopen:

# kprobe 'r:myopen do_sys_open $retval'

Trace do_sys_open() file mode:

# kprobe 'p:myopen do_sys_open mode=%cx:u16'

Trace do_sys_open() file mode for PID 81:

# kprobe -p 81 'p:myopen do_sys_open mode=%cx:u16'

Trace do_sys_open() with filename string:

# kprobe 'p:myopen do_sys_open filename=+0(%si):string'

Trace do_sys_open() for filenames ending in "stat":

# kprobe 'p:myopen do_sys_open fn=+0(%si):string' 'fn ~ "*stat" '

Trace tcp_retransmit_skb() and show kernel stack traces, showing the path that led to it (can help explain why):

# kprobe -s 'p:myprobe tcp_retransmit_skb'

👑 FIELDS 👑

the output format depends on the kernel version, and headings can be printed using -H. the format is the same as the ftrace function trace format, described in the kernel source under Documentation/trace/ftrace.txt.

typical fields are:

  • TASK-PID — The process name (which could include dashes), a dash, and the process ID.
  • CPU# — The CPU ID, in brackets.
  • |||| — Kernel state flags. For example, on Linux 3.16 these are for irqs-off, need-resched, hardirq/softirq, and preempt-depth.
  • TIMESTAMP — Time of event, in seconds.
  • FUNCTION — Kernel function name.

🔥 OVERHEAD 🔥

this can generate a lot of trace data quickly, depending on the frequency of the traced events. such data will cause performance overheads. it's giving chaos, honestly. 😭

this also works without buffering by default, printing function events as they happen (uses trace_pipe), context switching and consuming CPU to do so. so yeah, that's a cost.

if needed, you can try the "-d secs" option, which buffers events instead, reducing overhead. and if you think the buffer option is losing events, try increasing the buffer size (buffer_size_kb). 🍓

it's a good idea to use funccount(8) first, which is lower overhead, to help you select which functions you may want to trace using kprobe(8). scout first, then commit. main character planning energy ✨

⭐ SOURCE ⭐

this is from the perf-tools collection:

https://github.com/brendangregg/perf-tools

also look under the examples directory for a text file containing example usage, output, and commentary for this tool. 🫶

🌸 OS 🌸

Linux

🦋 STABILITY 🦋

Unstable — in development. she's growing, okay? 🙈

💜 AUTHOR 💜

Brendan Gregg

🌈 SEE ALSO 🌈

functrace(8), funccount(8)

okay that's the whole page, bestie — you ate. now go trace responsibly 💅✨

this lil beauty is from the USER COMMANDS section, version dated 2014-07-20, and yes she's still out here doing kprobe(8) things. love that for her. 🎀🦋💖

📦 where this page comes from