PLY(1)

💖 PLY(1) 💖

okay bestie, so this is ply from the General Commands Manual section 1, and lemme tell you, she is that girl 💅

---

✨ NAME ✨

ply — dynamically instrument the kernel 🧚

🌸 SYNOPSIS 🌸

ply program-file
ply program-text

🦋 DESCRIPTION 🦋

so basically, ply dynamically instruments the running kernel to aggregate and extract user-defined data. no cap.

she compiles an input program into one or more Linux bpf(2) binaries and attaches them to arbitrary points in the kernel using kprobes and tracepoints. literally main character energy for your kernel 🔥

---

👑 OPTIONS 👑

• -c command, --command=command — when all the probes are running, run command. then when the command exits, stop all the probes and exit too. it's giving "run as if invoked with sh -c <command>" energy ✨
• -d, --debug — turn on debugging output. we love a verbose queen 🙈
• -e, --dry-run — exit right after compilation, without actually instrumenting the system. typically used together with --dump. dip out before anything gets real 💅
• -h, --help — print the usage message, bestie. read it fr fr 💖
• -k, --keep-going — instead of printing a warning and exiting whenever any trace data is lost, it only prints the warning and keeps going. she's resilient, periodt 🫶
• -S, --dump — after compilation, dump the internal AST, the generated BPF instructions and other internal information. this is very useful to include when reporting a bug 🔥 (literally do this, iykyk)
• -T, --self-test — run the built-in self-test, which verifies that the required features in the kernel are available and that all providers are operational. a lil vibe check for your whole setup 🌈
• -u, --unbuffer — unconditionally disable buffering of stdout/stderr, even when they are not connected to a TTY. super useful in interactive sessions where the output is further processed by another program 🍓
• -v, --version — print version information. iconique ⭐

---

💜 SYNTAX 💜

the syntax is C-like in general, taking its inspiration from dtrace(1) and, by extension, from awk(1). the range bestie 😭

🎀 Probes 🎀

a program consists of one or more probes, which are analogous to awk's pattern-action statements. the syntax for a probe goes like this:

provider:probe-definition ['/' predicate '/']
{
     statement ';'
    [statement ';' ... ]
}

the provider selects which probe interface to use. see the PROVIDERS section for more info about each provider. then it's up to the provider to parse the probe-definition to determine the point(s) of instrumentation 👀

when tracing, it's often desirable to filter events to match some criteria. because of this, ply lets you provide a predicate, i.e. an expression that must evaluate to a non-zero value in order for the probe to be executed. no nonzero, no run, simple as that bestie 💅

then follows a block of statements that perform the actual information gathering. that's the whole point, the gathering, the slay ✨

a provider may define a default probe clause to be used if the user does not supply one. generous of her fr fr 🫶

🌸 Control of Flow 🌸

probes support basic conditional control of flow via an if-statement, which conforms to the same rules as C's equivalent:

'if' '(' expr ')'
    statement ';' | block
[else
    statement ';' | block]

in order to ensure that a probe will have a finite run-time, the kernel does not allow backwards branching 😭 as a result, ply does not have any loop construct like for or while. a simple for statement with an invariant that is known at compile-time could be added later — and in that case we could unroll the loop when generating BPF. manifesting it tbh 🙈

🦋 Type System 🦋

the type system is modeled after C. as such, ply understands the difference between signed and unsigned integers, the difference between a short and a long long, what separates an integer from a pointer, how a struct is laid out in memory and so on. she's thorough! but it is not complete though — notably floating point numbers and unions are missing 💀

programs are statically typed, but all types are inferred automatically. so the type system is mostly hidden from the user. plans are to expose more of it in the future by allowing casts, type declarations and so on. (future era, we're ready 👑)

numbers and string literals are specified in the same way as in C. easy 🍓

🧚 Maps 🧚

the primary way to extract information is to store it in a map, i.e. in a hash table. like awk(1), ply dynamically creates any referenced maps and their key and value types are inferred from the context in which they are used. all maps are in the global scope and can thus be used both for extracting data to the end-user, and for carrying data between probes. map names follow the rules of identifiers from C.

mapname[exprs]

data can be stored in a map by assigning a value to a given key:

mapname[exprs] = expr

the delete keyword can be used to remove an association from a map:

delete mapname[exprs]

you can also remove all elements in the map using the clear function. clean girl aesthetic 🎀

💅 Aggregations 💅

more often than not, looking at each individual datum from a trace is not nearly as helpful as an aggregation of the data. so ply supports aggregating data at the source, thereby reducing tracing overhead. slay, efficient queen ✨

aggregations are syntactically similar to maps — indeed they are a kind of map — but they are distinguished by a leading @. also, they can only be assigned the result of one of the following aggregation functions:

• @agg[exprs] = count() — bump a counter. that's it. one little bump 📈
• @agg[exprs] = sum(scalr-expr) — evaluates the argument and aggregates the result.
• @agg[exprs] = quantize(scalar-expr) — evaluates the argument and aggregates on the most significant bit of the result. in other words, it stores the distribution of the expression. it's giving histogram energy and we're obsessed 🌈

---

highkey iconic, and that's only part 1 bestie 💖

🌈 PROVIDERS 💖

A provider is basically the plug that feeds data to your probe — it exports functions and variables straight into it. Function calls use the same syntax as, like, every language that ever inherited from C. And on top of whatever a provider ships with, every provider also inherits a lil starter pack of common functions and variables:

• char[16] comm, char[16] execname — name of the running process's executable.
• u32 cpu — the CPU ID of the processor the probe fired on.
• u32 gid — group ID of the running process.
• u32 kpid — kernel PID of the running process. The kernel also calls this pid. For a single-threaded process kpid equals pid. For multi-threaded processes, kpid is unique per thread while pid stays the same across all threads. slay, that's the whole point.
• char[N] mem(void *address [, int size]) — copy size bytes from address. Omit size and it copies 64 bytes.
• s64 time, s64 walltime — nanoseconds elapsed since system boot. time is for time deltas, walltime is for timestamps. Same data underneath, just different default output formats, iykyk.
• u32 pid — process ID of the running process. The kernel also calls this the thread group ID (tgid).
• void print(...) — print each expression with its default output format, comma-separated and terminated with a newline, out to ply's standard out.
• void printf(format, ...) — prints formatted output to ply's standard out. In addition to the formats recognized by the printf sitting in your
• int strcmp(char *a, char *b) — returns -1, 0 or 1 when the first argument is less than, equal to or greater than the second argument. Strings get compared by lexicographical order.
• u32 uid — user ID of the running process.

🦋 kprobe and kretprobe

These two use the matching kernel features to instrument arbitrary instructions in the kernel. The probe-definition can be either an address or a symbol name. Give it a symbol name and glob expansion happens, so one probe can land in multiple locations. You can also tack on an offset relative to a symbol for kprobes. periodt.

Examples:

• kretprobe:schedule — trace every time schedule returns.
• kprobe:SyS_* — trace every time a syscall is made.
• kprobe:dev_hard_start_xmit+8 — trace function with offset.

Shared variables:

• struct pt_regs *regs — hardware register contents from when the probe was triggered. This matches the definition in <sys/ptrace.h> on your system.
• u32 stack — stack trace ID of the current probe. This just returns an index into a separate map holding the actual instruction pointers. As a user though, you can think of this function as returning a string containing the stack trace at the current location. Indeed print(stack) will produce exactly that.

CAUTION: on some architectures (looking at you, ARM 👀), capturing stack traces at the entry of a function — before the prologue has run — does not work. Setting your probe after the prologue works around it (typically two instructions, or +8, on ARM). bestie do not forget this one.

kprobe specific functions:

• arg0, arg1 ... argN:
• void *caller — the program counter, as recorded in regs, at the time the probe was triggered. was attached. The default output format will resolve it to a symbolic name if one is available.

kretprobe specific function:

• retval — return value of the probed function.

🌸 tracepoint

The tracepoint provider can instrument all the stable tracepoints in the kernel ✨. They're identified by their relative path from the /sys/kernel/debug/tracing/events directory, where each leaf directory corresponds to a tracepoint.

Examples:

• tracepoint:sched/sched_wakeup — trace every time a process is awoken.
• tracepoint:irq/irq_handler_entry — trace every time an interrupt is handled.

Variables:

• struct <X> *data — a struct gets dynamically generated for each tracepoint by parsing its format file. I.e. if the contents of the format file look like the following:

name: tcp_send_reset
ID: 1304
format:
     field:unsigned short common_type;  offset:0; size:2;   signed:0;
     field:unsigned char common_flags;  offset:2; size:1;   signed:0;
     field:unsigned char common_preempt_count;    offset:3; size:1;   signed:0;
     field:int common_pid;    offset:4; size:4;   signed:1;

     field:const void * skbaddr;   offset:8; size:8;   signed:0;
     field:const void * skaddr;    offset:16;     size:8;   signed:0;
     field:__u16 sport;  offset:24;     size:2;   signed:0;
     field:__u16 dport;  offset:26;     size:2;   signed:0;
     field:__u8 saddr[4];     offset:28;     size:4;   signed:0;
     field:__u8 daddr[4];     offset:32;     size:4;   signed:0;
     field:__u8 saddr_v6[16]; offset:36;     size:16;  signed:0;
     field:__u8 daddr_v6[16]; offset:52;     size:16;  signed:0;

Then data would point to a struct of the following type:

struct data {
    unsigned short common_type;
    unsigned char common_flags;
    unsigned char common_preempt_count;
    int common_pid;

    const void * skbaddr;
    const void * skaddr;
    __u16 sport;
    __u16 dport;
    __u8 saddr[4];
    __u8 daddr[4];
    __u8 saddr_v6[16];
    __u8 daddr_v6[16];
};

Functions:

• char[N] dyn(void *address [, int size]) — copy size bytes from a dynamic data pointer in data, i.e. a member marked with __data_loc. Omit size and the default string size decides how many bytes get copied.

💜 BEGIN and END

These special providers are called at the beginning and the end of the tracing session, exactly like awk and bpftrace do it 🫶. The names are case sensitive (yes fr). Users can print some messages or fill maps to known info.

🎀 interval

The interval provider will be trigger at each given interval. Users can specify time and unit (optional). If unit is omitted, then second is used. The supported units are:

• m — minutes
• s — seconds (default)
• ms — milli-seconds
• us — micro-seconds
• ns — nano-seconds

Examples:

• interval:1 — called for every second
• interval:500ms — called for every 500 milli-second

👑 profile

The profile provider supports profiling by letting the user specify how many times it will fire per-second. Values of 1-1000 are supported, and the profile provider supports two probe formats:

• profile:[N]hz — profile on all CPUs N times per second
• profile:[C]:[N]hz — profile on CPU C N times per second

---

🍓 EXAMPLE 🍓

Extracting data

Print all openated files on the system, and who opened them:

kprobe:SyS_openat
{
    print(comm, pid, str(arg1));
}

Quantize

Record the distribution of the return value of read(2):

kretprobe:SyS_read
{
    @["dist"] = quantize(retval);
}

Wildcards

Count all syscalls made on the system, grouped by function:

kprobe:SyS_*
{
    @[caller] = count();
}

Count all syscalls made by every dd(1) process, grouped by function:

kprobe:SyS_* / !strcmp(execname, "dd") /
{
    @[caller] = count();
}

Object Tracking

Record the distribution of the time it takes an skb to go from netif_receive to ip_rcv:

kprobe:__netif_receive_skb_core
{
    rx[arg0] = time;
}

kprobe:ip_rcv / rx[arg0] /
{
    @["diff"] = quantize(time - rx[arg0]);
}

---

⭐ RETURN VALUE ⭐

• 0 — program was successfully compiled and loaded into the kernel. ate 💅
• Non-Zero — an error occurred during compilation or during kernel setup. 😭

🫶 AUTHORS

Tobias Waldekranz tobias@waldekranz.com

🌈 COPYRIGHT

Copyright 2018 Tobias Waldekranz

License: GPLv2

🦋 SEE ALSO

awk(1) dtrace(1) bpf(2)

---

and that's the whole page bestie!! this one's the PLY(1) manual from January 2026 — source and version and date, all accounted for, no cap 💖 now go trace something iconic ✨👑