SDT(9)

💖 SDT(9) 💖 — the DTrace framework for adding statically-defined tracing probes

okay bestie, this is the page for SDT, straight out of the Kernel Developer's Manual. it's giving "I write kernel code and I want pretty little probes in it" energy and honestly? iconic. let's get into it 🫶

---

✨ NAME ✨

SDT — a DTrace framework for adding statically-defined tracing probes. yep that's the whole vibe check 💅

---

🌸 SYNOPSIS 🌸

first, the includes. you gotta bring these to the party:

#include <sys/param.h>
#include <sys/queue.h>
#include <sys/sdt.h>

then the provider macros:

SDT_PROVIDER_DECLARE(prov);

SDT_PROVIDER_DEFINE(prov);

then declaring and defining probes:

SDT_PROBE_DECLARE(prov, mod, func, name);

SDT_PROBE_DEFINE(prov, mod, func, name);

SDT_PROBE_DEFINE0(prov, mod, func, name);

SDT_PROBE_DEFINE1(prov, mod, func, name, arg0);

SDT_PROBE_DEFINE2(prov, mod, func, name, arg0, arg1);

SDT_PROBE_DEFINE3(prov, mod, func, name, arg0, arg1, arg2);

SDT_PROBE_DEFINE4(prov, mod, func, name, arg0, arg1, arg2, arg3);

SDT_PROBE_DEFINE5(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4);

SDT_PROBE_DEFINE6(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4,
    arg5);

SDT_PROBE_DEFINE7(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4,
    arg5, arg6);

and the ones that translate your args on the fly, the _XLATE girlies ✨:

SDT_PROBE_DEFINE0_XLATE(prov, mod, func, name);

SDT_PROBE_DEFINE1_XLATE(prov, mod, func, name, arg0, xarg0);

SDT_PROBE_DEFINE2_XLATE(prov, mod, func, name, arg0, xarg0, arg1,
    xarg1);

SDT_PROBE_DEFINE3_XLATE(prov, mod, func, name, arg0, xarg0, arg1, xarg1,
    arg2, xarg2);

SDT_PROBE_DEFINE4_XLATE(prov, mod, func, name, arg0, xarg0, arg1, xarg1,
    arg2, xarg2, arg3, xarg3);

SDT_PROBE_DEFINE5_XLATE(prov, mod, func, name, arg0, xarg0, arg1, xarg1,
    arg2, xarg2, arg3, xarg3, arg4, xarg4);

SDT_PROBE_DEFINE6_XLATE(prov, mod, func, name, arg0, xarg0, arg1, xarg1,
    arg2, xarg2, arg3, xarg3, arg4, xarg4, arg5, xarg5);

SDT_PROBE_DEFINE7_XLATE(prov, mod, func, name, arg0, xarg0, arg1, xarg1,
    arg2, xarg2, arg3, xarg3, arg4, xarg4, arg5, xarg5, arg6, xarg6);

and finally the trace points themselves, the ones you actually drop into your code:

SDT_PROBE0(prov, mod, func, name);

SDT_PROBE1(prov, mod, func, name, arg0);

SDT_PROBE2(prov, mod, func, name, arg0, arg1);

SDT_PROBE3(prov, mod, func, name, arg0, arg1, arg2);

SDT_PROBE4(prov, mod, func, name, arg0, arg1, arg2, arg3);

SDT_PROBE5(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4);

SDT_PROBE6(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4, arg5);

SDT_PROBE7(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4, arg5,
    arg6);

---

🦋 DESCRIPTION 🦋

so. the SDT macros let programmers define static trace points in kernel code. these trace points get picked up by the SDT framework to make DTrace probes, which means your code can be instrumented using dtrace(1). cute, right?

by default? trace points are disabled and have no effect on the surrounding code. but the second a DTrace probe matching a given trace point gets enabled, the threads executing that trace point call a handler and the probe fires 🔥. and bonus: trace points can take arguments, so you can pass data into the DTrace framework when an enabled probe fires. data delivery, slay.

multiple trace points can map to a single DTrace probe. this means you can build DTrace probes that represent logical system events instead of tying them to one specific code execution path. for example (fr, this is the real one from the page): a DTrace probe for an IP packet arriving into the network stack could be defined with two SDT trace points — one for IPv4 packets and one for IPv6 packets. one probe, two entry points, main character energy 👑

and it's not just probes — the SDT macros also let you define new DTrace providers, so you can namespace logically-related probes. example: FreeBSD's sctp provider, which holds SDT probes for FreeBSD's sctp(4) implementation.

SDT_PROVIDER_DECLARE() and SDT_PROVIDER_DEFINE() declare and define (respectively) a DTrace provider named prov with the SDT framework. a provider only needs to be defined once, but heads up: it must be declared before you define any SDT probes belonging to it. no skipping steps bestie.

same situation for SDT_PROBE_DECLARE() and SDT_PROBE_DEFINE*() — declare and define DTrace probes with the SDT framework. once a probe is defined, you can add trace points for it in kernel code. DTrace probe identifiers are made of a provider, module, function and name, all of which you can specify in the SDT probe definition.

⚠️ but bestie, do not specify a module name on probes. the module name is what the framework uses to figure out whether a probe should be destroyed when a kernel module is unloaded. see the BUGS section. and especially: probes must not be defined across multiple kernel modules. periodt.

if you want a - (dash) in a probe name, you have to write it as __ (double underscore) in the probe name parameter you pass to the various SDT_*() macros — because, technical reasons, a dash isn't valid in C identifiers. iykyk 💅

SDT_PROBE_DEFINE*() also lets you declare the types of the arguments passed to probes. this part is optional — if you skip the argument types (by using SDT_PROBE_DEFINE()), then anyone wanting to use the arguments has to manually cast them to the right types in their D scripts. the page says it's strongly recommended that probe definitions include a declaration of their argument types. so like. do that. 🫶

SDT_PROBE_DEFINE*_XLATE() is for probes whose argument types get dynamically translated into the types given by the matching xarg arguments. mostly this is for porting probe definitions from other operating systems. as dtrace(1) sees it, the arguments of a probe defined with these macros have types matching the xarg types in the probe definition. but the arguments you actually pass in at the trace point have types matching the native argument types in the probe definition — so the native type gets dynamically translated into the translated type. and as long as a suitable translator is defined in /usr/lib/dtrace, scripts using the probe don't have to care at all about the underlying type of a given SDT probe argument. it just works. ✨

SDT_PROBE*() creates the actual SDT trace points. these are meant to be added to executable code, and they're what you use to instrument the code they're called in.

---

💜 PROVIDERS 💜

there are a number of kernel DTrace providers available. generally speaking, these providers define stable interfaces and should be treated as such: existing D scripts can break if a probe gets renamed or its arguments get modified. 😭

that said, it's often useful to define ad-hoc SDT probes for debugging a subsystem or driver. and similarly, a developer might want to ship a group of SDT probes without committing to their future stability. those probes should go in the sdt provider instead of defining a whole new provider. no pressure, no promises 💅

---

🌈 EXAMPLES 🌈

want to see the DTrace providers available on the current system? run this:

dtrace -l | sed 1d | awk '{print $2}' | sort -u

want a detailed list of the probes a given provider offers? specify the provider with the -P flag. for example, to see the probes and argument types for the sched provider:

dtrace -lv -P sched

okay now the fun part. this probe definition creates a DTrace probe called icmp:::receive-unreachable, which would (hypothetically) fire when the kernel receives an ICMP packet of type Destination Unreachable:

SDT_PROVIDER_DECLARE(icmp);

SDT_PROBE_DEFINE1(icmp, , , receive__unreachable,
    "struct icmp *");

this particular probe takes a single argument: a pointer to the struct containing the ICMP header for the packet. and note — the module name of this probe is not specified. (see that receive__unreachable? that's the dash-to-double-underscore thing in action 🎀)

next: imagine a DTrace probe that fires when the network stack receives an IP packet. that probe would be defined by multiple tracepoints:

SDT_PROBE_DEFINE3(ip, , , receive, "struct ifnet *",
    "struct ip *", "struct ip6_hdr *");

int
ip_input(struct mbuf *m)
{
        struct ip *ip;
        ...
        ip = mtod(m, struct ip *);
        SDT_PROBE3(ip, , , receive, m->m_pkthdr.rcvif, ip, NULL);
        ...
}

int
ip6_input(struct mbuf *m)
{
        struct ip6_hdr *ip6;
        ...
        ip6 = mtod(m, struct ip6_hdr *);
        SDT_PROBE3(ip, , , receive, m->m_pkthdr.rcvif, NULL, ip6);
        ...
}

so the probe fires when the kernel receives either an IPv4 packet or an IPv6 packet. one probe, both paths, we love to see it 🌸

now back to that ICMP probe. its second argument is of type struct icmp, which is a type defined in the FreeBSD kernel to represent the ICMP header of an ICMP packet, defined in RFC 792. Linux has a corresponding type, struct icmphdr, for the same purpose — but its field names are different from FreeBSD's struct icmp. and illumos defines icmph_t, again with different field names. so even with the icmp:::pkt-receive probes defined in all three operating systems, you'd still have to write OS-specific scripts to pull a given field out of the ICMP header argument. ugh.

dynamically-translated types fix this: define an OS-independent c(7) struct to represent an ICMP header — say, struct icmp_hdr_dt — and define translators from each of the three OS-specific types into struct icmp_hdr_dt, all in the dtrace(1) library path. then that FreeBSD probe above can be defined like this:

SDT_PROBE_DEFINE1_XLATE(ip, , , receive, "struct icmp *",
    "struct icmp_hdr_dt *");

---

⭐ SEE ALSO ⭐

dtrace(1), dtrace_io(4), dtrace_ip(4), dtrace_proc(4), dtrace_sched(4), dtrace_tcp(4), dtrace_udp(4)

go read those too bestie, they're the whole friend group 🫶

---

👑 AUTHORS 👑

DTrace and the SDT framework were originally ported to FreeBSD from Solaris by John Birrell <jb@FreeBSD.org>. and this manual page was written by Mark Johnston <markj@FreeBSD.org>. absolute legends, we thank them for their service 💖

---

💀 BUGS 💀

the SDT macros let you specify the module and function names of a probe as part of a probe definition. and here's the thing: the DTrace framework uses the module name of probes to figure out which probes should be destroyed when a kernel module gets unloaded. so the module name of a probe should match the name of the module it's defined in. if you leave the module name unspecified in the probe definition, SDT will set it properly for you — see the EXAMPLES section. don't fight it.

⚠️ second one, and it's a little sad: one of the goals of the original SDT implementation (and, by extension, of FreeBSD's port) was that inactive SDT probes should have no performance impact. that is unfortunately not the case — SDT trace points add a small but non-zero amount of latency to the code they're defined in. a more sophisticated implementation of the probes will help alleviate this problem. so like, someday. 🙈

---

and that's the whole page bestie!! this was the SDT(9) manual page from the Kernel Developer's Manual, and if you were wondering where it came from — Debian, dated April 18, 2015. 💅✨ go forth and instrument your kernel, you absolute icon 🎀🦋💖