DTRACE_FBT(4)

💅 DTRACE_FBT(4) — the dtrace_fbt page, but make it make sense

okay so this is dtrace_fbt from the Kernel Interfaces Manual (section 4, kernel interfaces, you know the vibe) — it's the DTrace provider for dynamic kernel tracing that works off function boundaries. bestie, let's get into it 🦋

---

✨ NAME ✨

dtrace_fbt — a DTrace provider for dynamic kernel tracing based on function boundaries. literally what it says on the tin, no cap 🎀

✨ SYNOPSIS ✨

fbt:module:function:entry
fbt:module:function:return

🌸 DESCRIPTION 🌸

The Function Boundary Tracing (fbt) provider instruments the entry and return of almost every kernel function corresponding to an elf(5) symbol in the kernel and loaded kernel modules. almost — we'll get to the exceptions, don't worry 💀

• fbt:module:function:entry fires whenever the function is called.
• fbt:module:function:return fires when the function returns.

The module in the probe description is either the name of the loaded kernel module or ‘kernel’ for functions compiled into the kernel. so if it's baked into the kernel itself, you just say kernel and move on ✨

🧚 Function Boundary Instrumentation

fbt will always instrument a function's entry, but its return will be instrumented so long as it can find a ‘ret’ instruction. entry? always. return? only if there's a ret to latch onto, iykyk.

In some cases, fbt cannot instrument a function's entry and/or return. Refer to subsection “Frame Pointer” for more details. (spoiler: it's the inline function situation 🙈)

🫶 Probe Arguments

The arguments of the entry probe (fbt:module:function:entry) are the arguments of the traced function call. so they're typed, and they're the actual args, which is honestly so slay:

• args[0] — Function's first argument, typed (e.g., malloc(9)'s size_t size)
• args[1] — Function's second argument, typed (e.g., malloc(9)'s struct malloc_type *type)
• args[2] — Function's third argument, typed (e.g., malloc(9)'s int flags)
• ... — ...

The arguments of the return probe (fbt:module:function:return) are args[0] (the offset of the firing return instruction within the function; useful to tell apart two different return statements in a single function) and args[1] (the return value, if any). two args, two jobs, iconic 👑

• args[0] — Offset of the traced return instruction
• args[1] — Function's return value (e.g., a kernel virtual address if returning from a successful malloc(9))

Subsection “Example 2: Getting Details About Probe's Arguments” shows how to get probe's argument count and types directly with dtrace(1) without having to resort to the reading function's source code or documentation. no source-diving required, we love a shortcut that isn't a crime 🍓

---

🔥 EXAMPLES 🔥

🎀 Example 1: Listing Available FBT Probes

The following example shows how to list all the available fbt probes.

# dtrace -l -P fbt
   ID   PROVIDER     MODULE              FUNCTION NAME
[...]
31868        fbt     kernel           hammer_time entry
31869        fbt     kernel           hammer_time return
[...]

Since hammer_time() is a part of the kernel and not a separate loaded module, the module column displays ‘kernel’. see? told you 💅

🎀 Example 2: Getting Details About Probe's Arguments

The following example shows how to generate a program stability report of malloc(9)'s entry and return probes. Those reports are useful to view the probe's number of arguments and their types. genuinely so useful, ngl.

# dtrace -l -v -n fbt::malloc:entry
[...]
        Argument Types
                args[0]: size_t
                args[1]: struct malloc_type *
                args[2]: int

The count and types of fbt::malloc:entry arguments match the function signature of malloc(9): args[0] is size_t, args[1] is struct malloc_type *, and args[2] is int. like, exactly, fr fr.

# dtrace -l -v -n fbt::malloc:return
[...]
        Argument Types
                args[0]: int
                args[1]: void *

The return probe reports two arguments and their types: the return instruction offset (the usual int) and the function's return value, which in this case is void *, as malloc(9) returns a kernel virtual address. two for two, ate 🌈

🎀 Example 3: Counting Kernel Slab Memory Allocation by Function

# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }'
dtrace: description 'fbt::kmem*:entry ' matched 47 probes
^C
  kmem_alloc_contig                                     1
  kmem_alloc_contig_domainset                           1
  kmem_cache_reap_active                                1
  kmem_alloc_contig_pages                               2
  kmem_free                                             2
  kmem_std_destructor                                  19
  kmem_std_constructor                                 26
  kmem_cache_free                                     151
  kmem_cache_alloc                                    181

🎀 Example 4: Counting Kernel Slab Memory Allocation by Calling Function

# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'
^C
                kernel`contigmalloc+0x33                1
                        kernel`free+0xd3                1
           kernel`kmem_alloc_contig+0x29                1
kernel`kmem_alloc_contig_domainset+0x19a                1
           zfs.ko`arc_reap_cb_check+0x16                1

🎀 Example 5: Counting Kernel malloc()'s by Calling Function

# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\n", @); }'
^C
        kernel`devclass_get_devices+0xa8                1
                   kernel`sys_ioctl+0xb7                1
           dtrace.ko`dtrace_ioctl+0x15c1                1
            dtrace.ko`dtrace_ioctl+0x972                2
        dtrace.ko`dtrace_dof_create+0x35                2
             kernel`kern_poll_kfds+0x2f0                4
             kernel`kern_poll_kfds+0x28a               19

🎀 Example 6: Counting Kernel malloc()'s by Kernel Stack Trace

# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }'
^C
              dtrace.ko`dtrace_dof_create+0x35
              dtrace.ko`dtrace_ioctl+0x827
              kernel`devfs_ioctl+0xd1
              kernel`VOP_IOCTL_APV+0x2a
              kernel`vn_ioctl+0xb6
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x286
              kernel`sys_ioctl+0x12f
              kernel`amd64_syscall+0x169
              kernel`0xffffffff81092b0b
                2

🎀 Example 7: Summarizing vmem_alloc()'s by Arena Name and Size Distribution

# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }'
^C

  kernel arena dom
           value  ------------- Distribution ------------- count
            2048 |                                         0
            4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@              4
            8192 |@@@@@@@@@@@@@                            2
           16384 |                                         0

🎀 Example 8: Measuring Total Time Spent Executing a Function

This DTrace script measures the total time spent in vm_page*() kernel functions. The quantize() aggregation organizes the measurements into power-of-two buckets, providing a time distribution in nanoseconds for each function. yes bestie, nanoseconds, we're measuring at the speed of light here 💜

fbt::vm_page*:entry {
    self->start = timestamp;
}

fbt::vm_page*:return /self->start/ {
    @[probefunc] = quantize(timestamp - self->start);
    self->start = 0;
}

---

🌈 SEE ALSO 🌈

• dtrace(1)
• dtrace_kinst(4)
• tracing(7)

Brendan Gregg and Jim Mauro, DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD, Prentice Hall, https://www.brendangregg.com/dtracebook/, pp. 898–903, 2011.

The illumos Dynamic Tracing Guide, https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt, 2008, Chapter fbt Provider.

🦋 AUTHORS 🦋

This manual page was written by Mateusz Piotrowski <0mp@FreeBSD.org>. thank you king 🙏

💀 CAVEATS 💀

⚠️ Stability and Portability

fbt probes are by definition tightly coupled to kernel code; if the code underlying a script changes, the script may fail to run or may produce incorrect results. Scripts written for one version of FreeBSD might not work on others, and almost certainly will not work on other operating systems. like, this is not portable code, this is vibes-based-on-this-exact-kernel code 😭

Individual fbt probes often do not correspond nicely to logical system events. For example, consider a DTrace script which prints the destination address of every IP packet as the kernel hands them over to the network card driver (NIC). An fbt-based implementation of such a script is a discouragingly difficult task: it involves instrumenting at least four different functions in different parts of the IPv4 and IPv6 code. At the same time, with the dtrace_ip(4) provider the script is a simple one-liner:

dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}'

Make sure to review available dtrace(1) providers first before implementing a custom script with the fbt provider. If none of the DTrace providers offer the desired probes, consider adding new statically-defined tracing probes (SDT(9)). check your options first, bestie, don't hand-roll a whole thing for no reason 🧚

⚠️ Frame Pointer

Inline functions are not instrumentable by fbt as they lack a frame pointer. A developer might explicitly disable inlining by adding the ‘__noinline’ attribute to a function definition, but of course this requires a recompilation of the kernel. Building the kernel with -fno-omit-frame-pointer is another way of preserving frame pointers. Note, that sometimes compilers will omit the frame pointer in leaf functions, even when configured with -fno-omit-frame-pointer. so even when you do the thing, the compiler might still do its own thing. rude, honestly 💀

Function returns via a tail call are also not instrumentable by fbt. As a result, a function might have an entry probe and a mix of instrumented and uninstrumentable returns. entry probe ✅, returns: some yes, some no, good luck 🙂

Use dtrace_kinst(4) to trace arbitrary instructions inside kernel functions and work around some of the limitations of fbt. that's the workaround, go use it ✨

⚠️ Tracing DTrace

The fbt provider cannot attach to functions inside DTrace provider kernel modules. no, it cannot spy on itself, that's the rule 🎀

---

and that's the whole page, bestie 💖 this was DTRACE_FBT(4) from the Kernel Interfaces Manual, Debian build, dated July 16, 2025 — go forth and instrument responsibly, mwah 🫶