GDB(4)

📦 version 12.2, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

okay bestie, sit down, hydrate, because this one is a whole saga 💖 this is gdb(4), straight out of the Kernel Interfaces Manual — the external kernel debugger. It's long, it's got firewire in it, and by the end you're gonna feel so powerful. Let's go. ✨

💅 NAME 💅

gdb — external kernel debugger 👑

🎀 SYNOPSIS 🎀

makeoptions DEBUG=-g
options DDB

✨ DESCRIPTION ✨

The gdb kernel debugger is a whole variation of gdb(1) that actually Gets It — it understands a bunch of stuff about the FreeBSD kernel environment. And the way you can use it? Several. Multiple. A whole menu, bestie:

  • You can use it to peek at the memory of the processor it's literally running on. 👀
  • You can use it to analyse a processor dump after a panic. (the drama of it all)
  • You can use it to debug another system interactively over a serial or firewire link. In this mode the processor can be stopped and single stepped. main character energy fr.
  • And with a firewire link? You can examine the memory of a remote system without that system's participation. Like, without its consent. In this mode you cannot stop the processor or single step it, but honestly it's a lifesaver when the remote system has crashed and is just… not responding anymore. 💀
Heads up: when you're doing remote debugging, gdb requires the ddb(4) kernel debugger to be there. There are commands that let you switch between gdb and ddb(4), so you're not stuck, don't worry. 🫶

🌸 PREPARING FOR DEBUGGING 🌸

Okay so, when you're debugging kernels it is practically essential to have built a kernel with debugging symbols, aka makeoptions DEBUG=-g. No symbols, no vibes, no peace. 🧚

It's easiest to do all your operations from the kernel build directory — by default that's /usr/obj/usr/src/sys/GENERIC.

First thing first: make sure you have a copy of the debug macros in the directory:

make gdbinit

This command does some transformations on the macros installed in /usr/src/tools/debugscripts to adapt them to your local environment. Slay, now they fit you. 💅

🔍 Inspecting the environment of the local machine

To look at (and change) the contents of the memory of the system you're running on:

gdb -k -wcore kernel.debug /dev/mem

In this mode you need the -k flag, because that's how you tell gdb(1) that the "dump file" /dev/mem is a kernel data file. You can look at live data, and if you include the -wcore option you can change it — at your peril, bestie, I said what I said. 😭

The system does not stop (obviously). So a bunch of stuff will just not work. You can set breakpoints, but you cannot "continue" execution, so those breakpoints will not work. It's giving decorative. 🎀

💥 Debugging a crash dump

By default, crash dumps get stored in the directory /var/crash. Investigate them from the kernel build directory with:

gdb -k kernel.debug /var/crash/vmcore.29

In this mode the system is obviously stopped, so you can only look at it. Museum rules. 🖼️

Real quick, the vocabulary so nobody gets confused:

  • "local system" = the system running the debugger
  • "remote system" = the live system being debugged

To debug a live system with a remote link, the kernel must be compiled with the option options DDB. And the option options BREAK_TO_DEBUGGER is what lets the debugging machine stop the debugged machine once a connection has been established — you do that by pressing ‘^C’. Iconic. ⭐

When you're using a serial port for the remote link on the i386 platform, that serial port has to be identified by setting flag bit 0x80 for the specified interface. Generally this port will also be used as a serial console (that's flag bit 0x10), so the entry in /boot/device.hints should look like this:

hint.sio.0.flags="0x90"

Same as serial debugging — to debug a live system with a firewire link, the kernel must be compiled with the option options DDB. No skipping this step, bestie. 🫶

A number of steps must be performed to set up a firewire link:

  • Ensure that both systems have firewire(4) support, and that the kernel of the remote system includes the dcons(4) and dcons_crom(4) drivers. If they are not compiled into the kernel, load the KLDs:
kldload firewire

On the remote system only:

kldload dcons
kldload dcons_crom

You should see something like this in the dmesg(8) output of the remote system:

fwohci0: BUS reset
fwohci0: node_id=0x8800ffc0, gen=2, non CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1
firewire0: bus manager 1
firewire0: New S400 device ID:00c04f3226e88061
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x22a000

It is a good idea to load these modules at boot time with the following entry in /boot/loader.conf:

dcons_crom_enable="YES"

This ensures that all three modules are loaded. There is no harm in loading dcons(4) and dcons_crom(4) on the local system, but if you only want to load the firewire(4) module, include the following in /boot/loader.conf:

firewire_enable="YES"
  • Next, use fwcontrol(8) to find the firewire node corresponding to the remote machine. On the local machine you might see:
# fwcontrol
2 devices (info_len=2)
node        EUI64        status
   1  0x00c04f3226e88061      0
   0  0x000199000003622b      1

The first node is always the local system, so in this case node 0 is the remote system. If there are more than two systems, check from the other end to find which node corresponds to the remote system. On the remote machine, it looks like this:

# fwcontrol
2 devices (info_len=2)
node        EUI64        status
   0  0x000199000003622b      0
   1  0x00c04f3226e88061      1
dconschat -br -G 5556 -t 0x000199000003622b

0x000199000003622b is the EUI64 address of the remote node, as determined from the output of fwcontrol(8) above. When started in this manner, dconschat(8) establishes a local tunnel connection from port localhost:5556 to the remote debugger. You can also establish a console port connection with the -C option to the same invocation dconschat(8). See the dconschat(8) manpage for further details.

The dconschat(8) utility does not return control to the user. It displays error messages and console output for the remote system, so it is a good idea to start it in its own window. 🪟
  • Finally, establish connection:
# gdb kernel.debug
GNU gdb 5.2.1 (FreeBSD)
(political statements omitted)
Ready to go.  Enter 'tr' to connect to the remote target
with /dev/cuau0, 'tr /dev/cuau1' to connect to a different port
or 'trf portno' to connect to the remote target with the firewire
interface.  portno defaults to 5556.

Type 'getsyms' after connection to load kld symbols.

If you are debugging a local system, you can use 'kldsyms' instead
to load the kld symbols.  That is a less obnoxious interface.
(gdb) trf
0xc21bd378 in ?? ()

The trf macro assumes a connection on port 5556. If you want to use a different port (by changing the invocation of dconschat(8) above), use the tr macro instead. For example, if you want to use port 4711, run dconschat(8) like this:

dconschat -br -G 4711 -t 0x000199000003622b

Then establish connection with:

(gdb) tr localhost:4711
0xc21bd378 in ?? ()

Okay this one is lowkey the coolest part. In addition to the conventional debugging via firewire described in the previous section, it is possible to debug a remote system without its cooperation, once an initial connection has been established. This corresponds to debugging a local machine using /dev/mem.

It can be very useful if a system crashes and the debugger no longer responds. Rude of it, honestly. To use this method, set the sysctl(8) variables hw.firewire.fwmem.eui64_hi and hw.firewire.fwmem.eui64_lo to the upper and lower halves of the EUI64 ID of the remote system, respectively.

From the previous example, the remote machine shows:

# fwcontrol
2 devices (info_len=2)
node        EUI64        status
   0  0x000199000003622b      0
   1  0x00c04f3226e88061      1

Enter:

# sysctl -w hw.firewire.fwmem.eui64_hi=0x00019900
hw.firewire.fwmem.eui64_hi: 0 -> 104704
# sysctl -w hw.firewire.fwmem.eui64_lo=0x0003622b
hw.firewire.fwmem.eui64_lo: 0 -> 221739
Note that the variables must be explicitly stated in hexadecimal. No decimal shortcuts, bestie, it will not go well. 🙈

After this, you can examine the remote machine's state with the following input:

# gdb -k kernel.debug /dev/fwmem0.0
GNU gdb 5.2.1 (FreeBSD)
(messages omitted)
Reading symbols from /boot/kernel/dcons.ko...done.
Loaded symbols for /boot/kernel/dcons.ko
Reading symbols from /boot/kernel/dcons_crom.ko...done.
Loaded symbols for /boot/kernel/dcons_crom.ko
#0  sched_switch (td=0xc0922fe0) at /usr/src/sys/kern/sched_4bsd.c:621
0xc21bd378 in ?? ()

In this case, it is not necessary to load the symbols explicitly. And the remote system just keeps on running. No cap. 🌈

💖 COMMANDS 💖

The whole user interface to gdb runs through gdb(1), so literally every gdb(1) command works in here too, bestie. This section is only about the kernel-debugging extensions that get installed into your kernel build directory. It's giving bonus DLC energy. ✨

🛠️ Debugging environment

These macros are the ones that mess with your debugging environment:

  • ddb — flips you back to ddb(4). Only actually does anything when you're doing remote debugging, so don't come for it otherwise. 💅
  • getsyms — shows you the kldstat info for the target machine and then politely asks you to paste it back in. Why, you ask? Because gdb is not about letting data get passed to shell scripts, that's why. You need this for remote debugging and crash dumps. For local memory debugging, use kldsyms instead, periodt.
  • kldsyms — reads in the symbol tables for the debugging machine. Lowkey does not work for remote debugging or crash dumps, so use getsyms for those instead. 🙈
  • tr interface — debug a remote system through the serial or firewire interface you name.
  • tr0 — debug a remote system via the serial interface /dev/cuau0.
  • tr1 — debug a remote system via the serial interface /dev/cuau1.
  • trf — debug a remote system via the firewire interface at the default port 5556.

And listen: tr0, tr1 and trf are just convenience commands that invoke tr. They're the fast-pass line, not a different ride. 🎢

🫶 The current process environment

These macros are little convenience functions meant to make your life easier than the standard gdb(1) commands would. It's giving "I got you, bestie." 💖

  • f0 — select stack frame 0 and show assembler-level details.
  • f1 — select stack frame 1 and show assembler-level details.
  • f2 — select stack frame 2 and show assembler-level details.
  • f3 — select stack frame 3 and show assembler-level details.
  • f4 — select stack frame 4 and show assembler-level details.
  • f5 — select stack frame 5 and show assembler-level details.
  • xb — show 12 words in hex, starting at the current ebp value.
  • xi — list the next 10 instructions from the current eip value.
  • xp — show the register contents and the first four parameters of the current stack frame.
  • xp0 — show the first parameter of the current stack frame in various formats.
  • xp1 — show the second parameter of the current stack frame in various formats.
  • xp2 — show the third parameter of the current stack frame in various formats.
  • xp3 — show the fourth parameter of the current stack frame in various formats.
  • xp4 — show the fifth parameter of the current stack frame in various formats.
  • xs — show the last 12 words on the stack in hexadecimal.
  • xxp — show the register contents and the first ten parameters.
  • z — single step 1 instruction (over calls) and show the next instruction.
  • zs — single step 1 instruction (through calls) and show the next instruction.

👀 Examining other processes

These macros reach into other processes. Here's the tea: gdb does not understand the concept of multiple processes at all — it's got zero comprehension, it's not even trying — so these effectively bypass the entire gdb environment. Iconic workaround honestly. 🦋

  • btp pid — show a backtrace for the process pid.
  • btpa — show backtraces for all processes in the system.
  • btpp — show a backtrace for the process previously selected with defproc.
  • btr ebp — show a backtrace from the ebp address you specified.
  • defproc pid — specify the PID of the process, for use by some of the other commands in this section.
  • fr frame — show frame frame of the stack of the process previously selected with defproc.
  • pcb proc — show some PCB contents of the process proc.

🧐 Examining data structures

You can use the standard gdb(1) commands to look at most data structures — that's totally fine, no notes. The macros in this section are convenience functions that typically display the data in a more readable format, or that just omit the less interesting parts of the structure. Curated content, basically. ✨

  • bp — show info about the buffer header pointed to by the variable bp in the current frame.
  • bpd — show the contents (char *) of bp->data in the current frame.
  • bpl — show detailed information about the buffer header (struct bp) pointed at by the local variable bp.
  • bpp bp — show summary information about the buffer header (struct bp) pointed at by the parameter bp.
  • bx — print a number of fields from the buffer header pointed at by the pointer bp in the current environment.
  • vdev — show some information about the vnode pointed to by the local variable vp.

🎀 Miscellaneous macros

  • checkmem — check unallocated memory for modifications. This assumes the kernel was compiled with options DIAGNOSTIC. It causes the contents of free memory to be set to 0xdeadc0de. (Cute constant. No further comment. 💀)
  • dmesg — print the system message buffer. This corresponds to the dmesg(8) utility. This macro used to be called msgbuf, for the lore keepers. Fair warning though: it can take a very long time over a serial line, and it's even slower via firewire or local memory because of inefficiencies in gdb. When you're debugging a crash dump or working over firewire, you do not need to start gdb to access the message buffer — instead, use an appropriate variation of:
dmesg -M /var/crash/vmcore.0 -N kernel.debug
dmesg -M /dev/fwmem0.0 -N kernel.debug
  • kldstat — equivalent of the kldstat(8) utility, without options.
  • pname — print the command name of the current process.
  • ps — show process status. This corresponds in concept, but not in appearance, to the ps(1) utility. When you're debugging a crash dump or working over firewire, you don't have to start gdb to display the ps(1) output — instead, use an appropriate variation of:
ps -M /var/crash/vmcore.0 -N kernel.debug
ps -M /dev/fwmem0.0 -N kernel.debug
  • y — a kludge for writing macros. When you're writing macros, it's convenient to paste them back into the gdb window. Unfortunately, if the macro is already defined, gdb insists on asking
Redefine foo?

and it will not give up until you answer ‘y’. This command is that answer. It does nothing else except print a warning message to remind you to remove it again. Extremely "just here to say yes" behavior. 🙈

🌈 SEE ALSO 🌈

gdb(1), ps(1), ddb(4), firewire(4), dconschat(8), dmesg(8), fwcontrol(8), kldload(8)

👑 AUTHORS 👑

This man page was written by Greg Lehey <grog@FreeBSD.org>. Respect the legend. 🫶

💀 BUGS 💀

Okay so, the gdb(1) debugger was never designed to debug kernels, and honestly it is not a very good match. Many problems exist. Many. 😭

The gdb implementation is very inefficient, and many operations are slow. Not slow-cute. Just slow.

Serial debugging is even slower, and race conditions can make it difficult to run the link at more than 9600 bps. Firewire connections do not have this problem, so that's the one slaying. 🔥

The debugging macros “just grew.” In general, the person who wrote them did so while looking for a specific problem, so they may not be general enough, and they may behave badly when used in ways for which they were not intended — even if those ways make sense. It's giving "wrote it at 3am for one specific bug" energy, and no cap, that's exactly what happened.

Many of these commands only work on the ia32 architecture.

That's the whole page, bestie! 💅 This one comes from <strong>Debian</strong>, dated <strong>May 17, 2016</strong> — so yeah, it's a lil vintage, but it still ate. Go debug something and text me. 🦋💖

📦 where this page comes from