DDB(4)

๐Ÿ“ฆ
version 15.1 ยท 5 versions
, in freebsd 15.1

๐Ÿ’– ddb(4) โ€” interactive kernel debugger ๐Ÿ’–

โœจ NAME โœจ

okay bestie, ddb is the interactive kernel debugger โ€” yes, the actual โ€œtalk to the kernel while itโ€™s freaking outโ€ girlie ๐Ÿ’….

๐ŸŒธ SYNOPSIS ๐ŸŒธ

To enable kernel debugging facilities, include:

options KDB
options DDB

To prevent activation of the debugger on kernel panic(9):

options KDB_UNATTENDED

To print a stack trace of the current thread on the console for a panic:

options KDB_TRACE

To print the numerical value of symbols in addition to the symbolic representation, define:

options DDB_NUMSYM

To enable the gdb(4) backend, so that remote debugging with kgdb(1) (ports/devel/gdb) is possible, include:

options GDB

๐Ÿฆ‹ DESCRIPTION ๐Ÿฆ‹

The ddb kernel debugger is an interactive debugger with syntax inspired by gdb(1) (ports/devel/gdb). If itโ€™s linked into the running kernel, you can invoke it locally with the debug keymap(5) action, usually mapped to Ctrl+Alt+Esc, or by setting the debug.kdb.enter sysctl to 1. The debugger is also invoked on kernel panic(9) if the debug.debugger_on_panic sysctl(8) MIB variable is set non-zero โ€” which is the default unless the KDB_UNATTENDED option is specified. Similarly, if debug.debugger_on_recursive_panic is set to 1, then the debugger will be invoked on a recursive kernel panic. This variable has a default value of 0, and has no effect if debug.debugger_on_panic is already set non-zero.

The current location is called dot. The dot is displayed with a hexadecimal format at a prompt. The commands examine and write update dot to the address of the last line examined or the last location modified, and set next to the address of the next location to be examined or changed. Other commands do not change dot, and set next to be the same as dot.

The general command syntax is:

command[/modifier] [addr][,count]

A blank line repeats the previous command from the address next with count 1 and no modifiers. Specifying addr sets dot to the address. Omitting addr uses dot. A missing count is taken to be 1 for printing commands or infinity for stack traces. A count of -1 is equivalent to a missing count. Options that are supplied but not supported by the given command are usually ignored.

The ddb debugger has a pager feature (like the more(1) command) for the output. If an output line exceeds the number set in the lines variable, it displays โ€œ--More--โ€ and waits for a response. The valid responses for it are:

  • SPC โ€” one more page
  • RET โ€” one more line
  • q โ€” abort the current command, and return to the command input mode

Finally, ddb provides a small (currently 10 items) command history, and offers simple emacs-style command line editing capabilities. In addition to the emacs control keys, the usual ANSI arrow keys may be used to browse through the history buffer, and move the cursor within the current line.

๐Ÿ”ฅ COMMANDS ๐Ÿ”ฅ

๐Ÿ’ฌ COMMON DEBUGGER COMMANDS ๐Ÿ’ฌ

  • help โ€” print a short summary of the available commands and command abbreviations.
  • examine[/AISabcdghilmorsuxz ...] [addr][,count] / x[/AISabcdghilmorsuxz ...] [addr][,count] โ€” display the addressed locations according to the formats in the modifier. Multiple modifier formats display multiple locations. If no format is specified, the last format specified for this command is used.

The format characters are:

  • b โ€” look at by bytes (8 bits)
  • h โ€” look at by half words (16 bits)
  • l โ€” look at by long words (32 bits)
  • g โ€” look at by quad words (64 bits)
  • a โ€” print the location being displayed
  • A โ€” print the location with a line number if possible
  • x โ€” display in unsigned hex
  • z โ€” display in signed hex
  • o โ€” display in unsigned octal
  • d โ€” display in signed decimal
  • u โ€” display in unsigned decimal
  • r โ€” display in current radix, signed
  • c โ€” display low 8 bits as a character. Non-printing characters are displayed as an octal escape code (e.g., \000).
  • s โ€” display the null-terminated string at the location. Non-printing characters are displayed as octal escapes.
  • m โ€” display in unsigned hex with character dump at the end of each line. The location is also displayed in hex at the beginning of each line.
  • i โ€” display as a disassembled instruction
  • I โ€” display as a disassembled instruction with possible alternate formats depending on the machine. On i386, this selects the alternate format for the instruction decoding (16 bits in a 32-bit code segment and vice versa).
  • S โ€” display a symbol name for the pointer stored at the address
  • xf โ€” Examine forward: execute an examine command with the last specified parameters to it except that the next address displayed by it is used as the start address.
  • xb โ€” Examine backward: execute an examine command with the last specified parameters to it except that the last start address subtracted by the size displayed by it is used as the start address.
  • print[/acdoruxz] / p[/acdoruxz] โ€” print addrs according to the modifier character (as described above for examine). Valid formats are: a, x, z, o, d, u, r, and c. If no modifier is specified, the last one specified to it is used. The argument addr can be a string, in which case it is printed as it is. For example:
print/x "eax = " $eax "\necx = " $ecx "\n"

will print like:

eax = xxxxxx
ecx = yyyyyy
  • pprint[/d depth] [name] โ€” pretty-print symbol specified by name using CTF debugging data. Works for all symbols exported by the kernel and loaded kernel modules.

If the d modifier has been specified, contents of structs nested up to depth levels deep will also be included in the output.

  • pprint struct[/d depth ] [name][addr] โ€” print memory at addr as struct name. Works for all structs defined by the kernel and loaded kernel modules.

If the d modifier has been specified, contents of structs nested up to depth levels deep will also be included in the output.

  • write[/bhl] addr expr1 [expr2 ...] / w[/bhl] addr expr1 [expr2 ...] โ€” write the expressions specified after addr on the command line at succeeding locations starting with addr. The write unit size can be specified in the modifier with a letter b (byte), h (half word) or l (long word) respectively. If omitted, long word is assumed.
โš ๏ธ Warning: since there is no delimiter between expressions, strange things may happen. It is best to enclose each expression in parentheses.
  • set $variable [=] expr โ€” set the named variable or register with the value of expr. Valid variable names are described below.
  • break[/u] [addr][,count] / b[/u] [addr][,count] โ€” set a break point at addr. If count is supplied, the continue command will not stop at this break point on the first count - 1 times that it is hit. If the break point is set, a break point number is printed with #. This number can be used in deleting the break point or adding conditions to it.

If the u modifier is specified, this command sets a break point in user address space. Without the u option, the address is considered to be in the kernel space, and a wrong space address is rejected with an error message. This modifier can be used only if it is supported by machine dependent routines.

โš ๏ธ Warning: If a user text is shadowed by a normal user space debugger, user space break points may not work correctly. Setting a break point at the low-level code paths may also cause strange behavior.
  • delete [addr] / d [addr] / delete #number / d #number โ€” delete the specified break point. The break point can be specified by a break point number with #, or by using the same addr specified in the original break command, or by omitting addr to get the default address of dot.
  • halt โ€” halt the system.
  • watch [addr][,size] โ€” set a watchpoint for a region. Execution stops when an attempt to modify the region occurs. The size argument defaults to 4. If you specify a wrong space address, the request is rejected with an error message.
โš ๏ธ Warning: Attempts to watch wired kernel memory may cause unrecoverable error in some systems such as i386. Watchpoints on user addresses work best.
  • hwatch [addr][,size] โ€” set a hardware watchpoint for a region if supported by the architecture. Execution stops when an attempt to modify the region occurs. The size argument defaults to 4.
โš ๏ธ Warning: The hardware debug facilities do not have a concept of separate address spaces like the watch command does. Use hwatch for setting watchpoints on kernel address locations only, and avoid its use on user mode address spaces.
  • dhwatch [addr][,size] โ€” delete specified hardware watchpoint.
  • kill sig pid โ€” send signal sig to process pid. The signal is acted on upon returning from the debugger. This command can be used to kill a process causing resource contention in the case of a hung system. See signal(3) for a list of signals. Note that the arguments are reversed relative to kill(2).
  • step[/p][,count] / s[/p][,count] โ€” single step count times. If the p modifier is specified, print each instruction at each step. Otherwise, only print the last instruction.
โš ๏ธ Warning: depending on machine type, it may not be possible to single-step through some low-level code paths or user space code. On machines with software-emulated single-stepping (e.g., pmax), stepping through code executed by interrupt handlers will probably do the wrong thing.
  • continue[/c] / c[/c] โ€” continue execution until a breakpoint or watchpoint. If the c modifier is specified, count instructions while executing. Some machines (e.g., pmax) also count loads and stores.
โš ๏ธ Warning: when counting, the debugger is really silently single-stepping. This means that single-stepping on low-level code may cause strange behavior.
  • until[/p] โ€” keeps going until it hits the next call or return instruction, then stops. Add the p modifier and it'll spill the call nesting depth + the cumulative instruction count at every call or return. Without /p, it only prints when the matching return is hit. ngl the p version is way more noise but lowkey more informative ๐Ÿ’…
  • next[/p] โ€” same energy as until but the actual name for it. Stops at the matching return instruction. With the p modifier you get the call nesting depth and the cumulative instruction count printed at each call or return; without it, it only prints when the matching return is hit. โœจ
  • match[/p] โ€” literally the same thing as next[/p] bestie, and yes it also stops at the matching return instruction, and yes the p modifier also prints the call nesting depth and the cumulative instruction count at each call or return, and without p it also only prints when the matching return is hit. ๐Ÿซถ
  • trace[/u] [pid | tid][,count], t[/u] [pid | tid][,count], where[/u] [pid | tid][,count], bt[/u] [pid | tid][,count] โ€” stack trace, the icon of debugging ๐Ÿ”ฅ. The u option traces user space; if you leave it off, trace only traces kernel space. That optional count argument is how many frames get traced โ€” omit count and literally all frames get printed (yes, all of them, buckle up).
โš ๏ธ Warning bestie: user space stack traces are only valid if the machine dependent code supports it. no cap, check first.
  • search[/bhl] addr value [mask][,count] โ€” searches memory for value. That optional count argument caps how far the search goes. ๐Ÿ•ต๏ธโ€โ™€๏ธ
  • reboot[/s] [seconds], reset[/s] [seconds] โ€” hard reset the system, bye ๐Ÿ‘‹. If you pass the optional seconds argument, the debugger waits that long โ€” at most a week โ€” before rebooting. And when the s modifier is given, the command straight up skips running any registered shutdown handlers and attempts the most basic reset. It's giving "no thoughts, just vibes" but for rebooting ๐Ÿ’€
  • thread addr | tid โ€” switches the debugger over to the thread with ID tid if the argument is a decimal number, or to address addr otherwise. โœจ
  • watchdog [exp] โ€” programs the watchdog(4) timer to fire in 2^exp seconds. No argument at all? The watchdog timer just gets disabled. ๐Ÿ•

๐ŸŽ€ SPECIALIZED HELPER COMMANDS ๐ŸŽ€

  • findstack addr โ€” prints the address of the thread whose kernel-mode stack contains addr, if any such thread exists. ๐Ÿ”
  • show active trace, acttrace โ€” shows a stack trace for every thread running on a CPU. ๐ŸŒ€
  • show all procs[/a], ps[/a] โ€” displays all process info. Heads up: it might just not show anything if the machine doesn't support it, or if the bottom of the target process's stack isn't in main memory at that moment ๐Ÿ˜ญ. The a modifier prints command line arguments for each process. ๐Ÿงพ
  • show all tcpcbs[/bil] โ€” same output as "show tcpcb", but for all the TCP control blocks in the system. The b modifier requests that BBLog entries get printed. Pass the i modifier and the corresponding IP control block shows up too. And the l modifier limits output to only those TCP control blocks which are locked. ๐ŸŒ
  • show all trace, alltrace โ€” a stack trace for every single thread in the system. All of them. ๐Ÿ’–
  • show all ttys โ€” shows every TTY in the system. The output is similar to pstat(8), except it also includes the address of the TTY structure. ๐Ÿ–ฅ๏ธ
  • show all vnets โ€” same output as "show vnet", but it lists all the virtualized network stacks within the system. ๐ŸŒˆ
  • show allchains โ€” the same info as "show lockchain", but for every thread in the system. ๐Ÿ”—
  • show alllocks โ€” shows all locks that are currently held. Only available if witness(4) is included in the kernel. ๐Ÿ”’
  • show allpcpu โ€” same as "show pcpu", but for every CPU present in the system. ๐Ÿ‘‘
  • show allrman โ€” shows resource management info, including interrupt request lines, DMA request lines, I/O ports, I/O memory addresses, and Resource IDs. ๐Ÿ—‚๏ธ
  • show apic โ€” dumps data about APIC IDT vector mappings. ๐Ÿ
  • show badstacks โ€” walks the witness(4) graph and prints any lock-order violations it finds. Only available if witness(4) is included in the kernel. ๐Ÿ’€
  • show breaks โ€” shows the breakpoints you set with the "break" command. ๐Ÿ“
  • show bio addr โ€” shows info about the bio structure struct bio present at addr. Peep the sys/bio.h header file and g_bio(9) for more details on the exact meaning of the structure fields. ๐Ÿงฌ
  • show buffer addr โ€” shows info about the buf structure struct buf present at addr. See the sys/buf.h header file for more details on the exact meaning of the structure fields. ๐Ÿ“ฆ
  • show callout addr โ€” shows info about the callout structure struct callout present at addr. โฐ
  • show cdev [addr] โ€” shows the internal devfs state of the cdev structure located at addr. No argument? Then it shows the list of all created cdevs instead, consisting of the devfs node name and the struct cdev address. ๐ŸŽ›๏ธ
  • show conifhk โ€” lists the hooks currently waiting for completion in run_interrupt_driven_config_hooks(). ๐Ÿช
  • show cpusets โ€” prints numbered root and assigned CPU affinity sets. See cpuset(2) for more details. ๐Ÿง 
  • show cyrixreg โ€” shows registers specific to the Cyrix processor. (vintage!! ๐Ÿบ)
  • show devmap โ€” prints the contents of the static device mapping table. Currently only available on the ARM architecture. ๐Ÿฆพ
  • show domain addr โ€” prints the protocol domain structure struct domain at address addr. See the sys/domain.h header file for more details on the exact meaning of the structure fields. ๐ŸŒ
  • show ffs [addr] โ€” shows brief info about the ffs mount at the address addr, if the argument is given. Otherwise it gives you the summary about each ffs mount. ๐Ÿ“
  • show file addr โ€” shows info about the file structure struct file present at address addr. ๐Ÿ“„
  • show files โ€” shows info about every file structure in the system. Every. Single. One. ๐Ÿ˜ฎโ€๐Ÿ’จ
  • show freepages โ€” shows the number of physical pages in each of the free lists. ๐Ÿงฎ
  • show geom [addr] โ€” no addr argument? It displays the entire GEOM topology. Give it addr and it displays details about that given GEOM object (class, geom, provider or consumer). ๐Ÿงฑ
  • show idt โ€” shows the IDT layout. First column = the IDT vector. Second column = the name of the interrupt/trap handler. Those functions are machine dependent, iykyk. ๐Ÿ“Š
  • show igi_list addr โ€” shows info about the IGMP structure struct igmp_ifsoftc present at addr. ๐Ÿ“ก
  • show iosched addr โ€” shows info about the I/O scheduler struct cam_iosched_softc located at addr. ๐Ÿ’ฝ
  • show inodedeps [addr] โ€” shows brief info about each inodedep structure. If addr is given, only the inodedeps belonging to the fs located at the supplied address are shown. ๐Ÿ—ƒ๏ธ
  • show inpcb addr โ€” shows info on the IP Control Block struct in_pcb present at addr. ๐Ÿงท
  • show intr โ€” dumps info about interrupt handlers. โšก
  • show intrcnt โ€” dumps the interrupt statistics. ๐Ÿ”ข
  • show irqs โ€” shows interrupt lines and their respective kernel threads. ๐Ÿงต
  • show ktr[/avV] โ€” prints the contents of the ktr(4) trace buffer. The v modifier asks for fully verbose output, which makes the file, line number, and timestamp get printed for each trace entry. The V modifier asks for only the timestamps to be printed. And the a modifier requests that the output be unpaginated. ๐ŸŽž๏ธ
  • show lapic โ€” shows info from the local APIC registers for this CPU. ๐Ÿ
  • show lock addr โ€” shows the lock structure. The output format is literally this:
    • class: โ€” the class of the lock. Possible types include mutex(9), rmlock(9), rwlock(9), sx(9). ๐Ÿท๏ธ
    • name: โ€” the name of the lock. โœ๏ธ
    • flags: โ€” the flags passed to the lock initialization function. flags values are lock class specific. ๐Ÿšฉ
    • state: โ€” the current state of a lock. state values are lock class specific. ๐Ÿ“
    • owner: โ€” the lock owner. ๐Ÿ‘ค
  • show lockchain addr โ€” shows all the threads that the particular thread at address addr is waiting on, based on non-spin locks. โ›“๏ธ
  • show lockedbufs โ€” same info as "show buf", but for every locked struct buf object. ๐Ÿ”
  • show lockedvnods โ€” lists all the locked vnodes in the system. ๐ŸŒณ
  • show locks โ€” prints all locks that are currently acquired. Only available if witness(4) is included in the kernel. ๐Ÿ”
  • show locktree โ€” (that's it, that's the whole helper ๐ŸŽ€)
  • show malloc[/i] โ€” prints malloc(9) memory allocator statistics. If the i modifier is specified, it formats the output as machine-parseable comma-separated values ("CSV"). The output columns are as follows:
    • Type โ€” specifies a type of memory. It's the same as a description string used while defining the given memory type with MALLOC_DECLARE(9).
    • InUse โ€” number of memory allocations of the given type, for which free(9) has not been called yet.
    • MemUse โ€” total memory consumed by the given allocation type.
    • Requests โ€” number of memory allocation requests for the given memory type.

The same information can be gathered in userspace with โ€œvmstat -mโ€. ๐Ÿ“ˆ

  • show map[/f] addr โ€” prints the VM map at addr. If the f modifier is specified, the complete map gets printed. ๐Ÿ—บ๏ธ
  • show msgbuf โ€” prints the system's message buffer. It's the same output as in the โ€œdmesgโ€ case. Super useful if you got a kernel panic, attached a serial cable to the machine and want to get the boot messages from before the system hang. ๐Ÿ˜ญ๐Ÿ’–
  • show mount [addr] โ€” displays details about the mount point located at addr. No addr specified? It displays short info about all currently mounted file systems. ๐Ÿ“‚
  • show object[/f] addr โ€” prints the VM object at addr. If the f option is specified, the complete object gets printed. ๐ŸงŠ
  • show panic โ€” prints the panic message, if one is set. ๐Ÿšจ
  • show page โ€” shows statistics on VM pages. ๐Ÿ“ƒ
  • show pageq โ€” shows statistics on VM page queues. ๐Ÿšถโ€โ™€๏ธ
  • show pciregs โ€” prints PCI bus registers. The same info can be gathered in userspace by running โ€œpciconf -lvโ€. ๐Ÿงฉ
  • show pcpu โ€” prints the current processor state. The output format is as follows: ๐Ÿ‘€
  • cpuid โ€” the processor identifier ๐Ÿ†”
    • curthread โ€” the thread pointer, the process identifier, and the name of the process, all in one little bundle ๐Ÿงต
    • curpcb โ€” the control block pointer
    • fpcurthread โ€” the FPU thread pointer
    • idlethread โ€” the idle thread pointer ๐Ÿ˜ด
    • APIC ID โ€” the CPU identifier, coming straight from APIC
    • currentldt โ€” the LDT pointer
    • spin locks held โ€” the names of the spin locks currently held ๐Ÿ”’
  • show pgrpdump โ€” dumps all the process groups hanging out in the system ๐Ÿ’…
  • show prison [addr] โ€” shows the prison structure sitting at addr. No addr? Then it spills all the tea on every prison in the system ๐Ÿฐ
  • show proc [addr] โ€” info about the process structure at address addr, or about the current process if you don't pass one ๐Ÿซถ
  • show procvm [addr] โ€” the process virtual memory layout for the process at addr, or for the current process if you're feeling lazy and pass nothing ๐Ÿง 
  • show protosw addr โ€” prints the protocol switch structure struct protosw at address addr ๐Ÿ“ก
  • show registers[/u] โ€” displays the register set. Add the u modifier and instead you get the register contents of the thread's previous trapframe โ€” which usually means the saved state from userspace โœจ
  • show rman addr โ€” shows the resource manager object struct rman at address addr. Wanna collect addresses of particular pointers? Hit up the "show allrman" command ๐Ÿ—บ๏ธ
  • show route addr โ€” shows the route table result for destination addr. As of right now, INET and INET6 formatted addresses are the ones supported ๐Ÿ›ฃ๏ธ
  • show routetable [af] โ€” shows the full route table (or tables). Pass af and you only get routes for that numeric address family. Pass nothing and it dumps the route table for all address families, no cap ๐ŸŒˆ
  • show rtc โ€” shows the real time clock value. Genuinely useful for those long debugging sessions where you lose all sense of time ๐Ÿ˜ญโฐ
  • show sleepchain โ€” deprecated, bestie. It's now just an alias for show lockchain ๐Ÿ’€
  • show sleepq addr
  • show sleepqueue addr โ€” both show the sleepqueue(9) structure located at addr ๐Ÿ˜ด
  • show sockbuf addr โ€” shows the socket buffer struct sockbuf located at addr ๐Ÿงฆ
  • show socket addr โ€” shows the socket object struct socket located at addr ๐Ÿ”Œ
  • show sysregs โ€” shows system registers (think cr0-4 on i386). Heads up: it's not present on some platforms ๐Ÿ™ˆ
  • show tcpcb[/bi] addr โ€” prints the TCP control block struct tcpcb lying at address addr. For the exact interpretation of the output, go visit the netinet/tcp.h header file. The b modifier requests that BBLog entries get printed. And if you provide the i modifier, the corresponding IP control block gets shown too ๐Ÿ“ฆ
  • show thread [addr | tid] โ€” no addr or tid? Then you get detailed info about the current thread. Otherwise it prints info about the thread with ID tid or kernel address addr. (Pro tip: if the argument is a decimal number, it's assumed to be a tid.) ๐Ÿงต
  • show threads โ€” shows every thread in the system. The output format goes like this:
    • First column โ€” thread identifier (TID)
    • Second column โ€” thread structure address
    • Third column โ€” backtrace

๐ŸŽ€

  • show tty addr โ€” displays the contents of a TTY structure in a readable form ๐Ÿ–ฅ๏ธ
  • show turnstile addr โ€” shows the turnstile struct turnstile structure at address addr. Turnstiles are the structures used inside the FreeBSD kernel to implement synchronization primitives which, while holding a specific type of lock, cannot sleep or context switch to another thread. Currently the cool kids in that club are: mutex(9), rwlock(9), rmlock(9) ๐Ÿšช
  • show uma[/i] โ€” shows UMA allocator stats. Pass the i modifier and the output gets formatted as machine-parseable comma-separated values ("CSV") ๐Ÿค–. The output has these columns:
    • Zone โ€” name of the UMA zone. Same string that was passed to uma_zcreate(9) as a first argument.
    • Size โ€” size of a given memory object (slab).
    • Used โ€” number of slabs currently being used.
    • Free โ€” number of free slabs within the UMA zone.
    • Requests โ€” number of allocation requests to the given zone.
    • Total Mem โ€” total memory in use (either allocated or free) by a zone, in bytes.
    • XFree โ€” number of free slabs within the UMA zone that were freed on a different NUMA domain than allocated. (The count in the Free column is inclusive of XFree.)

Fun fact: you can gather the same info in userspace with the help of "vmstat -z" ๐Ÿ“Š

  • show unpcb addr โ€” shows the UNIX domain socket private control block struct unpcb present at the address addr ๐Ÿงฆ
  • show vmochk โ€” prints whether the internal VM objects are in a map somewhere, and whether none of them have zero ref counts โœ…
  • show vmopag โ€” walks the list of VM objects in the system, printing the indices and physical addresses of the VM pages belonging to each object ๐Ÿšถโ€โ™€๏ธ
  • show vnet addr โ€” prints the virtualized network stack struct vnet structure present at the address addr ๐ŸŒ
  • show vnode addr โ€” prints the vnode struct vnode structure lying at addr. For the exact interpretation of the output, go look at the sys/vnode.h header file ๐Ÿ“„
  • show vnodebufs addr โ€” shows the clean/dirty buffer lists of the vnode located at addr ๐Ÿงบ
  • show vpath addr โ€” walks the namecache to look up the pathname of the vnode located at addr ๐Ÿ›ค๏ธ
  • show watches โ€” displays all watchpoints. Specifically, the ones you set with the "watch" command ๐Ÿ‘€
  • show witness โ€” shows info about lock acquisition coming from the witness(4) subsystem ๐Ÿ‘‘

๐ŸŒ™ OFFLINE DEBUGGING COMMANDS ๐ŸŒ™

  • dump โ€” initiates a kernel core dump to the device(s) configured by dumpon(8) ๐Ÿ’พ
  • gdb โ€” switches to remote GDB mode. Heads up: in remote GDB mode you need another machine running gdb(1) (ports/devel/gdb) using the remote debug feature, connected to the serial console port on the target machine ๐Ÿ›
  • netdump -s server [-g gateway -c client -i iface] โ€” configures netdump(4) with the parameters you gave it, and immediately performs a netdump.

There are some known limitations, ngl. Principally, netdump(4) only supports IPv4 at this time. The address arguments to the netdump command must be dotted decimal IPv4 addresses. (Hostnames are not supported.) At present, the command only works if the machine is in a panic state. And finally, the ddb netdump command does not provide any way to configure compression or encryption ๐Ÿ’€

  • netgdb -s server [-g gateway -c client -i iface] โ€” initiates a netgdb(4) session with the provided parameters. netgdb has identical limitations to netdump ๐Ÿ›
  • capture on
  • capture off
  • capture reset
  • capture status โ€” ddb supports a basic output capture facility, which you can use to retrieve the results of debugging commands from userspace using sysctl(3). capture on enables output capture; capture off disables it. capture reset clears the capture buffer and disables capture. capture status reports the current buffer use, the buffer size, and the disposition of output capture ๐Ÿ“ธ

Userspace processes may inspect and manage ddb capture state using sysctl(8):

  • debug.ddb.capture.bufsize may be used to query or set the current capture buffer size.
  • debug.ddb.capture.maxbufsize may be used to query the compile-time limit on the capture buffer size.
  • debug.ddb.capture.bytes may be used to query the number of bytes of output currently in the capture buffer.
  • debug.ddb.capture.data returns the contents of the buffer as a string, to an appropriately privileged process.

This facility is particularly useful in concert with the scripting and textdump(4) facilities โ€” it lets scripted debugging output be captured and committed to disk as part of a textdump for later analysis. The contents of the capture buffer may also be inspected in a kernel core dump using kgdb(1) (ports/devel/gdb) ๐Ÿซถ

  • run
  • script
  • scripts
  • unscript โ€” run, define, list, and delete scripts. See the "SCRIPTING" section for more on the scripting facility ๐Ÿ“œ
  • textdump dump
  • textdump set
  • textdump status
  • textdump unset โ€” use the textdump dump command to immediately perform a textdump. More info lives in textdump(4). The textdump set command may be used to force the next kernel core dump to be a textdump rather than a traditional memory dump or minidump. textdump status reports whether a textdump has been scheduled. textdump unset cancels a request to perform a textdump as the next kernel core dump ๐Ÿ“

๐Ÿ’œ VARIABLES ๐Ÿ’œ

The debugger accesses registers and variables as $name. Register names are as in the "show registers" command. Some variables are suffixed with numbers, and may have some modifier following a colon immediately after the variable name. For example, register variables can have a u modifier to indicate user register (e.g., "$eax:u") โœจ

Built-in variables currently supported are:

  • radix โ€” input and output radix ๐Ÿ”ข
  • maxoff โ€” addresses are printed as "symbol+offset" unless offset is greater than maxoff
  • maxwidth โ€” the width of the displayed line ๐Ÿ“
  • lines โ€” the number of lines. It's used by the built-in pager. Setting it to 0 disables paging ๐Ÿ“ƒ
  • tabstops โ€” tab stop width โฉ
  • workxx โ€” work variable; xx can take values from 0 to 31 ๐Ÿ› ๏ธ

โœจ EXPRESSIONS โœจ

okay so ddb does math ๐Ÿงฎ and honestly it's lowkey C-coded โ€” most C expression operators work here, except for ~, ^, and the unary &. those three? not invited to the party ๐Ÿ™…โ€โ™€๏ธ. and there are some special ddb rules you gotta know:

  • Identifiers โ€” the name of a symbol gets translated into the value of the symbol, which is the address of the corresponding object. you can use . and : inside the identifier too ๐Ÿ’…. and if it's supported by an object format dependent routine, then [filename:]func:lineno, [filename:]variable, and [filename:]lineno can all be accepted as a symbol. fancy!
  • Numbers โ€” the radix is decided by the first two letters: 0x means hex, 0o means octal, 0t means decimal. otherwise, it just follows whatever the current radix is. iykyk ๐ŸŽ€
  • . โ€” dot ๐Ÿซถ
  • + โ€” next
  • .. โ€” the address of the start of the last line examined. and here's the thing: unlike dot or next, this one only changes when you use an examine or write command. icon behavior honestly โœจ
  • ' โ€” the last address explicitly specified
  • $variable โ€” translated to the value of the specified variable. it may be followed by a : and modifiers as described above ๐Ÿ‘€
  • a#b โ€” a binary operator that rounds up the left hand side to the next multiple of the right hand side. math slay ๐Ÿ”ฅ
  • *expr โ€” Indirection. it may be followed by a : and modifiers as described above ๐Ÿ’ซ

๐Ÿฆ‹ SCRIPTING ๐Ÿฆ‹

so ddb has a basic scripting facility and it's genuinely so good โ€” it lets you automate tasks or responses to specific events ๐ŸŒธ. each script is just a list of DDB commands executed sequentially, and it gets assigned a unique name. and bestie, some script names are special โ€” those get automatically run on various ddb events if scripts by those names have been defined ๐Ÿ‘‘

the script command defines a script by name. scripts are a series of ddb commands separated with the ; character. for example:

script kdb.enter.panic=bt; show pcpu
script lockinfo=show alllocks; show lockedvnods

the scripts command lists currently defined scripts ๐Ÿ“.

the run command executes a script by name. for example:

run lockinfo

the unscript command deletes a script by name ๐Ÿ’…. for example:

unscript kdb.enter.panic

and psst โ€” these functions can also be performed from userspace using the ddb(8) command ๐Ÿ˜Œ.

some scripts run automatically, if defined, for specific ddb events. here are the scripts that run when various events occur:

  • kdb.enter.acpi โ€” the kernel debugger was entered as a result of an acpi(4) event.
  • kdb.enter.bootflags โ€” the kernel debugger was entered at boot as a result of the debugger boot flag being set.
  • kdb.enter.break โ€” the kernel debugger was entered as a result of a serial or console break.
  • kdb.enter.cam โ€” the kernel debugger was entered as a result of a CAM(4) event.
  • kdb.enter.mac โ€” the kernel debugger was entered as a result of an assertion failure in the mac_test(4) module of the TrustedBSD MAC Framework.
  • kdb.enter.netgraph โ€” the kernel debugger was entered as a result of a netgraph(4) event.
  • kdb.enter.panic โ€” panic(9) was called. ๐Ÿ˜ญ
  • kdb.enter.powerpc โ€” the kernel debugger was entered as a result of an unimplemented interrupt type on the powerpc platform.
  • kdb.enter.sysctl โ€” the kernel debugger was entered as a result of the debug.kdb.enter sysctl being set.
  • kdb.enter.unionfs โ€” the kernel debugger was entered as a result of an assertion failure in the union file system.
  • kdb.enter.unknown โ€” the kernel debugger was entered, but no reason has been set. ๐Ÿ™ˆ
  • kdb.enter.vfslock โ€” the kernel debugger was entered as a result of a VFS lock violation.
  • kdb.enter.watchdog โ€” the kernel debugger was entered as a result of a watchdog firing.
  • kdb.enter.witness โ€” the kernel debugger was entered as a result of a witness(4) violation.

and in the event that none of these scripts is found, ddb will attempt to execute a default script:

  • kdb.enter.default โ€” the kernel debugger was entered, but a script exactly matching the reason for entering was not defined. this can be used as a catch-all to handle cases not specifically of interest ๐ŸŽฏ โ€” for example, kdb.enter.witness might be defined to have special handling, and kdb.enter.default might be defined to simply panic and reboot. main character fallback energy fr fr ๐Ÿ’€

๐Ÿ’œ HINTS ๐Ÿ’œ

on machines with an ISA expansion bus, you can construct a simple NMI generation card by connecting a push button between the A01 and B01 (CHCHK# and GND) card fingers ๐Ÿ”Œ. momentarily shorting these two fingers together may cause the bridge chipset to generate an NMI, which causes the kernel to pass control to ddb. some bridge chipsets do not generate an NMI on CHCHK#, so your mileage may vary bestie ๐Ÿ˜ฌ. the NMI lets you break into the debugger on a wedged machine to diagnose problems. other bus' bridge chipsets may be able to generate NMI using bus specific methods. there are many PCI and PCIe add-in cards which can generate NMI for debugging ๐Ÿ› ๏ธ. modern server systems typically use IPMI to generate signals to enter the debugger โ€” the devel/ipmitool port can be used to send the chassis power diag command, which delivers an NMI to the processor. embedded systems often use JTAG for debugging, but rarely use it in combination with ddb ๐Ÿ‘€.

serial consoles can break to the debugger by sending a BREAK condition on the serial line. this requires a kernel built with options BREAK_TO_DEBUGGER specified in the kernel ๐Ÿงš. most terminal emulation programs can send a break sequence with a special key sequence or menu selection. sending the break can be difficult, or even happen spuriously in some setups ๐Ÿ™ƒ. an alternative method is to build a kernel with options ALT_BREAK_TO_DEBUGGER โ€” then the sequence of CR TILDE CTRL-B enters the debugger; CR TILDE CTRL-P causes a panic; and CR TILDE CTRL-R causes an immediate reboot ๐Ÿ”ฅ. in all these sequences, CR represents Carriage Return and is usually sent by pressing the Enter or Return key. TILDE is the ASCII tilde character (~). CTRL-x is Control x, sent by pressing the Control key, then x, then releasing both ๐Ÿ’….

the break-to-debugger behavior can be enabled by setting sysctl(8) debug.kdb.break_to_debugger to 1. the alt-break-to-debugger behavior can be enabled by setting sysctl(8) debug.kdb.alt_break_to_debugger to 1. and the debugger can be entered by setting sysctl(8) debug.kdb.enter to 1 โœจ.

output can be interrupted, paused, and resumed with the control characters CTRL-C, CTRL-S, and CTRL-Q ๐Ÿซถ. and because these control characters are received as in-band data from the console, there is an input buffer โ€” and once that buffer fills, ddb must either stop responding to control characters or drop additional input while continuing to search for control characters. this behavior is controlled by the tunable sysctl(8) debug.ddb.prioritize_control_input, which defaults to 1. the input buffer size is 512 bytes ๐Ÿ“ฆ.


๐ŸŽ€ FILES ๐ŸŽ€

header files mentioned in this manual page can be found below the /usr/include directory ๐Ÿ’–:

  • sys/buf.h
  • sys/domain.h
  • netinet/in_pcb.h
  • sys/socket.h
  • sys/vnode.h

๐ŸŒˆ SEE ALSO ๐ŸŒˆ

gdb(1) (ports/devel/gdb), kgdb(1) (ports/devel/gdb), acpi(4), CAM(4), gdb(4), mac_ddb(4), mac_test(4), netgraph(4), textdump(4), witness(4), ddb(8), sysctl(8), panic(9) ๐Ÿ’ซ


โญ HISTORY โญ

the ddb debugger was developed for Mach, and ported to 386BSD 0.1 ๐Ÿ•ฐ๏ธ. this manual page was translated from man(7) macros by Garrett Wollman.

Robert N. M. Watson added support for ddb output capture, textdump(4) and scripting in FreeBSD 7.1 ๐Ÿ‘‘


and that's the whole page bestie, you ate that ๐Ÿซถ๐Ÿ’…โœจ โ€” this one's from Debian, dated October 31, 2025, and it's page DDB(4). go forth and debug, icon ๐ŸŽ€๐Ÿฆ‹

๐Ÿ“ฆ where this page comes from
  • freebsd 15.1: base 15.1
  • ๐Ÿ’– girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms