CRASH(8)

📦
version 9.0.3 · 6 versions
, in arch rolling, centos-stream 10, ubuntu stonking

💅 CRASH(8) 💅

okay bestie, welcome to the CRASH(8) page from the System Manager's Manual. This is the manual for crash, and it is giving kernel detective main character energy. ✨

✨ NAME ✨

crash — analyze Linux crash dump data or a live system. periodt. 💖

🌈 SYNOPSIS 🌈

okay so there are two forms, and they are exactly this:

crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)
crash [OPTION]... [NAMELIST]                         (live system form)

🌸 DESCRIPTION 🌸

okay so crash is THE tool for interactively analyzing the state of the Linux system while it is running, or after a kernel crash has occurred and a core dump has been created by the netdump, diskdump, LKCD, kdump, xendump, kvmdump or VMware facilities. it's giving forensic slay. 💅

It is loosely based on the SVR4 UNIX crash command, but it has been significantly enhanced by completely merging it with the gdb(1) debugger. The marriage of the two effectively combines the kernel-specific nature of the traditional UNIX crash utility with the source code level debugging capabilities of gdb(1). no cap, that's a power couple. ✨

In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument must be entered. In the live system form, the NAMELIST argument must be entered if the kernel's vmlinux file is not located in a known location, such as the /usr/lib/debug/lib/modules/<kernel-version> directory. 🫶

The crash utility has also been extended to support the analysis of dumpfiles generated by a crash of the Xen hypervisor. In that case, the NAMELIST argument must be that of the xen-syms binary. Live system analysis is not supported for the Xen hypervisor. 🚫

The crash utility command set consists of common kernel core analysis tools such as kernel stack back traces of all processes, source code disassembly, formatted kernel structure and variable displays, virtual memory data, dumps of linked-lists, etc., along with several commands that delve deeper into specific kernel subsystems. Appropriate gdb commands may also be entered, which in turn are passed on to the gdb module for execution. If desired, commands may be placed in either a $HOME/.crashrc file and/or in a .crashrc file in the current directory. During initialization, the commands in $HOME/.crashrc are executed first, followed by those in the ./.crashrc file. 💖

The crash utility is designed to be independent of Linux version dependencies. When new kernel source code impacts the correct functionality of crash and its command set, the utility will be updated to recognize new kernel code changes, while maintaining backwards compatibility with earlier releases. 🌈

👑 OPTIONS 👑

okay bestie, here is the full options menu, and yes, we are listing every single one. no skipping, no summarizing. 💅

  • NAMELIST — This is a pathname to an uncompressed kernel image (a vmlinux file), or a Xen hypervisor image (a xen-syms file) which has been compiled with the "-g" option. If using the dumpfile form, a vmlinux file may be compressed in either gzip or bzip2 formats. 💅
  • MEMORY-IMAGE[@ADDRESS] — A kernel core dump file created by the netdump, diskdump, LKCD, kdump, xendump, kvmdump or VMware facilities.
    • If a MEMORY-IMAGE argument is not entered, the session will be invoked on the live system, which typically requires root privileges because of the device file used to access system RAM. By default, /dev/crash will be used if it exists. If it does not exist, then /dev/mem will be used; but if the kernel has been configured with CONFIG_STRICT_DEVMEM, then /proc/kcore will be used. It is permissible to explicitly enter /dev/crash, /dev/mem or /proc/kcore.
    • An @ADDRESS value must be appended to the MEMORY-IMAGE if the dumpfile is a raw RAM dumpfile that has no header information describing the file contents. Multiple MEMORY-IMAGE@ADDRESS ordered pairs may be entered, with each dumpfile containing a contiguous block of RAM, where the ADDRESS value is the physical start address of the block expressed in hexadecimal. The physical address value(s) will be used to create a temporary ELF header in /var/tmp, which will only exist during the crash session. If a raw RAM dumpile represents a live memory source, such as that specified by the QEMU mem-path argument of a memory-backend-file object, then "live:" must be prepended to the MEMORY-IMAGE name.
    • As a VMware facility, the crash utility is able to process VMware VM memory dump generated by VM suspend or guest core dump. In that case, .vmss or .guest file should be used as a MEMORY-IMAGE and .vmem file must be located in the same folder.
  • mapfile — If the NAMELIST file is not the same kernel that is running (live system form), or the kernel that was running when the system crashed (dumpfile form), then the System.map file of the original kernel should be entered on the command line. 🌸
  • -h [option], --help [option] — Without an option argument, display a crash usage help message. If the option argument is a crash command name, the help page for that command is displayed. If it is the string "input", a page describing the various crash command line input options is displayed. If it is the string "output", a page describing command line output options is displayed. If it is the string "all", then all of the possible help messages are displayed. After the help message is displayed, crash exits. ✨
  • -s — Silently proceed directly to the "crash>" prompt without displaying any version, GPL, or crash initialization data during startup, and by default, runtime command output is not passed to any scrolling command. 🤫
  • -i file — Execute the command(s) contained in file prior to displaying the "crash>" prompt for interactive user input. 📜
  • -d num — Set the internal debug level. The higher the number, the more debugging data will be printed when crash initializes and runs. 🔍
  • -S — Use /boot/System.map as the mapfile. 🗺️
  • -e vi | emacs — Set the readline(3) command line editing mode to "vi" or "emacs". The default editing mode is "vi". ⌨️
  • -f — Force the usage of a compressed vmlinux file if its original name does not start with "vmlinux". 🗜️
  • -k — Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo file. 🧠
  • -g [namelist] — Determine if a vmlinux or xen-syms namelist file contains debugging data. 🔎
  • -t — Display the system-crash timestamp and exit. ⏰
  • -L — Attempt to lock all of its virtual address space into memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization. If the system call fails, an error message will be displayed, but the session continues. 🔒
  • -c tty-device — Open the tty-device as the console used for debug messages. 🖥️
  • -p page-size — If a processor's page size cannot be determined by the dumpfile, and the processor default cannot be used, use page-size. 📏
  • -o filename — Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dumpfiles, specifies a filename of a new ELF vmcore that will be created and used as the dumpfile. It will be saved to allow future use as a standalone vmcore, replacing the original raw RAM dumpfile. 💾
  • -m option=value, --machdep option=value — Pass an option and value pair to machine-dependent code. These architecture-specific option/pairs should only be required in very rare circumstances:
X86_64:
  phys_base=<physical-address>
  irq_eframe_link=<value>
  irq_stack_gap=<value>
  max_physmem_bits=<value>
  kernel_image_size=<value>
  vm=orig       (pre-2.6.11 virtual memory address ranges)
  vm=2.6.11     (2.6.11 and later virtual memory address ranges)
  vm=xen        (Xen kernel virtual memory address ranges)
  vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
  vm=5level     (5-level page tables)
  page_offset=<PAGE_OFFSET-value>
PPC64:
  vm=orig
  vm=2.6.14     (4-level page tables)
IA64:
  phys_start=<physical-address>
  init_stack_size=<size>
  vm=4l         (4-level page tables)
ARM:
  phys_base=<physical-address>
ARM64:
  phys_offset=<physical-address>
  kimage_voffset=<kimage_voffset-value>
  max_physmem_bits=<value>
  vabits_actual=<value>
X86:
  page_offset=<CONFIG_PAGE_OFFSET-value>
  • -x — Automatically load extension modules from a particular directory. If a directory is specified in the CRASH_EXTENSIONS shell environment variable, then that directory will be used. Otherwise /usr/lib64/crash/extensions (64-bit architectures) or /usr/lib/crash/extensions (32-bit architectures) will be used; if they do not exist, then the ./extensions directory will be used. 📦
  • --active — Track only the active task on each cpu. 🎯
  • --buildinfo — Display the crash binary's build date, the user ID of the builder, the hostname of the machine where the build was done, the target architecture, the version number, and the compiler version. 🏗️
  • --memory_module modname — Use the modname as an alternative kernel module to the crash.ko module that creates the /dev/crash device. 🔌
  • --memory_device device — Use device as an alternative device to the /dev/crash, /dev/mem or /proc/kcore devices. 🔧
  • --log dumpfile — Dump the contents of the kernel log buffer. A kernel namelist argument is not necessary, but the dumpfile must contain the VMCOREINFO data taken from the original /proc/vmcore ELF header. Note: this option is deprecated and will no longer work for kernel(>=v5.10). ⚠️
  • --no_kallsyms — Do not use kallsyms-generated symbol information contained within kernel module object files. 🚫
  • --no_modules — Do not access or display any kernel module related information. 📵
  • --no_ikconf — Do not attempt to read configuration data that was built into kernels configured with CONFIG_IKCONFIG. 🛑
  • --no_data_debug — Do not verify the validity of all structure member offsets and structure sizes that it uses. 🧐
  • --no_kmem_cache — Do not initialize the kernel's slab cache infrastructure, and commands that use kmem_cache-related data will not work. 🧱
  • --no_elf_notes — Do not use the registers from the ELF NT_PRSTATUS notes saved in a compressed kdump header for backtraces. 📝
  • --kmem_cache_delay — Delay the initialization of the kernel's slab cache infrastructure until it is required by a run-time command. ⏳
  • --readnow — passing this little guy over to the embedded gdb module, and it overrides gdb's whole two-stage strategy for reading symbol tables out of the NAMELIST. bestie said "no more stages, we read now" 📖✨
  • --smp — declares that the system you're analyzing is an SMP kernel. like yes she's multiprocessor, we been knew 💅
  • -v, --version — shows the version of the crash utility, the version of the embedded gdb module, all the GPL info, and the copyright notices. it's giving receipts ✨
  • --cpus number — tell it how many cpus are in the SMP system you're analyzing. number, pls and ty 🔢
  • --osrelease dumpfile — prints out the OSRELEASE vmcoreinfo string from a kdump dumpfile header. dumpfile lore unlocked 🩷
  • --build-id dumpfile — prints the BUILD-ID vmcoreinfo string from a kdump dumpfile header.
⚠️ heads up bestie: this only works for kernel >=v5.9. otherwise it prints "unknown" and exits with a non-zero status. not a vibe but okay 😭
  • --hyper — forces the session to be that of a Xen hypervisor. manifesting hypervisor era 👑
  • --p2m_mfn pfn — okay so when a Xen Hypervisor or its dom0 kernel crashes, the dumpfile is typically analyzed with either the Xen hypervisor or the dom0 kernel. but plot twist: you can also analyze any of the guest domU kernels, as long as you pass that guest kernel's pfn_to_mfn_list_list pfn value on the command line, along with its NAMELIST and the dumpfile. multitasking queen ✨
  • --xen_phys_start physical-address — supplies the base physical address of the Xen hypervisor's text and static data, for those older xendump dumpfiles that didn't pass that info along in the dumpfile header. doing the work the header forgot 🫶
  • --zero_excluded — so here's the tea: if the makedumpfile(8) facility has filtered a compressed kdump dumpfile to exclude various types of non-essential pages, or has marked a compressed or ELF kdump dumpfile as incomplete because of an ENOSPC or some other error while it was being created, then any attempt to read those missing pages will just fail. with this flag? reads from any of those pages return zero-filled memory instead. no failure, just zeros 🫧
  • --no_panic — do not attempt to find the task that was running when the kernel crashed. instead it sets the initial context to that of the "swapper" task on cpu 0. we're not dwelling on the past bestie 💅
  • --more — uses /bin/more as the command output scroller, overriding the default of /usr/bin/less and any settings in either ./.crashrc or $HOME/.crashrc. old school era 🎀
  • --less — uses /usr/bin/less as the command output scroller, overriding any settings in either ./.crashrc or $HOME/.crashrc. less is more, literally 💖
  • --hex — sets the default command output radix to 16, overriding the default radix of 10 and any radix settings in either ./.crashrc or $HOME/.crashrc. hexadecimal serve 🌈
  • --dec — sets the default command output radix to 10, overriding any radix settings in either ./.crashrc or $HOME/.crashrc. and fun fact bestie: this is the default radix setting. the original, the blueprint, the moment ⭐
  • --CRASHPAGER — uses the output paging command defined in the CRASHPAGER shell environment variable, overriding any settings in either ./.crashrc or $HOME/.crashrc. env var said it's my turn now 🌸
  • --no_scroll — does not pass run-time command output to any scrolling command. no pager, no scroll, just vibes 🦋
  • --no_strip — does not strip cloned kernel text symbol names. leave her names alone 💅
  • --no_crashrc — does not execute the commands in either $HOME/.crashrc or ./.crashrc. skipping the startup lore entirely 🙈
  • --mod directory — when you're loading the debuginfo data of kernel modules with the mod -S command, this searches for their object files in directory instead of in the standard location. custom directory, custom era 📁✨
  • --src directory — searches for the kernel source code in directory instead of in the standard location that's compiled into the debuginfo data. source code scavenger hunt 🩷
  • --kaslr offset|auto — if an x86, x86_64, s390x or loongarch64 kernel was configured with CONFIG_RANDOMIZE_BASE, then the offset value equals the difference between the symbol values compiled into the vmlinux file and their relocated KASLR values. and if you set it to auto? the KASLR offset value gets automatically calculated. let the machine do the math 🔥
  • --reloc size — when you're analyzing live x86 kernels that were configured with a CONFIG_PHYSICAL_START value that is larger than its CONFIG_PHYSICAL_ALIGN value, then you'll need to enter a relocation size equal to the difference between those two values. math is math 💀
  • --hash count — sets the number of internal hash queue heads used for list gathering and verification. the default count is 32768. iconic number, no cap 🎀
  • --max-malloc-bufs size — sets the value of MAX_MALLOC_BUFS to size. the minimum allowed value is 3072. don't go below it bestie, she said minimum 🚫
  • --minimal — brings up a session that is restricted to just the log, dis, rd, sym, eval, set and exit commands. this option may give you a way to extract some minimal/quick info from a corrupted or truncated dumpfile, or in situations where one of the several kernel subsystem initialization routines would abort the crash session. small but mighty, minimal main character energy 👑
  • --kvmhost [32|64] — when you're examining an x86 KVM guest dumpfile, this option specifies that the KVM host that created the dumpfile was an x86 (32-bit) or an x86_64 (64-bit) machine, overriding the automatically determined value. pick your fighter: 32 or 64 ⚔️
  • --kvmio <size> — overrides the automatically-calculated KVM guest I/O hole size. auto-calc could never 💅
  • --offline [show|hide] — show or hide command output that is related to offline cpus. the default setting is show. off the grid but make it configurable 🌙

💫 COMMANDS 💫

okay so every single crash command lowkey falls into one of these four vibes:

  • Symbolic display — these show off kernel text/data, and they fully use the power of gdb to format and display data structures symbolically. it's giving glow-up ✨
  • System state — the majority, bestie. a whole set of "kernel-aware" commands that dive into various kernel subsystems, either system-wide or per-task. main character energy 🌸
  • Utility functions — a set of useful helper commands serving various purposes: some simple, others quite powerful 👑
  • Session control — commands that control the crash session itself 💅

The following alphabetical list is a very simple overview of each crash command. BUT — and this is the important part — since individual commands often have several options that change the output significantly, you should really view the full description of each command by executing crash -h <command>, or during a crash session by simply entering help command. no cap, do the full read 🫶

  • * — "pointer to" is shorthand for either the struct or union commands. it displays the contents of a kernel structure or union. ⭐
  • alias — creates a single-word alias for a command.
  • ascii — displays an ascii chart, or translates a numeric value into its ascii components.
  • bpf — provides information on currently-loaded eBPF programs and maps.
  • bt — displays a task's kernel-stack backtrace. if you hand it the -a option, it displays the stack traces of the active tasks on all CPUs. it is often used with the foreach command to display the backtraces of all tasks with one command. iconic combo 💖
  • btop — translates a byte value (physical offset) to its page number.
  • dev — displays data concerning the character and block device assignments, I/O port usage, I/O memory usage, and PCI device data.
  • dis — disassembles memory: either entire kernel functions, from a location for a specified number of instructions, or from the start of a function up to a specified memory location.
  • eval — evaluates an expression or numeric type and displays the result in hexadecimal, decimal, octal and binary.
  • exit — causes crash to exit.
  • extend — dynamically loads or unloads crash shared object extension modules.
  • files — displays information about open files in a context.
  • foreach — repeats a specified command for the specified (or all) tasks in the system.
  • fuser — displays the tasks using the specified file or socket.
  • gdb — passes its argument to the embedded gdb module. it's useful for executing gdb commands that have the same name as crash commands. 🤝
  • help — alone, it displays the command menu; if followed by a command name, a full description of a command, its options, and examples are displayed. its output is far more complete and useful than this man page. (not me telling you to go read that instead, but... yeah 🙈)
  • ipcs — displays data about the System V IPC facilities.
  • irq — displays data concerning interrupt request numbers and bottom-half interrupt handling.
  • kmem — displays information about the use of kernel memory.
  • list — displays the contents of a linked list.
  • log — displays the kernel log_buf contents in chronological order.
  • mach — displays data specific to the machine type.
  • mod — displays information about the currently installed kernel modules, or adds or deletes symbolic or debugging information about specified kernel modules.
  • mount — displays information about the currently-mounted filesystems.
  • net — display various network related data.
  • p — passes its arguments to the gdb "print" command for evaluation and display.
  • ps — displays process status for specified, or all, processes in the system.
  • pte — translates the hexadecimal contents of a PTE into its physical page address and page bit settings.
  • ptob — translates a page frame number to its byte value.
  • ptov — translates a hexadecimal physical address into a kernel virtual address.
  • q — is an alias for the "exit" command.
  • rd — displays the contents of memory, with the output formatted in several different manners.
  • repeat — repeats a command indefinitely, optionally delaying a given number of seconds between each command execution.
  • runq — displays the tasks on the run queue.
  • sbitmapq — dumps the contents of the sbitmap_queue structure and the used bits in the bitmap. also, it shows the dump of a structure array associated with the sbitmap_queue. 🧚
  • search — searches a range of user or kernel memory space for given value.
  • set — either sets a new context, or gets the current context for display.
  • sig — displays signal-handling data of one or more tasks.
  • struct — displays either a structure definition or the contents of a kernel structure at a specified address.
  • swap — displays information about each configured swap device.
  • sym — translates a symbol to its virtual address, or a static kernel virtual address to its symbol — or to a symbol-plus-offset value, if appropriate.
  • sys — displays system-specific data.
  • task — displays the contents of a task_struct.
  • tree — displays the contents of a red-black tree or a radix tree.
  • timer — displays the timer queue entries, both old- and new-style, in chronological order.
  • union — is similar to the struct command, except that it works on kernel unions.
  • vm — displays basic virtual memory information of a context.
  • vtop — translates a user or kernel virtual address to its physical address.
  • waitq — walks the wait queue list displaying the tasks which are blocked on the specified wait queue.
  • whatis — displays the definition of structures, unions, typedefs or text/data symbols.
  • wr — modifies the contents of memory on a live system. 👀 it can only be used if /dev/mem is the device file being used to access system RAM, and should obviously be used with great care. like, bestie, be careful, please 😭

🦋 the Xen plot twist 🦋

When crash is invoked with a Xen hypervisor binary as the NAMELIST, the command set is slightly modified. the *, alias, ascii, bt, dis, eval, exit, extend, gdb, help, list, log, p, pte, rd, repeat, search, set, struct, sym, sys, union, whatis, wr and q commands are all the same as above. ✨

The following commands are specific to the Xen hypervisor:

  • domain — displays the contents of the domain structure for selected, or all, domains.
  • doms — displays domain status for selected, or all, domains.
  • dumpinfo — displays Xen dump information for selected, or all, cpus.
  • pcpus — displays physical cpu information for selected, or all, cpus.
  • vcpus — displays vcpu status for selected, or all, vcpus.

🎀 FILES 🎀

  • .crashrc — initialization commands. the file can be located in the user's HOME directory and/or the current directory. commands found in the .crashrc file in the HOME directory are executed before those in the current directory's .crashrc file. order matters, babes 💅

🌈 ENVIRONMENT 🌈

  • EDITOR — command input is read using readline(3). if EDITOR is set to emacs or vi then suitable keybindings are used. if EDITOR is not set, then vi is used. this can be overridden by set vi or set emacs commands located in a .crashrc file, or by entering -e emacs on the crash command line.
  • CRASHPAGER — if CRASHPAGER is set, its value is used as the name of the program to which command output will be sent. if not, then command output is sent to /usr/bin/less -E -X by default.
  • CRASH_MODULE_PATH — specifies an alternative directory tree to search for kernel module object files.
  • CRASH_EXTENSIONS — specifies a directory containing extension modules that will be loaded automatically if the -x command line option is used.

⭐ NOTES ⭐

If crash does not work, look for a newer version: kernel evolution frequently makes crash updates necessary. fr fr, the kernel moves fast and crash has to keep up 🏃‍♀️

The command set scroll off will cause output to be sent directly to the terminal rather than through a paging program. this is useful, for example, if you are running crash in a window of emacs.

lowkey a lifesaver if your pager and your emacs window are beefing 🙈

👑 AUTHOR 👑

Dave Anderson <anderson@redhat.com> wrote crash.

Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@redhat.com> wrote this man page.

💖 SEE ALSO 💖

The help command within crash provides more complete and accurate documentation than this man page. (again: go read it, bestie 💅)

https://github.com/crash-utility — the home page of the crash utility.

netdump(8), gdb(1), makedumpfile(8)

and that's the whole page, bestie!! 🫶 the footer down at the bottom signs the whole thing off as CRASH(8), with the source/version info and the date tucked in there too — your little timestamped receipt that this manual is that girl. go read your kernel, drink some water, and remember: help command is always more complete than this man page ✨💜

📦 where this page comes from