KDUMP(7)

πŸ“¦ version 2.1.6+git9.g60a2898, in opensuse-leap 16.0

🌸 KDUMP(7) β€” kdump, aka how SUSE saves kernel dumps 🌸

okay so bestie, this is KDUMP(7) from the User Manuals section and it's literally the whole vibe of "my kernel crashed and I need the receipts." buckle up πŸ’…

✨ SYNOPSIS ✨

(not applicable)

no synopsis, it's giving "concept page, not a command" πŸŽ€

πŸ’– DESCRIPTION πŸ’–

This manual page gives an overview about kdump configuration on SUSE. fr fr, that's the whole assignment.

Kdump is a technology to save the memory contents of a crashed system and save it to disk or network to analyse it later and find the cause of the crash. Think of it as the ultimate "no because WHAT happened" recorder 😭. When the system crashes, the mechanism uses kexec to boot a normal Linux kernel (that has been loaded into the system previously) which then has access to the old memory contents via /proc/vmcore interface and can save that away. iconic behaviour honestly.

After the memory has been saved, the system reboots (without kexec).

So, as mentioned above, that panic kernel has to be loaded into the system. That is accomplished via kexec(8) called by the kdump service at system bootup. And to have memory for that panic kernel β€” and also have RAM for the execution of that panic kernel β€” one has to reserve kernel memory with a special kernel command line option (crashkernel). The option is added to the kernel command line by the kdump-commandline service, but any change will only have effect for the next boot. lowkey important, highkey easy to forget πŸ™ˆ


πŸ¦‹ AUTOMATIC CONFIGURATION πŸ¦‹

πŸ’œ Non-transactional systems

An easy method to configure kdump on SUSE non-transactional systems is to install the kdump package and enable the kdump.service:

# zypper install kdump
# systemctl enable kdump

To update the bootloader with the default kdump commandline options and generate the kdump initrd, you can call

# systemctl start kdump

This will fail, because the currently running kernel has not been started with the crashkernel commandline option. But bestie, that's fine β€” the service will also start the kdump-commandline.service, which will by default update the bootloader to provide the option on the next boot. so the "failure" is literally part of the plan πŸ’€

If you don't perform this step, the service will fail during the next boot and kdump should be available on the second-next boot. that's a two-boot pipeline, 😭 plan accordingly.

🧚 Transactional systems

On a transactional system the kdump-commandline.service won't be able to update the bootloader. Use the transactional-update tool instead:

# transactional-update setup-kdump

This will install kdump, set KDUMP_UPDATE_BOOTLOADER to false and run kdumptool calibrate and kdumptool commandline to update the bootloader. slay, one command, three jobs 🫢


🌈 MANUAL SETUP 🌈

πŸŽ€ Bootloader configuration

It's necessary to reserve a certain amount of memory in the normal system at boot time which will be used by kexec(8) to load the panic kernel. To achieve that, you have to add a parameter called crashkernel in bootloader configuration. The syntax is:

crashkernel=size[@offset]

The optional offset is the load offset, i.e. the physical base address on which the memory reservation area starts.

To find a suggested value for size, use the kdumptool calibrate command. The suggested value is the sum of the Low and High values in its output. πŸ‘‘

Example: crashkernel=256M (on a normal PC system)

Note There's also a more advanced syntax that makes the amount of memory dependent on system RAM. See the section called "Extended crashkernel commandline".

You can either supply a hand-crafted crashkernel option to the kdump-commandline.service via the KDUMP_CRASHKERNEL config option and leave the job of updating the bootloader to the service.

Or you can disable automatic bootloader updates by setting KDUMP_UPDATE_BOOTLOADER to "false" and managing the bootloader config yourself. main character energy, do it your way ✨

⭐ Enable kdump service

The kdump systemd service loads the kdump kernel at boot. To enable it, run

# systemctl enable kdump

πŸ“ Configure kdump

The default configuration should work out of the box. You can tweak several configuration options in the /etc/sysconfig/kdump configuration file.

Important If you make changes in that configuration file, you always have to execute systemctl restart kdump manually to make that changes apply. If you don't, that changes will only apply after system reboot. not me forgetting the restart and waiting a whole reboot 😭

See the section "CONFIGURATION" later and/or kdump(5) for a description of the configuration options.


πŸ”₯ TESTING πŸ”₯

It perfectly makes sense to test the kdump configuration in a sane system state, i.e. not when the system really crashes but to trigger the dump manually. To perform that, use the SysRq mechanism, i.e. just execute

# echo s > /proc/sysrq-trigger
# echo u > /proc/sysrq-trigger
# echo c > /proc/sysrq-trigger

After that, the panic kernel should boot and the dump should be saved. πŸŽ‰

The most common problem is that the panic kernel will fail because the default amount of memory reserved by the crashkernel option is not sufficient. This typically demonstrates by either the panic kernel failing early at boot or running out of memory when running the kdump userspace processes (systemd, makedumpfile, ...). In that case you will want to manually set the KDUMP_CRASHKERNEL config option.

The default value is provided by the kdumptool_calibrate command and set in the bootloader by the kdumptool_commandline command.

You can use the values suggested by kdumptool_calibrate as a starting point for finding a correct value and setting KDUMP_CRASHKERNEL manually.

The value of KDUMP_CPUS influences the amount of memory required by kdump. You may want to decrease the default value to limit the memory required. Note that previously the default was to use a single CPU. the way the defaults changed and nobody told us πŸ’€


πŸ’… CONFIGURATION πŸ’…

The configuration file is /etc/sysconfig/kdump. Just edit this file with a plain text editor to adjust the settings. All variables are described in kdump(5). Here's a brief overview about some variables that are worth tweaking.

🌸 Save Directory

The most important setting is where the dump should be saved. Different methods are available: local file, FTP, SFTP (SSH), NFS, CIFS.

The configuration variable KDUMP_SAVEDIR has to be filled with a URL to where the dump should be saved. The syntax is described in kdump(5).

Note If you want to use SSH or SFTP with public key authentication, make sure to read the "Secure File Transfer Protocol" section in kdump(5).

🫢 Preventing running out of disk space

If you save the dumps to your local file system, you may want kdump to delete old dumps automatically. See the KDUMP_KEEP_OLD_DUMPS option.

If the partition has less than KDUMP_FREE_DISK_SIZE megabytes free disk space after saving the dump, the dump is not saved at all.

Important That two options don't apply to network dump targets. nope, networking dumps are on their own πŸ’€

πŸ¦‹ Dump Filtering and Compression

Dump size can be reduced by

  • using different compresison algorithms
  • filtering unnecessary pages from the dump

see KDUMP_DUMPFORMAT and KDUMP_DUMPLEVEL in kdump(5) β€” smaller dumps, happier disks 🌈

🌟 Notification

If you enable notification support, then you get an email when the system reboots after the dump has been copied. This only works for locally saved dumps. See KDUMP_NOTIFICATION_TO in kdump(5). an email saying "hey bestie your kernel crashed" 😭 iconic.

πŸ‘‘ Debugging options

Normally the machine is rebooted when kdump finishes and all errors are ignored to get the production system runningg as soon as possible,

To get a shell for debugging kdump itself, see the KDUMP_IMMEDIATE_REBOOT and KDUMP_CONTINUE_ON_ERROR options.

Also see the KDUMP_VERBOSE option to turn on more logging during kdump service start and during the dump itself.

To debug the creation of the kdump initrd, you may want to run mkdumprd -d.

Warning If you use a VGA console and trigger the dump when X11 is running (i.e. your graphical desktop), you might not see any output. Use a serial console in that case, or try to trigger the dump from Linux console (i.e. press Ctrl+Alt+F1 in your graphical desktop and log in there). yes the tty is your friend, always has been 🫢

🧚 ADVANCED CONFIGURATION 🧚

πŸ’œ Trigger Kdump on NMI (i386/x86_64 only)

Some systems (mostly "Enterprise" servers) have a so-called NMI button (physically or via the remote management consoles) that triggers an NMI manually if the system hangs completely and even SysRQ does not work any more.

If you want to trigger a kdump in that case, you have to execute

# sysctl kernel.panic_on_unrecovered_nmi=1

manually or (if you want to make that a permanent setting) add

kernel.panic_on_unrecovered_nmi=1

in /etc/sysctl.conf. set it and forget it, but like, set it properly πŸ’…

⭐ Extended crashkernel commandline

While the "crashkernel=size[@offset]" syntax is sufficient for most configurations, sometimes it's handy to have the reserved memory dependent on the value of System RAM β€” that's mostly for distributors that pre-setup the kernel command line to avoid a unbootable system after some memory has been removed from the machine.

The syntax is:

crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset]
range=start-[end]

while start is inclusive and end is exclusive. (yes, half-open interval, no cap 🧠)

For example:

crashkernel=512M-2G:64M,2G-:128M

This would mean:

  1. If the RAM is smaller than 512M, then don't reserve anything (this is the "rescue" case).
  2. If the RAM size is between 512M and 2G (exclusive), then reserve 64M.
  3. If the RAM size is larger than 2G, then reserve 128M.

ate, honestly. conditional memory reservation is kind of a flex πŸ”₯


πŸŽ€ SEE ALSO πŸŽ€

kexec(8), kdump(5), makedumpfile(8), crash(8), The Kexec and Kdump chapter in the SUSE System Analysis and Tuning Guide

🌸 AUTHOR 🌸

Bernhard Walle <bwalle@suse.de>

  • Author.

okay that's the whole page bestie, go save some dumps πŸ’–βœ¨

kdump 1.0.2 Β· 04/01/2026 β€” that's the version and date stamped on this whole thing, mwah πŸ’…πŸŽ€