1. Event : FUDCon PUNE 2015
Topic : Introduction to kdump
Track : Workshop
Date : June 26, 2015
Time : 10:00 am to 11:30 am
Hall : D 405 [4th Floor]
Presenter : Buland Singh
Email : buland.kumar.singh@gmail.com
IRC : bsingh
**** Introduction to Kdump ****
o What is kexec ?
o How does it work ?
o What is kdump ?
o How does it work ?
o How to configure kdump ?
o How to test kdump setup ?
o How to verify kernel crash dump (vmcore) ?
o References
** What is kexec ? **
o kexec is a fast­boot mechanism that allows booting a Linux kernel from the context of an already running kernel
without going through BIOS and passing through the boot­loader stage.
o kexec performs the function of the boot loader from within the kernel and when the system boots a kernel with kexec,
it skips the boot loader stage.
o Since BIOS checks at startup can be very time consuming (especially on big servers with numerous peripherals), kexec
can save a lot of time for developers who need to reboot a machine often for testing purposes.
o The total amount of time saved when rebooting is highly dependent on your server, and can range from several tens of
seconds to several minutes.
o The primary difference between a standard system boot and a kexec boot is that the hardware initialization normally
performed by the BIOS or firmware (depending on architecture) is not performed during a kexec boot. This has the
effect of reducing the time required for a reboot.
o kexec is useful for several reasons:
o Faster system rebooting
o Booting without GRUB or LILO configuration
o Avoiding unreliable firmware and hardware
o Saving the dump of a crashed kernel
o kexec include two components:
a) User Space Tool :­ /usr/sbin/kexec
o The binary /usr/sbin/kexec is provided "kexec­tools" package.
o It can be easily installed by using following command on Fedora­21.
# dnf install kexec­tools
b) Kernel System Call :­ kexec_load()
o The Linux kernel configuration "CONFIG_KEXEC" enables the kexec system call.
o The kernel packages shipped with Fedora/Centos/RHEL are complied with "CONFIG_KEXEC" enabled.
# grep "CONFIG_KEXEC=" /boot/config­4.0.4­301.fc22.x86_64
CONFIG_KEXEC=y
** How does it works ? **
o Using kexec consists of;
a) Loading the kernel into the physical memory.
# kexec ­l
b) Rebooting with the pre­loaded kernel.
# kexec ­e
[ Refer: kexec_design1.jpg ]

2. > To load a kernel, the syntax is as follows:
# kexec ­l <kernel­image> ­­initrd=<initrd­image> ­­append=<command­line­options>
Where;
o "kernel­image" is the kernel file that you intend to reboot to.
o "initrd­image" is the initrd image to be used during boot.
o "command­line­options" is the command­line parameters that must be passed to the new kernel.
Note: Passing the exact contents of /proc/cmdline into "command­line­options" is the safest way
to ensure that correct values are passed to the rebooting kernel.
> The loaded kernel can be booted to at any time using the command:
# kexec ­e
Eg: 1
# kexec ­l /boot/vmlinuz­4.0.4­301.fc22.x86_64 ­­initrd=/boot/initramfs­4.0.4­301.fc22.x86_64.img ­­command­line="$
( cat /proc/cmdline )"
# cat /sys/kernel/kexec_loaded
# kexec ­e
**OR**
Eg: 2
# kexec ­l /boot/vmlinuz­4.0.4­301.fc22.x86_64 ­­initrd=/boot/initramfs­4.0.4­301.fc22.x86_64.img ­­reuse­cmdline
# cat /sys/kernel/kexec_loaded
# kexec ­e
o kexec ­l loads the kernel to the address space of a production kernel for a regular reboot.
o /sys/kernel/kexec_loaded file is used to check whether the kexec kernel is loaded.
1 ­ kexec kernel is loaded in memory.
0 ­ kexec kernel is not loaded in memory.
o kexec ­u unload the kernel that was loaded with the kdump ­l option.
Eg: 3
# kexec ­p /boot/vmlinuz­4.0.4­301.fc22.x86_64 ­­initrd=/boot/initramfs­4.0.4­301.fc22.x86_64.img ­­command­line="$
( cat /proc/cmdline )"
# cat /sys/kernel/kexec_loaded
o kexec ­p loads the kernel to a reserved area of memory. This kernel will be booted automatically
when the system crashes.
o kexec ­p ­u unload the kernel that was loaded with the kdump ­p option.
** What is kdump ? **
o Kdump is a very reliable kexec based crash dumping mechanism for Linux.
o Kdump functionality is broken mainly in two components, user space (kexec) and kernel space (kdump).
o Kernel space patches are already part of main line kernel tree.
o kdump support in kernel:
# grep CRASH_DUMP /boot/config­4.0.4­301.fc22.x86_64
CONFIG_CRASH_DUMP=y
o User space component is nothing but a patch on top of existing kexec tools.
# grep "CONFIG_KEXEC=" /boot/config­4.0.4­301.fc22.x86_64
CONFIG_KEXEC=y
o Kdump need two different kernel, one we can call standard (production) kernel and another we can call crash
(capture) kernel.
o The crash dumps are captured from the context of a freshly booted kernel; not from the context of the crashed
kernel.
o Kdump can be configured to dump either directly to a device, to a file, or to some location on the network via NFS
or SSH.
o Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64, and s390x architectures.

3. ** How does it works ? ***
o Whenever the standard (production) kernel crashes, kdump uses kexec to boot into a crash (capture) kernel.
o This crash (capture) kernel boots with very little pre­reserved memory and captures the dump image.
o The standard (production) kernel reserves a section of memory that the crash (capture) kernel uses to boot.
o Be aware that the memory reserved for the kdump kernel at boot time cannot be used by the standard kernel.
o Kexec enables booting the capture kernel without going through the BIOS, so contents of the first kernel's
memory are preserved, which is essentially the kernel crash dump.
[ Refer: kdump_design1.jpg ]
** What is kernel crash dump (vmcore) ? **
o Kernel crash dump (vmcore) is a dump of all the physical memory (RAM, registers, but not swap) at a particular point
in time.
o Kernel crash dump captures the state of kernel at the moment of panic.
"Kernel crash dump (vmcore) is a snapshot of the physical memory at the time of crash"
** Who needs kernel crash dump (vmcore) and why ? **
o Kernel developers : To identify the cause of kernel panic. It helps them to fix the bug.
o Enterprise customer : To patch the servers with a bug fix package in­order to avoid the downtime in future.
** Kdump Configuration file **
o The following two configuration files are used to configure kdump.
o /etc/sysconfig/kdump { configuraiton file for kexec kernel }
o /etc/kdump.conf { configuration file for kdump service }
o Detail information of different directives present in these configuration files.
** /etc/sysconfig/kdump **
KDUMP_KERNELVER
o Use to set the kernel version string for the kdump kernel (optional).
o Leave it empty to use the auto­detection mechanism (strongly recommended).
Eg:
KDUMP_KERNELVER=""
OR
KDUMP_KERNELVER="4.0.4­301.fc22"
KDUMP_COMMANDLINE
o Use to overwrite the automatically generated kdump command line (optional).
o If a command line is not specified, the default will be taken from /proc/cmdline.
o Leave it empty to use the auto­detection mechanism (strongly recommended).
Eg:
KDUMP_COMMANDLINE=""
KDUMP_COMMANDLINE_APPEND
o Use to set additional command line parameters for the kdump kernel.
o Leave it as it is (recommended).
Eg:
KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children­max=2
panic=10 rootflags=nofail acpi_no_memhotplug"
KEXEC_ARGS
o Use to set additional kexec­tools command line arguments.
o Leave it empty (strongly recommended).
Eg:
KEXEC_ARGS="­­elf32­core­headers"
KDUMP_BOOTDIR
o Used to set the directory to find the boot image.
o Leave it as it is (strongly recommended).
Eg:
KDUMP_BOOTDIR="/boot"
KDUMP_IMG
o Used to set the image type used for kdump.
o Leave it as it is (strongly recommended).

4. Eg:
KDUMP_IMG="vmlinuz"
KDUMP_IMG_EXT
o Used to set the images extension.
o Leave it as it is (strongly recommended).
Eg:
KDUMP_IMG_EXT=""
** /etc/kdump.conf **
raw <partition>
o Use to set dump target to raw <partition>.
o Use persistent device names for partition devices, such as /dev/vg/<devname>.
o Kdump will dd /proc/vmcore into raw <partition>
Eg:
#raw /dev/vg/lv_kdump
<fs type> <partition>
o Use to set dump target to a specific partition of a specific filesystem type.
o <partition> can be a device node, label or uuid.
o It's recommended to use persistent device names such as /dev/vg/<devname>.
o Otherwise it's suggested to use label or uuid.
o kdump will mount ­t <fs type> <partition> /mnt and copy /proc/vmcore to /mnt/var/crash/%DATE/.
Eg:
ext4 /dev/vg/lv_kdump
ext4 LABEL=/boot
ext4 UUID=03138356­5e61­4ab3­b58e­27507ac41937
nfs <nfs mount>
o Use to set dump target to NFS mount point.
o Kdump will mount fs and copy /proc/vmcore to <mnt>/var/crash/%HOST­%DATE/, supports DNS.
Eg:
nfs server.example.com:/export/tmp
ssh <user@server>
o Used to set dump target on remote machine.
o Kdump will scp /proc/vmcore to <user@server>:/var/crash/%HOST­%DATE/, supports DNS.
Note: Make sure user has necessary write permissions on server.
Eg:
ssh user@server.example.com
sshkey <path>
o Use to specify the path of the ssh key you want to use when do ssh dump, the default value is
/root/.ssh/kdump_id_rsa
o kdump will use the sshkey to do ssh dump.
Eg:
sshkey /root/.ssh/kdump_id_rsa
path <path>
o "path" represents the filesystem path in which vmcore will be saved.
o If a dump target is specified in kdump.conf, then "path" is relative to the specified dump target.
o Interpretation of path changes a bit if user has not specified a dump target explicitly in kdump.conf. In this case,
"path" represents the absolute path from root. And dump target and adjusted path are arrived at automatically
depending on what's mounted in the current system.
o Ignored for raw device dumps.
o If unset, will default to /var/crash.
Eg:
path /var/crash
core_collector <command> <options>
o This allows you to specify the command to copy the vmcore.
o Default core_collector for other targets is: "makedumpfile ­c ­­message­level 1 ­d 31"
Eg:
core_collector makedumpfile ­c ­­message­level 1 ­d 31

5. o The ­c tells makedumpfile command to compress the vmcore file.
o The "­d" option is used to set the dump_level.
o "dump_level" is used to decide which pages to be removed from the resultant vmcore file.
o The option is a bit mask, having each page type specified like so:
zero pages = 1
cache pages = 2
cache private = 4
user pages = 8
free pages = 16
o In general, these pages don't contain relevant information. To set all these flags and leave out these pages, use a
value of ­d 31.
kdump_post <binary | script>
o This directive allows you to run a specified executable just after the memory dump process terminates.
o The exit status from the dump process is fed to the kdump_post executable, which can be used to trigger different
actions for success or failure.
Eg:
kdump_post /var/crash/scripts/kdump­post.sh
kdump_pre <binary | script>
o Works just like the kdump_post directive, but instead of running after the dump process, runs immediately before.
o Exit status of this binary is interpreted as follows:
0 ­ continue with dump process as usual
!0 ­ reboot the system
Eg:
kdump_pre /var/crash/scripts/kdump­pre.sh
extra_bins <binaries | shell scripts>
o This directive allows you to specify additional binaries or shell scripts you'd like to include in your kdump
initrd.
o Generally only useful in conjunction with a kdump_post binary or script that relies on other binaries or scripts.
Eg:
extra_bins /usr/bin/lftp
extra_modules <module(s)>
o This directive allows you to specify extra kernel modules that you want to be loaded in the kdump initrd, typically
used to set up access to non­boot­path dump targets that might otherwise not be accessible in the kdump environment.
o Multiple modules can be listed, separated by a space, and any dependent modules will automatically be included.
Eg:
extra_modules gfs2
default <reboot | halt | poweroff | shell | dump_to_rootfs>
o Used to set action to preform in case dumping to intended target fails.
o If no default action is specified, "reboot" is assumed default.
reboot: If the default action is reboot simply reboot the system and loose the core that you are trying to retrieve.
halt: If the default action is halt, then simply halt the system after attempting to capture a vmcore, regardless
of success or failure.
poweroff: The system will be powered down
shell: If the default action is shell, then drop to an shell session inside the initramfs from where you can try to
record the core manually. Exiting this shell reboots the system. Note: kdump uses bash as the default shell.
dump_to_rootfs: If non­root dump target is specified, the default action can be set as dump_to_rootfs. That means when
dump to target fails, dump vmcore to rootfs from initramfs context and reboot.
Eg:
default shell
force_rebuild <0 | 1>
o By default, kdump initrd only will be rebuilt when necessary.
o Specify 1 to force rebuilding kdump initrd every time when kdump service starts.
Eg:
force_rebuild 1

6. dracut_args <arg(s)>
o Use to pass extra dracut options when rebuilding kdump initrd.
Eg:
dracut_args ­­omit­drivers "cfg80211 snd" ­­add­drivers "ext2 ext3"
fence_kdump_args <arg(s)>
o Command line arguments for fence_kdump_send (it can contain all valid arguments except hosts to send notification
to).
Eg:
fence_kdump_args ­p 7410 ­f auto ­c 0 ­i 10
fence_kdump_nodes <node(s)>
o List of cluster node(s) separated by space to send fence_kdump notification to (this option is mandatory to enable
fence_kdump).
Eg:
fence_kdump_nodes node1 node2
** How to configure kdump ? **
o Install "kexec­tools" package.
Eg:
# dnf install kexec­tools
o Verify "kexec­tools" package installed on the system.
# rpm ­q kexec­tools
kexec­tools­2.0.8­8.fc22.x86_64
o Set kernel boot parameter "crashkernel" to reserve memory for crash/kdump kernel.
Eg:
# cat /etc/sysconfig/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system­release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet crashkernel=128M"
GRUB_DISABLE_RECOVERY="true"
Two different formats for "crashkernel" parameter.
crashkernel=128M
crashkernel=512M­2G:64M,2G­:128M
The size of crashkernel depends on;
o Machine Architecture.
o Total amount of installed system memory.
o [128 MB + 4 bits for every 4KB page]
*************************************************************************************************************
| Platform | Recommended values |
*************************************************************************************************************
| i386 and x86­64 | 64M for small machines (about 2 GiB of RAM, 4 cores) and 128M for larger machines. |
| PPC64 | 128M for small machines and 256M for larger machines. |
| IA64 | 256M for small machines, 512M for medium machines and 1G and more for large machines. |
*************************************************************************************************************
o Rebuild the grub2 configuration file for changes to reflect in grub.cfg file.
Eg:
# grub2­mkconfig ­o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz­4.0.4­301.fc22.x86_64
Found initrd image: /boot/initramfs­4.0.4­301.fc22.x86_64.img
Found linux image: /boot/vmlinuz­0­rescue­7b1d220372f00d7c2c3bb933e28756e2
Found initrd image: /boot/initramfs­0­rescue­7b1d220372f00d7c2c3bb933e28756e2.img
done

7. o Reboot the system for changes to take effect.
Eg:
# reboot
o Verify the memory reserved for crash/kdump kernel.
Eg:
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz­4.0.4­301.fc22.x86_64 root=/dev/mapper/fedora­root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root
rhgb quiet crashkernel=128M
# cat /sys/kernel/kexec_crash_loaded
1
# cat /sys/kernel/kexec_crash_size
134217728
# bc ­q
scale=2
134217728/2^20
128.00 <<<­­­­{ 128 MiB }
**OR**
# grep ­i crash /proc/iomem
2d000000­34ffffff : Crash kernel
# printf "%dn" 0x34ffffff
889192447
# printf "%dn" 0x2d000000
754974720
# bc ­q
scale=2
(889192447­754974720)
134217727
./2^20
127.99 <<<­­­­{ 128 MiB }
o Configure the dump target (default is /var/crash )
Eg:
# vi /etc/kdump.conf
path /var/crash
Note: 1 Currently only one dump target and path may be configured at once.
Note: 2 The free disk space on the target dump device must be equal to the physical memory on the system.
o Set the message level and dump level to filter the pages from vmcore.
Eg:
# vi /ect/kdump.conf
path /var/crash
core_collector makedumpfile ­l ­­message­level 1 ­d 31
o Set the action to preform in case dumping to intended target fails.
Eg:
# vi /ect/kdump.conf
path /var/crash
core_collector makedumpfile ­l ­­message­level 1 ­d 31
default reboot
Note: If no default action is specified, "reboot" is assumed default.
o Enable the kdump service.
Eg:
# systemctl enable kdump
Created symlink from /etc/systemd/system/multi­user.target.wants/kdump.service to
/usr/lib/systemd/system/kdump.service.
o Start the kdump service.
Eg:
# systemctl start kdump.service
OR

8. # kdumpctl start
o Verify kdump service status.
Eg:
# systemctl status kdump.service
kdump.service ­ Crash recovery kernel arming●
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: disabled)
Active: active (exited) since Sat 2015­06­20 21:49:06 IST; 7h ago
Process: 866 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
Main PID: 866 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/kdump.service
Jun 20 21:49:03 fedora22.kernel.org systemd[1]: Starting Crash recovery kernel arming...
Jun 20 21:49:06 fedora22.kernel.org kdumpctl[866]: kexec: loaded kdump kernel
Jun 20 21:49:06 fedora22.kernel.org kdumpctl[866]: Starting kdump: [OK]
Jun 20 21:49:06 fedora22.kernel.org systemd[1]: Started Crash recovery kernel arming.
OR
# kdumpctl status
Kdump is operational
** How to test kdump setup ? **
o Enable magic key "SysRq" to trigger kernel panic.
Eg:
# echo 1 > /proc/sys/kernel/sysrq
o Trigger SysRq panic event (c) to crash the system manually.
Eg:
# echo c > /proc/sysrq­trigger
** How to verify kernel crash dump (vmcore) ? **
o Check directory set by "path" in kdump.conf file for "vmcore" and "vmcore­dmesg.txt" files.
Eg:
# tree /var/crash/
/var/crash/
127.0.0.1­2015.06.08­17:25:32└──
vmcore├──
vmcore­dmesg.txt└──
1 directory, 2 files
o Install crash package.
Eg:
# dnf install crash
o Open kernel crash dump (vmcore) using "crash" utility.
Eg:
# crash ­­osrelease vmcore
OR
# crash ­d 1 vmcore| egrep "nodename|release|machine|dump_level"
compressed kdump: header­>utsname.machine: x86_64
machine_type: 62 (EM_X86_64)
nodename: fedora22.kernel.org
release: 4.0.4­301.fc22.x86_64
machine: x86_64
dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
**Reference:**
# less /usr/share/doc/kexec­tools/kexec­kdump­howto.txt
https://www.kernel.org/doc/Documentation/kdump/kdump.txt
https://www.kernel.org/doc/ols/2007/ols2007v1­pages­167­178.pdf
http://people.redhat.com/nhorman/papers/ols­slides.pdf