This document provides an introduction to kdump and kernel crash dump analysis. It discusses kexec, which allows fast rebooting by loading a new kernel from an already running kernel. Kdump uses kexec to boot a capture kernel to analyze the state of a crashed production kernel and capture a vmcore dump file. The document outlines how to configure kdump by reserving memory, setting the dump target, enabling the kdump service, and testing a crash. Kernel crash dumps can be analyzed using the crash utility to help debug issues.

1. Kdump & Kernel crash dump analysis
Part: 1
Introduction to kdump
Part: 2
Introduction to kernel crash dump analysis
http://people.redhat.com/gtiwari/fudcon/

2. Introduction to Kdump
Buland Singh
Red Hat

3. Introduction to Kdump
What is kexec ?
How does it work ?
What is kdump ?
How does it work ?
How to configure kdump ?
How to test kdump setup ?
How to verify kernel crash dump (vmcore) ?
References

4. What is kexec ?
kexec is a fast­boot mechanism that allows booting a kernel
from the context of an already running kernel without going
through BIOS or passing through the boot­loader stage.
kexec performs the function of the boot loader from within the
kernel and when the system boots a kernel with kexec, it skips
the boot­loader stage.
This has the effect of reducing the time required for a reboot.

5. Kexec include two components:
User Space Tool : /usr/sbin/kexec
o The binary /usr/sbin/kexec is provided "kexec­tools" package.
Kernel System Call : kexec_load()
o The Linux kernel configuration "CONFIG_KEXEC" enables the
kexec system call.
# grep "CONFIG_KEXEC=" /boot/config­4.0.4­301.fc22.x86_64
CONFIG_KEXEC=y

6. Use of kexec:
Faster system reboots.
Booting without GRUB or LILO boot loader.
Saving the dump of a crashed kernel.

7. Using kexec: 2 Step process
Loading a new kernel into the physical memory.
# kexec ­l
Rebooting with the pre­loaded kernel.
# kexec ­e

8. Kexec Design:

9. Kexec command:
Command to load the kernel in memory:
# kexec ­l <kernel­image> ­­initrd=<initrd­image> ­­append=<command­line­options>
kernel­image : Path of the kernel file.
initrd­image : Path of the initrd image to be used during boot.
Command­line­options : The command­line parameters for new kernel.
Command to boot with the pre­loaded kernel:
# kexec ­e
Command to unload the kernel which was loaded with the kexec ­l command.
# kexec ­u

10. Kexec command:
Command to load a kernel to boot at the time of kernel panic.
# kexec ­p <kernel­image> ­­initrd=<initrd­image> ­­append=<command­line­options>
Command to unload the kernel which was loaded with the kexec ­p command.
# kexec ­u ­p
Command to check the status of kexec kernel.
# cat /sys/kernel/kexec_loaded
0 – kernel is not loaded in the memory.
1 – kernel is loaded in the memory.

11. Demo:

12. What is kdump ?
Kdump is a reliable kexec based crash dumping mechanism for Linux.
Kdump functionality is broken mainly in two components;
o User space (kexec)
/usr/sbin/kexec
# grep "CONFIG_KEXEC=" /boot/config­4.0.4­301.fc22.x86_64
CONFIG_KEXEC=y
o Kernel space (kdump)
# grep CRASH_DUMP /boot/config­4.0.4­301.fc22.x86_64
CONFIG_CRASH_DUMP=y

13. What is kdump ?
Kdump need two different kernel:
o Standard (production) kernel.
o Crash (capture) kernel.
The crash dumps are captured from the context of a freshly booted kernel; not from the context of the
crashed kernel.
Kdump can be configured to dump either directly to a device, to a file, or to some remote location
on the network via NFS or SSH.
Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64, and s390/s390x architectures.

14. Kdump Design:

15. How does it works ?
Whenever the standard (production) kernel crashes, kdump uses kexec to boot
into a crash (capture) kernel.
This crash (capture) kernel boots with very little pre­reserved memory and
captures the dump image.
The standard (production) kernel reserves a section of memory that the
crash (capture) kernel uses to boot.
Be aware that the memory reserved for the kdump kernel at boot time cannot
be used by the standard kernel.
Kexec enables booting the capture kernel without going through the BIOS, so
contents of the first kernel's memory are preserved, which is essentially
the kernel crash dump.

16. What is kernel crash dump (vmcore) ?
Kernel crash dump captures the state of kernel at the moment
of panic.
Kernel crash dump is a snapshot of the physical memory at the
time of crash.

17. Who needs kernel crash dump & why ?
Kernel developers :
To identify the cause of kernel panic. It helps them to fix
the bug.
Enterprise customer :
To patch the servers with a bug fix kernel package in­order to
avoid the downtime in future.

18. Kdump Configuration files:
The following two configuration files are used to configure kdump.
/etc/sysconfig/kdump
o Configuration file for kexec kernel.
o Use to pass additional boot parameters to kexec kernel.
/etc/kdump.conf
o Configuration file for kdump service.
o Use to set dump taget (local/remote).
o Use to set default action in case kdump failed.

19. How to configure kdump ?
Install kexec­tools package.
Reserve memory for kdump kernel.
Reboot.
Set dump target (Local/Remote) for kdump.
Enable and start kdump service.

20. Demo:

21. Step: 1
Install "kexec­tools" package.
# dnf install kexec­tools
Verify "kexec­tools" package.
# rpm ­q kexec­tools
kexec­tools­2.0.8­8.fc22.x86_64

22. Step: 2
Set kernel boot parameter "crashkernel" to reserve memory for crash/kdump kernel.
# vi /etc/sysconfig/grub
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet
crashkernel=128M"
The size of crashkernel depends on;
o Machine Architecture.
o Total amount of installed system memory.
o [128 MB + 4 bits for every 4KB page]

23. Step: 3
Rebuild the grub2 configuration file.
# grub2­mkconfig ­o /boot/grub2/grub.cfg
Reboot the system for changes to take effect.
# reboot

24. Step: 4
Verify the memory reserved for crash/kdump kernel.
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz­4.0.4­301.fc22.x86_64 root=/dev/mapper/fedora­root ro
rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet crashkernel=128M
# cat /sys/kernel/kexec_crash_loaded
1
# cat /sys/kernel/kexec_crash_size
134217728

25. Step: 5
Configure the dump target (default is /var/crash )
# vi /etc/kdump.conf
path /var/crash
Note: 1
Currently only one dump target and path can be configured at once.
Note: 2
The free disk space on the target dump device must be equal to the physical
memory on the system.

26. /etc/kdump.conf
path <path>
o "path" represents the filesystem path in which vmcore will be saved.
path /var/crash
raw <partition>
o Use to set dump target to raw <partition>.
raw /dev/vg/lv_kdump

27. <fs type> <partition>
o Use to set dump target to a specific partition of a specific
filesystem type.
ext4 /dev/vg/lv_kdump
ext4 UUID=03138356­5e61­4ab3­b58e­27507ac41937

28. /etc/kdump.conf
nfs <nfs mount>
o Use to set dump target to NFS mount point.
nfs server.example.com:/export/tmp
ssh <user@server>
o Used to set dump target on remote machine.
ssh user@server.example.com

29. Step: 6
Set the message level and dump level to filter the pages from vmcore.
# vi /etc/kdump.conf
path /var/crash
core_collector makedumpfile ­l ­­message­level 1 ­d 31

30. /etc/kdump.conf
core_collector <command> <options>
o This allows you to specify the command to copy the vmcore.
core_collector makedumpfile ­c ­­message­level 1 ­d 31
­c : Use to compress the vmcore file.
­d : Use to set the dump_level.
o "dump_level" is use to decide which pages to be filter from the vmcore file.

31. Step: 7
Set the action to preform in case dumping to intended target fails.
# vi /ect/kdump.conf
path /var/crash
core_collector makedumpfile ­l ­­message­level 1 ­d 31
default reboot
Note:
If no default action is specified, "reboot" is assumed default.

32. Step: 8
Enable the kdump service.
# systemctl enable kdump
Start the kdump service.
# systemctl start kdump.service
# kdumpctl start
Verify kdump service status.
# systemctl status kdump.service
# kdumpctl status

33. How to test kdump setup ?
Enable magic key "SysRq" to trigger kernel panic.
# echo 1 > /proc/sys/kernel/sysrq
Trigger SysRq panic event (c) to crash the system.
# echo c > /proc/sysrq­trigger

34. How to verify kernel crash dump (vmcore) ?
Verify /var/crash directory for vmcore and vmcore­dmesg.txt file.
# tree /var/crash/
/var/crash/
└── 127.0.0.1­2015.06.08­17:25:32
├── vmcore
└── vmcore­dmesg.txt
Open kernel crash dump (vmcore) using crash utility.
# crash ­­osrelease vmcore
# crash ­d 1 vmcore| egrep "nodename|release|machine|dump_level"

35. References:
https://www.kernel.org/doc/Documentation/kdump/kdump.txt
https://www.kernel.org/doc/ols/2007/ols2007v1­pages­167­178.pdf
http://people.redhat.com/nhorman/papers/ols­slides.pdf
/usr/share/doc/kexec­tools/kexec­kdump­howto.txt

36. Questions ?