Kdump is a kernel crash dumping mechanism that uses kexec to load a separate crash kernel to capture a kernel memory dump (vmcore file) when the primary kernel crashes. It can be configured to dump the vmcore file to local storage or over the network. Testing involves triggering a kernel panic using SysRq keys which causes the crash kernel to load and dump diagnostic information to the configured target path for analysis.

1. Kdump and Kernel Crash Dump Analysis
Buland Singh

2. The Linux Kernel
10101010101001010101010101010
10101010100101010101011010101
1100110001111010101
24+ Million
lines of code
Largest & Successful
collaborative development project in the
history of computing
15,600+
individual Linux developers worldwide
from over 1,400 diferent companies
Runs on
Mobile Devices to
Super Computers

3. The Linux kernel is the heart of the operating system
Memory
Management
Process
Management
IO Device ManagementPower Management
The Linux Kernel

4. 10100101010
10101010101
01010101001
01010101011
01010111001
10001111011
The Linux Kernel
Bug in diferent subsystems of kernel

Memory management

Scheduler

Networking

File Systems

Power Management

Hardware Architecture

USB

Device driver Infrastructure
Deadlock conditions Memory Corruption
ffff8101d9ba5c00: aeb038654d2a6c47 ffff81024d2ed3c0
ffff8101d9ba5c10: ffff81024d2ed3c0 0000000000000246
ffff8101d9ba5c20: 0000000000000008 00007fff41966040
ffff8101d9ba5c30: 0000000000000007 ffffffffffffff7f
ffff8101d9ba5c40: 0000000000000007 0000000000000000
ffff8101d9ba5c50: ffff81024d2ed3c0 aeb038654d2a6c47
ffff8101d9ba5c60: ffffffffffffffff 0000000000000000
ffff8101d9ba5c70: ffffffff8007bd5a 0000000001689000
ffff8101d9ba5c80: ffffffff80061000 00000000006c3000
ffff8101d9ba5c90: ffffffff804c3000 00000000006c4000
ffff8101d9ba5ca0: ffffffff804c4000 00000000006c5000
ffff8101d9ba5cb0: ffffffff804c5000 00000000006c6000
ffff8101d9ba5cc0: ffffffff804c6000 00000000006c7000
ffff8101d9ba5cd0: ffffffff804c7000 00000000006c8000
ffff8101d9ba5ce0: ffffffff804c8000 00000000006c9000
ffff8101d9ba5cf0: ffffffff804c9000 0000000001688000
ffff8101d9ba5d00: ffffffff800aef27 00000000029ee3ff
ffff8101d9ba5d10: 0000000000000000 ffff8101d9ba5e98
ffff8101d9ba5d20: ffffffff800b0037 0000000000000000
ffff8101d9ba5d30: 0000000000000000 000000001c6829f0
P0 P1
Resource 2
Resource 1
WaitingforAssignedto
AssignedtoWaitingfor

5. The Linux Kernel
90%
of the public cloud workload runs on Linux
80%
of the IOT market runs on Linux
99%
of the supercomputers runs on Linux
90%
of the world’s stock exchanges are powered
by Linux

6. Imagine what would happen
if kernel crashed on those systems?

7. Business impact
Production down
No “tatkal” reservation
No FacebookNo online banking
No Whatsapp
No Twitter
Unavailability of services

8. How do you troubleshoot the kernel crash?

9. Kexec
Basics of kexec
Usage of kexec
Kdump
Basics of kdump
Confguration of kdump service
Kernel Crash Dump Analysis
Environment setup for kernel crash dump analysis
Basic commands of crash utility
Agenda

10. Kexec

11. 
Kexec is a mechanism to boot second kernel from the context of frst kernel.

It performs the function of the boot loader from within the kernel.

It skips hardware initialization stage (performed by BIOS/UEFI) thus reboot is faster.

Supported architectures:

Primary architectures (eg: x86 and x86_64).

Secondary architectures (eg: ppc, ppc64, s390, s390x, arm, arm64 etc).
Kexec

12. Kexec mechanism include two components:

Kernel space support:

kexec_load() and reboot(LINUX_REBOOT_CMD_KEXEC) system calls.

kexec_load() system call loads a new kernel into memory.

reboot() system call executes that kernel later.

The Linux kernel confguration items:
CONFIG_KEXEC & CONFIG_CRASH_DUMP enables kexec_load() & reboot() system call.
Kexec

User space support:

/usr/sbin/kexec binary provided by 'kexec-tools' package.

13. Kexec
# kexec -l <kernel-image> --initrd=<initramfs-image> --reuse-cmdline
Kexec mechanism can be used to load a new kernel for regular boot.
Booting regular kernel using kexec mechanism is a two step process
kernel-image : Path of the kernel fle.
initramfs-image : Path of the initramfs image to be used during boot.
--reuse-cmdline : The command line parameters for new kernel
Boot with the loaded kernel.
# kexec -e
1
2
Load a new kernel into the physical memory from the context of running kernel.

14. Kexec Control Flow
First Kernel First Kernel Second Kernel
Setup Page
initramfs
kexec -l
Physical Memory
Load
Second
Kernel
kexec -e
Execute
Second
Kernel
Second Kernel Pages

15. Steps to boot with regular kernel using kexec
# yum install kexec-tools
Load a new kernel into the physical memory.
# kexec -l /boot/vmlinuz-3.10.0-693.2.2.el7.x86_64 --initrd=/boot/initramfs-3.10.0-
693.2.2.el7.x86_64.img --reuse-cmdline
Check the status of kexec kernel.
# cat /sys/kernel/kexec_loaded
1
0 – kernel is not loaded.
1 – kernel is loaded
Install kexec-tools package.
# kexec -e
Boot to the loaded kernel.
Note: Unload the kernel which was
loaded with the kexec -l command.
# kexec -u
1
2
3
4

16. Kexec
Booting with crash kernel using kexec mechanism is a three step process:
Reserve memory using kernel boot parameter 'crashkernel'.
Syntax:
The size of 'crashkernel' depends on:

Size of initramfs fle, machine IO devices complexity

Number of CPUs to be used in crash kernel

Total amount of installed system memory
128 MiB + 4 bits for every 4KB page
crashkernel=size[KMG][@offset[KMG]]
Eg: crashkernel=128M@16M or crashkernel=128M
crashkernel=range1:size1[,range2:size2,...][@offset]
Eg: crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-:768M
crashkernel=auto
1
Kexec mechanism can also be used to load a new kernel to boot at the time of panic.

17. Kexec
# kexec -p <kernel-image> --initrd=<initramfs-image> --reuse-cmdline
Load a new kernel into the physical memory to boot at the time of kernel panic.
kernel-image : Path of the kernel fle.
initramfs-image : Path of the initramfs image to be used during boot.
--reuse-cmdline : The command line parameters for new kernel
Trigger a panic/crash by using SysRq facility.
# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger
2
3

18. Kexec Control Flow
Regular Kernel Regular Kernel
ELF Core Headers
Regular Kernel
kexec -p
Physical Memory
Load capture
kernel
Crash
Boot into
capture kernel
initramfs
Crash Kernel
Setup Code
ELF Core Headers
initramfs
Crash Kernel
Setup Code
Reserved Memory
for Crash Kernel
using
‘crashkernel’
Panic/SysRq + c

19. Steps to boot with kernel using kexec at the time of panic
# vi /etc/sysconfig/grub
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap crashkernel=auto"
# grep "Crash kernel" /proc/iomem
2d000000-370fffff : Crash kernel
# cat /sys/kernel/kexec_crash_size
168820736
Verify the reserved memory
Reserve memory using kernel boot parameter 'crashkernel'
# grub2-mkconfig -o /boot/grub2/grub.cfg
Rebuild grub2 confguration fle
# reboot
Reboot the system
1
2
3
4

20. Load a new kernel into the physical memory to boot at the time of kernel panic.
Steps to boot with kernel using kexec at the time of panic
# kexec -p /boot/vmlinuz-3.10.0-693.2.2.el7.x86_64 --initrd=/boot/initramfs-3.10.0-
693.2.2.el7.x86_64.img --reuse-cmdline
Check the status of kexec crash kernel.
# cat /sys/kernel/kexec_crash_loaded
1
0 – crash kernel is not loaded.
1 – crash kernel is loaded
Note: Unload the crash kernel which was
loaded with the kexec -p command.
# kexec -u
# echo 1 > /proc/sys/kernel/sysrq
# cat /proc/sys/kernel/sysrq
1
0 – SysRq disabled 1 – SysRq enabled
# echo c > /proc/sysrq-trigger
Trigger a crash by using SysRq
5
6
7

21. 
Reduces the time required for the system reboot.

Allows booting without bootloader (GRUB or LILO etc).

Allows to capture the kernel crash dump for later investigation.
Usage of kexec

22. Kdump

23. 
Kdump is a reliable kexec based crash dumping mechanism for Linux kernel.

Kdump needs two diferent kernels:

Standard (production) kernel.

Crash (capture) kernel.
Kdump

24. 
Kdump can be confgured to dump on local or remote system.

The supported dump targets are:

ext2/3/4, btrfs and xfs flesystems on direct attached single drives.

Hardware RAID logical drives or mdraid arrays.

LVM devices.

NFS flesystem.

Remote SSH connections.

Multipath based storage over iSCSI software initiator

Supported architectures:

Primary architectures (eg: x86, x86_64).

Secondary architectures (eg: ppc, ppc64, s390, s390x, arm, arm64 etc).
Kdump

25. Kdump Control Flow
Standard
(Production)
Kernel
Physical Memory
Crash
Boot into
capture kernel
Crash (Capture)
Kernel Panic/SysRq + c
Standard
(Production)
Kernel
Crash (Capture)
Kernel
Remote
Disk
Local Disk
NFS / ssh
makedumpfile
cp / dd
/proc/vmcore

26. The control fow between the two works as follows:

Standard (production) kernel crashed.

Crash (capture) kernel boots.

kdump initramfs loads and runs /init.

Extract kernel ring bufers from /proc/vmcore

save as vmcore-dmesg.txt at </path> location.

Copies memory image with 'makedumpfile' from /proc/vmcore

save as vmcore at </path> location.

Reboot the system with standard (production) kernel.
Kdump Control Flow

27. The following two fles are used to confgure kdump.
/etc/sysconfig/kdump

Confguration fle for kexec kernel.

Use to pass additional boot parameters to kexec kernel.
Kdump configuration files
/etc/kdump.conf

Confguration fle for kdump service.

Use to set dump taget (local/remote).

Use to set default action in case kdump failed.

Use to flter pages from the crash dump.

28. Install 'kexec-tools' package.
# yum install kexec-tools
How to configure kdump
# vi /etc/sysconfig/grub
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap crashkernel=auto"
Reserve memory using kernel boot parameter 'crashkernel' for crash/kdump kernel.
# grub2-mkconfig -o /boot/grub2/grub.cfg
Rebuild grub2 confguration fle
# reboot
Reboot the system
# grep "Crash kernel" /proc/iomem
2d000000-370fffff : Crash kernel
Verify the reserved memory
1
2
3
4
5

29. Install 'kexec-tools' package.
# yum install kexec-tools
How to configure kdump
# vi /etc/sysconfig/grub
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap crashkernel=auto"
Reserve memory using kernel boot parameter 'crashkernel' for crash/kdump kernel.
# grub2-mkconfig -o /boot/grub2/grub.cfg
Rebuild grub2 confguration fle
# reboot
Reboot the system
# grep "Crash kernel" /proc/iomem
2d000000-370fffff : Crash kernel
Verify the reserved memory
1
2
3
4
5

30. Confgure the dump target (default is /var/crash).
# vi /etc/kdump.conf
path /var/crash
How to configure kdump
Set dump level to flter the pages from vmcore (optional).
# vi /etc/kdump.conf
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
core_collector option allows you to specify the command to copy vmcore.
c : Use to compress vmcore file.
d : Use to set the dump_level.
dump_level is use to decide which pages to be filter from vmcore file.
1 : Exclude the pages filled with zero.
2 : Exclude the non-private cache pages.
4 : Exclude all cache pages.
8 : Exclude the user process data pages.
16 : Exclude the free pages.
6
7

31. Set the action to preform in case dumping to intended target fails (default is reboot).
# vi /etc/kdump.conf
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
default reboot
How to configure kdump
Enable and start the kdump service.
# systemctl enable kdump
# systemctl start kdump.service or # kdumpctl start
Verify kdump service status.
# systemctl status kdump.service
● kdump.service - Crash recovery kernel arming
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2018-07-21 15:34:19 IST; 8min ago
Main PID: 2038 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/kdump.service
8
9
10

32. Enable magic key 'SysRq' to trigger kernel panic.
# echo 1 > /proc/sys/kernel/sysrq
How to test kdump setup
Trigger SysRq panic event (c) to crash the system.
# echo c > /proc/sysrq-trigger
1
2

33. Verify /var/crash directory for vmcore and vmcore-dmesg.txt fle.
# tree /var/crash/
/var/crash/
└── 127.0.0.1-2018-07-21-15:43:10
├── vmcore
└── vmcore-dmesg.txt
1 directory, 2 files
How to verify kernel crash dump (vmcore)

34. 
Kernel crash dump (aka vmcore) is a snapshot of the physical memory
(RAM) at the time of crash.

It contains all the data which are present in the physical memory at the time
of crash.
What is kernel crash dump (vmcore)

35. Kernel developers :

To understand the nature and cause of kernel panic/system hang.

To fx the bug in kernel space.
Enterprise customers :

To determine the cause of an unexpected system reboot or system hang.

To patch the system with bug fx kernel package.
Who needs kernel crash dump

36. Kernel Crash Dump Analysis

37. Kernel symbol fle (vmlinux) of the crashed
kernel.
Provided by 'kernel-debuginfo' package.
Eg: kernel-debuginfo-3.10.0-862.3.2.el7.x86_64.rpm
crash binary (/usr/bin/crash)
Provided by 'crash' package.
Eg: crash-7.2.0-6.el7.x86_64
No cross platform support.
Use 64 bit version of crash for vmcore
captured from 64 bit kernel.
Environment Setup
# yum install crash
# yum install kernel-debuginfo-3.10.0-862.3.2.el7
Kernel crash dump fle (vmcore)
Captured using kdump/xendump/vmss2core etc
mechanisms.
System of same architecture as of
kernel from which vmcore was
captured.
Kernel crash dump captured from x86_64
machine can only be viewed on x86_64
machine.
Source code of the crashed kernel
(optional)
Provided by "src" rpm package of kernel.
Eg: kernel-3.10.0-862.3.2.src.rpm
2
1
5
4
3
# uname -m
x86_64

38. Invoking crash
Typical postmortem debugging: [ ofine ]
# crash --osrelease vmcore
3.10.0-862.3.2.el7.x86_64
# crash vmcore /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/vmlinux
# crash </path/to/vmlinux> </path/to/vmcore>
Live memory debugging: [ online ]
# crash /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/vmlinux
/dev/crash used by default for live memory image.
# crash </path/to/vmlinux>
1
2

39. Basic commands of crash utility

40. sys command: system-specific data
crash> sys
KERNEL: /usr/lib/debug/lib/modules/3.10.0-693.el7.x86_64/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 16
DATE: Fri May 11 15:34:17 2018
UPTIME: 83 days, 05:38:37
LOAD AVERAGE: 0.31, 0.54, 0.52
TASKS: 1222
NODENAME: server1.redhat.com
RELEASE: 3.10.0-693.el7.x86_64
VERSION: #1 SMP Thu Jul 6 19:56:57 EDT 2017
MACHINE: x86_64 (2600 Mhz)
MEMORY: 127.9 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"
sys - System information
Date & time of crash
Cause of Panic
Version of running Kernel
Load on the system: 1-5-15 minute interval

41. sys command: system-specific data
crash> sys -i
DMI_BIOS_VENDOR: Dell Inc.
DMI_BIOS_VERSION: 2.6.0
DMI_BIOS_DATE: 10/26/2017
DMI_SYS_VENDOR: Dell Inc.
DMI_PRODUCT_NAME: PowerEdge R730
DMI_PRODUCT_VERSION:
DMI_PRODUCT_SERIAL: 5PMD8M2
DMI_PRODUCT_UUID: 4C4E4844-5000-4D10-8077-B58032F384D32
DMI_BOARD_VENDOR: Dell Inc.
DMI_BOARD_NAME: 073T6E
DMI_BOARD_VERSION: A07
DMI_BOARD_SERIAL: .5PMCD8M2.CNFCP007EDGUH.
DMI_BOARD_ASSET_TAG:
DMI_CHASSIS_VENDOR: Dell Inc.
DMI_CHASSIS_TYPE: 23
DMI_CHASSIS_VERSION:
DMI_CHASSIS_SERIAL: 5PMCD8M2
DMI_CHASSIS_ASSET_TAG:
sys -i - Display the DMI string data if available in the kernel
Hardware manufacturer
Model Name
BIOS Revision

42. sys command: system-specific data
crash> sys -c | head
NUM SYSTEM CALL FILE AND LINE NUMBER
0 sys_read ../fs/read_write.c: 569
1 sys_write ../fs/read_write.c: 584
2 sys_open ../fs/open.c: 1063
3 sys_close ../fs/open.c: 1124
4 sys_newstat ../fs/stat.c: 247
5 sys_newfstat ../fs/stat.c: 285
6 sys_newlstat ../fs/stat.c: 258
7 sys_poll ../fs/select.c: 963
8 sys_lseek ../fs/read_write.c: 336
sys -c - Display the system call table:
Syscall
number
System call name Kernel source file

43. sys command: system-specific data
crash> sys -t
TAINTED_MASK: 3000 OE
crash> eval -b 0x3000
hexadecimal: 3000 (12KB)
decimal: 12288
octal: 30000
binary: 0000000000000000000000000000000000000000000000000011000000000000
bits set: 13 12
Kernel Source: include/linux/kernel.h
...
513 #define TAINT_OOT_MODULE 12
514 #define TAINT_UNSIGNED_MODULE 13
...
sys -t - Display kernel taint information.
Kernel Taint flag

44. crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 2002525 7.6 GB ----
FREE 368440 1.4 GB 18% of TOTAL MEM
USED 1634085 6.2 GB 81% of TOTAL MEM
SHARED 1034345 3.9 GB 51% of TOTAL MEM
BUFFERS 741 2.9 MB 0% of TOTAL MEM
CACHED 1178665 4.5 GB 58% of TOTAL MEM
SLAB 36254 141.6 MB 1% of TOTAL MEM
TOTAL HUGE 0 0 ----
HUGE FREE 0 0 0% of TOTAL HUGE
TOTAL SWAP 1992702 7.6 GB ----
SWAP USED 0 0 0% of TOTAL SWAP
SWAP FREE 1992702 7.6 GB 100% of TOTAL SWAP
COMMIT LIMIT 2993964 11.4 GB ----
COMMITTED 2364322 9 GB 78% of TOTAL LIMIT
kmem command: System memory
kmem -i – Display memory usage information.
General memory usage
Hugepage details
Swap details
Commit limit

45. crash> kmem -h
HSTATE SIZE FREE TOTAL NAME
ffffffff8d183c60 2MB 0 0 hugepages-2048kB
kmem command: System memory
kmem -h - Display hugepage information.
crash> kmem -s | head
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff8e3a3fb17600 rfs_file_cache 288 37 98 7 4k
ffff8e3a3fb17500 rfs_inode_cache 384 44805 44835 2135 8k
ffff8e383689ab00 nf_conntrack_ffffffff8ccfc940 312 61 276 23 4k
ffff8e383689aa00 xfs_dqtrx 528 0 0 0 8k
ffff8e383689a900 xfs_dquot 488 0 0 0 8k
ffff8e383689a800 xfs_ili 168 2372 2376 99 4k
ffff8e383689a700 xfs_inode 936 36821 36839 2167 16k
ffff8e383689a600 xfs_efd_item 416 6 285 15 8k
ffff8e383689a500 xfs_btree_cur 208 0 38 2 4k
kmem -s – Displays basic kmalloc() slab data.
Hugepage details
Slab cache name

46. crash> swap
SWAP_INFO_STRUCT TYPE SIZE USED PCT PRI FILENAME
ffff8e3835c60600 PARTITION 1679356k 0k 0% -1 /dev/dm-0
ffff8e3835c64c00 PARTITION 6291452k 0k 0% -2 /dev/dm-2
swap command: Swap device info
swap - Displays swap device information.

47. crash> ps | head
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff8cc16480 RU 0.0 0 0 [swapper/0]
0 0 1 ffff8e397cef2f70 RU 0.0 0 0 [swapper/1]
1 0 0 ffff8e397ce58000 IN 0.0 131616 3976 systemd
2 0 1 ffff8e397ce58fd0 IN 0.0 0 0 [kthreadd]
3 2 0 ffff8e397ce59fa0 IN 0.0 0 0 [ksoftirqd/0]
5 2 0 ffff8e397ce5bf40 IN 0.0 0 0 [kworker/0:0H]
7 2 0 ffff8e397ce5dee0 IN 0.0 0 0 [migration/0]
8 2 0 ffff8e397ce5eeb0 IN 0.0 0 0 [rcu_bh]
9 2 0 ffff8e397cef0000 RU 0.0 0 0 [rcu_sched]
ps command: Display process status information
ps - Show the process status of all current tasks.
PID Of
process
Struct task_struct *
Process
State
Percent of
Memory used
Resident
Set Size
Process name

48. crash> ps -u | head
PID PPID CPU TASK ST %MEM VSZ RSS COMM
1 0 0 ffff8e397ce58000 IN 0.0 131616 3976 systemd
403 21790 1 ffff8e385f510000 IN 0.3 1009284 28036 db2fmp
404 21790 1 ffff8e3a1aef3f40 IN 0.3 1009284 28036 db2fmp
558 1 1 ffff8e383694cf10 IN 0.0 36988 1516 systemd-journal
582 1 1 ffff8e3a32d12f70 IN 0.0 348200 3768 lvmetad
598 1 0 ffff8e3a32d16eb0 IN 0.0 44944 3220 systemd-udevd
769 1 0 ffff8e3a2fee3f40 IN 0.0 114624 1816 auditd
787 1 0 ffff8e3a2fee2f70 IN 0.0 114624 1816 auditd
788 769 0 ffff8e3a2fee0000 IN 0.0 80268 988 audispd
ps command: Display process status information
ps -u - Restrict the output to only user tasks.
crash> ps -k | head
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff8cc16480 RU 0.0 0 0 [swapper/0]
0 0 1 ffff8e397cef2f70 RU 0.0 0 0 [swapper/1]
2 0 1 ffff8e397ce58fd0 IN 0.0 0 0 [kthreadd]
3 2 0 ffff8e397ce59fa0 IN 0.0 0 0 [ksoftirqd/0]
5 2 0 ffff8e397ce5bf40 IN 0.0 0 0 [kworker/0:0H]
7 2 0 ffff8e397ce5dee0 IN 0.0 0 0 [migration/0]
8 2 0 ffff8e397ce5eeb0 IN 0.0 0 0 [rcu_bh]
9 2 0 ffff8e397cef0000 RU 0.0 0 0 [rcu_sched]
10 2 0 ffff8e397cef0fd0 IN 0.0 0 0 [lru-add-drain]
ps -k - Restrict the output to only kernel threads.

49. crash> ps -S
RU: 18
IN: 438
ps command: Display process status information
ps -S - Display a summary consisting of the number of tasks in a task state.
crash> ps -p 558
PID: 0 TASK: ffffffff8cc16480 CPU: 0 COMMAND: "swapper/0"
PID: 1 TASK: ffff8e397ce58000 CPU: 0 COMMAND: "systemd"
PID: 558 TASK: ffff8e383694cf10 CPU: 1 COMMAND: "systemd-journal"
ps -p - Display the parental hierarchy of selected or all, tasks.
State of process
Number of tasks
Parent task
Child task

50. crash> ps -c 558
PID: 558 TASK: ffff8e383694cf10 CPU: 1 COMMAND: "systemd-journal"
(no children)
ps command: Display process status information
ps -c - display the children of selected or all, tasks.
crash> ps -m | head
[0 00:00:00.000] [RU] PID: 20684 TASK: ffff8e39f9f39fa0 CPU: 0 COMMAND: "awk"
[0 00:00:00.000] [RU] PID: 1786 TASK: ffff8e3a36228000 CPU: 1 COMMAND: "ds_am"
[0 00:00:00.000] [IN] PID: 558 TASK: ffff8e383694cf10 CPU: 1 COMMAND: "systemd-journal"
[0 00:00:00.010] [RU] PID: 20685 TASK: ffff8e39f9f3bf40 CPU: 0 COMMAND: "awk"
[0 00:00:00.010] [IN] PID: 7 TASK: ffff8e397ce5dee0 CPU: 0 COMMAND: "migration/0"
[0 00:00:00.010] [RU] PID: 20686 TASK: ffff8e3a358e3f40 CPU: 1 COMMAND: "env"
[0 00:00:00.010] [RU] PID: 20683 TASK: ffff8e39f9f3cf10 CPU: 1 COMMAND: "awk"
[0 00:00:00.022] [RU] PID: 20687 TASK: ffff8e3a1aef2f70 CPU: 0 COMMAND: "awk"
[0 00:00:00.021] [IN] PID: 20179 TASK: ffff8e3a29683f40 CPU: 1 COMMAND: "kworker/u4:1"
[0 00:00:00.032] [RU] PID: 20681 TASK: ffff8e3840d28fd0 CPU: 0 COMMAND: "awk"
ps -m - Display the task's last-run timestamp value.
Days Hours Minutes Seconds Milliseconds

51. crash> runq
CPU 0 RUNQUEUE: ffff8e3a3fc18b00
CURRENT: PID: 20684 TASK: ffff8e39f9f39fa0 COMMAND: "awk"
RT PRIO_ARRAY: ffff8e3a3fc18ca0
[no tasks queued]
CFS RB_ROOT: ffff8e3a3fc18ba8
[120] PID: 22270 TASK: ffff8e3a34543f40 COMMAND: "db2sysc"
[120] PID: 21796 TASK: ffff8e3a3fce1fa0 COMMAND: "db2sysc"
[120] PID: 749 TASK: ffff8e3a32d14f10 COMMAND: "xfsaild/dm-3"
[120] PID: 9 TASK: ffff8e397cef0000 COMMAND: "rcu_sched"
[120] PID: 20681 TASK: ffff8e3840d28fd0 COMMAND: "awk"
[120] PID: 20685 TASK: ffff8e39f9f3bf40 COMMAND: "awk"
[120] PID: 20687 TASK: ffff8e3a1aef2f70 COMMAND: "awk"
CPU 1 RUNQUEUE: ffff8e3a3fd18b00
CURRENT: PID: 1786 TASK: ffff8e3a36228000 COMMAND: "ds_am"
RT PRIO_ARRAY: ffff8e3a3fd18ca0
[no tasks queued]
CFS RB_ROOT: ffff8e3a3fd18ba8
[120] PID: 474 TASK: ffff8e3a32a1cf10 COMMAND: "xfsaild/dm-1"
[120] PID: 815 TASK: ffff8e3a32d10000 COMMAND: "vmtoolsd"
[120] PID: 858 TASK: ffff8e3a3622dee0 COMMAND: "in:imjournal"
[120] PID: 20686 TASK: ffff8e3a358e3f40 COMMAND: "env"
[120] PID: 20688 TASK: ffff8e39eaa28fd0 COMMAND: "env"
[120] PID: 20683 TASK: ffff8e39f9f3cf10 COMMAND: "awk"
[120] PID: 20689 TASK: ffff8e39eaa2dee0 COMMAND: "awk"
runq command: Runqueue
runq - Display the tasks on an O(1) scheduler run queue.
Running task
Task waiting in run
queue of CPU 0
Struct rq * pointer of CPU 1

52. crash> ipcs -s | head
SEM_ARRAY KEY SEMID UID PERMS NSEMS
ffff8e3a28f98240 33b0b874 502824960 1004 667 1
ffff8e38a4bec140 00000000 502857729 1004 661 1
ipcs command: System V IPC facilities
ipcs -s - Show semaphore arrays.
crash> ipcs -q | head
MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES
ffff8e38ba86f040 3d5d6f65 32768 1004 711 0 0
ffff8e38ba86f640 4d042c7f 65537 1004 711 0 0
ipcs -q - Show message queues
crash> ipcs -m
SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS
ffff8e38b8c40940 00000000 0 1004 667 34156016 1 dest
ffff8e384068a640 33b0b874 451608577 1004 667 34156016 8
ipcs -m - Show shared memory segments.

53. crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff8e397cdb6000 ffff8e397cdc8000 rootfs rootfs /
ffff8e397cf96a80 ffff8e3a344e2800 sysfs sysfs /sys
ffff8e397cf96c00 ffff8e397cdca800 proc proc /proc
ffff8e397cf96d80 ffff8e397cfc8000 devtmpfs devtmpfs /dev
ffff8e397cf96f00 ffff8e3a3fb3e000 securityfs securityfs /sys/kernel/security
ffff8e397cf97080 ffff8e3a344e3000 tmpfs tmpfs /dev/shm
ffff8e397cf97200 ffff8e397cfca000 devpts devpts /dev/pts
ffff8e397cf97380 ffff8e3a344e3800 tmpfs tmpfs /run
ffff8e397cdb6f00 ffff8e383693f800 configfs configfs /sys/kernel/config
ffff8e3a34581200 ffff8e3a3202a000 xfs /dev/mapper/rhel-root /
ffff8e3a364e7e00 ffff8e3a3202d800 autofs systemd-1 /proc/sys/fs/binfmt_misc
ffff8e3a3459a600 ffff8e3836a9c000 hugetlbfs hugetlbfs /dev/hugepages
ffff8e3a3459af00 ffff8e397cfcb000 mqueue mqueue /dev/mqueue
ffff8e3a3459b080 ffff8e397cdcc800 debugfs debugfs /sys/kernel/debug
ffff8e3a31f05080 ffff8e3a3202e800 binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/
ffff8e3a2ffea300 ffff8e383693f000 xfs /dev/mapper/datavg/lv /opt/IBM
ffff8e3a3feb0900 ffff8e38369b8800 xfs /dev/sda1 /boot
mount command: Mounted filesystem information
mount - Displays basic information about the currently-mounted flesystems.
Filesystem Device name Mount point

54. crash> files
PID: 1786 TASK: ffff8e3a36228000 CPU: 1 COMMAND: "ds_am"
ROOT: / CWD: /var/opt/ds_agent/am
FD FILE DENTRY INODE TYPE PATH
0 ffff8e38ba842c00 ffff8e397f823240 ffff8e3a3fbd6850 CHR /dev/null
1 ffff8e38ba842a00 ffff8e397f823240 ffff8e3a3fbd6850 CHR /dev/null
2 ffff8e38b8c97400 ffff8e397f823240 ffff8e3a3fbd6850 CHR /dev/null
3 ffff8e3a267f9100 ffff8e3a353f2b40 ffff8e3a353c4160 REG /var/opt/ds_agent/diag/ds_am.log
4 ffff8e3a2cac4d00 ffff8e3a352b8c00 ffff8e38b8d34300 REG /dev/shm/dsam
5 ffff8e3a2ab43200 ffff8e3840ad1000 ffff8e386bafcca0 REG /var/opt/ds_agent/diag/ds_am-icrc.log
6 ffff8e3a28f94000 ffff8e3a34c10300 ffff8e38b8d345a8 REG /dev/shm/ICRC_PERF_COUNTER
7 ffff8e3a267f9700 ffff8e3a34c103c0 ffff8e3a353b6e60 REG /var/opt/ds_agent/diag/icrc.dat
8 ffff8e3a29ead800 ffff8e3a34c106c0 ffff8e3a353c27b0 SOCK UNIX
9 ffff8e38b8c5a300 ffff8e3a315fe480 ffff8e3a353b4ca0 REG /var/opt/ds_agent/am/crcz.ptn
10 ffff8e38a487a800 ffff8e39e0c6d0c0 ffff8e39217bf230 REG /proc/driver/gsch/syshook/enable
11 ffff8e3a28a3be00 ffff8e3a353f0780 ffff8e3a3533f6b0 SOCK UNIX
12 ffff8e3a28f93a00 ffff8e39217cd6c0 ffff8e3819d0d1b0 SOCK TCP
14 ffff8e385f5df000 ffff8e3819e8f480 ffff8e3819e5ea30 SOCK TCP
16 ffff8e38ba9ac400 ffff8e39e0c16300 ffff8e3a30af0af8 CHR /dev/dsa_rtscan
17 ffff8e3a34552a00 ffff8e39217c5840 ffff8e3819d0c7b0 SOCK TCP
files command: Information about open files in a context
fles - Display the open fles of the current context
List of open files
Struct file * Struct dentry * Struct inode *
Current working directory
of process

55. crash> dev -d
MAJOR GENDISK NAME REQUEST_QUEUE TOTAL ASYNC SYNC DRV
2 ffff8e3836937c00 fd0 ffff8e3a33420000 0 0 0 0
8 ffff8e3a347cbc00 sda ffff8e3a33424bc0 0 0 0 0
8 ffff8e3a347c2800 sdb ffff8e3a33425538 0 0 0 0
8 ffff8e3a347c1800 sdc ffff8e3a33425eb0 0 0 0 0
8 ffff8e3a347c0c00 sdd ffff8e3a33426828 0 0 0 0
11 ffff8e3836a7c000 sr0 ffff8e3a33535eb0 0 0 0 0
253 ffff8e3a347cdc00 dm-0 ffff8e3a33535538 10 7 3 0
253 ffff8e3836a7f400 dm-1 ffff8e3a32030000 0 0 0 0
253 ffff8e3836ad5800 dm-2 ffff8e3a32030978 0 0 0 0
253 ffff8e383616ec00 dm-3 ffff8e3a33534bc0 0 0 0 0
SYNC : I/O requests that are synchronous
ASYNC : I/O requests that are asynchronous
DRV : I/O requests that are in-flight in the device driver.
dev command: Device data
dev -d – Display disk I/O statistics.
Device name

56. crash> net
NET_DEVICE NAME IP ADDRESS(ES)
ffff8e3a3fc93000 lo 127.0.0.1
ffff8e38369a8000 eth0 192.168.68.51
net command: Network related information
net - Display the list of network devices, names and IP addresses.
crash> net -a
NEIGHBOUR IP ADDRESS HW TYPE HW ADDRESS DEVICE STATE
ffff8e3a3126d600 127.0.0.1 UNKNOWN 00 00 00 00 00 00 lo NOARP
ffff8e3a33405600 224.0.0.22 ETHER 01:00:5e:00:00:16 eth0 NOARP
ffff8e38bb62de00 192.168.68.4 ETHER 8c:60:4f:63:4a:bc eth0 STALE
ffff8e3a34598800 192.168.68.3 ETHER 8c:60:4f:63:4b:c1 eth0 STALE
ffff8e3a33404800 224.0.0.251 ETHER 01:00:5e:00:00:fb eth0 NOARP
ffff8e38bb62d400 192.168.68.2 ETHER 8c:60:4f:63:4e:01 eth0 STALE
ffff8e38bb62ce00 192.168.68.5 ETHER 8c:60:4f:63:4e:c1 eth0 STALE
ffff8e3a29fce200 192.168.68.1 ETHER 00:00:0c:9f:f2:38 eth0 REACHABLE
net -a - Display the ARP cache.
Struct net_device *
IP Address
Network Interface name

57. crash> mod | head
MODULE NAME SIZE OBJECT FILE
ffffffffc0235780 dm_mod 123941 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/drivers/md/dm-mod.ko.debug
ffffffffc0248000 crct10dif_common 12595 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/crypto/crct10dif_common.ko.debug
ffffffffc024f160 dm_log 18411 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/drivers/md/dm-log.ko.debug
ffffffffc02561c0 dm_mirror 22289 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/drivers/md/dm-mirror.ko.debug
ffffffffc0268b00 floppy 69432 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/drivers/block/floppy.ko.debug
ffffffffc0273380 vmw_pvscsi 23130 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/drivers/scsi/vmw_pvscsi.ko.debug
ffffffffc02802a0 i2c_core 63151 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/drivers/i2c/i2c-core.ko.debug
mod command: Module information
mod - Displays basic information of the currently installed modules.
crash> mod -t
NAME TAINTS
redirfs OE
gsch OE
mod -t – Display the contents of the module's "taints" bitmask.
Struct module *
Module size
List of unsigned modules
Object file path

58. crash> mod -s xfs
MODULE NAME SIZE OBJECT FILE
ffffffffc050d9e0 xfs 1003971 /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/kernel/fs/xfs/xfs.ko.debug
mod command: Module information
mod -s - Load the symbolic and debugging data of the xfs module from its known location.
crash> mod -d xfs
crash> mod | grep xfs
ffffffffc050d9e0 xfs 1003971 (not loaded) [CONFIG_KALLSYMS]
mod -d - Deletes the symbolic and debugging data of the module specifed.

59. log command: Kernel ring buffer
crash> log
[2559941.344390] SysRq : Trigger a crash
[2559942.957195] BUG: unable to handle kernel NULL pointer dereference at (null)
[2559943.099378] IP: [<ffffffff8135f166>] sysrq_handle_crash+0x16/0x20
[2559944.783241] PGD 8000001331cce067 PUD 13a4d32067 PMD 0
[2559945.486950] Oops: 0002 [#1] SMP
[2559946.422806] last sysfs file: /sys/devices/system/cpu/online
[2559946.684623] CPU 6
[2559947.107669] Modules linked in: nfs lockd fscache auth_rpcgss nfs_acl tcp_diag inet_diag
oracleacfs(P)(U) oracleadvm(P)(U) dm_region_hash dm_log dm_mod [last unloaded: cbsensor]
[2559951.437296]
[2559951.945506] Pid: 5833, comm: cssdagent Tainted: P -------------- 2.6.32-696.18.7.el6.x86_64
#1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[2559953.580469] RIP: 0010:[<ffffffff8135f166>] [<ffffffff8135f166>] sysrq_handle_crash+0x16/0x20
...
log – Display the kernel ring bufer.

60. bt command: Backtrace
crash> bt
PID: 28438 TASK: ffff88131860bf40 CPU: 11 COMMAND: "ls"
#0 [ffff882001a17728] machine_kexec at ffffffff8105c4cb
...
#9 [ffff882001a179d0] page_fault at ffffffff816ac508
[exception RIP: vnlayer_linux_new_clrvnode+21]
RIP: ffffffffc08b5f03 RSP: ffff882001a17a88 RFLAGS: 00010246
RAX: ffffffff81a8fdc0 RBX: 0000000000000000 RCX: 0000000000000002
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff880144beae40
RBP: ffff882001a17ab0 R8: ffff88014055fa10 R9: 00000001802a002a
R10: ffff88203df59b80 R11: ffffea004599e8c0 R12: 0000000000000000
R13: ffff880144beae40 R14: ffff881fc87462f0 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff882001a17ab8] mvop_linux_lookup_component at ffffffffc08b289f [mvfs]
#11 [ffff882001a17af0] mvfs_lookup_ctx at ffffffffc086e57c [mvfs]
#12 [ffff882001a17b78] mvfs_linux_lookup_wrapper at ffffffffc08a8257 [mvfs]
#13 [ffff882001a17bd8] vnode_iop_lookup at ffffffffc08b056e [mvfs]
#14 [ffff882001a17c60] lookup_real at ffffffff8120b45d
#15 [ffff882001a17c80] __lookup_hash at ffffffff8120bd32
#16 [ffff882001a17cb0] lookup_slow at ffffffff816a1322
#17 [ffff882001a17ce8] path_lookupat at ffffffff8120f2db
#18 [ffff882001a17d80] filename_lookup at ffffffff8120f33b
#19 [ffff882001a17db8] user_path_at_empty at ffffffff81212e47
#20 [ffff882001a17e88] user_path_at at ffffffff81212eb1
#21 [ffff882001a17e98] vfs_fstatat at ffffffff81206463
#22 [ffff882001a17ee8] SYSC_newlstat at ffffffff81206a31
#23 [ffff882001a17f70] sys_newlstat at ffffffff81206cbe
#24 [ffff882001a17f80] system_call_fastpath at ffffffff816b4fc9
...
bt - Displays a task's kernel-stack backtrace.
Panic Location
Panic task
Content of CPU registers
at the time of panic
Return address
Module name
Stackframe
Functions on the stack

61. bt command: Backtrace
crash> bt -a
PID: 20684 TASK: ffff8e39f9f39fa0 CPU: 0 COMMAND: "awk"
#0 [ffff8e3840ec7ea8] crash_nmi_callback at ffffffff8c0537c7
#1 [ffff8e3840ec7eb8] nmi_handle at ffffffff8c71893c
#2 [ffff8e3840ec7f10] do_nmi at ffffffff8c718b5d
#3 [ffff8e3840ec7f50] nmi at ffffffff8c717d00
RIP: 00007fabec9c9c44 RSP: 00007fff4f634750 RFLAGS: 00000206
RAX: 0000000076036301 RBX: 0000000000f0b2ff RCX: 0000000000000000
RDX: 0000000000ca0000 RSI: 00007fff4f634848 RDI: 00000000000000bc
RBP: 00000000000000bc R8: 0000000000000007 R9: 0000000000000002
R10: 00007fff4f6345a0 R11: 00007fff4f6346a0 R12: 00007fff4f634770
R13: 00007fff4f634760 R14: 0000000000000001 R15: 00007fabecce5730
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
PID: 1786 TASK: ffff8e3a36228000 CPU: 1 COMMAND: "ds_am"
#0 [ffff8e38b8e9fa78] machine_kexec at ffffffff8c060f9a
#1 [ffff8e38b8e9fad8] __crash_kexec at ffffffff8c113732
#2 [ffff8e38b8e9fba8] crash_kexec at ffffffff8c113820
#3 [ffff8e38b8e9fbc0] oops_end at ffffffff8c718798
#4 [ffff8e38b8e9fbe8] no_context at ffffffff8c707fbe
#5 [ffff8e38b8e9fc38] __bad_area_nosemaphore at ffffffff8c708055
#6 [ffff8e38b8e9fc88] bad_area_nosemaphore at ffffffff8c7081c6
#7 [ffff8e38b8e9fc98] __do_page_fault at ffffffff8c71b750
#8 [ffff8e38b8e9fd00] do_page_fault at ffffffff8c71b945
#9 [ffff8e38b8e9fd30] page_fault at ffffffff8c717788
[exception RIP: gsch_install_hooks+136]
RIP: ffffffffc073b8d8 RSP: ffff8e38b8e9fde0 RFLAGS: 00010202
RAX: 0000000000000000 RBX: ffffffffc074a000 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffffffffc074a960 RDI: ffffffffc074a180
RBP: ffff8e38b8e9fe40 R8: 0000000030323863 R9: 0000000000000003
R10: 00000000000005dd R11: 3138666666666666 R12: ffffffffc074a180
R13: ffffffffc074a960 R14: ffffffff81803310 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
...
bt -a - Displays the backtraces of the active tasks on all CPUs.
Backtrace of task on CPU 0
Backtrace of task on CPU 1

62. bt command: Backtrace
crash> bt -f
PID: 1786 TASK: ffff8e3a36228000 CPU: 1 COMMAND: "ds_am"
…
#5 [ffff8e38b8e9fc38] __bad_area_nosemaphore at ffffffff8c708055
ffff8e38b8e9fc40: 0000000000226c6c 0000000000000246
ffff8e38b8e9fc50: 0000000000000000 0000000000000000
ffff8e38b8e9fc60: 0000000000000000 ffffffff81803310
ffff8e38b8e9fc70: ffff8e38b8e9fd38 ffff8e3a36228000
ffff8e38b8e9fc80: ffff8e38b8e9fc90 ffffffff8c7081c6
#6 [ffff8e38b8e9fc88] bad_area_nosemaphore at ffffffff8c7081c6
ffff8e38b8e9fc90: ffff8e38b8e9fcf8 ffffffff8c71b750
#7 [ffff8e38b8e9fc98] __do_page_fault at ffffffff8c71b750
ffff8e38b8e9fca0: ffff8e38b8e9fcf8 ffff8e38b8e9fcb8
ffff8e38b8e9fcb0: 00000000f0a3d3eb 0000000000000000
ffff8e38b8e9fcc0: ffff8e3a29f49978 ffff8e3a29f49900
ffff8e38b8e9fcd0: ffff8e38b8e9fd38 0000000000000000
ffff8e38b8e9fce0: 0000000000000000 ffffffff81803310
ffff8e38b8e9fcf0: 0000000000000000 ffff8e38b8e9fd28
ffff8e38b8e9fd00: ffffffff8c71b945
#8 [ffff8e38b8e9fd00] do_page_fault at ffffffff8c71b945
ffff8e38b8e9fd08: ffffffffc074ab60 0000000000000001
ffff8e38b8e9fd18: ffffffffc074a180 ffffffffc074a960
ffff8e38b8e9fd28: ffff8e38b8e9fe40 ffffffff8c717788
#9 [ffff8e38b8e9fd30] page_fault at ffffffff8c717788
[exception RIP: gsch_install_hooks+136]
RIP: ffffffffc073b8d8 RSP: ffff8e38b8e9fde0 RFLAGS: 00010202
RAX: 0000000000000000 RBX: ffffffffc074a000 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffffffffc074a960 RDI: ffffffffc074a180
RBP: ffff8e38b8e9fe40 R8: 0000000030323863 R9: 0000000000000003
R10: 00000000000005dd R11: 3138666666666666 R12: ffffffffc074a180
R13: ffffffffc074a960 R14: ffffffff81803310 R15: 0000000000000000
bt -f – Display all stack data contained in a frame.
Return address
Data on the stack

63. dis command: Disassemble
crash> dis -r sysrq_handle_crash+22
0xffffffff9102f170 <sysrq_handle_crash>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffff9102f175 <sysrq_handle_crash+5>: push %rbp
0xffffffff9102f176 <sysrq_handle_crash+6>: mov %rsp,%rbp
0xffffffff9102f179 <sysrq_handle_crash+9>: movl $0x1,0x8142d1(%rip) #
0xffffffff91843454
0xffffffff9102f183 <sysrq_handle_crash+19>: sfence
0xffffffff9102f186 <sysrq_handle_crash+22>: movb $0x1,0x0
dis -r <function+ofset> - Disassemble a function from the start to the specifed ofset.
CPU InstructionsFunction Offset

64. dis command: Disassemble
crash> dis -l sysrq_handle_crash+22
/usr/src/debug/kernel-3.10.0-862.3.2.el7/linux-3.10.0-862.3.2.el7.x86_64/drivers/tty/sysrq.c: 145
0xffffffff9102f186 <sysrq_handle_crash+22>: movb $0x1,0x0
dis -l <function+ofset> - Shows the line number in kernel code.
CPU InstructionsFunction Offset
Kernel source file
Line number
Kernel Source: drivers/tty/sysrq.c
...
133 static void sysrq_handle_crash(int key)
134 {
135 char *killer = NULL;
136
137 /* we need to release the RCU read lock here,
138 * otherwise we get an annoying
139 * 'BUG: sleeping function called from invalid context'
140 * complaint from the kernel before the panic.
141 */
142 rcu_read_unlock();
143 panic_on_oops = 1; /* force panic */
144 wmb();
145 *killer = 1;
146 }
Kernel source file
Line number

65. crash> whatis vfs_write
ssize_t vfs_write(struct file *, const char *, size_t, loff_t *);
whatis command: Definition of function
whatis <function name>- Display the type data of function
Function Arguments

66. whatis command: Definition of structure
crash> whatis file
struct file {
union {
struct list_head fu_list;
struct callback_head fu_rcuhead;
} f_u;
struct path f_path;
struct inode *f_inode;
const struct file_operations *f_op;
spinlock_t f_lock;
int f_sb_list_cpu_deprecated;
atomic_long_t f_count;
unsigned int f_flags;
fmode_t f_mode;
loff_t f_pos;
struct fown_struct f_owner;
const struct cred *f_cred;
struct file_ra_state f_ra;
u64 f_version;
void *f_security;
void *private_data;
struct list_head f_ep_links;
struct list_head f_tfile_llink;
struct address_space *f_mapping;
struct mutex f_pos_lock;
}
SIZE: 256
whatis <structure tag> - Display the defnition of a structure tag.
Structure name
Members of structure

67. crash> struct file -ox
struct file {
union {
struct list_head fu_list;
struct callback_head fu_rcuhead;
[0x0] } f_u;
[0x10] struct path f_path;
[0x20] struct inode *f_inode;
[0x28] const struct file_operations *f_op;
[0x30] spinlock_t f_lock;
[0x34] int f_sb_list_cpu_deprecated;
[0x38] atomic_long_t f_count;
[0x40] unsigned int f_flags;
[0x44] fmode_t f_mode;
[0x48] loff_t f_pos;
[0x50] struct fown_struct f_owner;
[0x70] const struct cred *f_cred;
[0x78] struct file_ra_state f_ra;
[0x98] u64 f_version;
[0xa0] void *f_security;
[0xa8] void *private_data;
[0xb0] struct list_head f_ep_links;
[0xc0] struct list_head f_tfile_llink;
[0xd0] struct address_space *f_mapping;
[0xd8] struct mutex f_pos_lock;
}
SIZE: 0x100
struct command: Structure contents
struct <struct-tag-name> [-o] [-x | -d] – display defnition and member ofsets.
Structure name
Members of structure
o is offset
d in decimal
x in hexadecimal

68. crash> p (0x1+0x2)
$1 = 3
crash> pd 0x11
$2 = 17
crash> px 11
$3 = 0xb
p command : Print
p - Print the value of an expression.

69. crash> eval (0x1+0x2)
hexadecimal: 3
decimal: 3
octal: 3
binary: 0000000000000000000000000000000000000000000000000000000000000011
crash> eval -b (0x11)
hexadecimal: 11
decimal: 17
octal: 21
binary: 0000000000000000000000000000000000000000000000000000000000010001
bits set: 4 0
eval command: Evaluate
Enabled bits
eval - Evaluates an expression or numeric value

70. crash> rd ffffffff9102f186 0x4
ffffffff9102f186: 01000000002504c6 550000441f0fc35d ..%.....]...D..U
ffffffff9102f196: 0081454e05c7c031 53e5894800000007 1...NE......H..S
crash> rd ffffffff9102f186
ffffffff9102f186: 01000000002504c6 ..%.....
rd command: Read memory
rd - read memory contents.
Memory location Data stored
Memory location Data stored
Address + range
ASCII representation
crash> rd c009bf2c -e c009bf60
c009bf2c: c009bf64 c01328c3 c009bf64 c0132838 d....(..d...8(..
c009bf3c: 0000002a 00000004 c57d77e8 00000104 *........w}.....
c009bf4c: 0000000b c009a000 7fffffff 00000000 ................
C009bf5c: 00000000
Read the range of memory from c009bf2c to c009bf60:

71. Example

72. System information
crash> sys
KERNEL: /usr/lib/debug/lib/modules/3.10.0-862.3.2.el7.x86_64/vmlinux
DUMPFILE: /var/crash/127.0.0.1-2018-07-30-17:38:26/vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Mon Jul 30 17:38:19 2018
UPTIME: 00:02:26
LOAD AVERAGE: 0.10, 0.09, 0.04
TASKS: 260
NODENAME: localhost.localdomain
RELEASE: 3.10.0-862.3.2.el7.x86_64
VERSION: #1 SMP Tue May 15 18:22:15 EDT 2018
MACHINE: x86_64 (2808 Mhz)
MEMORY: 6 GB
PANIC: "divide error: 0000 [#1] SMP "

73. Kernel ring buffer
crash> log
[..]
[ 148.101287] DevConf: loading out-of-tree module taints kernel.
[ 148.111590] DevConf: module verification failed: signature and/or required key missing - tainting kernel
[ 148.132504] Hello, DevConf !
[ 148.133688] divide error: 0000 [#1] SMP
[ 148.135147] Modules linked in: DevConf(OE+) ip6table_filter
[ 148.161198] CPU: 1 PID: 2403 Comm: insmod Kdump: loaded Tainted: GOE ------------ 3.10.0-862.3.2.el7.x86_64 #1
[ 148.163796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26 04/01/2014
[ 148.165553] task: ffff97e0a9a28fd0 ti: ffff97e0b6780000 task.ti: ffff97e0b6780000
[ 148.167094] RIP: 0010:[<ffffffffc075301f>] [<ffffffffc075301f>] DevConf_module_init+0x1f/0x30 [DevConf]
[ 148.169081] RSP: 0018:ffff97e0b6783d30 EFLAGS: 00010246
[ 148.170157] RAX: 0000000000000001 RBX: ffffffff94016020 RCX: 0000000000000000
[ 148.171722] RDX: 0000000000000000 RSI: ffff97e0bfc93938 RDI: ffff97e0bfc93938
[ 148.173263] RBP: ffff97e0b6783d30 R08: 0000000000000000 R09: 0000000000000100
[ 148.174718] R10: 0000000000000417 R11: 0000000000aaaaaa R12: ffff97e0a2b09860
[ 148.176185] R13: ffffffffc0753000 R14: 0000000000000000 R15: ffffffffc0755000
[ 148.177636] FS: 00007efd309a1740(0000) GS:ffff97e0bfc80000(0000) knlGS:0000000000000000
[ 148.179305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 148.180515] CR2: 0000000000413e60 CR3: 00000000b30e4000 CR4: 00000000003606e0
[ 148.182022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 148.183467] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 148.184919] Call Trace:
[ 148.185424] [<ffffffff9340210a>] do_one_initcall+0xba/0x240
[ 148.186591] [<ffffffff9350f8dc>] load_module+0x272c/0x2bc0
[ 148.187757] [<ffffffff93776be0>] ? ddebug_proc_write+0xf0/0xf0
[ 148.189006] [<ffffffff9350b513>] ? copy_module_from_fd.isra.43+0x53/0x150
[ 148.190491] [<ffffffff9350ff26>] SyS_finit_module+0xa6/0xd0
[ 148.191952] [<ffffffff93b2082f>] system_call_fastpath+0x1c/0x21
[ 148.193418] Code: <f7> f9 5d c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[ 148.198450] RIP [<ffffffffc075301f>] DevConf_module_init+0x1f/0x30 [DevConf]
[ 148.201937] RSP <ffff97e0b6783d30>

74. Panic task and List of open files
crash> set -p
PID: 2403
COMMAND: "insmod"
TASK: ffff97e0a9a28fd0 [THREAD_INFO: ffff97e0b6780000]
CPU: 1
STATE: TASK_RUNNING (PANIC)
crash> files
PID: 2403 TASK: ffff97e0a9a28fd0 CPU: 1 COMMAND: "insmod"
ROOT: / CWD: /root/module
FD FILE DENTRY INODE TYPE PATH
0 ffff97dfbb56b300 ffff97e0ab111900 ffff97e0b72cf2e0 CHR /dev/pts/0
1 ffff97dfbb56b300 ffff97e0ab111900 ffff97e0b72cf2e0 CHR /dev/pts/0
2 ffff97dfbb56b300 ffff97e0ab111900 ffff97e0b72cf2e0 CHR /dev/pts/0
3 ffff97e09e691900 ffff97e0ab1e3540 ffff97dfafcb6e60 REG /root/module/DevConf.ko

75. Backtrace of panic task
crash> bt
PID: 2403 TASK: ffff97e0a9a28fd0 CPU: 1 COMMAND: "insmod"
#0 [ffff97e0b67839e0] machine_kexec at ffffffff93460f9a
#1 [ffff97e0b6783a40] __crash_kexec at ffffffff93513732
#2 [ffff97e0b6783b10] crash_kexec at ffffffff93513820
#3 [ffff97e0b6783b28] oops_end at ffffffff93b18798
#4 [ffff97e0b6783b50] die at ffffffff9342e96b
#5 [ffff97e0b6783b80] do_trap at ffffffff93b17f10
#6 [ffff97e0b6783bd0] do_divide_error at ffffffff9342b08e
#7 [ffff97e0b6783c80] divide_error at ffffffff93b23e4e
[exception RIP: init_module+31]
RIP: ffffffffc075301f RSP: ffff97e0b6783d30 RFLAGS: 00010246
RAX: 0000000000000001 RBX: ffffffff94016020 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff97e0bfc93938 RDI: ffff97e0bfc93938
RBP: ffff97e0b6783d30 R8: 0000000000000000 R9: 0000000000000100
R10: 0000000000000417 R11: 0000000000aaaaaa R12: ffff97e0a2b09860
R13: ffffffffc0753000 R14: 0000000000000000 R15: ffffffffc0755000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff97e0b6783d38] do_one_initcall at ffffffff9340210a
#9 [ffff97e0b6783d68] load_module at ffffffff9350f8dc
#10 [ffff97e0b6783eb8] sys_finit_module at ffffffff9350ff26
#11 [ffff97e0b6783f50] system_call_fastpath at ffffffff93b2082f
RIP: 00007efd2fe787f9 RSP: 00007ffc3eb99528 RFLAGS: 00010246
RAX: 0000000000000139 RBX: 0000000001b5d2e0 RCX: 000000000000001f
RDX: 0000000000000000 RSI: 000000000041a2d8 RDI: 0000000000000003
RBP: 000000000041a2d8 R8: 0000000000000000 R9: 00007ffc3eb99738
R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000000000
R13: 0000000001b5d2c0 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 0000000000000139 CS: 0033 SS: 002b

76. Dis-assembly of exception RIP
crash> dis -rl ffffffffc075301f
0xffffffffc0753000 <DevConf_module_init>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffc0753005 <init_module+5> : push %rbp
0xffffffffc0753006 <init_module+6> : mov $0xffffffffc0754024,%rdi
0xffffffffc075300d <init_module+13>: xor %eax,%eax
0xffffffffc075300f <init_module+15>: mov %rsp,%rbp
0xffffffffc0753012 <init_module+18>: callq 0xffffffff93b08bc7 <printk>
0xffffffffc0753017 <init_module+23>: mov $0x1,%eax
0xffffffffc075301c <init_module+28>: xor %ecx,%ecx
0xffffffffc075301e <init_module+30>: cltd
0xffffffffc075301f <init_module+31>: idiv %ecx
[exception RIP: init_module+31]
RIP: ffffffffc075301f RSP: ffff97e0b6783d30 RFLAGS: 00010246
RAX: 0000000000000001 RBX: ffffffff94016020 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff97e0bfc93938 RDI: ffff97e0bfc93938
RBP: ffff97e0b6783d30 R8: 0000000000000000 R9: 0000000000000100
R10: 0000000000000417 R11: 0000000000aaaaaa R12: ffff97e0a2b09860
R13: ffffffffc0753000 R14: 0000000000000000 R15: ffffffffc0755000

77. Dis-assembly of exception RIP
crash> mod -s DevConf DevConf.o
MODULE NAME SIZE OBJECT FILE
ffffffffc0755000 DevConf 12514 DevConf.o
crash> dis -rl ffffffffc075301f
/root/module/DevConf.c: 10
0xffffffffc0753000 <DevConf_module_init>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffc0753005 <init_module+5> : push %rbp
/root/module/DevConf.c: 11
0xffffffffc0753006 <init_module+6> : mov $0xffffffffc0754024,%rdi
0xffffffffc075300d <init_module+13> : xor %eax,%eax
/root/module/DevConf.c: 10
0xffffffffc075300f <init_module+15> : mov %rsp,%rbp
/root/module/DevConf.c: 11
0xffffffffc0753012 <init_module+18> : callq 0xffffffff93b08bc7 <printk>
/root/module/DevConf.c: 12
0xffffffffc0753017 <init_module+23> : mov $0x1,%eax
0xffffffffc075301c <init_module+28> : xor %ecx,%ecx
0xffffffffc075301e <init_module+30> : cltd
0xffffffffc075301f <init_module+31> : idiv %ecx
crash> mod | grep -e NAME -e DevConf
MODULE NAME SIZE OBJECT FILE
ffffffffc0755000 DevConf 12514 (not loaded) [CONFIG_KALLSYMS]

78. DevConf module source code
# vi /root/module/DevConf.c
1 /* Kernel Panic Example Module */
2
3 #ifdef __KERNEL__
4
5 #include <linux/init.h>
6 #include <linux/module.h>
7 #include <linux/kernel.h>
8
9 static int DevConf_module_init(void)
10 {
11 printk(KERN_INFO "Hello, DevConf !n");
12 return 1/0;
13 }
14
15 static void DevConf_module_exit(void)
16 {
17 printk(KERN_INFO "Goodbye, DevConf !n");
18 }
19
20 module_init(DevConf_module_init);
21 module_exit(DevConf_module_exit);
22 MODULE_LICENSE("GPL");
23 MODULE_DESCRIPTION("DevConf Module");
24 MODULE_AUTHOR("reanimationBK");
25
26 #endif

79. Questions ?

80. More questions ?
@reanimationbk

81. Thank you