Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Linux crashdump analysis
Dumping and analysing system state
Kernel-Recipes 2015
Adrien Mahieux - Sysadmin & microsecond hu...
0 - Agenda
1. What’s a (crash)dump ?
2. Dump analysis
3. Live analysis (+ edition)
4. Tools & Links
- Get a dump - from hy...
What A snapshot of a system memory at a specific time
Who Mostly for sysadmins and guardians of production
Where Physical ...
1.1 - Get a dump - hypervisor
VMWare
- Suspend / resume (.vmss file) or Snapshot with memory (.vmsn file)
- Use tool vmss2...
1.2 - Get a crashdump - kexec / kdump
Kernel configuration
- CONFIG_KEXEC=y to boot the secondary kernel
- CONFIG_SYSFS=y ...
1.2 - Get a crashdump - kexec / kdump
Configure kdump
- Feature of the kernel that exports an ELF memory image via /proc/v...
1.2 - Get a crashdump - kexec / kdump
Dumping an unresponsive system : PANIC !
Manually
- SysRq echo c > /proc/sysrq-trigg...
1.2 - Get a crashdump - kexec / kdump (non-server)
Desktops / Laptops usually don’t have external source to generate NMI
K...
1.2 - Get a crashdump - kexec / kdump
2 - Dump Analysis
2 - Dump Analysis
Your weapon : Crash
- Tool by Dave Anderson (RedHat)
- Based on GDB
- x86, x86_64, arm, ia64, ppc64, s39...
2.1 - What to look for
Summup of the system state : sys
KERNEL: /var/crash/127.0.0.1-2015-08-20-20:00:00/vmcore
DUMPFILE: ...
2.2 - Crash : GDB for Kernel
Let’s check for a real kernel bug
KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.29.2.el6.x86_...
2.2 - Crash : GDB for Kernel
crash> bt
PID: 2120 TASK: ffff880437dcf540 CPU: 50 COMMAND: "scsi_eh_6"
#0 [ffff880435a95890]...
2.2 - Crash : GDB for Kernel
Let’s check the structures involved in this bug :
crash> struct scsi_cmnd
struct scsi_cmnd {
...
3 - Live modifications
3 - Live modifications
Yes, you can tinkle with the Kernel memory too !
Through /dev/mem, you can access memory… but not o...
3.1 - Live modifications - Network Parameters
Get the list of the NICs :
crash> net
NET_DEVICE NAME IP ADDRESS(ES)
ffff880...
4 - Tools and useful links
4.1 - Tools : OpenGrok
Wicked fast code source browser
http://opengrok.github.io/OpenGrok/
Grok : "to understand intuitive...
4.2 - Tools : kdumptools
Set of scripts to ease your kdump usage (try to work with all distributions)
https://github.com/s...
4.3 - Links - kdump
Kdump-Tool : Kexec is part of kexec-tools
Sources : https://git.kernel.org/cgit/utils/kernel/kexec/kex...
4.4 - Links - crash
Official Page : Download, tools and help
http://people.redhat.com/anderson
Linux Crash Cook Book : Det...
4.5 - Links - Kernel
Linux Insides : https://0xax.gitbooks.io/linux-insides
Understanding the Linux Kernel
ISBN 10 : 0-596...
Thank you
Upcoming SlideShare
Loading in …5
×

Kernel Recipes 2015 - Kernel dump analysis

2,311 views

Published on

Kernel dump analysis

Cloud this, cloud that…It’s making everything easier, especially for web hosted services. But what about the servers that are not supposed to crash ? For applications making the assumption the OS won’t do any fault or go down, what can you write in your post-mortem once the server froze and has been restarted ? How to track down the bug that lead to service unavailability ?
In this talk, we’ll see how to setup kdump and how to panic a server to generate a coredump. Once you have the vmcore file, how to track the issue with “crash” tool to find why your OS went down. Last but not least : with “crash” you can also modify your live kernel, the same way you would do with gdb.


Adrien Mahieux – System administrator obsessed with performance and uptime, tracking down microseconds from hardware to software since 2011. The application must be seen as a whole to provide efficiently the requested service. This includes searching for bottlenecks and tradeoffs, design issues or hardware optimization.

Published in: Software
  • Login to see the comments

Kernel Recipes 2015 - Kernel dump analysis

  1. 1. Linux crashdump analysis Dumping and analysing system state Kernel-Recipes 2015 Adrien Mahieux - Sysadmin & microsecond hunter gh: github.com/Saruspete tw: @Saruspete
  2. 2. 0 - Agenda 1. What’s a (crash)dump ? 2. Dump analysis 3. Live analysis (+ edition) 4. Tools & Links - Get a dump - from hypervisor - Get a crashdump - with kdump - GDB based tool : crash - Requirements : debuginfo - What to look for - Using crash on a live system - Source browsing - Script helpers - Analysis
  3. 3. What A snapshot of a system memory at a specific time Who Mostly for sysadmins and guardians of production Where Physical and Virtual Linux-based servers When Your server is unresponsive (from ssh / console / application…) Why To know what happened (kernel bug, external attack, limit missing…) How Physical : kexec & panic the server Virtual : same, or from hypervisor H. Much Uses between 64M and 512M of RAM to boot the sec. kernel On Virtual, you may do it from hypervisor at no cost 1 - What’s a (crash)dump ?
  4. 4. 1.1 - Get a dump - hypervisor VMWare - Suspend / resume (.vmss file) or Snapshot with memory (.vmsn file) - Use tool vmss2core (VMWare Labs) to transform the raw dump into ELF dump libvirt - virsh : virsh dump MyGuestName /storage/MyGuestName.dump - QEMU Monitor : dump-guest-memory [-z|-l|-s] FILENAME Xen - xl : dump-core domain-id filename
  5. 5. 1.2 - Get a crashdump - kexec / kdump Kernel configuration - CONFIG_KEXEC=y to boot the secondary kernel - CONFIG_SYSFS=y for /sys/kernel/kexec_crash_{loaded,size} - CONFIG_CRASH_DUMP=y - CONFIG_PROC_VMCORE=y Export dump to /proc/vmcore - (CONFIG_DEBUG_INFO=y) Will not be in live kernel - (CONFIG_RELOCATABLE=y) To use the same kernel for live & dump - boot option : crashkernel=X@Y - X is the amount of memory to be reserved + 2 bytes for each 4KB - Y is the offset at which memory will be reserved - You can specify only X and the Kernel will find Y - If you have more than 2G of RAM, you can use “auto”
  6. 6. 1.2 - Get a crashdump - kexec / kdump Configure kdump - Feature of the kernel that exports an ELF memory image via /proc/vmcore - kdump often refers to the whole process to dump a core - Relies on kexec to boot a secondary kernel / initrd to do the job - Uses the memory reserved by “crashkernel” bootopt to load the “dump- capture” kernel & initrd - Upon panic, the running kernel will start the new one, which will do the dump (ssh, ftp, local disk.. depending on your script) and reboot the system - kdump can use makedumpfile to filter memory data by type (free pages, userland pages, private cache, cache pages, zero pages). - Check status with /sys/kernel/kexec_crash_{loaded,size}
  7. 7. 1.2 - Get a crashdump - kexec / kdump Dumping an unresponsive system : PANIC ! Manually - SysRq echo c > /proc/sysrq-trigger - NMI via IPMI ipmitool power diag - NMI via virsh virsh inject-nmi MyGuestName - Beware of kernel.unknown_nmi_panic=1 Automatically - Watchdog Boot cmdline: nmi_watchdog=1 - Softlockup sysctl kernel.softlockup_panic=1 - Out Of Memory sysctl vm.panic_on_oom=1
  8. 8. 1.2 - Get a crashdump - kexec / kdump (non-server) Desktops / Laptops usually don’t have external source to generate NMI Kernel provides other ways : Hard/Soft lockup detectors - Kernel config {SOFT,HARD}LOCKUP_DETECTOR / BOOTPARAM_{SOFT,HARD} LOCKUP_PANIC - Hard : Stay in kernel for more than 10sec - Soft : Task is hung for 120sec Watchdog daemon - Kernel config {SOFT,CLOCKSOURCE}_WATCHDOG - Boot option “nmi_watchdog=1” - watchdog daemon (http://sourceforge.net/projects/watchdog)
  9. 9. 1.2 - Get a crashdump - kexec / kdump
  10. 10. 2 - Dump Analysis
  11. 11. 2 - Dump Analysis Your weapon : Crash - Tool by Dave Anderson (RedHat) - Based on GDB - x86, x86_64, arm, ia64, ppc64, s390 - Extensible (snap, trace, appdump, memory, dm, ipcs, cgroups, sockets, openvz…) - Quick evolution and active Mailing List Your gunsmith : debuginfos - We don’t want debug in production, but we’d like to be able to debug - Split debuginfo are Dwarf debug data in separate files to be used on demand - Most distributions provides them for stock kernel Redhat : debuginfo-install kernel Debian : apt-get install linux- image-$(uname -r)-dbg
  12. 12. 2.1 - What to look for Summup of the system state : sys KERNEL: /var/crash/127.0.0.1-2015-08-20-20:00:00/vmcore DUMPFILE: vmcore.myserver [PARTIAL DUMP] CPUS: 24 DATE: Mon Aug 20 20:00:00 2015 UPTIME: 32 days, 17:12:02 LOAD AVERAGE: 1625.88, 1603.11, 1509.73 TASKS: 25639 NODENAME: myserver RELEASE: 2.6.18-371.8.1.el5 VERSION: #1 SMB Fri Mar 28 05:53:58 EDT 2014 MACHINE: x86_64 (2933Mhz) MEMORY: 284 GB PANIC: “Kernel panic - not syncing: An NMI occured” PID: 61015 COMMAND: "java" TAKS: ffff8135b50e5830 [THREAD_INFO: ffff8104bd256000] CPU: 0 STATE: TASK_RUNNING (PANIC) System logs log Memory Usage kmem Swap Usage swap Running processps Set PID to analyze set Task struct of PID task Files opened by PID files Backtrace of PID bt Available devices dev Available NICs net Interrupts irq Mountpoints mount Process using a file fuser IPC Show ipcs Kernernel Modules mod RunQueue runq Symbols info sym
  13. 13. 2.2 - Crash : GDB for Kernel Let’s check for a real kernel bug KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.29.2.el6.x86_64/vmlinux DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 64 DATE: Wed Jun 14 11:23:14 2015 UPTIME: 44 days, 04:14:21 LOAD AVERAGE: 0.70, 0.58, 0.55 TASKS: 1917 NODENAME: myredhat65 RELEASE: 2.6.32-431.29.2.el6.x86_64 VERSION: #1 SMP Sun Jul 27 15:55:46 EDT 2014 MACHINE: x86_64 (1997 Mhz) MEMORY: 64 GB PANIC: "BUG: unable to handle kernel NULL pointer dereference at (null)" PID: 2120 COMMAND: "scsi_eh_6" TASK: ffff880437dcf540 [THREAD_INFO: ffff880435a94000] CPU: 50 STATE: TASK_RUNNING (PANIC)
  14. 14. 2.2 - Crash : GDB for Kernel crash> bt PID: 2120 TASK: ffff880437dcf540 CPU: 50 COMMAND: "scsi_eh_6" #0 [ffff880435a95890] machine_kexec at ffffffff81038f3b #1 [ffff880435a958f0] crash_kexec at ffffffff810c5af2 #2 [ffff880435a959c0] oops_end at ffffffff8152ca50 #3 [ffff880435a959f0] no_context at ffffffff8104a00b #4 [ffff880435a95a40] __bad_area_nosemaphore at ffffffff8104a295 #5 [ffff880435a95a90] bad_area_nosemaphore at ffffffff8104a363 #6 [ffff880435a95aa0] __do_page_fault at ffffffff8104aabf #7 [ffff880435a95bc0] do_page_fault at ffffffff8152e99e #8 [ffff880435a95bf0] page_fault at ffffffff8152bd55 [exception RIP: scsi_send_eh_cmnd+99] RIP: ffffffff813860e3 RSP: ffff880435a95ca0 RFLAGS: 00010286 RAX: 0000000000000000 RBX: ffff880c2d600ec0 RCX: 0000000000002710 RDX: ffff880c3002f000 RSI: ffffffff82017288 RDI:ffff880c2d600ec0 RBP: ffff880435a95da0 R8: 0000000000000000 R9: 0000000000000000 R10: 000d8f6a631f7b23 R11: 0000000000000001 R12: 0000000000000001 R13: ffff880435a95e90 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff880435a95da8] scsi_eh_tur at ffffffff81386672 #10 [ffff880435a95dd8] scsi_eh_test_devices at ffffffff8138675a #11 [ffff880435a95e28] scsi_error_handler at ffffffff81387d4c #12 [ffff880435a95ee8] kthread at ffffffff8109abf6 #13 [ffff880435a95f48] kernel_thread at ffffffff8100c20a crash> gdb set disassemble-flavor intel crash> dis scsi_send_eh_cmnd 0xffffffff81386080 <scsi_send_eh_cmnd>: push rbp 0xffffffff81386081 <scsi_send_eh_cmnd+1>: mov rbp,rsp 0xffffffff81386084 <scsi_send_eh_cmnd+4>: push r15 0xffffffff81386086 <scsi_send_eh_cmnd+6>: push r14 0xffffffff81386088 <scsi_send_eh_cmnd+8>: push r13 0xffffffff8138608a <scsi_send_eh_cmnd+10>: push r12 0xffffffff8138608c <scsi_send_eh_cmnd+12>: push rbx 0xffffffff8138608d <scsi_send_eh_cmnd+13>: sub rsp,0xd8 0xffffffff81386094 <scsi_send_eh_cmnd+20>: nop DWORD PTR [rax+rax*1+0x0] 0xffffffff81386099 <scsi_send_eh_cmnd+25>: mov rax,QWORD PTR gs:0x28 0xffffffff813860a2 <scsi_send_eh_cmnd+34>: mov QWORD PTR [rbp-0x38],rax 0xffffffff813860a6 <scsi_send_eh_cmnd+38>: xor eax,eax 0xffffffff813860a8 <scsi_send_eh_cmnd+40>: mov QWORD PTR [rbp-0xc8],rsi 0xffffffff813860af <scsi_send_eh_cmnd+47>: mov DWORD PTR [rbp-0xcc],edx 0xffffffff813860b5 <scsi_send_eh_cmnd+53>: mov rbx,rdi 0xffffffff813860b8 <scsi_send_eh_cmnd+56>: mov rax,QWORD PTR [rdi+0x80] crash> rd -o 0x80 0xffff880c2d600ec0 ffff880c2d600f40: ffff880c372afd00 0xffffffff813860bf <scsi_send_eh_cmnd+63>: mov rdx,QWORD PTR [rdi] 0xffffffff813860c2 <scsi_send_eh_cmnd+66>: mov r14d,r8d 0xffffffff813860c5 <scsi_send_eh_cmnd+69>: mov rax,QWORD PTR [rax+0xb0] crash> rd -64 -o 0xb0 ffff880c372afd00 ffff880c372afdb0: ffff880c2a4d0400 0xffffffff813860cc <scsi_send_eh_cmnd+76>: mov QWORD PTR [rbp-0xe8],0x0 0xffffffff813860d7 <scsi_send_eh_cmnd+87>: test rax,rax 0xffffffff813860da <scsi_send_eh_cmnd+90>: je 0xffffffff813860ed <scsi_send_eh_cmnd+109> 0xffffffff813860dc <scsi_send_eh_cmnd+92>: mov rax,QWORD PTR [rax+0x2c8] crash> rd -64 -o 0x2c8 ffff880c2a4d0400 ffff880c2a4d06c8: 0000000000000000 0xffffffff813860e3 <scsi_send_eh_cmnd+99>: mov rax,QWORD PTR [rax] Looking on the kernel code, there is this function : static inline struct scsi_driver *scsi_cmd_to_driver(struct scsi_cmnd *cmd) { if (!cmd->request->rq_disk) return NULL; return *(struct scsi_driver **)cmd->request->rq_disk->private_data; } The "test/je" matches the "if (!cmd->request->rq_disk)".
  15. 15. 2.2 - Crash : GDB for Kernel Let’s check the structures involved in this bug : crash> struct scsi_cmnd struct scsi_cmnd { … unsigned int transfersize; __struct request *request;__ unsigned char *sense_buffer; … } crash> struct request struct request { … struct gendisk *rq_disk; … } crash> struct -xo gendisk struct gendisk { … [0x2c0] struct request_queue *queue; [0x2c8] void *private_data; [0x2d0] int flags; … } So here we have a scsi_cmnd, which contains a “request”, which contains a “gendisk”. Here are the addresses of our different instances : ffff880c2d600ec0 = scsi_cmnd ffff880c372afd00 = request ffff880c2a4d0400 = gendisk Offset 0x2c8 matches the code just before the crash : 0xffffffff813860dc <scsi_send_eh_cmnd+92>: mov rax,QWORD PTR [rax+0x2c8] crash> struct gendisk.disk_name ffff880c2a4d0400 disk_name = "sg96000000000000...000" Disk is /dev/sg96. From "scsi_cmnd", the first element is an scsi_device object (addr : 0xffff880c3002f000 ) crash> scsi_device.vendor 0xffff880c3002f000 vendor = 0xffff880c2a3a2ac8 "QUANTUM Scalar i6000 656Q656Q.GS01501 001" crash> scsi_device.model 0xffff880c3002f000 model = 0xffff880c2a3a2ad0 "Scalar i6000 656Q656Q.GS01501 001" crash> scsi_device.rev 0xffff880c3002f000 rev = 0xffff880c2a3a2ae0 "656Q656Q.GS01501 001" Redhat Bug : https://access.redhat.com/solutions/1231363
  16. 16. 3 - Live modifications
  17. 17. 3 - Live modifications Yes, you can tinkle with the Kernel memory too ! Through /dev/mem, you can access memory… but not on most distributions. Dave Anderson says : Defeat CONFIG_STRICT_DEVMEM with kretprobes http: //www.redhat.com/archives/crash-utility/2008-March/msg00036.html /* Return-probe handler: force return value to be 1. */ static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs) { #if defined(__i386__) && !defined(__KERNEL__) regs->eax = 1; #else regs->ax = 1; #endif return 0; }
  18. 18. 3.1 - Live modifications - Network Parameters Get the list of the NICs : crash> net NET_DEVICE NAME IP ADDRESS(ES) ffff88003e999020 lo 127.0.0.1 ffff88003e228020 eth0 192.168.122.13 Check the value (net_device) crash> struct net_device.mtu ffff88003e228020 mtu = 1500 Get the offset crash> struct -o net_device.mtu ffff88003e228020 struct net_device { [ffff88003e22818c] unsigned int mtu; } Read the memory crash> rd -32 -D ffff88003e22818c ffff88003e22818c: 1500 And change it crash> wr -32 ffff88003e22818c 1400 [root@centos6 ~]# ifconfig eth0 |grep -Po 'MTU:[0-9]+' MTU:1400
  19. 19. 4 - Tools and useful links
  20. 20. 4.1 - Tools : OpenGrok Wicked fast code source browser http://opengrok.github.io/OpenGrok/ Grok : "to understand intuitively or by empathy; to establish rapport with" / "to empathize or communicate sympathetically (with); also, to experience enjoyment" Uses ctags and lucene to index code with context : Search for “text”, “definitions”, “symbols”, “file path” and “history” Understand : Mercurial, Git, SCCS, RCS, CVS, Subversion, Teamware, ClearCase, Perforce, Monotone and Bazaar
  21. 21. 4.2 - Tools : kdumptools Set of scripts to ease your kdump usage (try to work with all distributions) https://github.com/saruspete/kdumptools kdump_setup.sh Helper: setup kdump on your distrib kdump_analyze.sh Helper: analyze a crashdump (retrieve dbg + crash) kdump_live.sh Helper: analyze your running system kdump_getdbg.sh Helper: retrieve debuginfos for a given OS / Release src/crash Crash + compile scripts (latest version) src/allow_devmem Kernel module to allow /dev/mem usage
  22. 22. 4.3 - Links - kdump Kdump-Tool : Kexec is part of kexec-tools Sources : https://git.kernel.org/cgit/utils/kernel/kexec/kexec-tools.git Distrib : https://kernel.org/pub/linux/utils/kernel/kexec/ Kernel Doc : http://www.kernel.org/doc/Documentation/kdump/kdump.txt MakeDumpFile : Select the memory regions to be stripped of the dump https://github.com/chitranshi/makedumpfile Fence Kdump : Avoid kdump being interrupted by sending heartbeats http://www.ovirt.org/Fence_kdump
  23. 23. 4.4 - Links - crash Official Page : Download, tools and help http://people.redhat.com/anderson Linux Crash Cook Book : Detailed and step-by-step details http://www.dedoimedo.com/computers/crash-book.html Defeating /dev/mem restrictions : Howto tinkle with /dev/mem http://www.redhat. com/archives/crash-utility/2008-March/msg00036.html Dwarf debuginfo format : Details on the Dwarf format compatible with ELF binaries http://dwarfstd.org
  24. 24. 4.5 - Links - Kernel Linux Insides : https://0xax.gitbooks.io/linux-insides Understanding the Linux Kernel ISBN 10 : 0-596-00565-2 Linux Kernel Development ISBN 10 : 0-672-32946-8 Linux Kernel Architecture ISBN 10 : 0-470-34343-5 The Linux Programming Interface ISBN 10 : 1-59327-220-0
  25. 25. Thank you

×