Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BUD17-TR04: Kernel Debug Stories

1,194 views

Published on

"Session ID: BUD17-TR04
Session Name: Kernel Debug Stories - BUD17-TR04
Speaker:
Track:


★ Session Summary ★
This session is a short introductory course on Linux kernel debugging. The course is will examine a number of different debugging challenges and discuss the techniques and tools that can be employed to overcome them. By focusing on stories rather than the minute details of each tool we can cover a lot of topics in a short space of time, providing a springboard for further independent study by trainees.
---------------------------------------------------
★ Resources ★
Event Page: http://connect.linaro.org/resource/bud17/bud17-tr04/
Presentation: https://www.slideshare.net/linaroorg/bud17tr04-kernel-debug-stories
Video: https://youtu.be/LA2dxBMatnw
---------------------------------------------------

★ Event Details ★
Linaro Connect Budapest 2017 (BUD17)
6-10 March 2017
Corinthia Hotel, Budapest,
Erzsébet krt. 43-49,
1073 Hungary

---------------------------------------------------
Keyword: kernel, debug
http://www.linaro.org
http://connect.linaro.org
---------------------------------------------------
Follow us on Social Media
https://www.facebook.com/LinaroOrg
https://twitter.com/linaroorg
https://www.youtube.com/user/linaroorg?sub_confirmation=1
https://www.linkedin.com/company/1026961"

Published in: Technology
  • Be the first to comment

BUD17-TR04: Kernel Debug Stories

  1. 1. ENGINEERS AND DEVICES WORKING TOGETHER
  2. 2. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ○ ○ ○ ● ○ ●
  3. 3. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ printk() ● ○ ○ ○ ● ○ ○ ○ ○ ● ○
  4. 4. ENGINEERS AND DEVICES WORKING TOGETHER 0x0 0x0000ffffffffffff 256TB 0xffff000000000000 0xffff000008000000 128MB 0xffff000008000000 0xffff00000807ffff 512KB 0xffff000008080000 10MB+ Kernel image specific 0xffff7dffbfff0000 ~126TB 0xffff7dfffe7fd000 0xffff7dfffec00000 4MB + 12KB 0xffff7dfffee00000 0xffff7dffffe00000 16MB 0xffff7e0000000000 0xffff800000000000 2048G 0xffff800000000000 0xffffffffffffffff 128TB
  5. 5. ENGINEERS AND DEVICES WORKING TOGETHER
  6. 6. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ⇒ ○ ⇒ ○ ⇒ ○ ⇒ ● ○ printk ○ git grep printk ● ○ ○ ○
  7. 7. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ FRAME_WARN ○ SCHED_STACK_END_CHECK ● ○ slub_debug= ■ SLUB_DEBUG ○ DEBUG_PAGEALLOC ○ PAGE_POISONING ● ○ DEBUG_SPINLOCK ○ DEBUG_MUTEXES ○ DEBUG_ATOMIC_SLEEP ○ DEBUG_LOCK_ALLOC ○ LOCKUP_DETECTOR ●
  8. 8. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ● ○ ○ PROVE_LOCKING ○ KASAN
  9. 9. ENGINEERS AND DEVICES WORKING TOGETHER PROVE_LOCKING lock(subsys) lock(driver) lock(driver) lock(subsys) DEBUG_SPINLOCK DEBUG_MUTEXES lock(subsys) lock(driver) unlock(driver) unlock(subsys) lock(driver) lock(subsys) PROVE_LOCKING PROVE_LOCKING
  10. 10. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ○ ○ ○ ● ○ ●
  11. 11. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ●
  12. 12. ENGINEERS AND DEVICES WORKING TOGETHER make cscope git grep 'struct foo {' printk() [ 0.001636] xyz: Found 10 widgets ⇒ git grep "Found .* widgets" I can’t reproduce but my customer can
  13. 13. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ⇒ ○ quiet LOG_BUF_SHIFT log_bug_len= ● pr_debug() ○ <8> <7> ○ ○ DYNAMIC_DEBUG I can’t reproduce but my customer can
  14. 14. ENGINEERS AND DEVICES WORKING TOGETHER I can’t reproduce but my customer can SP LR (r30) FP (r29) r19..r28 r18 IP0/IP1 (r16-r17) r9..r15 r8 r0..r7
  15. 15. ENGINEERS AND DEVICES WORKING TOGETHER git grep REGMAP_ALLOW_WRITE_DEBUGFS I can’t reproduce but my customer can config DEBUG_FS bool "Debug Filesystem" select SRCU help debugfs is a virtual file system that kernel developers use to put debugging files into. Enable this option to be able to read and write to these files. If unsure, say N.
  16. 16. ENGINEERS AND DEVICES WORKING TOGETHER I can’t reproduce but my customer can <release_thread>: stp x29, x30, [sp,#-16]! mov x29, sp mov x0, x30 bl ffff000008092ea0 <_mcount> ldp x29, x30, [sp],#16 ret <release_thread>: stp x29, x30, [sp,#-16]! mov x29, sp mov x0, x30 nop ldp x29, x30, [sp],#16 ret <release_thread>: stp x29, x30, [sp,#-16]! mov x29, sp mov x0, x30 bl ftrace_caller ldp x29, x30, [sp],#16 ret
  17. 17. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ○ ftrace=function ftrace_filter=mydrv_* tp_printk ○ ftrace=function_graph ftrace_graph_notrace=rcu*,*lock,*spin* ● cd /sys/kernel/debug/tracing echo function > current_tracer echo 1 > tracing_on ● ○ ○ ○ ftrace_dump_on_oops ○ ● trace_printk() I can’t reproduce but my customer can
  18. 18. ENGINEERS AND DEVICES WORKING TOGETHER /proc/vmcore console=ttyS0,1115200 … … crashkernel=128M -p ./kexec -p vmlinux --dtb=xxx.dtb --append="root=/dev/mmcblk0p9 rw 1 maxcpus=1 reset_devices" I can’t reproduce but my customer can
  19. 19. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ● ● printk()
  20. 20. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ○ ○ ○ echo 'sched:*' > /sys/kernel/debug/tracing/set_event My widget missed its deadline
  21. 21. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ● My widget missed its deadline
  22. 22. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● .
  23. 23. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ●
  24. 24. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ initcall_debug ○ initcall_debug clk_disable_unused() clk_ignore_unused ○ DEBUG_SPINLOCK DEBUG_MUTEX LOCKUP_DETECTOR ○ ● ○ ○ ○ ○ ○ My board just stopped dead
  25. 25. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ● ○ ○ ○ ● ○ ○ ○ ● ○ ○ My board just stopped dead
  26. 26. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ○ ○ ○ ● ● ● My board just stopped dead
  27. 27. ENGINEERS AND DEVICES WORKING TOGETHER My board just stopped dead [ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78 [ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78 [ 277.600833] PCSR of cluster0 cpu2 is 0xffff00000809a934 [ 277.609564] PCSR of cluster0 cpu3 is 0xffff00000809a934
  28. 28. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ○ ○ ○
  29. 29. ENGINEERS AND DEVICES WORKING TOGETHER git bisect start <bad_sha1> <good_sha1> git bisect run compile_kernel_and_run_test.sh compile_kernel_and_test.sh compile_kernel_and_test.sh I’m sure this used to work
  30. 30. ENGINEERS AND DEVICES WORKING TOGETHER ktest.pl compile_kernel_and_test.sh Documentation/ tools/testing/ktest/sample.conf tools/testing/ktest/examples I’m sure this used to work
  31. 31. ENGINEERS AND DEVICES WORKING TOGETHER ● ● ● ● ● ●
  32. 32. ENGINEERS AND DEVICES WORKING TOGETHER My board just randomly failed
  33. 33. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ○ ○ ○ ● ○ ●
  34. 34. ENGINEERS AND DEVICES WORKING TOGETHER
  35. 35. ENGINEERS AND DEVICES WORKING TOGETHER Graphic by Jost, CC0-PD
  36. 36. ENGINEERS AND DEVICES WORKING TOGETHER ● ○ ○ ○ ● ○ ○ ○ ○ ○ ● ○ ●
  37. 37. ENGINEERS AND DEVICES WORKING TOGETHERom: http://linaro.co/bud17-kdb

×