The document discusses kernel debugging techniques. It covers analyzing kernel error logs, debugging memory errors, detecting kernel deadlocks, and troubleshooting login issues. The key debugging techniques mentioned are using kernel log functions to output error information, understanding the structure of kernel bug logs, tracing from assembly to C code, and using tools like AddressSanitizer to detect memory errors.

1. © 2015 Wind River. All Rights Reserved.1 © 2015 Wind River. All Rights Reserved.
Kernel Debugging
Firo Yang
yongming.yang@windriver.com

2. 2 © 2015 Wind River. All Rights Reserved.
Agenda
What is debugging
内核错误日志的分析
内存错误调试手段
内核死锁
无法登陆时的手段

3. 3 © 2015 Wind River. All Rights Reserved.
What is debugging?
● It’s abductive reasoning
● Theory + Observations => Explanation
● Explanation → Observations
● Bug types
● nvd.nist.gov/cwe.cfm
● Debugging techniques

4. 4 © 2015 Wind River. All Rights Reserved.
内核错误日志的分析
4 © 2015 Wind River. All Rights Reserved.

5. 5 © 2015 Wind River. All Rights Reserved.
General procedures
● Prerequisites
● Don’t Panic
● Hardware specifications
● Kernel Source code
● Read the F**ing Source Code :-) torvalds@cc.helsinki.fi
● The basic principle: assembly → C → Transactions

6. 6 © 2015 Wind River. All Rights Reserved.
Kernel log functions
● Kernel mostly use die() to output log info
• Panic, oops, traps
● Other functions
• Custom outputs

7. 7 © 2015 Wind River. All Rights Reserved.
The structure of kernel bug log
●
Components of oops messages
●
Bug title
●
Print modules
●
Registers
●
Stack & calltrace
●
Instruction code
●
Summary of this bug
●
Example by Linus
yarchive.net/comp/linux/oops_decoding.html

8. 8 © 2015 Wind River. All Rights Reserved.
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000008
printing eip: c022a7b5 *pde = 00000000 Oops: 0000 [#1] SMP
Modules linked in: thinkpad_acpi ppdev speedstep_lib cpufreq_conservative cpufreq_userspace cpufreq_ondemand cpufreq_stats
cpufreq_powersave freq_table video bay dock ac sbs button container battery lp irtty_sir sir_dev pcmcia parport_pc parport snd_cs46xx
gameport snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss nsc_ircc snd_pcm snd_seq_dummy irda crc_ccitt snd_seq_oss psmouse i2c_piix4
snd_seq_midi snd_rawmidi snd_seq_midi_event serio_raw pcspkr snd_seq i2c_core snd_timer snd_seq_device snd soundcore snd_page_alloc shpchp
pci_hotplug intel_agp yenta_socket rsrc_nonstatic pcmcia_core agpgart evdev ext3 jbd mbcache sg sr_mod cdrom sd_mod uhci_hcd usbcore
ata_piix ata_generic libata scsi_mod e100 mii thermal processor fan fuse apparmor commoncap
CPU: 0
EIP: 0060:[<c022a7b5>] Not tainted VLI
EFLAGS: 00010202 (2.6.22-12-generic #1)
EIP is at acpi_ns_internalize_name+0xd/0x83
eax: 00000008 ebx: 00000000 ecx: 00000000 edx: c7879e54
esi: d0b980c0 edi: c7879e54 ebp: c7879e70 esp: c7879de8
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process modprobe (pid: 4467, ti=c7878000 task=ce5c94c0 task.ti=c7878000)
Stack: 00000000 00000000 d0b97e60 00008080 c01c4390 d0b97e60 00000000 00000000
d0b980c0 00000000 c7879e70 c022a85c d0b97e60 c795d030 c7c604e0 c01c44ef
00000004 d0b97e60 c7acea18 c01c3884 00008080 00000004 00000004 00000080
Call Trace:
[<c01c4390>] __sysfs_new_dirent+0x20/0x50
[<c022a85c>] acpi_ns_get_node+0x31/0x93
[<c01c44ef>] sysfs_make_dirent+0x2f/0x50
[<c01c3884>] sysfs_add_file+0x74/0x90
[<d0b910b7>] drv_acpi_handle_init+0x37/0x90 [thinkpad_acpi]
[<c0231aef>] acpi_ut_release_mutex+0x5b/0x63
[<c0233ac0>] acpi_method_notify_enable+0x15/0x34
[<d0b5ba32>] cmos_init+0x52/0x70 [thinkpad_acpi]
[<d0b5c21f>] thinkpad_acpi_module_init+0x27f/0x69a [thinkpad_acpi]
[<c014a811>] sys_init_module+0x151/0x1a00
[<c01fb8cf>] prio_tree_insert+0x1f/0x250
[<c01041d2>] sysenter_past_esp+0x6b/0xa9
=======================
Code: c7 44 24 14 01 00 00 00 8b 54 24 14 8d 04 96 e9 f2 fe ff ff 83 c4 18 89 d0 5b 5e 5f 5d c3 55 57 89 d7 56 53 83 ec 1c 85 c0 74 67 <80>
38 00 74 62 85 d2 74 5e 89 04 24 89 e0 e8 b5 fb ff ff 8b 4c
EIP: [<c022a7b5>] acpi_ns_internalize_name+0xd/0x83 SS:ESP 0068:c7879de8
Bug title
Modules
Registers info
Stack and calltrace
Instructions
Summary

9. 9 © 2015 Wind River. All Rights Reserved.
CPU: 0
EIP: 0060:[<c022a7b5>] Not tainted VLI
EFLAGS: 00010202 (2.6.22-12-generic #1)
EIP is at acpi_ns_internalize_name+0xd/0x83
eax: 00000008 ebx: 00000000 ecx: 00000000 edx: c7879e54
esi: d0b980c0 edi: c7879e54 ebp: c7879e70 esp: c7879de8
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process modprobe (pid: 4467, ti=c7878000 task=ce5c94c0 task.ti=c7878000)
Stack: 00000000 00000000 d0b97e60 00008080 c01c4390 d0b97e60 00000000 00000000
d0b980c0 00000000 c7879e70 c022a85c d0b97e60 c795d030 c7c604e0 c01c44ef
00000004 d0b97e60 c7acea18 c01c3884 00008080 00000004 00000004 00000080
Call Trace:
[<c01c4390>] __sysfs_new_dirent+0x20/0x50
[<c022a85c>] acpi_ns_get_node+0x31/0x93
[<c01c44ef>] sysfs_make_dirent+0x2f/0x50
Crash-function
Crash-offset
Size of function

10. 10 © 2015 Wind River. All Rights Reserved.
From assembly to C
● CONFIG_DEBUG_INFO
● Gdb vmlinux
● Objdump -S vmlinux

11. 11 © 2015 Wind River. All Rights Reserved.
From C to specific transactions
● Marcos – the disaster
● Solutions
● Gcc option: -save-temps
● Use DWARF debug info

12. 12 © 2015 Wind River. All Rights Reserved.
DWARF macro debug info
Build kernel with -g3

13. 13 © 2015 Wind River. All Rights Reserved.
内存越界
13 © 2015 Wind River. All Rights Reserved.

14. 14 © 2015 Wind River. All Rights Reserved.
Weaknesses that Affect Memory

Improper Restriction of Operations within the Bounds of a
Memory Buffer

Memory Resource Management Errors

Information Exposure

15. 15 © 2015 Wind River. All Rights Reserved.
Improper Restriction of Operations within
the Bounds of a Memory Buffer

Out-of-bounds Read/Write

Stack-based Buffer Overflow

Heap-based Buffer Overflow

Else?

Access of Resource Using Incompatible Type ('Type
Confusion')

16. 16 © 2015 Wind River. All Rights Reserved.
Memory Resource Management Errors

Improper Release of Memory Before Removing Last
Reference ('Memory Leak')

Use After Free

Double Free

17. 17 © 2015 Wind River. All Rights Reserved.
Detection Methods

Static Analysis

Static syntax checker: smatch

Code review

Dynamic Analysis

User space: Valgrind, AddressSanitizer

Kernel: KASAN, memleak

18. 18 © 2015 Wind River. All Rights Reserved.
AddressSanitizer

A compiled and fast memory error detector for c/c++.

Buffer overflows in heap, stack and globals

heap-use-after-free, stack-use-after-return

The average slowdown is ~2x

The average memory overhead is ~2-3x

10000+ bugs found (Chromium, Firefox, ...)


How to use

# gcc -fsanitize=address main.c

19. 19 © 2015 Wind River. All Rights Reserved.
Example of AddressSnitizer
# cat sanitizer.c
static short zero = 0;
int main(int argc, char **argv) {
unsigned long *p = &zero;
return *p;
}
# gcc -g -O -fsanitize=address -o sanitizer sanitizer.c

20. 20 © 2015 Wind River. All Rights Reserved.

21. 21 © 2015 Wind River. All Rights Reserved.
How does it work?
?

22. 22 © 2015 Wind River. All Rights Reserved.
How does it work?

Based on compiler instrumentation

23. 23 © 2015 Wind River. All Rights Reserved.
How does it work?

Based on compiler instrumentation

an instrumentation module – gcc/asan.c

a run-time library – libsanitizer or kernel src

24. 24 © 2015 Wind River. All Rights Reserved.
How does it work?

Instrumentation - Inject codes

Poison shadow memory

Check the shadow state for each memory
access


25. 25 © 2015 Wind River. All Rights Reserved.
VMA to shadow memory

Shadow memory = 1/8 of the Virtual memory space

On x86_64, offset is 0x100000000000


26. 26 © 2015 Wind River. All Rights Reserved.
Shadow states

Each 8 bytes of Virtual memory space is tracked by 1 byte in shadow
memory

Shadow states

27. 27 © 2015 Wind River. All Rights Reserved.
InstrumentationDump of assembler code for function main:
0x0000000000400846 <+0>: sub $0x8,%rsp
0x000000000040084a <+4>: mov $0x4009a0,%edi
0x000000000040084f <+9>:callq 0x400720 <puts@plt>
0x0000000000400854 <+14>: mov $0x4009e0,%eax
0x0000000000400859 <+19>: mov %rax,%rdx
0x000000000040085c <+22>: shr $0x3,%rdx
0x0000000000400860 <+26>: movzbl 0x7fff8000(%rdx),%ecx
0x0000000000400867 <+33>: mov $0x4009e7,%edx
0x000000000040086c <+38>: mov %rdx,%rsi
0x000000000040086f <+41>: shr $0x3,%rsi
0x0000000000400873 <+45>: movzbl 0x7fff8000(%rsi),%esi
0x000000000040087a <+52>: and $0x7,%eax
0x000000000040087d <+55>: cmp %al,%cl
0x000000000040087f <+57>: setle %dil
0x0000000000400883 <+61>: test %cl,%cl
0x0000000000400885 <+63>: setne %al
0x0000000000400888 <+66>: test %al,%dil
0x000000000040088b <+69>: jne 0x4008ab <main+101>
0x000000000040088d <+71>: and $0x7,%edx
0x0000000000400890 <+74>: cmp %dl,%sil
0x0000000000400893 <+77>: setle %dl
0x0000000000400896 <+80>: test %sil,%sil
0x0000000000400899 <+83>: setne %al
0x000000000040089c <+86>: test %al,%dl
0x000000000040089e <+88>: jne 0x4008ab <main+101>
0x00000000004008a0 <+90>: mov 0x13a(%rip),%eax # 0x4009e0
<zero>
0x00000000004008a6 <+96>: add $0x8,%rsp
0x00000000004008aa <+100>: retq
0x00000000004008ab <+101>: mov $0x8,%esi
0x00000000004008b0 <+106>: mov $0x4009e0,%edi
0x00000000004008b5 <+111>: callq 0x4006e0 <__asan_report_load_n@plt>
Dump of assembler code for function main:
0x00000000004004d6 <+0>: mov 0x94(%rip),%eax
# 0x400570 <zero>
0x00000000004004dc <+6>: retq
static short zero = 0;
int main(int argc, char **argv)
{
unsigned long *p = &zero;
return *p;
}

28. 28 © 2015 Wind River. All Rights Reserved.
Instrumentation
static short zero = 0;
int main(int argc, char **argv)
{
unsigned long *p = &zero;
unsigned long k = (p >> 3) + Offset;
int size = sizeof(unsigned long);
if (k != 0 && ((p & 7) + size > k))
ReportAndCrash(p);
return *p;
}

29. 29 © 2015 Wind River. All Rights Reserved.
Interceptors & reporter

Libsanitizer for userspace

KASAN for kernel

Originate from Google

100+ bug founded

30. 30 © 2015 Wind River. All Rights Reserved.
KASAN shadow memory layout
0000000000000000 - 00007fffffffffff (=47 bits) user space, different per mm
hole caused by [48:63] sign extension
ffff800000000000 - ffff87ffffffffff (=43 bits) guard hole, reserved for hypervisor
ffff880000000000 - ffffc7ffffffffff (=64 TB) direct mapping of all phys. memory
ffffc80000000000 - ffffc8ffffffffff (=40 bits) ho le
ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space
ffffe90000000000 - ffffe9ffffffffff (=40 bits) hole
ffffea0000000000 - ffffeaffffffffff (=40 bits) virtual memory map (1TB) ... unused hole ...
ffffec0000000000 - fffffc0000000000 (=44 bits) kasan shadow memory (16TB)
ffffff0000000000 - ffffff7fffffffff (=39 bits) %esp fixup stacks ... unused hole ...
ffffffff80000000 - ffffffffa0000000 (=512 MB) kernel text mapping, from phys 0
ffffffffa0000000 - ffffffffff5fffff (=1525 MB) module mapping space
ffffffffff600000 - ffffffffffdfffff (=8 MB) vsyscalls
ffffffffffe00000 - ffffffffffffffff (=2 MB) unused hole

31. 31 © 2015 Wind River. All Rights Reserved.
Setup shadow memory

32. 32 © 2015 Wind River. All Rights Reserved.
Implementations

Memcpy/memset/memmove

Kmalloc/kfree

kasan_report

mm/kasan/

33. 33 © 2015 Wind River. All Rights Reserved.
Implementations

34. 34 © 2015 Wind River. All Rights Reserved.
Implementations

35. 35 © 2015 Wind River. All Rights Reserved.
Other kernel facilities
DEBUG_PAGE_ALLOC
Poison SLAB

36. 36 © 2015 Wind River. All Rights Reserved.
操作系统挂死（内核报错）
36 © 2015 Wind River. All Rights Reserved.

37. 37 © 2015 Wind River. All Rights Reserved.
Lockup
● Soft lockup
– No schedule lasts more than 20 secs
● Hard lockup
– No interrupts lasts more than 10 secs
● Lock detectors
– Watchdog

38. 38 © 2015 Wind River. All Rights Reserved.
Soft Lockup detector
[ 584.399202] BUG: soft lockup - CPU#0 stuck for 23s! [rsync:3603]
[ 584.496357] [<c001369c>] (unwind_backtrace+0x0/0xe0) from [<c00645ac>] (watchdog_timer_fn+0xe0/0x134)
[ 584.505615] [<c00645ac>] (watchdog_timer_fn+0xe0/0x134) from [<c004365c>] (__run_hrtimer+0x118/0x1ec)
[ 584.514875] [<c004365c>] (__run_hrtimer+0x118/0x1ec) from [<c0043e80>] (hrtimer_interrupt+0xe8/0x230)
[ 584.524138] [<c0043e80>] (hrtimer_interrupt+0xe8/0x230) from [<c001a1dc>] (orion_timer_interrupt+0x20/0x30)
[ 584.533921] [<c001a1dc>] (orion_timer_interrupt+0x20/0x30) from [<c0064e9c>] (handle_irq_event_percpu+0x7c/0x23c)
[ 584.544226] [<c0064e9c>] (handle_irq_event_percpu+0x7c/0x23c) from [<c0065084>] (handle_irq_event+0x28/0x38)
[ 584.554098] [<c0065084>] (handle_irq_event+0x28/0x38) from [<c0067240>] (handle_level_irq+0xac/0xc0)
[ 584.563271] [<c0067240>] (handle_level_irq+0xac/0xc0) from [<c006486c>] (generic_handle_irq+0x28/0x44)
[ 584.572621] [<c006486c>] (generic_handle_irq+0x28/0x44) from [<c000ed94>] (handle_IRQ+0x60/0x84)
[ 584.581443] [<c000ed94>] (handle_IRQ+0x60/0x84) from [<c000dab4>] (__irq_svc+0x34/0x78)

39. 39 © 2015 Wind River. All Rights Reserved.
Dispose soft lockup detector
● IRQ
– Hrtimer
● Watchdog thread
– RT thread

40. 40 © 2015 Wind River. All Rights Reserved.
Hard lockup
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 18
Pid: 12410, comm: xxxx Not tainted 2.6.32-220.el6.x86_64 #1
Call Trace:
<NMI> [<ffffffff814f8464>] ? panic+0x8b/0x156
[<ffffffff810dac1a>] ? watchdog_overflow_callback+0x1fa/0x200
[<ffffffff8110cb3d>] ? __perf_event_overflow+0x9d/0x230
[<ffffffff8110d0f4>] ? perf_event_overflow+0x14/0x20
[<ffffffff8101e396>] ? intel_pmu_handle_irq+0x336/0x550
[<ffffffff814fe156>] ? kprobe_exceptions_notify+0x16/0x450
[<ffffffff814fcc39>] ? perf_event_nmi_handler+0x39/0xb0
[<ffffffff814fe7a5>] ? notifier_call_chain+0x55/0x80
[<ffffffff814fe80a>] ? atomic_notifier_call_chain+0x1a/0x20
[<ffffffff81097dce>] ? notify_die+0x2e/0x30
[<ffffffff814fc3c3>] ? do_nmi+0x173/0x2c0
[<ffffffff814fbcd0>] ? nmi+0x20/0x30
[<ffffffff814fb465>] ? _spin_lock_irq+0x25/0x40

41. 41 © 2015 Wind River. All Rights Reserved.
Discompose hard lockup
detector
● NMI
– Despite irq disabling
● Performance event subsystem
– Portable

42. 42 © 2015 Wind River. All Rights Reserved.
Common reasons
● hard-lockup
– Disable interrupts but not re-enable them.
– Deadlock
● soft lockup
– Interrupts fire continuously (staging new driver)
– Infinitive loop because of integer overflow
– Deadlock

43. 43 © 2015 Wind River. All Rights Reserved.
Types of deadlocks
● Non-recursive lock
– Take twice
● Lock AB-BA
– Complex deadlock chain
● Special case
– Interrupt

44. 44 © 2015 Wind River. All Rights Reserved.
Non-recursive
● Take twice
● What about spinlock?
● What about mutex?

45. 45 © 2015 Wind River. All Rights Reserved.
Lock AB-BA
● Uni-core?
● Mutex or Spinlock?

46. 46 © 2015 Wind River. All Rights Reserved.
DEADLOCK
CPU 0 CPU 1
Lock A
Lock B
Lock B
Lock C
CPU 2
Lock C
Lock A
Lock chain: AB BC CA

47. 47 © 2015 Wind River. All Rights Reserved.
Mess up with interrupt
● If a lock will be taken in interrupt, then it must have
interrupts disabled when that lock is ever taken.
●
DEADLOCK
Process
Lock A
Interrupt
Lock A

48. 48 © 2015 Wind River. All Rights Reserved.
CPU 0
lock A
lock B
IRQ disable
CPU 1
lock B
IRQ enabled
DEADLOCKLock A
Interrupt

49. 49 © 2015 Wind River. All Rights Reserved.
Interrupt CPU 0
lock A
unlock A
lock A
lock B
IRQ disable
unlock B
unlock A
Enable
IRQ
CPU 1
lock B
lock C
IRQ disable
unlock C
unlock B
Enable
IRQ
CPU 2
lock C
IRQ enabled
unlock C
Deadlock?

50. 50 © 2015 Wind River. All Rights Reserved.
Lockdep
● Writen by Ingo Molnar in 2006
– Countless bugs found
● Lock correctness validator

51. 51 © 2015 Wind River. All Rights Reserved.
Lock classes
● Locks are tracked via “classes”, not
instances
● Spinlock in inode

52. 52 © 2015 Wind River. All Rights Reserved.
Dynamic detecting deadlock
● When locks are taken at runtime, the class
is updated.
● Lock_acquire() embedded in spinlock,
mutex, semaphore.

53. 53 © 2015 Wind River. All Rights Reserved.
[ INFO: possible circular locking dependency detected ]
2.6.32-504.16.2.1chaos.ch5.3.x86_64.debug #1
-------------------------------------------------------
txg_sync/1894 is trying to acquire lock:
(&db->db_mtx){+.+.+.}, at: [<ffffffffa033e558>] dbuf_sync_leaf+0x48/0x4f0 [zfs]
but task is already holding lock:
(&dr->dt.di.dr_mtx){+.+...}, at: [<ffffffffa033eb62>] dbuf_sync_indirect+0x102/0x330 [zfs]
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&dr->dt.di.dr_mtx){+.+...}:
[<ffffffff810bfbbe>] __lock_acquire+0xede/0x1560
[<ffffffff810c02e4>] lock_acquire+0xa4/0x120
[<ffffffff8155f24c>] mutex_lock_nested+0x5c/0x3b0
[<ffffffffa033f727>] dbuf_dirty+0x4a7/0xdf0 [zfs]
-> #0 (&db->db_mtx){+.+.+.}:
[<ffffffff810c013a>] __lock_acquire+0x145a/0x1560
[<ffffffff810c02e4>] lock_acquire+0xa4/0x120
[<ffffffff8155f24c>] mutex_lock_nested+0x5c/0x3b0
[<ffffffffa033e558>] dbuf_sync_leaf+0x48/0x4f0 [zfs]

54. 54 © 2015 Wind River. All Rights Reserved.
(&dr>dt.di.dr_mtx){+. +. ..},
Hardirqs Softirqs reclaim_fs?

55. 55 © 2015 Wind River. All Rights Reserved.
Lockdep implementations
● Three lists
– Current held locks
– Lock-class’s before list and after list.

56. 56 © 2015 Wind River. All Rights Reserved.
内核网口串口均无法登陆时的处
理方法
56 © 2015 Wind River. All Rights Reserved.

57. 57 © 2015 Wind River. All Rights Reserved.
Can not login?
 Common solutions
− Use early printk to print more log
− If keyboard available, Sysrq
− Use watchdog to dump current status


58. 58 © 2015 Wind River. All Rights Reserved.
Questions is welcomed
“Have a nice day:-)
Firo Yang