The document discusses bug hunting in the Linux kernel, emphasizing its complexity with millions of lines of code and frequent updates. It covers various debugging techniques, tools, and the use of eBPF for performance monitoring and tracing. The conclusion highlights the benefits of virtualization and tracing in kernel development, encouraging a fun approach to understanding and improving the kernel.

1. LINUX KERNEL BUG HUNTINGLINUX KERNEL BUG HUNTING
Andrea Righi
<andrea.righi@canonical.com>
Twitter: @arighi

2. LINUX KERNEL IS COMPLEXLINUX KERNEL IS COMPLEX
26,947,416 lines of code right now
438 patches last week
813 files changed, 7714 insertions(+), 31289
deletions(-)
find -type f -name '*.[chS]' -exec wc -l {} ; |
awk 'BEGIN{sum=0}{sum+=$1}END{print sum}'
git log --oneline v5.10-rc1..v5.10-rc2 | wc -l
git diff --stat v5.10-rc1..v5.10-rc2 | tail -1

3. PREPARING THE ENVIRONMENTPREPARING THE ENVIRONMENT
Development
git, vim, ctags, mutt, b4
Build
make, gcc
Testing
qemu, virtme, uvt-kvm, tmux
Debugging
gdb, crash, bp race

4. GET THE SOURCE CODEGET THE SOURCE CODE
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torval

5. KERNEL BUGSKERNEL BUGS
Kernel panic
Fatal error, system becomes unusable
Kernel oops
Non-fatal error
Wrong result
Fata error from user perspective
Security vulnerability
Data leakage
Performance regression
Everything correct, but slower

6. CRASHING THE KERNELCRASHING THE KERNEL

7. DEBUGGING TECHNIQUESDEBUGGING TECHNIQUES
blinking LED
printk() / dump_stack()
procfs
debugger (i.e., kgbd)
in-kernel tools (i.e., lockdep)
virtualization
profiling / tracing

8. PROFILING VS TRACINGPROFILING VS TRACING
Profiling
Tracing
Periodic timed interrupt that collects current
program counter, function address, stack back
trace
Record invocations of a specific event

9. PROFILING EXAMPLEPROFILING EXAMPLE
perf top

10. TRACING EXAMPLETRACING EXAMPLE
strace: system call tracer in Linux
It uses ptrace() syscall that pauses the targer
process any time it executes a syscall so that the
debugger can read the state
... and it's doing it twice: when the syscall begins
and when it ends!

11. STRACE OVERHEADSTRACE OVERHEAD
### Regular execution
$ dd if=/dev/zero of=/dev/null bs=1 count=500k
512000+0 records in
512000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 0.147195 s, 3.5 MB/s
### Strace execution (tracing a syscall that is never called)
$ strace -e trace=bind
dd if=/dev/zero of=/dev/null bs=1 count=500k
512000+0 records in
512000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 8.0567 s, 63.5 kB/s
+++ exited with 0 +++

12. eBPFeBPF
Highly efficeint VM that lives in the kernel
Inject safe bytecode into the kernel that runs in a
sandbox
In-kernel JIT compiler
Can use kprobe to attach code to kernel functions
Access kernel memory without the risk of
crashing, hanging or breaking the system

13. KPROBESKPROBES
>

14. BTF & CO-REBTF & CO-RE
BPF type format
Metadata format which encodes debugging
information of kernel data structures
CO-RE
BPF Compile-Once Run-Everywhere
Allows eBPF bytecode to be relocatable
http://www.brendangregg.com/blog/2020-11-
04/bpf-co-re-btf-libbpf.html

15. eBPFeBPF TRACING TOOLSTRACING TOOLS
BPF Compiler Collection
git://github.com/iovisor/bcc.git
(deprecated)
bp race
git://github.com/iovisor/bp race.git

16. FLAME GRAPHSFLAME GRAPHS

17. EXAMPLESEXAMPLES
keylogger in eBPF
tracing 'ping' with eBPF
tracing task wait / wakeup events

18. KEYLOGGER WITH BPFTRACEKEYLOGGER WITH BPFTRACE

19. KEYLOGGER WITH BPFTRACE:KEYLOGGER WITH BPFTRACE:
COMMANDSCOMMANDS
sudo bpftrace -e 'kprobe:*interrupt* { @[probe] = count() }'
sudo bpftrace -e 'kprobe:atkbd_interrupt
{ printf("%xn", arg1) }'

20. TRACING 'PING' WITH BPFTRACETRACING 'PING' WITH BPFTRACE

21. TRACING 'PING' WITH BPFTRACE:TRACING 'PING' WITH BPFTRACE:
COMMANDSCOMMANDS
sudo bpftrace -e 'kprobe:*icmp* { @[probe] = count(); }'
sudo bpftrace -e 'kprobe:icmp_out_count
{ printf("%s: type=%d%sn", probe, arg1, kstack); }'

22. TRACING TASK WAIT / WAKUPTRACING TASK WAIT / WAKUP
EVENTS WITH BPFTRACEEVENTS WITH BPFTRACE
#!/usr/bin/env bpftrace
#include <linux/sched.h>
BEGIN
{
printf("Tracing off-CPU time (us) with waker stacks. Ctrl-C
}
kprobe:try_to_wake_up
{
$p = (struct task_struct *)arg0;
@waker[$p->pid] = kstack;
}

23. CONCLUSIONCONCLUSION
Virtualization is your friend to speed up kernel
development
Tracing is your friend to understand how the kernel
works
Kernel development can be fun!

24. REFERENCESREFERENCES
https://git.launchpad.net/~arighi/+git/kernel-tools
http://arighi.blogspot.com/2019/08/kerneldebuggingusingqemu.html
https://github.com/arighi/virtme
http://www.brendangregg.com/BPF/bp race-cheat-sheet.html
https://github.com/iovisor/bcc

25. THANK YOUTHANK YOU
Questions?
Andrea Righi
<andrea.righi@canonical.com>
twitter: @arighi