Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Andrea Righi - Spying on the Linux kernel for fun and profit


Published on

Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.

This talk explores methodologies that allow to take a look “live” at kernel internal operations, from a network perspective, to I/O paths, CPU usage, memory allocations, etc., using in-kernel technologies, like eBPF and ftrace. Understanding such kernel internals can be really helpful to track down performance bottlenecks, debug system failures and it can be also a very effective way to approach to kernel development.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Andrea Righi - Spying on the Linux kernel for fun and profit

  1. 1. Spying on the Linux kernel for fun and profit Andrea Righi Twitter: @arighi Github profile:
  2. 2. Linux kernel is complex
  3. 3.
  4. 4. Linux kernel changes
  5. 5. How to keep up with changes ● ● ● ● kernel source: Documentation/
  6. 6. Real-time tracing
  7. 7. strace ● strace(1): system call tracer in Linux ● It uses the ptrace() system call that pauses the target process for each syscall so that the debugger can read the state ● And it’s doing this twice: when the syscall begins and when it ends!
  8. 8. strace overhead ### Regular execution ### $ dd if=/dev/zero of=/dev/null bs=1 count=500k 512000+0 records in 512000+0 records out 512000 bytes (512 kB, 500 KiB) copied, 0,501455 s, 1.0 MB/s ### Strace execution (tracing a syscall that is never called) ### $ strace -e trace=accept dd if=/dev/zero of=/dev/null bs=1 count=500k 512000+0 records in 512000+0 records out 512000 bytes (512 kB, 500 KiB) copied, 44.0216 s, 11,6 kB/s +++ exited with 0 +++
  9. 9. Advanced tracing techniques
  10. 10. eBPF
  11. 11. eBPF features ● Highly efficient VM that lives in the kernel ● Inject safe sanboxed bytecode into the kernel ● Attach code to kernel functions / events ● In-kernel JIT compiler – Dynamically translate eBPF bytecode into native opcodes ● eBPF makes kernel programmable without having to cross kernel/user-space boundaries ● Access in-kernel data structures directly without the risk of crashing, hanging or breaking the kernel in any way
  12. 12. eBPF history ● Initially it was BPF: Berkeley Packet Filter ● It has its roots in BSD in the very early 1990’s ● Originally designed as a mechanism for fast filtering network packets ● 3.15: Linux introduced eBPF: extended Berkeley Packet Filter ● More efficient / more generic than the original BPF ● 3.18: eBPF VM exposed to user-space ● 4.9: eBPF programs can be attached to perf_events ● 4.10: eBPF programs can be attached to cgroups ● 4.15: eBPF LSM hooks
  13. 13. eBPF as a VM ● Example assembly of a simple eBPF filter ● Load 16-bit quantity from offset 12 in the packet to the accumulator (ethernet type) ● Compare the value to see if the packet is an IP packet ● If the packet is IP, return TRUE (packet is accepted) ● otherwise return 0 (packet is rejected) ● Only 4 VM instructions to filter IP packets! ldh [12] jeq #ETHERTYPE_IP, l1, l2 l1: ret #TRUE l2: ret #0
  14. 14. eBPF use cases
  15. 15. kprobe
  16. 16. BCC tracing tools ● BPF Compiler Collection ● Front-end to eBPF ● BCC makes eBPF programs easier to write – Include C wrapper around LLVM – Python – Lua – C++ – C helper libs ● golang (gobpf) ●
  17. 17. Examples
  18. 18. Example #1: trace exec() ● Intercept all the processes executed in the system
  19. 19. Example #2: keylogger ● Identify where and how keyboard characters are received and processed by the kernel
  20. 20. Example #3: ping ● Identify where ICMP packets (ECHO_REQUEST / ECHO_REPLY) are received and processed by the kernel
  21. 21. Example #4: task wait / wakeup ● Determine the stack trace of a sleeping process and the stack trace of the process that wakes up a sleeping process
  22. 22. Conclusion ● Real-time tracing as a method to study the kernel ● Understanding what the kernel is doing can help to improve your application / service in terms of performance, reliability and security
  23. 23. References ● Brendan Gregg blog ● ● BCC tools ● ● gobpf (BPF bindings for go): ● ● The BSD Packet Filter: A New Architecture for User-level Packet Capture - S. McCanne and V. Jacobson ●
  24. 24. Thanks ● twitter: @arighi ● email: