Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Staring into the eBPF Abyss

1,348 views

Published on

eBPF (extended Berkeley Packet Filters) is a modern kernel technology that can be used to introduce dynamic tracing into a system that wasn't prepared or instrumented in any way. The tracing programs run in the kernel, are guaranteed to never crash or hang your system, and can probe every module and function -- from the kernel to user-space frameworks such as Node and Ruby.

In this workshop, you will experiment with Linux dynamic tracing first-hand. First, you will explore BCC, the BPF Compiler Collection, which is a set of tools and libraries for dynamic tracing. Many of your tracing needs will be answered by BCC, and you will experiment with memory leak analysis, generic function tracing, kernel tracepoints, static tracepoints in user-space programs, and the "baked" tools for file I/O, network, and CPU analysis. You'll be able to choose between working on a set of hands-on labs prepared by the instructors, or trying the tools out on your own test system.

Next, you will hack on some of the bleeding edge tools in the BCC toolkit, and build a couple of simple tools of your own. You'll be able to pick from a curated list of GitHub issues for the BCC project, a set of hands-on labs with known "school solutions", and an open-ended list of problems that need tools for effective analysis. At the end of this workshop, you will be equipped with a toolbox for diagnosing issues in the field, as well as a framework for building your own tools when the generic ones do not suffice.

Published in: Software
  • Be the first to comment

Staring into the eBPF Abyss

  1. 1. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Staring into the eBPF Abyss Sasha Goldshtein CTO, Sela Group @goldshtn
  2. 2. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Agenda • Modern Linux tracing landscape • BPF • BCC – BPF Compiler Collection • Using BCC tools • Authoring BCC tools
  3. 3. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Prerequisites • You should … • Have experience developing on or administering a Linux deployment • Be familiar with C/Python/Lua (a bonus) • To use your own machines for this workshop … • You will need Linux 4.6+ • Clone or install some open source tools (perf, bcc) • You can also use the instructor-provided VirtualBox appliance or Strigo workspace • Instructions and labs: https://github.com/goldshtn/linux-tracing-workshop
  4. 4. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Tracing on the Performance Spectrum Invasiveness Envelope System completeness Metrics & simulations Development Testing Production Profilers jprof, valgrind Counters top, vmstat Debuggers Function tracers Event tracers Aggregators SystemTap, BPF Load tools ab Light-weight profilers perf
  5. 5. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Tracing Objectives • Trace function execution, arguments, call graph • Print lightweight log messages (kernel/user) • Aggregate statistics (min/max/avg, histogram) • Low overhead • Continuous monitoring
  6. 6. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Linux Tracing Tools, Today Ease of use BPF/BCC SysDig ktap SystemTap LTTng ftrace perf custom .ko new stable dead Level of detail, features
  7. 7. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Berkeley Packet Filters (BPF) • Originally designed for, well, packet filtering: dst port 80 and len >= 100 • Custom instruction set, interpreted/JIT compiled 0: (bf) r6 = r1 1: (85) call 14 2: (67) r0 <<= 32 3: (77) r0 >>= 32 4: (15) if r0 == 0x49f goto pc+40
  8. 8. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Extended BPF (3.18 and ongoing) • Attach BPF programs to kprobes/uprobes (4.1) and tracepoints (4.7) • Data structures: array, hash (expandable), stack map (4.6) • Output to trace buffer (4.3) and perf cyclic buffer (4.4) • Helper functions: get time, get current comm, get current CPU, etc.
  9. 9. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 BCC: BPF Compiler Collection • Library and Python/Lua module for compiling, loading, and executing BPF programs • Compile BPF program from C source • Attach BPF program to kprobe/uprobe/tracepoint/USDT/socket • Poll data from BPF program using Python/Lua • Can do in-kernel aggregation and filtering • Growing collection of tracing, networking, and performance tools
  10. 10. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07
  11. 11. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 BCC • The BCC repository contains a variety of existing scripts and tools to get you started • The BPF module (Python/Lua) can be used to build new tools or one- off scripts $ ls *.py argdist.py bashreadline.py biolatency.py biosnoop.py biotop.py bitesize.py btrfsdist.py btrfsslower.py cachestat.py cpudist.py dcsnoop.py dcstat.py execsnoop.py ext4dist.py ext4slower.py filelife.py fileslower.py filetop.py funccount.py funclatency.py gethostlatency.py hardirqs.py killsnoop.py mdflush.py memleak.py offcputime.py offwaketime.py oomkill.py opensnoop.py pidpersec.py runqlat.py softirqs.py solisten.py stackcount.py stacksnoop.py statsnoop.py syncsnoop.py tcpaccept.py tcpconnect.py tcpconnlat.py tcpretrans.py tplist.py trace.py vfscount.py vfsstat.py wakeuptime.py xfsdist.py xfsslower.py zfsdist.py zfsslower.py
  12. 12. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Specialized Tools # ./hardirqs.py Tracing hard irq event time... Hit Ctrl-C to end. ^C HARDIRQ TOTAL_usecs virtio0-input.0 959 ahci[0000:00:1f.2] 1290 # ./biolatency.py Tracing block device I/O... Hit Ctrl-C to end. ^C usecs : count distribution 64 -> 127 : 7 |********* | 128 -> 255 : 14 |****************** | 256 -> 511 : 5 |****** | 512 -> 1023 : 30 |****************************************| 1024 -> 2047 : 1 |* |
  13. 13. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Specialized Tools # ./filetop.py 01:35:51 loadavg: 0.01 0.04 0.03 2/139 3611 PID COMM READS WRITES R_Kb W_Kb T FILE 2496 sshd 3 1 48 0 O ptmx 2939 screen 4 1 16 0 O ptmx 2496 sshd 1 3 16 0 S TCP 3611 clear 2 0 8 0 R screen 2939 screen 1 3 4 0 O 0 3589 filetop.py 2 0 2 0 R loadavg 3611 clear 1 0 0 0 R libtinfo.so.5.9 3611 clear 1 0 0 0 R libc-2.21.so 3611 filetop.py 3 0 0 0 R clear 3611 filetop.py 2 0 0 0 R ld-2.21.so 3611 clear 0 1 0 0 O 2 3589 filetop.py 0 3 0 0 O 2 # ./cachestat.py HITS MISSES DIRTIES READ_HIT% WRITE_HIT% BUFFERS_MB CACHED_MB 0 0 0 0.0% 0.0% 54 482 842 0 0 100.0% 0.0% 54 482 889 128 0 87.4% 12.6% 54 482
  14. 14. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Specialized Tools # ./stackcount.py __kmalloc Tracing 1 functions for "__kmalloc"... Hit Ctrl-C to end. ^C __kmalloc alloc_fdtable dup_fd copy_process.part.31 _do_fork sys_clone do_syscall_64 return_from_SYSCALL_64 4 __kmalloc create_pipe_files __do_pipe_flags sys_pipe entry_SYSCALL_64_fastpath 6 __kmalloc htree_dirblock_to_tree ext4_htree_fill_tree ext4_readdir iterate_dir SyS_getdents entry_SYSCALL_64_fastpath 14
  15. 15. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 BPF Tracing Targets (circa July 2016) Target Support Overhead kprobes Native Low uprobes Native Medium handler runs in KM Kernel tracepoints NativeNEW Low USDT tracepoints Temporary through uprobes Medium handler runs in KM
  16. 16. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Multi-Tools: argdist # ./argdist.py -C 'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1' [01:49:00] p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1 COUNT EVENT 1 count = 3134 1 count = 170 1 count = 181 2 count = 18 3 count = 30
  17. 17. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Multi-Tools: argdist # ./argdist.py -i 5 -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t:$entry(count):$latency > 1000000' [01:51:40] count : count distribution 0 -> 1 : 20 |****************************************| 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 6 |************ | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 1 |** |
  18. 18. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Multi-Tools: trace # ./trace.py 'r:/usr/bin/bash:readline "%s", retval' TIME PID COMM FUNC - 02:02:26 3711 bash readline ls –la 02:02:36 3711 bash readline wc -l src.c # ./tplist.py -v block:block_rq_complete block:block_rq_complete dev_t dev; sector_t sector; unsigned int nr_sector; int errors; char rwbs[8]; # ./trace.py 't:block:block_rq_complete "sectors=%d", tp.nr_sector' TIME PID COMM FUNC - 02:03:56 0 swapper/0 block_rq_complete sectors=16 02:03:56 0 swapper/0 block_rq_complete sectors=8 02:03:58 0 swapper/0 block_rq_complete sectors=24 02:04:00 0 swapper/0 block_rq_complete sectors=0
  19. 19. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Multi-Tools: trace # ./tplist.py -l pthread -v libpthread:pthread_create /usr/lib64/libpthread-2.21.so libpthread:pthread_create location 0x7c63 8 unsigned bytes @ register %rax 8 unsigned bytes @ -192(%rbp) 8 unsigned bytes @ -168(%rbp) 8 unsigned bytes @ -176(%rbp) # ./trace.py 'u:pthread:pthread_create "%llx", arg3' TIME PID COMM FUNC - 02:07:29 4051 contentions pthread_create 400e00 02:07:29 4051 contentions pthread_create 400e00 02:07:29 4051 contentions pthread_create 400e00 02:07:29 4051 contentions pthread_create 400e00 ^C
  20. 20. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Multi-Tools: trace # trace -p $(pidof node) 'u:node:http__server__request "%s %s (from %s:%d)" arg5, arg6, arg3, arg4' TIME PID COMM FUNC - 04:50:44 22185 node http__server__request GET /foofoo (from ::1:51056) 04:50:46 22185 node http__server__request GET / (from ::1:51056) ^C # ./trace.py 'u:/tmp/libjvm.so:thread__start "%s [%d]", arg1, arg4' 'u:/tmp/libjvm.so:thread__stop "%s [%d]", arg1, arg4' TIME PID COMM FUNC - 06:55:24 32157 java thread__start Reference Handler [32157] 06:55:24 32158 java thread__start Finalizer [32158] 06:55:24 32159 java thread__start Signal Dispatcher [32159] 06:55:24 32160 java thread__start C2 CompilerThread0 [32160] 06:55:24 32161 java thread__start C2 CompilerThread1 [32161] 06:55:24 32162 java thread__start C1 CompilerThread2 [32162] 06:55:24 32163 java thread__start Service Thread [32163] 06:55:28 32159 java thread__stop Signal Dispatcher [32159] ^C
  21. 21. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Lab #3 – Chasing a C++ Memory Leak #4 – MySQL and Disk Stats and Stacks #5 – Node and JVM USDT Probes https://s.sashag.net/sreconlabs
  22. 22. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Kernel Custom Tool Design BPF program Tracepoint kprobe Python/Lua driver App process uprobeUSDT Probe handler Probe handler Hash or histogram Cyclic buffer U K
  23. 23. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 BPF Program: Counting Allocations #include <linux/ptrace.h> struct alloc_info_t { u64 count; u64 size; }; BPF_HASH(allocs, u32, struct alloc_info_t); int handler(struct pt_regs *ctx, size_t size) { u32 pid = bpf_get_current_pid_tgid(); struct alloc_info_t init = { 0 }, *info; info = allocs.lookup_or_init(&pid, &init); info->count += 1; info->size += size; return 0; }
  24. 24. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 BPF Driver #!/usr/bin/env python from bcc import BPF from time import sleep program = BPF(src_file="allocs.c") program.attach_kprobe(event="__kmalloc", fn_name="handler") allocs = program["allocs"] while True: sleep(5) print("n%-8s %-8s %-10s" % ("PID", "COUNT", "SIZE")) for key, value in sorted(allocs.items(), key=lambda (k, v): k.value): print("%-8d %-8d %-8d" % (key.value, value.count, value.size))
  25. 25. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 BPF Execution # ./allocs.py PID COUNT SIZE 28064 3 456 28157 10 76 28158 5 1116 PID COUNT SIZE 28001 113 1828 28064 8 1216 28110 38 683 28157 46 328 28158 5 1116 28159 41 12894 ^C
  26. 26. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Inline BPF Program #!/usr/bin/env python from bcc import BPF from time import sleep program = BPF(text="""BPF_HASH(counts, u32, u32); TRACEPOINT_PROBE(irq, irq_handler_entry) { u32 zero = 0, *existing, irq = args->irq; existing = counts.lookup_or_init(&irq, &zero); ++(*existing); return 0; }""") counts = program["counts"] sleep(9999999) print("n%-8s %-8s" % ("IRQ", "COUNT")) for key, value in counts.items(): print("%-8d %-8d" % (key.value, value.value))
  27. 27. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Data Types • Array • Hash • Histogram • Perf buffer (4.4+)NEW • Stack map (4.6+)NEW
  28. 28. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Example: Histogram struct dist_key_t { char op[OP_NAME_LEN]; u64 slot; }; BPF_HISTOGRAM(dist, struct dist_key_t); ... struct dist_key_t key = { .slot=bpf_log2l(elapsed_time) }; __builtin_memcpy(&key.op, op, sizeof(key.op)); dist.increment(key); ... bpf.get_table("dist").print_log2_hist("operation")
  29. 29. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Example: Perf Buffer bpf = BPF(text="""#include <linux/ptrace.h> struct data_t { u64 pid; char str[80]; }; BPF_PERF_OUTPUT(events); int print(struct pt_regs *ctx) { struct data_t data = {0}; ... events.perf_submit(ctx, &data, sizeof(data)); return 0; }""") class Data(ct.Structure): _fields_ = [ ("pid", ct.c_ulonglong), ("str", ct.c_char*80) ] bpf.attach_uretprobe(name="/bin/bash", sym="readline", fn_name="print") b["events"].open_perf_buffer(lambda cpu, data, size: event = ct.cast(data, ct.POINTER(Data)).contents print(event) ) while True: bpf.kprobe_poll()
  30. 30. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Example: Stack Map BPF_HASH(counts, int); BPF_STACK_TRACE(stacks, 1024); ... int key = stacks.get_stackid(ctx, BPF_F_REUSE_STACKID); u64 zero = 0; u64 *val = counts.lookup_or_init(&key, &zero); ++(*val); ... counts, stacks = bpf["counts"], bpf["stacks"] for k, v in counts: for addr in stacks.walk(k.value): print(BPF.ksym(addr)) print(v.value)
  31. 31. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Custom Tool Design Tips • Try to perform all aggregations in the BPF program and keep UM copying to a minimum • Limit hash/histogram/stackmap sizes, prune, keep only top entries • Clear cyclic buffer often and quickly
  32. 32. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Deployment • For Python tools, deploy Python + libbcc.so • For Lua tools, deploy only bcc-lua • Statically links libbcc.a but allows plugging libbcc.so • Kernel build flags: • CONFIG_BPF=y • CONFIG_BPF_SYSCALL=y • CONFIG_BPF_JIT=y • CONFIG_HAVE_BPF_JIT=y • CONFIG_BPF_EVENTS=y
  33. 33. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Lab #6 – Contention Stats and Stacks #7 – From BCC GitHub Issues https://s.sashag.net/sreconlabs
  34. 34. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Summary • Tracing can identify bugs and performance issues that no debugger or profiler can catch • Tools make low-overhead, dynamic, production tracing possible • BPF is the next-generation backend for Linux tracing tools
  35. 35. SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 SRECon Europe 2016 @goldshtn https://s.sashag.net/bpfws07 Thank You! Sasha Goldshtein @goldshtn

×