Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Meet-cute between eBPF and Kernel Tracing Viller Hsiao <villerhsiao@gmail.com> Jul. 5, 2016
03/09/2016 2 Who am I ? Viller Hsiao Embedded Linux / RTOS engineer    http://image.dfdaily.com/201 2/5/4/6347169311287512...
03/09/2016 3 BPF Berkeley Packet Filter by Steven McCanne and Van Jacobson, 1993
03/09/2016 4 Who am I ? Viller Hsiao Embedded Linux / RTOS engineer    http://image.dfdaily.com/201 2/5/4/6347169311287512...
03/09/2016 5 Berkeley Packet Filter Packet filter: tcpdump -nnnX port 3000
03/09/2016 6 network stack sniffer kernel user net if Applications tcpdump ­nnnX  port 3000 port 3000 VM filter http://www...
03/09/2016 7 Berkeley Packet Filter Improve unix packet filter
03/09/2016 8 Berkeley Packet Filter Improve unix packet filter Replace stack-based VM with register-based VM
03/09/2016 9 Berkeley Packet Filter Improve unix packet filter Replace stack-based VM with register-based VM 20 times fast...
03/09/2016 10 In­Kernel VM for Filtering Flexibility Efficiency Security
03/09/2016 11 BPF in Linux a.k.a. Linux Socket Filter kernel 2.1.75, in 1997
03/09/2016 12 Areas Use BPF in Linux Nowadays ● Linux­3.4 (2012), Seccomp filters of syscalls (chrome sandboxing) ● Packet...
03/09/2016 13 Story today, When kernel tracing meets ebpf http://2.blog.xuite.net/2/4/7/8/11001626/blog_70864/txt/17378250...
03/09/2016 14 Examples of BPF Program   ldh [12]   jne #0x806, drop   ret #­1   drop: ret #0 ARP packets ICMP random packe...
03/09/2016 15 BPF Example: Translate to Binary $ ./bpf_asm ­c foo  Opcode   JT   JF          K { 0x28,       0,    0,   0x...
03/09/2016 16 Userspace Application struct sock_filter code[] = { { 0x28,  0,  0, 0x0000000c }, { 0x15,  0,  8, 0x000086dd...
03/09/2016 17 BPF JIT Compiler in 2011 ● Linux­3.0, by Eric Dumazet ● Architecture support – x86_64, SPARC, PowerPC, ARM, ...
03/09/2016 18 extended BPF Linux-3.15 by Alexei Starovoitov, 2013
03/09/2016 19 Classic BPF vs Internal BPF (a.k.a extended BPF)
03/09/2016 20 eBPF Design Goals ● Just­in­time map to modern 64­bit CPU with minimal  performance overhead ● Write program...
03/09/2016 21 cBPF vs eBPF BPF eBPF registers A, X R0 ­ R10 width 32 bit  64 bit opcode op:16, jt:8, jf:8, k:32 op:8, dst_...
03/09/2016 22 BPF Calling Convention ● R0 ● Return value from in­kernel function, and exit value for eBPF  program ● R1 – ...
03/09/2016 23 Designed to be JITed for 64­bit Architecture  /* restore ctx for next call */     bpf_mov R6, R1x     bpf_mo...
03/09/2016 24 How does it work?
03/09/2016 25 BPF Internals (1) subsys BPF binary kernel user     app BPF VM
03/09/2016 26 BPF  Internals (2) BPF binary subsys BPF binary kernel user Interpreter JIT bpf syscall BPF_PROG_LOAD     app
03/09/2016 27 BPF  Internals (3) BPF binary subsys BPF binary kernel user Interpreter JIT bpf syscall verifier     app
03/09/2016 28 BPF Verifier ● Do static check in verifier as possible ● Directed Acyclic Graph(DAG) program – Max 4096 inst...
03/09/2016 29 BPF  Internals (4) BPF binary MAP subsys BPF binary kernel user Interpreter JIT bpf syscall verifier BPF_MAP...
03/09/2016 30 BPF MAP ● BPF_MAP_TYPE_HASH ● BPF_MAP_TYPE_ARRAY ● BPF_MAP_TYPE_PROG_ARRAY ● BPF_MAP_TYPE_PERF_EVENT_ARRAY m...
03/09/2016 31 BPF  Internals (5) BPF binary MAP subsys BPF binary kernel user Interpreter JIT bpf syscall verifier BPF_PRO...
03/09/2016 32 BPF  Internals  (6) BPF binary MAP helper subsys Other subsys BPF_PROG_RUN BPF binary kernel user Interprete...
03/09/2016 33 BPF Helpers map netsystem perf trace ● bpf_func_id
03/09/2016 34 BPF  Internals (7) BPF binary MAP helper subsys Other subsys BPF_PROG_RUN BPF binary kernel user Interpreter...
03/09/2016 35 Kernel Instrumentation
03/09/2016 36 Dynamic Probe Kernel user Kprobe Kretprobe Jprobe Uprobe
03/09/2016 37 Kprobe INST BREAK register_kprobe() pre_handler() post_handler() address sym + offset Write kernel module to...
03/09/2016 38 Kprobe BREAKBREAK INST pre_handler() post_handler() exception address Note: More details are not revealed
03/09/2016 39 Kprobe­based Event Tracing # echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/tracing/kprobe_events # ...
03/09/2016 40 Uprobe  echo 'p:myapp /bin/bash:0x4245c0' > /sys/kernel/tracing/uprobe_events ● Linux­3.5 ● userspace breakp...
03/09/2016 41 User Tools for Kprobe ● tracefs files ● systemtap
03/09/2016 42 ftrace ● Linux­2.6.27 ● Linux kernel internal tracer
03/09/2016 43 ftrace Interface tracefs (debugfs in past)  README available_events available_filter_functions available_tra...
03/09/2016 44 ftrace Function Tracer   void Func ( … )   {       Line 1;       Line 2;       …   }      void Func ( … )   ...
03/09/2016 45 Dynamic Function Tracer Function trace enabled on Func()      void Func ( … )   {       nop;       Line 1;  ...
03/09/2016 46 Tracepoint      #include <trace/events/subsys.h>        DEFINE_TRACE(subsys_eventname);        void somefct(...
03/09/2016 47 perf Statistics data $ perf stat my­app args Sampling record $ perf record my­app args perf­tool perf framew...
03/09/2016 48 Summary of Kernel Tracing http://www.slideshare.net/brendangregg/linux-systems-performance-2016
03/09/2016 49 https://i.ytimg.com/vi/elc3FdKxaOk/maxresdefault.jpg Before BPF Integration Complex filters and scripts can ...
03/09/2016 50 People desire more powerful tool  like dtrace Some attemptation: systemtap, ktap
03/09/2016 51 Linux­4.1 “One of the more interesting features in this cycle is the  ability to attach eBPF programs (user­...
03/09/2016 52 Instrument powered by eBPF “If DTrace is Kixy Hawk, eBPF is a jet engine” ~ Brendan Gregg http://www.ait.org...
03/09/2016 53 Attach to Kprobe as well as tracepoint By Alexei Starovoitov – tracing: attach BPF programs to kprobes – tra...
03/09/2016 54 BPF for Tracing ● The output data is not limited to PMU counters but data like  time latencies, cache misses...
03/09/2016 55 Ftrace Filter Interpreter on eBPF (not merged yet?) "field1 == 1 || field2 == 2"
03/09/2016 56 The Evolution of eBPF Userspace Utilities  http://www.bitrebels.com/wp-content/uploads/2011/04/Evolution-Of-...
03/09/2016 57 Program on eBPF Restrict C BPF Binary  LLVM ( up 3.7) userspace program eBPF assembly or Kernel
03/09/2016 58 Write a eBPF Program in C Looks Good. But, What's the rule of “restrict C” ?
03/09/2016 59 Restrict C [9] ● No support for  – Global variables  – Arbitrary function calls,  – Floating point, varargs,...
03/09/2016 60 BPF Utilities 1: Kernel Samples foo_user.c     +      foo_kern.c All prog/data needed when loading bpf ● bpf...
03/09/2016 61 foo_kern.c struct bpf_map_def SEC("maps") my_map = { .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY, .max_entries = 3...
03/09/2016 62 foo_user.c   Take kprobe as example map 1 map 2 bpf_prog1 bpf_prog2 bpf_prog3 version sec(“maps”) sec(“kprob...
03/09/2016 63 BPF Utilities 2: BCC in IOVisor The project enables developers to build, innovate, and  share open, programm...
03/09/2016 64 BPF Compiler Collection Frontend python, lua llvm library BPF bytecode libbcc.so BPF C text/code BCC module ...
03/09/2016 65 BPF_HASH(start, struct request *); void trace_start(struct pt_regs *ctx, struct request *req) {             ...
03/09/2016 66 BCC Example: Python Frontend from bcc import BPF b = BPF (src_file="disksnoop.c") b.attach_kprobe (event="bl...
03/09/2016 67 Current Tracing Scripts in BCC https://raw.githubusercontent.com/iovisor/bcc/master/images/bcc_tracing_tools...
03/09/2016 68 BPF Utilities 3: perf tools $ perf bpf record --object sample_bpf.o -- -a sleep 4 ● Introduced by Wang Nan
03/09/2016 69 Summary ● eBPF: In­kernel VM designed to be JITed ● Used by many subsystems as a filtering engine – Packet m...
03/09/2016 70 Other Topics: How to use in embedded system?
03/09/2016 71 Other Topics: Linux­4.7: hist trigger Another mechanism other than eBPF http://www.brendangregg.com/blog/201...
03/09/2016 72 Q & A
9/3/16 73/75 Reference [1] Alexei Starovoitov (May. 2014), “tracing: accelerate tracing filters with BPF”, KERNEL PATCH [2...
9/3/16 74/75 ● HCSM is the community of Hsinchu Coders in Taiwan. ● iovisor is a project of Linux Foundation ● ARM are tra...
9/3/16 Viller Hsiao THE END
Upcoming SlideShare
Loading in …5
×

Meet cute-between-ebpf-and-tracing

3,823 views

Published on

Introduction to Linux kernel tracing and ebpf integration

Published in: Software
License: CC Attribution-ShareAlike License
no profile picture user
Show More

Meet cute-between-ebpf-and-tracing

  1. 1. Meet-cute between eBPF and Kernel Tracing Viller Hsiao <villerhsiao@gmail.com> Jul. 5, 2016
  2. 2. 03/09/2016 2 Who am I ? Viller Hsiao Embedded Linux / RTOS engineer    http://image.dfdaily.com/201 2/5/4/634716931128751250504b 050c1_nEO_IMG.jpg
  3. 3. 03/09/2016 3 BPF Berkeley Packet Filter by Steven McCanne and Van Jacobson, 1993
  4. 4. 03/09/2016 4 Who am I ? Viller Hsiao Embedded Linux / RTOS engineer    http://image.dfdaily.com/201 2/5/4/634716931128751250504b 050c1_nEO_IMG.jpg
  5. 5. 03/09/2016 5 Berkeley Packet Filter Packet filter: tcpdump -nnnX port 3000
  6. 6. 03/09/2016 6 network stack sniffer kernel user net if Applications tcpdump ­nnnX  port 3000 port 3000 VM filter http://www.ic onsdb.com/ico ns/download/g ray/empty-fil ter-512.png In­kernel Packet Filter
  7. 7. 03/09/2016 7 Berkeley Packet Filter Improve unix packet filter
  8. 8. 03/09/2016 8 Berkeley Packet Filter Improve unix packet filter Replace stack-based VM with register-based VM
  9. 9. 03/09/2016 9 Berkeley Packet Filter Improve unix packet filter Replace stack-based VM with register-based VM 20 times faster than original design
  10. 10. 03/09/2016 10 In­Kernel VM for Filtering Flexibility Efficiency Security
  11. 11. 03/09/2016 11 BPF in Linux a.k.a. Linux Socket Filter kernel 2.1.75, in 1997
  12. 12. 03/09/2016 12 Areas Use BPF in Linux Nowadays ● Linux­3.4 (2012), Seccomp filters of syscalls (chrome sandboxing) ● Packet classifier for traffic contol  ● Actions for traffic control ● Xtables packet filtering ● Tracing
  13. 13. 03/09/2016 13 Story today, When kernel tracing meets ebpf http://2.blog.xuite.net/2/4/7/8/11001626/blog_70864/txt/17378250/0.jpg
  14. 14. 03/09/2016 14 Examples of BPF Program   ldh [12]   jne #0x806, drop   ret #­1   drop: ret #0 ARP packets ICMP random packet sampling 1 in 4   ldh [12]   jne #0x800, drop   ldb [23]   jneq #1, drop   ld rand                   mod #4   jneq #1, drop   ret #­1   drop: ret #0 helper extensions
  15. 15. 03/09/2016 15 BPF Example: Translate to Binary $ ./bpf_asm ­c foo  Opcode   JT   JF          K { 0x28,       0,    0,   0x0000000c }, { 0x15,       0,    1,   0x00000806 }, { 0x06,       0,    0,   0xffffffff }, { 0x06,       0,    0,   0000000000 },
  16. 16. 03/09/2016 16 Userspace Application struct sock_filter code[] = { { 0x28,  0,  0, 0x0000000c }, { 0x15,  0,  8, 0x000086dd },        … }; struct sock_fprog bpf = { .len = ARRAY_SIZE(code), .filter = code, }; sock = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL)); if (sock < 0) /* ... bail out ... */ ret = setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &bpf, sizeof(bpf)); if (ret < 0) /* ... bail out ... */ BPF Binary
  17. 17. 03/09/2016 17 BPF JIT Compiler in 2011 ● Linux­3.0, by Eric Dumazet ● Architecture support – x86_64, SPARC, PowerPC, ARM, ARM64, MIPS and  s390   $ echo 1 > /proc/sys/net/core/bpf_jit_enable
  18. 18. 03/09/2016 18 extended BPF Linux-3.15 by Alexei Starovoitov, 2013
  19. 19. 03/09/2016 19 Classic BPF vs Internal BPF (a.k.a extended BPF)
  20. 20. 03/09/2016 20 eBPF Design Goals ● Just­in­time map to modern 64­bit CPU with minimal  performance overhead ● Write programs in restricted C and compile into BPF with  GCC/LLVM ● Guarantee termination and safety of BPF program in kernel  with simple algorithm
  21. 21. 03/09/2016 21 cBPF vs eBPF BPF eBPF registers A, X R0 ­ R10 width 32 bit  64 bit opcode op:16, jt:8, jf:8, k:32 op:8, dst_reg:4, src_reg:4, off:16, imm:32 JIT support x86_64, SPARC,  PowerPC, ARM,  ARM64, MIPS and  s390 x86­64, aarch64, s390x
  22. 22. 03/09/2016 22 BPF Calling Convention ● R0 ● Return value from in­kernel function, and exit value for eBPF  program ● R1 – R5 ● Arguments from eBPF program to in­kernel function ● R6 – R9 ● Callee saved registers that in­kernel function will preserve ● R10 ● Read­only frame pointer to access stack
  23. 23. 03/09/2016 23 Designed to be JITed for 64­bit Architecture  /* restore ctx for next call */     bpf_mov R6, R1x     bpf_mov R2, 2     bpf_mov R3, 3     bpf_mov R4, 4     bpf_mov R5, 5     bpf_call foo  /* save foo() return value */     bpf_mov R7, R0  /* restore ctx for next call */     bpf_mov R1, R6     bpf_mov R2, 6     bpf_mov R3, 7     bpf_mov R4, 8     bpf_mov R5, 9     bpf_call bar     bpf_add R0, R7     bpf_exit     push %rbp     mov %rsp,%rbp     sub $0x228,%rsp     mov %rbx,­0x228(%rbp)     mov %r13,­0x220(%rbp)     mov %rdi,%rbx     mov $0x2,%esi     mov $0x3,%edx     mov $0x4,%ecx     mov $0x5,%r8d     callq foo     mov %rax,%r13     mov %rbx,%rdi     mov $0x2,%esi     mov $0x3,%edx     mov $0x4,%ecx     mov $0x5,%r8d     callq bar     add %r13,%rax     mov ­0x228(%rbp),%rbx     mov ­0x220(%rbp),%r13     leaveq     retq x86_64
  24. 24. 03/09/2016 24 How does it work?
  25. 25. 03/09/2016 25 BPF Internals (1) subsys BPF binary kernel user     app BPF VM
  26. 26. 03/09/2016 26 BPF  Internals (2) BPF binary subsys BPF binary kernel user Interpreter JIT bpf syscall BPF_PROG_LOAD     app
  27. 27. 03/09/2016 27 BPF  Internals (3) BPF binary subsys BPF binary kernel user Interpreter JIT bpf syscall verifier     app
  28. 28. 03/09/2016 28 BPF Verifier ● Do static check in verifier as possible ● Directed Acyclic Graph(DAG) program – Max 4096 instructions – No loop – unreachable insns exist ● Instruction walk – Read a never­written register – Do arithmetic of two valid pointer – Load/store registers of invalid types – Read stack before writing data into
  29. 29. 03/09/2016 29 BPF  Internals (4) BPF binary MAP subsys BPF binary kernel user Interpreter JIT bpf syscall verifier BPF_MAP_CREATE BPF_MAP_LOOKUP_ELEM BPF_MAP_UPDATE_ELEM ….     app
  30. 30. 03/09/2016 30 BPF MAP ● BPF_MAP_TYPE_HASH ● BPF_MAP_TYPE_ARRAY ● BPF_MAP_TYPE_PROG_ARRAY ● BPF_MAP_TYPE_PERF_EVENT_ARRAY map1 map2 map3 Tracing prog_1 sock prog_3 Tracing prog_2 sk_buff on eth0 Tracepoint Event C Tracepoint Event B Tracepoint Event A
  31. 31. 03/09/2016 31 BPF  Internals (5) BPF binary MAP subsys BPF binary kernel user Interpreter JIT bpf syscall verifier BPF_PROG_RUN     app
  32. 32. 03/09/2016 32 BPF  Internals  (6) BPF binary MAP helper subsys Other subsys BPF_PROG_RUN BPF binary kernel user Interpreter/ JIT bpf syscall verifier     app
  33. 33. 03/09/2016 33 BPF Helpers map netsystem perf trace ● bpf_func_id
  34. 34. 03/09/2016 34 BPF  Internals (7) BPF binary MAP helper subsys Other subsys BPF_PROG_RUN BPF binary kernel user Interpreter/JIT bpf syscall verifier     app
  35. 35. 03/09/2016 35 Kernel Instrumentation
  36. 36. 03/09/2016 36 Dynamic Probe Kernel user Kprobe Kretprobe Jprobe Uprobe
  37. 37. 03/09/2016 37 Kprobe INST BREAK register_kprobe() pre_handler() post_handler() address sym + offset Write kernel module to register a kprobe
  38. 38. 03/09/2016 38 Kprobe BREAKBREAK INST pre_handler() post_handler() exception address Note: More details are not revealed
  39. 39. 03/09/2016 39 Kprobe­based Event Tracing # echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/tracing/kprobe_events # echo 1 > /sys/kernel/tracing/events/kprobes/myretprobe/enable # cat /sys/kernel/tracing/trace # tracer: nop # #           TASK­PID   CPU#  ||||    TIMESTAMP  FUNCTION #              | |       |   ||||       |         |               sh­746   [000] d...   40.96: myretprobe: (SyS_open+0x2c/0x30 <­ do_sys_open) arg1=0x3               sh­746   [000] d...   42.19: myretprobe: (SyS_open+0x2c/0x30 <­ do_sys_open) arg1=0x3 …..
  40. 40. 03/09/2016 40 Uprobe  echo 'p:myapp /bin/bash:0x4245c0' > /sys/kernel/tracing/uprobe_events ● Linux­3.5 ● userspace breakpoints in kernel
  41. 41. 03/09/2016 41 User Tools for Kprobe ● tracefs files ● systemtap
  42. 42. 03/09/2016 42 ftrace ● Linux­2.6.27 ● Linux kernel internal tracer
  43. 43. 03/09/2016 43 ftrace Interface tracefs (debugfs in past)  README available_events available_filter_functions available_tracers buffer_size_kb buffer_total_size_kb current_tracer dyn_ftrace_total_info enabled_functions events free_buffer instances kprobe_events kprobe_profile max_graph_depth options per_cpu printk_formats saved_cmdlines saved_cmdlines_size set_event set_event_pid set_ftrace_filter set_ftrace_notrace set_ftrace_pid set_graph_function set_graph_notrace trace trace_clock trace_marker trace_options trace_pipe tracing_cpumask tracing_on tracing_thresh $ ls /sys/kernel/tracing
  44. 44. 03/09/2016 44 ftrace Function Tracer   void Func ( … )   {       Line 1;       Line 2;       …   }      void Func ( … )   {       mcount (pc, ra);       Line 1;       Line 2;       …   } gcc ­pg
  45. 45. 03/09/2016 45 Dynamic Function Tracer Function trace enabled on Func()      void Func ( … )   {       nop;       Line 1;       Line 2;       …   }      void Func ( … )   {       mcount (pc, ra);       Line 1;       Line 2;       …   } Function trace disabled on Func()
  46. 46. 03/09/2016 46 Tracepoint      #include <trace/events/subsys.h>        DEFINE_TRACE(subsys_eventname);        void somefct(void)      {          ...          trace_subsys_eventname(arg, task);          ...      }     DECLARE_TRACE( subsys_eventname,                                     TP_PROTO(int firstarg, struct task_struct *p),                                     TP_ARGS(firstarg, p)); include/trace/events/subsys.h subsys/file.c
  47. 47. 03/09/2016 47 perf Statistics data $ perf stat my­app args Sampling record $ perf record my­app args perf­tool perf framework kernel user HW event perf_event SW event PMU trace event trace point dynamic event kprobe uprobe
  48. 48. 03/09/2016 48 Summary of Kernel Tracing http://www.slideshare.net/brendangregg/linux-systems-performance-2016
  49. 49. 03/09/2016 49 https://i.ytimg.com/vi/elc3FdKxaOk/maxresdefault.jpg Before BPF Integration Complex filters and scripts can be expensive Components are isolated
  50. 50. 03/09/2016 50 People desire more powerful tool  like dtrace Some attemptation: systemtap, ktap
  51. 51. 03/09/2016 51 Linux­4.1 “One of the more interesting features in this cycle is the  ability to attach eBPF programs (user­defined, sandboxed  bytecode executed by the kernel) to kprobes. This allows  user­defined instrumentation on a live kernel image that  can never crash, hang or interfere with the kernel  negatively. “ ~Ingo Molnár  https://lkml.org/lkml/2015/4/14/232
  52. 52. 03/09/2016 52 Instrument powered by eBPF “If DTrace is Kixy Hawk, eBPF is a jet engine” ~ Brendan Gregg http://www.ait.org.tw/infousa/zhtw/american_story/assets/es/nc/es_nc_kttyhwk_1_e.jpg
  53. 53. 03/09/2016 53 Attach to Kprobe as well as tracepoint By Alexei Starovoitov – tracing: attach BPF programs to kprobes – tracing: allow BPF programs to call bpf_ktime_get_ns() – tracing: allow BPF programs to call bpf_trace_printk() prog_fd = bpf_prog_load(...); struct perf_event_attr attr = { .type = PERF_TYPE_TRACEPOINT, .config = event_id, /* ID of just created kprobe event */ }; event_fd = perf_event_open(&attr,...); ioctl(event_fd, PERF_EVENT_IOC_SET_BPF, prog_fd);
  54. 54. 03/09/2016 54 BPF for Tracing ● The output data is not limited to PMU counters but data like  time latencies, cache misses or other things users want to  record. http://www.slideshare.net/brendangregg/linux-bpf-superpowers
  55. 55. 03/09/2016 55 Ftrace Filter Interpreter on eBPF (not merged yet?) "field1 == 1 || field2 == 2"
  56. 56. 03/09/2016 56 The Evolution of eBPF Userspace Utilities  http://www.bitrebels.com/wp-content/uploads/2011/04/Evolution-Of-Man-Parodies-333.jpg
  57. 57. 03/09/2016 57 Program on eBPF Restrict C BPF Binary  LLVM ( up 3.7) userspace program eBPF assembly or Kernel
  58. 58. 03/09/2016 58 Write a eBPF Program in C Looks Good. But, What's the rule of “restrict C” ?
  59. 59. 03/09/2016 59 Restrict C [9] ● No support for  – Global variables  – Arbitrary function calls,  – Floating point, varargs, exceptions, indirect jumps, arbitrary  pointer arithmetic, alloca, etc.   ● Kernel rejects all programs that it cannot prove safe – programs with loops  – with memory accesses via arbitrary pointers. 
  60. 60. 03/09/2016 60 BPF Utilities 1: Kernel Samples foo_user.c     +      foo_kern.c All prog/data needed when loading bpf ● bpf programs ● map ● license ● … etc   Userspace ● Load BPF ● Cretae maps ● Flow control ● Data presentaion
  61. 61. 03/09/2016 61 foo_kern.c struct bpf_map_def SEC("maps") my_map = { .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY, .max_entries = 32, …. }; SEC("kprobe/sys_write") int bpf_prog1(struct pt_regs *ctx) { u64 count; u32 key = bpf_get_smp_processor_id(); char fmt[] = "CPU­%d   %llun"; count = bpf_perf_event_read(&my_map, key); bpf_trace_printk(fmt, sizeof(fmt), key, count); return 0; } u32 _version SEC("version") = LINUX_VERSION_CODE; BPF programs MAPs Others
  62. 62. 03/09/2016 62 foo_user.c   Take kprobe as example map 1 map 2 bpf_prog1 bpf_prog2 bpf_prog3 version sec(“maps”) sec(“kprobe/prog1”) sec(“kprobe/prog2”) sec(“kprobe/prog3”) sec(“version”) foo_kern.c foo_kern.o (elf) clang ­­target=bpf Create map (maps section) Load bpf_progx (kprobe/xxx, license,  … sections) Setup  /sys/.../krpobe_events (kprobe/xxx sections) libbpf foo_user.c bpf_prog_load
  63. 63. 03/09/2016 63 BPF Utilities 2: BCC in IOVisor The project enables developers to build, innovate, and  share open, programmable data plane with dynamic IO and  networking functions https://www.iovisor.org/sites /cpstandard/files/pages/image s/io_visor.jpg
  64. 64. 03/09/2016 64 BPF Compiler Collection Frontend python, lua llvm library BPF bytecode libbcc.so BPF C text/code BCC module BCC bpf syscallperf event / trace_fs User program
  65. 65. 03/09/2016 65 BPF_HASH(start, struct request *); void trace_start(struct pt_regs *ctx, struct request *req) {                   …... } void trace_completion(struct pt_regs *ctx, struct request *req) { u64 *tsp, delta; tsp = start.lookup(&req); if (tsp != 0) { delta = bpf_ktime_get_ns() ­ *tsp; bpf_trace_printk("%d %x %dn", req­>__data_len,     req­>cmd_flags, delta / 1000); start.delete(&req); } } BCC Example: BPF c Program Simpler than kernel samples
  66. 66. 03/09/2016 66 BCC Example: Python Frontend from bcc import BPF b = BPF (src_file="disksnoop.c") b.attach_kprobe (event="blk_start_request", fn_name="trace_start") b.attach_kprobe (event="blk_mq_start_request", fn_name="trace_start") b.attach_kprobe (event="blk_account_io_completion",                                              fn_name="trace_completion")                     ….... while 1: (task, pid, cpu, flags, ts, msg) = b.trace_fields()                     ….... print("%­18.9f %­2s %­7s %8.2f" % (ts, type_s, bytes_s, ms))
  67. 67. 03/09/2016 67 Current Tracing Scripts in BCC https://raw.githubusercontent.com/iovisor/bcc/master/images/bcc_tracing_tools_2016.png Tools for BPF­based Linux IO analysis, networking, monitoring, and  more
  68. 68. 03/09/2016 68 BPF Utilities 3: perf tools $ perf bpf record --object sample_bpf.o -- -a sleep 4 ● Introduced by Wang Nan
  69. 69. 03/09/2016 69 Summary ● eBPF: In­kernel VM designed to be JITed ● Used by many subsystems as a filtering engine – Packet monitor filtering – Tracing and perf – Seccomp – Networking ● Tools – BCC  ● Easy to customized script for probe kernel ● Kernel >=4.1, LLVM >= 3.7 – perf
  70. 70. 03/09/2016 70 Other Topics: How to use in embedded system?
  71. 71. 03/09/2016 71 Other Topics: Linux­4.7: hist trigger Another mechanism other than eBPF http://www.brendangregg.com/blog/2016­06­08/linux­hist­triggers.html
  72. 72. 03/09/2016 72 Q & A
  73. 73. 9/3/16 73/75 Reference [1] Alexei Starovoitov (May. 2014), “tracing: accelerate tracing filters with BPF”, KERNEL PATCH [2] Alexei Starovoitov, (Feb. 2015), "BPF – in-kernel virtual machine", presented at Collaboration Summit 2015 [3] Brendan Gregg, (Feb. 2016), "Linux 4.x Performance Using BPF Superpowers ", presented at Performance@ scale 2016 [4] Elena Zannoni (Jun. 2015), “New (and Exciting!) Developments in Linux Tracing ”, presented at Linuxcon Japan 2015 [5] Gary Lin (Mar. 2016), “eBPF: Trace from Kernel to Userspace ”, presented at OpenSUSE Technology Sharing Day 2016 [6] Jonathan Corbet. (May. 2014), “BPF: the universal in-kernel virtual machine ”, LWN [7] Kernel documentation, “Using the Linux Kernel Tracepoints” [8] Suchakrapani D. Sharma (Dec. 2014), “Towards Faster Trace Filtersvusing eBPF and JIT ” [9] Michael Larabel, (Jan. 2015), “ BPF Backend Merged Into LLVM To Make Use Of New Kernel Functionality ”, Phoronix
  74. 74. 9/3/16 74/75 ● HCSM is the community of Hsinchu Coders in Taiwan. ● iovisor is a project of Linux Foundation ● ARM are trademarks or registered trademarks of ARM Holdings. ● Linux Foundation is a registered trademark of The Linux Foundation. ● Linux is a registered trademark of Linus Torvalds. ● Other company, product, and service names may be trademarks or service marks of others. ● The license of each graph belongs to each website listed individually. ● The others of my work in the slide is licensed under a CC-BY-SA License. ● License text: http://creativecommons.org/licenses/by-sa/4.0/legalcode Rights to Copy copyright © 2016 Viller Hsiao
  75. 75. 9/3/16 Viller Hsiao THE END

×