Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

presentation

261 views

Published on

  • Be the first to comment

presentation

  1. 1. 1Benchmarking and comparison of kernel and userspace tracers Benchmarking and comparison of kernel and userspace tracers 5 May, 2016 École Polytechnique de Montreal Mohamad Gebai Michel Dagenais
  2. 2. 2Benchmarking and comparison of kernel and userspace tracers Content The tracers The setup The benchmarks Results Kernel space System calls Filtering Userspace
  3. 3. 3Benchmarking and comparison of kernel and userspace tracers The tracers Kernel tracers LTTng Ftrace Perf eBPF (kind of) Userspace tracers LTTng Printf() LTTng using tracef() Extrae Lightweight homemade basic tracer
  4. 4. 4Benchmarking and comparison of kernel and userspace tracers The setup Intel i7 @ 3.40 GHz 16 GB of RAM Linux kernel version 4.5.0 LTTng suite built from tip of the branch stabe­2.7 Deactivated: Hyperthreading C­states CPU idle Intel Turbo Boost LTTng Snapshot mode Subbuffer size 32 KB Ftrace Global clock
  5. 5. 5Benchmarking and comparison of kernel and userspace tracers The benchmarks Objective: measure the cost of writing a tracepoint Kernel tracers: Microbenchmark in kernel space Payload: 32 bytes Userspace tracers: Microbenchmark in userspace Payload: 32 bytes Steady state (drop initial page faults, cache misses, initialization, etc.) Do a full buffer write and start overwriting
  6. 6. 6Benchmarking and comparison of kernel and userspace tracers Kernel tracing: the outline Final numbers: Average 90th percentile Value (ns) Overhead Value (ns) Overhead None 17 0% 17 0% LTTng 92 441% 89 424% Ftrace 118 594% 114 571% Perf 120 606% 118 594% LTTng-kprobe 122 618% 120 606% eBPF 131 671% 126 641% None LTTng Ftrace Perf LTTng-kprobe eBPF 0 20 40 60 80 100 120 140 Average latency of a tracepoint Latencyinns
  7. 7. 7Benchmarking and comparison of kernel and userspace tracers Kernel tracing: the outline Final numbers: None LTTng Ftrace Perf 0 50 100 150 200 250 300 Average of tracepoint latency in steady vs transient states Transient Steady Kernel tracers Latencyinns
  8. 8. 8Benchmarking and comparison of kernel and userspace tracers Kernel tracing: LTTng vs Ftrace
  9. 9. 9Benchmarking and comparison of kernel and userspace tracers Kernel tracing: closer look
  10. 10. 10Benchmarking and comparison of kernel and userspace tracers Kernel tracing: closer look
  11. 11. 11Benchmarking and comparison of kernel and userspace tracers Kernel tracing: ioctl() syscall Final numbers: Average 90th percentile Value (ns) Overhead Value (ns) Overhead None 99 0% 98 0% Ftrace 311 214% 302 208% LTTng 341 244% 333 240% Perf 600 506% 586 498% None Ftrace LTTng Perf 0 100 200 300 400 500 600 700 Average latency of a syscall with tracing Kernel tracers Latencyinns
  12. 12. 12Benchmarking and comparison of kernel and userspace tracers Kernel tracing: open() syscall 0 10 20 30 40 50 60 0 100 200 300 400 500 600 700 800 900 1000 Latency of tracing syscall open() according to filename length LTTng Ftrace Filename length in bytes Latencyinns
  13. 13. 13Benchmarking and comparison of kernel and userspace tracers Kernel tracing: filtering with LTTng 0 10 20 30 40 50 60 0 50 100 150 200 250 300 350 400 450 Average latency of a tracepoint with filtering failed conditions average latency on success no filter Filter bytecode size (number of conditions) Latencyinns
  14. 14. 14Benchmarking and comparison of kernel and userspace tracers Userspace tracing: the outline Final numbers, but... LTTng is signal­safe reentrant LTTng is scalable Average 90th percentile Value (ns) Overhead Value (ns) Overhead None 18 0% 17 0% LW-ust 67 272% 64 276% Extrae 159 783% 118 594% Printf 245 1261% 243 1329% LTTng 255 1317% 249 1365% LTTng tracef() 536 2878% 523 2976% None LW-ust Extrae Printf LTTng LTTng tracef() 0 100 200 300 400 500 600 Average latency of a tracepoint Latencyinns
  15. 15. 15Benchmarking and comparison of kernel and userspace tracers Userspace tracing: closer look
  16. 16. 16Benchmarking and comparison of kernel and userspace tracers Userspace tracing: scalability
  17. 17. 17Benchmarking and comparison of kernel and userspace tracers Related work More scalability runs Closer look at aggregators (eBPF, Systemtap)
  18. 18. 18Benchmarking and comparison of kernel and userspace tracers Acknowledgements Professor Michel Dagenais Julien Desfossez Mathieu Desnoyers Ericsson

×