This slide is an introduction to eBPF. This is a slide of a talk I gave on Pycon 19 Israel

1. Root Cause Analysis with
eBPF & Python
Pavel Rogovoy
PyCon IL 2019

4. eBPF

5. Waiting on a lock
20 seconds sleep

6. Pavel Rogovoy
Performance & Cloud Engineer @

7. LogicInput Output

9. Our App

11. Our App
extended Berkley
Packet Filter

12. Our App

13. A real life example

15. from bcc import BPF
sensor=''int kprobe__sys_clone(void *ctx) {
bpf_trace_printk("Hello, World!n");
return 0;
}'
BPF(text=sensor).trace_print()

18. Applications
System Libraries
System Call Interface
Sockets
TCP/UDP
IP
Ethernet
VFS
File Systems
Volume Manager
Block Device
Interface
Device Drivers
Scheduler
Virtual
Memory
syscount

19. summarize aggregated latencies of syscalls
./syscount.py -L -p `pgrep -nx my_process`
Tracing syscalls, printing top 10... Ctrl+C to quit.
[09:41:32]
SYSCALL COUNT TIME (us)
nanosleep 9 7020894.226
futex 6 220.009
write 4 90.292
clone 2 70.935
Detaching...

20. top 3 failed syscalls
/usr/share/bcc/tools/syscount -x -T 3
Tracing failed syscalls, printing top 3... Ctrl+C to quit.
[13:27:46]
SYSCALL COUNT
recvmsg 6152
futex 346
read 32
^C
Detaching...

21. Applications
System Libraries
System Call Interface
Sockets
TCP/UDP
IP
Ethernet
VFS
File Systems
Volume Manager
Block Device
Interface
Device Drivers
Scheduler
Virtual
Memory
offcputime

22. What is off-cpu time?

24. offcputime

25. finish_task_switch
Previous task Next task
Stack
Trace TSTS

26. @
CPYTHON 3.6+
USDT: User Statically-Defined Tracing

27. System Libraries
System Call Interface
Sockets
TCP/UDP
IP
Ethernet
VFS
File Systems
Volume Manager
Block Device
Interface
Device Drivers
Scheduler
Virtual
Memory
Trace
PYTHON
CPYTHON 3.6+

28. Listing tracepoints of CPython interpreter
tplist.py -l ./python
./python python:line
./python python:function__entry
./python python:function__return
./python python:import__find__load__start
./python python:import__find__load__done
./python python:gc__start
./python python:gc__done

29. import time
def my_func():
print('Hello')
while True:
time.sleep(3)
my_func()

31. Things You Must Be Aware
Of
Not everything is perfect...

32. ● Maximum of 4096 ebpf assembly
instructions
● No loops allowed
● Poor ecosystem

33. Brendan Gregg
Performance Engineer
@ Netflix
Big contributor to
BCC, eBPF &
FlameGraphs projects

34. QUESTIONS?

35. https://www.final.co.il

36. THANKS
Email: pavelr@final.co.il
LinkedIn: progovoy
Twitter: @p_rogovoy