The document discusses the use of eBPF (extended Berkeley Packet Filter) for distributed performance analysis, detailing its functionalities, processes, and various applications. eBPF serves as a tracing framework that enables dynamic instrumentation of the Linux kernel to monitor events and trace behavior without requiring extensive modifications to the kernel itself. It also highlights practical examples of using eBPF to trace database queries, network activity, and device I/O latency, among others.

1. © 2018 InfluxData. All rights reserved.1
Distributed Performance Analysis
Using InfluxDB and the Linux
eBPF Virtual Machine

2. eBPF (extended BPF)
Extended Berkeley Packet Filter

3. © 2018 InfluxData. All rights reserved.3
BPF is a Tracing Framework*
Used to access kernel trace backend instrumentation tools
*Actually, it’s not just that. And there’s also XDP.

4. © 2018 InfluxData. All rights reserved.4
Static tracepoints
# cat /sys/kernel/debug/tracing/available_events
sched:
task:
signal:
timer:
workqueue:
irq:
kvm:
# ls /sys/kernel/debug/tracing/events/irq/
enable filter irq_handler_entry
irq_handler_exit softirq_entry softirq_exit
softirq_raise
tcp:

5. © 2018 InfluxData. All rights reserved.5
Dynamic trace functionalities
uprobes kprobes
XDP

6. © 2018 InfluxData. All rights reserved.6
eBPF program lifecycle
Kernel space User space
User program
eBPF bytecode
Static verifier
BPF
BPF_MAP_* Results
(bpf_lookup_elem)
uprobes
kprobes
tracepoints
BPF_PROG_LOAD
see man 2 bpf
The mustache parrot warns!
eBPF programs can’t be turing complete!
XDP
socketfilter

7. In today’s world

8. © 2018 InfluxData. All rights reserved.8
In today’s world: tcpdump
Documentation about the instruction set: https://www.kernel.org/doc/Documentation/networking/filter.txt
# tcpdump -d 'ip and tcp port 80'
(000) ldh [12]
(001) jeq #0x800 jt 2 jf 12
(002) ldb [23]
(003) jeq #0x6 jt 4 jf 12
(004) ldh [20]
(005) jset #0x1fff jt 12 jf 6
(006) ldxb 4*([14]&0xf)
(007) ldh [x + 14]
(008) jeq #0x50 jt 11 jf 9
(009) ldh [x + 16]
(010) jeq #0x50 jt 11 jf 12
(011) ret #262144
(012) ret #0
-d stands for: Dump the compiled packet-matching code in a human readable form to standard output and stop.
Is src (x+14) on port 80 (0x50)?
Is src (x+16) on port 80 (0x50)?
Is it an ethernet IP IPv4 packet?

9. In today’s world: seccomp
gcc -lseccomp seccomp-test.c
./a.out
hey there!
something's gonna happen!!
[1] 19463 killed ./a.out

10. © 2018 InfluxData. All rights reserved.10
10
More practical examples?
¨ Trace file opens by filename
¨ Trace queries done against a database, like InfluxDB or MySQL
¨ Trace TCP retransmissions
¨ Trace all commands done in a bash shell
¨ Trace block device I/O latency over time
¨ JVM events
¨ Go Runtime Events
¨ Firewalls, packet rewriting, dropping etc..

11. High-level APIs are there!

12. © 2018 InfluxData. All rights reserved.12
Write every executed bash command to Influx
Kernel space User space
main.go
eBPF bytecode
Static verifier
BPF
BPF_PERF_OUTPUT Results
(bpf_open_perf_buffer)
uretprobe
get_return_value
BPF_PROG_LOAD LLVM Backend “ebpf”
Results channel
Results consumer
consumes
writes
Populates the channel
Populates the userspace map

13. © 2018 InfluxData. All rights reserved.13
As a Daemonset in a Kubernetes Cluster!
Kernel space (for privileged POD) Pod User space
main.go
eBPF bytecode
Static verifier
BPF
BPF_PERF_OUTPUT Results
(bpf_open_perf_buffer)
uretprobe
get_return_value
BPF_PROG_LOAD LLVM Backend “ebpf”
Results channel
Results consumer
consumes
writes
Populates the channel
Populates the userspace map

14. © 2018 InfluxData. All rights reserved.14

15. © 2018 InfluxData. All rights reserved.15

16. © 2018 InfluxData. All rights reserved.16
References
1. https://www.iovisor.org/
2. https://github.com/cilium/cilium
3. https://github.com/iovisor/gobpf
4. https://landlock.io/
5. https://github.com/iovisor/bpftrace
6. https://github.com/iovisor/bpf-docs
7. https://medium.com/@fntlnz/load-xdp-programs-using-the-ip-iproute2-command-502043898263
8. https://www.youtube.com/watch?v=JRFNIKUROPE
9. https://cilium.readthedocs.io/en/latest/bpf/
1. iovisor BCC
2. Cilium: HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP
3. iovisor/gobpf - To load eBPF programs using Go
4. Landlock LSM
5. iovisor bpftrace
6. iovisor BPF docs
7. Blog post on how to load xdp programs using iproute2
8. BPF Tracing Talk from Brendan Gregg
9. Cilium documentation for BPF

17. Thank you!lorenzo@influxdata.com
twitter.com/fntlnz
github.com/fntlnz
If you’re that kind of person can find my pgp key here:
https://fntlnz.wtf/downloads/pubkey-B2400EE4.asc