The document outlines the evolution and functionality of eBPF (extended Berkeley Packet Filter) for low-overhead system tracing, covering various aspects such as its historical development, usage in containers, and tools available for implementation. It emphasizes the importance of eBPF in improving observability, networking, and security in modern systems, particularly within container environments. Additionally, it provides references and credits to notable figures and resources in the eBPF community.

1. Low-Overhead
System Tracing
With eBPF
May 2018
Akshay Kapoor
DevOps Engineer @ SAP Labs

2. Low-Overhead System Tracing With eBPF

3. Low-Overhead System Tracing With eBPF

4. Low-Overhead System Tracing With eBPF

5. You don't need to know how to operate an X-ray machine,
but you do need to know that if you swallow a penny, an X-ray is an option!
~ www.bredangregg.com

6. -----------------------------------------------------------------------
CLASSIC PKT. FILTERING
BPF VIRTUAL
MACHINE
EXTENDED BPF
BPF SYSTEM CALL
ADDTL. PROBES
1993
Before 1992 2013
2014
Today
EVOLUTION OF BPF
STACK BASED
KERNEL -> USPACE COPIES
REGISTER BASED (2)
LESSER COPIES
IMPROVED ISA & eBPF MAPS
MORE REGISTERS (10)
EXPOSED TO USER SPACE
KERNEL FILTERS
UPROBES, KPROBES
USDT, TRACEPOINTS

7. tcpdump -n "dst host 192.168.1.1 and dst port 23"

8. # Credits : https://suchakra.wordpress.com/
HOW BPF WORKS ?

9. BCC (BPF COMPILER COLLECTION)
• https://github.com/iovisor/bcc
• Lead Developer – Brenden Blanco
#Credits : Sasha Goldshtein

11. BCC Tools [examples…]

12. BCC Tools [examples…]

13. BCC Tools [examples…]

14. BCC Tools [examples…]

15. Flamegraphs
[ BCC/BPF Visualizations ]

16. Flamegraphs
[ BCC/BPF Visualizations]
(Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/)
CallStacks
No. of Samples

17. BPF and
Containers

18. • Namespaces
BPF and containers…
PID Y
PID X
CONTAINER
HOST
Restricts Visibility
• mnt
• pid
• net
• . . .

19. • Namespaces
• CGroups
BPF and containers…
CONTAINER 1
HOST
Restricts Quota/Usage
• cpu
• mem
• blkio
• . . .
CONTAINER 2
CPU SHARES

20. • Namespaces
• CGroups
• Analysis from the host
• PID Mappings (/sys/fs/cgroup/docker/*)
• Symbol file locations
BPF and containers…
0x7f82b510ddda
0x7f82b510999d
0x7f82b510f665
0x7f82b510t546

21. • Namespaces
• CGroups
• Analysis from the host
• PID Mappings (/sys/fs/cgroup*)
• Symbol file locations
• Deployment Methodologies
BPF and containers…
APP
CONTAINER 1
APP
CONTAINER 2
BCC
TOOLS ON
HOST
KERNEL EVENTS

22. • Namespaces
• CGroups
• Analysis from the host
• PID Mappings (/sys/fs/cgroup*)
• Symbol file locations
• Deployment Methodologies
BPF and containers…
APP
CONTAINER 1
APP
CONTAINER 2
KERNEL EVENTS
BCC
CONTAINER 3

23. DEMO
OBSERVABILITYNETWORKING SECURITY

24. BPF Implementations…
- Seccomp
• Control system calls made by a process
- Cilium
• Controls Networking, Security and Load Balancing for containers
- Weavescope
• Observability into containerized application stacks like Docker and Kubernetes
- Iptables
• Bpfilter implementations to optimize ingress/outgress security rules
- Systemtap
• BPF backend for optimizations

25. References
Sasha Goldshtein (goldshtn)
Brendan Gregg (brendangregg)
Suchakra (tuxology)
Julia Evans (b0rk)
@Follow
https://github.com/iovisor/bcc
http://man7.org/linux/man-pages/man2/bpf.2.html
http://brendangregg.com/ebpf.html
https://github.com/goldshtn/linux-tracing-workshop
https://suchakra.wordpress.com/ - eBPF
https://blog.yadutaf.fr/ - Networking & eBPF
https://jvns.ca/blog/2017/07/05/linux-tracing-systems/
https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF
https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/
https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/
https://lwn.net/Articles/740157/ - Thorough eBPF intro
https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/
https://lwn.net/Articles/747551/ - BPF comes to firewalls

26. Thank You !
akshay.kapoor@sap.com
akskap
akskap
akskap