1.
Low-Overhead
System Tracing
With eBPF
May 2018
Akshay Kapoor
DevOps Engineer @ SAP Labs

2.
Low-Overhead System Tracing With eBPF

3.
Low-Overhead System Tracing With eBPF

4.
Low-Overhead System Tracing With eBPF

5.
You don't need to know how to operate an X-ray machine,
but you do need to know that if you swallow a penny, an X-ray is an option!
~ www.bredangregg.com

6.
-----------------------------------------------------------------------
CLASSIC PKT. FILTERING
BPF VIRTUAL
MACHINE
EXTENDED BPF
BPF SYSTEM CALL
ADDTL. PROBES
1993
Before 1992 2013
2014
Today
EVOLUTION OF BPF
STACK BASED
KERNEL -> USPACE COPIES
REGISTER BASED (2)
LESSER COPIES
IMPROVED ISA & eBPF MAPS
MORE REGISTERS (10)
EXPOSED TO USER SPACE
KERNEL FILTERS
UPROBES, KPROBES
USDT, TRACEPOINTS

7.
tcpdump -n "dst host 192.168.1.1 and dst port 23"

8.
# Credits : https://suchakra.wordpress.com/
HOW BPF WORKS ?

9.
BCC (BPF COMPILER COLLECTION)
• https://github.com/iovisor/bcc
• Lead Developer – Brenden Blanco
#Credits : Sasha Goldshtein

10.
BCC Tools [examples…]

11.
BCC Tools [examples…]

12.
BCC Tools [examples…]

13.
BCC Tools [examples…]

14.
Flamegraphs
[ BCC/BPF Visualizations ]

15.
Flamegraphs
[ BCC/BPF Visualizations]
(Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/)
CallStacks
No. of Samples

16.
BPF and
Containers

17.
• Namespaces
BPF and containers…
PID Y
PID X
CONTAINER
HOST
Restricts Visibility
• mnt
• pid
• net
• . . .

18.
• Namespaces
• CGroups
BPF and containers…
CONTAINER 1
HOST
Restricts Quota/Usage
• cpu
• mem
• blkio
• . . .
CONTAINER 2
CPU SHARES

19.
• Namespaces
• CGroups
• Analysis from the host
• PID Mappings (/sys/fs/cgroup/docker/*)
• Symbol file locations
BPF and containers…
0x7f82b510ddda
0x7f82b510999d
0x7f82b510f665
0x7f82b510t546

20.
• Namespaces
• CGroups
• Analysis from the host
• PID Mappings (/sys/fs/cgroup*)
• Symbol file locations
• Deployment Methodologies
BPF and containers…
APP
CONTAINER 1
APP
CONTAINER 2
BCC
TOOLS ON
HOST
KERNEL EVENTS

21.
• Namespaces
• CGroups
• Analysis from the host
• PID Mappings (/sys/fs/cgroup*)
• Symbol file locations
• Deployment Methodologies
BPF and containers…
APP
CONTAINER 1
APP
CONTAINER 2
KERNEL EVENTS
BCC
CONTAINER 3

22.
DEMO
OBSERVABILITYNETWORKING SECURITY

23.
BPF Implementations…
- Seccomp
• Control system calls made by a process
- Cilium
• Controls Networking, Security and Load Balancing for containers
- Weavescope
• Observability into containerized application stacks like Docker and Kubernetes
- Iptables
• Bpfilter implementations to optimize ingress/outgress security rules
- Systemtap
• BPF backend for optimizations

24.
References
Sasha Goldshtein (goldshtn)
Brendan Gregg (brendangregg)
Suchakra (tuxology)
Julia Evans (b0rk)
@Follow
https://github.com/iovisor/bcc
http://man7.org/linux/man-pages/man2/bpf.2.html
http://brendangregg.com/ebpf.html
https://github.com/goldshtn/linux-tracing-workshop
https://suchakra.wordpress.com/ - eBPF
https://blog.yadutaf.fr/ - Networking & eBPF
https://jvns.ca/blog/2017/07/05/linux-tracing-systems/
https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF
https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/
https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/
https://lwn.net/Articles/740157/ - Thorough eBPF intro
https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/
https://lwn.net/Articles/747551/ - BPF comes to firewalls

25.
Thank You !
akshay.kapoor@sap.com
akskap
akskap
akskap