Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Low Overhead System Tracing with eBPF

45 views

Published on

In the Container world, there is a need to build observability around apps and backing services running in containers. The observability should allow to capture on demand low level metrics at a low overhead. The proposal is to use ebpf as the tracing technology to capture details at kernel & user level, and generate on demand flamegraphs, heat maps for applications & backing services. The Linux kernel has a built-in BPF JIT compiler, and an in-kernel verifier which is used to validate eBPF programs. This allows user defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively. eBPF provides in-kernel implementation of storage maps such as histograms and hash-maps, which helps in efficient copy of summarized monitoring data from kernel to user space with low overhead.

These features make eBPF programs safe to run in production and allow admins and engineers to collect crucial data from systems for performance analysis and monitoring.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Low Overhead System Tracing with eBPF

  1. 1. Low-Overhead System Tracing With eBPF May 2018 Akshay Kapoor DevOps Engineer @ SAP Labs
  2. 2. Low-Overhead System Tracing With eBPF
  3. 3. Low-Overhead System Tracing With eBPF
  4. 4. Low-Overhead System Tracing With eBPF
  5. 5. You don't need to know how to operate an X-ray machine, but you do need to know that if you swallow a penny, an X-ray is an option! ~ www.bredangregg.com
  6. 6. ----------------------------------------------------------------------- CLASSIC PKT. FILTERING BPF VIRTUAL MACHINE EXTENDED BPF BPF SYSTEM CALL ADDTL. PROBES 1993 Before 1992 2013 2014 Today EVOLUTION OF BPF STACK BASED KERNEL -> USPACE COPIES REGISTER BASED (2) LESSER COPIES IMPROVED ISA & eBPF MAPS MORE REGISTERS (10) EXPOSED TO USER SPACE KERNEL FILTERS UPROBES, KPROBES USDT, TRACEPOINTS
  7. 7. tcpdump -n "dst host 192.168.1.1 and dst port 23"
  8. 8. # Credits : https://suchakra.wordpress.com/ HOW BPF WORKS ?
  9. 9. BCC (BPF COMPILER COLLECTION) • https://github.com/iovisor/bcc • Lead Developer – Brenden Blanco #Credits : Sasha Goldshtein
  10. 10. BCC Tools [examples…]
  11. 11. BCC Tools [examples…]
  12. 12. BCC Tools [examples…]
  13. 13. BCC Tools [examples…]
  14. 14. Flamegraphs [ BCC/BPF Visualizations ]
  15. 15. Flamegraphs [ BCC/BPF Visualizations] (Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/) CallStacks No. of Samples
  16. 16. BPF and Containers
  17. 17. • Namespaces BPF and containers… PID Y PID X CONTAINER HOST Restricts Visibility • mnt • pid • net • . . .
  18. 18. • Namespaces • CGroups BPF and containers… CONTAINER 1 HOST Restricts Quota/Usage • cpu • mem • blkio • . . . CONTAINER 2 CPU SHARES
  19. 19. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup/docker/*) • Symbol file locations BPF and containers… 0x7f82b510ddda 0x7f82b510999d 0x7f82b510f665 0x7f82b510t546
  20. 20. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations • Deployment Methodologies BPF and containers… APP CONTAINER 1 APP CONTAINER 2 BCC TOOLS ON HOST KERNEL EVENTS
  21. 21. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations • Deployment Methodologies BPF and containers… APP CONTAINER 1 APP CONTAINER 2 KERNEL EVENTS BCC CONTAINER 3
  22. 22. DEMO OBSERVABILITYNETWORKING SECURITY
  23. 23. BPF Implementations… - Seccomp • Control system calls made by a process - Cilium • Controls Networking, Security and Load Balancing for containers - Weavescope • Observability into containerized application stacks like Docker and Kubernetes - Iptables • Bpfilter implementations to optimize ingress/outgress security rules - Systemtap • BPF backend for optimizations
  24. 24. References Sasha Goldshtein (goldshtn) Brendan Gregg (brendangregg) Suchakra (tuxology) Julia Evans (b0rk) @Follow https://github.com/iovisor/bcc http://man7.org/linux/man-pages/man2/bpf.2.html http://brendangregg.com/ebpf.html https://github.com/goldshtn/linux-tracing-workshop https://suchakra.wordpress.com/ - eBPF https://blog.yadutaf.fr/ - Networking & eBPF https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/ https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/ https://lwn.net/Articles/740157/ - Thorough eBPF intro https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/ https://lwn.net/Articles/747551/ - BPF comes to firewalls
  25. 25. Thank You ! akshay.kapoor@sap.com akskap akskap akskap

×