Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 26

Low Overhead System Tracing with eBPF

May. 24, 2018
• 2 likes • 767 views

2

Share

Download to read offline

Software

In the Container world, there is a need to build observability around apps and backing services running in containers. The observability should allow to capture on demand low level metrics at a low overhead. The proposal is to use ebpf as the tracing technology to capture details at kernel & user level, and generate on demand flamegraphs, heat maps for applications & backing services. The Linux kernel has a built-in BPF JIT compiler, and an in-kernel verifier which is used to validate eBPF programs. This allows user defined instrumentation on a live kernel image that can never crash, hang or interfere with the kernel negatively. eBPF provides in-kernel implementation of storage maps such as histograms and hash-maps, which helps in efficient copy of summarized monitoring data from kernel to user space with low overhead.

These features make eBPF programs safe to run in production and allow admins and engineers to collect crucial data from systems for performance analysis and monitoring.

Recommended

Related Books

Free with a 30 day trial from Scribd

See all
The Impulse Economy: Understanding Mobile Shoppers and What Makes Them Buy Gary Schwartz
(4.5/5)
Free
Emergence: The Connected Lives of Ants, Brains, Cities, and Software Steven Johnson
(4.5/5)
Free
Tubes: A Journey to the Center of the Internet Andrew Blum
(4/5)
Free
World Wide Mind: The Coming Integration of Humanity, Machines, and the Internet Michael Chorost
(4/5)
Free
An Army of Davids: How Markets and Technology Empower Ordinary People to Beat Big Media, Big Government, and Other Goliaths Glenn Reynolds
(4/5)
Free
In the Plex: How Google Thinks, Works, and Shapes Our Lives Steven Levy
(4.5/5)
Free
Hamlet's BlackBerry: A Practical Philosophy for Building a Good Life in the Digital Age William Powers
(4/5)
Free
The Thank You Economy Gary Vaynerchuk
(4/5)
Free
Talking Back to Facebook: The Common Sense Guide to Raising Kids in the Digital Age James P. Steyer
(4.5/5)
Free
Public Parts: How Sharing in the Digital Age Improves the Way We Work and Live Jeff Jarvis
(3.5/5)
Free
The Nature of the Future: Dispatches from the Socialstructed World Marina Gorbis
(4/5)
Free
Socialnomics: How Social Media Transforms the Way We Live and Do Business Erik Qualman
(3/5)
Free
The End of Business As Usual: Rewire the Way You Work to Succeed in the Consumer Revolution Brian Solis
(5/5)
Free
Blog Schmog: The Truth About What Blogs Can (and Can't) Do for Your Business Robert W. Bly
(4/5)
Free
From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network, and the Rise of Digital Utopianism Fred Turner
(2.5/5)
Free
How to Be Invisible: Protect Your Home, Your Children, Your Assets, and Your Life J. J. Luna
(4/5)
Free

Related Audiobooks

Free with a 30 day trial from Scribd

See all
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Seth Stephens-Davidowitz
(4.5/5)
Free
The New New Thing: A Silicon Valley Story Michael Lewis
(4.5/5)
Free
Cognitive Surplus: Creativity and Generosity in a Connected Age Clay Shirky
(3.5/5)
Free
So You Want to Start a Podcast: Finding Your Voice, Telling Your Story, and Building a Community that Will Listen Kristen Meinzer
(4.5/5)
Free
The Dark Net: Inside the Digital Underworld Jamie Bartlett
(3.5/5)
Free
Kill All Normies: Online Culture Wars From 4Chan And Tumblr To Trump And The Alt-Right Angela Nagle
(4/5)
Free
The Art of Social Media: Power Tips for Power Users Guy Kawasaki
(4/5)
Free
Who Owns the Future? Jaron Lanier
(4/5)
Free
An Introduction to Information Theory: Symbols, Signals and Noise John R. Pierce
(4.5/5)
Free
Algorithms to Live By: The Computer Science of Human Decisions Tom Griffiths
(4.5/5)
Free
The Death of Expertise: The Campaign Against Established Knowledge and Why it Matters Tom Nichols
(4.5/5)
Free
Alone Together: Why We Expect More from Technology and Less from Each Other Sherry Turkle
(4/5)
Free
Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, and the World Don Tapscott
(4/5)
Free
The Emperor's New Mind: Concerning Computers, Minds, and the Laws of Physics Roger Penrose
(3.5/5)
Free
New Dark Age: Technology and the End of the Future James Bridle
(4.5/5)
Free
The History of the Future: Oculus, Facebook, and the Revolution That Swept Virtual Reality Blake J. Harris
(4.5/5)
Free

Low Overhead System Tracing with eBPF

  1. 1. Low-Overhead System Tracing With eBPF May 2018 Akshay Kapoor DevOps Engineer @ SAP Labs
  2. 2. Low-Overhead System Tracing With eBPF
  3. 3. Low-Overhead System Tracing With eBPF
  4. 4. Low-Overhead System Tracing With eBPF
  5. 5. You don't need to know how to operate an X-ray machine, but you do need to know that if you swallow a penny, an X-ray is an option! ~ www.bredangregg.com
  6. 6. ----------------------------------------------------------------------- CLASSIC PKT. FILTERING BPF VIRTUAL MACHINE EXTENDED BPF BPF SYSTEM CALL ADDTL. PROBES 1993 Before 1992 2013 2014 Today EVOLUTION OF BPF STACK BASED KERNEL -> USPACE COPIES REGISTER BASED (2) LESSER COPIES IMPROVED ISA & eBPF MAPS MORE REGISTERS (10) EXPOSED TO USER SPACE KERNEL FILTERS UPROBES, KPROBES USDT, TRACEPOINTS
  7. 7. tcpdump -n "dst host 192.168.1.1 and dst port 23"
  8. 8. # Credits : https://suchakra.wordpress.com/ HOW BPF WORKS ?
  9. 9. BCC (BPF COMPILER COLLECTION) • https://github.com/iovisor/bcc • Lead Developer – Brenden Blanco #Credits : Sasha Goldshtein
  10. 10. BCC Tools [examples…]
  11. 11. BCC Tools [examples…]
  12. 12. BCC Tools [examples…]
  13. 13. BCC Tools [examples…]
  14. 14. Flamegraphs [ BCC/BPF Visualizations ]
  15. 15. Flamegraphs [ BCC/BPF Visualizations] (Source : https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/) CallStacks No. of Samples
  16. 16. BPF and Containers
  17. 17. • Namespaces BPF and containers… PID Y PID X CONTAINER HOST Restricts Visibility • mnt • pid • net • . . .
  18. 18. • Namespaces • CGroups BPF and containers… CONTAINER 1 HOST Restricts Quota/Usage • cpu • mem • blkio • . . . CONTAINER 2 CPU SHARES
  19. 19. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup/docker/*) • Symbol file locations BPF and containers… 0x7f82b510ddda 0x7f82b510999d 0x7f82b510f665 0x7f82b510t546
  20. 20. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations • Deployment Methodologies BPF and containers… APP CONTAINER 1 APP CONTAINER 2 BCC TOOLS ON HOST KERNEL EVENTS
  21. 21. • Namespaces • CGroups • Analysis from the host • PID Mappings (/sys/fs/cgroup*) • Symbol file locations • Deployment Methodologies BPF and containers… APP CONTAINER 1 APP CONTAINER 2 KERNEL EVENTS BCC CONTAINER 3
  22. 22. DEMO OBSERVABILITYNETWORKING SECURITY
  23. 23. BPF Implementations… - Seccomp • Control system calls made by a process - Cilium • Controls Networking, Security and Load Balancing for containers - Weavescope • Observability into containerized application stacks like Docker and Kubernetes - Iptables • Bpfilter implementations to optimize ingress/outgress security rules - Systemtap • BPF backend for optimizations
  24. 24. References Sasha Goldshtein (goldshtn) Brendan Gregg (brendangregg) Suchakra (tuxology) Julia Evans (b0rk) @Follow https://github.com/iovisor/bcc http://man7.org/linux/man-pages/man2/bpf.2.html http://brendangregg.com/ebpf.html https://github.com/goldshtn/linux-tracing-workshop https://suchakra.wordpress.com/ - eBPF https://blog.yadutaf.fr/ - Networking & eBPF https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ https://www.youtube.com/watch?v=aaTQM7wcmfk – Kernel Meetup | eBPF https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/ https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/ https://lwn.net/Articles/740157/ - Thorough eBPF intro https://developers.redhat.com/blog/2017/12/13/introducing-stapbpf-systemtaps-new-bpf-backend/ https://lwn.net/Articles/747551/ - BPF comes to firewalls
  25. 25. Thank You ! akshay.kapoor@sap.com akskap akskap akskap

×