DevOpsDays Zurich 2023 — Bridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards
•
4 likes•63 views
eBPF (extended Berkeley Packet Filter) is a powerful and versatile technology that can be used to extend observability in Linux systems. In this talk, we will explore how eBPF can be used to bridge the gap between dev and ops by providing a deeper understanding of the kernel and OS internals as well as the applications running on top. We will discuss how eBPF can be used to extend observability downwards by enabling access to low-level system information and how it can be used to extend observability upwards by providing application-level tracing capabilities.
Recommended
More Related Content
Similar to DevOpsDays Zurich 2023 — Bridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards
Similar to DevOpsDays Zurich 2023 — Bridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards (20)
More from Raphaël PINSON
More from Raphaël PINSON (20)
Recently uploaded
Recently uploaded (20)
DevOpsDays Zurich 2023 — Bridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards
- 1. Bridging Dev and Ops with eBPF Raphaël Pinson | @raphink | @raphink@mastodon.social Extending Observability Upwards and Downwards Solutions Architect, Isovalent
- 2. Dumb Monitoring Observability over the Wall @raphink | @raphink@mastodon.social
- 3. Who am I Raphaël Pinson Solutions Architect @ Isovalent | CNCF Ambassador @raphink | @raphink@mastodon.social
- 4. ⬢ From Dumb to Expertise-Driven Observability Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 5. ⬢ From Dumb to Expertise-Driven Observability ⬢ eBPF Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 6. ⬢ From Dumb to Expertise-Driven Observability ⬢ eBPF ⬢ Observing Downwards & Upwards Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 7. ⬢ From Dumb to Expertise-Driven Observability ⬢ eBPF ⬢ Observing Downwards & Upwards ⬢ The Bridge Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 8. ⬢ From Dumb to Expertise-Driven Observability Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards Image: Unsplash / Wolfgang Hasselmann
- 9. Privilege-Driven Monitoring With Great Power comes great Responsibility @raphink | @raphink@mastodon.social
- 10. Expertise-Driven Responsibility The Ownership Principle @raphink | @raphink@mastodon.social
- 11. Platforms, Interfaces & Responsibilities (But DevOps is not dead) @raphink | @raphink@mastodon.social
- 12. Observability Ownership Developers ❤ Observability @raphink | @raphink@mastodon.social
- 13. Data Collection Services & Application Instrumentation @raphink | @raphink@mastodon.social
- 14. ⬢ From Dumb to Expertise-Driven Observability ⬢ eBPF Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 15. Have you used eBPF? eBPF is already used in many places - Load balancing - DDOS protection on large Internet platforms - Kernel live-patching (5.7+ with LSM/eBPF) - Android (e.g. app data stats) @raphink | @raphink@mastodon.social
- 16. Makes the Linux kernel programmable in a secure and efficient way. “What JavaScript is to the browser, eBPF is to the Linux Kernel” @raphink | @raphink@mastodon.social
- 18. @raphink | @raphink@mastodon.social Cloud Native Identities
- 19. ⬢ From Dumb to Expertise-Driven Observability ⬢ eBPF ⬢ Observing Downwards & Upwards Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 20. Cilium & Friends - performance gains (no need for iptables, bypass TCP/IP) - simpler architecture (e.g. no sidecar proxy for Service Mesh) Cilium @raphink | @raphink@mastodon.social
- 21. Cilium & Friends Hubble - fine-grained network observability - exports to SIEM - support for OpenTelemetry - performance gains (no need for iptables, bypass TCP/IP) - simpler architecture (e.g. no sidecar proxy for Service Mesh) Cilium @raphink | @raphink@mastodon.social
- 22. Cilium & Friends Tetragon - observe & export kernel events - act on events (e.g. SIGKILL) - performance gains (no need for iptables, bypass TCP/IP) - simpler architecture (e.g. no sidecar proxy for Service Mesh) Cilium Hubble - fine-grained network observability - exports to SIEM - support for OpenTelemetry @raphink | @raphink@mastodon.social
- 23. Deep Down in Kernel Space Observe directly in the kernel - Low-overhead tracing/observability - Example: network performance / SRTT / micro-bursts - HTTP / TLS in-kernel visibility - Troubleshooting prod on the fly (see bpftrace) @raphink | @raphink@mastodon.social
- 24. Deep Down in Kernel Space Example software - BCC - bpftrace - Pixie - Cilium (network) - Cilium Tetragon (system) @raphink | @raphink@mastodon.social Observe directly in the kernel - Low-overhead tracing/observability - Example: network performance / SRTT / micro-bursts - HTTP / TLS in-kernel visibility - Troubleshooting prod on the fly (see bpftrace)
- 25. Observability: bpftrace @raphink | @raphink@mastodon.social
- 26. Networking: Hubble (CLI) @raphink | @raphink@mastodon.social $ kubectl get pods NAME READY STATUS RESTARTS AGE tiefighter 1/1 Running 0 2m34s xwing 1/1 Running 0 2m34s deathstar-5b7489bc84-crlxh 1/1 Running 0 2m34s deathstar-5b7489bc84-j7qwq 1/1 Running 0 2m34s $ hubble observe --follow -l class=xwing # DNS lookup to coredns default/xwing:41391 (ID:16092) -> kube-system/coredns-66bff467f8-28dgp:53 (ID:453) to-proxy FORWARDED (UDP) kube-system/coredns-66bff467f8-28dgp:53 (ID:453) -> default/xwing:41391 (ID:16092) to-endpoint FORWARDED (UDP) # ... # Successful HTTPS request to www.disney.com default/xwing:37836 (ID:16092) -> www.disney.com:443 (world) to-stack FORWARDED (TCP Flags: SYN) www.disney.com:443 (world) -> default/xwing:37836 (ID:16092) to-endpoint FORWARDED (TCP Flags: SYN, ACK) www.disney.com:443 (world) -> default/xwing:37836 (ID:16092) to-endpoint FORWARDED (TCP Flags: ACK, FIN) default/xwing:37836 (ID:16092) -> www.disney.com:443 (world) to-stack FORWARDED (TCP Flags: RST) # ... # Blocked HTTP request to deathstar backend default/xwing:49610 (ID:16092) -> default/deathstar:80 (ID:16081) Policy denied DROPPED (TCP Flags: SYN)
- 27. Networking: Hubble (UI) @raphink | @raphink@mastodon.social
- 28. Observability: Cilium + Grafana ❤ @raphink | @raphink@mastodon.social
- 29. Network Metrics (Hubble) @raphink | @raphink@mastodon.social
- 30. HTTP Metrics (Hubble) @raphink | @raphink@mastodon.social
- 31. OpenTelemetry (Hubble OTEL) @raphink | @raphink@mastodon.social
- 32. Network Policy Verdicts @raphink | @raphink@mastodon.social
- 33. TLS (Tetragon) @raphink | @raphink@mastodon.social
- 34. Combined Network & Runtime @raphink | @raphink@mastodon.social
- 35. Continuous Profiling (Parca) @raphink | @raphink@mastodon.social https://www.parca.dev Continuous profiling of Cilium: https://www.youtube.com/watch?v=tScFlySdSAc
- 36. APM (groundcover) @raphink | @raphink@mastodon.social https://groundcover.com
- 37. ⬢ From Dumb to Expertise-Driven Observability ⬢ eBPF ⬢ Observing Downwards & Upwards ⬢ The Bridge Bridging Dev and Ops with eBPF Extending Observability Upwards and Downwards
- 38. To Infinity… … and beyond 🚀 - more integration (Grafana, etc.) - more links between sources (metrics, logs, traces) - APM @raphink | @raphink@mastodon.social
- 39. eBPF resources eCHO eBPF YouTube podcast: https://www.youtube.com/channel/UCJFUxkVQTBJh3LD1wYB WvuQ eBPF & Cilium Slack http://slack.cilium.io/ eCHO News Bi-weekly eBPF newsletter: https://cilium.io/newsletter/ @raphink | @raphink@mastodon.social
- 40. Practical Labs … to become a Cilium & eBPF Jedi @raphink | @raphink@mastodon.social isovalent.com/labs
- 41. Practical Labs … to become a Cilium & eBPF Jedi @raphink | @raphink@mastodon.social isovalent.com/labs
- 42. Thank you!