eBPF (extended Berkeley Packet Filter) is a powerful and versatile technology that can be used to extend observability in Linux systems. In this talk, we will explore how eBPF can be used to bridge the gap between dev and ops by providing a deeper understanding of the kernel and OS internals as well as the applications running on top. We will discuss how eBPF can be used to extend observability downwards by enabling access to low-level system information and how it can be used to extend observability upwards by providing application-level tracing capabilities.
3. Who am I
Raphaël Pinson
Solutions Architect @ Isovalent | CNCF Ambassador
@raphink | @raphink@mastodon.social
4. ⬢ From Dumb to Expertise-Driven
Observability
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
5. ⬢ From Dumb to Expertise-Driven
Observability
⬢ eBPF
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
6. ⬢ From Dumb to Expertise-Driven
Observability
⬢ eBPF
⬢ Observing Downwards & Upwards
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
7. ⬢ From Dumb to Expertise-Driven
Observability
⬢ eBPF
⬢ Observing Downwards & Upwards
⬢ The Bridge
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
8. ⬢ From Dumb to Expertise-Driven
Observability
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
Image:
Unsplash
/
Wolfgang
Hasselmann
14. ⬢ From Dumb to Expertise-Driven
Observability
⬢ eBPF
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
15. Have you used eBPF?
eBPF is already used in many places
- Load balancing
- DDOS protection on large Internet platforms
- Kernel live-patching (5.7+ with LSM/eBPF)
- Android (e.g. app data stats)
@raphink | @raphink@mastodon.social
16. Makes the Linux kernel
programmable in a
secure and efficient way.
“What JavaScript is to the
browser, eBPF is to the
Linux Kernel”
@raphink | @raphink@mastodon.social
19. ⬢ From Dumb to Expertise-Driven
Observability
⬢ eBPF
⬢ Observing Downwards & Upwards
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
20. Cilium & Friends
- performance gains
(no need for iptables, bypass TCP/IP)
- simpler architecture
(e.g. no sidecar proxy for Service Mesh)
Cilium
@raphink | @raphink@mastodon.social
21. Cilium & Friends
Hubble
- fine-grained network observability
- exports to SIEM
- support for OpenTelemetry
- performance gains
(no need for iptables, bypass TCP/IP)
- simpler architecture
(e.g. no sidecar proxy for Service Mesh)
Cilium
@raphink | @raphink@mastodon.social
22. Cilium & Friends
Tetragon
- observe & export kernel events
- act on events (e.g. SIGKILL)
- performance gains
(no need for iptables, bypass TCP/IP)
- simpler architecture
(e.g. no sidecar proxy for Service Mesh)
Cilium
Hubble
- fine-grained network observability
- exports to SIEM
- support for OpenTelemetry
@raphink | @raphink@mastodon.social
23. Deep Down in Kernel Space
Observe directly in the kernel
- Low-overhead tracing/observability
- Example: network performance / SRTT / micro-bursts
- HTTP / TLS in-kernel visibility
- Troubleshooting prod on the fly (see bpftrace)
@raphink | @raphink@mastodon.social
24. Deep Down in Kernel Space
Example software
- BCC
- bpftrace
- Pixie
- Cilium (network)
- Cilium Tetragon (system)
@raphink | @raphink@mastodon.social
Observe directly in the kernel
- Low-overhead tracing/observability
- Example: network performance / SRTT / micro-bursts
- HTTP / TLS in-kernel visibility
- Troubleshooting prod on the fly (see bpftrace)
37. ⬢ From Dumb to Expertise-Driven
Observability
⬢ eBPF
⬢ Observing Downwards & Upwards
⬢ The Bridge
Bridging Dev and Ops with eBPF
Extending Observability Upwards and Downwards
38. To Infinity…
… and beyond 🚀
- more integration (Grafana, etc.)
- more links between sources (metrics, logs, traces)
- APM
@raphink | @raphink@mastodon.social