The Linux kernel is undergoing the most fundamental architecture evolution in history and is becoming a microkernel. Why is the Linux kernel evolving into a microkernel? The potentially biggest fundamental change ever happening to the Linux kernel. This talk covers how companies like Facebook and Google use BPF to patch 0-day exploits, how BPF will change the way features are added to the kernel forever, and how BPF is introducing a new type of application deployment method for the Linux kernel.

1. Rethinking the
Linux kernel
Thomas Graf
Cilium Project, Co-Founder & CTO, Isovalent

2. 2Cameron Askin: Cameron’s World
Remember
GeoCities?

3. 3
Markup Only (HTML)
What enabled this evolution?
Programmable Platform

4. Programmability Essentials
4
Untrusted code runs
in the browser of the
user.
→ Sandboxing
Allow evolution of
logic without requiring
to constantly ship new
browser versions.
→ Deploy anytime with
seamless upgrades
Programmability must
be provided with
minimal overhead.
→ Native Execution
(JIT compiler)
Safety
Continuous
Delivery
Performance

5. Kernel Architecture
5
TCP/IPVFS
Linux
Kernel
Network DeviceBlock Device
AdminProcess Process
Network
Hardware
Storage
Hardware
Configuration
(sysfs,netlink,procfs,...)
Sockets
recvmsg()sendmsg()
Syscall
read()
File Descriptor
write()
Syscall
User
Space
HW

6. Cons:
● You likely need to ship a different
module for each kernel version
● Might crash your kernel
● Change kernel source code
● Expose configuration API
● Wait 5 years for your users
to upgrade
6
Kernel Development 101
● Write kernel module
● Every kernel release will break it
Cons:
Option 1
Native Support
Option 2
Kernel Module

7. How about we add
JavaScript-like capabilities
to the Linux Kernel?
7

8. 8

9. 9
Process
Scheduler
execve()
Linux
Kernel
Syscall

10. eBPF Runtime
10
Controller
Sockets
bpf()
Linux
Kernel
TCP/IP
Network Device
recvmsg()sendmsg()
Process
Syscall
Verifier
JIT Compiler
BPF
Program
BPF
Program
BPF
Program
approved
x86_64
Syscall
Safety & Security
The verifier will reject any
unsafe program and
provides a sandbox.
Continuous Delivery
Programs can be exchanged
without disrupting workloads.
Performance
The JIT compiler ensures
native execution
performance.
bytecode

11. eBPF Hooks
11
Process
Storage
Hardware
Sockets
TCP/IP
Network Device
read()
File Descriptor
VFS
Block Device
write()
Linux
Kernel
Network
Hardware
Process
SyscallSyscall
Where can you hook? kernel functions (kprobes), userspace functions (uprobes), system calls,
fentry/fexit, tracepoints, network devices (tc/xdp), network routes, TCP congestion algorithms,
sockets (data level)
recvmsg()sendmsg()

12. eBPF Maps
12
Controller
Sockets
Linux
Kernel
TCP/IP
Network Device
Process
Syscall Syscall
Admin
BPF
Map
Syscall
Map Types:
- Hash tables, Arrays
- LRU (Least Recently Used)
- Ring Buffer
- Stack Trace
- LPM (Longest Prefix match)
What are Maps used for?
● Program state
● Program configuration
● Share data between programs
● Share state, metrics, and
statistics with user space
recvmsg()sendmsg()

13. eBPF Helpers
13
Sockets
Linux
Kernel
TCP/IP
Network Device
Process
Syscall
What helpers exist?
● Random numbers
● Get current time
● Map access
● Get process/cgroup context
● Manipulate network packets and
forwarding
● Access socket data
● Perform tail call
● Access process stack
● Access syscall arguments
● ...
[...]
num = bpf_get_prandom_u32();
[...]
recvmsg()sendmsg()

14. eBPF Tail and Function Calls
14
Linux
Kernel
What are Tail Calls used for?
● Chain programs together
● Split programs into independent
logical components
● Make BPF programs composable
What are Functions Calls used for?
● Reuse functionality inside of a
program
● Reduce program size (avoid
inlining)

15. 15
Community
287 contributors:
(Jan 2016 to Jan 2020)
● 466 Daniel Borkmann (Cilium; maintainer)
● 290 Andrii Nakryiko (Facebook)
● 279 Alexei Starovoitov (Facebook; maintainer)
● 217 Jakub Kicinski (Facebook)
● 173 Yonghong Song (Facebook)
● 168 Martin KaFai Lau (Facebook)
● 159 Stanislav Fomichev (Google)
● 148 Quentin Monnet (Cilium)
● 148 John Fastabend (Cilium)
● 118 Jesper Dangaard Brouer (Red Hat)
● [...]

16. 16
eBPF Projects
Katran
High-performance L4
Loadbalancer
facebookincubator/katran
Android & Security
kernel runtime security
instrumentation (KRSI),
Android BPF loader,
eBPF traffic monitor
bcc, bpftrace
Performance
troubleshooting &
profiling
iovisor/bcc
Traffic Optimization
DDoS mitigation, QoS,
traffic optimization,
load balancer
cloudflare/bpftools
Falco
Container runtime
security, behavior
analysis
falcosecurity/falco
Cilium
Networking, security and
load-balancing for k8s
cilium/cilium
et al.

17. Tracing & Profiling with
17
Sockets
Linux
Kernel
TCP/IP
Process
Syscall
Verifier
JIT Compiler
Syscall
BPF
Program
Python
BCC
BPF
Maps
BCC:
github.com/iovisor/bcc
recvmsg()sendmsg()
# tcptop
Tracing... Output every 1 secs. Hit Ctrl-C to end
<screen clears>
19:46:24 loadavg: 1.86 2.67 2.91 3/362 16681
PID COMM LADDR RADDR RX_KB TX_KB
16648 16648 100.66.3.172:22 100.127.69.165:6684 1 0
16647 sshd 100.66.3.172:22 100.127.69.165:6684 0 2149
14374 sshd 100.66.3.172:22 100.127.69.165:25219 0 0
14458 sshd 100.66.3.172:22 100.127.69.165:7165 0 0

18. bpftrace
bpftrace - DTrace for Linux
18
File Descriptors
Linux
Kernel
VFS
Process
Syscall
Verifier
JIT Compiler
Syscall
bpftrace
Program
BPF
Maps
bpftrace:
github.com/iovisor/bpftrace
# bpftrace -e 'kprobe:do_sys_open { printf("%s: %sn", comm, str(arg1)) }'
Attaching 1 probe...
git: .git/objects/da
git: .git/objects/pack
git: /etc/localtime
systemd-journal: /var/log/journal/72d0774c88dc4943ae3d34ac356125dd
DNS Res~ver #15: /etc/hosts
^C
open()

19. Networking, load-balancing
and security for Kubernetes
19
Sockets
Linux
Kernel
TCP/IP
Container
Syscall
Verifier
JIT Compiler
Syscall
Clium
BPF
Maps
Network Device
Sockets
Container
Syscall
Network Device
Network
Hardware
TCP/IP
Kubernetes

20. 20
Container Networking
● Highly efficient and flexible networking
● Routing, Overlay, Cloud-provider native
● IPv4, IPv6, NAT46
● Multi cluster routing
Service Load balancing:
● Highly scalable L3-L4 load balancing
● Kubernetes services (replaces
kube-proxy)
● Multi-cluster
● Service affinity (prefer zones)
Container Security
● Identity-based network security
● API-aware security (HTTP, gRPC, Kafka,
Cassandra, memcached, ..)
● DNS-aware policies
● Encryption
● SSL data visibility via kTLS
Visibility
● Service topology map & live visualization
● Advanced network metrics & alerting
Servicemesh:
● Minimize overhead when injecting
servicemesh sidecar proxies
● Istio integration

21. 21
Hubble: eBPF Visibility for Kubernetes
# hubble observe --since=1m -t l7 -j
| jq 'select(.l7.dns.rcode==3) | .destination.namespace + "/" + .destination.pod_name'
| sort | uniq -c | sort -r
42 "starwars/jar-jar-binks-6f5847c97c-qmggv"

22. Development
Program Maps
Runtime
Go Development Toolchain
22
clang -target bpf
Sockets
Linux
Kernel
TCP/IP
recvmsg()sendmsg()
Process
Verifier
JIT Compiler
Syscall
BPF
Program
C source
BPF
Program
bytecode
BPF
Map
Syscall
Go Library
Go Library: https://github.com/cilium/ebpf

23. 23
Outlook: Future of
is turning the Linux
kernel into a microkernel.
● An increasing amount of new kernel
functionality is implemented with eBPF.
● 100% modular and composable.
● New additions can evolve at a rapid pace.
Much quicker than normal kernel
development.
Example: The linux kernel is not aware of
containers and microservices (it only knows
about namespaces). Cilium is making the
Linux kernel container and Kubernetes
aware.
could enable the Linux kernel
hotpatching we always dreamed about.
Problem:
● Linux kernel vulnerability requires to
patch kernel.
● Rebooting 20’000 servers takes a very
long time without risking extensive
downtime.
Function
Function
Function
Hotfix
Linux
Kernel

24. Thank You
eBPF Maintainers
Daniel Borkmann, Alexei Starovoitov
Cilium Team
André Martins, Jarno Rajahalme, Joe Stringer,
John Fastabend, Maciej Kwiek, Martynas
Pumputis, Paul Chaignon, Quentin Monnet,
Ray Bejjani, Tobias Klauser
Facebook Team
Andrii Nakryiko, Andrey Ignatov, Jakub
Kicinski, Martin KaFai Lau, Roman Gushchin,
Song Liu, Yonghong Song
Google Team
Chenbo Feng, KP Singh, Lorenzo Colitti,
Maciej Żenczykowski, Stanislav Fomichev,
BCC & bpftrace
Alastair Robertson, Brendan Gregg, Brenden
Blanco
Kernel Team
Björn Töpel, David S. Miller, Edward Cree,
Jesper Brouer, Toke Høiland-Jørgensen
24
● BPF Getting Started Guide
BPF and XDP Reference Guide
● Cilium
github.com/cilium/cilium
● Twitter
@ciliumproject
● Contact the speaker
@tgraf__
All images: Pixabay