SlideShare a Scribd company logo
BPF – in-kernel virtual machine
1!
BPF is
•  Berkeley Packet Filter
•  low level instruction set
•  kernel infrastructure around it
•  interpreter
•  JITs
•  maps
•  helper functions
Agenda
•  status and new use cases
•  architecture and design
•  demo
extended BPF JITs and compilers
•  x64 JIT upstreamed
•  arm64 JIT upstreamed
•  s390 JIT in progress
•  ppc JIT in progress
•  LLVM backend is upstreamed
•  gcc backend is in progress
extended BPF use cases
1.  networking
2.  tracing (analytics, monitoring, debugging)
3.  in-kernel optimizations
4.  hw modeling
5.  crazy stuff...
1. extended BPF in networking
•  socket filters
•  four use cases of bpf in openvswitch (bpf+ovs)
•  bpf as an action on flow-hit
•  bpf as fallback on flow-miss
•  bpf as packet parser before flow lookup
•  bpf to completely replace ovs datapath
•  two use cases in traffic control (bpf+tc)
•  cls – packet parser and classifier
•  act – action
•  bpf as net_device
2. extended BPF in tracing
•  bpf+kprobe – dtrace/systemtap like
•  bpf+syscalls – analytics and monitoring
•  bpf+tracepoints – faster alternative to kprobes
•  TCP stack instrumentation with bpf+tracepoints as non-
intrusive alternative to web10g
•  disk latency monitoring
•  live kernel debugging (with and without debug info)
3. extended BPF for in-kernel optimizations
•  kernel interface is kept unmodified. subsystems use bpf to
accelerate internal execution
•  predicate tree walker of tracing filters -> bpf
•  nft (netfilter tables) -> bpf
4. extended BPF for HW modeling
•  p4 – language for programing flexible network switches
•  p4 compiler into bpf (userspace)
•  pass bpf into kernel via switchdev abstraction
•  rocker device (part of qemu) to execute bpf
5. other crazy uses of BPF
•  'reverse BPF' was proposed
•  in-kernel NIC drivers expose BPF back to user space as generic program to
construct hw specific data structures
•  bpf -> NPUs
•  some networking HW vendors planning to translate bpf directly to HW
classic BPF
•  BPF - Berkeley Packet Filter
•  inspired by BSD
•  introduced in linux in 1997 in version 2.1.75
•  initially used as socket filter by packet capture tool tcpdump
(via libpcap)
classic BPF
•  two 32-bit registers: A, X
•  implicit stack of 16 32-bit slots (LD_MEM, ST_MEM insns)
•  full integer arithmetic
•  explicit load/store from packet (LD_ABS, LD_IND insns)
•  conditional branches (with two destinations: jump true/false)
Ex: tcpdump syntax and classic BPF assembler
•  tcpdump –d 'ip and tcp port 22’
(000) ldh [12] // fetch eth proto

(001) jeq #0x800 jt 2"jf 12 // is it IPv4 ?

(002) ldb [23] // fetch ip proto

(003) jeq #0x6 jt 4"jf 12 // is it TCP ?

(004) ldh [20] // fetch frag_off

(005) jset #0x1fff jt 12 jf 6 // is it a frag?

(006) ldxb 4*([14]&0xf) // fetch ip header len

(007) ldh [x + 14] // fetch src port

(008) jeq #0x16 jt 11 jf 9 // is it 22 ?

(009) ldh [x + 16] // fetch dest port

(010) jeq #0x16 jt 11 jf 12 // is it 22 ?

(011) ret #65535 // trim packet and pass

(012) ret #0 // ignore packet"
Classic BPF for use cases
•  socket filters (drop or trim packet and pass to user space)
•  used by tcpdump/libpcap, wireshark, nmap, dhcp, arpd, ...
•  in networking subsystems
•  cls_bpf (TC classifier), xt_bpf, ppp, team, ...
•  seccomp (chrome sandboxing)
•  introduced in 2012 to filter syscall arguments with bpf program
Classic BPF safety
•  verifier checks all instructions, forward jumps only, stack slot
load/store, etc
•  instruction set has some built-in safety (no exposed stack
pointer, instead load instruction has ‘mem’ modifier)
•  dynamic packet-boundary checks
Classic BPF extensions
•  over years multiple extensions were added in the form of ‘load
from negative hard coded offset’
•  LD_ABS -0x1000 – skb->protocol
LD_ABS -0x1000+4 – skb->pkt_type
LD_ABS -0x1000+56 – get_random()
Extended BPF
•  design goals:
•  parse, lookup, update, modify network packets
•  loadable as kernel modules on demand, on live traffic
•  safe on production system
•  performance equal to native x86 code
•  fast interpreter speed (good performance on all architectures)
•  calls into bpf and calls from bpf to kernel should be free (no FFI overhead)
in kernel 3.15
tcpdump dhclient chrome
cls,
xt,
team,
ppp,
…
classic -> extended
bpf engine
in kernel 3.18
tcpdump
dhclient
chrome
gcc/llvm
libbpf
classic -> extended
bpf engine
bpf syscall
verifier
x64 JIT
arm64 JIT
Early prototypes
•  Failed approach #1 (design a VM from scratch)
•  performance was too slow, user tools need to be developed from scratch as
well
•  Failed approach #2 (have kernel disassemble and verify x86
instructions)
•  too many instruction combinations, disasm/verifier needs to be rewritten for
every architecture
Extended BPF
•  take a mix of real CPU instructions
•  10% classic BPF + 70% x86 + 25% arm64 + 5% risc
•  rename every x86 instruction ‘mov rax, rbx’ into ‘mov r1, r2’
•  analyze x86/arm64/risc calling conventions and define a
common one for this ‘renamed’ instruction set
•  make instruction encoding fixed size (for high interpreter
speed)
•  reuse classic BPF instruction encoding (for trivial classic-
>extended conversion)
extended vs classic BPF
•  ten 64-bit registers vs two 32-bit registers
•  arbitrary load/store vs stack load/store
•  call instruction
Performance
•  user space compiler ‘thinks’ that it’s emitting simplified x86
code
•  kernel verifies this ‘simplified x86’ code
•  kernel JIT translates each ‘simplified x86’ insn into real x86
•  all registers map one-to-one
•  most of instructions map one-to-one
•  bpf ‘call’ instruction maps to x86 ‘call’
Extended BPF calling convention
•  BPF calling convention was carefully selected to match a
subset of amd64/arm64 ABIs to avoid extra copy in calls:
•  R0 – return value
•  R1..R5 – function arguments
•  R6..R9 – callee saved
•  R10 – frame pointer
Mapping of BPF registers to x86
•  R0 – rax return value from function

R1 – rdi 1st argument

R2 – rsi 2nd argument

R3 – rdx 3rd argument

R4 – rcx 4th argument

R5 – r8 5th argument

R6 – rbx callee saved

R7 - r13 callee saved

R8 - r14 callee saved

R9 - r15 callee saved

R10 – rbp frame pointer"
calls and helper functions
•  bpf ‘call’ and set of in-kernel helper functions define what bpf
programs can do
•  bpf code itself is a ‘glue’ between calls to in-kernel helper
functions
•  helpers
•  map_lookup/update/delete
•  ktime_get
•  packet_write
•  fetch
BPF maps
•  maps is a generic storage of different types for sharing data between
kernel and userspace
•  The maps are accessed from user space via BPF syscall, which has
commands:
•  create a map with given type and attributes
map_fd = bpf(BPF_MAP_CREATE, union bpf_attr *attr, u32 size)
•  lookup key/value, update, delete, iterate, delete a map
•  userspace programs use this syscall to create/access maps that BPF
programs are concurrently updating
BPF compilers
•  BPF backend for LLVM is in trunk and will be released as part of 3.7
•  BPF backend for GCC is being worked on
•  C front-end (clang) is used today to compile C code into BPF
•  tracing and networking use cases may need custom languages
•  BPF backend only knows how to emit instructions (calls to helper
functions look like normal calls)
Extended BPF assembler
0: r1 = *(u64 *)(r1 +8)

1: *(u64 *)(r10 -8) = r1

2: r1 = 1

3: *(u64 *)(r10 -16) = r1

4: r1 = map_fd

6: r2 = r10

7: r2 += -8

8: call 1

9: if r0 == 0x0 goto pc+4

10: r1 = *(u64 *)(r0 +0)

11: r1 += 1

12: *(u64 *)(r0 +0) = r1

13: goto pc+8

14: r1 = map_fd

16: r2 = r10

17: r2 += -8

18: r3 = r10

19: r3 += -16

20: r4 = 0

21: call 2

22: r0 = 0

23: exit"
int bpf_prog(struct bpf_context *ctx)

{

u64 loc = ctx->arg2;

u64 init_val = 1;

u64 *value;



value = bpf_map_lookup_elem(&my_map, &loc);

if (value)

*value += 1;

else

bpf_map_update_elem(&my_map, &loc,

&init_val, BPF_ANY);

return 0;

}"
"
compiled by LLVM from C to bpf asm
compiler as a library
tracing script in .txt file
bpf_create_map
.txt parser
llvm mcjit api
bpf
backend
x64
backend
bpf code x86 code
bpf_prog_load
user
kernel
libllvm
perf
binary
run it
BPF verifier (CFG check)
•  To minimize run-time overhead anything that can be checked
statically is done by verifier
•  all jumps of a program form a CFG which is checked for loops
•  DAG check = non-recursive depth-first-search
•  if back-edge exists -> there is a loop -> reject program
•  jumps back are allowed if they don’t form loops
•  bpf compiler can move cold basic blocks out of critical path
•  likely/unlikely() hints give extra performance
BPF verifier (instruction walking)
•  once it’s known that all paths through the program reach final ‘exit’
instruction, brute force analyzer of all instructions starts
•  it descents all possible paths from the 1st insn till ‘exit’ insn
•  it simulates execution of every insn and updates the state change of
registers and stack
BPF verifier
•  at the start of the program:
•  type of R1 = PTR_TO_CTX
type of R10 = FRAME_PTR
other registers and stack is unreadable
•  when verifier sees:
•  ‘R2 = R1’ instruction it copies the type of R1 into R2
•  ‘R3 = 123’ instruction, the type of R3 becomes CONST_IMM
•  ‘exit’ instruction, it checks that R0 is readable
•  ‘if (R4 == 456) goto pc+5’ instruction, it checks that R4 is readable and forks current
state of registers and stack into ‘true’ and ‘false’ branches
BPF verifier (state pruning)
•  every branch adds another fork for verifier to explore, therefore
branch pruning is important
•  when verifiers sees an old state that has more strict register state and
more strict stack state then the current branch doesn't need to be
explored further, since verifier already concluded that more strict state
leads to valid ‘exit’
•  two states are equivalent if register state is more conservative and
explored stack state is more conservative than the current one
unprivileged programs?
•  today extended BPF is root only
•  to consider unprivileged access:
•  teach verifier to conditionally reject programs that expose kernel addresses to
user space
•  constant blinding pass
BPF for tracing
•  BPF is seen as alternative to systemtap/dtrace
•  provides in-kernel aggregation, event filtering
•  can be 'always on'
•  must have minimal overhead
BPF for tracing (kernel part)
struct bpf_map_def SEC("maps") my_hist_map = {
.type = BPF_MAP_TYPE_ARRAY,
.key_size = sizeof(u32),
.value_size = sizeof(u64),
.max_entries = 64,
};
SEC("events/syscalls/sys_enter_write")
int bpf_prog(struct bpf_context *ctx)
{
u64 write_size = ctx->arg3;
u32 index = log2(write_size);
u64 *value;
value = bpf_map_lookup_elem(&my_hist_map, &index);
if (value)
__sync_fetch_and_add(value, 1);
return 0;
}
sent to kernel as bpf map via bpf() syscall
compiled by llvm into .o and
loaded via bpf() syscall
name of elf section - tracing event to attach via perf_event ioctl
BPF for tracing (user part)
u64 data[64] = {}; u32 key; u64 value;
for (key = 0; key < 64; key++) {
bpf_lookup_elem(fd, &key, &value);
data[key] = value;
if (value && key > max_ind)
max_ind = key;
if (value > max_value)
max_value = value;
}
printf("syscall write() statsn");
user space walks the map and fetches elements via bpf() syscall
syscall write() stats"
byte_size : count distribution"
1 -> 1 : 9 |*************************** |"
2 -> 3 : 0 | |"
4 -> 7 : 0 | |"
8 -> 15 : 2 |***** |"
16 -> 31 : 0 | |"
32 -> 63 : 10 |****************************** |"
64 -> 127 : 12 |************************************* |"
128 -> 255 : 1 |** |"
256 -> 511 : 2 |***** |"
(Brendan Gregg’s slide)
Extended BPF
demo
41!

More Related Content

What's hot

Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
Brendan Gregg
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
Netronome
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
Brendan Gregg
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
Brendan Gregg
 
Cilium - Fast IPv6 Container Networking with BPF and XDP
Cilium - Fast IPv6 Container Networking with BPF and XDPCilium - Fast IPv6 Container Networking with BPF and XDP
Cilium - Fast IPv6 Container Networking with BPF and XDP
Thomas Graf
 
eBPF maps 101
eBPF maps 101eBPF maps 101
eBPF maps 101
SUSE Labs Taipei
 
BPF / XDP 8월 세미나 KossLab
BPF / XDP 8월 세미나 KossLabBPF / XDP 8월 세미나 KossLab
BPF / XDP 8월 세미나 KossLab
Taeung Song
 
Using eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in CiliumUsing eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in Cilium
ScyllaDB
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
SUSE Labs Taipei
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep Dive
Michal Rostecki
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
Brendan Gregg
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
Brendan Gregg
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
Thomas Graf
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking Walkthrough
Thomas Graf
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
Brendan Gregg
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF Superpowers
Brendan Gregg
 
Cfgmgmtcamp 2023 — eBPF Superpowers
Cfgmgmtcamp 2023 — eBPF SuperpowersCfgmgmtcamp 2023 — eBPF Superpowers
Cfgmgmtcamp 2023 — eBPF Superpowers
Raphaël PINSON
 
Linux Linux Traffic Control
Linux Linux Traffic ControlLinux Linux Traffic Control
Linux Linux Traffic Control
SUSE Labs Taipei
 
ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!
Affan Syed
 
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDPDockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
Thomas Graf
 

What's hot (20)

Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
 
Cilium - Fast IPv6 Container Networking with BPF and XDP
Cilium - Fast IPv6 Container Networking with BPF and XDPCilium - Fast IPv6 Container Networking with BPF and XDP
Cilium - Fast IPv6 Container Networking with BPF and XDP
 
eBPF maps 101
eBPF maps 101eBPF maps 101
eBPF maps 101
 
BPF / XDP 8월 세미나 KossLab
BPF / XDP 8월 세미나 KossLabBPF / XDP 8월 세미나 KossLab
BPF / XDP 8월 세미나 KossLab
 
Using eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in CiliumUsing eBPF for High-Performance Networking in Cilium
Using eBPF for High-Performance Networking in Cilium
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep Dive
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking Walkthrough
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF Superpowers
 
Cfgmgmtcamp 2023 — eBPF Superpowers
Cfgmgmtcamp 2023 — eBPF SuperpowersCfgmgmtcamp 2023 — eBPF Superpowers
Cfgmgmtcamp 2023 — eBPF Superpowers
 
Linux Linux Traffic Control
Linux Linux Traffic ControlLinux Linux Traffic Control
Linux Linux Traffic Control
 
ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!
 
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDPDockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP
 

Similar to BPF - in-kernel virtual machine

Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
Sasha Goldshtein
 
Ebpf ovsconf-2016
Ebpf ovsconf-2016Ebpf ovsconf-2016
Ebpf ovsconf-2016
Cheng-Chun William Tu
 
eBPF Basics
eBPF BasicseBPF Basics
eBPF Basics
Michael Kehoe
 
Berkeley Packet Filters
Berkeley Packet FiltersBerkeley Packet Filters
Berkeley Packet Filters
Kernel TLV
 
MM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary Demos
MM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary DemosMM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary Demos
MM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary Demos
AMD Developer Central
 
Kernel Recipes 2019 - BPF at Facebook
Kernel Recipes 2019 - BPF at FacebookKernel Recipes 2019 - BPF at Facebook
Kernel Recipes 2019 - BPF at Facebook
Anne Nicolas
 
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
OpenEBS
 
OSN days 2019 - Open Networking and Programmable Switch
OSN days 2019 - Open Networking and Programmable SwitchOSN days 2019 - Open Networking and Programmable Switch
OSN days 2019 - Open Networking and Programmable Switch
Chun Ming Ou
 
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
Yuuki Takano
 
Comprehensive XDP Off‌load-handling the Edge Cases
Comprehensive XDP Off‌load-handling the Edge CasesComprehensive XDP Off‌load-handling the Edge Cases
Comprehensive XDP Off‌load-handling the Edge Cases
Netronome
 
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
Kernel TLV
 
Network Programming: Data Plane Development Kit (DPDK)
Network Programming: Data Plane Development Kit (DPDK)Network Programming: Data Plane Development Kit (DPDK)
Network Programming: Data Plane Development Kit (DPDK)
Andriy Berestovskyy
 
Netlink-Optimization.pptx
Netlink-Optimization.pptxNetlink-Optimization.pptx
Netlink-Optimization.pptx
KalimuthuVelappan
 
BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!
Linaro
 
15 ia64
15 ia6415 ia64
15 ia64
dilip kumar
 
DEF CON 27 - JEFF DILEO - evil e bpf in depth
DEF CON 27 - JEFF DILEO - evil e bpf in depthDEF CON 27 - JEFF DILEO - evil e bpf in depth
DEF CON 27 - JEFF DILEO - evil e bpf in depth
Felipe Prado
 
Routing basics/CEF
Routing basics/CEFRouting basics/CEF
Routing basics/CEF
Dmitry Figol
 
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThe Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
Thomas Graf
 
Microchip's PIC Micro Controller
Microchip's PIC Micro ControllerMicrochip's PIC Micro Controller
Microchip's PIC Micro Controller
Midhu S V Unnithan
 
Lec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIW
Lec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIWLec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIW
Lec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIW
Hsien-Hsin Sean Lee, Ph.D.
 

Similar to BPF - in-kernel virtual machine (20)

Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
 
Ebpf ovsconf-2016
Ebpf ovsconf-2016Ebpf ovsconf-2016
Ebpf ovsconf-2016
 
eBPF Basics
eBPF BasicseBPF Basics
eBPF Basics
 
Berkeley Packet Filters
Berkeley Packet FiltersBerkeley Packet Filters
Berkeley Packet Filters
 
MM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary Demos
MM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary DemosMM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary Demos
MM-4105, Realtime 4K HDR Decoding with GPU ACES, by Gary Demos
 
Kernel Recipes 2019 - BPF at Facebook
Kernel Recipes 2019 - BPF at FacebookKernel Recipes 2019 - BPF at Facebook
Kernel Recipes 2019 - BPF at Facebook
 
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
 
OSN days 2019 - Open Networking and Programmable Switch
OSN days 2019 - Open Networking and Programmable SwitchOSN days 2019 - Open Networking and Programmable Switch
OSN days 2019 - Open Networking and Programmable Switch
 
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
 
Comprehensive XDP Off‌load-handling the Edge Cases
Comprehensive XDP Off‌load-handling the Edge CasesComprehensive XDP Off‌load-handling the Edge Cases
Comprehensive XDP Off‌load-handling the Edge Cases
 
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
 
Network Programming: Data Plane Development Kit (DPDK)
Network Programming: Data Plane Development Kit (DPDK)Network Programming: Data Plane Development Kit (DPDK)
Network Programming: Data Plane Development Kit (DPDK)
 
Netlink-Optimization.pptx
Netlink-Optimization.pptxNetlink-Optimization.pptx
Netlink-Optimization.pptx
 
BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!BKK16-103 OpenCSD - Open for Business!
BKK16-103 OpenCSD - Open for Business!
 
15 ia64
15 ia6415 ia64
15 ia64
 
DEF CON 27 - JEFF DILEO - evil e bpf in depth
DEF CON 27 - JEFF DILEO - evil e bpf in depthDEF CON 27 - JEFF DILEO - evil e bpf in depth
DEF CON 27 - JEFF DILEO - evil e bpf in depth
 
Routing basics/CEF
Routing basics/CEFRouting basics/CEF
Routing basics/CEF
 
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThe Next Generation Firewall for Red Hat Enterprise Linux 7 RC
The Next Generation Firewall for Red Hat Enterprise Linux 7 RC
 
Microchip's PIC Micro Controller
Microchip's PIC Micro ControllerMicrochip's PIC Micro Controller
Microchip's PIC Micro Controller
 
Lec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIW
Lec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIWLec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIW
Lec15 Computer Architecture by Hsien-Hsin Sean Lee Georgia Tech -- EPIC VLIW
 

Recently uploaded

TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
aeeva
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
kalichargn70th171
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert
vaishalijagtap12
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Jhone kinadey
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
DevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps ServicesDevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps Services
seospiralmantra
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
campbellclarkson
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
OnePlan Solutions
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid
 
Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
Massimo Artizzu
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISDECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
Tier1 app
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
Severalnines
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 

Recently uploaded (20)

TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
 
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
 
42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert
 
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management AppsBoost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
DevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps ServicesDevOps Consulting Company | Hire DevOps Services
DevOps Consulting Company | Hire DevOps Services
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
 
Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISDECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
 
Kubernetes at Scale: Going Multi-Cluster with Istio
Kubernetes at Scale:  Going Multi-Cluster  with IstioKubernetes at Scale:  Going Multi-Cluster  with Istio
Kubernetes at Scale: Going Multi-Cluster with Istio
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 

BPF - in-kernel virtual machine

  • 1. BPF – in-kernel virtual machine 1!
  • 2. BPF is •  Berkeley Packet Filter •  low level instruction set •  kernel infrastructure around it •  interpreter •  JITs •  maps •  helper functions
  • 3. Agenda •  status and new use cases •  architecture and design •  demo
  • 4. extended BPF JITs and compilers •  x64 JIT upstreamed •  arm64 JIT upstreamed •  s390 JIT in progress •  ppc JIT in progress •  LLVM backend is upstreamed •  gcc backend is in progress
  • 5. extended BPF use cases 1.  networking 2.  tracing (analytics, monitoring, debugging) 3.  in-kernel optimizations 4.  hw modeling 5.  crazy stuff...
  • 6. 1. extended BPF in networking •  socket filters •  four use cases of bpf in openvswitch (bpf+ovs) •  bpf as an action on flow-hit •  bpf as fallback on flow-miss •  bpf as packet parser before flow lookup •  bpf to completely replace ovs datapath •  two use cases in traffic control (bpf+tc) •  cls – packet parser and classifier •  act – action •  bpf as net_device
  • 7. 2. extended BPF in tracing •  bpf+kprobe – dtrace/systemtap like •  bpf+syscalls – analytics and monitoring •  bpf+tracepoints – faster alternative to kprobes •  TCP stack instrumentation with bpf+tracepoints as non- intrusive alternative to web10g •  disk latency monitoring •  live kernel debugging (with and without debug info)
  • 8. 3. extended BPF for in-kernel optimizations •  kernel interface is kept unmodified. subsystems use bpf to accelerate internal execution •  predicate tree walker of tracing filters -> bpf •  nft (netfilter tables) -> bpf
  • 9. 4. extended BPF for HW modeling •  p4 – language for programing flexible network switches •  p4 compiler into bpf (userspace) •  pass bpf into kernel via switchdev abstraction •  rocker device (part of qemu) to execute bpf
  • 10. 5. other crazy uses of BPF •  'reverse BPF' was proposed •  in-kernel NIC drivers expose BPF back to user space as generic program to construct hw specific data structures •  bpf -> NPUs •  some networking HW vendors planning to translate bpf directly to HW
  • 11. classic BPF •  BPF - Berkeley Packet Filter •  inspired by BSD •  introduced in linux in 1997 in version 2.1.75 •  initially used as socket filter by packet capture tool tcpdump (via libpcap)
  • 12. classic BPF •  two 32-bit registers: A, X •  implicit stack of 16 32-bit slots (LD_MEM, ST_MEM insns) •  full integer arithmetic •  explicit load/store from packet (LD_ABS, LD_IND insns) •  conditional branches (with two destinations: jump true/false)
  • 13. Ex: tcpdump syntax and classic BPF assembler •  tcpdump –d 'ip and tcp port 22’ (000) ldh [12] // fetch eth proto
 (001) jeq #0x800 jt 2"jf 12 // is it IPv4 ?
 (002) ldb [23] // fetch ip proto
 (003) jeq #0x6 jt 4"jf 12 // is it TCP ?
 (004) ldh [20] // fetch frag_off
 (005) jset #0x1fff jt 12 jf 6 // is it a frag?
 (006) ldxb 4*([14]&0xf) // fetch ip header len
 (007) ldh [x + 14] // fetch src port
 (008) jeq #0x16 jt 11 jf 9 // is it 22 ?
 (009) ldh [x + 16] // fetch dest port
 (010) jeq #0x16 jt 11 jf 12 // is it 22 ?
 (011) ret #65535 // trim packet and pass
 (012) ret #0 // ignore packet"
  • 14. Classic BPF for use cases •  socket filters (drop or trim packet and pass to user space) •  used by tcpdump/libpcap, wireshark, nmap, dhcp, arpd, ... •  in networking subsystems •  cls_bpf (TC classifier), xt_bpf, ppp, team, ... •  seccomp (chrome sandboxing) •  introduced in 2012 to filter syscall arguments with bpf program
  • 15. Classic BPF safety •  verifier checks all instructions, forward jumps only, stack slot load/store, etc •  instruction set has some built-in safety (no exposed stack pointer, instead load instruction has ‘mem’ modifier) •  dynamic packet-boundary checks
  • 16. Classic BPF extensions •  over years multiple extensions were added in the form of ‘load from negative hard coded offset’ •  LD_ABS -0x1000 – skb->protocol LD_ABS -0x1000+4 – skb->pkt_type LD_ABS -0x1000+56 – get_random()
  • 17. Extended BPF •  design goals: •  parse, lookup, update, modify network packets •  loadable as kernel modules on demand, on live traffic •  safe on production system •  performance equal to native x86 code •  fast interpreter speed (good performance on all architectures) •  calls into bpf and calls from bpf to kernel should be free (no FFI overhead)
  • 18. in kernel 3.15 tcpdump dhclient chrome cls, xt, team, ppp, … classic -> extended bpf engine
  • 19. in kernel 3.18 tcpdump dhclient chrome gcc/llvm libbpf classic -> extended bpf engine bpf syscall verifier x64 JIT arm64 JIT
  • 20. Early prototypes •  Failed approach #1 (design a VM from scratch) •  performance was too slow, user tools need to be developed from scratch as well •  Failed approach #2 (have kernel disassemble and verify x86 instructions) •  too many instruction combinations, disasm/verifier needs to be rewritten for every architecture
  • 21. Extended BPF •  take a mix of real CPU instructions •  10% classic BPF + 70% x86 + 25% arm64 + 5% risc •  rename every x86 instruction ‘mov rax, rbx’ into ‘mov r1, r2’ •  analyze x86/arm64/risc calling conventions and define a common one for this ‘renamed’ instruction set •  make instruction encoding fixed size (for high interpreter speed) •  reuse classic BPF instruction encoding (for trivial classic- >extended conversion)
  • 22. extended vs classic BPF •  ten 64-bit registers vs two 32-bit registers •  arbitrary load/store vs stack load/store •  call instruction
  • 23. Performance •  user space compiler ‘thinks’ that it’s emitting simplified x86 code •  kernel verifies this ‘simplified x86’ code •  kernel JIT translates each ‘simplified x86’ insn into real x86 •  all registers map one-to-one •  most of instructions map one-to-one •  bpf ‘call’ instruction maps to x86 ‘call’
  • 24. Extended BPF calling convention •  BPF calling convention was carefully selected to match a subset of amd64/arm64 ABIs to avoid extra copy in calls: •  R0 – return value •  R1..R5 – function arguments •  R6..R9 – callee saved •  R10 – frame pointer
  • 25. Mapping of BPF registers to x86 •  R0 – rax return value from function
 R1 – rdi 1st argument
 R2 – rsi 2nd argument
 R3 – rdx 3rd argument
 R4 – rcx 4th argument
 R5 – r8 5th argument
 R6 – rbx callee saved
 R7 - r13 callee saved
 R8 - r14 callee saved
 R9 - r15 callee saved
 R10 – rbp frame pointer"
  • 26. calls and helper functions •  bpf ‘call’ and set of in-kernel helper functions define what bpf programs can do •  bpf code itself is a ‘glue’ between calls to in-kernel helper functions •  helpers •  map_lookup/update/delete •  ktime_get •  packet_write •  fetch
  • 27. BPF maps •  maps is a generic storage of different types for sharing data between kernel and userspace •  The maps are accessed from user space via BPF syscall, which has commands: •  create a map with given type and attributes map_fd = bpf(BPF_MAP_CREATE, union bpf_attr *attr, u32 size) •  lookup key/value, update, delete, iterate, delete a map •  userspace programs use this syscall to create/access maps that BPF programs are concurrently updating
  • 28. BPF compilers •  BPF backend for LLVM is in trunk and will be released as part of 3.7 •  BPF backend for GCC is being worked on •  C front-end (clang) is used today to compile C code into BPF •  tracing and networking use cases may need custom languages •  BPF backend only knows how to emit instructions (calls to helper functions look like normal calls)
  • 29. Extended BPF assembler 0: r1 = *(u64 *)(r1 +8)
 1: *(u64 *)(r10 -8) = r1
 2: r1 = 1
 3: *(u64 *)(r10 -16) = r1
 4: r1 = map_fd
 6: r2 = r10
 7: r2 += -8
 8: call 1
 9: if r0 == 0x0 goto pc+4
 10: r1 = *(u64 *)(r0 +0)
 11: r1 += 1
 12: *(u64 *)(r0 +0) = r1
 13: goto pc+8
 14: r1 = map_fd
 16: r2 = r10
 17: r2 += -8
 18: r3 = r10
 19: r3 += -16
 20: r4 = 0
 21: call 2
 22: r0 = 0
 23: exit" int bpf_prog(struct bpf_context *ctx)
 {
 u64 loc = ctx->arg2;
 u64 init_val = 1;
 u64 *value;
 
 value = bpf_map_lookup_elem(&my_map, &loc);
 if (value)
 *value += 1;
 else
 bpf_map_update_elem(&my_map, &loc,
 &init_val, BPF_ANY);
 return 0;
 }" " compiled by LLVM from C to bpf asm
  • 30. compiler as a library tracing script in .txt file bpf_create_map .txt parser llvm mcjit api bpf backend x64 backend bpf code x86 code bpf_prog_load user kernel libllvm perf binary run it
  • 31. BPF verifier (CFG check) •  To minimize run-time overhead anything that can be checked statically is done by verifier •  all jumps of a program form a CFG which is checked for loops •  DAG check = non-recursive depth-first-search •  if back-edge exists -> there is a loop -> reject program •  jumps back are allowed if they don’t form loops •  bpf compiler can move cold basic blocks out of critical path •  likely/unlikely() hints give extra performance
  • 32. BPF verifier (instruction walking) •  once it’s known that all paths through the program reach final ‘exit’ instruction, brute force analyzer of all instructions starts •  it descents all possible paths from the 1st insn till ‘exit’ insn •  it simulates execution of every insn and updates the state change of registers and stack
  • 33. BPF verifier •  at the start of the program: •  type of R1 = PTR_TO_CTX type of R10 = FRAME_PTR other registers and stack is unreadable •  when verifier sees: •  ‘R2 = R1’ instruction it copies the type of R1 into R2 •  ‘R3 = 123’ instruction, the type of R3 becomes CONST_IMM •  ‘exit’ instruction, it checks that R0 is readable •  ‘if (R4 == 456) goto pc+5’ instruction, it checks that R4 is readable and forks current state of registers and stack into ‘true’ and ‘false’ branches
  • 34. BPF verifier (state pruning) •  every branch adds another fork for verifier to explore, therefore branch pruning is important •  when verifiers sees an old state that has more strict register state and more strict stack state then the current branch doesn't need to be explored further, since verifier already concluded that more strict state leads to valid ‘exit’ •  two states are equivalent if register state is more conservative and explored stack state is more conservative than the current one
  • 35. unprivileged programs? •  today extended BPF is root only •  to consider unprivileged access: •  teach verifier to conditionally reject programs that expose kernel addresses to user space •  constant blinding pass
  • 36. BPF for tracing •  BPF is seen as alternative to systemtap/dtrace •  provides in-kernel aggregation, event filtering •  can be 'always on' •  must have minimal overhead
  • 37. BPF for tracing (kernel part) struct bpf_map_def SEC("maps") my_hist_map = { .type = BPF_MAP_TYPE_ARRAY, .key_size = sizeof(u32), .value_size = sizeof(u64), .max_entries = 64, }; SEC("events/syscalls/sys_enter_write") int bpf_prog(struct bpf_context *ctx) { u64 write_size = ctx->arg3; u32 index = log2(write_size); u64 *value; value = bpf_map_lookup_elem(&my_hist_map, &index); if (value) __sync_fetch_and_add(value, 1); return 0; } sent to kernel as bpf map via bpf() syscall compiled by llvm into .o and loaded via bpf() syscall name of elf section - tracing event to attach via perf_event ioctl
  • 38. BPF for tracing (user part) u64 data[64] = {}; u32 key; u64 value; for (key = 0; key < 64; key++) { bpf_lookup_elem(fd, &key, &value); data[key] = value; if (value && key > max_ind) max_ind = key; if (value > max_value) max_value = value; } printf("syscall write() statsn"); user space walks the map and fetches elements via bpf() syscall syscall write() stats" byte_size : count distribution" 1 -> 1 : 9 |*************************** |" 2 -> 3 : 0 | |" 4 -> 7 : 0 | |" 8 -> 15 : 2 |***** |" 16 -> 31 : 0 | |" 32 -> 63 : 10 |****************************** |" 64 -> 127 : 12 |************************************* |" 128 -> 255 : 1 |** |" 256 -> 511 : 2 |***** |"