This document discusses how eBPF (extended Berkeley Packet Filter) can be used for kernel tracing. It provides an overview of BPF and eBPF, how eBPF programs are compiled and run in the kernel, the use of BPF maps, and how eBPF enables new possibilities for dynamic kernel instrumentation through techniques like Kprobes and ftrace.

1. Meet-cute between
eBPF and Kernel Tracing
Viller Hsiao <villerhsiao@gmail.com>
Jul. 5, 2016

2. 03/09/2016 2
Who am I ?
Viller Hsiao
Embedded Linux / RTOS engineer
http://image.dfdaily.com/201
2/5/4/634716931128751250504b
050c1_nEO_IMG.jpg

3. 03/09/2016 3
BPF
Berkeley Packet Filter
by Steven McCanne and Van Jacobson, 1993

4. 03/09/2016 4
Who am I ?
Viller Hsiao
Embedded Linux / RTOS engineer
http://image.dfdaily.com/201
2/5/4/634716931128751250504b
050c1_nEO_IMG.jpg

5. 03/09/2016 5
Berkeley Packet Filter
Packet filter: tcpdump -nnnX port 3000

6. 03/09/2016 6
network
stack
sniffer
kernel
user
net if
Applications
tcpdump ­nnnX port 3000
port 3000
VM filter
http://www.ic
onsdb.com/ico
ns/download/g
ray/empty-fil
ter-512.png
In­kernel Packet Filter

7. 03/09/2016 7
Berkeley Packet Filter
Improve unix packet filter

8. 03/09/2016 8
Berkeley Packet Filter
Improve unix packet filter
Replace stack-based VM with register-based VM

9. 03/09/2016 9
Berkeley Packet Filter
Improve unix packet filter
Replace stack-based VM with register-based VM
20 times faster than original design

10. 03/09/2016 10
In­Kernel VM for Filtering
Flexibility
Efficiency Security

11. 03/09/2016 11
BPF in Linux
a.k.a. Linux Socket Filter
kernel 2.1.75, in 1997

12. 03/09/2016 12
Areas Use BPF
in Linux Nowadays
●
Linux­3.4 (2012), Seccomp filters of syscalls (chrome sandboxing)
●
Packet classifier for traffic contol
●
Actions for traffic control
●
Xtables packet filtering
●
Tracing

13. 03/09/2016 13
Story today,
When kernel tracing meets ebpf
http://2.blog.xuite.net/2/4/7/8/11001626/blog_70864/txt/17378250/0.jpg

14. 03/09/2016 14
Examples of BPF Program
ldh [12]
jne #0x806, drop
ret #­1
drop: ret #0
ARP packets
ICMP
random packet sampling
1 in 4
ldh [12]
jne #0x800, drop
ldb [23]
jneq #1, drop
ld rand
mod #4
jneq #1, drop
ret #­1
drop: ret #0
helper
extensions

15. 03/09/2016 15
BPF Example: Translate to Binary
$ ./bpf_asm ­c foo
Opcode JT JF K
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 1, 0x00000806 },
{ 0x06, 0, 0, 0xffffffff },
{ 0x06, 0, 0, 0000000000 },

16. 03/09/2016 16
Userspace Application
struct sock_filter code[] = {
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 8, 0x000086dd },
…
};
struct sock_fprog bpf = {
.len = ARRAY_SIZE(code),
.filter = code,
};
sock = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
if (sock < 0)
/* ... bail out ... */
ret = setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &bpf, sizeof(bpf));
if (ret < 0)
/* ... bail out ... */
BPF Binary

17. 03/09/2016 17
BPF JIT Compiler
in 2011
●
Linux­3.0, by Eric Dumazet
●
Architecture support
– x86_64, SPARC, PowerPC, ARM, ARM64, MIPS and
s390
$ echo 1 > /proc/sys/net/core/bpf_jit_enable

18. 03/09/2016 18
extended BPF
Linux-3.15
by Alexei Starovoitov, 2013

19. 03/09/2016 19
Classic BPF
vs
Internal BPF (a.k.a extended BPF)

20. 03/09/2016 20
eBPF Design Goals
●
Just­in­time map to modern 64­bit CPU with minimal
performance overhead
●
Write programs in restricted C and compile into BPF with
GCC/LLVM
●
Guarantee termination and safety of BPF program in kernel
with simple algorithm

21. 03/09/2016 21
cBPF vs eBPF
BPF eBPF
registers A, X R0 ­ R10
width 32 bit 64 bit
opcode op:16, jt:8, jf:8, k:32 op:8, dst_reg:4, src_reg:4, off:16, imm:32
JIT support
x86_64, SPARC,
PowerPC, ARM,
ARM64, MIPS and
s390
x86­64, aarch64, s390x

22. 03/09/2016 22
BPF Calling Convention
●
R0
●
Return value from in­kernel function, and exit value for eBPF
program
●
R1 – R5
●
Arguments from eBPF program to in­kernel function
●
R6 – R9
●
Callee saved registers that in­kernel function will preserve
●
R10
●
Read­only frame pointer to access stack

23. 03/09/2016 23
Designed to be JITed
for 64­bit Architecture
/* restore ctx for next call */
bpf_mov R6, R1x
bpf_mov R2, 2
bpf_mov R3, 3
bpf_mov R4, 4
bpf_mov R5, 5
bpf_call foo
/* save foo() return value */
bpf_mov R7, R0
/* restore ctx for next call */
bpf_mov R1, R6
bpf_mov R2, 6
bpf_mov R3, 7
bpf_mov R4, 8
bpf_mov R5, 9
bpf_call bar
bpf_add R0, R7
bpf_exit
push %rbp
mov %rsp,%rbp
sub $0x228,%rsp
mov %rbx,­0x228(%rbp)
mov %r13,­0x220(%rbp)
mov %rdi,%rbx
mov $0x2,%esi
mov $0x3,%edx
mov $0x4,%ecx
mov $0x5,%r8d
callq foo
mov %rax,%r13
mov %rbx,%rdi
mov $0x2,%esi
mov $0x3,%edx
mov $0x4,%ecx
mov $0x5,%r8d
callq bar
add %r13,%rax
mov ­0x228(%rbp),%rbx
mov ­0x220(%rbp),%r13
leaveq
retq
x86_64

24. 03/09/2016 24
How does it work?

25. 03/09/2016 25
BPF Internals (1)
subsys
BPF
binary
kernel
user
app
BPF VM

26. 03/09/2016 26
BPF Internals (2)
BPF
binary
subsys
BPF
binary
kernel
user
Interpreter
JIT
bpf syscall
BPF_PROG_LOAD
app

27. 03/09/2016 27
BPF Internals (3)
BPF
binary
subsys
BPF
binary
kernel
user
Interpreter
JIT
bpf syscall
verifier
app

28. 03/09/2016 28
BPF Verifier
●
Do static check in verifier as possible
●
Directed Acyclic Graph(DAG) program
– Max 4096 instructions
– No loop
– unreachable insns exist
●
Instruction walk
– Read a never­written register
– Do arithmetic of two valid pointer
– Load/store registers of invalid types
– Read stack before writing data into

29. 03/09/2016 29
BPF Internals (4)
BPF
binary
MAP
subsys
BPF
binary
kernel
user
Interpreter
JIT
bpf syscall
verifier
BPF_MAP_CREATE
BPF_MAP_LOOKUP_ELEM
BPF_MAP_UPDATE_ELEM
….
app

30. 03/09/2016 30
BPF MAP
●
BPF_MAP_TYPE_HASH
●
BPF_MAP_TYPE_ARRAY
●
BPF_MAP_TYPE_PROG_ARRAY
●
BPF_MAP_TYPE_PERF_EVENT_ARRAY
map1 map2 map3
Tracing
prog_1
sock
prog_3
Tracing
prog_2
sk_buff on
eth0
Tracepoint
Event C
Tracepoint
Event B
Tracepoint
Event A

31. 03/09/2016 31
BPF Internals (5)
BPF
binary
MAP
subsys
BPF
binary
kernel
user
Interpreter
JIT
bpf syscall
verifier
BPF_PROG_RUN
app

32. 03/09/2016 32
BPF Internals (6)
BPF
binary
MAP
helper
subsys
Other
subsys
BPF_PROG_RUN
BPF
binary
kernel
user
Interpreter/ JIT
bpf syscall
verifier
app

33. 03/09/2016 33
BPF Helpers
map netsystem
perf trace
●
bpf_func_id

34. 03/09/2016 34
BPF Internals (7)
BPF
binary
MAP
helper
subsys
Other
subsys
BPF_PROG_RUN
BPF
binary
kernel
user
Interpreter/JIT
bpf syscall
verifier
app

35. 03/09/2016 35
Kernel Instrumentation

36. 03/09/2016 36
Dynamic Probe
Kernel
user
Kprobe
Kretprobe
Jprobe
Uprobe

37. 03/09/2016 37
Kprobe
INST BREAK
register_kprobe()
pre_handler()
post_handler()
address
sym + offset
Write kernel module
to register a kprobe

38. 03/09/2016 38
Kprobe
BREAKBREAK INST
pre_handler()
post_handler()
exception
address
Note: More details are not revealed

39. 03/09/2016 39
Kprobe­based Event Tracing
# echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/tracing/kprobe_events
# echo 1 > /sys/kernel/tracing/events/kprobes/myretprobe/enable
# cat /sys/kernel/tracing/trace
# tracer: nop
#
# TASK­PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
sh­746 [000] d... 40.96: myretprobe: (SyS_open+0x2c/0x30 <­ do_sys_open) arg1=0x3
sh­746 [000] d... 42.19: myretprobe: (SyS_open+0x2c/0x30 <­ do_sys_open) arg1=0x3
…..

40. 03/09/2016 40
Uprobe
echo 'p:myapp /bin/bash:0x4245c0' > /sys/kernel/tracing/uprobe_events
●
Linux­3.5
●
userspace breakpoints in kernel

41. 03/09/2016 41
User Tools for Kprobe
●
tracefs files
●
systemtap

42. 03/09/2016 42
ftrace
●
Linux­2.6.27
●
Linux kernel internal tracer

43. 03/09/2016 43
ftrace Interface
tracefs (debugfs in past)
README
available_events
available_filter_functions
available_tracers
buffer_size_kb
buffer_total_size_kb
current_tracer
dyn_ftrace_total_info
enabled_functions
events
free_buffer
instances
kprobe_events
kprobe_profile
max_graph_depth
options
per_cpu
printk_formats
saved_cmdlines
saved_cmdlines_size
set_event
set_event_pid
set_ftrace_filter
set_ftrace_notrace
set_ftrace_pid
set_graph_function
set_graph_notrace
trace
trace_clock
trace_marker
trace_options
trace_pipe
tracing_cpumask
tracing_on
tracing_thresh
$ ls /sys/kernel/tracing

44. 03/09/2016 44
ftrace Function Tracer
void Func ( … )
{
Line 1;
Line 2;
…
}
void Func ( … )
{
mcount (pc, ra);
Line 1;
Line 2;
…
}
gcc ­pg

45. 03/09/2016 45
Dynamic Function Tracer
Function trace enabled
on Func()
void Func ( … )
{
nop;
Line 1;
Line 2;
…
}
void Func ( … )
{
mcount (pc, ra);
Line 1;
Line 2;
…
}
Function trace disabled
on Func()

46. 03/09/2016 46
Tracepoint
#include <trace/events/subsys.h>
DEFINE_TRACE(subsys_eventname);
void somefct(void)
{
...
trace_subsys_eventname(arg, task);
...
}
DECLARE_TRACE( subsys_eventname,
TP_PROTO(int firstarg, struct task_struct *p),
TP_ARGS(firstarg, p));
include/trace/events/subsys.h
subsys/file.c

47. 03/09/2016 47
perf
Statistics data
$ perf stat my­app args
Sampling record
$ perf record my­app args
perf­tool
perf framework
kernel
user
HW
event
perf_event
SW
event
PMU
trace
event
trace
point
dynamic
event
kprobe
uprobe

48. 03/09/2016 48
Summary of Kernel Tracing
http://www.slideshare.net/brendangregg/linux-systems-performance-2016

49. 03/09/2016 49
https://i.ytimg.com/vi/elc3FdKxaOk/maxresdefault.jpg
Before BPF Integration
Complex filters and scripts can be expensive
Components are isolated

50. 03/09/2016 50
People desire more powerful tool
like dtrace
Some attemptation: systemtap, ktap

51. 03/09/2016 51
Linux­4.1
“One of the more interesting features in this cycle is the
ability to attach eBPF programs (user­defined, sandboxed
bytecode executed by the kernel) to kprobes. This allows
user­defined instrumentation on a live kernel image that
can never crash, hang or interfere with the kernel
negatively. “
~Ingo Molnár
https://lkml.org/lkml/2015/4/14/232

52. 03/09/2016 52
Instrument powered by eBPF
“If DTrace is Kixy Hawk, eBPF is a jet engine”
~ Brendan Gregg
http://www.ait.org.tw/infousa/zhtw/american_story/assets/es/nc/es_nc_kttyhwk_1_e.jpg

53. 03/09/2016 53
Attach to Kprobe
as well as tracepoint
By Alexei Starovoitov
– tracing: attach BPF programs to kprobes
– tracing: allow BPF programs to call bpf_ktime_get_ns()
– tracing: allow BPF programs to call bpf_trace_printk()
prog_fd = bpf_prog_load(...);
struct perf_event_attr attr = {
.type = PERF_TYPE_TRACEPOINT,
.config = event_id, /* ID of just created kprobe event */
};
event_fd = perf_event_open(&attr,...);
ioctl(event_fd, PERF_EVENT_IOC_SET_BPF, prog_fd);

54. 03/09/2016 54
BPF for Tracing
●
The output data is not limited to PMU counters but data like
time latencies, cache misses or other things users want to
record.
http://www.slideshare.net/brendangregg/linux-bpf-superpowers

55. 03/09/2016 55
Ftrace Filter Interpreter on eBPF
(not merged yet?)
"field1 == 1 || field2 == 2"

56. 03/09/2016 56
The Evolution of
eBPF Userspace Utilities
http://www.bitrebels.com/wp-content/uploads/2011/04/Evolution-Of-Man-Parodies-333.jpg

57. 03/09/2016 57
Program on eBPF
Restrict C
BPF Binary
LLVM
( up 3.7)
userspace
program
eBPF
assembly
or
Kernel

58. 03/09/2016 58
Write a eBPF Program in C Looks Good.
But,
What's the rule of “restrict C” ?

59. 03/09/2016 59
Restrict C [9]
●
No support for
– Global variables
– Arbitrary function calls,
– Floating point, varargs, exceptions, indirect jumps, arbitrary
pointer arithmetic, alloca, etc.
●
Kernel rejects all programs that it cannot prove safe
– programs with loops
– with memory accesses via arbitrary pointers.

60. 03/09/2016 60
BPF Utilities 1:
Kernel Samples
foo_user.c + foo_kern.c
All prog/data needed
when loading bpf
●
bpf programs
●
map
●
license
●
… etc
Userspace
●
Load BPF
●
Cretae maps
●
Flow control
●
Data presentaion

61. 03/09/2016 61
foo_kern.c
struct bpf_map_def SEC("maps") my_map = {
.type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
.max_entries = 32, ….
};
SEC("kprobe/sys_write")
int bpf_prog1(struct pt_regs *ctx)
{
u64 count;
u32 key = bpf_get_smp_processor_id();
char fmt[] = "CPU­%d %llun";
count = bpf_perf_event_read(&my_map, key);
bpf_trace_printk(fmt, sizeof(fmt), key, count);
return 0;
}
u32 _version SEC("version") = LINUX_VERSION_CODE;
BPF
programs
MAPs
Others

62. 03/09/2016 62
foo_user.c
Take kprobe as example
map 1
map 2
bpf_prog1
bpf_prog2
bpf_prog3
version
sec(“maps”)
sec(“kprobe/prog1”)
sec(“kprobe/prog2”)
sec(“kprobe/prog3”)
sec(“version”)
foo_kern.c foo_kern.o
(elf)
clang
­­target=bpf
Create map
(maps section)
Load bpf_progx
(kprobe/xxx, license,
… sections)
Setup
/sys/.../krpobe_events
(kprobe/xxx sections)
libbpf
foo_user.c
bpf_prog_load

63. 03/09/2016 63
BPF Utilities 2:
BCC in IOVisor
The project enables developers to build, innovate, and
share open, programmable data plane with dynamic IO and
networking functions
https://www.iovisor.org/sites
/cpstandard/files/pages/image
s/io_visor.jpg

64. 03/09/2016 64
BPF Compiler Collection
Frontend
python, lua
llvm library
BPF bytecode
libbcc.so
BPF C text/code
BCC module
BCC
bpf syscallperf event / trace_fs
User
program

65. 03/09/2016 65
BPF_HASH(start, struct request *);
void trace_start(struct pt_regs *ctx, struct request *req) {
…...
}
void trace_completion(struct pt_regs *ctx, struct request *req) {
u64 *tsp, delta;
tsp = start.lookup(&req);
if (tsp != 0) {
delta = bpf_ktime_get_ns() ­ *tsp;
bpf_trace_printk("%d %x %dn", req­>__data_len,
req­>cmd_flags, delta / 1000);
start.delete(&req);
}
}
BCC Example: BPF c Program
Simpler than kernel samples

66. 03/09/2016 66
BCC Example: Python Frontend
from bcc import BPF
b = BPF (src_file="disksnoop.c")
b.attach_kprobe (event="blk_start_request", fn_name="trace_start")
b.attach_kprobe (event="blk_mq_start_request", fn_name="trace_start")
b.attach_kprobe (event="blk_account_io_completion",
fn_name="trace_completion")
…....
while 1:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
…....
print("%­18.9f %­2s %­7s %8.2f" % (ts, type_s, bytes_s, ms))

67. 03/09/2016 67
Current Tracing Scripts
in BCC
https://raw.githubusercontent.com/iovisor/bcc/master/images/bcc_tracing_tools_2016.png
Tools for BPF­based Linux IO analysis, networking, monitoring, and
more

68. 03/09/2016 68
BPF Utilities 3:
perf tools
$ perf bpf record --object sample_bpf.o -- -a sleep 4
●
Introduced by Wang Nan

69. 03/09/2016 69
Summary
●
eBPF: In­kernel VM designed to be JITed
●
Used by many subsystems as a filtering engine
– Packet monitor filtering
– Tracing and perf
– Seccomp
– Networking
●
Tools
– BCC
●
Easy to customized script for probe kernel
●
Kernel >=4.1, LLVM >= 3.7
– perf

70. 03/09/2016 70
Other Topics:
How to use in embedded system?

71. 03/09/2016 71
Other Topics:
Linux­4.7: hist trigger
Another mechanism other than eBPF
http://www.brendangregg.com/blog/2016­06­08/linux­hist­triggers.html

72. 03/09/2016 72
Q & A

73. 9/3/16 73/75
Reference
[1] Alexei Starovoitov (May. 2014), “tracing: accelerate tracing filters with BPF”, KERNEL
PATCH
[2] Alexei Starovoitov, (Feb. 2015), "BPF – in-kernel virtual machine", presented at
Collaboration Summit 2015
[3] Brendan Gregg, (Feb. 2016), "Linux 4.x Performance Using BPF Superpowers ",
presented at Performance@ scale 2016
[4] Elena Zannoni (Jun. 2015), “New (and Exciting!) Developments in Linux Tracing ”,
presented at Linuxcon Japan 2015
[5] Gary Lin (Mar. 2016), “eBPF: Trace from Kernel to Userspace ”, presented at OpenSUSE
Technology Sharing Day 2016
[6] Jonathan Corbet. (May. 2014), “BPF: the universal in-kernel virtual machine ”, LWN
[7] Kernel documentation, “Using the Linux Kernel Tracepoints”
[8] Suchakrapani D. Sharma (Dec. 2014), “Towards Faster Trace Filtersvusing eBPF and JIT ”
[9] Michael Larabel, (Jan. 2015), “
BPF Backend Merged Into LLVM To Make Use Of New Kernel Functionality ”, Phoronix

74. 9/3/16 74/75
● HCSM is the community of Hsinchu Coders in Taiwan.
● iovisor is a project of Linux Foundation
● ARM are trademarks or registered trademarks of ARM Holdings.
● Linux Foundation is a registered trademark of The Linux Foundation.
● Linux is a registered trademark of Linus Torvalds.
● Other company, product, and service names may be trademarks or service marks
of others.
● The license of each graph belongs to each website listed individually.
● The others of my work in the slide is licensed under a CC-BY-SA License.
● License text: http://creativecommons.org/licenses/by-sa/4.0/legalcode
Rights to Copy
copyright © 2016 Viller Hsiao

75. 9/3/16 Viller Hsiao
THE END