SlideShare a Scribd company logo
# readahead.bt
Attaching 5 probes...
^C
Readahead unused pages: 128
Readahead used page age (ms):
@age_ms:
[1] 2455 |@@@@@@@@@@@@@@@ |
[2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[32, 64) 0 | |
[64, 128) 0 | |
[128, 256) 384 |@@ |
BPF
Observability
Brendan Gregg
LSFMM
Apr 2019
eBPF
Superpowers Demo
eBPF: extended Berkeley Packet Filter
Kernel
kprobes
uprobes
tracepoints
sockets
SDN Configuration
User-Defined BPF Programs
…
Event TargetsRuntime
perf_events
BPF
actions
BPF
verifier
DDoS Mitigation
Intrusion Detection
Container Security
Observability
Firewalls (bpfilter)
Device Drivers
>150k AWS EC2 server instances
~34% US Internet traffic at night
>130M members
Performance is customer satisfaction & Netflix cost
Experience: ps(1) failure
(This is from last week...)
Experience: ps(1) failure
# wait for $pid to finish:
while ps -p $pid >/dev/null; do
sleep 1
done
# do stuff...
Experience: ps(1) failure
Problem: ps(1) sometimes fails to find the process
Hypothesis: kernel bug!
# wait for $pid to finish:
while ps -p $pid >/dev/null; do
sleep 1
done
# do stuff...
Experience: ps(1) failure
# bpftrace -e 't:syscalls:sys_exit_* /comm == "ps"/ {
@[probe, args->ret > 0 ? 0 : - args->ret] = count(); }'
Attaching 316 probes...
[...]
@[tracepoint:syscalls:sys_exit_openat, 2]: 120
@[tracepoint:syscalls:sys_exit_newfstat, 0]: 180
@[tracepoint:syscalls:sys_exit_mprotect, 0]: 230
@[tracepoint:syscalls:sys_exit_rt_sigaction, 0]: 240
@[tracepoint:syscalls:sys_exit_mmap, 0]: 350
@[tracepoint:syscalls:sys_exit_newstat, 0]: 5000
@[tracepoint:syscalls:sys_exit_read, 0]: 10170
@[tracepoint:syscalls:sys_exit_close, 0]: 10190
@[tracepoint:syscalls:sys_exit_openat, 0]: 10190
Which syscall is abnormally failing (without strace(1) )?
Experience: ps(1) failure
# bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ {
@[args->id, args->ret > 0 ? 0 : - args->ret] = count(); }'
Attaching 1 probe...
[...]
@[21, 2]: 120
@[5, 0]: 180
@[10, 0]: 230
@[13, 0]: 240
@[9, 0]: 350
@[4, 0]: 5000
@[0, 0]: 10170
@[3, 0]: 10190
@[257, 0]: 10190
Which syscall is abnormally failing (without multi-probe)?
Experience: ps(1) failure
# bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ {
@[ksym(*(kaddr("sys_call_table") + args->id * 8)),
args->ret > 0 ? 0 : - args->ret] = count(); }'
[...]
@[sys_brk, 0]: 8202
@[sys_ioctl, 25]: 8203
@[sys_access, 2]: 32808
@[SyS_openat, 2]: 32808
@[sys_newfstat, 0]: 49213
@[sys_newstat, 2]: 60820
@[sys_mprotect, 0]: 62882
[...]
Which syscall is abnormally failing (without multi-probe)?
caught 1 extra failure
ioctl() was a dead end
Experience: ps(1) failure
# bpftrace -e 't:syscalls:sys_exit_getdents /comm == "ps"/ {
printf("ret: %dn", args->ret); }'
[...]
ret: 9192
ret: 0
ret: 9216
ret: 0
ret: 9168
ret: 0
ret: 5640
ret: 0
^C
Which syscall is successfully failing?
Experience: ps(1) failure
# bpftrace -e 't:syscalls:sys_enter_getdents /comm == "ps"/ {
@start[tid] = nsecs; }
t:syscalls:sys_exit_getdents /@start[tid]/ {
printf("%8d us, ret: %dn", (nsecs - @start[tid]) / 1000,
args->ret); delete(@start[tid]); }'
[...]
559 us, ret: 9640
3 us, ret: 0
516 us, ret: 9576
3 us, ret: 0
373 us, ret: 7720
2 us, ret: 0
^C
Which syscall is successfully failing?
Experience: ps(1) failure
# funccount '*proc*'
Tracing "*proc*"... Ctrl-C to end.^C
FUNC COUNT
[…]
proc_readdir 1492
proc_readdir_de 1492
proc_root_getattr 1492
process_measurement 1669
kick_process 1671
wake_up_process 2188
proc_pid_readdir 2984
proc_root_readdir 2984
proc_fill_cache 977263
/proc debugging
Experience: ps(1) failure
# bpftrace -e 'kr:proc_fill_cache /comm == "ps"/ {
@[retval] = count(); }'
Some quick dead ends
# bpftrace -e 'kr:nr_processes /comm == "ps"/ {
printf("%dn", retval); }'
# bpftrace -e 'kr:proc_readdir_de /comm == "ps"/ {
printf("%dn", retval); }'
# bpftrace -e 'kr:proc_root_readdir /comm == "ps"/ {
printf("%dn", retval); }'
Note: this is all in production
Experience: ps(1) failure
Getting closer to the cause
15020
15281
15323
15414
15746
15773
15778
# bpftrace -e 'k:find_ge_pid /comm == "ps"/ { printf("%dn", arg0);
}'
30707
31546
31913
31944
31945
31946
32070
success failure
Experience: ps(1) failure
# bpftrace -e 'k:find_ge_pid /comm == "ps"/ { @nr[tid] = arg0; }
kr:find_ge_pid /@nr[tid]/ {
printf("%d: %llxn", @nr[tid], retval); delete(@nr[tid]); }'
[…]
15561: ffff8a3ee70ad280
15564: ffff8a400244bb80
15569: ffff8a3f6f1a1840
15570: ffff8a3ffe890c00
15571: ffff8a3ffd23bdc0
15575: ffff8a40024fdd80
15576: 0
find_ge_pid() entry argument & return value:
Experience: ps(1) failure
struct pid *find_ge_pid(int nr, struct pid_namespace *ns)
{
return idr_get_next(&ns->idr, &nr);
}
[…]
void *idr_get_next(struct idr *idr, int *nextid)
{
[…]
slot = radix_tree_iter_find(&idr->idr_rt, &iter, id);
Subject [RFC 2/2] pid: Replace PID bitmap implementation with IDR API
Date Sat, 9 Sep 2017 18:03:17 +0530
[…]
Kernel source:
Experience: ps(1) failure
So far we have moved from:
To:
I’ll keep digging after this keynote
ps(1) sometimes fails. Kernel bug!
find_ge_pid() sometimes returns NULL
instead of the next struct *pid
Takeaway:
BPF enables better bug reports
bpftrace: BPF observability front-end
Linux 4.9+ https://github.com/iovisor/bpftrace
# Files opened by process
bpftrace -e 't:syscalls:sys_enter_open { printf("%s %sn", comm,
str(args->filename)) }'
# Read size distribution by process
bpftrace -e 't:syscalls:sys_exit_read { @[comm] = hist(args->ret) }'
# Count VFS calls
bpftrace -e 'kprobe:vfs_* { @[func]++ }'
# Show vfs_read latency as a histogram
bpftrace -e 'k:vfs_read { @[tid] = nsecs }
kr:vfs_read /@[tid]/ { @ns = hist(nsecs - @[tid]); delete(@tid) }’
# Trace user-level function
bpftrace -e 'uretprobe:bash:readline { printf(“%sn”, str(retval)) }’
…
Raw BPF
samples/bpf/sock_example.c
87 lines truncated
C/BPF
samples/bpf/tracex1_kern.c
58 lines truncated
bcc/BPF (C & Python)
bcc examples/tracing/bitehist.py
entire program
bpftrace/BPF
https://github.com/iovisor/bpftrace
entire program
bpftrace -e 'kr:vfs_read { @ = hist(retval); }'
The Tracing Landscape, Apr 2019
Scope & Capability
Easeofuse
sysdig
perf
ftrace
C/BPF
stap
Stage of
Development
(my opinion)
(brutal)(lessbrutal)
(alpha) (mature)
bcc/BPF
ply/BPF
Raw BPF
LTTng
(hist triggers)
recent changes
(many)
bpftrace
(eBPF)
(0.9)
Experience: readahead
Experience: readahead
Is readahead polluting the cache?
Experience: readahead
# readahead.bt
Attaching 5 probes...
^C
Readahead unused pages: 128
Readahead used page age (ms):
@age_ms:
[1] 2455 |@@@@@@@@@@@@@@@ |
[2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[32, 64) 0 | |
[64, 128) 0 | |
[128, 256) 384 |@@ |
Is readahead polluting the cache?
#!/usr/local/bin/bpftrace
kprobe:__do_page_cache_readahead { @in_readahead[tid] = 1; }
kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
kretprobe:__page_cache_alloc
/@in_readahead[tid]/
{
@birth[retval] = nsecs;
@rapages++;
}
kprobe:mark_page_accessed
/@birth[arg0]/
{
@age_ms = hist((nsecs - @birth[arg0]) / 1000000);
delete(@birth[arg0]);
@rapages--;
}
END
{
printf("nReadahead unused pages: %dn", @rapages);
printf("nReadahead used page age (ms):n");
print(@age_ms); clear(@age_ms);
clear(@birth); clear(@in_readahead); clear(@rapages);
}
Takeaway:
bpftrace is good for short tools
bpftrace Syntax
bpftrace -e ‘k:do_nanosleep /pid > 100/ { @[comm]++ }’
Probe
Filter
(optional)
Action
Probes
Probe Type Shortcuts
tracepoint t Kernel static tracepoints
usdt U User-level statically defined tracing
kprobe k Kernel function tracing
kretprobe kr Kernel function returns
uprobe u User-level function tracing
uretprobe ur User-level function returns
profile p Timed sampling across all CPUs
interval i Interval output
software s Kernel software events
hardware h Processor hardware events
Filters
● /pid == 181/
● /comm != “sshd”/
● /@ts[tid]/
Actions
●
Per-event output
– printf()
– system()
– join()
– time()
●
Map Summaries
– @ = count() or @++
– @ = hist()
– …
The following is in the https://github.com/iovisor/bpftrace/blob/master/docs/reference_
Functions
●
hist(n) Log2 histogram
●
lhist(n, min, max, step) Linear hist.
●
count() Count events
●
sum(n) Sum value
●
min(n) Minimum value
●
max(n) Maximum value
●
avg(n) Average value
●
stats(n) Statistics
●
str(s) String
●
ksym(p) Resolve kernel addr
●
usym(p) Resolve user addr
●
kaddr(n) Resolve kernel symbol
●
uaddr(n) Resolve user symbol
●
printf(fmt, ...) Print formatted
●
print(@x[, top[, div]]) Print map
●
delete(@x) Delete map element
●
clear(@x) Delete all keys/values
●
reg(n) Register lookup
●
join(a) Join string array
●
time(fmt) Print formatted time
●
system(fmt) Run shell command
●
cat(file) Print file contents
●
exit() Quit bpftrace
Variable Types
●
Basic Variables
– @global
– @thread_local[tid]
– $scratch
●
Associative Arrays
– @array[key] = value
●
Buitins
– pid
– ...
Builtin Variables
● pid Process ID (kernel tgid)
● tid Thread ID (kernel pid)
● cgroup Current Cgroup ID
● uid User ID
● gid Group ID
● nsecs Nanosecond timestamp
● cpu Processor ID
● comm Process name
● kstack Kernel stack trace
● ustack User stack trace
● arg0, arg1, ... Function args
● retval Return value
● args Tracepoint args
● func Function name
● probe Full probe name
● curtask Curr task_struct (u64)
● rand Random number (u32)
bpftrace: biolatency
#!/usr/local/bin/bpftrace
BEGIN
{
printf("Tracing block device I/O... Hit Ctrl-C to end.n");
}
kprobe:blk_account_io_start
{
@start[arg0] = nsecs;
}
kprobe:blk_account_io_completion
/@start[arg0]/
{
@usecs = hist((nsecs - @start[arg0]) / 1000);
delete(@start[arg0]);
}
Experience: superping!
Experience: superping
# ping 172.20.0.1
PING 172.20.0.1 (172.20.0.1) 56(84) bytes of data.
64 bytes from 172.20.0.1: icmp_seq=1 ttl=64 time=2.87 ms
64 bytes from 172.20.0.1: icmp_seq=2 ttl=64 time=1.66 ms
64 bytes from 172.20.0.1: icmp_seq=3 ttl=64 time=1.55 ms
64 bytes from 172.20.0.1: icmp_seq=4 ttl=64 time=1.11 ms
64 bytes from 172.20.0.1: icmp_seq=5 ttl=64 time=2.48 ms
64 bytes from 172.20.0.1: icmp_seq=6 ttl=64 time=2.39 ms
[...]
How much is scheduler latency?
Experience: superping
# ./superping.bt
Attaching 6 probes...
Tracing ICMP echo request latency. Hit Ctrl-C to end.
IPv4 ping, ID 9827 seq 1: 2883 us
IPv4 ping, ID 9827 seq 2: 1682 us
IPv4 ping, ID 9827 seq 3: 1568 us
IPv4 ping, ID 9827 seq 4: 1078 us
IPv4 ping, ID 9827 seq 5: 2486 us
IPv4 ping, ID 9827 seq 6: 2394 us
[...]
How much is scheduler latency?
?!
#!/usr/local/bin/bpftrace
#include <linux/skbuff.h>
#include <linux/icmp.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/in.h>
BEGIN { printf("Tracing ICMP echo request latency. Hit Ctrl-C to end.n"); }
kprobe:ip_send_skb
{
$skb = (struct sk_buff *)arg1;
// get IPv4 header; see skb_network_header():
$iph = (struct iphdr *)($skb->head + $skb->network_header);
if ($iph->protocol == IPPROTO_ICMP) {
// get ICMP header; see skb_transport_header():
$icmph = (struct icmphdr *)($skb->head + $skb->transport_header);
if ($icmph->type == ICMP_ECHO) {
$id = $icmph->un.echo.id;
$seq = $icmph->un.echo.sequence;
@start[$id, $seq] = nsecs;
}
}
}
[...]
Note: no debuginfo required
Takeaway:
BPF tracing can walk structs
bpftrace Internals
Major Features (v1)
Known Bug Fixes
Packaging
API Stability Stable Docs
Oct 2018
v0.90
Mar-2019
v1.0
?2019Dec 2016
More Bug Fixes
v0.80
Jan-2019
Minor Features (v1) ...
bpftrace Development
v0.10
?Jun-2019
bpftrace Tools Linux 4.9+
https://github.com/iovisor/
Experience: tcplife
Experience: tcplife
Which processes are connecting to which port?
Experience: tcplife
# ./tcplife
PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS
22597 recordProg 127.0.0.1 46644 127.0.0.1 28527 0 0 0.23
3277 redis-serv 127.0.0.1 28527 127.0.0.1 46644 0 0 0.28
22598 curl 100.66.3.172 61620 52.205.89.26 80 0 1 91.79
22604 curl 100.66.3.172 44400 52.204.43.121 80 0 1 121.38
22624 recordProg 127.0.0.1 46648 127.0.0.1 28527 0 0 0.22
3277 redis-serv 127.0.0.1 28527 127.0.0.1 46648 0 0 0.27
22647 recordProg 127.0.0.1 46650 127.0.0.1 28527 0 0 0.21
3277 redis-serv 127.0.0.1 28527 127.0.0.1 46650 0 0 0.26
[...]
Which processes are connecting to which port?
bcc Linux 4.4+
https://github.com/iovisor/bcc
int kprobe__tcp_set_state(struct pt_regs *ctx, struct sock *sk, int state)
{
u32 pid = bpf_get_current_pid_tgid() >> 32;
// lport is either used in a filter here, or later
u16 lport = sk->__sk_common.skc_num;
[…]
struct tcp_sock *tp = (struct tcp_sock *)sk;
rx_b = tp->bytes_received;
tx_b = tp->bytes_acked;
u16 family = sk->__sk_common.skc_family;
if (family == AF_INET) {
struct ipv4_data_t data4 = {};
data4.span_us = delta_us;
data4.rx_b = rx_b;
data4.tx_b = tx_b;
data4.ts_us = bpf_ktime_get_ns() / 1000;
data4.saddr = sk->__sk_common.skc_rcv_saddr;
data4.daddr = sk->__sk_common.skc_daddr;
[…]
bcc: tcplife
Experience: tcplife
# bpftrace -lv t:tcp:tcp_set_state
tracepoint:tcp:tcp_set_state
const void * skaddr;
int oldstate;
int newstate;
__u16 sport;
__u16 dport;
__u8 saddr[4];
__u8 daddr[4];
__u8 saddr_v6[16];
__u8 daddr_v6[16];
From kprobes to tracepoints
# bpftrace -lv t:sock:inet_sock_set_state
tracepoint:sock:inet_sock_set_state
const void * skaddr;
int oldstate;
int newstate;
__u16 sport;
__u16 dport;
__u16 family;
__u8 protocol;
__u8 saddr[4];
__u8 daddr[4];
__u8 saddr_v6[16];
__u8 daddr_v6[16];Linux 4.15
Linux 4.16+
Takeaways:
bcc for complex tools
kprobes can prototype tracepoints
tracepoints can change (best effort)
Experience: cachestat
Experience: cachestat
What is the page cache hit ratio?
Experience: cachestat
# cachestat
HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
1132 0 4 100.00% 277 4367
161 0 36 100.00% 277 4372
16 0 28 100.00% 277 4372
17154 13750 15 55.51% 277 4422
19 0 1 100.00% 277 4422
83 0 83 100.00% 277 4421
16 0 1 100.00% 277 4423
[...]
What is the page cache hit ratio?
b.attach_kprobe(event="add_to_page_cache_lru", fn_name="do_count")
b.attach_kprobe(event="mark_page_accessed", fn_name="do_count")
b.attach_kprobe(event="account_page_dirtied", fn_name="do_count")
b.attach_kprobe(event="mark_buffer_dirty", fn_name="do_count")
[…]
# total = total cache accesses without counting dirties
# misses = total of add to lru because of read misses
total = mpa - mbd
misses = apcl - apd
if misses < 0:
misses = 0
if total < 0:
total = 0
hits = total - misses
# If hits are < 0, then its possible misses are overestimated
# due to possibly page cache read ahead adding more pages than
# needed. In this case just assume misses as total and reset hits.
if hits < 0:
misses = total
hits = 0
[…]
This is a sandcastle
Takeaway:
BPF tracing can prototype /proc stats
Many of our perf wins are from CPU flame graphs
not CLI tracing
Reality Check
Java
JVM
Kernel
GC
CPU Flame Graphs
Alphabetical frame sort (A - Z)
Stackdepth(0-max)
BPF-based CPU Flame Graphs
perf record
perf script
stackcollapse-perf.pl
flamegraph.pl
perf.data
flamegraph.pl
profile.py
Linux 4.9Linux 2.6
Takeaway:
BPF all the things!
Take Aways
●
BPF observability:
– bpftrace: one-liners, short scripts
– bcc: complex tools
– Production safe, and no debuginfo needed
●
kprobe tools can prototype tracepoints, /proc stats
●
I’m ok with tracepoints are best effort
●
BPF all the things!
URLs
- https://github.com/iovisor/bpftrace
– https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
– https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md
- https://github.com/iovisor/bcc
– https://github.com/iovisor/bcc/blob/master/docs/tutorial.md
– https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md
- http://www.brendangregg.com/ebpf.html
Update: this keynote was summarized by
https://lwn.net/Articles/787131/
Thanks
●
bpftrace
– Alastair Robertson (creator)
– Netflix: myself, Matheus Marchini
– Sthima: Willian Gaspar
– Facebook: Jon Haslam, Dan Xu
– Augusto Mecking Caringi, Dale Hamel, ...
●
eBPF & bcc
– Facebook: Alexei Starovoitov, Teng Qin, Yonghong Song, Martin Lau,
Mark Drayton, …
– Netflix: myself
– VMware: Brenden Blanco
– Daniel Borkmann, David S. Miller, Sasha Goldsthein, Paul Chaignon, ...

More Related Content

What's hot

Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)
Brendan Gregg
 
Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!
Ray Jenkins
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
Brendan Gregg
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
Brendan Gregg
 
netfilter and iptables
netfilter and iptablesnetfilter and iptables
netfilter and iptables
Kernel TLV
 
Linux kernel tracing
Linux kernel tracingLinux kernel tracing
Linux kernel tracing
Viller Hsiao
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame Graphs
Brendan Gregg
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
Brendan Gregg
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
Brendan Gregg
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and Monitoring
Georg Schönberger
 
eBPF maps 101
eBPF maps 101eBPF maps 101
eBPF maps 101
SUSE Labs Taipei
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
Brendan Gregg
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
Brendan Gregg
 
Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)
Brendan Gregg
 
Staring into the eBPF Abyss
Staring into the eBPF AbyssStaring into the eBPF Abyss
Staring into the eBPF Abyss
Sasha Goldshtein
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
Thomas Graf
 
BPF - in-kernel virtual machine
BPF - in-kernel virtual machineBPF - in-kernel virtual machine
BPF - in-kernel virtual machine
Alexei Starovoitov
 
Accelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelAccelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux Kernel
Thomas Graf
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
Adrien Mahieux
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
Brendan Gregg
 

What's hot (20)

Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)
 
Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
 
netfilter and iptables
netfilter and iptablesnetfilter and iptables
netfilter and iptables
 
Linux kernel tracing
Linux kernel tracingLinux kernel tracing
Linux kernel tracing
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame Graphs
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and Monitoring
 
eBPF maps 101
eBPF maps 101eBPF maps 101
eBPF maps 101
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
 
Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)
 
Staring into the eBPF Abyss
Staring into the eBPF AbyssStaring into the eBPF Abyss
Staring into the eBPF Abyss
 
Linux Networking Explained
Linux Networking ExplainedLinux Networking Explained
Linux Networking Explained
 
BPF - in-kernel virtual machine
BPF - in-kernel virtual machineBPF - in-kernel virtual machine
BPF - in-kernel virtual machine
 
Accelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux KernelAccelerating Envoy and Istio with Cilium and the Linux Kernel
Accelerating Envoy and Istio with Cilium and the Linux Kernel
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
 

Similar to LSFMM 2019 BPF Observability

eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019
Brendan Gregg
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
PROIDEA
 
Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨
flyinweb
 
Global Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the SealGlobal Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the Seal
Tzung-Bi Shih
 
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
Teddy Hsiung
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
Brendan Gregg
 
Linux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudLinux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloud
Andrea Righi
 
Beyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisBeyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic Analysis
Fastly
 
A miało być tak... bez wycieków
A miało być tak... bez wyciekówA miało być tak... bez wycieków
A miało być tak... bez wycieków
Konrad Kokosa
 
C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략 C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략
명신 김
 
Improving go-git performance
Improving go-git performanceImproving go-git performance
Improving go-git performance
source{d}
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernel
lcplcp1
 
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Gavin Guo
 
Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"
Yulia Tsisyk
 
State of the .Net Performance
State of the .Net PerformanceState of the .Net Performance
State of the .Net Performance
CUSTIS
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018
Brendan Gregg
 
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in GoCapturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
ScyllaDB
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
Sasha Goldshtein
 
Quiz 9
Quiz 9Quiz 9
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
Kernel TLV
 

Similar to LSFMM 2019 BPF Observability (20)

eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
 
Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨
 
Global Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the SealGlobal Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the Seal
 
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
 
Linux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudLinux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloud
 
Beyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisBeyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic Analysis
 
A miało być tak... bez wycieków
A miało być tak... bez wyciekówA miało być tak... bez wycieków
A miało być tak... bez wycieków
 
C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략 C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략
 
Improving go-git performance
Improving go-git performanceImproving go-git performance
Improving go-git performance
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernel
 
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
 
Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"
 
State of the .Net Performance
State of the .Net PerformanceState of the .Net Performance
State of the .Net Performance
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018
 
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in GoCapturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
 
Quiz 9
Quiz 9Quiz 9
Quiz 9
 
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
 

More from Brendan Gregg

YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing Performance
Brendan Gregg
 
IntelON 2021 Processor Benchmarking
IntelON 2021 Processor BenchmarkingIntelON 2021 Processor Benchmarking
IntelON 2021 Processor Benchmarking
Brendan Gregg
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
Brendan Gregg
 
LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing Tools
Brendan Gregg
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflix
Brendan Gregg
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
Brendan Gregg
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF Observability
Brendan Gregg
 
FlameScope 2018
FlameScope 2018FlameScope 2018
FlameScope 2018
Brendan Gregg
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)
Brendan Gregg
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for Performance
Brendan Gregg
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
Brendan Gregg
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
Brendan Gregg
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis Methodologies
Brendan Gregg
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
Brendan Gregg
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
Brendan Gregg
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
Brendan Gregg
 

More from Brendan Gregg (16)

YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing Performance
 
IntelON 2021 Processor Benchmarking
IntelON 2021 Processor BenchmarkingIntelON 2021 Processor Benchmarking
IntelON 2021 Processor Benchmarking
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
 
LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing Tools
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflix
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF Observability
 
FlameScope 2018
FlameScope 2018FlameScope 2018
FlameScope 2018
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for Performance
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis Methodologies
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
 

Recently uploaded

Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
DianaGray10
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
Fwdays
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Fwdays
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
ScyllaDB
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 

Recently uploaded (20)

Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 

LSFMM 2019 BPF Observability

  • 1. # readahead.bt Attaching 5 probes... ^C Readahead unused pages: 128 Readahead used page age (ms): @age_ms: [1] 2455 |@@@@@@@@@@@@@@@ | [2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 384 |@@ | BPF Observability Brendan Gregg LSFMM Apr 2019
  • 4. eBPF: extended Berkeley Packet Filter Kernel kprobes uprobes tracepoints sockets SDN Configuration User-Defined BPF Programs … Event TargetsRuntime perf_events BPF actions BPF verifier DDoS Mitigation Intrusion Detection Container Security Observability Firewalls (bpfilter) Device Drivers
  • 5. >150k AWS EC2 server instances ~34% US Internet traffic at night >130M members Performance is customer satisfaction & Netflix cost
  • 6. Experience: ps(1) failure (This is from last week...)
  • 7. Experience: ps(1) failure # wait for $pid to finish: while ps -p $pid >/dev/null; do sleep 1 done # do stuff...
  • 8. Experience: ps(1) failure Problem: ps(1) sometimes fails to find the process Hypothesis: kernel bug! # wait for $pid to finish: while ps -p $pid >/dev/null; do sleep 1 done # do stuff...
  • 9. Experience: ps(1) failure # bpftrace -e 't:syscalls:sys_exit_* /comm == "ps"/ { @[probe, args->ret > 0 ? 0 : - args->ret] = count(); }' Attaching 316 probes... [...] @[tracepoint:syscalls:sys_exit_openat, 2]: 120 @[tracepoint:syscalls:sys_exit_newfstat, 0]: 180 @[tracepoint:syscalls:sys_exit_mprotect, 0]: 230 @[tracepoint:syscalls:sys_exit_rt_sigaction, 0]: 240 @[tracepoint:syscalls:sys_exit_mmap, 0]: 350 @[tracepoint:syscalls:sys_exit_newstat, 0]: 5000 @[tracepoint:syscalls:sys_exit_read, 0]: 10170 @[tracepoint:syscalls:sys_exit_close, 0]: 10190 @[tracepoint:syscalls:sys_exit_openat, 0]: 10190 Which syscall is abnormally failing (without strace(1) )?
  • 10. Experience: ps(1) failure # bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ { @[args->id, args->ret > 0 ? 0 : - args->ret] = count(); }' Attaching 1 probe... [...] @[21, 2]: 120 @[5, 0]: 180 @[10, 0]: 230 @[13, 0]: 240 @[9, 0]: 350 @[4, 0]: 5000 @[0, 0]: 10170 @[3, 0]: 10190 @[257, 0]: 10190 Which syscall is abnormally failing (without multi-probe)?
  • 11. Experience: ps(1) failure # bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ { @[ksym(*(kaddr("sys_call_table") + args->id * 8)), args->ret > 0 ? 0 : - args->ret] = count(); }' [...] @[sys_brk, 0]: 8202 @[sys_ioctl, 25]: 8203 @[sys_access, 2]: 32808 @[SyS_openat, 2]: 32808 @[sys_newfstat, 0]: 49213 @[sys_newstat, 2]: 60820 @[sys_mprotect, 0]: 62882 [...] Which syscall is abnormally failing (without multi-probe)? caught 1 extra failure ioctl() was a dead end
  • 12. Experience: ps(1) failure # bpftrace -e 't:syscalls:sys_exit_getdents /comm == "ps"/ { printf("ret: %dn", args->ret); }' [...] ret: 9192 ret: 0 ret: 9216 ret: 0 ret: 9168 ret: 0 ret: 5640 ret: 0 ^C Which syscall is successfully failing?
  • 13. Experience: ps(1) failure # bpftrace -e 't:syscalls:sys_enter_getdents /comm == "ps"/ { @start[tid] = nsecs; } t:syscalls:sys_exit_getdents /@start[tid]/ { printf("%8d us, ret: %dn", (nsecs - @start[tid]) / 1000, args->ret); delete(@start[tid]); }' [...] 559 us, ret: 9640 3 us, ret: 0 516 us, ret: 9576 3 us, ret: 0 373 us, ret: 7720 2 us, ret: 0 ^C Which syscall is successfully failing?
  • 14. Experience: ps(1) failure # funccount '*proc*' Tracing "*proc*"... Ctrl-C to end.^C FUNC COUNT […] proc_readdir 1492 proc_readdir_de 1492 proc_root_getattr 1492 process_measurement 1669 kick_process 1671 wake_up_process 2188 proc_pid_readdir 2984 proc_root_readdir 2984 proc_fill_cache 977263 /proc debugging
  • 15. Experience: ps(1) failure # bpftrace -e 'kr:proc_fill_cache /comm == "ps"/ { @[retval] = count(); }' Some quick dead ends # bpftrace -e 'kr:nr_processes /comm == "ps"/ { printf("%dn", retval); }' # bpftrace -e 'kr:proc_readdir_de /comm == "ps"/ { printf("%dn", retval); }' # bpftrace -e 'kr:proc_root_readdir /comm == "ps"/ { printf("%dn", retval); }' Note: this is all in production
  • 16. Experience: ps(1) failure Getting closer to the cause 15020 15281 15323 15414 15746 15773 15778 # bpftrace -e 'k:find_ge_pid /comm == "ps"/ { printf("%dn", arg0); }' 30707 31546 31913 31944 31945 31946 32070 success failure
  • 17. Experience: ps(1) failure # bpftrace -e 'k:find_ge_pid /comm == "ps"/ { @nr[tid] = arg0; } kr:find_ge_pid /@nr[tid]/ { printf("%d: %llxn", @nr[tid], retval); delete(@nr[tid]); }' […] 15561: ffff8a3ee70ad280 15564: ffff8a400244bb80 15569: ffff8a3f6f1a1840 15570: ffff8a3ffe890c00 15571: ffff8a3ffd23bdc0 15575: ffff8a40024fdd80 15576: 0 find_ge_pid() entry argument & return value:
  • 18. Experience: ps(1) failure struct pid *find_ge_pid(int nr, struct pid_namespace *ns) { return idr_get_next(&ns->idr, &nr); } […] void *idr_get_next(struct idr *idr, int *nextid) { […] slot = radix_tree_iter_find(&idr->idr_rt, &iter, id); Subject [RFC 2/2] pid: Replace PID bitmap implementation with IDR API Date Sat, 9 Sep 2017 18:03:17 +0530 […] Kernel source:
  • 19. Experience: ps(1) failure So far we have moved from: To: I’ll keep digging after this keynote ps(1) sometimes fails. Kernel bug! find_ge_pid() sometimes returns NULL instead of the next struct *pid
  • 21. bpftrace: BPF observability front-end Linux 4.9+ https://github.com/iovisor/bpftrace # Files opened by process bpftrace -e 't:syscalls:sys_enter_open { printf("%s %sn", comm, str(args->filename)) }' # Read size distribution by process bpftrace -e 't:syscalls:sys_exit_read { @[comm] = hist(args->ret) }' # Count VFS calls bpftrace -e 'kprobe:vfs_* { @[func]++ }' # Show vfs_read latency as a histogram bpftrace -e 'k:vfs_read { @[tid] = nsecs } kr:vfs_read /@[tid]/ { @ns = hist(nsecs - @[tid]); delete(@tid) }’ # Trace user-level function bpftrace -e 'uretprobe:bash:readline { printf(“%sn”, str(retval)) }’ …
  • 24. bcc/BPF (C & Python) bcc examples/tracing/bitehist.py entire program
  • 26. The Tracing Landscape, Apr 2019 Scope & Capability Easeofuse sysdig perf ftrace C/BPF stap Stage of Development (my opinion) (brutal)(lessbrutal) (alpha) (mature) bcc/BPF ply/BPF Raw BPF LTTng (hist triggers) recent changes (many) bpftrace (eBPF) (0.9)
  • 28. Experience: readahead Is readahead polluting the cache?
  • 29. Experience: readahead # readahead.bt Attaching 5 probes... ^C Readahead unused pages: 128 Readahead used page age (ms): @age_ms: [1] 2455 |@@@@@@@@@@@@@@@ | [2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 384 |@@ | Is readahead polluting the cache?
  • 30. #!/usr/local/bin/bpftrace kprobe:__do_page_cache_readahead { @in_readahead[tid] = 1; } kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; } kretprobe:__page_cache_alloc /@in_readahead[tid]/ { @birth[retval] = nsecs; @rapages++; } kprobe:mark_page_accessed /@birth[arg0]/ { @age_ms = hist((nsecs - @birth[arg0]) / 1000000); delete(@birth[arg0]); @rapages--; } END { printf("nReadahead unused pages: %dn", @rapages); printf("nReadahead used page age (ms):n"); print(@age_ms); clear(@age_ms); clear(@birth); clear(@in_readahead); clear(@rapages); }
  • 31. Takeaway: bpftrace is good for short tools
  • 32. bpftrace Syntax bpftrace -e ‘k:do_nanosleep /pid > 100/ { @[comm]++ }’ Probe Filter (optional) Action
  • 34. Probe Type Shortcuts tracepoint t Kernel static tracepoints usdt U User-level statically defined tracing kprobe k Kernel function tracing kretprobe kr Kernel function returns uprobe u User-level function tracing uretprobe ur User-level function returns profile p Timed sampling across all CPUs interval i Interval output software s Kernel software events hardware h Processor hardware events
  • 35. Filters ● /pid == 181/ ● /comm != “sshd”/ ● /@ts[tid]/
  • 36. Actions ● Per-event output – printf() – system() – join() – time() ● Map Summaries – @ = count() or @++ – @ = hist() – … The following is in the https://github.com/iovisor/bpftrace/blob/master/docs/reference_
  • 37. Functions ● hist(n) Log2 histogram ● lhist(n, min, max, step) Linear hist. ● count() Count events ● sum(n) Sum value ● min(n) Minimum value ● max(n) Maximum value ● avg(n) Average value ● stats(n) Statistics ● str(s) String ● ksym(p) Resolve kernel addr ● usym(p) Resolve user addr ● kaddr(n) Resolve kernel symbol ● uaddr(n) Resolve user symbol ● printf(fmt, ...) Print formatted ● print(@x[, top[, div]]) Print map ● delete(@x) Delete map element ● clear(@x) Delete all keys/values ● reg(n) Register lookup ● join(a) Join string array ● time(fmt) Print formatted time ● system(fmt) Run shell command ● cat(file) Print file contents ● exit() Quit bpftrace
  • 38. Variable Types ● Basic Variables – @global – @thread_local[tid] – $scratch ● Associative Arrays – @array[key] = value ● Buitins – pid – ...
  • 39. Builtin Variables ● pid Process ID (kernel tgid) ● tid Thread ID (kernel pid) ● cgroup Current Cgroup ID ● uid User ID ● gid Group ID ● nsecs Nanosecond timestamp ● cpu Processor ID ● comm Process name ● kstack Kernel stack trace ● ustack User stack trace ● arg0, arg1, ... Function args ● retval Return value ● args Tracepoint args ● func Function name ● probe Full probe name ● curtask Curr task_struct (u64) ● rand Random number (u32)
  • 40. bpftrace: biolatency #!/usr/local/bin/bpftrace BEGIN { printf("Tracing block device I/O... Hit Ctrl-C to end.n"); } kprobe:blk_account_io_start { @start[arg0] = nsecs; } kprobe:blk_account_io_completion /@start[arg0]/ { @usecs = hist((nsecs - @start[arg0]) / 1000); delete(@start[arg0]); }
  • 42. Experience: superping # ping 172.20.0.1 PING 172.20.0.1 (172.20.0.1) 56(84) bytes of data. 64 bytes from 172.20.0.1: icmp_seq=1 ttl=64 time=2.87 ms 64 bytes from 172.20.0.1: icmp_seq=2 ttl=64 time=1.66 ms 64 bytes from 172.20.0.1: icmp_seq=3 ttl=64 time=1.55 ms 64 bytes from 172.20.0.1: icmp_seq=4 ttl=64 time=1.11 ms 64 bytes from 172.20.0.1: icmp_seq=5 ttl=64 time=2.48 ms 64 bytes from 172.20.0.1: icmp_seq=6 ttl=64 time=2.39 ms [...] How much is scheduler latency?
  • 43. Experience: superping # ./superping.bt Attaching 6 probes... Tracing ICMP echo request latency. Hit Ctrl-C to end. IPv4 ping, ID 9827 seq 1: 2883 us IPv4 ping, ID 9827 seq 2: 1682 us IPv4 ping, ID 9827 seq 3: 1568 us IPv4 ping, ID 9827 seq 4: 1078 us IPv4 ping, ID 9827 seq 5: 2486 us IPv4 ping, ID 9827 seq 6: 2394 us [...] How much is scheduler latency? ?!
  • 44. #!/usr/local/bin/bpftrace #include <linux/skbuff.h> #include <linux/icmp.h> #include <linux/ip.h> #include <linux/ipv6.h> #include <linux/in.h> BEGIN { printf("Tracing ICMP echo request latency. Hit Ctrl-C to end.n"); } kprobe:ip_send_skb { $skb = (struct sk_buff *)arg1; // get IPv4 header; see skb_network_header(): $iph = (struct iphdr *)($skb->head + $skb->network_header); if ($iph->protocol == IPPROTO_ICMP) { // get ICMP header; see skb_transport_header(): $icmph = (struct icmphdr *)($skb->head + $skb->transport_header); if ($icmph->type == ICMP_ECHO) { $id = $icmph->un.echo.id; $seq = $icmph->un.echo.sequence; @start[$id, $seq] = nsecs; } } } [...] Note: no debuginfo required
  • 47. Major Features (v1) Known Bug Fixes Packaging API Stability Stable Docs Oct 2018 v0.90 Mar-2019 v1.0 ?2019Dec 2016 More Bug Fixes v0.80 Jan-2019 Minor Features (v1) ... bpftrace Development v0.10 ?Jun-2019
  • 48. bpftrace Tools Linux 4.9+ https://github.com/iovisor/
  • 50. Experience: tcplife Which processes are connecting to which port?
  • 51. Experience: tcplife # ./tcplife PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 22597 recordProg 127.0.0.1 46644 127.0.0.1 28527 0 0 0.23 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46644 0 0 0.28 22598 curl 100.66.3.172 61620 52.205.89.26 80 0 1 91.79 22604 curl 100.66.3.172 44400 52.204.43.121 80 0 1 121.38 22624 recordProg 127.0.0.1 46648 127.0.0.1 28527 0 0 0.22 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46648 0 0 0.27 22647 recordProg 127.0.0.1 46650 127.0.0.1 28527 0 0 0.21 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46650 0 0 0.26 [...] Which processes are connecting to which port?
  • 53. int kprobe__tcp_set_state(struct pt_regs *ctx, struct sock *sk, int state) { u32 pid = bpf_get_current_pid_tgid() >> 32; // lport is either used in a filter here, or later u16 lport = sk->__sk_common.skc_num; […] struct tcp_sock *tp = (struct tcp_sock *)sk; rx_b = tp->bytes_received; tx_b = tp->bytes_acked; u16 family = sk->__sk_common.skc_family; if (family == AF_INET) { struct ipv4_data_t data4 = {}; data4.span_us = delta_us; data4.rx_b = rx_b; data4.tx_b = tx_b; data4.ts_us = bpf_ktime_get_ns() / 1000; data4.saddr = sk->__sk_common.skc_rcv_saddr; data4.daddr = sk->__sk_common.skc_daddr; […] bcc: tcplife
  • 54. Experience: tcplife # bpftrace -lv t:tcp:tcp_set_state tracepoint:tcp:tcp_set_state const void * skaddr; int oldstate; int newstate; __u16 sport; __u16 dport; __u8 saddr[4]; __u8 daddr[4]; __u8 saddr_v6[16]; __u8 daddr_v6[16]; From kprobes to tracepoints # bpftrace -lv t:sock:inet_sock_set_state tracepoint:sock:inet_sock_set_state const void * skaddr; int oldstate; int newstate; __u16 sport; __u16 dport; __u16 family; __u8 protocol; __u8 saddr[4]; __u8 daddr[4]; __u8 saddr_v6[16]; __u8 daddr_v6[16];Linux 4.15 Linux 4.16+
  • 55. Takeaways: bcc for complex tools kprobes can prototype tracepoints tracepoints can change (best effort)
  • 57. Experience: cachestat What is the page cache hit ratio?
  • 58. Experience: cachestat # cachestat HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB 1132 0 4 100.00% 277 4367 161 0 36 100.00% 277 4372 16 0 28 100.00% 277 4372 17154 13750 15 55.51% 277 4422 19 0 1 100.00% 277 4422 83 0 83 100.00% 277 4421 16 0 1 100.00% 277 4423 [...] What is the page cache hit ratio?
  • 59. b.attach_kprobe(event="add_to_page_cache_lru", fn_name="do_count") b.attach_kprobe(event="mark_page_accessed", fn_name="do_count") b.attach_kprobe(event="account_page_dirtied", fn_name="do_count") b.attach_kprobe(event="mark_buffer_dirty", fn_name="do_count") […] # total = total cache accesses without counting dirties # misses = total of add to lru because of read misses total = mpa - mbd misses = apcl - apd if misses < 0: misses = 0 if total < 0: total = 0 hits = total - misses # If hits are < 0, then its possible misses are overestimated # due to possibly page cache read ahead adding more pages than # needed. In this case just assume misses as total and reset hits. if hits < 0: misses = total hits = 0 […] This is a sandcastle
  • 60. Takeaway: BPF tracing can prototype /proc stats
  • 61. Many of our perf wins are from CPU flame graphs not CLI tracing Reality Check
  • 62. Java JVM Kernel GC CPU Flame Graphs Alphabetical frame sort (A - Z) Stackdepth(0-max)
  • 63. BPF-based CPU Flame Graphs perf record perf script stackcollapse-perf.pl flamegraph.pl perf.data flamegraph.pl profile.py Linux 4.9Linux 2.6
  • 65. Take Aways ● BPF observability: – bpftrace: one-liners, short scripts – bcc: complex tools – Production safe, and no debuginfo needed ● kprobe tools can prototype tracepoints, /proc stats ● I’m ok with tracepoints are best effort ● BPF all the things!
  • 66. URLs - https://github.com/iovisor/bpftrace – https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md – https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md - https://github.com/iovisor/bcc – https://github.com/iovisor/bcc/blob/master/docs/tutorial.md – https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md - http://www.brendangregg.com/ebpf.html Update: this keynote was summarized by https://lwn.net/Articles/787131/
  • 67. Thanks ● bpftrace – Alastair Robertson (creator) – Netflix: myself, Matheus Marchini – Sthima: Willian Gaspar – Facebook: Jon Haslam, Dan Xu – Augusto Mecking Caringi, Dale Hamel, ... ● eBPF & bcc – Facebook: Alexei Starovoitov, Teng Qin, Yonghong Song, Martin Lau, Mark Drayton, … – Netflix: myself – VMware: Brenden Blanco – Daniel Borkmann, David S. Miller, Sasha Goldsthein, Paul Chaignon, ...