SlideShare a Scribd company logo
1 of 67
Download to read offline
# readahead.bt
Attaching 5 probes...
^C
Readahead unused pages: 128
Readahead used page age (ms):
@age_ms:
[1] 2455 |@@@@@@@@@@@@@@@ |
[2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[32, 64) 0 | |
[64, 128) 0 | |
[128, 256) 384 |@@ |
BPF
Observability
Brendan Gregg
LSFMM
Apr 2019
eBPF
Superpowers Demo
eBPF: extended Berkeley Packet Filter
Kernel
kprobes
uprobes
tracepoints
sockets
SDN Configuration
User-Defined BPF Programs
…
Event TargetsRuntime
perf_events
BPF
actions
BPF
verifier
DDoS Mitigation
Intrusion Detection
Container Security
Observability
Firewalls (bpfilter)
Device Drivers
>150k AWS EC2 server instances
~34% US Internet traffic at night
>130M members
Performance is customer satisfaction & Netflix cost
Experience: ps(1) failure
(This is from last week...)
Experience: ps(1) failure
# wait for $pid to finish:
while ps -p $pid >/dev/null; do
sleep 1
done
# do stuff...
Experience: ps(1) failure
Problem: ps(1) sometimes fails to find the process
Hypothesis: kernel bug!
# wait for $pid to finish:
while ps -p $pid >/dev/null; do
sleep 1
done
# do stuff...
Experience: ps(1) failure
# bpftrace -e 't:syscalls:sys_exit_* /comm == "ps"/ {
@[probe, args->ret > 0 ? 0 : - args->ret] = count(); }'
Attaching 316 probes...
[...]
@[tracepoint:syscalls:sys_exit_openat, 2]: 120
@[tracepoint:syscalls:sys_exit_newfstat, 0]: 180
@[tracepoint:syscalls:sys_exit_mprotect, 0]: 230
@[tracepoint:syscalls:sys_exit_rt_sigaction, 0]: 240
@[tracepoint:syscalls:sys_exit_mmap, 0]: 350
@[tracepoint:syscalls:sys_exit_newstat, 0]: 5000
@[tracepoint:syscalls:sys_exit_read, 0]: 10170
@[tracepoint:syscalls:sys_exit_close, 0]: 10190
@[tracepoint:syscalls:sys_exit_openat, 0]: 10190
Which syscall is abnormally failing (without strace(1) )?
Experience: ps(1) failure
# bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ {
@[args->id, args->ret > 0 ? 0 : - args->ret] = count(); }'
Attaching 1 probe...
[...]
@[21, 2]: 120
@[5, 0]: 180
@[10, 0]: 230
@[13, 0]: 240
@[9, 0]: 350
@[4, 0]: 5000
@[0, 0]: 10170
@[3, 0]: 10190
@[257, 0]: 10190
Which syscall is abnormally failing (without multi-probe)?
Experience: ps(1) failure
# bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ {
@[ksym(*(kaddr("sys_call_table") + args->id * 8)),
args->ret > 0 ? 0 : - args->ret] = count(); }'
[...]
@[sys_brk, 0]: 8202
@[sys_ioctl, 25]: 8203
@[sys_access, 2]: 32808
@[SyS_openat, 2]: 32808
@[sys_newfstat, 0]: 49213
@[sys_newstat, 2]: 60820
@[sys_mprotect, 0]: 62882
[...]
Which syscall is abnormally failing (without multi-probe)?
caught 1 extra failure
ioctl() was a dead end
Experience: ps(1) failure
# bpftrace -e 't:syscalls:sys_exit_getdents /comm == "ps"/ {
printf("ret: %dn", args->ret); }'
[...]
ret: 9192
ret: 0
ret: 9216
ret: 0
ret: 9168
ret: 0
ret: 5640
ret: 0
^C
Which syscall is successfully failing?
Experience: ps(1) failure
# bpftrace -e 't:syscalls:sys_enter_getdents /comm == "ps"/ {
@start[tid] = nsecs; }
t:syscalls:sys_exit_getdents /@start[tid]/ {
printf("%8d us, ret: %dn", (nsecs - @start[tid]) / 1000,
args->ret); delete(@start[tid]); }'
[...]
559 us, ret: 9640
3 us, ret: 0
516 us, ret: 9576
3 us, ret: 0
373 us, ret: 7720
2 us, ret: 0
^C
Which syscall is successfully failing?
Experience: ps(1) failure
# funccount '*proc*'
Tracing "*proc*"... Ctrl-C to end.^C
FUNC COUNT
[…]
proc_readdir 1492
proc_readdir_de 1492
proc_root_getattr 1492
process_measurement 1669
kick_process 1671
wake_up_process 2188
proc_pid_readdir 2984
proc_root_readdir 2984
proc_fill_cache 977263
/proc debugging
Experience: ps(1) failure
# bpftrace -e 'kr:proc_fill_cache /comm == "ps"/ {
@[retval] = count(); }'
Some quick dead ends
# bpftrace -e 'kr:nr_processes /comm == "ps"/ {
printf("%dn", retval); }'
# bpftrace -e 'kr:proc_readdir_de /comm == "ps"/ {
printf("%dn", retval); }'
# bpftrace -e 'kr:proc_root_readdir /comm == "ps"/ {
printf("%dn", retval); }'
Note: this is all in production
Experience: ps(1) failure
Getting closer to the cause
15020
15281
15323
15414
15746
15773
15778
# bpftrace -e 'k:find_ge_pid /comm == "ps"/ { printf("%dn", arg0);
}'
30707
31546
31913
31944
31945
31946
32070
success failure
Experience: ps(1) failure
# bpftrace -e 'k:find_ge_pid /comm == "ps"/ { @nr[tid] = arg0; }
kr:find_ge_pid /@nr[tid]/ {
printf("%d: %llxn", @nr[tid], retval); delete(@nr[tid]); }'
[…]
15561: ffff8a3ee70ad280
15564: ffff8a400244bb80
15569: ffff8a3f6f1a1840
15570: ffff8a3ffe890c00
15571: ffff8a3ffd23bdc0
15575: ffff8a40024fdd80
15576: 0
find_ge_pid() entry argument & return value:
Experience: ps(1) failure
struct pid *find_ge_pid(int nr, struct pid_namespace *ns)
{
return idr_get_next(&ns->idr, &nr);
}
[…]
void *idr_get_next(struct idr *idr, int *nextid)
{
[…]
slot = radix_tree_iter_find(&idr->idr_rt, &iter, id);
Subject [RFC 2/2] pid: Replace PID bitmap implementation with IDR API
Date Sat, 9 Sep 2017 18:03:17 +0530
[…]
Kernel source:
Experience: ps(1) failure
So far we have moved from:
To:
I’ll keep digging after this keynote
ps(1) sometimes fails. Kernel bug!
find_ge_pid() sometimes returns NULL
instead of the next struct *pid
Takeaway:
BPF enables better bug reports
bpftrace: BPF observability front-end
Linux 4.9+ https://github.com/iovisor/bpftrace
# Files opened by process
bpftrace -e 't:syscalls:sys_enter_open { printf("%s %sn", comm,
str(args->filename)) }'
# Read size distribution by process
bpftrace -e 't:syscalls:sys_exit_read { @[comm] = hist(args->ret) }'
# Count VFS calls
bpftrace -e 'kprobe:vfs_* { @[func]++ }'
# Show vfs_read latency as a histogram
bpftrace -e 'k:vfs_read { @[tid] = nsecs }
kr:vfs_read /@[tid]/ { @ns = hist(nsecs - @[tid]); delete(@tid) }’
# Trace user-level function
bpftrace -e 'uretprobe:bash:readline { printf(“%sn”, str(retval)) }’
…
Raw BPF
samples/bpf/sock_example.c
87 lines truncated
C/BPF
samples/bpf/tracex1_kern.c
58 lines truncated
bcc/BPF (C & Python)
bcc examples/tracing/bitehist.py
entire program
bpftrace/BPF
https://github.com/iovisor/bpftrace
entire program
bpftrace -e 'kr:vfs_read { @ = hist(retval); }'
The Tracing Landscape, Apr 2019
Scope & Capability
Easeofuse
sysdig
perf
ftrace
C/BPF
stap
Stage of
Development
(my opinion)
(brutal)(lessbrutal)
(alpha) (mature)
bcc/BPF
ply/BPF
Raw BPF
LTTng
(hist triggers)
recent changes
(many)
bpftrace
(eBPF)
(0.9)
Experience: readahead
Experience: readahead
Is readahead polluting the cache?
Experience: readahead
# readahead.bt
Attaching 5 probes...
^C
Readahead unused pages: 128
Readahead used page age (ms):
@age_ms:
[1] 2455 |@@@@@@@@@@@@@@@ |
[2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[32, 64) 0 | |
[64, 128) 0 | |
[128, 256) 384 |@@ |
Is readahead polluting the cache?
#!/usr/local/bin/bpftrace
kprobe:__do_page_cache_readahead { @in_readahead[tid] = 1; }
kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
kretprobe:__page_cache_alloc
/@in_readahead[tid]/
{
@birth[retval] = nsecs;
@rapages++;
}
kprobe:mark_page_accessed
/@birth[arg0]/
{
@age_ms = hist((nsecs - @birth[arg0]) / 1000000);
delete(@birth[arg0]);
@rapages--;
}
END
{
printf("nReadahead unused pages: %dn", @rapages);
printf("nReadahead used page age (ms):n");
print(@age_ms); clear(@age_ms);
clear(@birth); clear(@in_readahead); clear(@rapages);
}
Takeaway:
bpftrace is good for short tools
bpftrace Syntax
bpftrace -e ‘k:do_nanosleep /pid > 100/ { @[comm]++ }’
Probe
Filter
(optional)
Action
Probes
Probe Type Shortcuts
tracepoint t Kernel static tracepoints
usdt U User-level statically defined tracing
kprobe k Kernel function tracing
kretprobe kr Kernel function returns
uprobe u User-level function tracing
uretprobe ur User-level function returns
profile p Timed sampling across all CPUs
interval i Interval output
software s Kernel software events
hardware h Processor hardware events
Filters
● /pid == 181/
● /comm != “sshd”/
● /@ts[tid]/
Actions
●
Per-event output
– printf()
– system()
– join()
– time()
●
Map Summaries
– @ = count() or @++
– @ = hist()
– …
The following is in the https://github.com/iovisor/bpftrace/blob/master/docs/reference_
Functions
●
hist(n) Log2 histogram
●
lhist(n, min, max, step) Linear hist.
●
count() Count events
●
sum(n) Sum value
●
min(n) Minimum value
●
max(n) Maximum value
●
avg(n) Average value
●
stats(n) Statistics
●
str(s) String
●
ksym(p) Resolve kernel addr
●
usym(p) Resolve user addr
●
kaddr(n) Resolve kernel symbol
●
uaddr(n) Resolve user symbol
●
printf(fmt, ...) Print formatted
●
print(@x[, top[, div]]) Print map
●
delete(@x) Delete map element
●
clear(@x) Delete all keys/values
●
reg(n) Register lookup
●
join(a) Join string array
●
time(fmt) Print formatted time
●
system(fmt) Run shell command
●
cat(file) Print file contents
●
exit() Quit bpftrace
Variable Types
●
Basic Variables
– @global
– @thread_local[tid]
– $scratch
●
Associative Arrays
– @array[key] = value
●
Buitins
– pid
– ...
Builtin Variables
● pid Process ID (kernel tgid)
● tid Thread ID (kernel pid)
● cgroup Current Cgroup ID
● uid User ID
● gid Group ID
● nsecs Nanosecond timestamp
● cpu Processor ID
● comm Process name
● kstack Kernel stack trace
● ustack User stack trace
● arg0, arg1, ... Function args
● retval Return value
● args Tracepoint args
● func Function name
● probe Full probe name
● curtask Curr task_struct (u64)
● rand Random number (u32)
bpftrace: biolatency
#!/usr/local/bin/bpftrace
BEGIN
{
printf("Tracing block device I/O... Hit Ctrl-C to end.n");
}
kprobe:blk_account_io_start
{
@start[arg0] = nsecs;
}
kprobe:blk_account_io_completion
/@start[arg0]/
{
@usecs = hist((nsecs - @start[arg0]) / 1000);
delete(@start[arg0]);
}
Experience: superping!
Experience: superping
# ping 172.20.0.1
PING 172.20.0.1 (172.20.0.1) 56(84) bytes of data.
64 bytes from 172.20.0.1: icmp_seq=1 ttl=64 time=2.87 ms
64 bytes from 172.20.0.1: icmp_seq=2 ttl=64 time=1.66 ms
64 bytes from 172.20.0.1: icmp_seq=3 ttl=64 time=1.55 ms
64 bytes from 172.20.0.1: icmp_seq=4 ttl=64 time=1.11 ms
64 bytes from 172.20.0.1: icmp_seq=5 ttl=64 time=2.48 ms
64 bytes from 172.20.0.1: icmp_seq=6 ttl=64 time=2.39 ms
[...]
How much is scheduler latency?
Experience: superping
# ./superping.bt
Attaching 6 probes...
Tracing ICMP echo request latency. Hit Ctrl-C to end.
IPv4 ping, ID 9827 seq 1: 2883 us
IPv4 ping, ID 9827 seq 2: 1682 us
IPv4 ping, ID 9827 seq 3: 1568 us
IPv4 ping, ID 9827 seq 4: 1078 us
IPv4 ping, ID 9827 seq 5: 2486 us
IPv4 ping, ID 9827 seq 6: 2394 us
[...]
How much is scheduler latency?
?!
#!/usr/local/bin/bpftrace
#include <linux/skbuff.h>
#include <linux/icmp.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/in.h>
BEGIN { printf("Tracing ICMP echo request latency. Hit Ctrl-C to end.n"); }
kprobe:ip_send_skb
{
$skb = (struct sk_buff *)arg1;
// get IPv4 header; see skb_network_header():
$iph = (struct iphdr *)($skb->head + $skb->network_header);
if ($iph->protocol == IPPROTO_ICMP) {
// get ICMP header; see skb_transport_header():
$icmph = (struct icmphdr *)($skb->head + $skb->transport_header);
if ($icmph->type == ICMP_ECHO) {
$id = $icmph->un.echo.id;
$seq = $icmph->un.echo.sequence;
@start[$id, $seq] = nsecs;
}
}
}
[...]
Note: no debuginfo required
Takeaway:
BPF tracing can walk structs
bpftrace Internals
Major Features (v1)
Known Bug Fixes
Packaging
API Stability Stable Docs
Oct 2018
v0.90
Mar-2019
v1.0
?2019Dec 2016
More Bug Fixes
v0.80
Jan-2019
Minor Features (v1) ...
bpftrace Development
v0.10
?Jun-2019
bpftrace Tools Linux 4.9+
https://github.com/iovisor/
Experience: tcplife
Experience: tcplife
Which processes are connecting to which port?
Experience: tcplife
# ./tcplife
PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS
22597 recordProg 127.0.0.1 46644 127.0.0.1 28527 0 0 0.23
3277 redis-serv 127.0.0.1 28527 127.0.0.1 46644 0 0 0.28
22598 curl 100.66.3.172 61620 52.205.89.26 80 0 1 91.79
22604 curl 100.66.3.172 44400 52.204.43.121 80 0 1 121.38
22624 recordProg 127.0.0.1 46648 127.0.0.1 28527 0 0 0.22
3277 redis-serv 127.0.0.1 28527 127.0.0.1 46648 0 0 0.27
22647 recordProg 127.0.0.1 46650 127.0.0.1 28527 0 0 0.21
3277 redis-serv 127.0.0.1 28527 127.0.0.1 46650 0 0 0.26
[...]
Which processes are connecting to which port?
bcc Linux 4.4+
https://github.com/iovisor/bcc
int kprobe__tcp_set_state(struct pt_regs *ctx, struct sock *sk, int state)
{
u32 pid = bpf_get_current_pid_tgid() >> 32;
// lport is either used in a filter here, or later
u16 lport = sk->__sk_common.skc_num;
[…]
struct tcp_sock *tp = (struct tcp_sock *)sk;
rx_b = tp->bytes_received;
tx_b = tp->bytes_acked;
u16 family = sk->__sk_common.skc_family;
if (family == AF_INET) {
struct ipv4_data_t data4 = {};
data4.span_us = delta_us;
data4.rx_b = rx_b;
data4.tx_b = tx_b;
data4.ts_us = bpf_ktime_get_ns() / 1000;
data4.saddr = sk->__sk_common.skc_rcv_saddr;
data4.daddr = sk->__sk_common.skc_daddr;
[…]
bcc: tcplife
Experience: tcplife
# bpftrace -lv t:tcp:tcp_set_state
tracepoint:tcp:tcp_set_state
const void * skaddr;
int oldstate;
int newstate;
__u16 sport;
__u16 dport;
__u8 saddr[4];
__u8 daddr[4];
__u8 saddr_v6[16];
__u8 daddr_v6[16];
From kprobes to tracepoints
# bpftrace -lv t:sock:inet_sock_set_state
tracepoint:sock:inet_sock_set_state
const void * skaddr;
int oldstate;
int newstate;
__u16 sport;
__u16 dport;
__u16 family;
__u8 protocol;
__u8 saddr[4];
__u8 daddr[4];
__u8 saddr_v6[16];
__u8 daddr_v6[16];Linux 4.15
Linux 4.16+
Takeaways:
bcc for complex tools
kprobes can prototype tracepoints
tracepoints can change (best effort)
Experience: cachestat
Experience: cachestat
What is the page cache hit ratio?
Experience: cachestat
# cachestat
HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
1132 0 4 100.00% 277 4367
161 0 36 100.00% 277 4372
16 0 28 100.00% 277 4372
17154 13750 15 55.51% 277 4422
19 0 1 100.00% 277 4422
83 0 83 100.00% 277 4421
16 0 1 100.00% 277 4423
[...]
What is the page cache hit ratio?
b.attach_kprobe(event="add_to_page_cache_lru", fn_name="do_count")
b.attach_kprobe(event="mark_page_accessed", fn_name="do_count")
b.attach_kprobe(event="account_page_dirtied", fn_name="do_count")
b.attach_kprobe(event="mark_buffer_dirty", fn_name="do_count")
[…]
# total = total cache accesses without counting dirties
# misses = total of add to lru because of read misses
total = mpa - mbd
misses = apcl - apd
if misses < 0:
misses = 0
if total < 0:
total = 0
hits = total - misses
# If hits are < 0, then its possible misses are overestimated
# due to possibly page cache read ahead adding more pages than
# needed. In this case just assume misses as total and reset hits.
if hits < 0:
misses = total
hits = 0
[…]
This is a sandcastle
Takeaway:
BPF tracing can prototype /proc stats
Many of our perf wins are from CPU flame graphs
not CLI tracing
Reality Check
Java
JVM
Kernel
GC
CPU Flame Graphs
Alphabetical frame sort (A - Z)
Stackdepth(0-max)
BPF-based CPU Flame Graphs
perf record
perf script
stackcollapse-perf.pl
flamegraph.pl
perf.data
flamegraph.pl
profile.py
Linux 4.9Linux 2.6
Takeaway:
BPF all the things!
Take Aways
●
BPF observability:
– bpftrace: one-liners, short scripts
– bcc: complex tools
– Production safe, and no debuginfo needed
●
kprobe tools can prototype tracepoints, /proc stats
●
I’m ok with tracepoints are best effort
●
BPF all the things!
URLs
- https://github.com/iovisor/bpftrace
– https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
– https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md
- https://github.com/iovisor/bcc
– https://github.com/iovisor/bcc/blob/master/docs/tutorial.md
– https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md
- http://www.brendangregg.com/ebpf.html
Update: this keynote was summarized by
https://lwn.net/Articles/787131/
Thanks
●
bpftrace
– Alastair Robertson (creator)
– Netflix: myself, Matheus Marchini
– Sthima: Willian Gaspar
– Facebook: Jon Haslam, Dan Xu
– Augusto Mecking Caringi, Dale Hamel, ...
●
eBPF & bcc
– Facebook: Alexei Starovoitov, Teng Qin, Yonghong Song, Martin Lau,
Mark Drayton, …
– Netflix: myself
– VMware: Brenden Blanco
– Daniel Borkmann, David S. Miller, Sasha Goldsthein, Paul Chaignon, ...

More Related Content

What's hot

Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Ray Jenkins
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceBrendan Gregg
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceSUSE Labs Taipei
 
eBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KerneleBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KernelThomas Graf
 
Performance Tuning EC2 Instances
Performance Tuning EC2 InstancesPerformance Tuning EC2 Instances
Performance Tuning EC2 InstancesBrendan Gregg
 
LinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking WalkthroughLinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking WalkthroughThomas Graf
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedBrendan Gregg
 
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCKernel TLV
 
Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Brendan Gregg
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)Brendan Gregg
 
Cilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDPCilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDPThomas Graf
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFBrendan Gregg
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019Brendan Gregg
 

What's hot (20)

Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!Understanding eBPF in a Hurry!
Understanding eBPF in a Hurry!
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
 
eBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux KerneleBPF - Rethinking the Linux Kernel
eBPF - Rethinking the Linux Kernel
 
Performance Tuning EC2 Instances
Performance Tuning EC2 InstancesPerformance Tuning EC2 Instances
Performance Tuning EC2 Instances
 
eBPF maps 101
eBPF maps 101eBPF maps 101
eBPF maps 101
 
OVS v OVS-DPDK
OVS v OVS-DPDKOVS v OVS-DPDK
OVS v OVS-DPDK
 
LinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking WalkthroughLinuxCon 2015 Linux Kernel Networking Walkthrough
LinuxCon 2015 Linux Kernel Networking Walkthrough
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting Started
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
 
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
 
Dpdk performance
Dpdk performanceDpdk performance
Dpdk performance
 
Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
 
Cilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDPCilium - Container Networking with BPF & XDP
Cilium - Container Networking with BPF & XDP
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
 

Similar to LSFMM 2019 BPF Observability

CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
 
Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨flyinweb
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at NetflixBrendan Gregg
 
Global Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the SealGlobal Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the SealTzung-Bi Shih
 
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321Teddy Hsiung
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFBrendan Gregg
 
Linux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudLinux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudAndrea Righi
 
Beyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisBeyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisFastly
 
A miało być tak... bez wycieków
A miało być tak... bez wyciekówA miało być tak... bez wycieków
A miało być tak... bez wyciekówKonrad Kokosa
 
C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략 C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략 명신 김
 
Improving go-git performance
Improving go-git performanceImproving go-git performance
Improving go-git performancesource{d}
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernellcplcp1
 
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)Gavin Guo
 
Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"Yulia Tsisyk
 
State of the .Net Performance
State of the .Net PerformanceState of the .Net Performance
State of the .Net PerformanceCUSTIS
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018Brendan Gregg
 
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in GoCapturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in GoScyllaDB
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing LandscapeSasha Goldshtein
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLCommand Prompt., Inc
 

Similar to LSFMM 2019 BPF Observability (20)

CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
 
Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
 
Global Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the SealGlobal Interpreter Lock: Episode I - Break the Seal
Global Interpreter Lock: Episode I - Break the Seal
 
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
ExperiencesSharingOnEmbeddedSystemDevelopment_20160321
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
 
Linux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloudLinux kernel tracing superpowers in the cloud
Linux kernel tracing superpowers in the cloud
 
Beyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic AnalysisBeyond Breakpoints: A Tour of Dynamic Analysis
Beyond Breakpoints: A Tour of Dynamic Analysis
 
A miało być tak... bez wycieków
A miało być tak... bez wyciekówA miało być tak... bez wycieków
A miało być tak... bez wycieków
 
C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략 C++ AMP 실천 및 적용 전략
C++ AMP 실천 및 적용 전략
 
Improving go-git performance
Improving go-git performanceImproving go-git performance
Improving go-git performance
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernel
 
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
Spectre(v1%2 fv2%2fv4) v.s. meltdown(v3)
 
Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"Adam Sitnik "State of the .NET Performance"
Adam Sitnik "State of the .NET Performance"
 
State of the .Net Performance
State of the .Net PerformanceState of the .Net Performance
State of the .Net Performance
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018
 
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in GoCapturing NIC and Kernel TX and RX Timestamps for Packets in Go
Capturing NIC and Kernel TX and RX Timestamps for Packets in Go
 
Modern Linux Tracing Landscape
Modern Linux Tracing LandscapeModern Linux Tracing Landscape
Modern Linux Tracing Landscape
 
Quiz 9
Quiz 9Quiz 9
Quiz 9
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 

More from Brendan Gregg

YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing PerformanceBrendan Gregg
 
Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Brendan Gregg
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedBrendan Gregg
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceBrendan Gregg
 
LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsBrendan Gregg
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixBrendan Gregg
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixBrendan Gregg
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityBrendan Gregg
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Brendan Gregg
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceBrendan Gregg
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance AnalysisBrendan Gregg
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFBrendan Gregg
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesBrendan Gregg
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFBrendan Gregg
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsBrendan Gregg
 

More from Brendan Gregg (18)

YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing Performance
 
Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
 
LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing Tools
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflix
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at Netflix
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF Observability
 
FlameScope 2018
FlameScope 2018FlameScope 2018
FlameScope 2018
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for Performance
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis Methodologies
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
 

Recently uploaded

Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?Paolo Missier
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligencePrecisely
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfAnubhavMangla3
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxFIDO Alliance
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuidePixlogix Infotech
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxFIDO Alliance
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfdanishmna97
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdfMuhammad Subhan
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Paige Cruz
 

Recently uploaded (20)

Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial IntelligenceRevolutionizing SAP® Processes with Automation and Artificial Intelligence
Revolutionizing SAP® Processes with Automation and Artificial Intelligence
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 

LSFMM 2019 BPF Observability

  • 1. # readahead.bt Attaching 5 probes... ^C Readahead unused pages: 128 Readahead used page age (ms): @age_ms: [1] 2455 |@@@@@@@@@@@@@@@ | [2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 384 |@@ | BPF Observability Brendan Gregg LSFMM Apr 2019
  • 4. eBPF: extended Berkeley Packet Filter Kernel kprobes uprobes tracepoints sockets SDN Configuration User-Defined BPF Programs … Event TargetsRuntime perf_events BPF actions BPF verifier DDoS Mitigation Intrusion Detection Container Security Observability Firewalls (bpfilter) Device Drivers
  • 5. >150k AWS EC2 server instances ~34% US Internet traffic at night >130M members Performance is customer satisfaction & Netflix cost
  • 6. Experience: ps(1) failure (This is from last week...)
  • 7. Experience: ps(1) failure # wait for $pid to finish: while ps -p $pid >/dev/null; do sleep 1 done # do stuff...
  • 8. Experience: ps(1) failure Problem: ps(1) sometimes fails to find the process Hypothesis: kernel bug! # wait for $pid to finish: while ps -p $pid >/dev/null; do sleep 1 done # do stuff...
  • 9. Experience: ps(1) failure # bpftrace -e 't:syscalls:sys_exit_* /comm == "ps"/ { @[probe, args->ret > 0 ? 0 : - args->ret] = count(); }' Attaching 316 probes... [...] @[tracepoint:syscalls:sys_exit_openat, 2]: 120 @[tracepoint:syscalls:sys_exit_newfstat, 0]: 180 @[tracepoint:syscalls:sys_exit_mprotect, 0]: 230 @[tracepoint:syscalls:sys_exit_rt_sigaction, 0]: 240 @[tracepoint:syscalls:sys_exit_mmap, 0]: 350 @[tracepoint:syscalls:sys_exit_newstat, 0]: 5000 @[tracepoint:syscalls:sys_exit_read, 0]: 10170 @[tracepoint:syscalls:sys_exit_close, 0]: 10190 @[tracepoint:syscalls:sys_exit_openat, 0]: 10190 Which syscall is abnormally failing (without strace(1) )?
  • 10. Experience: ps(1) failure # bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ { @[args->id, args->ret > 0 ? 0 : - args->ret] = count(); }' Attaching 1 probe... [...] @[21, 2]: 120 @[5, 0]: 180 @[10, 0]: 230 @[13, 0]: 240 @[9, 0]: 350 @[4, 0]: 5000 @[0, 0]: 10170 @[3, 0]: 10190 @[257, 0]: 10190 Which syscall is abnormally failing (without multi-probe)?
  • 11. Experience: ps(1) failure # bpftrace -e 't:raw_syscalls:sys_exit /comm == "ps"/ { @[ksym(*(kaddr("sys_call_table") + args->id * 8)), args->ret > 0 ? 0 : - args->ret] = count(); }' [...] @[sys_brk, 0]: 8202 @[sys_ioctl, 25]: 8203 @[sys_access, 2]: 32808 @[SyS_openat, 2]: 32808 @[sys_newfstat, 0]: 49213 @[sys_newstat, 2]: 60820 @[sys_mprotect, 0]: 62882 [...] Which syscall is abnormally failing (without multi-probe)? caught 1 extra failure ioctl() was a dead end
  • 12. Experience: ps(1) failure # bpftrace -e 't:syscalls:sys_exit_getdents /comm == "ps"/ { printf("ret: %dn", args->ret); }' [...] ret: 9192 ret: 0 ret: 9216 ret: 0 ret: 9168 ret: 0 ret: 5640 ret: 0 ^C Which syscall is successfully failing?
  • 13. Experience: ps(1) failure # bpftrace -e 't:syscalls:sys_enter_getdents /comm == "ps"/ { @start[tid] = nsecs; } t:syscalls:sys_exit_getdents /@start[tid]/ { printf("%8d us, ret: %dn", (nsecs - @start[tid]) / 1000, args->ret); delete(@start[tid]); }' [...] 559 us, ret: 9640 3 us, ret: 0 516 us, ret: 9576 3 us, ret: 0 373 us, ret: 7720 2 us, ret: 0 ^C Which syscall is successfully failing?
  • 14. Experience: ps(1) failure # funccount '*proc*' Tracing "*proc*"... Ctrl-C to end.^C FUNC COUNT […] proc_readdir 1492 proc_readdir_de 1492 proc_root_getattr 1492 process_measurement 1669 kick_process 1671 wake_up_process 2188 proc_pid_readdir 2984 proc_root_readdir 2984 proc_fill_cache 977263 /proc debugging
  • 15. Experience: ps(1) failure # bpftrace -e 'kr:proc_fill_cache /comm == "ps"/ { @[retval] = count(); }' Some quick dead ends # bpftrace -e 'kr:nr_processes /comm == "ps"/ { printf("%dn", retval); }' # bpftrace -e 'kr:proc_readdir_de /comm == "ps"/ { printf("%dn", retval); }' # bpftrace -e 'kr:proc_root_readdir /comm == "ps"/ { printf("%dn", retval); }' Note: this is all in production
  • 16. Experience: ps(1) failure Getting closer to the cause 15020 15281 15323 15414 15746 15773 15778 # bpftrace -e 'k:find_ge_pid /comm == "ps"/ { printf("%dn", arg0); }' 30707 31546 31913 31944 31945 31946 32070 success failure
  • 17. Experience: ps(1) failure # bpftrace -e 'k:find_ge_pid /comm == "ps"/ { @nr[tid] = arg0; } kr:find_ge_pid /@nr[tid]/ { printf("%d: %llxn", @nr[tid], retval); delete(@nr[tid]); }' […] 15561: ffff8a3ee70ad280 15564: ffff8a400244bb80 15569: ffff8a3f6f1a1840 15570: ffff8a3ffe890c00 15571: ffff8a3ffd23bdc0 15575: ffff8a40024fdd80 15576: 0 find_ge_pid() entry argument & return value:
  • 18. Experience: ps(1) failure struct pid *find_ge_pid(int nr, struct pid_namespace *ns) { return idr_get_next(&ns->idr, &nr); } […] void *idr_get_next(struct idr *idr, int *nextid) { […] slot = radix_tree_iter_find(&idr->idr_rt, &iter, id); Subject [RFC 2/2] pid: Replace PID bitmap implementation with IDR API Date Sat, 9 Sep 2017 18:03:17 +0530 […] Kernel source:
  • 19. Experience: ps(1) failure So far we have moved from: To: I’ll keep digging after this keynote ps(1) sometimes fails. Kernel bug! find_ge_pid() sometimes returns NULL instead of the next struct *pid
  • 21. bpftrace: BPF observability front-end Linux 4.9+ https://github.com/iovisor/bpftrace # Files opened by process bpftrace -e 't:syscalls:sys_enter_open { printf("%s %sn", comm, str(args->filename)) }' # Read size distribution by process bpftrace -e 't:syscalls:sys_exit_read { @[comm] = hist(args->ret) }' # Count VFS calls bpftrace -e 'kprobe:vfs_* { @[func]++ }' # Show vfs_read latency as a histogram bpftrace -e 'k:vfs_read { @[tid] = nsecs } kr:vfs_read /@[tid]/ { @ns = hist(nsecs - @[tid]); delete(@tid) }’ # Trace user-level function bpftrace -e 'uretprobe:bash:readline { printf(“%sn”, str(retval)) }’ …
  • 24. bcc/BPF (C & Python) bcc examples/tracing/bitehist.py entire program
  • 26. The Tracing Landscape, Apr 2019 Scope & Capability Easeofuse sysdig perf ftrace C/BPF stap Stage of Development (my opinion) (brutal)(lessbrutal) (alpha) (mature) bcc/BPF ply/BPF Raw BPF LTTng (hist triggers) recent changes (many) bpftrace (eBPF) (0.9)
  • 28. Experience: readahead Is readahead polluting the cache?
  • 29. Experience: readahead # readahead.bt Attaching 5 probes... ^C Readahead unused pages: 128 Readahead used page age (ms): @age_ms: [1] 2455 |@@@@@@@@@@@@@@@ | [2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 384 |@@ | Is readahead polluting the cache?
  • 30. #!/usr/local/bin/bpftrace kprobe:__do_page_cache_readahead { @in_readahead[tid] = 1; } kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; } kretprobe:__page_cache_alloc /@in_readahead[tid]/ { @birth[retval] = nsecs; @rapages++; } kprobe:mark_page_accessed /@birth[arg0]/ { @age_ms = hist((nsecs - @birth[arg0]) / 1000000); delete(@birth[arg0]); @rapages--; } END { printf("nReadahead unused pages: %dn", @rapages); printf("nReadahead used page age (ms):n"); print(@age_ms); clear(@age_ms); clear(@birth); clear(@in_readahead); clear(@rapages); }
  • 31. Takeaway: bpftrace is good for short tools
  • 32. bpftrace Syntax bpftrace -e ‘k:do_nanosleep /pid > 100/ { @[comm]++ }’ Probe Filter (optional) Action
  • 34. Probe Type Shortcuts tracepoint t Kernel static tracepoints usdt U User-level statically defined tracing kprobe k Kernel function tracing kretprobe kr Kernel function returns uprobe u User-level function tracing uretprobe ur User-level function returns profile p Timed sampling across all CPUs interval i Interval output software s Kernel software events hardware h Processor hardware events
  • 35. Filters ● /pid == 181/ ● /comm != “sshd”/ ● /@ts[tid]/
  • 36. Actions ● Per-event output – printf() – system() – join() – time() ● Map Summaries – @ = count() or @++ – @ = hist() – … The following is in the https://github.com/iovisor/bpftrace/blob/master/docs/reference_
  • 37. Functions ● hist(n) Log2 histogram ● lhist(n, min, max, step) Linear hist. ● count() Count events ● sum(n) Sum value ● min(n) Minimum value ● max(n) Maximum value ● avg(n) Average value ● stats(n) Statistics ● str(s) String ● ksym(p) Resolve kernel addr ● usym(p) Resolve user addr ● kaddr(n) Resolve kernel symbol ● uaddr(n) Resolve user symbol ● printf(fmt, ...) Print formatted ● print(@x[, top[, div]]) Print map ● delete(@x) Delete map element ● clear(@x) Delete all keys/values ● reg(n) Register lookup ● join(a) Join string array ● time(fmt) Print formatted time ● system(fmt) Run shell command ● cat(file) Print file contents ● exit() Quit bpftrace
  • 38. Variable Types ● Basic Variables – @global – @thread_local[tid] – $scratch ● Associative Arrays – @array[key] = value ● Buitins – pid – ...
  • 39. Builtin Variables ● pid Process ID (kernel tgid) ● tid Thread ID (kernel pid) ● cgroup Current Cgroup ID ● uid User ID ● gid Group ID ● nsecs Nanosecond timestamp ● cpu Processor ID ● comm Process name ● kstack Kernel stack trace ● ustack User stack trace ● arg0, arg1, ... Function args ● retval Return value ● args Tracepoint args ● func Function name ● probe Full probe name ● curtask Curr task_struct (u64) ● rand Random number (u32)
  • 40. bpftrace: biolatency #!/usr/local/bin/bpftrace BEGIN { printf("Tracing block device I/O... Hit Ctrl-C to end.n"); } kprobe:blk_account_io_start { @start[arg0] = nsecs; } kprobe:blk_account_io_completion /@start[arg0]/ { @usecs = hist((nsecs - @start[arg0]) / 1000); delete(@start[arg0]); }
  • 42. Experience: superping # ping 172.20.0.1 PING 172.20.0.1 (172.20.0.1) 56(84) bytes of data. 64 bytes from 172.20.0.1: icmp_seq=1 ttl=64 time=2.87 ms 64 bytes from 172.20.0.1: icmp_seq=2 ttl=64 time=1.66 ms 64 bytes from 172.20.0.1: icmp_seq=3 ttl=64 time=1.55 ms 64 bytes from 172.20.0.1: icmp_seq=4 ttl=64 time=1.11 ms 64 bytes from 172.20.0.1: icmp_seq=5 ttl=64 time=2.48 ms 64 bytes from 172.20.0.1: icmp_seq=6 ttl=64 time=2.39 ms [...] How much is scheduler latency?
  • 43. Experience: superping # ./superping.bt Attaching 6 probes... Tracing ICMP echo request latency. Hit Ctrl-C to end. IPv4 ping, ID 9827 seq 1: 2883 us IPv4 ping, ID 9827 seq 2: 1682 us IPv4 ping, ID 9827 seq 3: 1568 us IPv4 ping, ID 9827 seq 4: 1078 us IPv4 ping, ID 9827 seq 5: 2486 us IPv4 ping, ID 9827 seq 6: 2394 us [...] How much is scheduler latency? ?!
  • 44. #!/usr/local/bin/bpftrace #include <linux/skbuff.h> #include <linux/icmp.h> #include <linux/ip.h> #include <linux/ipv6.h> #include <linux/in.h> BEGIN { printf("Tracing ICMP echo request latency. Hit Ctrl-C to end.n"); } kprobe:ip_send_skb { $skb = (struct sk_buff *)arg1; // get IPv4 header; see skb_network_header(): $iph = (struct iphdr *)($skb->head + $skb->network_header); if ($iph->protocol == IPPROTO_ICMP) { // get ICMP header; see skb_transport_header(): $icmph = (struct icmphdr *)($skb->head + $skb->transport_header); if ($icmph->type == ICMP_ECHO) { $id = $icmph->un.echo.id; $seq = $icmph->un.echo.sequence; @start[$id, $seq] = nsecs; } } } [...] Note: no debuginfo required
  • 47. Major Features (v1) Known Bug Fixes Packaging API Stability Stable Docs Oct 2018 v0.90 Mar-2019 v1.0 ?2019Dec 2016 More Bug Fixes v0.80 Jan-2019 Minor Features (v1) ... bpftrace Development v0.10 ?Jun-2019
  • 48. bpftrace Tools Linux 4.9+ https://github.com/iovisor/
  • 50. Experience: tcplife Which processes are connecting to which port?
  • 51. Experience: tcplife # ./tcplife PID COMM LADDR LPORT RADDR RPORT TX_KB RX_KB MS 22597 recordProg 127.0.0.1 46644 127.0.0.1 28527 0 0 0.23 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46644 0 0 0.28 22598 curl 100.66.3.172 61620 52.205.89.26 80 0 1 91.79 22604 curl 100.66.3.172 44400 52.204.43.121 80 0 1 121.38 22624 recordProg 127.0.0.1 46648 127.0.0.1 28527 0 0 0.22 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46648 0 0 0.27 22647 recordProg 127.0.0.1 46650 127.0.0.1 28527 0 0 0.21 3277 redis-serv 127.0.0.1 28527 127.0.0.1 46650 0 0 0.26 [...] Which processes are connecting to which port?
  • 53. int kprobe__tcp_set_state(struct pt_regs *ctx, struct sock *sk, int state) { u32 pid = bpf_get_current_pid_tgid() >> 32; // lport is either used in a filter here, or later u16 lport = sk->__sk_common.skc_num; […] struct tcp_sock *tp = (struct tcp_sock *)sk; rx_b = tp->bytes_received; tx_b = tp->bytes_acked; u16 family = sk->__sk_common.skc_family; if (family == AF_INET) { struct ipv4_data_t data4 = {}; data4.span_us = delta_us; data4.rx_b = rx_b; data4.tx_b = tx_b; data4.ts_us = bpf_ktime_get_ns() / 1000; data4.saddr = sk->__sk_common.skc_rcv_saddr; data4.daddr = sk->__sk_common.skc_daddr; […] bcc: tcplife
  • 54. Experience: tcplife # bpftrace -lv t:tcp:tcp_set_state tracepoint:tcp:tcp_set_state const void * skaddr; int oldstate; int newstate; __u16 sport; __u16 dport; __u8 saddr[4]; __u8 daddr[4]; __u8 saddr_v6[16]; __u8 daddr_v6[16]; From kprobes to tracepoints # bpftrace -lv t:sock:inet_sock_set_state tracepoint:sock:inet_sock_set_state const void * skaddr; int oldstate; int newstate; __u16 sport; __u16 dport; __u16 family; __u8 protocol; __u8 saddr[4]; __u8 daddr[4]; __u8 saddr_v6[16]; __u8 daddr_v6[16];Linux 4.15 Linux 4.16+
  • 55. Takeaways: bcc for complex tools kprobes can prototype tracepoints tracepoints can change (best effort)
  • 57. Experience: cachestat What is the page cache hit ratio?
  • 58. Experience: cachestat # cachestat HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB 1132 0 4 100.00% 277 4367 161 0 36 100.00% 277 4372 16 0 28 100.00% 277 4372 17154 13750 15 55.51% 277 4422 19 0 1 100.00% 277 4422 83 0 83 100.00% 277 4421 16 0 1 100.00% 277 4423 [...] What is the page cache hit ratio?
  • 59. b.attach_kprobe(event="add_to_page_cache_lru", fn_name="do_count") b.attach_kprobe(event="mark_page_accessed", fn_name="do_count") b.attach_kprobe(event="account_page_dirtied", fn_name="do_count") b.attach_kprobe(event="mark_buffer_dirty", fn_name="do_count") […] # total = total cache accesses without counting dirties # misses = total of add to lru because of read misses total = mpa - mbd misses = apcl - apd if misses < 0: misses = 0 if total < 0: total = 0 hits = total - misses # If hits are < 0, then its possible misses are overestimated # due to possibly page cache read ahead adding more pages than # needed. In this case just assume misses as total and reset hits. if hits < 0: misses = total hits = 0 […] This is a sandcastle
  • 60. Takeaway: BPF tracing can prototype /proc stats
  • 61. Many of our perf wins are from CPU flame graphs not CLI tracing Reality Check
  • 62. Java JVM Kernel GC CPU Flame Graphs Alphabetical frame sort (A - Z) Stackdepth(0-max)
  • 63. BPF-based CPU Flame Graphs perf record perf script stackcollapse-perf.pl flamegraph.pl perf.data flamegraph.pl profile.py Linux 4.9Linux 2.6
  • 65. Take Aways ● BPF observability: – bpftrace: one-liners, short scripts – bcc: complex tools – Production safe, and no debuginfo needed ● kprobe tools can prototype tracepoints, /proc stats ● I’m ok with tracepoints are best effort ● BPF all the things!
  • 66. URLs - https://github.com/iovisor/bpftrace – https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md – https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md - https://github.com/iovisor/bcc – https://github.com/iovisor/bcc/blob/master/docs/tutorial.md – https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md - http://www.brendangregg.com/ebpf.html Update: this keynote was summarized by https://lwn.net/Articles/787131/
  • 67. Thanks ● bpftrace – Alastair Robertson (creator) – Netflix: myself, Matheus Marchini – Sthima: Willian Gaspar – Facebook: Jon Haslam, Dan Xu – Augusto Mecking Caringi, Dale Hamel, ... ● eBPF & bcc – Facebook: Alexei Starovoitov, Teng Qin, Yonghong Song, Martin Lau, Mark Drayton, … – Netflix: myself – VMware: Brenden Blanco – Daniel Borkmann, David S. Miller, Sasha Goldsthein, Paul Chaignon, ...