SlideShare a Scribd company logo
1
Brendan Gregg
Jun 2021
Performance Wins with BPF
Getting Started
2
This article is
not for beginners
… not the best start
… out of date
BPF has evolved
Many docs were true in 2015
but not today
Newcomers keep re-posting
old info as new
This talk is Jun 2021
3
BPF is no longer an
acronym
BPF is a bytecode and
execution environment
4
How to get quick and easy
BPF performance wins
Think like a
sysadmin
Not like a
programmer
6
Think like a sysadmin
# apt-get install bcc-tools
# PATH=$PATH:/usr/share/bcc/tools
# execsnoop
# opensnoop
# tcplife
# ext4slower
# biosnoop
[...]
Get it installed everywhere and use it.
7
Think like a sysadmin
# apt-get install bcc-tools
# PATH=$PATH:/usr/share/bcc/tools
# execsnoop
# opensnoop
# tcplife
# ext4slower
# biosnoop
[...]
Get it installed everywhere and use it.
Anything periodic running? crontab?
Any misconfigurations? File not found?
Any unexpected TCP sessions?
Any file system I/O slower than 10ms?
Any unusual disk access patterns? Outliers?
8
Case Study: BCC biosnoop
# iostat -xz 1
Linux 4.15.0-1052-aws (cass-xxx) 12/04/2019 _x86_64_ (8 CPU)
[…]
avg-cpu: %user %nice %system %iowait %steal %idle
4.67 0.13 0.25 0.00 0.00 94.95
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 1.00 0.00 8.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
xvdb 0.00 0.00 32.00 0.00 152.00 0.00 9.50 0.16 5.00 5.00 0.00 5.00 16.00
xvdc 0.00 0.00 36.00 1.00 192.00 4.00 10.59 0.19 5.19 5.33 0.00 5.19 19.20
xvdd 4.00 0.00 37.00 0.00 181.00 0.00 9.78 0.18 4.97 4.97 0.00 4.54 16.80
xvde 0.00 0.00 27.00 0.00 156.00 0.00 11.56 0.18 6.67 6.67 0.00 6.67 18.00
xvdf 0.00 0.00 35.00 0.00 164.00 0.00 9.37 0.19 5.37 5.37 0.00 5.37 18.80
xvdg 0.00 0.00 25.00 1.00 136.00 4.00 10.77 0.14 5.23 5.44 0.00 5.23 13.60
md0 0.00 0.00 195.00 2.00 965.00 8.00 9.88 0.00 0.00 0.00 0.00 0.00 0.00
[…]
9
Case Study: BCC biosnoop, cont.
# /usr/share/bcc/tools/biosnoop
TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
0.000000 perl 7755 xvdc R 610822184 4096 8.57
0.000812 biosnoop 32196 xvda R 269480 4096 0.43
0.006197 perl 3285 xvde R 610737856 4096 6.10
0.006390 perl 23937 xvde R 377704624 4096 0.10
0.015040 perl 7755 xvdb R 732825200 4096 8.51
0.022842 perl 3285 xvdc R 732953880 4096 7.72
0.023019 perl 23937 xvdb R 377707064 4096 0.09
0.034443 perl 7755 xvdg R 732998328 4096 11.28
0.039648 perl 23937 xvdd R 733127392 4096 5.08
0.039863 perl 31913 xvdg R 732868048 4096 0.10
0.049431 perl 3285 xvdg R 732906896 4096 9.45
0.058521 perl 27565 xvdg R 610744920 4096 8.99
0.070843 perl 27565 xvdg R 377706520 4096 12.26
0.080564 perl 31913 xvdc R 610951744 4096 9.62
0.080804 perl 7755 xvdc R 732858664 4096 0.14
0.086932 perl 27565 xvdg R 732937416 4096 6.01
0.087093 perl 27565 xvde R 610853240 4096 0.09
[…]
10
Case Study: BCC biosnoop, cont.
# ps -ef | grep perl
root 3285 3274 1 14:16 ? 00:04:24 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 7755 7748 1 04:16 ? 00:10:20 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 11366 11359 1 10:16 ? 00:06:39 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 15054 15049 2 16:16 ? 00:03:07 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 19675 19670 1 06:16 ? 00:08:53 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 23937 23930 1 12:16 ? 00:05:30 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 27565 27561 2 18:16 ? 00:00:28 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 28232 28223 1 02:16 ? 00:11:43 /usr/bin/perl /apps/…ec2rotatelogs.pl
root 31913 31907 1 08:15 ? 00:07:40 /usr/bin/perl /apps/…ec2rotatelogs.pl
[…]
Many more
tools to try!
bcc tools
bpftrace tools
(from my book, open source)
Solve >90% of perf
issues with canned
observability
(tracing) tools
This is BPF
observability
in one pic
Print on your
office/home
wall, use as a
checklist
Suspected disk issue?
Try these first:
Print on your
office/home
wall, use as a
checklist
Suspected disk issue?
Try these first:
Then walk up the
stack: file system tools,
VFS tools, syscall tools,
etc.
14
The future of BPF perf observability
… is GUIs. The end user may not even know it’s BPF.
Tool output, visualized
This GUI is in development by Susie Xia, Netflix
15
bcc
tools
Instance
agent
Server
Application
bcc
tools
Instance
agent
Server
Application
Client
GUI GUI
Webpage
Web Server
Example real-time BPF observability UI
Netflix Vector (now retired)
uses this model
●
Instance agent pcp pmdabcc
●
https://github.com/Netflix/vector
16
Instance
agent
Server
Application
Instance
agent
Server
Application
Client
GUI GUI
Webpage
Web Server
Example real-time BPF observability UI #2
bpftrace tools
bpf-
trace
bpf-
trace
Netflix FlameCommander UI
●
(not yet open source)
17
Many tools are sandcastles, and require frequent rebuilding to match kernel changes
Fortunately many companies and engineers maintain these versions
Think like a sysadmin
Please try to use bcc/bpftrace tools as-is and fetch regular updates
bcc & bpftrace
tools
Facebook
Isovalent
Netflix
(many more)
18
Sysadmins sometimes program
shell scripting
awk
sed one-liners
bpftrace tools
bpftrace one-liners
19
Think like a programmer if
You have a real-world problem that tools don’t solve
You’re a BPF-based startup
You’re debugging your own code*
You’re doing networking/security/etc.
You really want to learn BPF internals
* JIT-ed runtimes like Java are currently complex to trace
20
Recommended tracing front-ends
I want to run some tools
●
bcc, bpftrace
I want to hack up some new tools
●
bpftrace
I want to spend weeks developing a BPF product
●
bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs
Requires LLVM; becoming
obsolete / special-use only
New, lightweight,
CO-RE & BTF based
21
Recommended tracing front-ends
I want to run some tools
●
bcc, bpftrace /usr/bin/*
I want to hack up some new tools
●
bpftrace bash, awk
I want to spend weeks developing a BPF product
●
bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs C, C++
Unix analogies
Requires LLVM; becoming
obsolete / special-use only
New, lightweight,
CO-RE & BTF based
22
bpftrace example
# readahead.bt
Attaching 5 probes...
^C
Readahead unused pages: 128
Readahead used page age (ms):
@age_ms:
[1] 2455 |@@@@@@@@@@@@@@@ |
[2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[32, 64) 0 | |
[64, 128) 0 | |
[128, 256) 384 |@@ |
23
#!/usr/local/bin/bpftrace
kprobe:__do_page_cache_readahead { @in_readahead[tid] = 1; }
kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; }
kretprobe:__page_cache_alloc
/@in_readahead[tid]/
{
@birth[retval] = nsecs;
@rapages++;
}
kprobe:mark_page_accessed
/@birth[arg0]/
{
@age_ms = hist((nsecs - @birth[arg0]) / 1000000);
delete(@birth[arg0]);
@rapages--;
}
END
{
printf("nReadahead unused pages: %dn", @rapages);
printf("nReadahead used page age (ms):n");
print(@age_ms); clear(@age_ms);
clear(@birth); clear(@in_readahead); clear(@rapages);
}
Fits on one slide!
bpftrace example
24
BCC libbpf tool example
# ./opensnoop
PID COMM FD ERR PATH
27974 opensnoop 28 0 /etc/localtime
1482 redis-server 7 0 /proc/1482/stat
[…]
# ldd opensnoop
linux-vdso.so.1 (0x00007ffddf3f1000)
libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f9fb7836000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9fb7619000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fb7228000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9fb7c76000)
# ls -lh opensnoop opensnoop.stripped
-rwxr-xr-x 1 root root 645K Feb 28 23:18 opensnoop
-rwxr-xr-x 1 root root 151K Feb 28 23:33 opensnoop.stripped
151 Kbytes for a stand-alone BPF program!
(Note: A static bpftrace/BTF + scripts will also have a small average tool size)
25
PSA
CONFIG_DEBUG_INFO_BTF=y
E.g., Ubuntu 20.10, Fedora 30, and RHEL 8.2 have it
26
Kernel
User-mode
Applications
Hardware Events (incl. clock)
BPF Future: Event-based Applications
Kernel-mode
Applications (BPF)
Scheduler Kernel
Events
U.E.
27
A New Type of Software
Execution
model
User
defined
Compil-
ation
Security Failure
mode
Resource
access
User task yes any user
based
abort syscall,
fault
Kernel task no static none panic direct
BPF event yes JIT,
CO-RE
verified,
JIT
error
message
restricted
helpers
28
Take Away
To get started with BPF performance wins,
think like a sysadmin:
1. Install BCC & bpftrace tools
2. Run them
3. Get some wins
29
References
This is also where I recommend you go to learn more:
●
https://github.com/iovisor/bcc/blob/master/docs/tutorial.md
●
https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
●
BPF Performance Tools, Addison Wesley 2020
●
Systems Performance 2nd
Edition, Addison Wesley 2021
●
http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
●
http://www.brendangregg.com/ebpf.html
●
https://ebpf.io/what-is-ebpf
30
Thanks
BPF: Alexei Starovoitov (Facebook), Daniel Borkmann (Isovalent), David S. Miller (Red
Hat), Jakub Kicinski (Facebook), Yonghong Song (Facebook), Martin KaFai Lau
(Facebook), John Fastabend (Isovalent), Quentin Monnet (Isovalent), Jesper Dangaard
Brouer (Red Hat), Andrey Ignatov (Facebook), and Stanislav Fomichev (Google), Linus
Torvalds, and many more in the BPF community
BCC: Brenden Blanco (VMware), Yonghong Song, Sasha Goldsthein (Google), Teng Qin
(Facebook), Paul Chaignon (Isovalent), Vicent Martí (PlanetScale), and many more in the
BCC community
bpftrace: Alastair Robertson (Yellowbrick Data), Dan Xu (Facebook), Bas Smit, Mary
Marchini (Netflix), Masanori Misono, Jiri Olsa, Viktor Malík, Dale Hamel, Willian Gaspar,
Augusto Mecking Caringi, and many more in the bpftrace community

More Related Content

What's hot

Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)
Brendan Gregg
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
Brendan Gregg
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
Brendan Gregg
 
Meet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingMeet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracing
Viller Hsiao
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
Brendan Gregg
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
SUSE Labs Taipei
 
BPF - in-kernel virtual machine
BPF - in-kernel virtual machineBPF - in-kernel virtual machine
BPF - in-kernel virtual machine
Alexei Starovoitov
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF Superpowers
Brendan Gregg
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Brendan Gregg
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Anne Nicolas
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
Netronome
 
Introduction to eBPF
Introduction to eBPFIntroduction to eBPF
Introduction to eBPF
RogerColl2
 
Linux kernel tracing
Linux kernel tracingLinux kernel tracing
Linux kernel tracing
Viller Hsiao
 
New Ways to Find Latency in Linux Using Tracing
New Ways to Find Latency in Linux Using TracingNew Ways to Find Latency in Linux Using Tracing
New Ways to Find Latency in Linux Using Tracing
ScyllaDB
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019
Brendan Gregg
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
Brendan Gregg
 
YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing Performance
Brendan Gregg
 
LSFMM 2019 BPF Observability
LSFMM 2019 BPF ObservabilityLSFMM 2019 BPF Observability
LSFMM 2019 BPF Observability
Brendan Gregg
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
Brendan Gregg
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
Brendan Gregg
 

What's hot (20)

Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
 
Meet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracingMeet cute-between-ebpf-and-tracing
Meet cute-between-ebpf-and-tracing
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
eBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to UserspaceeBPF Trace from Kernel to Userspace
eBPF Trace from Kernel to Userspace
 
BPF - in-kernel virtual machine
BPF - in-kernel virtual machineBPF - in-kernel virtual machine
BPF - in-kernel virtual machine
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF Superpowers
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
 
eBPF/XDP
eBPF/XDP eBPF/XDP
eBPF/XDP
 
Introduction to eBPF
Introduction to eBPFIntroduction to eBPF
Introduction to eBPF
 
Linux kernel tracing
Linux kernel tracingLinux kernel tracing
Linux kernel tracing
 
New Ways to Find Latency in Linux Using Tracing
New Ways to Find Latency in Linux Using TracingNew Ways to Find Latency in Linux Using Tracing
New Ways to Find Latency in Linux Using Tracing
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
 
YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing Performance
 
LSFMM 2019 BPF Observability
LSFMM 2019 BPF ObservabilityLSFMM 2019 BPF Observability
LSFMM 2019 BPF Observability
 
Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
 

Similar to Systems@Scale 2021 BPF Performance Getting Started

Efficient System Monitoring in Cloud Native Environments
Efficient System Monitoring in Cloud Native EnvironmentsEfficient System Monitoring in Cloud Native Environments
Efficient System Monitoring in Cloud Native Environments
Gergely Szabó
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
Brendan Gregg
 
Security Monitoring with eBPF
Security Monitoring with eBPFSecurity Monitoring with eBPF
Security Monitoring with eBPF
Alex Maestretti
 
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Jian-Hong Pan
 
Check the version with fixes. Link in description
Check the version with fixes. Link in descriptionCheck the version with fixes. Link in description
Check the version with fixes. Link in description
Przemyslaw Koltermann
 
Getting started with LinuxBoot Firmware on AArch64 Server
Getting started with LinuxBoot Firmware on AArch64 Server Getting started with LinuxBoot Firmware on AArch64 Server
Getting started with LinuxBoot Firmware on AArch64 Server
Naohiro Tamura
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
Brendan Gregg
 
Debugging 2013- Jesper Brouer
Debugging 2013- Jesper BrouerDebugging 2013- Jesper Brouer
Debugging 2013- Jesper Brouer
Mediehuset Ingeniøren Live
 
Docker as an every day work tool
Docker as an every day work toolDocker as an every day work tool
Docker as an every day work tool
Przemyslaw Koltermann
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Valeriy Kravchuk
 
Share the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development BoardShare the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development Board
Jian-Hong Pan
 
Bpf performance tools chapter 4 bcc
Bpf performance tools chapter 4   bccBpf performance tools chapter 4   bcc
Bpf performance tools chapter 4 bcc
Viller Hsiao
 
story_of_bpf-1.pdf
story_of_bpf-1.pdfstory_of_bpf-1.pdf
story_of_bpf-1.pdf
hegikip775
 
Docker 활용법: dumpdocker
Docker 활용법: dumpdockerDocker 활용법: dumpdocker
Docker 활용법: dumpdocker
Jaehwa Park
 
Not breaking userspace: the evolving Linux ABI
Not breaking userspace: the evolving Linux ABINot breaking userspace: the evolving Linux ABI
Not breaking userspace: the evolving Linux ABI
Alison Chaiken
 
Kernel bug hunting
Kernel bug huntingKernel bug hunting
Kernel bug hunting
Andrea Righi
 
Loadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitLoadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkit
Frederic Descamps
 
ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!
Affan Syed
 
Scaling Development Environments with Docker
Scaling Development Environments with DockerScaling Development Environments with Docker
Scaling Development Environments with Docker
Docker, Inc.
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
Brendan Gregg
 

Similar to Systems@Scale 2021 BPF Performance Getting Started (20)

Efficient System Monitoring in Cloud Native Environments
Efficient System Monitoring in Cloud Native EnvironmentsEfficient System Monitoring in Cloud Native Environments
Efficient System Monitoring in Cloud Native Environments
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
 
Security Monitoring with eBPF
Security Monitoring with eBPFSecurity Monitoring with eBPF
Security Monitoring with eBPF
 
Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021Let's trace Linux Lernel with KGDB @ COSCUP 2021
Let's trace Linux Lernel with KGDB @ COSCUP 2021
 
Check the version with fixes. Link in description
Check the version with fixes. Link in descriptionCheck the version with fixes. Link in description
Check the version with fixes. Link in description
 
Getting started with LinuxBoot Firmware on AArch64 Server
Getting started with LinuxBoot Firmware on AArch64 Server Getting started with LinuxBoot Firmware on AArch64 Server
Getting started with LinuxBoot Firmware on AArch64 Server
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
 
Debugging 2013- Jesper Brouer
Debugging 2013- Jesper BrouerDebugging 2013- Jesper Brouer
Debugging 2013- Jesper Brouer
 
Docker as an every day work tool
Docker as an every day work toolDocker as an every day work tool
Docker as an every day work tool
 
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
Dynamic tracing of MariaDB on Linux - problems and solutions (MariaDB Server ...
 
Share the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development BoardShare the Experience of Using Embedded Development Board
Share the Experience of Using Embedded Development Board
 
Bpf performance tools chapter 4 bcc
Bpf performance tools chapter 4   bccBpf performance tools chapter 4   bcc
Bpf performance tools chapter 4 bcc
 
story_of_bpf-1.pdf
story_of_bpf-1.pdfstory_of_bpf-1.pdf
story_of_bpf-1.pdf
 
Docker 활용법: dumpdocker
Docker 활용법: dumpdockerDocker 활용법: dumpdocker
Docker 활용법: dumpdocker
 
Not breaking userspace: the evolving Linux ABI
Not breaking userspace: the evolving Linux ABINot breaking userspace: the evolving Linux ABI
Not breaking userspace: the evolving Linux ABI
 
Kernel bug hunting
Kernel bug huntingKernel bug hunting
Kernel bug hunting
 
Loadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkitLoadays managing my sql with percona toolkit
Loadays managing my sql with percona toolkit
 
ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!ebpf and IO Visor: The What, how, and what next!
ebpf and IO Visor: The What, how, and what next!
 
Scaling Development Environments with Docker
Scaling Development Environments with DockerScaling Development Environments with Docker
Scaling Development Environments with Docker
 
Velocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPFVelocity 2017 Performance analysis superpowers with Linux eBPF
Velocity 2017 Performance analysis superpowers with Linux eBPF
 

More from Brendan Gregg

LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing Tools
Brendan Gregg
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflix
Brendan Gregg
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at Netflix
Brendan Gregg
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
Brendan Gregg
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF Observability
Brendan Gregg
 
FlameScope 2018
FlameScope 2018FlameScope 2018
FlameScope 2018
Brendan Gregg
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018
Brendan Gregg
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)
Brendan Gregg
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for Performance
Brendan Gregg
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
Brendan Gregg
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
Brendan Gregg
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
Brendan Gregg
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis Methodologies
Brendan Gregg
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
Brendan Gregg
 

More from Brendan Gregg (14)

LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing Tools
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflix
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at Netflix
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF Observability
 
FlameScope 2018
FlameScope 2018FlameScope 2018
FlameScope 2018
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for Performance
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
 
Kernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPFKernel Recipes 2017: Performance Analysis with BPF
Kernel Recipes 2017: Performance Analysis with BPF
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis Methodologies
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
 

Recently uploaded

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Jeffrey Haguewood
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Wask
 

Recently uploaded (20)

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
 

Systems@Scale 2021 BPF Performance Getting Started

  • 1. 1 Brendan Gregg Jun 2021 Performance Wins with BPF Getting Started
  • 2. 2 This article is not for beginners … not the best start … out of date BPF has evolved Many docs were true in 2015 but not today Newcomers keep re-posting old info as new This talk is Jun 2021
  • 3. 3 BPF is no longer an acronym BPF is a bytecode and execution environment
  • 4. 4 How to get quick and easy BPF performance wins
  • 5. Think like a sysadmin Not like a programmer
  • 6. 6 Think like a sysadmin # apt-get install bcc-tools # PATH=$PATH:/usr/share/bcc/tools # execsnoop # opensnoop # tcplife # ext4slower # biosnoop [...] Get it installed everywhere and use it.
  • 7. 7 Think like a sysadmin # apt-get install bcc-tools # PATH=$PATH:/usr/share/bcc/tools # execsnoop # opensnoop # tcplife # ext4slower # biosnoop [...] Get it installed everywhere and use it. Anything periodic running? crontab? Any misconfigurations? File not found? Any unexpected TCP sessions? Any file system I/O slower than 10ms? Any unusual disk access patterns? Outliers?
  • 8. 8 Case Study: BCC biosnoop # iostat -xz 1 Linux 4.15.0-1052-aws (cass-xxx) 12/04/2019 _x86_64_ (8 CPU) […] avg-cpu: %user %nice %system %iowait %steal %idle 4.67 0.13 0.25 0.00 0.00 94.95 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util xvda 0.00 0.00 1.00 0.00 8.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 xvdb 0.00 0.00 32.00 0.00 152.00 0.00 9.50 0.16 5.00 5.00 0.00 5.00 16.00 xvdc 0.00 0.00 36.00 1.00 192.00 4.00 10.59 0.19 5.19 5.33 0.00 5.19 19.20 xvdd 4.00 0.00 37.00 0.00 181.00 0.00 9.78 0.18 4.97 4.97 0.00 4.54 16.80 xvde 0.00 0.00 27.00 0.00 156.00 0.00 11.56 0.18 6.67 6.67 0.00 6.67 18.00 xvdf 0.00 0.00 35.00 0.00 164.00 0.00 9.37 0.19 5.37 5.37 0.00 5.37 18.80 xvdg 0.00 0.00 25.00 1.00 136.00 4.00 10.77 0.14 5.23 5.44 0.00 5.23 13.60 md0 0.00 0.00 195.00 2.00 965.00 8.00 9.88 0.00 0.00 0.00 0.00 0.00 0.00 […]
  • 9. 9 Case Study: BCC biosnoop, cont. # /usr/share/bcc/tools/biosnoop TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms) 0.000000 perl 7755 xvdc R 610822184 4096 8.57 0.000812 biosnoop 32196 xvda R 269480 4096 0.43 0.006197 perl 3285 xvde R 610737856 4096 6.10 0.006390 perl 23937 xvde R 377704624 4096 0.10 0.015040 perl 7755 xvdb R 732825200 4096 8.51 0.022842 perl 3285 xvdc R 732953880 4096 7.72 0.023019 perl 23937 xvdb R 377707064 4096 0.09 0.034443 perl 7755 xvdg R 732998328 4096 11.28 0.039648 perl 23937 xvdd R 733127392 4096 5.08 0.039863 perl 31913 xvdg R 732868048 4096 0.10 0.049431 perl 3285 xvdg R 732906896 4096 9.45 0.058521 perl 27565 xvdg R 610744920 4096 8.99 0.070843 perl 27565 xvdg R 377706520 4096 12.26 0.080564 perl 31913 xvdc R 610951744 4096 9.62 0.080804 perl 7755 xvdc R 732858664 4096 0.14 0.086932 perl 27565 xvdg R 732937416 4096 6.01 0.087093 perl 27565 xvde R 610853240 4096 0.09 […]
  • 10. 10 Case Study: BCC biosnoop, cont. # ps -ef | grep perl root 3285 3274 1 14:16 ? 00:04:24 /usr/bin/perl /apps/…ec2rotatelogs.pl root 7755 7748 1 04:16 ? 00:10:20 /usr/bin/perl /apps/…ec2rotatelogs.pl root 11366 11359 1 10:16 ? 00:06:39 /usr/bin/perl /apps/…ec2rotatelogs.pl root 15054 15049 2 16:16 ? 00:03:07 /usr/bin/perl /apps/…ec2rotatelogs.pl root 19675 19670 1 06:16 ? 00:08:53 /usr/bin/perl /apps/…ec2rotatelogs.pl root 23937 23930 1 12:16 ? 00:05:30 /usr/bin/perl /apps/…ec2rotatelogs.pl root 27565 27561 2 18:16 ? 00:00:28 /usr/bin/perl /apps/…ec2rotatelogs.pl root 28232 28223 1 02:16 ? 00:11:43 /usr/bin/perl /apps/…ec2rotatelogs.pl root 31913 31907 1 08:15 ? 00:07:40 /usr/bin/perl /apps/…ec2rotatelogs.pl […]
  • 11. Many more tools to try! bcc tools bpftrace tools (from my book, open source) Solve >90% of perf issues with canned observability (tracing) tools This is BPF observability in one pic
  • 12. Print on your office/home wall, use as a checklist Suspected disk issue? Try these first:
  • 13. Print on your office/home wall, use as a checklist Suspected disk issue? Try these first: Then walk up the stack: file system tools, VFS tools, syscall tools, etc.
  • 14. 14 The future of BPF perf observability … is GUIs. The end user may not even know it’s BPF. Tool output, visualized This GUI is in development by Susie Xia, Netflix
  • 15. 15 bcc tools Instance agent Server Application bcc tools Instance agent Server Application Client GUI GUI Webpage Web Server Example real-time BPF observability UI Netflix Vector (now retired) uses this model ● Instance agent pcp pmdabcc ● https://github.com/Netflix/vector
  • 16. 16 Instance agent Server Application Instance agent Server Application Client GUI GUI Webpage Web Server Example real-time BPF observability UI #2 bpftrace tools bpf- trace bpf- trace Netflix FlameCommander UI ● (not yet open source)
  • 17. 17 Many tools are sandcastles, and require frequent rebuilding to match kernel changes Fortunately many companies and engineers maintain these versions Think like a sysadmin Please try to use bcc/bpftrace tools as-is and fetch regular updates bcc & bpftrace tools Facebook Isovalent Netflix (many more)
  • 18. 18 Sysadmins sometimes program shell scripting awk sed one-liners bpftrace tools bpftrace one-liners
  • 19. 19 Think like a programmer if You have a real-world problem that tools don’t solve You’re a BPF-based startup You’re debugging your own code* You’re doing networking/security/etc. You really want to learn BPF internals * JIT-ed runtimes like Java are currently complex to trace
  • 20. 20 Recommended tracing front-ends I want to run some tools ● bcc, bpftrace I want to hack up some new tools ● bpftrace I want to spend weeks developing a BPF product ● bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs Requires LLVM; becoming obsolete / special-use only New, lightweight, CO-RE & BTF based
  • 21. 21 Recommended tracing front-ends I want to run some tools ● bcc, bpftrace /usr/bin/* I want to hack up some new tools ● bpftrace bash, awk I want to spend weeks developing a BPF product ● bcc libbpf C, bcc Python (maybe), gobpf, libbbpf-rs C, C++ Unix analogies Requires LLVM; becoming obsolete / special-use only New, lightweight, CO-RE & BTF based
  • 22. 22 bpftrace example # readahead.bt Attaching 5 probes... ^C Readahead unused pages: 128 Readahead used page age (ms): @age_ms: [1] 2455 |@@@@@@@@@@@@@@@ | [2, 4) 8424 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4, 8) 4417 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8, 16) 7680 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [16, 32) 4352 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32, 64) 0 | | [64, 128) 0 | | [128, 256) 384 |@@ |
  • 23. 23 #!/usr/local/bin/bpftrace kprobe:__do_page_cache_readahead { @in_readahead[tid] = 1; } kretprobe:__do_page_cache_readahead { @in_readahead[tid] = 0; } kretprobe:__page_cache_alloc /@in_readahead[tid]/ { @birth[retval] = nsecs; @rapages++; } kprobe:mark_page_accessed /@birth[arg0]/ { @age_ms = hist((nsecs - @birth[arg0]) / 1000000); delete(@birth[arg0]); @rapages--; } END { printf("nReadahead unused pages: %dn", @rapages); printf("nReadahead used page age (ms):n"); print(@age_ms); clear(@age_ms); clear(@birth); clear(@in_readahead); clear(@rapages); } Fits on one slide! bpftrace example
  • 24. 24 BCC libbpf tool example # ./opensnoop PID COMM FD ERR PATH 27974 opensnoop 28 0 /etc/localtime 1482 redis-server 7 0 /proc/1482/stat […] # ldd opensnoop linux-vdso.so.1 (0x00007ffddf3f1000) libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f9fb7836000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9fb7619000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fb7228000) /lib64/ld-linux-x86-64.so.2 (0x00007f9fb7c76000) # ls -lh opensnoop opensnoop.stripped -rwxr-xr-x 1 root root 645K Feb 28 23:18 opensnoop -rwxr-xr-x 1 root root 151K Feb 28 23:33 opensnoop.stripped 151 Kbytes for a stand-alone BPF program! (Note: A static bpftrace/BTF + scripts will also have a small average tool size)
  • 25. 25 PSA CONFIG_DEBUG_INFO_BTF=y E.g., Ubuntu 20.10, Fedora 30, and RHEL 8.2 have it
  • 26. 26 Kernel User-mode Applications Hardware Events (incl. clock) BPF Future: Event-based Applications Kernel-mode Applications (BPF) Scheduler Kernel Events U.E.
  • 27. 27 A New Type of Software Execution model User defined Compil- ation Security Failure mode Resource access User task yes any user based abort syscall, fault Kernel task no static none panic direct BPF event yes JIT, CO-RE verified, JIT error message restricted helpers
  • 28. 28 Take Away To get started with BPF performance wins, think like a sysadmin: 1. Install BCC & bpftrace tools 2. Run them 3. Get some wins
  • 29. 29 References This is also where I recommend you go to learn more: ● https://github.com/iovisor/bcc/blob/master/docs/tutorial.md ● https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md ● BPF Performance Tools, Addison Wesley 2020 ● Systems Performance 2nd Edition, Addison Wesley 2021 ● http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html ● http://www.brendangregg.com/ebpf.html ● https://ebpf.io/what-is-ebpf
  • 30. 30 Thanks BPF: Alexei Starovoitov (Facebook), Daniel Borkmann (Isovalent), David S. Miller (Red Hat), Jakub Kicinski (Facebook), Yonghong Song (Facebook), Martin KaFai Lau (Facebook), John Fastabend (Isovalent), Quentin Monnet (Isovalent), Jesper Dangaard Brouer (Red Hat), Andrey Ignatov (Facebook), and Stanislav Fomichev (Google), Linus Torvalds, and many more in the BPF community BCC: Brenden Blanco (VMware), Yonghong Song, Sasha Goldsthein (Google), Teng Qin (Facebook), Paul Chaignon (Isovalent), Vicent Martí (PlanetScale), and many more in the BCC community bpftrace: Alastair Robertson (Yellowbrick Data), Dan Xu (Facebook), Bas Smit, Mary Marchini (Netflix), Masanori Misono, Jiri Olsa, Viktor Malík, Dale Hamel, Willian Gaspar, Augusto Mecking Caringi, and many more in the bpftrace community