1. Linux
Performance
Tools
Brendan Gregg
Senior Performance Architect
Performance Engineering Team
bgregg@netflix.com
@brendangregg

2. This
Tutorial
• A tour of many Linux performance tools
– To show you what can be done
– With guidance for how to do it
• This includes objectives, discussion, live demos
– See the video of this tutorial
Observability
Benchmarking
Tuning
Sta<c
Tuning

3. • Massive AWS EC2 Linux cloud
– 10s of thousands of cloud instances
• FreeBSD for content delivery
– ~33% of US Internet traffic at night
• Over 50M subscribers
– Recently launched in ANZ
• Use Linux server tools as needed
– After cloud monitoring (Atlas, etc.) and
instance monitoring (Vector) tools

4. Agenda
• Methodologies
• Tools
• Tool Types:
– Observability
– Benchmarking
– Tuning
– Static
• Profiling
• Tracing

5. Methodologies

6. Methodologies
• Objectives:
– Recognize the Streetlight Anti-Method
– Perform the Workload Characterization Method
– Perform the USE Method
– Learn how to start with the questions, before using tools
– Be aware of other methodologies

7. My
system
is
slow…
DEMO
&
DISCUSSION

8. Methodologies
• There are dozens of performance tools for Linux
– Packages: sysstat, procps, coreutils, …
– Commercial products
• Methodologies can provide guidance for choosing and
using tools effectively
• A starting point, a process, and an ending point

9. An#-­‐Methodologies
• The lack of a deliberate methodology…

10. Street
Light
An<-­‐Method
1. Pick observability tools that are:
– Familiar
– Found on the Internet
– Found at random
2. Run tools
3. Look for obvious issues

11. Drunk
Man
An<-­‐Method
• Tune things at random until the problem goes away

12. Blame
Someone
Else
An<-­‐Method
1. Find a system or environment component you are not
responsible for
2. Hypothesize that the issue is with that component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1

13. Actual
Methodologies
• Problem Statement Method
• Workload Characterization Method
• USE Method
• Off-CPU Analysis
• CPU Profile Method
• RTFM Method
• Active Benchmarking (covered later)
• Static Performance Tuning (covered later)
• …

14. Problem
Statement
Method