LISA2010 visualizations

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Performance
Visualizations
Brendan Gregg
Software Engineer
brendan.gregg@joyent.com

USENIX/LISA’10
November, 2010

• G’Day, I’m Brendan
• ... also known as “shouting guy”

2

硬

也會鬧情緒

•

3

• I do performance analysis
• and I’m a DTrace addict

4

•

5

Agenda
•

Performance

•
•

•

Workload Analysis and Resource Monitoring
Understanding available and ideal metrics before ...

Visualizations like these
•

The “rainbow pterodactyl”

Latency

Time

•

7

... which needs quite a bit of explanation

Primary Objectives
•

•

See the value of visualizations

•

8

Consider performance metrics before plotting

Remember key...

Secondary Objectives
•

Consider performance metrics before plotting

•
•

•

Why studying latency is good
... and studyin...

Content based on
•
•

10

“Visualizing System Latency”, Communications of the ACM
July 2010, by Brendan Gregg
and more

Performance
Understanding the metrics before we
visualize them

11

Performance Activities
•

Workload analysis

•
•

Where is the issue?

•

•

Is there an issue? Is an issue real?

Will th...

Workload Analysis
•

Applied during:

•
•

proof of concept testing

•

regression testing

•

benchmarking

•

13

softwa...

Workload Performance Issues
•

•

14

Load

Architecture

Workload Performance Issues
•

Load

•
•

Too much for the system?

•

•

Workload applied

Poorly constructed?

Architect...

Workload Analysis Steps
•

Identify or conﬁrm if a workload has a performance issue

•

•

Locate issue

•

•

Quantify

D...

Quantify

•

17

Finding a performance issue isn’t the problem ... it’s ﬁnding
the issue that matters

bugs.mysql.com “performance”

•

18

bugs.opensolaris.org “performance”

•

19

bugs.mozilla.org: “performance”

•

20

“performance” bugs

•
•

21

... and those are just the known performance bugs
... and usually only of a certain type (arc...

How to Quantify
•

Observation based

•
•

•

Choose a reliable metric
Estimate performance gain from resolving issue

Exp...

Observation based
•

For example:

•
•

Suggestion: replace disks with ﬂash-memory based SSDs, with
an expected latency of...

Experimentation based
•

For example:

•
•

Observed: Application transaction latency average 2 ms

•

24

Experiment: Add...

Metrics to Quantify Performance
•

Choose reliable metrics to quantify performance:

•
•

transactions/second

•

throughp...

Metrics to Quantify Performance
•

Choose reliable metrics to quantify performance:

•
•

transactions/second

•

throughp...

Metrics Availability
•

Ideally (given the luxury of time):

•
•

then see if they exist, or,

•

•

design the desired me...

Assumptions to avoid
•

Available metrics are implemented correctly

•

all software has bugs
•

•

•

sometimes added by ...

Getting technical
•

This will be explained using two examples:

•
•

29

Workload Analysis
Capacity Planning

Example: Workload Analysis
•

Quantifying performance issues with IOPS vs latency

•
•

30

IOPS is commonly presented by ...

IOPS
•

Depends on where the I/O is measured

•

•

app -> library -> syscall -> VFS -> ﬁlesystem -> RAID -> device

Depen...

Some disk IOPS problems
•

IOPS Inﬂation

•
•

Filesystem metadata

•

•

Library or Filesystem prefetch/read-ahead

RAID ...

IOPS example: iostat -xnz 1
•

Consider this disk: 86 IOPS == 99% busy
r/s
86.6

•

%w
0

%b device
99 c1d0

Versus this d...

IOPS example: iostat -xnz 1
•

Consider this disk: 86 IOPS == 99% busy
r/s
86.6

•

%b device
99 c1d0

extended device sta...

Using IOPS to quantify issues
•

to identify

•

•

to locate

•

•

90% of IOPS are random. Is that the problem?

to veri...

Using IOPS to quantify issues
•

to identify

•

•

to locate

•

•

36

90% of IOPS are random. Is that the problem? (dep...

Using latency to quantify issues
•

to identify

•

•

to locate

•

•

90ms of the 100ms is lock contention. Is that the ...

Using latency to quantify issues
•

to identify

•

•

to locate

•

•

38

90ms of the 100ms is lock contention. Is that ...

Latency
•

Time from I/O or transaction request to completion

•

Synchronous latency has a direct impact on performance

...

Turning other metrics into latency
•

Currency converter (* -> ms):

•
•

CPU utilization == code path execution latency

...

Example: Resource Monitoring
•

Different performance activity

•
•

Information is used for capacity planning

•

41

inc...

IOPS vs Utilization
•

Another look at this disk:
r/s
86.6
[...]
r/s
21284.7

•

42

w/s
0.0

extended device statistics
k...

IOPS vs Utilization
•

Another look at this disk:
r/s
86.6
[...]
r/s
21284.7

•

w/s
0.0

extended device statistics
kr/s
...

IOPS vs Utilization
•

Another look at this disk:
r/s
86.6
[...]
r/s
21284.7

•

w/s
0.0

extended device statistics
kr/s
...

Performance Summary
•

Metrics matter - need to reliably quantify performance

•
•

•

to identify, locate, verify
try to ...

Objectives
•

Consider performance metrics before plotting

•
•

•

Why latency is good
... and IOPS can be bad

See the v...

Visualizations
Current Examples
Latency

47

Visualizations
•

So far we’ve picked:

•

Latency

•

•

Utilization

•

48

for workload analysis

for resource monitori...

Latency
•

For example, disk I/O

•

Raw data looks like this:
# iosnoop -o
DTIME
UID
PID
125
100
337
138
100
337
127
100
...

Latency Data
•

tuples

•
•

•

50

I/O completion time
I/O latency

can be 1,000s of these per second

Summarizing Latency
•

iostat(1M) can show per second average:
$ iostat -xnz 1
[...]
r/s
471.0
r/s
631.0

w/s
0.0

r/s
472...

Per second
•

Condenses I/O completion time

•

Almost always a sufﬁcient resolution

•

52

(So far I’ve only had one cas...

Average/second
•

Average loses latency outliers

•

Average loses latency distribution

•

... but not disk distribution:...

Latency outliers
•

Occasional high-latency I/O

•

Can be the sole reason for performance issues

•

Can be lost in an av...

Maximum/second
•
•

can be visualized along with average/second

•

does identify outliers

•

55

iostat(1M) doesn’t show...

Latency distribution
•

Apart from outliers and average, it can be useful to examine
the full proﬁle of latency - all the ...

Latency Distribution Example
•

Using DTrace:

# ./disklatency.d
Tracing... Hit Ctrl-C to end.
^C
sd4 (28,256), us:
value
...

disklatency.d
•

not why we are here, but before someone asks...
#!/usr/sbin/dtrace -s
#pragma D option quiet
dtrace:::BEG...

Latency Distribution Example
# ./disklatency.d
Tracing... Hit Ctrl-C to end.
^C
sd4 (28,256), us:
value
16
32
64
128
256
5...

Column Quantized Visualization
aka “heat map”
•

For example:

•

60

Heat Map: Offset Distribution

•

x-axis: time

•

y-axis: offset

•

z-axis (color scale): I/O count for that time/offset...

Heat Map: Latency Distribution
•

For example:

•
•

y-axis: latency

•

62

x-axis: time

z-axis (color saturation): I/O ...

Heat Map: Latency Distribution
•

... in fact, this is a great example:

•

reads
served
from:

DRAM
disk

DRAM
ﬂash-memor...

Heat Map: Latency Distribution
•

... in fact, this is a great example:

•

reads
served
from:

DRAM
disk

DRAM
ﬂash-memor...

Latency Heat Map
•

A color shaded matrix of pixels

•

Each pixel is a time and latency range

•

Color shade picked base...

Pixel Size
•

Large pixels (and corresponding time/latency ranges)

•

•

•

increases likelyhood that adjacent pixels
inc...

Color Palette
•

Linear scale can make subtle details (outliers) difﬁcult to see

•
•

outliers are usually < 1% of the I/...

Outliers
•

Heat maps show these very well

•

However, latency outliers can
compress the bulk of the heat
map data

•

•
...

Data Storage
•

Since heat-maps are three dimensions, storing this data can
become costly (volume)

•

Most of the data po...

Other Interesting Latency Heat Maps
•
•

The “Rainbow Pterodactyl”

•

70

The “Icy Lake”

Latency Levels

The “Icy Lake” Workload
•

About as simple as it gets:

•
•

•

71

Single client, single thread, sequential synchronous 8...

The “Icy Lake”

•

72

“Icy Lake” Analysis: Observation
•

Examining single disk latency:

•

Pattern match with NFS latency: similar lines

•
73...

Pattern Match?
•

We just associated NFS latency with disk device latency,
using our eyeballs

•

see the titles on the pr...

“Icy Lake” Analysis: Experimentation
•

Same workload, single disk pool:

•

No diagonal lines

•

75

but more questions ...

“Icy Lake” Analysis: Experimentation
•

Same workload, two disk striped pool:

•

Ah-hah! Diagonal lines.

•

76

... but ...

“Icy Lake” Analysis: Experimentation
•

... each disk from that pool:

•

77

“Icy Lake” Analysis: Questions
•

Remaining Questions:

•
•

78

Why does the slope sometimes change?
What exactly seeds t...

“Icy Lake” Analysis: Mirroring
•

79

Trying mirroring the pool disks instead of striping:

Another Example: “X marks the spot”

80

The “Rainbow Pterodactyl” Workload
•

48 x 7,200 RPM disks, 2 disk enclosures

•

Sequential 128 Kbyte reads to each disk ...

The “Rainbow Pterodactyl”

82

The “Rainbow Pterodactyl”

83

The “Rainbow Pterodactyl”
Buldge

Beak
84

Head

Wing

Neck

Shoulders

Body

The “Rainbow Pterodactyl”: Analysis
•

Hasn’t been understood in detail

•

•

85

Would never be understood (or even know...

The “Rainbow Pterodactyl”: Theories
•
•

“Head”: 9th disk, contention on the 2 x4 SAS ports

•

“Buldge”: ?

•

“Neck”: ?
...

Latency Levels Workload
•
•

87

Same as “Rainbow Pterodactyl”, stepping disks
Instead of sequential reads, this is repeat...

Latency Levels

•

88

Latency Levels Theories
•

89

???

Bonus Latency Heat Map

•

90

This time we do know the source of the latency...

硬

也會鬧情緒

•

91

Latency Heat Maps: Summary
•

Shows latency distribution over time

•

Shows outliers (maximums)

•

Indirectly shows aver...

Similar Heat Map Uses
•

These all have a dynamic y-axis scale:

•
•

•

93

I/O size
I/O offset

These aren’t a primary m...

Heat Map: I/O Offset

•

94

y-axis: I/O offset (in this case, NFSv3 ﬁle location)

Heat Map: I/O Size

•

95

y-axis: I/O size (bytes)

Heat Map Abuse
•

96

What can we ‘paint’ by adjusting the workload?

I/O Size

•

97

How was this done?

I/O Offset

•

98

How was this done?

I/O Latency

•

99

How was this done?

Visualizations
Current Examples
Utilization

100

CPU Utilization
•

Commonly used indicator of CPU performance

•

eg, vmstat(1M)

$ vmstat 1 5
kthr
memory
r b w
swap free...

CPU Utilization: Line Graph
•

102

Easy to plot:

CPU Utilization: Line Graph
•

Easy to plot:

•

Average across all CPUs:

•

103

Identiﬁes how utilized all CPUs are, in...

CPU Utilization by CPU
•

mpstat(1M) can show utilization by-CPU:

$ mpstat 1
[...]
CPU minf mjf xcal
0
0
0
2
1
0
0
0
2
0
...

CPU Resource Monitoring
•

Monitor overall utilization for capacity planning

•

Also valuable to monitor individual CPUs
...

Heat Map: CPU Utilization

100%

0%
60s

•
•

y-axis: percent utilization

•

106

x-axis: time

z-axis (color saturation)...

Heat Map: CPU Utilization
idle CPUs

single ‘hot’ CPU
100%

0%
60s

•
•

107

Single ‘hot’ CPUs are a common problem due t...

Heat Map: Disk Utilization
•

Ditto for disks

•

Disk Utilization heat map can identify:

•
•

unbalanced conﬁgurations

...

Back to Line Graphs...
•

Are typically used to visualize performance, be it IOPS or
utilization

•

Show patterns over ti...

Resource Utilization Heat Map Summary
•

Can exist for any resource with multiple components:

•

CPUs

•

Disks

•

Netwo...

Visualizations
Future Opportunities

111

Cloud Computing
So far analysis has been for a single server

What about the cloud?

112

From one to thousands of servers
Workload Analysis:
latency I/O x cloud
Resource Monitoring:
# of CPUs x cloud
# of disks ...

Heat Maps for the Cloud
•

Heat Maps are promising for cloud computing observability:

•

•

Find outliers regardless of n...

Proposed Visualizations
•

Include:

•
•

Latency heat maps for cloud application components

•

CPU utilization by cloud ...

Cloud Latency Heat Map
•

Latency at different layers:

•
•

PHP/Ruby/...

•

MySQL

•

DNS

•

Disk I/O

•

CPU dispatche...

Latency Example: MySQL
•

Query latency (DTrace):
query time (ns)
value ------------- Distribution ------------- count
102...

Latency Example: MySQL
•

Query latency (DTrace):
query time (ns)
value ------------- Distribution ------------1024 |
2048...

Latency Example: MySQL
query time (ns)
value ------------- Distribution ------------- count
1024 |
0
2048 |
2
4096 |@
99
8...

Latency Example: MySQL
•
•

innodb thread concurrency back-off sleep latency: 8 - 16 ms

•

Both have a similar magnitude ...

Cloud Latency Heat Map
•

Identify latency outliers, distributions, patterns

•

Can add more functionality to identify th...

4th Dimension!
•

Bryan Cantrill @Joyent coded this 11 hours ago

•
•

122

assuming it’s now about 10:30am during this ta...

4th Dimension Example: Thread Runtime
100 ms

0 ms

•
•

y-axis: thread runtime

•

123

x-axis: time

z-axis (color satur...

4th Dimension Example: Thread Runtime
100 ms

0 ms

•
•

z-axis (color saturation): count at that time/runtime range

•

1...

4th Dimension Example: Thread Runtime
100 ms

0 ms

•
•

z-axis (color saturation): count at that time/runtime range

•

1...

4th Dimension Example: Thread Runtime
100 ms

0 ms

•
•

z-axis (color saturation): count at that time/runtime range

•

1...

4th Dimension Example: Thread Runtime
100 ms

0 ms

•
•

z-axis (color saturation): count at that time/runtime range

•

1...

“Dimensionality”
•

While the data supports the 4th dimension, visualizing this
properly may become difﬁcult (we are eager...

More Visualizations
•

We plan much more new stuff

•
•

129

We are building a team of engineers to work on it; including...

Beyond Performance Analysis
•

Visualizations such as heat maps could also be applied to:

•

Security, with pattern match...

Objectives
•

Consider performance metrics before plotting

•
•

•

Why latency is good
... and IOPS can be bad

See the v...

Heat Map: I/O Latency

•

Latency matters

•

•

Heat map shows

•
132

synchronous latency has a direct impact on perform...

Heat Map: CPU Utilization
100%

0%
60s

•

Identify single threaded issues

•

•

Heat map shows

•
133

single CPU hittin...

Tools Demonstrated
•

For Reference:

•

DTraceTazTool

•

•

Analytics

•

•

2008; Oracle Sun ZFS Storage Appliance

“ne...

Question Time
•

Thank you!

•

How to ﬁnd me on the web:

•
•

http://blogs.sun.com/brendan <-- is my old blog

•
135

ht...

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 135 Ad

1 of 135 Ad