The Statistics of Web Performance Analysis

• Philip Tellis

• .com
• philip@lognormal.com
• @bluesmoon
• geek paranoid speedfreak
• http://bluesmoon.info/

Boston #WebPerf Meetup / 2012-08-14 The Statistics of Web Performance Analysis 1

I’m a Web Speedfreak


We measure real user website performance


This talk is about the Statistics we learned while building it


The Statistics of Web Performance Analysis

Philip Tellis / philip@lognormal.com

Boston #WebPerf Meetup / 2012-08-14


0
Numbers


Accurately measure page performance∗


Be unintrusive

If you try to measure something accurately, you will change
something related
– Heisenberg’s uncertainty principle


And one number to rule them all


What do we measure?

• Network Throughput
• Network Latency
• User perceived page load time


We measure real user data


Which is noisy


1
Statistics - 1


Disclaimer

I am not a statistician


1-1 Random Sampling


Population

All possible users of your system


Sample

Representative subset of the population


Bad sample

Sometimes it’s not


How to randomize?

http://xkcd.com/221/


How to randomize?

• Pick 10% of users at random and always test them

OR

• For each user, decide at random if they should be tested

http://tech.bluesmoon.info/2010/01/statistics-of-performance-measurement.html


Select 10% of users - I

if($sessionid % 10 === 0) {
// instrument code for measurement
}

• Once a user enters the measurement bucket, they stay
there until they log out
• Fixed set of users, so tests may be more consistent
• Error in the sample results in positive feedback


Select 10% of users - II

if(rand() < 0.1 * getrandmax()) {
// instrument code for measurement
}

• For every request, a user has a 10% chance of being
tested
• Gets rid of positive feedback errors, but sample size !=
10% of population


How big a sample is representative?

Select n such that
σ
1.96 √n ≤ 5%µ


1-2 Margin of Error


Standard Deviation

• Standard deviation tells you the spread of the curve
• The narrower the curve, the more conﬁdent you can be


MoE at 95% conﬁdence

σ
±1.96 √n


MoE & Sample size

There is an inverse square root correlation between sample size
and margin of error


1-3 Central Tendency


One number

• Mean (Arithmetic)
• Good for symmetric curves
• Affected by outliers

Mean(10, 11, 12, 11, 109) = 30


One number

• Median
• Middle value measures central tendency well
• Not trivial to pull out of a DB

Median(10, 11, 12, 11, 109) = 11


One number

• Mode
• Not often used
• Multi-modal distributions suggest problems

Mode(10, 11, 12, 11, 109) = 11


Other numbers

• A percentile point in the distribution: 95th , 98.5th or 99th
• Used to find out the worst user experience
• Makes more sense if you filter data first

P95th (10, 11, 12, 11, 109) = 12


Other means

• Geometric mean
• Good if your data is exponential in nature
(with the tail on the right)

GMean(10, 11, 12, 11, 109) = 16.68


Wait... how did I get that?

N
ΠN xi — could lead to overﬂow
i=1

ΣN loge (xi )
i=1
N
e — computationally simpler


Other means

And there is also the Harmonic mean, but forget about that


...though consequently

We have other margins of error
• Geometric margin of error
• Uses geometric standard deviation
• Median margin of error
• Uses ranges of actual values from data set
• Stick to the arithmetic MoE
– simpler to calculate, simpler to read and not incorrect


2
Statistics - 2


2-1 Distributions


Let’s look at some real charts


Sparse Distribution


Log-normal distribution


Bimodal distribution


What does all of this mean?


Distributions

• Sparse distribution suggests that you don’t have enough
data points
• Log-normal distribution is typical
• Bi-modal distribution suggests two (or more) distributions
combined


In practice, a bi-modal distribution is not uncommon


Hint: Does your site do a lot of back-end caching?


2-2 Filtering


Outliers

• Out of range data points
• Nothing you can ﬁx here
• There’s even a book about
them


DNS problems can cause outliers

• 2 or 3 DNS servers for an ISP
• 30 second timeout if ﬁrst fails
• ... 30 second increase in page load time
• Maybe measure both and ﬁx what you can
• http://nms.lcs.mit.edu/papers/dns-ton2002.pdf


Band-pass ﬁltering


Band-pass ﬁltering

• Strip everything outside a reasonable range
• Bandwidth range: 4kbps - 4Gbps
• Page load time: 50ms - 120s
• You may need to relook at the ranges all the time


IQR ﬁltering


IQR ﬁltering

Here, we derive the range from the data


Further Reading

lognormal.com/blog/2012/08/13/analysing-performance-data/


Summary

• Choose a reasonable sample size and sampling factor
• Tune sample size for minimal margin of error
• Decide based on your data whether to use mode, median
or one of the means
• Figure out whether your data is Normal, Log-Normal or
something else
• Filter out anomalous outliers


• Philip Tellis

• .com
• philip@lognormal.com
• @bluesmoon
• geek paranoid speedfreak
• http://bluesmoon.info/


Thank you


Photo credits

• http://www.flickr.com/photos/leoffreitas/332360959/ by leoffreitas
• http://www.flickr.com/photos/cobalt/56500295/ by cobalt123
• http://www.flickr.com/photos/sophistechate/4264466015/ by Lisa
Brewster


List of ﬁgures

• http://en.wikipedia.org/wiki/File:Standard_deviation_diagram.svg
• http://en.wikipedia.org/wiki/File:Normal_Distribution_PDF.svg
• http://en.wikipedia.org/wiki/File:KilroySchematic.svg
• http://en.wikipedia.org/wiki/File:Boxplot_vs_PDF.png


The Statistics of Web Performance Analysis

More Related Content

What's hot

Viewers also liked

Similar to The Statistics of Web Performance Analysis

More from Philip Tellis

Recently uploaded

The Statistics of Web Performance Analysis