0
Velocity 2012 / 2012-06-26   RUM for Breakfast   1
RUM for BreakfastBuddy Brewer, Carlos Bueno, Philip Tellis            Velocity 2012 / 2012-06-26Velocity 2012 / 2012-06-26...
Real Users             Velocity 2012 / 2012-06-26   RUM for Breakfast   3
Real Browsers            Velocity 2012 / 2012-06-26   RUM for Breakfast   4
https://github.com/lognormal/boomerang/            Velocity 2012 / 2012-06-26   RUM for Breakfast   5
Carlos built the dns plugin             Velocity 2012 / 2012-06-26   RUM for Breakfast   6
Buddy built the navtiming plugin             Velocity 2012 / 2012-06-26   RUM for Breakfast   7
tl;dr        1   Measure a bunch of stuff in the browser        2   Use high school stats that we vaguely remember        ...
1                    MeasureVelocity 2012 / 2012-06-26   RUM for Breakfast   9
2                     AnalyzeVelocity 2012 / 2012-06-26   RUM for Breakfast   10
Log-Normal Distribution            Velocity 2012 / 2012-06-26   RUM for Breakfast   11
Log-Normal Distribution     The logarithm of the x-axis follows a Normal distribution             Velocity 2012 / 2012-06-...
Log-Normal Distribution    Use the Geometric Mean for pure Log-Normal distributions             Velocity 2012 / 2012-06-26...
Log-Normal Distribution   Performance data does not always follow a "pure" Log-Normal                          distributio...
Look at the entire spread                             ...Velocity 2012 / 2012-06-26   RUM for Breakfast   14
Look at the entire spreadwhich often approaches an infinite widthVelocity 2012 / 2012-06-26   RUM for Breakfast   14
Distill          Velocity 2012 / 2012-06-26   RUM for Breakfast   15
• 0.8% of hits are fake/abusive• 0.2-0.5% of hits are from a stale cache• 0.1% of hits are absurd• Timestamps in the futur...
Even with beacons, you need to sanitize your input    Velocity 2012 / 2012-06-26   RUM for Breakfast   17
Band-pass filtering            Velocity 2012 / 2012-06-26   RUM for Breakfast   18
Band-pass filtering     • Strip everything outside a reasonable range          • Bandwidth range: 4kbps - 4Gbps          • ...
IQR filtering               Velocity 2012 / 2012-06-26   RUM for Breakfast   19
IQR filtering                       Derive the range from the data               Velocity 2012 / 2012-06-26   RUM for Break...
Sampling           Velocity 2012 / 2012-06-26   RUM for Breakfast   20
Margin of Error                                               σ                                         ±1.96 √n          ...
MoE & Sample size   There is an inverse square root correlation between sample size                         and margin of ...
How big a sample is representative?                              Select nsuch that                                   σ    ...
This needs to be at your lowest drilldown level  Velocity 2012 / 2012-06-26   RUM for Breakfast   24
3                       InsightVelocity 2012 / 2012-06-26   RUM for Breakfast   25
How does performanceimpact human behavior?
8 million pages1.5 million visits50 different dimensions
very fast sessions had high bounce rates70.00%52.50%35.00%17.50%   0%         0   2    4   6   8   10   12   14   16   18 ...
bounce rate vs. load time70.00%52.50%35.00%17.50%   0%         1   3   5   7    9   11   13   15   17   19   21   23   25 ...
bounce rate vs. DOM interactive70.00%52.50%35.00%17.50%   0%         1   1.5   2   2.5   3    3.5   4   4.5   5   5.5   6 ...
bounce rate vs. front end time80.00%60.00%40.00%20.00%   0%         0.5   2   3.5   5   6.5   8   9.5   11   12.5   14   1...
is my web site performance toxic to my                   users?http://www.flickr.com/photos/21560098@N06/3796822070
LD50 - when do half the users bounce?http://www.flickr.com/photos/thecosmopolitan/6117530924
Bounce rate =50% Back end time    1.7 sec DOM Loading      1.8 secDOM Interactive   2.75 secFront end time    3.5 secDOM C...
Future directionsWhat is the LD50 for your site?Other bounce rates? 40%? 30%?Other variables? (critical contentvisible, et...
Thank youVelocity 2012 / 2012-06-26   RUM for Breakfast   26
Questions?Buddy Brewer @bbrewer      Carlos Bueno @archivdPhilip Tellis @bluesmoon
Photo credits     • Rum on Ice – wiserbailey on flickr     • Laptop-top Cat – wabisabi2015 on flickr     • About NCSA Mosaic...
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Rum for Breakfast
Upcoming SlideShare
Loading in...5
×

Rum for Breakfast

2,566

Published on

Real user monitoring is one of the best ways of learning “the truth” about what visitors experience on your web site, but it comes at a cost. The real world is messy and noisy making it hard to know exactly what’s going on. Filtering your data, splitting it along multiple dimensions, and determining what to discard are important second steps on the path to insightful RUM analysis, and in this session, we’ll go into some of the details.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,566
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Rum for Breakfast"

  1. 1. Velocity 2012 / 2012-06-26 RUM for Breakfast 1
  2. 2. RUM for BreakfastBuddy Brewer, Carlos Bueno, Philip Tellis Velocity 2012 / 2012-06-26Velocity 2012 / 2012-06-26 RUM for Breakfast 2
  3. 3. Real Users Velocity 2012 / 2012-06-26 RUM for Breakfast 3
  4. 4. Real Browsers Velocity 2012 / 2012-06-26 RUM for Breakfast 4
  5. 5. https://github.com/lognormal/boomerang/ Velocity 2012 / 2012-06-26 RUM for Breakfast 5
  6. 6. Carlos built the dns plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 6
  7. 7. Buddy built the navtiming plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 7
  8. 8. tl;dr 1 Measure a bunch of stuff in the browser 2 Use high school stats that we vaguely remember 3 Randomly invent insights Velocity 2012 / 2012-06-26 RUM for Breakfast 8
  9. 9. 1 MeasureVelocity 2012 / 2012-06-26 RUM for Breakfast 9
  10. 10. 2 AnalyzeVelocity 2012 / 2012-06-26 RUM for Breakfast 10
  11. 11. Log-Normal Distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  12. 12. Log-Normal Distribution The logarithm of the x-axis follows a Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  13. 13. Log-Normal Distribution Use the Geometric Mean for pure Log-Normal distributions Velocity 2012 / 2012-06-26 RUM for Breakfast 12
  14. 14. Log-Normal Distribution Performance data does not always follow a "pure" Log-Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 13
  15. 15. Look at the entire spread ...Velocity 2012 / 2012-06-26 RUM for Breakfast 14
  16. 16. Look at the entire spreadwhich often approaches an infinite widthVelocity 2012 / 2012-06-26 RUM for Breakfast 14
  17. 17. Distill Velocity 2012 / 2012-06-26 RUM for Breakfast 15
  18. 18. • 0.8% of hits are fake/abusive• 0.2-0.5% of hits are from a stale cache• 0.1% of hits are absurd• Timestamps in the future (or past depending on how you interpret it)• Bots ignore robots.txt across domains• "Interesting" caches/copies Velocity 2012 / 2012-06-26 RUM for Breakfast 16
  19. 19. Even with beacons, you need to sanitize your input Velocity 2012 / 2012-06-26 RUM for Breakfast 17
  20. 20. Band-pass filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  21. 21. Band-pass filtering • Strip everything outside a reasonable range • Bandwidth range: 4kbps - 4Gbps • Page load time: 0ms - 600s • You may need to relook at the ranges all the time Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  22. 22. IQR filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  23. 23. IQR filtering Derive the range from the data Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  24. 24. Sampling Velocity 2012 / 2012-06-26 RUM for Breakfast 20
  25. 25. Margin of Error σ ±1.96 √n Velocity 2012 / 2012-06-26 RUM for Breakfast 21
  26. 26. MoE & Sample size There is an inverse square root correlation between sample size and margin of error Velocity 2012 / 2012-06-26 RUM for Breakfast 22
  27. 27. How big a sample is representative? Select nsuch that σ 1.96 √n ≤ 5%µ Velocity 2012 / 2012-06-26 RUM for Breakfast 23
  28. 28. This needs to be at your lowest drilldown level Velocity 2012 / 2012-06-26 RUM for Breakfast 24
  29. 29. 3 InsightVelocity 2012 / 2012-06-26 RUM for Breakfast 25
  30. 30. How does performanceimpact human behavior?
  31. 31. 8 million pages1.5 million visits50 different dimensions
  32. 32. very fast sessions had high bounce rates70.00%52.50%35.00%17.50% 0% 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
  33. 33. bounce rate vs. load time70.00%52.50%35.00%17.50% 0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
  34. 34. bounce rate vs. DOM interactive70.00%52.50%35.00%17.50% 0% 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5
  35. 35. bounce rate vs. front end time80.00%60.00%40.00%20.00% 0% 0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29
  36. 36. is my web site performance toxic to my users?http://www.flickr.com/photos/21560098@N06/3796822070
  37. 37. LD50 - when do half the users bounce?http://www.flickr.com/photos/thecosmopolitan/6117530924
  38. 38. Bounce rate =50% Back end time 1.7 sec DOM Loading 1.8 secDOM Interactive 2.75 secFront end time 3.5 secDOM Complete 4.75 sec Load event 5.5 sec
  39. 39. Future directionsWhat is the LD50 for your site?Other bounce rates? 40%? 30%?Other variables? (critical contentvisible, etc)Other behaviors? Conversions,revenue, pages per session, actions,when do people make tea?
  40. 40. Thank youVelocity 2012 / 2012-06-26 RUM for Breakfast 26
  41. 41. Questions?Buddy Brewer @bbrewer Carlos Bueno @archivdPhilip Tellis @bluesmoon
  42. 42. Photo credits • Rum on Ice – wiserbailey on flickr • Laptop-top Cat – wabisabi2015 on flickr • About NCSA Mosaic – ncsa • Distilled – Lost Albatross on flickr • Anthon Berg Chocolates – ulterior epicure on flickr • KilroySchematic – on WikiPedia • Boxplot vs PDF – on WikiPedia Velocity 2012 / 2012-06-26 RUM for Breakfast 27
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×