Successfully reported this slideshow.
Your SlideShare is downloading. ×

RUM for Breakfast - distilling insights from the noise

RUM for Breakfast - distilling insights from the noise

Download to read offline

From Velocity 2012 in Santa Clara, CA. Buddy Brewer, Philip Tellis, and Carlos Bueno talk about real user measurement collection, analysis, and insights.

From Velocity 2012 in Santa Clara, CA. Buddy Brewer, Philip Tellis, and Carlos Bueno talk about real user measurement collection, analysis, and insights.

More Related Content

Related Audiobooks

Free with a 30 day trial from Scribd

See all

RUM for Breakfast - distilling insights from the noise

  1. 1. Velocity 2012 / 2012-06-26 RUM for Breakfast 1
  2. 2. RUM for Breakfast Buddy Brewer, Carlos Bueno, Philip Tellis Velocity 2012 / 2012-06-26 Velocity 2012 / 2012-06-26 RUM for Breakfast 2
  3. 3. Real Users Velocity 2012 / 2012-06-26 RUM for Breakfast 3
  4. 4. Real Browsers Velocity 2012 / 2012-06-26 RUM for Breakfast 4
  5. 5. https://github.com/lognormal/boomerang/ Velocity 2012 / 2012-06-26 RUM for Breakfast 5
  6. 6. Carlos built the dns plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 6
  7. 7. Buddy built the navtiming plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 7
  8. 8. tl;dr 1 Measure a bunch of stuff in the browser 2 Use high school stats that we vaguely remember 3 Randomly invent insights Velocity 2012 / 2012-06-26 RUM for Breakfast 8
  9. 9. 1 Measure Velocity 2012 / 2012-06-26 RUM for Breakfast 9
  10. 10. 2 Analyze Velocity 2012 / 2012-06-26 RUM for Breakfast 10
  11. 11. Log-Normal Distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  12. 12. Log-Normal Distribution The logarithm of the x-axis follows a Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  13. 13. Log-Normal Distribution Use the Geometric Mean for pure Log-Normal distributions Velocity 2012 / 2012-06-26 RUM for Breakfast 12
  14. 14. Log-Normal Distribution Performance data does not always follow a "pure" Log-Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 13
  15. 15. Look at the entire spread ... Velocity 2012 / 2012-06-26 RUM for Breakfast 14
  16. 16. Look at the entire spread which often approaches an infinite width Velocity 2012 / 2012-06-26 RUM for Breakfast 14
  17. 17. Distill Velocity 2012 / 2012-06-26 RUM for Breakfast 15
  18. 18. • 0.8% of hits are fake/abusive • 0.2-0.5% of hits are from a stale cache • 0.1% of hits are absurd • Timestamps in the future (or past depending on how you interpret it) • Bots ignore robots.txt across domains • "Interesting" caches/copies Velocity 2012 / 2012-06-26 RUM for Breakfast 16
  19. 19. Even with beacons, you need to sanitize your input Velocity 2012 / 2012-06-26 RUM for Breakfast 17
  20. 20. Band-pass filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  21. 21. Band-pass filtering • Strip everything outside a reasonable range • Bandwidth range: 4kbps - 4Gbps • Page load time: 0ms - 600s • You may need to relook at the ranges all the time Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  22. 22. IQR filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  23. 23. IQR filtering Derive the range from the data Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  24. 24. Sampling Velocity 2012 / 2012-06-26 RUM for Breakfast 20
  25. 25. Margin of Error σ ±1.96 √n Velocity 2012 / 2012-06-26 RUM for Breakfast 21
  26. 26. MoE & Sample size There is an inverse square root correlation between sample size and margin of error Velocity 2012 / 2012-06-26 RUM for Breakfast 22
  27. 27. How big a sample is representative? Select nsuch that σ 1.96 √n ≤ 5%µ Velocity 2012 / 2012-06-26 RUM for Breakfast 23
  28. 28. This needs to be at your lowest drilldown level Velocity 2012 / 2012-06-26 RUM for Breakfast 24
  29. 29. 3 Insight Velocity 2012 / 2012-06-26 RUM for Breakfast 25
  30. 30. How does performance impact human behavior?
  31. 31. 8 million pages 1.5 million visits 50 different dimensions
  32. 32. very fast sessions had high bounce rates 70.00% 52.50% 35.00% 17.50% 0% 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
  33. 33. bounce rate vs. load time 70.00% 52.50% 35.00% 17.50% 0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
  34. 34. bounce rate vs. DOM interactive 70.00% 52.50% 35.00% 17.50% 0% 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5
  35. 35. bounce rate vs. front end time 80.00% 60.00% 40.00% 20.00% 0% 0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29
  36. 36. is my web site performance toxic to my users? http://www.flickr.com/photos/21560098@N06/3796822070
  37. 37. LD50 - when do half the users bounce? http://www.flickr.com/photos/thecosmopolitan/6117530924
  38. 38. Bounce rate =50% Back end time 1.7 sec DOM Loading 1.8 sec DOM Interactive 2.75 sec Front end time 3.5 sec DOM Complete 4.75 sec Load event 5.5 sec
  39. 39. Future directions What is the LD50 for your site? Other bounce rates? 40%? 30%? Other variables? (critical content visible, etc) Other behaviors? Conversions, revenue, pages per session, actions, when do people make tea?
  40. 40. Numbers don’t lie Velocity 2012 / 2012-06-26 RUM for Breakfast 26
  41. 41. Questions? Buddy Brewer @bbrewer Carlos Bueno @archivd Philip Tellis @bluesmoon
  42. 42. Thank you Velocity 2012 / 2012-06-26 RUM for Breakfast 27

×