RUM for Breakfast - distilling insights from the noise

1,730 views

Published on

From Velocity 2012 in Santa Clara, CA. Buddy Brewer, Philip Tellis, and Carlos Bueno talk about real user measurement collection, analysis, and insights.

Published in: Technology, Travel
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,730
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

RUM for Breakfast - distilling insights from the noise

  1. 1. Velocity 2012 / 2012-06-26 RUM for Breakfast 1
  2. 2. RUM for BreakfastBuddy Brewer, Carlos Bueno, Philip Tellis Velocity 2012 / 2012-06-26Velocity 2012 / 2012-06-26 RUM for Breakfast 2
  3. 3. Real Users Velocity 2012 / 2012-06-26 RUM for Breakfast 3
  4. 4. Real Browsers Velocity 2012 / 2012-06-26 RUM for Breakfast 4
  5. 5. https://github.com/lognormal/boomerang/ Velocity 2012 / 2012-06-26 RUM for Breakfast 5
  6. 6. Carlos built the dns plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 6
  7. 7. Buddy built the navtiming plugin Velocity 2012 / 2012-06-26 RUM for Breakfast 7
  8. 8. tl;dr 1 Measure a bunch of stuff in the browser 2 Use high school stats that we vaguely remember 3 Randomly invent insights Velocity 2012 / 2012-06-26 RUM for Breakfast 8
  9. 9. 1 MeasureVelocity 2012 / 2012-06-26 RUM for Breakfast 9
  10. 10. 2 AnalyzeVelocity 2012 / 2012-06-26 RUM for Breakfast 10
  11. 11. Log-Normal Distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  12. 12. Log-Normal Distribution The logarithm of the x-axis follows a Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 11
  13. 13. Log-Normal Distribution Use the Geometric Mean for pure Log-Normal distributions Velocity 2012 / 2012-06-26 RUM for Breakfast 12
  14. 14. Log-Normal Distribution Performance data does not always follow a "pure" Log-Normal distribution Velocity 2012 / 2012-06-26 RUM for Breakfast 13
  15. 15. Look at the entire spread ...Velocity 2012 / 2012-06-26 RUM for Breakfast 14
  16. 16. Look at the entire spreadwhich often approaches an infinite widthVelocity 2012 / 2012-06-26 RUM for Breakfast 14
  17. 17. Distill Velocity 2012 / 2012-06-26 RUM for Breakfast 15
  18. 18. • 0.8% of hits are fake/abusive• 0.2-0.5% of hits are from a stale cache• 0.1% of hits are absurd• Timestamps in the future (or past depending on how you interpret it)• Bots ignore robots.txt across domains• "Interesting" caches/copies Velocity 2012 / 2012-06-26 RUM for Breakfast 16
  19. 19. Even with beacons, you need to sanitize your input Velocity 2012 / 2012-06-26 RUM for Breakfast 17
  20. 20. Band-pass filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  21. 21. Band-pass filtering • Strip everything outside a reasonable range • Bandwidth range: 4kbps - 4Gbps • Page load time: 0ms - 600s • You may need to relook at the ranges all the time Velocity 2012 / 2012-06-26 RUM for Breakfast 18
  22. 22. IQR filtering Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  23. 23. IQR filtering Derive the range from the data Velocity 2012 / 2012-06-26 RUM for Breakfast 19
  24. 24. Sampling Velocity 2012 / 2012-06-26 RUM for Breakfast 20
  25. 25. Margin of Error σ ±1.96 √n Velocity 2012 / 2012-06-26 RUM for Breakfast 21
  26. 26. MoE & Sample size There is an inverse square root correlation between sample size and margin of error Velocity 2012 / 2012-06-26 RUM for Breakfast 22
  27. 27. How big a sample is representative? Select nsuch that σ 1.96 √n ≤ 5%µ Velocity 2012 / 2012-06-26 RUM for Breakfast 23
  28. 28. This needs to be at your lowest drilldown level Velocity 2012 / 2012-06-26 RUM for Breakfast 24
  29. 29. 3 InsightVelocity 2012 / 2012-06-26 RUM for Breakfast 25
  30. 30. How does performanceimpact human behavior?
  31. 31. 8 million pages1.5 million visits50 different dimensions
  32. 32. very fast sessions had high bounce rates70.00%52.50%35.00%17.50% 0% 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
  33. 33. bounce rate vs. load time70.00%52.50%35.00%17.50% 0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
  34. 34. bounce rate vs. DOM interactive70.00%52.50%35.00%17.50% 0% 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5 11 11.5 12 12.5
  35. 35. bounce rate vs. front end time80.00%60.00%40.00%20.00% 0% 0.5 2 3.5 5 6.5 8 9.5 11 12.5 14 15.5 17 18.5 20 21.5 23 24.5 26 27.5 29
  36. 36. is my web site performance toxic to my users?http://www.flickr.com/photos/21560098@N06/3796822070
  37. 37. LD50 - when do half the users bounce?http://www.flickr.com/photos/thecosmopolitan/6117530924
  38. 38. Bounce rate =50% Back end time 1.7 sec DOM Loading 1.8 secDOM Interactive 2.75 secFront end time 3.5 secDOM Complete 4.75 sec Load event 5.5 sec
  39. 39. Future directionsWhat is the LD50 for your site?Other bounce rates? 40%? 30%?Other variables? (critical contentvisible, etc)Other behaviors? Conversions,revenue, pages per session, actions,when do people make tea?
  40. 40. Numbers don’t lie Velocity 2012 / 2012-06-26 RUM for Breakfast 26
  41. 41. Questions?Buddy Brewer @bbrewer Carlos Bueno @archivdPhilip Tellis @bluesmoon
  42. 42. Thank youVelocity 2012 / 2012-06-26 RUM for Breakfast 27

×