MeasureWorks - Velocity Europe - Real World Rum

1,548 views

Published on

Slides from Velocity Conference Europe 2013 from my joint session with @bbrewer (Soasta). The talk covers implementation, analysis techniques and how you as the performance team can use RUM data to persuade and justify investments in changes that lead to improved performance....

Published in: Design, Technology, Business
  • Be the first to comment

MeasureWorks - Velocity Europe - Real World Rum

  1. 1. Real World RUM
  2. 2. @jeroentjepkema Founder & CEO
  3. 3. Eliminate the noise, create insights...
  4. 4. What type of performance data am I?
  5. 5. Right now? Everything working? Is it me or the internet? What’s the impact? How to fix it? 24h
  6. 6. Right now? Service Levels Everything working? Yesterday Is it me or the internet? How did we perform? What’s the impact? Versus targets How to fix it? Historical context 24h 7days
  7. 7. Right now? Service Levels Trends & Optimization Everything working? Yesterday Future Is it me or the internet? How did we perform? Big Picture What’s the impact? Versus targets Actionable How to fix it? Historical context 24h Optimization/Conversion 7days ??
  8. 8. Mapping Synthetic versus RUM
  9. 9. Keywords: Error focused Root Cause Object driven information Real time Reporting life cycle Keywords: Alerting Incident driven MTTR Keywords: User experience Conversion metrics Volume/Trends Data required Day Week Month Relevance for Application Life Cycle Quarter Keywords: Business impact Front End Optimization Capacity Management
  10. 10. Keywords: Error focused Root Cause Object driven information Synthetic Real time Reporting life cycle Keywords: Alerting Incident driven MTTR Keywords: User experience Conversion metrics Volume/Trends Data required Day Week Month Quarter RUM Relevance for Application Life Cycle Keywords: Business impact Front End Optimization Capacity Management
  11. 11. Real World RUM Unsampled, Actionable
  12. 12. RUM for Ops
  13. 13. Season Readiness Dashboard (example 2: Real Time Performance Dashboard) Real Time Visibility
  14. 14. Example.com Desktop
  15. 15. Example.com Performance Culture
  16. 16. Unknown Unknowns
  17. 17. Newsletter Marketing page Case presented at Webperfdays Amsterdam
  18. 18. Email campaign 10Mb webpage!! Slow traffic? RUM Triage issue Case presented at Webperfdays Amsterdam
  19. 19. RUM for Analytics
  20. 20. #1. Group data
  21. 21. Put monitoring into your release cycle: ‣ Align to the development team(s) ‣ If possible, match with conversion targets
  22. 22. Home Product List Budget Segmented Matched per page type Matched per dev. team Search Matched per page template Payment
  23. 23. #2. Performance SLA
  24. 24. Wait a week or 2 before setting targets
  25. 25. Establish a baseline:
  26. 26. Establish a baseline: A pre-defined set of metrics
  27. 27. Establish a baseline: A pre-defined set of metrics that describes normal behavior
  28. 28. Establish a baseline: A pre-defined set of metrics that describes normal behavior in order to detect variancies
  29. 29. Establish a baseline: A pre-defined set of metrics that describes normal behavior in order to detect variancies and to be comparable within historic context
  30. 30. A practical (and easy DIY) example:
  31. 31. Purchasing a book, must be completed (speed), where every page loads under 3 sec., Customer journey Metric: Speed Target: Sec using IE9 and higher, User scenario from any location in the Netherlands, User locations for 90% of all users, every day between 6am and 12pm, measured with Real User Monitoring. Percentile Window Collection type Read more: Metrics 101, Velocityconf 2010
  32. 32. Performance SLA
  33. 33. 100% 150% 125% 100% 75% 50% 50% 25% 25% 0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Sessions Bounce rate 75%
  34. 34. LD50: 5.2 sec. 100% 150% 100% 125% 100% 75% 50% 50% 25% 25% 0% 0% 1 2 3 4 5 6 7 8 9 10 11 12 Baseline: 2,7 sec. Poverty Line: 10,6 sec. 13 14 Sessions Bounce rate 75%
  35. 35. #3. Understand User Behavior
  36. 36. Optimization begins with understanding how users actually use your product...
  37. 37. 0:0 0:0 1:0 0" 0:0 2:0 0" 0:0 3:0 0" 0:0 4:0 0" 0:0 5:0 0" 0:0 6:0 0" 0:0 7:0 0" 0:0 8:0 0" 0:0 9:0 0" 0:0 10 :00 0" 11 :00" :00 12 :00" :00 13 :00" :00 14 :00" :00 15 :00" :00 16 :00" :00 17 :00" :00 18 :00" :00 19 :00" :00 20 :00" :00 21 :00" :00 22 :00" :00 23 :00" :00 :00 " Patterns Pageviews (%) Daily pattern Totaal& 100% 450000" 400000" 75% 350000" 300000" 250000" 50% 200000" 150000" 25% 100000" 50000" 0% 0"
  38. 38. page%view%trend%(week)% Weekly Daily pattern Totaal& pattern 20000" Pageviews (%) 2000" 0:0 0:0 1:0 0" 0:0 2:0 0" 0:0 3:0 0" 0:0 4:0 0" 0:0 5:0 0" 0:0 6:0 0" 0:0 7:0 0" 0:0 8:0 0" 0:0 9:0 0" 0:0 10 :00 0" 11 :00" :00 12 :00" :00 13 :00" :00 14 :00" :00 15 :00" :00 16 :00" :00 17 :00" :00 18 :00" :00 19 :00" :00 20 :00" :00 21 :00" :00 22 :00" :00 23 :00" :00 :00 " Patterns 100% 450000" 18000" 400000" 16000" 75% 350000" 14000" 300000" 12000" 250000" 50% 10000" 200000" 8000" 150000" 25% 100000" 6000" 50000" 4000" 0% 0" 0" 03Jun2013" 04Jun2013" 05Jun2013" 06Jun2013" 07Jun2013" 08Jun2013" 09Jun2013" 10Jun2013"
  39. 39. 01 0:0 Jun2 020: 01 100 Ju00" 3" : 30 n201 :n Ju00 3" 200 2"0 : 40 Ju0 13" : n0 300 2"01 : 50 Ju0 3" : n0 400 2"01 : 60u J : n 3" 2 500 J 00"013 7u : 0 n :00 0 " 08 2" 1 6:0 Ju n 0:0 2 3" 09 0 01 7:0 Ju " 3" n2 100:00 01 8:0 Jun " 3" 110:0 201 9:0 Jun0" 3" 120:0 201 10 Jun0" 3" : 100: 201 11 3Ju00" 3" :4 n 100 201 :n 12 Ju00" 3" :5 100 201 :n 131 Ju00"0 3" :00 2 13 6Ju n2 141 :00"0 " 7J :00u 13 n2 1518 :00"01 " :00u J n 3" 2 1619 :00"01 J :00un 3" 2 1720J :00"013 :00un 2 " 21 :00 01 18 Ju " 3 :00 n2 " : 1922Ju00"013" : n 200:0 201 20 3Jun0" 3" : 200: 201 21 4Ju00" 3" :5 n 200 201 :n 22 Ju00" 3" 200 201 :6 J: n 232 u00"0 3" :00 2 13 7Ju :00 " n 28 2"01 Jun 3" 29 201 Jun 3" 30 201 Jun 3" 20 13 " Patterns Pageviews (%) 100% 450000" 18000" 250000" 400000" 16000" 75% 200000" 350000" 14000" 300000" 150000" 12000" 250000" 50% 10000" 100000" 200000" 8000" 150000" 25% 50000" 100000" 6000" 50000" 0" 4000" 0% 0" Pageviews 20000" 0" 03Jun2013" page%view%trend%(week)% Weekly pattern Daily Totaal& pattern Totaal& pattern Monthly 2000" 04Jun2013" 05Jun2013" 06Jun2013" 07Jun2013" 08Jun2013" 09Jun2013" 10Jun2013"
  40. 40. Top 5 DESKTOP Top Browser type Bouncerate avg. Back-End Time avg. Front-End time Total avg. load time 1 Chrome30 49% 0.26 1.31 1.61 2 IE10 51% 0.27 0.81 1.18 3 Firefox25 48% 0.35 1.42 1.88 4 IE9 51% 0.27 1.26 1.61 5 Mobile Safari7 48% 0.52 1.86 2.79 Browsers, usage, performance... Top 5 MOBILE Browser type Bouncerate Backend Time FrontEnd time Loadtime 1 Android 4 72% 0.27 2.38 2.64 2 Mobile Safari 7 53% 0.93 1.04 2.58 3 Chrome Mobile 30 75% 0.40 1.19 1.99 4 Chrome Mobile 18 82% 0.41 1.53 2.25 5 Android 2 59% 0.72 1.88 3.05
  41. 41. Homepage_MHP"10th"&"90th"percen0les" Mac"vs."Windows" Mac"P_10" Mac"P_90" Windows"P_10"" Windows"P_90" 35" 30" Load"0me"(sec)" 25" 20" 15" 10" 5" 0" 06(22(2013"0:00" 06(24(2013"0:00" 06(26(2013"0:00" 06(28(2013"0:00"
  42. 42. #4. What affects performance?
  43. 43. Understand the DNA of your website and what affects your performance...
  44. 44. Speed vs. Engagement per month (Non-Bounce) 90 78,75 56,25 Pageviews (#) Non-Bounce rate (%) 67,5 45 33,75 22,5 11,25 0 0-0,5 0,5-1 1-1,5 1,5-2 2-2,5 2,5-3 3-3,5 3,5-4 4-5 5-6 Non-Bounce CategorienaamPageviews # 6-7 7-8 8-9 9-10 >10
  45. 45. First view vs. Repeat view 7 7000 6 0% Patterns Load Time (Sec.) 6000 5 5000 4 4000 3 3000 2 2000 1 1000 0 0% 0 Monday Tuesday Wednesday Thursday First View IE9 Windows Friday Saturday Repeat View IE9 Windows Sunday Monday Tuesday First View Safari MacOS Wednesday Thursday Repeat View Safari MacOS Friday Saturday Sunday
  46. 46. Page views vs. Load time breakdown 0% 7 6 100% 5 2.5 Load (%) Page views Time (Sec.) Pageviews 4 75% 3 50% 1.5 Usage impacts Speed? 2 25% 1.0 0 0 0% 1 0% Time (days) TCP connection SSL Handshake DNS Resolution Front-End time DOM Ready Page Load Load Time (Sec.) 2.0
  47. 47. #5. Analyze & Optimize
  48. 48. Group & Graph: Pageviews 5,00 4,50 4,00 3,50 3,00 2,50 2,00 1,50 1,00 Load Time (sec.) Group all: Payment 100% 5,00 4,50 4,00 3,50 3,00 2,50 2,00 1,50 1,00 Load Time (sec.) Pageviews Payment page: Confirmation
  49. 49. Boxplots Outliers Incidents? Group: Home Avg. Load Time Load Time (sec.) Hours (#): 17 Avg. Loadtime (sec.): 7.34 Sum of Page Views: 11.327 Time (Hour) Time
  50. 50. Train your audience: ‣ Spread the word to key stakeholders ‣ Report within 24 hours after new release period ‣ Always compared to previous period
  51. 51. Getting fancy
  52. 52. Service levels Service Level Report KPI’s 082 084 Performance (avg) Page 1 www.homepage.nl Loadtime. Loadtime Availability (Document. (.fully.loaded.) complete) Competition You 5 97,0% 2,20 3,50 10% 85% Page 2 ge.nl/autoverzekeren 99,0% 2,50 3,30 20% 90% Page 3 overzekeren/afsluiten 95,0% 2,30 3,60 15% 70% 95,0% 2,43 3,57 20% 67% 1 Page 5 homepage.nl/zoeken 94,0% 2,48 3,62 23% 59% Branche 2 Page 4 homepage.nl/contact Competitors 0 Page 6 e.nl/productoverzicht 93,0% 2,53 3,67 25% 92,0% 2,58 3,72 28% 44% 3 33 34 35 ge.nl/ab8test/variant2 Page 8 91,0% 2,63 3,77 30% 37% ge.nl/facebookplugins Page 9 90,0% 2,68 3,82 33% 89,0% 2,73 3,87 35% 22% ww.homepage.nl/faq Page 11 88,0% 2,78 3,92 38% 14% 37 38 39 40 # 6 3,4 sec Is your ranking against competitors is the average performance of your competitors 40% 40% of your compeitors are faster than you 29% page.nl/muziekpagina Page 10 36 Week 52% Page 7 ge.nl/ab8test/variant1 4 Source: Availability: Gomez Loadtime: Soasta Competition: Webpagetest.org
  53. 53. Performance Trend report 75 4 45 3 2 1 0 Objects (#) 60 15 3d Party 5 30 URL’s 6 0 33 34 35 36 37 38 39 Load time (sec.) 90 Speed 40 Week (#) 33 34 35 36 37 38 39 40 Speed (sec.) 2.9 2.4 2.2 2.8 3.9 2.9 3.1 2.9 URL’s (#) 67 68 65 69 80 72 69 73 3d Party (#) 7 8 6 9 8 9 7 7 Page Size (Kb) 2000 1800 1850 2100 2700 2200 2100 2150 Source: Speed: Soasta Object data: Webpagetest.org
  54. 54. Revenue risked 100.000 visits (Performance/Time Period/Percentiles) Optimal Flow (P_25) Bouncerate (%) Potential Conversion 2.1 100 49% 51.000 Page 2 4.0 100 51% 24.990 Page 3 2.0 100 60% 9996 Page 4 3.5 85 25% 6372 Actual Flow (P_85) Speed (sec.) Availability (%) Bouncerate (%) Actual Conversion Page 1 4.1 100 59% 41.000 Page 2 6.3 100 54% 18.860 Page 3 Speed: Soasta Availability: Webpagetest.org Availability (%) Page 1 Source: Speed (sec.) 2.0 100 60% 7544 Page 4 4.7 85 53% 3013 0-2,7 sec. 2,7-3,6 sec. >3,6 sec.
  55. 55. From noise to insights...
  56. 56. 1 Datamodel vs. Development
  57. 57. 1 2 Datamodel vs. Development Collect data & set Performance KPI
  58. 58. 1 2 Datamodel vs. Development 3 Collect data & set Performance KPI Understand User Behavior
  59. 59. 1 2 Datamodel vs. Development 3 Collect data & set Performance KPI 4 Understand User Behavior Analyze, Correlate & Get Faster!
  60. 60. 1 2 Datamodel vs. Development RUM, FTW!! 3 Collect data & set Performance KPI 4 Understand User Behavior Analyze, Correlate & Optimize
  61. 61. Thanks! More questions? M: jtjepkema@measureworks.nl T: @jeroentjepkema W: www.measureworks.nl

×