Tracking Performance - Velocity NYC 2013

4,127 views

Published on

Slides from Velocity NYC 2013 presentation on measuring and tracking web performance.

Published in: Technology

Tracking Performance - Velocity NYC 2013

  1. 1. Tracking Performance Patrick Meenan @patmeenan
  2. 2. Slides will be available http://www.slideshare.net/patrickmeenan
  3. 3. Introducing shop-de-nile.com
  4. 4. System Architecture Web Server (App) Web Server (App) Load Balancer Load Balancer Product DB Reviews DB Web Server (App) Web Server (App) SOLR FT Search Web Server (Static) Web Server (Static) Product Images
  5. 5. Us Them
  6. 6. http://www.memegen.com/meme/dbuj21
  7. 7. New System Architecture Product DB Web Server (App) Web Server (App) Load Balancer Web Server (App) Load Balancer Web Server (App) Varnish Varnish Memcache Memcache Memcache Reviews DB SOLR FT Search Web Server (Static) Web Server (Static) Product Images
  8. 8. Testing Plan ● 14 Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Search Results Page Product Listing Page ● 1 Browser ● First View Only ● Every 5 Minutes (12 samples per hour)
  9. 9. 12,096 Tests Per Day
  10. 10. http://www.memegen.com/meme/ek1v79
  11. 11. CDN test plan ● Test 3 CDN’s ● Over 2 Weeks ● Compare against: ○ Each Other ○ Local Performance ○ No CDN
  12. 12. 36,288 Tests Per Day
  13. 13. Re-Run Competitive Comparison ● ● ● ● Globally 20 Page Loads Per Day 3 Representative Pages 1,680 Page Loads
  14. 14. http://www.memegen.com/meme/duqjfs
  15. 15. Real User Monitoring (RUM)
  16. 16. 5.28 1.38
  17. 17. http://www.memegen.com/meme/9vyywz
  18. 18. Instrument back-end http://www.slideshare.net/mikebrittain/metrics-driven-engineering-at-etsy http://codeascraft.com/ Access Logs -> logster -> statsd -> graphite
  19. 19. Application Performance Measurement (APM)
  20. 20. Cache Hits
  21. 21. Not no much
  22. 22. Testing Plan ● 14 Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Search Results Page Product Listing Page ● 1 Browser ● First View Only ● Every 5 Minutes (12 samples per hour)
  23. 23. Go buy an SSD @crucially http://www.youtube.com/watch?v=H7PJ1oeEyGg
  24. 24. New New System Architecture Product DB (SSD) Web Server (App) Web Server (App) Load Balancer Web Server (App) Load Balancer Web Server (App) Varnish (SSD) Web Server (Static) Memcache Reviews DB (SSD) SOLR FT Search (SSD) Web Server (Static) Varnish (SSD) Memcache Memcache Product Images (SSD)
  25. 25. 4.58
  26. 26. Not matching synthetic testing ● Backbone vs last-mile ● Long-tail images
  27. 27. http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
  28. 28. Our Testing http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
  29. 29. Our Users http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
  30. 30. Testing Plan Images always in edge cache ● 14 Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Search Results Page Product Listing Page ● 1 Browser ● First View Only ● Every 5 Minutes (12 samples per hour)
  31. 31. New Experience Testing Plan ● 14 Last-Mile Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Random Search Results Page Random Product Listing Page ● 2 Browsers ○ Desktop ○ Mobile (actual device) ● First View Only ● 20 Samples per day
  32. 32. Measured Load Times still too slow ● ● ● ● Social Sharing Widgets Ads Retargeting pixels Analytics All already Async and using best practices
  33. 33. Ummm…. ● Large drop in revenue ● Loss of tracking ● Doesn’t “feel” significantly faster
  34. 34. Speed Index FTW ● Measures visual experience ● Synthetic Only (WebPagetest)
  35. 35. Speed Index 3rd Party Evaluation
  36. 36. Speed Index FTW ● Measures visual experience ● Synthetic Only (WebPagetest)
  37. 37. We got this...it’s only code onload=”performance.mark(‘aft.Image Loaded’)”... window.onload: ● performance.getEntriesByType(“mark”) ● Report latest aft.* as custom time http://blog.patrickmeenan.com/2013/07/measuring-performance-of-user-experience.html
  38. 38. Roll-your-own ● Send http beacon to beacon server ○ All timings as query params ○ 204 or transparent 1px gif ○ Log requests to access log ● IP, User Agent and Timings all in each record ● access log -> logster -> statsd -> graphite ● Profit!
  39. 39. Google Analytics _gaq.push(['_trackTiming', 'UserTimings', 'aft', measuredTime);
  40. 40. Boomerang/Soasta mPulse BOOMR.plugins.RT.setTimer(‘custom0’, measuredTime); https://gist.github.com/pmeenan/5902672
  41. 41. AFT vs onload AFT Median 0.932 95th Percentile 4.141 98th Percentile 7.262 onLoad Median 2.235 95th Percentile 11.787 98th Percentile 26.72 http://www.soasta.com/products/mpulse/
  42. 42. Custom timings in WebPagetest
  43. 43. Where does that leave us? ● Competitive testing with synthetic last-mile ○ After builds pushed to staging/prod ● Performance tracking with RUM custom measurements ○ ○ Data can be noisy No competitive capability (opportunity)
  44. 44. Where does that leave us? ● Competitive testing with synthetic last-mile ○ After builds pushed to staging/prod ● Performance tracking with RUM custom measurements ○ ○ Data can be noisy No competitive capability (opportunity)
  45. 45. Synthetic testing in CI ● PhantomJS + CI ● WebPagetest Private Instance + CI ○ https://github.com/marcelduran/webpagetest-api ● Graphing/Trending artifacts ○ ○ ○ ○ httparchive graphite Show Slow tsviewdb
  46. 46. tsviewdb ● ● Trending Regression Detection https://github.com/google/tsviewdb
  47. 47. People will optimize for what you measure…. So measure what you want optimized.

×