Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Tracking Performance
Patrick Meenan
@patmeenan
Slides will be available

http://www.slideshare.net/patrickmeenan
Introducing shop-de-nile.com
System Architecture
Web Server (App)
Web Server (App)
Load Balancer
Load Balancer

Product
DB

Reviews
DB

Web Server (App...
Us

Them
http://www.memegen.com/meme/dbuj21
New System Architecture
Product
DB

Web Server (App)
Web Server (App)
Load Balancer

Web Server (App)

Load Balancer

Web ...
Testing Plan
● 14 Locations
● 3 “Representative” Pages
○
○
○

Landing Page
Search Results Page
Product Listing Page

● 1 B...
12,096 Tests Per Day
http://www.memegen.com/meme/ek1v79
CDN test plan
● Test 3 CDN’s
● Over 2 Weeks
● Compare against:
○ Each Other
○ Local Performance
○ No CDN
36,288 Tests Per Day
Re-Run Competitive Comparison
●
●
●
●

Globally
20 Page Loads Per Day
3 Representative Pages
1,680 Page Loads
http://www.memegen.com/meme/duqjfs
Real User Monitoring (RUM)
5.28

1.38
http://www.memegen.com/meme/9vyywz
Instrument back-end
http://www.slideshare.net/mikebrittain/metrics-driven-engineering-at-etsy

http://codeascraft.com/
Acc...
Application Performance Measurement (APM)
Cache Hits
Not no much
Testing Plan
● 14 Locations
● 3 “Representative” Pages
○
○
○

Landing Page
Search Results Page
Product Listing Page

● 1 B...
Go buy an SSD

@crucially
http://www.youtube.com/watch?v=H7PJ1oeEyGg
New New System Architecture
Product
DB (SSD)

Web Server (App)
Web Server (App)
Load Balancer

Web Server (App)

Load Bala...
4.58
Not matching synthetic testing
● Backbone vs last-mile
● Long-tail images
http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
Our
Testing
http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
Our Users
http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
Testing Plan
Images always in edge cache

● 14 Locations
● 3 “Representative” Pages
○
○
○

Landing Page
Search Results Pag...
New Experience Testing Plan
● 14 Last-Mile Locations
● 3 “Representative” Pages
○
○
○

Landing Page
Random Search Results ...
Measured Load Times still too slow
●
●
●
●

Social Sharing Widgets
Ads
Retargeting pixels
Analytics

All already Async and...
Ummm….
● Large drop in revenue
● Loss of tracking
● Doesn’t “feel” significantly faster
Speed Index FTW
● Measures visual experience
● Synthetic Only (WebPagetest)
Speed Index 3rd Party Evaluation
Speed Index FTW
● Measures visual experience
● Synthetic Only (WebPagetest)
We got this...it’s only code
onload=”performance.mark(‘aft.Image Loaded’)”...

window.onload:
● performance.getEntriesByTy...
Roll-your-own
● Send http beacon to beacon server
○ All timings as query params
○ 204 or transparent 1px gif
○ Log request...
Google Analytics
_gaq.push(['_trackTiming', 'UserTimings', 'aft', measuredTime);
Boomerang/Soasta mPulse
BOOMR.plugins.RT.setTimer(‘custom0’, measuredTime);

https://gist.github.com/pmeenan/5902672
AFT vs onload
AFT
Median

0.932

95th Percentile

4.141

98th Percentile

7.262

onLoad
Median

2.235

95th Percentile

11...
Custom timings in WebPagetest
Where does that leave us?
● Competitive testing with synthetic last-mile
○

After builds pushed to staging/prod

● Perform...
Where does that leave us?
● Competitive testing with synthetic last-mile
○

After builds pushed to staging/prod

● Perform...
Synthetic testing in CI
● PhantomJS + CI
● WebPagetest Private Instance + CI
○

https://github.com/marcelduran/webpagetest...
tsviewdb
●
●

Trending
Regression Detection

https://github.com/google/tsviewdb
People will optimize for what you measure….
So measure what you want optimized.
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Tracking Performance - Velocity NYC 2013
Upcoming SlideShare
Loading in …5
×

Tracking Performance - Velocity NYC 2013

4,645 views

Published on

Slides from Velocity NYC 2013 presentation on measuring and tracking web performance.

Published in: Technology
  • Dating direct: ❶❶❶ http://bit.ly/369VOVb ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❶❶❶ http://bit.ly/369VOVb ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Tracking Performance - Velocity NYC 2013

  1. 1. Tracking Performance Patrick Meenan @patmeenan
  2. 2. Slides will be available http://www.slideshare.net/patrickmeenan
  3. 3. Introducing shop-de-nile.com
  4. 4. System Architecture Web Server (App) Web Server (App) Load Balancer Load Balancer Product DB Reviews DB Web Server (App) Web Server (App) SOLR FT Search Web Server (Static) Web Server (Static) Product Images
  5. 5. Us Them
  6. 6. http://www.memegen.com/meme/dbuj21
  7. 7. New System Architecture Product DB Web Server (App) Web Server (App) Load Balancer Web Server (App) Load Balancer Web Server (App) Varnish Varnish Memcache Memcache Memcache Reviews DB SOLR FT Search Web Server (Static) Web Server (Static) Product Images
  8. 8. Testing Plan ● 14 Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Search Results Page Product Listing Page ● 1 Browser ● First View Only ● Every 5 Minutes (12 samples per hour)
  9. 9. 12,096 Tests Per Day
  10. 10. http://www.memegen.com/meme/ek1v79
  11. 11. CDN test plan ● Test 3 CDN’s ● Over 2 Weeks ● Compare against: ○ Each Other ○ Local Performance ○ No CDN
  12. 12. 36,288 Tests Per Day
  13. 13. Re-Run Competitive Comparison ● ● ● ● Globally 20 Page Loads Per Day 3 Representative Pages 1,680 Page Loads
  14. 14. http://www.memegen.com/meme/duqjfs
  15. 15. Real User Monitoring (RUM)
  16. 16. 5.28 1.38
  17. 17. http://www.memegen.com/meme/9vyywz
  18. 18. Instrument back-end http://www.slideshare.net/mikebrittain/metrics-driven-engineering-at-etsy http://codeascraft.com/ Access Logs -> logster -> statsd -> graphite
  19. 19. Application Performance Measurement (APM)
  20. 20. Cache Hits
  21. 21. Not no much
  22. 22. Testing Plan ● 14 Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Search Results Page Product Listing Page ● 1 Browser ● First View Only ● Every 5 Minutes (12 samples per hour)
  23. 23. Go buy an SSD @crucially http://www.youtube.com/watch?v=H7PJ1oeEyGg
  24. 24. New New System Architecture Product DB (SSD) Web Server (App) Web Server (App) Load Balancer Web Server (App) Load Balancer Web Server (App) Varnish (SSD) Web Server (Static) Memcache Reviews DB (SSD) SOLR FT Search (SSD) Web Server (Static) Varnish (SSD) Memcache Memcache Product Images (SSD)
  25. 25. 4.58
  26. 26. Not matching synthetic testing ● Backbone vs last-mile ● Long-tail images
  27. 27. http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
  28. 28. Our Testing http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
  29. 29. Our Users http://www.belshe.com/2010/05/24/more-bandwidth-doesnt-matter-much/
  30. 30. Testing Plan Images always in edge cache ● 14 Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Search Results Page Product Listing Page ● 1 Browser ● First View Only ● Every 5 Minutes (12 samples per hour)
  31. 31. New Experience Testing Plan ● 14 Last-Mile Locations ● 3 “Representative” Pages ○ ○ ○ Landing Page Random Search Results Page Random Product Listing Page ● 2 Browsers ○ Desktop ○ Mobile (actual device) ● First View Only ● 20 Samples per day
  32. 32. Measured Load Times still too slow ● ● ● ● Social Sharing Widgets Ads Retargeting pixels Analytics All already Async and using best practices
  33. 33. Ummm…. ● Large drop in revenue ● Loss of tracking ● Doesn’t “feel” significantly faster
  34. 34. Speed Index FTW ● Measures visual experience ● Synthetic Only (WebPagetest)
  35. 35. Speed Index 3rd Party Evaluation
  36. 36. Speed Index FTW ● Measures visual experience ● Synthetic Only (WebPagetest)
  37. 37. We got this...it’s only code onload=”performance.mark(‘aft.Image Loaded’)”... window.onload: ● performance.getEntriesByType(“mark”) ● Report latest aft.* as custom time http://blog.patrickmeenan.com/2013/07/measuring-performance-of-user-experience.html
  38. 38. Roll-your-own ● Send http beacon to beacon server ○ All timings as query params ○ 204 or transparent 1px gif ○ Log requests to access log ● IP, User Agent and Timings all in each record ● access log -> logster -> statsd -> graphite ● Profit!
  39. 39. Google Analytics _gaq.push(['_trackTiming', 'UserTimings', 'aft', measuredTime);
  40. 40. Boomerang/Soasta mPulse BOOMR.plugins.RT.setTimer(‘custom0’, measuredTime); https://gist.github.com/pmeenan/5902672
  41. 41. AFT vs onload AFT Median 0.932 95th Percentile 4.141 98th Percentile 7.262 onLoad Median 2.235 95th Percentile 11.787 98th Percentile 26.72 http://www.soasta.com/products/mpulse/
  42. 42. Custom timings in WebPagetest
  43. 43. Where does that leave us? ● Competitive testing with synthetic last-mile ○ After builds pushed to staging/prod ● Performance tracking with RUM custom measurements ○ ○ Data can be noisy No competitive capability (opportunity)
  44. 44. Where does that leave us? ● Competitive testing with synthetic last-mile ○ After builds pushed to staging/prod ● Performance tracking with RUM custom measurements ○ ○ Data can be noisy No competitive capability (opportunity)
  45. 45. Synthetic testing in CI ● PhantomJS + CI ● WebPagetest Private Instance + CI ○ https://github.com/marcelduran/webpagetest-api ● Graphing/Trending artifacts ○ ○ ○ ○ httparchive graphite Show Slow tsviewdb
  46. 46. tsviewdb ● ● Trending Regression Detection https://github.com/google/tsviewdb
  47. 47. People will optimize for what you measure…. So measure what you want optimized.

×