Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real User Monitoring: Getting Real Data from Real Users in the Real World - Steve Lerner, eBay


Published on

Improvements to user experience translate directly to real business metrics and the bottom line. To guide the business to making wise choices on user experience, you need an accurate picture of site performance for real users. In this talk, Steve Lerner will describe how eBay’s performance monitoring strategy has evolved, how the insights gained from real user monitoring have impacted eBay’s business, and some of the considerations that have shaped their in house implementation of Real User Monitoring to serve eBay’s massive global scale. See Steve Lerner's Edge Presentation:

The Akamai Edge Conference is a gathering of the industry revolutionaries who are committed to creating leading edge experiences, realizing the full potential of what is possible in a Faster Forward World. From customer innovation stories, industry panels, technical labs, partner and government forums to Web security and developers' tracks, there’s something for everyone at Edge 2013.

Learn more at

Published in: Technology
  • Be the first to comment

Real User Monitoring: Getting Real Data from Real Users in the Real World - Steve Lerner, eBay

  1. 1. Real User Monitoring Getting Real Data from Real Users in the Real World Steve Lerner, Senior Member of Technical Staff, Network Engineering, Global Network Services Rajasekhar Bhogi, Manager, Software Development, Platform Frameworks, US Sheldon Shao, Member of Technical Staff, Software Engineer, Platform Frameworks, China
  2. 2. PLATFORM
  3. 3. Thought Experiment Measuring Performance: Not As Obvious As It Seems Which Scenario Has Better Performance? 2meter 5watt radio Reaches 3 miles Perfect clarity How Can It Be Improved? 10meter 100watt radio Reaches Europe from NYC Slightly distorted 3
  4. 4. Pontificated Points Measurement metrics must contain clear path for action Otherwise suffer analysis paralysis Scale, geography, network, servers, users, etc. all influence performance • Performance measurement is multifaceted • No single method is enough There is no silver bullet Because werewolves don’t exist Image Source: Creative Commons via Wikipedia 4
  5. 5. Performance Measurement Synthetic testing (ThousandEyes, Catchpoint, Keynote, etc) • • • • Use small numbers of probes/agents to artificially measure your pages Can be useful for network measurements and triaging issues Small sample size and test agent structure makes unreliable for critical needs CDNs game synthetic tests with caches near agents so measurements are always perfect Real User Metrics Are 100% Accurate But • Cannot yet capture details about each object on page • Generate massive data- need massive infrastructure and experienced team to manage • Upstream problems can create confounding- i.e. problems on a web server may show overall slowness of RUM but not point to problem’s root cause • Standard browser hooks for timing not supported by Safari- need to use proprietary ones • Ultimate goal would be to analyze every log line of every object but not yet feasible Best Practice: Use Everything • TRIANGULATE AND INTERPRET: Results need to be viewed in context, double checked, and cross referenced 5
  6. 6. Goal: Continuous Improvement eBay’s Top 5 Pages For Analysis • Measured “from” a country’s users “to” a country site i.e. from Brazil users to US site: • Home page • Search results • View Item • My eBay • Login Clear Path For Improvement Is Laid Out By Proper Timing Measurement Tools For Example • • • • • • DNS resolution Connect First byte Navigation Object download Object start 6
  7. 7. eBay Synthetic Testing: ThousandEyes Standard Waterfall Used To Troubleshoot Object Issues Pubmatic Ad Server = Ouch 7
  8. 8. eBay Synthetic Testing: ThousandEyes Network Path View Used For Troubleshooting Internet Issues 8
  9. 9. eBay Synthetic Testing: ThousandEyes BGP View Used For Troubleshooting Routing Issues 9
  10. 10. eBay Real User Metrics aka Site Speed eBay Real User Metrics (RUM) • Program is called Site Speed and reporting is called Site Speed Gauge • Designed and built by global team Based On Standards • W3C Web Performance Working Group • “The mission of the Web Performance Working Group, part of the Rich Web Client Activity, is to provide methods to measure aspects of application performance of user agent features and APIs” Anyone Can Build This! • Open source libraries exist • Boomerang: Anyone Can Buy This! • Most synthetic test vendors already or will soon offer a RUM solution 10
  11. 11. Order of Operations for Site Speed Measurement Before Navigation Timing API Support Receive Page Request and Choose Sample RateUsually 10% Take time stamps • • • • Start Head of Page Page load Others Calculate commonly used metrics Send precalculated metrics back to server via beacon Generate reports, dashboards and alerts With Navigation Timing API Support Receive Page Request and Choose Sample Rate- Usually 10% Calculate commonly used metrics Send precalculated metrics back to server via beacon Generate reports, dashboards and alerts 11
  12. 12. Visualization of Web Page Timing Elements Proprietary code catches network layer timing events to pass to JavaScript beacon (Before NavigationTimer API support) Event Graph Source: 12
  13. 13. Site Speed: Sample Sizes for UK Search Results Page 10% Sampling = 350M views of Search Results per day during this time period, of UK page from UK users – to verify sample size is acceptable 13
  14. 14. Site Speed Example: Akamai DSA in Italy • eBay enabled Akamai Dynamic Site Acceleration (DSA) for as a “better bandwidth” service- no page caching, prefetching, or other page changes enabled Clear Path – Improve connection time and reduce latency to pages • Static objects already on Akamai EdgeSuite for acceleration • Speed Gauge example report: – Connection Time (Connections End – Connection start): Median – Content Download Time (Response End– Response Start): Median 14
  15. 15. Site Speed Results: Akamai DSA in Italy DSA Enabled Content Download 58% Reduction Connect Time 78% Reduction 15
  16. 16. TRIANGULATE! Synthetic Test View of DSA Activation 90% Reduction In Latency From ThousandEyes Agents To When DSA Enabled 16
  17. 17. Site Speed: Switching CDNs for View Item Photos •View Item pages contain images and objects delivered on their own domain i.e. Clear Path –Switch image domain from one CDN to another with better global coverage •Measure: Median response of start to on-load event (loadEventStart – responseStart) 17
  18. 18. Site Speed Results: Switching CDNs for View Item Photos Improvements: Philippines: 38% Malaysia: 30% China: 25% Brazil: 19% 18
  19. 19. Site Speed Results: Enabling Global DNS Service • eBay is enabling global DNS service – a global anycast deployment of DNS servers Clear Path – Enable global DNS service to reduce DNS resolution time • First test of service is in India on 57% Improvement For DNS Lookup Time In India When India DNS Servers Added USA DNS Shown For Scale 19
  20. 20. DNS: Triangulate Results Outsource DNS Service, Globally, Is 63% Faster Than Origin DNS 20
  21. 21. Site Speed: Global Heat Map Green = faster, Red = slower 21
  22. 22. Site Speed: USA Heat Map Green = Faster, Red = Slower 22
  23. 23. Site Speed: Browser View View Of Home Page, USA Users (Some Areas Redacted) 23
  24. 24. Site Speed: ISP View View Of Home Page, USA Users (Some Areas Redacted) 24
  25. 25. Site Speed Histogram: Speed at Percentile (BETA) Which Page Download Speed Is At Which Percentile Goal: Move Pages Left To Get Faster (Better) Speed 25
  26. 26. Site Speed Histogram: Count At Each Percentile (BETA) Quantity Of Datapoints At Each Percentile Of Speed Goal: Move Users Left To Get Faster (Better) Speed 26
  27. 27. Purchases Per Visitor / Week How Does Page Load Speed Impact Business? But there is a bigger economics story 10% Faster Page = 1% Increase in Purchases Per Visitor / Week Source: Published eBay study 27
  28. 28. Summary, Conclusions, and Wisdom Measurement Metrics Must Contain A Clear Path For Action • Don’t Fall For (Or Supply) A Single Summary Metric- It Doesn’t Exist Performance Measurement Requires A Multifaceted Approach • Synthetic Tests • Real User Metrics • Ideally Log File Analysis Triangulate Results To Understand Cause And Effect Identify Economic Outcome Keep Investigating, Evolving, And Breaking Things Never Assume What Worked Or Had Success In The Past Will Continue To Do So 28
  29. 29. Steve Lerner Senior Member of Technical Staff, Network Engineering Global Network Services Rajasekhar Bhogi Manager, Software Development Platform Frameworks, US 29