Real User Monitoring
Getting Real Data from Real
Users in the Real World
Steve Lerner, Senior Member of Technical Staff, N...
PLATFORM
Thought Experiment
Measuring Performance: Not As Obvious As It Seems

Which Scenario Has
Better Performance?
2meter 5watt ...
Pontificated Points

Measurement metrics must
contain clear path for action
Otherwise suffer
analysis paralysis

Scale, ge...
Performance Measurement
Synthetic testing (ThousandEyes, Catchpoint, Keynote, etc)
•
•
•
•

Use small numbers of probes/ag...
Goal: Continuous Improvement
eBay’s Top 5 Pages For Analysis
• Measured “from” a country’s users “to” a country site i.e. ...
eBay Synthetic Testing: ThousandEyes

Standard
Waterfall Used
To Troubleshoot
Object Issues

Pubmatic Ad
Server =
Ouch

7
eBay Synthetic Testing: ThousandEyes

Network Path
View Used For
Troubleshooting
Internet Issues

8
eBay Synthetic Testing: ThousandEyes

BGP View Used For
Troubleshooting
Routing Issues

9
eBay Real User Metrics aka Site Speed
eBay Real User Metrics (RUM)
• Program is called Site Speed and reporting is called ...
Order of Operations for Site Speed Measurement
Before Navigation Timing API Support
Receive
Page
Request and
Choose
Sample...
Visualization of Web Page Timing Elements

Proprietary code catches network
layer timing events to pass to
JavaScript beac...
Site Speed: Sample Sizes for UK Search Results
Page

10% Sampling = 350M views of Search
Results per day during this time ...
Site Speed Example: Akamai DSA in Italy
• eBay enabled Akamai Dynamic Site Acceleration (DSA) for www.ebay.it as a “better...
Site Speed Results: Akamai DSA in Italy

DSA Enabled

Content
Download
58% Reduction

Connect Time
78% Reduction

15
TRIANGULATE! Synthetic Test View of DSA Activation

90% Reduction In Latency
From ThousandEyes
Agents To www.ebay.it
When ...
Site Speed: Switching CDNs for View Item Photos
•View Item pages contain images and objects delivered on
their own domain ...
Site Speed Results: Switching CDNs for View Item Photos

Improvements:
Philippines: 38%
Malaysia: 30%
China: 25%
Brazil: 1...
Site Speed Results: Enabling Global DNS Service
• eBay is enabling global DNS service – a global anycast deployment of DNS...
DNS: Triangulate Results
Outsource DNS
Service, Globally,
Is 63% Faster
Than Origin DNS

20
Site Speed: Global Heat Map

Green = faster,
Red = slower

21
Site Speed: USA Heat Map

Green = Faster,
Red = Slower

22
Site Speed: Browser View
View Of Home
Page, USA Users
(Some Areas
Redacted)

23
Site Speed: ISP View
View Of Home
Page, USA Users
(Some Areas
Redacted)

24
Site Speed Histogram: Speed at Percentile (BETA)

Which Page
Download Speed Is At
Which Percentile

Goal: Move Pages Left ...
Site Speed Histogram: Count At Each Percentile (BETA)

Quantity Of
Datapoints At Each
Percentile Of
Speed

Goal: Move User...
Purchases Per Visitor / Week

How Does Page Load Speed Impact Business?

But there is a
bigger
economics
story

10% Faster...
Summary, Conclusions, and Wisdom
Measurement Metrics Must Contain A Clear Path For Action
• Don’t Fall For (Or Supply) A S...
Steve Lerner
Senior Member of Technical Staff, Network Engineering
Global Network Services
slerner@ebay.com
Rajasekhar Bho...
Upcoming SlideShare
Loading in...5
×

Real User Monitoring: Getting Real Data from Real Users in the Real World - Steve Lerner, eBay

1,368

Published on

Improvements to user experience translate directly to real business metrics and the bottom line. To guide the business to making wise choices on user experience, you need an accurate picture of site performance for real users. In this talk, Steve Lerner will describe how eBay’s performance monitoring strategy has evolved, how the insights gained from real user monitoring have impacted eBay’s business, and some of the considerations that have shaped their in house implementation of Real User Monitoring to serve eBay’s massive global scale. See Steve Lerner's Edge Presentation: http://www.akamai.com/html/custconf/edgetv-commerce.html#real-user-monitoring

The Akamai Edge Conference is a gathering of the industry revolutionaries who are committed to creating leading edge experiences, realizing the full potential of what is possible in a Faster Forward World. From customer innovation stories, industry panels, technical labs, partner and government forums to Web security and developers' tracks, there’s something for everyone at Edge 2013.

Learn more at http://www.akamai.com/edge

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,368
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
23
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Real User Monitoring: Getting Real Data from Real Users in the Real World - Steve Lerner, eBay

  1. 1. Real User Monitoring Getting Real Data from Real Users in the Real World Steve Lerner, Senior Member of Technical Staff, Network Engineering, Global Network Services Rajasekhar Bhogi, Manager, Software Development, Platform Frameworks, US Sheldon Shao, Member of Technical Staff, Software Engineer, Platform Frameworks, China
  2. 2. PLATFORM
  3. 3. Thought Experiment Measuring Performance: Not As Obvious As It Seems Which Scenario Has Better Performance? 2meter 5watt radio Reaches 3 miles Perfect clarity How Can It Be Improved? 10meter 100watt radio Reaches Europe from NYC Slightly distorted 3
  4. 4. Pontificated Points Measurement metrics must contain clear path for action Otherwise suffer analysis paralysis Scale, geography, network, servers, users, etc. all influence performance • Performance measurement is multifaceted • No single method is enough There is no silver bullet Because werewolves don’t exist Image Source: Creative Commons via Wikipedia 4
  5. 5. Performance Measurement Synthetic testing (ThousandEyes, Catchpoint, Keynote, etc) • • • • Use small numbers of probes/agents to artificially measure your pages Can be useful for network measurements and triaging issues Small sample size and test agent structure makes unreliable for critical needs CDNs game synthetic tests with caches near agents so measurements are always perfect Real User Metrics Are 100% Accurate But • Cannot yet capture details about each object on page • Generate massive data- need massive infrastructure and experienced team to manage • Upstream problems can create confounding- i.e. problems on a web server may show overall slowness of RUM but not point to problem’s root cause • Standard browser hooks for timing not supported by Safari- need to use proprietary ones • Ultimate goal would be to analyze every log line of every object but not yet feasible Best Practice: Use Everything • TRIANGULATE AND INTERPRET: Results need to be viewed in context, double checked, and cross referenced 5
  6. 6. Goal: Continuous Improvement eBay’s Top 5 Pages For Analysis • Measured “from” a country’s users “to” a country site i.e. from Brazil users to US site: • Home page • Search results • View Item • My eBay • Login Clear Path For Improvement Is Laid Out By Proper Timing Measurement Tools For Example • • • • • • DNS resolution Connect First byte Navigation Object download Object start 6
  7. 7. eBay Synthetic Testing: ThousandEyes Standard Waterfall Used To Troubleshoot Object Issues Pubmatic Ad Server = Ouch 7
  8. 8. eBay Synthetic Testing: ThousandEyes Network Path View Used For Troubleshooting Internet Issues 8
  9. 9. eBay Synthetic Testing: ThousandEyes BGP View Used For Troubleshooting Routing Issues 9
  10. 10. eBay Real User Metrics aka Site Speed eBay Real User Metrics (RUM) • Program is called Site Speed and reporting is called Site Speed Gauge • Designed and built by global team Based On Standards • W3C Web Performance Working Group http://www.w3.org/2010/webperf/ • “The mission of the Web Performance Working Group, part of the Rich Web Client Activity, is to provide methods to measure aspects of application performance of user agent features and APIs” Anyone Can Build This! • Open source libraries exist • Boomerang: http://lognormal.github.io/boomerang/doc/ Anyone Can Buy This! • Most synthetic test vendors already or will soon offer a RUM solution 10
  11. 11. Order of Operations for Site Speed Measurement Before Navigation Timing API Support Receive Page Request and Choose Sample RateUsually 10% Take time stamps • • • • Start Head of Page Page load Others Calculate commonly used metrics Send precalculated metrics back to server via beacon Generate reports, dashboards and alerts With Navigation Timing API Support Receive Page Request and Choose Sample Rate- Usually 10% Calculate commonly used metrics Send precalculated metrics back to server via beacon Generate reports, dashboards and alerts 11
  12. 12. Visualization of Web Page Timing Elements Proprietary code catches network layer timing events to pass to JavaScript beacon (Before NavigationTimer API support) Event Graph Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html 12
  13. 13. Site Speed: Sample Sizes for UK Search Results Page 10% Sampling = 350M views of Search Results per day during this time period, of UK page from UK users – to verify sample size is acceptable 13
  14. 14. Site Speed Example: Akamai DSA in Italy • eBay enabled Akamai Dynamic Site Acceleration (DSA) for www.ebay.it as a “better bandwidth” service- no page caching, prefetching, or other page changes enabled Clear Path – Improve connection time and reduce latency to pages • Static objects already on Akamai EdgeSuite for acceleration • Speed Gauge example report: – Connection Time (Connections End – Connection start): Median – Content Download Time (Response End– Response Start): Median 14
  15. 15. Site Speed Results: Akamai DSA in Italy DSA Enabled Content Download 58% Reduction Connect Time 78% Reduction 15
  16. 16. TRIANGULATE! Synthetic Test View of DSA Activation 90% Reduction In Latency From ThousandEyes Agents To www.ebay.it When DSA Enabled 16
  17. 17. Site Speed: Switching CDNs for View Item Photos •View Item pages contain images and objects delivered on their own domain i.e. i.ebayimg.com Clear Path –Switch image domain from one CDN to another with better global coverage •Measure: Median response of start to on-load event (loadEventStart – responseStart) 17
  18. 18. Site Speed Results: Switching CDNs for View Item Photos Improvements: Philippines: 38% Malaysia: 30% China: 25% Brazil: 19% 18
  19. 19. Site Speed Results: Enabling Global DNS Service • eBay is enabling global DNS service – a global anycast deployment of DNS servers Clear Path – Enable global DNS service to reduce DNS resolution time • First test of service is in India on www.ebay.in 57% Improvement For DNS Lookup Time In India When India DNS Servers Added USA DNS Shown For Scale 19
  20. 20. DNS: Triangulate Results Outsource DNS Service, Globally, Is 63% Faster Than Origin DNS 20
  21. 21. Site Speed: Global Heat Map Green = faster, Red = slower 21
  22. 22. Site Speed: USA Heat Map Green = Faster, Red = Slower 22
  23. 23. Site Speed: Browser View View Of Home Page, USA Users (Some Areas Redacted) 23
  24. 24. Site Speed: ISP View View Of Home Page, USA Users (Some Areas Redacted) 24
  25. 25. Site Speed Histogram: Speed at Percentile (BETA) Which Page Download Speed Is At Which Percentile Goal: Move Pages Left To Get Faster (Better) Speed 25
  26. 26. Site Speed Histogram: Count At Each Percentile (BETA) Quantity Of Datapoints At Each Percentile Of Speed Goal: Move Users Left To Get Faster (Better) Speed 26
  27. 27. Purchases Per Visitor / Week How Does Page Load Speed Impact Business? But there is a bigger economics story 10% Faster Page = 1% Increase in Purchases Per Visitor / Week Source: Published eBay study http://www.ebaytechblog.com/2013/03/29/measuring-real-user-experience-with-site-speed-gauge/ 27
  28. 28. Summary, Conclusions, and Wisdom Measurement Metrics Must Contain A Clear Path For Action • Don’t Fall For (Or Supply) A Single Summary Metric- It Doesn’t Exist Performance Measurement Requires A Multifaceted Approach • Synthetic Tests • Real User Metrics • Ideally Log File Analysis Triangulate Results To Understand Cause And Effect Identify Economic Outcome Keep Investigating, Evolving, And Breaking Things Never Assume What Worked Or Had Success In The Past Will Continue To Do So 28
  29. 29. Steve Lerner Senior Member of Technical Staff, Network Engineering Global Network Services slerner@ebay.com Rajasekhar Bhogi Manager, Software Development Platform Frameworks, US rbhogi@ebay.com 29
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×