Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SYNTHETIC AND RUM 
BEST OF BOTH WORLDS
Mark Zeman 
@SpeedCurve 
Cliff Crocker 
@cliffcrocker
ACTIVE 
MONITORING 
• Simulated health-checks of 
the system and it’s parts 
• Scripted journeys through 
an application 
...
PASSIVE 
MONITORING 
• Listening to actual traffic as 
it moves through a system 
• Observing behavior of the 
system and/...
Our focus is on the experience of the crowd 
allhumorpic.com
What is RUM? 
RUM 101 
Real User User Measurement Measurement (traditional (RUM) definition) 
is a technology for collecti...
THE GOOD PARTS - RUM 
• Always on 
• Every user, every browser, every network, anywhere in 
the world 
• Ability to captur...
What is RUM? 
Real User Measurement (traditional definition) 
Synthetic 101 
Synthetic monitoring (for purposes of this di...
THE GOOD PARTS - SYNTHETIC 
• Rich data collected (waterfall, videos/filmstrip, HTTP 
Headers) 
• Consistent “clean room” ...
COMMON THINGS WE HEAR ABOUT 
BOTH RUM AND SYNTHETIC 
• “Why are these numbers so different?” 
• “I don’t trust your data. ...
WHY ARE THE TWO NUMBERS SO FAR OFF? 
Real users are not normal
Page Load Times 
All Countries, User Agents, Carriers, Operating Systems 
3.23s – Median
Page Load Times 
All Countries, User Agents, Carriers, Operating Systems 
10.45s – p95
Page Load Times 
All Countries, User Agents, Carriers, Operating Systems 
17.26s – p98
Page Load Times 
By Operating System 
OS Median 95th Percentile 98th Percentile 
Windows 7 2.41s 9.29s 15.89s 
Mac OS X/10...
WHY ARE THE TWO NUMBERS SO FAR OFF? 
Median: 3.23s 
RUM 
SYNTHETIC 
FIRST 
SYNTHETIC 
REPEAT 
3 . 2 3 4 . 1 2 . 2 
0 1 2 3...
“BUT, IT LOADS SO MUCH FASTER FOR ME??!!” 
4 seconds 20 seconds 
• 2015 Macbook Pro 
• Warm browser cache 
• FIOS 
vs. 
• ...
RUM SYNTHETIC 
• Primary source of 
truth 
• User behavior/ 
conversion 
• Complete coverage 
• Diagnostics 
• Page Constr...
RUM SYNTHETIC 
• Primary source of 
truth 
• User behavior/ 
conversion 
• Complete coverage 
• Diagnostics 
• Page Constr...
START RENDER 
1.58S 
DNS 
110MS 
TCP 
40MS 
TTFB 
650MS 
DOM LOADING 
680MS 
DOM READY 
2.75S 
PAGE LOAD 
3.05 
FULLY LOAD...
THE MANY FACES OF PERFORMANCE 
C-Level Operations Developer Designer 
thenounproject.com
START RENDER DNS TCP TTFB 
DOM LOADING DOM READY PAGE LOAD FULLY LOADED 
USER TIMING 
RESOURCE 
TIMING 
REQUESTS BYTES IN ...
START RENDER 
1.58S 
DNS 
110MS 
TCP 
40MS 
TTFB 
DOM LOADING 6 5 0MS 
680MS 
DOM READY 
2.75S 
PAGE LOAD 
3.05 
FULLY LOA...
START RENDER 
1.58S 
DNS 
110MS 
TCP 
40MS 
TTFB 
DOM LOADING 6 5 0MS 
680MS 
DOM READY 
2.75S 
PAGE LOAD 
3.05 
FULLY LOA...
“What is the right number for me to communicate to the 
organization?” 
– CTO
REALISTICALLY, THERE IS NOT ONE NUMBER 
1s=$27M Page Load 
3.05s 
Start Render 
1.58s 
Speed Index 
3086
“How is site performance impacting revenue?” 
– CEO
TOP OF FUNNEL IMPACT (BROWSE PAGES) 
NOT ALL PAGES ARE CREATED EQUAL 
For a typical 
eCommerce site, 
conversion rate 
dro...
BOTTOM OF FUNNEL IMPACT (CHECKOUT PAGES) 
NOT ALL PAGES ARE CREATED EQUAL 
However, there is 
much less impact 
to convers...
HOW FAST SHOULD I BE? 
1s = $27M
“How do we compare to the competition?” 
– CEO
“How is our site doing in Spain? Are we winning there?” 
– CEO
SITE PERFORMANCE - SPAIN 
• Represents 1% of our overall traffic 
• 3rd overall in our competitive index 
3.91s Competitiv...
“How are third parties impacting my site?” 
– VP ECOMMERCE
Domains 
3RD PARTY STUDY - UNIQUE DOMAINS PER PAGE 
Alexa Top 25 Retail 
31 
10 
35 
23 
86 
41 
20 
37 
31 30 31 
4 
48 
...
91% 
9% 
91% of domains 
are 3rd parties 
3RD PARTY STUDY
HUFFINGTON POST - THIRD PART Y 
IMPACT USING PETTY CASH
SETTING MEANINGFUL SLAS 
Example SLA: Response time measured using resource timing 
from Chrome browsers in the United Sta...
“How should I measure my site in pre-production?” 
– FRONT END DEVELOPER
“Should I optimize my site for mobile?” 
– VP ECOMMERCE
Page Load Times 
Desktop vs. Mobile (Mobile and Tablet Optimized) 
Platform Median 95th Percentile 98th Percentile 
Mobile...
Page Load Times 
Desktop vs. Mobile (Full site on Tablet) 
Platform Median 95th Percentile 98th Percentile 
Mobile 4.70s 1...
3530 3530 
IMG 
JS 
CSS 
320 1024 1366 1920
768 1280 
1770 
637 
IMG 
JS 
320 1920
“How can I determine the effectiveness of my CDN 
provider?” 
– OPERATIONS
WHICH SITE IS USING A CDN? 
Site A Site B
“How has page construction changed over the last month, 
and what impact has that had on performance?” 
– FRONT END DEVELO...
“What is the performance of my single page web 
application?” 
– FRONT END DEVELOPER
THE CHALLENGE OF SINGLE-PAGE 
APPLICATIONS 
• Challenges with synthetic: Only measure first page 
load, not feasible to me...
“How do I know how fast my page becomes usable?” 
– DESIGNER
Start Render Loaded 
Usable
• Other questions?
SUMMARY 
• You need rich sets of data 
• There is not one number 
• Choose the right number(s) for the right audience
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
Synthetic and RUM - Best of bo
Upcoming SlideShare
Loading in …5
×

Synthetic and RUM - Best of bo

1,407 views

Published on

Velocity EU Presentation with Cliff Crocker (SOASTA) and Mark Zeman (SpeedCurve). Discussion on how to look at RUM and Synthetic performance data together.

Published in: Internet

Synthetic and RUM - Best of bo

  1. 1. SYNTHETIC AND RUM BEST OF BOTH WORLDS
  2. 2. Mark Zeman @SpeedCurve Cliff Crocker @cliffcrocker
  3. 3. ACTIVE MONITORING • Simulated health-checks of the system and it’s parts • Scripted journeys through an application • “Creation” of a singular or set of events and/or actions • Synthetic is a form of active monitoring AP Photo/ Eric Risberg
  4. 4. PASSIVE MONITORING • Listening to actual traffic as it moves through a system • Observing behavior of the system and/or its operator • “Measurement” of actions or events • RUM is a form of passive monitoring
  5. 5. Our focus is on the experience of the crowd allhumorpic.com
  6. 6. What is RUM? RUM 101 Real User User Measurement Measurement (traditional (RUM) definition) is a technology for collecting performance metrics directly from the browser of an end user. • Involves instrumentation of your website via JavaScript • Measurements are fired across the network to a collection point through a small request object (beacon) <JS> <beacon>
  7. 7. THE GOOD PARTS - RUM • Always on • Every user, every browser, every network, anywhere in the world • Ability to capture human behavior/events with performance data • Only getting better (waterfalls)
  8. 8. What is RUM? Real User Measurement (traditional definition) Synthetic 101 Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations. • A set ‘path’ or URL is defined • Tests are run either adhoc or scheduled and data is collected
  9. 9. THE GOOD PARTS - SYNTHETIC • Rich data collected (waterfall, videos/filmstrip, HTTP Headers) • Consistent “clean room” baseline • Nothing to install • Doesn’t require users/ability to measure pre-production, competition
  10. 10. COMMON THINGS WE HEAR ABOUT BOTH RUM AND SYNTHETIC • “Why are these numbers so different?” • “I don’t trust your data. Your numbers are wrong.” • “How are you calculating Page Load time?” • “I can’t share two sets of numbers with the business.”
  11. 11. WHY ARE THE TWO NUMBERS SO FAR OFF? Real users are not normal
  12. 12. Page Load Times All Countries, User Agents, Carriers, Operating Systems 3.23s – Median
  13. 13. Page Load Times All Countries, User Agents, Carriers, Operating Systems 10.45s – p95
  14. 14. Page Load Times All Countries, User Agents, Carriers, Operating Systems 17.26s – p98
  15. 15. Page Load Times By Operating System OS Median 95th Percentile 98th Percentile Windows 7 2.41s 9.29s 15.89s Mac OS X/10 2.30s 8.11s 13.45s iOS7 3.27s 10.64s 15.79s Android 4 4.06s 14.30s 27.93s iOS8 3.53s 11.54s 19.72s Windows 8 2.67s 10.75s 18.74s
  16. 16. WHY ARE THE TWO NUMBERS SO FAR OFF? Median: 3.23s RUM SYNTHETIC FIRST SYNTHETIC REPEAT 3 . 2 3 4 . 1 2 . 2 0 1 2 3 5 6 7 8 9 10 11 12 14 15 16 17 18 19
  17. 17. “BUT, IT LOADS SO MUCH FASTER FOR ME??!!” 4 seconds 20 seconds • 2015 Macbook Pro • Warm browser cache • FIOS vs. • x86 - Windows 7 VM • Completely cold cache/dns • Throttled bandwidth
  18. 18. RUM SYNTHETIC • Primary source of truth • User behavior/ conversion • Complete coverage • Diagnostics • Page Construction • Pre-production & Competitive benchmarking
  19. 19. RUM SYNTHETIC • Primary source of truth • User behavior/ conversion • Complete coverage • Diagnostics • Page Construction • Pre-production & Competitive benchmarking You need both
  20. 20. START RENDER 1.58S DNS 110MS TCP 40MS TTFB 650MS DOM LOADING 680MS DOM READY 2.75S PAGE LOAD 3.05 FULLY LOADED 19.68S USER TIMING 1.95S RESOURCE TIMING 237MS REQUESTS 116 BYTES IN 2,143KB SPEED INDEX 3086 PAGESPEED SCORE 83 1S=$27M DOM ELEMENTS 3931 DOM SIZE 3,143KB VISUALLY COMPLETE 15.2S REDIRECT 0S S S L NEGOTIATION 93MS THIS IS YOUR “ONE NUMBER”
  21. 21. THE MANY FACES OF PERFORMANCE C-Level Operations Developer Designer thenounproject.com
  22. 22. START RENDER DNS TCP TTFB DOM LOADING DOM READY PAGE LOAD FULLY LOADED USER TIMING RESOURCE TIMING REQUESTS BYTES IN SPEED INDEX PAGESPEED SCORE 1S=$$ DOM ELEMENTS DOM SIZE VISUALLY COMPLETE REDIRECT S S L NEGOTIATION START RENDER DNS TCP TTFB DOM LOADING DOM READY PAGE LOAD FULLY LOADED USER TIMING RESOURCE TIMING REQUESTS BYTES IN SPEED INDEX PAGESPEED SCORE 1S=$$ DOM ELEMENTS DOM SIZE VISUALLY COMPLETE REDIRECT S S L NEGOTIATION START RENDER DNS TCP TTFB DOM LOADING DOM READY PAGE LOAD FULLY LOADED USER TIMING RESOURCE TIMING REQUESTS BYTES IN SPEED INDEX PAGESPEED SCORE 1S=$$ DOM ELEMENTS DOM SIZE VISUALLY COMPLETE REDIRECT S S L NEGOTIATION START RENDER DNS TCP TTFB DOM LOADING DOM READY PAGE LOAD FULLY LOADED USER TIMING RESOURCE TIMING REQUESTS BYTES IN SPEED INDEX PAGESPEED SCORE 1S=$$ DOM ELEMENTS DOM SIZE VISUALLY COMPLETE REDIRECT S S L NEGOTIATION
  23. 23. START RENDER 1.58S DNS 110MS TCP 40MS TTFB DOM LOADING 6 5 0MS 680MS DOM READY 2.75S PAGE LOAD 3.05 FULLY LOADED US E R T IMING 1 9 . 6 8 S 1.95S RESOURCE TIMING 237MS REQUESTS 116 BYTES IN S P E ED INDE X 2 , 1 4 3KB 3086 PAGESPEED SCORE 83 1S=$27M DOM ELEMENTS DOM S I Z E VIZ COMPLETE REDIRECT 3 9 3 1 3,143KB 15.2S 0S SSL NEGOTIATION 93MS Analysis
  24. 24. START RENDER 1.58S DNS 110MS TCP 40MS TTFB DOM LOADING 6 5 0MS 680MS DOM READY 2.75S PAGE LOAD 3.05 FULLY LOADED US E R T IMING 1 9 . 6 8 S 1.95S RESOURCE TIMING 237MS REQUESTS 116 BYTES IN S P E ED INDE X 2 , 1 4 3KB 3086 PAGESPEED SCORE 83 1S=$27M DOM ELEMENTS DOM S I Z E VIZ COMPLETE REDIRECT 3 9 3 1 3,143KB 15.2S 0S SSL NEGOTIATION 93MS Analysis
  25. 25. “What is the right number for me to communicate to the organization?” – CTO
  26. 26. REALISTICALLY, THERE IS NOT ONE NUMBER 1s=$27M Page Load 3.05s Start Render 1.58s Speed Index 3086
  27. 27. “How is site performance impacting revenue?” – CEO
  28. 28. TOP OF FUNNEL IMPACT (BROWSE PAGES) NOT ALL PAGES ARE CREATED EQUAL For a typical eCommerce site, conversion rate drops by up to 50% when “browse” pages increase from 1 to 6 seconds
  29. 29. BOTTOM OF FUNNEL IMPACT (CHECKOUT PAGES) NOT ALL PAGES ARE CREATED EQUAL However, there is much less impact to conversion when “checkout” pages degrade
  30. 30. HOW FAST SHOULD I BE? 1s = $27M
  31. 31. “How do we compare to the competition?” – CEO
  32. 32. “How is our site doing in Spain? Are we winning there?” – CEO
  33. 33. SITE PERFORMANCE - SPAIN • Represents 1% of our overall traffic • 3rd overall in our competitive index 3.91s Competitive Index Site 1 Site 2 Site 3 Site 4 Site 5
  34. 34. “How are third parties impacting my site?” – VP ECOMMERCE
  35. 35. Domains 3RD PARTY STUDY - UNIQUE DOMAINS PER PAGE Alexa Top 25 Retail 31 10 35 23 86 41 20 37 31 30 31 4 48 38 42 16 11 6 28 68 31 36 4 8 19 2 1 2 5 3 3 1 5 2 2 1 1 5 3 2 3 4 2 3 5 1 7 3 7 17 Primary domains 3rd Party domains Domains: Median - 32 Min - 25 Max - 87
  36. 36. 91% 9% 91% of domains are 3rd parties 3RD PARTY STUDY
  37. 37. HUFFINGTON POST - THIRD PART Y IMPACT USING PETTY CASH
  38. 38. SETTING MEANINGFUL SLAS Example SLA: Response time measured using resource timing from Chrome browsers in the United States should not exceed a median (50th percentile) of 100ms or a 95th percentile of 500ms for a population of more than 500 users in a 24 hour period
  39. 39. “How should I measure my site in pre-production?” – FRONT END DEVELOPER
  40. 40. “Should I optimize my site for mobile?” – VP ECOMMERCE
  41. 41. Page Load Times Desktop vs. Mobile (Mobile and Tablet Optimized) Platform Median 95th Percentile 98th Percentile Mobile 3.62s 12.53s 20.04s Desktop 2.44s 9.31s 15.86s
  42. 42. Page Load Times Desktop vs. Mobile (Full site on Tablet) Platform Median 95th Percentile 98th Percentile Mobile 4.70s 17.87s 34.43s Desktop 3.81s 25.15 55.34s 0 1.5 3 4.5 6 7.5 9 10.5 12 13.5 15 16.5 18 19.5
  43. 43. 3530 3530 IMG JS CSS 320 1024 1366 1920
  44. 44. 768 1280 1770 637 IMG JS 320 1920
  45. 45. “How can I determine the effectiveness of my CDN provider?” – OPERATIONS
  46. 46. WHICH SITE IS USING A CDN? Site A Site B
  47. 47. “How has page construction changed over the last month, and what impact has that had on performance?” – FRONT END DEVELOPER
  48. 48. “What is the performance of my single page web application?” – FRONT END DEVELOPER
  49. 49. THE CHALLENGE OF SINGLE-PAGE APPLICATIONS • Challenges with synthetic: Only measure first page load, not feasible to measure secondary loads via XHR • Challenges with RUM: Determining whether call is a “page” or service call;Risk of diluting overall numbers
  50. 50. “How do I know how fast my page becomes usable?” – DESIGNER
  51. 51. Start Render Loaded Usable
  52. 52. • Other questions?
  53. 53. SUMMARY • You need rich sets of data • There is not one number • Choose the right number(s) for the right audience

×