Jon Fox
@jfox85
Performance Monitoring:
Synthetic
vs
RUM
Velocity 2013
Friday, June 28, 13
Performance Monitoring
Friday, June 28, 13
Performance Monitoring
Two primary methods:
- Synthetic (Active)
- Real User Measurement - RUM (Passive)
Friday, June 28, ...
Synthetic Overview
- Browser in the cloud
- Usually has extra instrumentation
Friday, June 28, 13
Synthetic Overview
- Often in controlled environments
- Reliable bandwidth
- Fixed specs
- Dedicated hardware
Friday, June...
Synthetic Overview
- Generally in several geographic locations
- Offer various browsers / rendering engines
Friday, June 2...
Synthetic - How It Works
Provider Your Website
Cloud Agent
Friday, June 28, 13
Synthetic - How It Works
Provider Your Website
Tell provider to run a test
Friday, June 28, 13
Synthetic - How It Works
Provider Your Website
Provider sends the test to an agent
Friday, June 28, 13
Synthetic - How It Works
Provider Your Website
The agent requests your website
Friday, June 28, 13
Synthetic - How It Works
Provider Your Website
The agent receives the page and renders the response
while measuring load t...
Synthetic - How It Works
Provider Your Website
The results are sent back to the provider to be viewed
and possibly aggrega...
Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13
Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13
Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13
Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13
Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13
Synthetic - What kind of data do we get?
Source: http://www.webpagetest.org/
Friday, June 28, 13
Real User Measurement (RUM)
On the other hand we have RUM
Friday, June 28, 13
RUM Overview
- Add JavaScript to the page
- Gather timing data from the browser
<script type='text/javascript'>var TBRUM=T...
RUM Overview
- Every page and every visitor
- Actual visitor timings
Friday, June 28, 13
RUM Overview
- Covers every device and browser that visits your site
Friday, June 28, 13
RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
Friday, June 28, 13
RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The site visitor decides to come to...
RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
They request your page and view it ...
RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The page includes the RUM Javascrip...
RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The timing data is then sent to the...
RUM - How It Works
Your Website
JavaScript
Beacon
Aggregation /
Processing
SiteVisitor
The data is then aggregated and pro...
RUM - Navigation Timing Spec
Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
Frida...
RUM - Navigation Timing Availability
Source: http://caniuse.com/nav-timing
Friday, June 28, 13
RUM - Other Timing Methods
- GoogleToolbar
Source: http://www.google.com/toolbar/ie/index.html
Friday, June 28, 13
RUM - Other Timing Methods
- Page unload cookie
Friday, June 28, 13
RUM - Other Timing Methods
- Start of document timer
Friday, June 28, 13
RUM - Other Data
- Estimated Bandwidth
- IP Address
- URL
- Referer
- Cookies
- Local Storage support
- Number of images, ...
RUM - Resource Timing Spec
Source: http://www.w3.org/TR/resource-timing/
Friday, June 28, 13
State of Resource Timing Spec
- IE10, Chrome, Chrome Mobile
-Various degrees of support / compatibility
- Same origin poli...
RUM - What kind of data do we get?
Friday, June 28, 13
RUM - What kind of data do we get?
Friday, June 28, 13
RUM - What kind of data do we get?
Friday, June 28, 13
RUM - What kind of data do we get?
Friday, June 28, 13
Comparison
So how do these methods compare?
Friday, June 28, 13
Methodology
Ran both on velocityconf.com
Torbit used for RUM
Webpagetest for Synthetic
Approximately 3 months (3/18/13 - 6...
Methodology
Synthetic hit the following URLs once an hour:
http://velocityconf.com/velocity2013/
http://velocityconf.com/v...
Methodology
From the following synthetic agents:
Dulles
- Chrome
- IE8
- IE9
- IE10
- Firefox
- Safari
Chicago
- Chrome
Sa...
Methodology
Real User Measurement:
RUM was run on every page
(as far as I know)
Friday, June 28, 13
Results - Data Overview
Synthetic Load Times
RUM Load Times
Friday, June 28, 13
Results - Data Overview
Synthetic Load Times
RUM Load Times
Friday, June 28, 13
Results - Browser Coverage
Synthetic RUM
- Chrome
- IE 10
- IE 9
- IE 8
- Firefox
- Safari
- Chrome
- IE 10
- IE 9
- IE 8
...
Results - Geographic Coverage
Synthetic:
Friday, June 28, 13
Results - Geographic Coverage
RUM:
Friday, June 28, 13
Results - Data Insights
RUM:
Synthetic:
0"
5"
10"
15"
20"
25"
30"
35"
40"
45"
50"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 1...
Results - Data Insights
0"
1"
2"
3"
4"
5"
6"
7"
8"
9"
10"
1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 1...
Results - Data Insights
Friday, June 28, 13
Results - Data Insights
Friday, June 28, 13
When To Use?
When should you use one over the other?
Friday, June 28, 13
When to use Synthetic
- Pre-production
- Low / no traffic pages
- Going deep on a problem
- Critical pages and paths
- Moni...
When to use RUM
- Ultimate source of truth
- Finding out you have a problem
- Monitoring trends over time
-Vendor or depar...
Build or Buy? - Synthetic
Should you build it yourself or buy a solution?
Friday, June 28, 13
Build - Synthetic
Open source
Self-host options
Friday, June 28, 13
Reasons to Buy - Synthetic
Need lots of agents
Friday, June 28, 13
Reasons to Buy - Synthetic
Need lots of locations
Friday, June 28, 13
Reasons to Buy - Synthetic
Simpler aggregation and reporting
Friday, June 28, 13
Build or Buy? - RUM
What about building your own RUM?
Friday, June 28, 13
Build - RUM
Collection is pretty easy...
Episodes - Steve Souders
http://stevesouders.com/episodes/
Boomerang - PhilipTell...
Reasons to Buy - RUM
Storage and processing are hard...
Friday, June 28, 13
Reasons to Buy - RUM
Dealing with outliers & bad data
Friday, June 28, 13
Conclusions
So which method should you use?
Friday, June 28, 13
Conclusions
A mix of both is best
Friday, June 28, 13
Why do we care?
And in case you need a reminder...
Friday, June 28, 13
Performance Matters
Aggregated data from 1,000+ Torbit Sites
Friday, June 28, 13
Performance Matters
Aggregated data from 1,000+ Torbit Sites
Friday, June 28, 13
Conclusions
Free options are available:
Synthetic RUM
Friday, June 28, 13
Conclusions
Start collecting data now!
Friday, June 28, 13
Questions?
Jon Fox
jon@torbit.com
@jfox85
Friday, June 28, 13
Upcoming SlideShare
Loading in...5
×

Velocity 2013 - Rum vs Synthetic

1,624

Published on

Everyone in the performance community is trying to better understand RUM and how to use it. In this presentation I'll introduce, compare, and contrast the two primary methods of performance monitoring. This will be done by using Synthetic and Real User Measurement side by side for 3 months and aggregating and presenting the data that the two methods yield.

Published in: Technology

Velocity 2013 - Rum vs Synthetic

  1. 1. Jon Fox @jfox85 Performance Monitoring: Synthetic vs RUM Velocity 2013 Friday, June 28, 13
  2. 2. Performance Monitoring Friday, June 28, 13
  3. 3. Performance Monitoring Two primary methods: - Synthetic (Active) - Real User Measurement - RUM (Passive) Friday, June 28, 13
  4. 4. Synthetic Overview - Browser in the cloud - Usually has extra instrumentation Friday, June 28, 13
  5. 5. Synthetic Overview - Often in controlled environments - Reliable bandwidth - Fixed specs - Dedicated hardware Friday, June 28, 13
  6. 6. Synthetic Overview - Generally in several geographic locations - Offer various browsers / rendering engines Friday, June 28, 13
  7. 7. Synthetic - How It Works Provider Your Website Cloud Agent Friday, June 28, 13
  8. 8. Synthetic - How It Works Provider Your Website Tell provider to run a test Friday, June 28, 13
  9. 9. Synthetic - How It Works Provider Your Website Provider sends the test to an agent Friday, June 28, 13
  10. 10. Synthetic - How It Works Provider Your Website The agent requests your website Friday, June 28, 13
  11. 11. Synthetic - How It Works Provider Your Website The agent receives the page and renders the response while measuring load times Friday, June 28, 13
  12. 12. Synthetic - How It Works Provider Your Website The results are sent back to the provider to be viewed and possibly aggregated with other results Friday, June 28, 13
  13. 13. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  14. 14. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  15. 15. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  16. 16. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  17. 17. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  18. 18. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  19. 19. Real User Measurement (RUM) On the other hand we have RUM Friday, June 28, 13
  20. 20. RUM Overview - Add JavaScript to the page - Gather timing data from the browser <script type='text/javascript'>var TBRUM=TBRUM||{};TBRUM.q=TBRUM.q|| [];TBRUM.q.push(['mark','firstbyte',(new Date).getTime()]);</script> Friday, June 28, 13
  21. 21. RUM Overview - Every page and every visitor - Actual visitor timings Friday, June 28, 13
  22. 22. RUM Overview - Covers every device and browser that visits your site Friday, June 28, 13
  23. 23. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor Friday, June 28, 13
  24. 24. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The site visitor decides to come to your site Friday, June 28, 13
  25. 25. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor They request your page and view it as normal Friday, June 28, 13
  26. 26. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The page includes the RUM Javascript, which collects timing data from the browser SiteVisitor Friday, June 28, 13
  27. 27. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The timing data is then sent to the collection service via a beacon image request SiteVisitor Friday, June 28, 13
  28. 28. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The data is then aggregated and processed to generate pretty charts and graphs SiteVisitor Friday, June 28, 13
  29. 29. RUM - Navigation Timing Spec Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html Friday, June 28, 13
  30. 30. RUM - Navigation Timing Availability Source: http://caniuse.com/nav-timing Friday, June 28, 13
  31. 31. RUM - Other Timing Methods - GoogleToolbar Source: http://www.google.com/toolbar/ie/index.html Friday, June 28, 13
  32. 32. RUM - Other Timing Methods - Page unload cookie Friday, June 28, 13
  33. 33. RUM - Other Timing Methods - Start of document timer Friday, June 28, 13
  34. 34. RUM - Other Data - Estimated Bandwidth - IP Address - URL - Referer - Cookies - Local Storage support - Number of images, JavaScript, CSS, etc - User actions on the page - Memory Usage (very limited availability) - Anything else you can get from JavaScript! Friday, June 28, 13
  35. 35. RUM - Resource Timing Spec Source: http://www.w3.org/TR/resource-timing/ Friday, June 28, 13
  36. 36. State of Resource Timing Spec - IE10, Chrome, Chrome Mobile -Various degrees of support / compatibility - Same origin policy by default - Override withTiming-Allow-Origin HTTP response header Friday, June 28, 13
  37. 37. RUM - What kind of data do we get? Friday, June 28, 13
  38. 38. RUM - What kind of data do we get? Friday, June 28, 13
  39. 39. RUM - What kind of data do we get? Friday, June 28, 13
  40. 40. RUM - What kind of data do we get? Friday, June 28, 13
  41. 41. Comparison So how do these methods compare? Friday, June 28, 13
  42. 42. Methodology Ran both on velocityconf.com Torbit used for RUM Webpagetest for Synthetic Approximately 3 months (3/18/13 - 6/14/13) Friday, June 28, 13
  43. 43. Methodology Synthetic hit the following URLs once an hour: http://velocityconf.com/velocity2013/ http://velocityconf.com/velocity2013/public/schedule/topic/935 http://velocityconf.com/velocity2013/public/content/exhibitors http://velocityconf.com/velocity2013/public/schedule/grid http://velocityconf.com/velocity2013/public/content/about http://velocityconf.com/velocity2013/public/schedule/speakers Friday, June 28, 13
  44. 44. Methodology From the following synthetic agents: Dulles - Chrome - IE8 - IE9 - IE10 - Firefox - Safari Chicago - Chrome San Jose - IE9 Los Angeles - Chrome Dublin - IE8 Sao Paulo - IE8 Toronto - IE8 Amsterdam - IE9 - Chrome Frankfurt - IE9 - Firefox Moscow - Chrome - IE8 Stockholm - Safari Delhi - IE8 China - IE 8 Tokyo - IE 8 Sydney - Chrome Friday, June 28, 13
  45. 45. Methodology Real User Measurement: RUM was run on every page (as far as I know) Friday, June 28, 13
  46. 46. Results - Data Overview Synthetic Load Times RUM Load Times Friday, June 28, 13
  47. 47. Results - Data Overview Synthetic Load Times RUM Load Times Friday, June 28, 13
  48. 48. Results - Browser Coverage Synthetic RUM - Chrome - IE 10 - IE 9 - IE 8 - Firefox - Safari - Chrome - IE 10 - IE 9 - IE 8 - IE 7 - Firefox - Safari - Opera - Android - iPhone - And more... Friday, June 28, 13
  49. 49. Results - Geographic Coverage Synthetic: Friday, June 28, 13
  50. 50. Results - Geographic Coverage RUM: Friday, June 28, 13
  51. 51. Results - Data Insights RUM: Synthetic: 0" 5" 10" 15" 20" 25" 30" 35" 40" 45" 50" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" Load%Time%(sec)% Time%(hrs)% RUM"Avg" 90th" Syn"Avg" Friday, June 28, 13
  52. 52. Results - Data Insights 0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29" Load%Times%(sec)% Time%(hrs)% RUM"Avg" RUM"90th" Syn"Avg" 0" 200" 400" 600" 800" 1000" 1200" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29" Pageviews% Pageviews" Friday, June 28, 13
  53. 53. Results - Data Insights Friday, June 28, 13
  54. 54. Results - Data Insights Friday, June 28, 13
  55. 55. When To Use? When should you use one over the other? Friday, June 28, 13
  56. 56. When to use Synthetic - Pre-production - Low / no traffic pages - Going deep on a problem - Critical pages and paths - Monitoring competitors Friday, June 28, 13
  57. 57. When to use RUM - Ultimate source of truth - Finding out you have a problem - Monitoring trends over time -Vendor or departmental SLAs Friday, June 28, 13
  58. 58. Build or Buy? - Synthetic Should you build it yourself or buy a solution? Friday, June 28, 13
  59. 59. Build - Synthetic Open source Self-host options Friday, June 28, 13
  60. 60. Reasons to Buy - Synthetic Need lots of agents Friday, June 28, 13
  61. 61. Reasons to Buy - Synthetic Need lots of locations Friday, June 28, 13
  62. 62. Reasons to Buy - Synthetic Simpler aggregation and reporting Friday, June 28, 13
  63. 63. Build or Buy? - RUM What about building your own RUM? Friday, June 28, 13
  64. 64. Build - RUM Collection is pretty easy... Episodes - Steve Souders http://stevesouders.com/episodes/ Boomerang - PhilipTellis https://github.com/lognormal/boomerang Friday, June 28, 13
  65. 65. Reasons to Buy - RUM Storage and processing are hard... Friday, June 28, 13
  66. 66. Reasons to Buy - RUM Dealing with outliers & bad data Friday, June 28, 13
  67. 67. Conclusions So which method should you use? Friday, June 28, 13
  68. 68. Conclusions A mix of both is best Friday, June 28, 13
  69. 69. Why do we care? And in case you need a reminder... Friday, June 28, 13
  70. 70. Performance Matters Aggregated data from 1,000+ Torbit Sites Friday, June 28, 13
  71. 71. Performance Matters Aggregated data from 1,000+ Torbit Sites Friday, June 28, 13
  72. 72. Conclusions Free options are available: Synthetic RUM Friday, June 28, 13
  73. 73. Conclusions Start collecting data now! Friday, June 28, 13
  74. 74. Questions? Jon Fox jon@torbit.com @jfox85 Friday, June 28, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×