Velocity 2013 - Rum vs Synthetic
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Velocity 2013 - Rum vs Synthetic

  • 1,611 views
Uploaded on

Everyone in the performance community is trying to better understand RUM and how to use it. In this presentation I'll introduce, compare, and contrast the two primary methods of performance ...

Everyone in the performance community is trying to better understand RUM and how to use it. In this presentation I'll introduce, compare, and contrast the two primary methods of performance monitoring. This will be done by using Synthetic and Real User Measurement side by side for 3 months and aggregating and presenting the data that the two methods yield.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,611
On Slideshare
1,539
From Embeds
72
Number of Embeds
4

Actions

Shares
Downloads
56
Comments
0
Likes
4

Embeds 72

http://velocityconf.com 64
https://confluence.softserveinc.com 6
https://jira.softserveinc.com 1
http://www.feweekly.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Jon Fox @jfox85 Performance Monitoring: Synthetic vs RUM Velocity 2013 Friday, June 28, 13
  • 2. Performance Monitoring Friday, June 28, 13
  • 3. Performance Monitoring Two primary methods: - Synthetic (Active) - Real User Measurement - RUM (Passive) Friday, June 28, 13
  • 4. Synthetic Overview - Browser in the cloud - Usually has extra instrumentation Friday, June 28, 13
  • 5. Synthetic Overview - Often in controlled environments - Reliable bandwidth - Fixed specs - Dedicated hardware Friday, June 28, 13
  • 6. Synthetic Overview - Generally in several geographic locations - Offer various browsers / rendering engines Friday, June 28, 13
  • 7. Synthetic - How It Works Provider Your Website Cloud Agent Friday, June 28, 13
  • 8. Synthetic - How It Works Provider Your Website Tell provider to run a test Friday, June 28, 13
  • 9. Synthetic - How It Works Provider Your Website Provider sends the test to an agent Friday, June 28, 13
  • 10. Synthetic - How It Works Provider Your Website The agent requests your website Friday, June 28, 13
  • 11. Synthetic - How It Works Provider Your Website The agent receives the page and renders the response while measuring load times Friday, June 28, 13
  • 12. Synthetic - How It Works Provider Your Website The results are sent back to the provider to be viewed and possibly aggregated with other results Friday, June 28, 13
  • 13. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  • 14. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  • 15. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  • 16. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  • 17. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  • 18. Synthetic - What kind of data do we get? Source: http://www.webpagetest.org/ Friday, June 28, 13
  • 19. Real User Measurement (RUM) On the other hand we have RUM Friday, June 28, 13
  • 20. RUM Overview - Add JavaScript to the page - Gather timing data from the browser <script type='text/javascript'>var TBRUM=TBRUM||{};TBRUM.q=TBRUM.q|| [];TBRUM.q.push(['mark','firstbyte',(new Date).getTime()]);</script> Friday, June 28, 13
  • 21. RUM Overview - Every page and every visitor - Actual visitor timings Friday, June 28, 13
  • 22. RUM Overview - Covers every device and browser that visits your site Friday, June 28, 13
  • 23. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor Friday, June 28, 13
  • 24. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The site visitor decides to come to your site Friday, June 28, 13
  • 25. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor They request your page and view it as normal Friday, June 28, 13
  • 26. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The page includes the RUM Javascript, which collects timing data from the browser SiteVisitor Friday, June 28, 13
  • 27. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The timing data is then sent to the collection service via a beacon image request SiteVisitor Friday, June 28, 13
  • 28. RUM - How It Works Your Website JavaScript Beacon Aggregation / Processing SiteVisitor The data is then aggregated and processed to generate pretty charts and graphs SiteVisitor Friday, June 28, 13
  • 29. RUM - Navigation Timing Spec Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html Friday, June 28, 13
  • 30. RUM - Navigation Timing Availability Source: http://caniuse.com/nav-timing Friday, June 28, 13
  • 31. RUM - Other Timing Methods - GoogleToolbar Source: http://www.google.com/toolbar/ie/index.html Friday, June 28, 13
  • 32. RUM - Other Timing Methods - Page unload cookie Friday, June 28, 13
  • 33. RUM - Other Timing Methods - Start of document timer Friday, June 28, 13
  • 34. RUM - Other Data - Estimated Bandwidth - IP Address - URL - Referer - Cookies - Local Storage support - Number of images, JavaScript, CSS, etc - User actions on the page - Memory Usage (very limited availability) - Anything else you can get from JavaScript! Friday, June 28, 13
  • 35. RUM - Resource Timing Spec Source: http://www.w3.org/TR/resource-timing/ Friday, June 28, 13
  • 36. State of Resource Timing Spec - IE10, Chrome, Chrome Mobile -Various degrees of support / compatibility - Same origin policy by default - Override withTiming-Allow-Origin HTTP response header Friday, June 28, 13
  • 37. RUM - What kind of data do we get? Friday, June 28, 13
  • 38. RUM - What kind of data do we get? Friday, June 28, 13
  • 39. RUM - What kind of data do we get? Friday, June 28, 13
  • 40. RUM - What kind of data do we get? Friday, June 28, 13
  • 41. Comparison So how do these methods compare? Friday, June 28, 13
  • 42. Methodology Ran both on velocityconf.com Torbit used for RUM Webpagetest for Synthetic Approximately 3 months (3/18/13 - 6/14/13) Friday, June 28, 13
  • 43. Methodology Synthetic hit the following URLs once an hour: http://velocityconf.com/velocity2013/ http://velocityconf.com/velocity2013/public/schedule/topic/935 http://velocityconf.com/velocity2013/public/content/exhibitors http://velocityconf.com/velocity2013/public/schedule/grid http://velocityconf.com/velocity2013/public/content/about http://velocityconf.com/velocity2013/public/schedule/speakers Friday, June 28, 13
  • 44. Methodology From the following synthetic agents: Dulles - Chrome - IE8 - IE9 - IE10 - Firefox - Safari Chicago - Chrome San Jose - IE9 Los Angeles - Chrome Dublin - IE8 Sao Paulo - IE8 Toronto - IE8 Amsterdam - IE9 - Chrome Frankfurt - IE9 - Firefox Moscow - Chrome - IE8 Stockholm - Safari Delhi - IE8 China - IE 8 Tokyo - IE 8 Sydney - Chrome Friday, June 28, 13
  • 45. Methodology Real User Measurement: RUM was run on every page (as far as I know) Friday, June 28, 13
  • 46. Results - Data Overview Synthetic Load Times RUM Load Times Friday, June 28, 13
  • 47. Results - Data Overview Synthetic Load Times RUM Load Times Friday, June 28, 13
  • 48. Results - Browser Coverage Synthetic RUM - Chrome - IE 10 - IE 9 - IE 8 - Firefox - Safari - Chrome - IE 10 - IE 9 - IE 8 - IE 7 - Firefox - Safari - Opera - Android - iPhone - And more... Friday, June 28, 13
  • 49. Results - Geographic Coverage Synthetic: Friday, June 28, 13
  • 50. Results - Geographic Coverage RUM: Friday, June 28, 13
  • 51. Results - Data Insights RUM: Synthetic: 0" 5" 10" 15" 20" 25" 30" 35" 40" 45" 50" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" Load%Time%(sec)% Time%(hrs)% RUM"Avg" 90th" Syn"Avg" Friday, June 28, 13
  • 52. Results - Data Insights 0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29" Load%Times%(sec)% Time%(hrs)% RUM"Avg" RUM"90th" Syn"Avg" 0" 200" 400" 600" 800" 1000" 1200" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15" 16" 17" 18" 19" 20" 21" 22" 23" 24" 25" 26" 27" 28" 29" Pageviews% Pageviews" Friday, June 28, 13
  • 53. Results - Data Insights Friday, June 28, 13
  • 54. Results - Data Insights Friday, June 28, 13
  • 55. When To Use? When should you use one over the other? Friday, June 28, 13
  • 56. When to use Synthetic - Pre-production - Low / no traffic pages - Going deep on a problem - Critical pages and paths - Monitoring competitors Friday, June 28, 13
  • 57. When to use RUM - Ultimate source of truth - Finding out you have a problem - Monitoring trends over time -Vendor or departmental SLAs Friday, June 28, 13
  • 58. Build or Buy? - Synthetic Should you build it yourself or buy a solution? Friday, June 28, 13
  • 59. Build - Synthetic Open source Self-host options Friday, June 28, 13
  • 60. Reasons to Buy - Synthetic Need lots of agents Friday, June 28, 13
  • 61. Reasons to Buy - Synthetic Need lots of locations Friday, June 28, 13
  • 62. Reasons to Buy - Synthetic Simpler aggregation and reporting Friday, June 28, 13
  • 63. Build or Buy? - RUM What about building your own RUM? Friday, June 28, 13
  • 64. Build - RUM Collection is pretty easy... Episodes - Steve Souders http://stevesouders.com/episodes/ Boomerang - PhilipTellis https://github.com/lognormal/boomerang Friday, June 28, 13
  • 65. Reasons to Buy - RUM Storage and processing are hard... Friday, June 28, 13
  • 66. Reasons to Buy - RUM Dealing with outliers & bad data Friday, June 28, 13
  • 67. Conclusions So which method should you use? Friday, June 28, 13
  • 68. Conclusions A mix of both is best Friday, June 28, 13
  • 69. Why do we care? And in case you need a reminder... Friday, June 28, 13
  • 70. Performance Matters Aggregated data from 1,000+ Torbit Sites Friday, June 28, 13
  • 71. Performance Matters Aggregated data from 1,000+ Torbit Sites Friday, June 28, 13
  • 72. Conclusions Free options are available: Synthetic RUM Friday, June 28, 13
  • 73. Conclusions Start collecting data now! Friday, June 28, 13
  • 74. Questions? Jon Fox jon@torbit.com @jfox85 Friday, June 28, 13