Who I Am
Cliff Crocker
VP Product
SOASTA, Inc.
@cliffcrocker
http://www.flickr.com/photos/luc/
Real Users
http://www.flickr.com/photos/spullara/
Happy Users
http://www.flickr.com/photos/sybrenstuvel/
Not So
Happy
RUM 101
Real User Measurement (RUM) is a technology for collecting
performance metrics directly from the browser of an end...
Origins of RUM
2005-6: Steve S. builds
JS tool to measure
page load from the
browser during a
Yahoo! hackday
2008: Steve o...
This helped…
Navigation Timing API
• Exposes network level timing details to JavaScript
• DNS lookup, TCP connect, SSL handshake, First...
C’mon Safari, be cool…
http://caniuse.com/nav-timing
~70%
Browser
Support
Why RUM is Awesome
• Reduces the level of convincing we have to do within our
organization that ‘faster is better’
• The o...
What gives??
No RUM
RUM • According to a study
done in 2012* only 1
in 5 of the top
ecommerce sites are
using some form of...
Challenges with RUM
1. It’s a “data” problem
2. People don’t like change
3. Hard to “sell” internally
For a modest sized site, in a given day
1-3 Million Page Views
Perf Timing
User Agent
OS/Device
Geography
Page
Segmentatio...
Many have advocated aggressive
sampling of data…
~1% sample, looking at a period of about 3 hours for a given page segment (Checkout)
1% > checkout
Filtered to the United States…
1% > >checkout
Looking specifically at California…
1% > > >checkout
IE9 Browsers only.
1% > > >checkout
Looking at all traffic.
100% > checkout
All filters applied, identified bimodal distribution driven by backend time.
100% > > >checkout
> backend time
Sampled at 1%
Not sampled
To be fair…
Read:
http://www.webperformancetoday.com/2013/03/06/big-
data-vs-big-enough-data/
Many use a batch solution…
• Batch processing large amounts of data has
been a traditional approach for RUM
– Great for looking at historical data, t...
In order for RUM to be operationally relevant,
you need the data in real-time
Synthetic vs. RUM
http://www.flickr.com/photos/myvector/
It is Not Either Or
• Synthetic and RUM coexist
• Important to understand:
– Geographic performance
– Object level detail ...
Performance by Geography
RUM
????
Performance by Geography
Synthetic
Understanding Performance by
Geography
• RUM provides
better visibility
into geographic
variability (simply
more data poin...
Object Level Detail
RUM 1
Synthetic 1
Waterfall available for most
major browsers available for
synthetic monitoring (IE, ...
Browser Coverage
RUM 2
Synthetic 1
Synthetic
Support for major browsers
Commercial
(mainstream
synthetic)
WebPagetest
supp...
Competitive Positioning
RUM 2
Synthetic 2
Synthetic
Ability to measure competitors easily
RUM
More complex, no standard
be...
Site Coverage
RUM 3
Synthetic 2
Synthetic
Choose a ‘happy path’ or spot check
RUM
No blind spots!
http://scoutmastercg.com...
Synthetic Monitoring
• Pros
– Technology is great
– Real Browsers (IE,
Chrome, FF)
– Built in Alerting
– Waterfall analysi...
RUM
Cast a wide net
• Identify key areas of concern
• Understand real user impact
• Tie this back to $$ (next)
Synthetic
D...
RUM in the
Workplace
Studies show…
However, you have to look at your own
data
Understand your KPIs and How to
Track Them
• Retail: Order Conversion, Add to Cart
Conversion, Revenue
• Media: In Line Di...
Bottom Line, if you want your business
to care – then make it relevant for
them (speak their language)
Strategy for Success – RUM First
RUM First
• All the data (or
statistically
relevant
sample)
• Real-time and
historical
Su...
mPulse Lite
http://www.soasta.com/free/
Rum first   london web perf meetup
Rum first   london web perf meetup
Upcoming SlideShare
Loading in …5
×

Rum first london web perf meetup

817 views
712 views

Published on

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
817
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Rum first london web perf meetup

  1. 1. Who I Am Cliff Crocker VP Product SOASTA, Inc. @cliffcrocker
  2. 2. http://www.flickr.com/photos/luc/ Real Users
  3. 3. http://www.flickr.com/photos/spullara/ Happy Users
  4. 4. http://www.flickr.com/photos/sybrenstuvel/ Not So Happy
  5. 5. RUM 101 Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user. • Involves instrumentation of the page with JavaScript • Measurements are fired across the network to a collection point through a small request object (beacon) <JS> MySite.com <beacon>
  6. 6. Origins of RUM 2005-6: Steve S. builds JS tool to measure page load from the browser during a Yahoo! hackday 2008: Steve open sources ‘Episodes’. White pages open sources ‘Jiffy’ 2010: Yahoo! Open sources ‘boomerang.js’ the same day Navigation Timing is introduced in IE9 beta • http://stevesouders.com/episodes/ • https://code.google.com/p/jiffy-web/ • http://lognormal.github.com/boomerang/doc/ • https://dvcs.w3.org/hg/webperf/raw- file/tip/specs/NavigationTiming/Overview.html
  7. 7. This helped…
  8. 8. Navigation Timing API • Exposes network level timing details to JavaScript • DNS lookup, TCP connect, SSL handshake, First Byte time, and more • For a given web request, you can tell exactly where the time was spent • Supported by IE >= 9, Chrome >=6, Firefox >=7, Android Browser .=4.0, others • No exposure into rendering times
  9. 9. C’mon Safari, be cool… http://caniuse.com/nav-timing ~70% Browser Support
  10. 10. Why RUM is Awesome • Reduces the level of convincing we have to do within our organization that ‘faster is better’ • The only way to measure perceived performance from your end user • Easy to maintain • Only getting better… – Check out:http://blog.patrickmeenan.com/2013/07/measuring- performance-of-user-experience.html • User Timing for RUM and Synthetic – Fun with resource timing: https://github.com/andydavies/waterfall
  11. 11. What gives?? No RUM RUM • According to a study done in 2012* only 1 in 5 of the top ecommerce sites are using some form of RUM *http://www.webperformancetoday.com/2012/11/13/real-user-monitoring-rum-ecommerce-sites/
  12. 12. Challenges with RUM 1. It’s a “data” problem 2. People don’t like change 3. Hard to “sell” internally
  13. 13. For a modest sized site, in a given day 1-3 Million Page Views Perf Timing User Agent OS/Device Geography Page Segmentation Bandwidth Classification A/B Bucket ‘Other’
  14. 14. Many have advocated aggressive sampling of data…
  15. 15. ~1% sample, looking at a period of about 3 hours for a given page segment (Checkout) 1% > checkout
  16. 16. Filtered to the United States… 1% > >checkout
  17. 17. Looking specifically at California… 1% > > >checkout
  18. 18. IE9 Browsers only. 1% > > >checkout
  19. 19. Looking at all traffic. 100% > checkout
  20. 20. All filters applied, identified bimodal distribution driven by backend time. 100% > > >checkout > backend time
  21. 21. Sampled at 1% Not sampled
  22. 22. To be fair… Read: http://www.webperformancetoday.com/2013/03/06/big- data-vs-big-enough-data/
  23. 23. Many use a batch solution…
  24. 24. • Batch processing large amounts of data has been a traditional approach for RUM – Great for looking at historical data, trending, etc. – Big Data is not necessarily Fast Data • Not ‘Operational’
  25. 25. In order for RUM to be operationally relevant, you need the data in real-time
  26. 26. Synthetic vs. RUM http://www.flickr.com/photos/myvector/
  27. 27. It is Not Either Or • Synthetic and RUM coexist • Important to understand: – Geographic performance – Object level detail (waterfall) – Browser performance – Competitive positioning – Site coverage
  28. 28. Performance by Geography RUM
  29. 29. ???? Performance by Geography Synthetic
  30. 30. Understanding Performance by Geography • RUM provides better visibility into geographic variability (simply more data points from more locations) RUM 1 Synthetic 0
  31. 31. Object Level Detail RUM 1 Synthetic 1 Waterfall available for most major browsers available for synthetic monitoring (IE, FF (Commercial), Chrome, Safari) Limited support for waterfall using resource timing (IE10 & Chrome stable) – using waterfall.js
  32. 32. Browser Coverage RUM 2 Synthetic 1 Synthetic Support for major browsers Commercial (mainstream synthetic) WebPagetest support RUM Support for any browser that executes JS
  33. 33. Competitive Positioning RUM 2 Synthetic 2 Synthetic Ability to measure competitors easily RUM More complex, no standard benchmarks (yet!) Commercial offerings WebPagetest.org
  34. 34. Site Coverage RUM 3 Synthetic 2 Synthetic Choose a ‘happy path’ or spot check RUM No blind spots! http://scoutmastercg.com/wp- content/uploads/typimg/green_laser.png http://www.flickr.com/photos/areese/with/4002609608/
  35. 35. Synthetic Monitoring • Pros – Technology is great – Real Browsers (IE, Chrome, FF) – Built in Alerting – Waterfall analysis – Screen shots & headers – Object level detail • Cons – False Positives – Impossible/impractical to measure everything – Fixed number of browser/OS combinations – Simulated bandwidth constraints at best – Too few data points for statistical relevance
  36. 36. RUM Cast a wide net • Identify key areas of concern • Understand real user impact • Tie this back to $$ (next) Synthetic Diagnostic tool • Identify issues in a ‘lab’ • Reproduce a problem found with RUM http://www.flickr.com/photos/84338444@N00/with/3780079044/ http://www.flickr.com/photos/ezioman/
  37. 37. RUM in the Workplace
  38. 38. Studies show…
  39. 39. However, you have to look at your own data
  40. 40. Understand your KPIs and How to Track Them • Retail: Order Conversion, Add to Cart Conversion, Revenue • Media: In Line Display Ad Views, Comments, • Other: Facebook likes/Social Media sharing
  41. 41. Bottom Line, if you want your business to care – then make it relevant for them (speak their language)
  42. 42. Strategy for Success – RUM First RUM First • All the data (or statistically relevant sample) • Real-time and historical Supplement with Synthetic • Reduce measurements based on coverage with RUM • Use for adhoc diagnostic & competitive benchmarking Map to business KPIs • YOUR goals • YOUR data • Single source of truth
  43. 43. mPulse Lite http://www.soasta.com/free/

×