Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Measuring What Matters
© AKAMAI - EDGE 2016
Introductions
Cliff Crocker
Akamai
Buddy Brewer
SOASTA
© AKAMAI - EDGE 2016
Three eras of modern performance measurement
2001-2006 Basically one browser
2007-2010 Transitional p...
© AKAMAI - EDGE 2016
2001-2006: Internet Explorer 6
2001 - 2006
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7
iPhone
Firefox 2
2008 2009 2010
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7
iPhone
Firefox 2
2008
IE7
iPhone 3G
Firefox 3
Android
Chrome ...
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7
iPhone
Firefox 2
2008
IE7
iPhone 3G
Firefox 3
Android
Chrome ...
© AKAMAI - EDGE 2016
2007-2010: Transitional period
2007
IE7
iPhone
Firefox 2
2008
IE7
iPhone 3G
Firefox 3
Android
Chrome ...
© AKAMAI - EDGE 2016
2011-2016: WTF just happened
?
How many browsers in a
typical Thanksgiving
weekend?
© AKAMAI - EDGE 2016
2011-2016: WTF just happened
572
How many browsers in a
typical Thanksgiving
weekend?
© AKAMAI - EDGE 2016
Meanwhile...
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
So what can I measure?
© AKAMAI - EDGE 2016
What is available today - Synthetic Tooling
Pros
CPU metrics
Test environments & other places without...
© AKAMAI - EDGE 2016
What is available today - RUM
Pros
Near perfect coverage of browsers, geos, pages
Ability to understa...
© AKAMAI - EDGE 2016
Engagement
Revenue
Coverage
Caching
CPU
Pre-prod
Availability
Page details
Resources
Headers
Syntheti...
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
https://www.w3.org/wiki/Web_Performance/Publications
W3C - Performance Working Group
© AKAMAI - EDGE 2016
Navigation Timing API
© AKAMAI - EDGE 2016
Resource Timing API
© AKAMAI - EDGE 2016
Coming soon to a browser
near you! Measure the size of
assets in addition to
performance.
Resource Ti...
© AKAMAI - EDGE 2016
● Measure performance using high-precision
timestamps
● Consists of “marks” and “measures”
● Performa...
© AKAMAI - EDGE 2016
Browser Support - Navigation Timing API
© AKAMAI - EDGE 2016
Browser Support - Resource Timing API
© AKAMAI - EDGE 2016
Browser Support - User Timing API
© AKAMAI - EDGE 2016
So what should I measure?
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
What matters depends on you
© AKAMAI - EDGE 2016
What matters to an ops engineer?
● Preference for binary metrics
● Looks for indicators of system per...
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Network
Server
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Network
Server
function getPerf() {
var timing = window.performance....
© AKAMAI - EDGE 2016
Help answers questions like...
● Do I have an issue with an upstream DNS resolver?
● Am I having Tier...
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Don’t be misled by “super fast” responses
● DNS Caching
● Prefetchin...
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Object Level
Timing
● Resource timing API
● Binary metric for testin...
© AKAMAI - EDGE 2016
What matters to an ops engineer?
Object Timing
var rUrl =
‘https://edge.akamai.com/ec/us/multimed
ia/...
© AKAMAI - EDGE 2016
What matters to a front-end developer?
Optimized
Load Times
● “Do no harm”
● Looks for indicators of ...
© AKAMAI - EDGE 2016
What matters to a front-end developer?
Client
© AKAMAI - EDGE 2016
What matters to a front-end developer?
© AKAMAI - EDGE 2016
What matters to a front-end developer?
function getPerf() {
var timing = window.performance.timing;
r...
© AKAMAI - EDGE 2016
Be wary of onload...
~1s OnLoad ~3.5s visibly complete
© AKAMAI - EDGE 2016
What matters to a designer?
UX
● Performance by design (D4P)
● Looks for indicators of human percepti...
© AKAMAI - EDGE 2016
RUM first paint and start render
Chrome - “First Paint” Visual Start Render
© AKAMAI - EDGE 2016
User perception
Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html
© AKAMAI - EDGE 2016
Custom Metrics
© AKAMAI - EDGE 2016
Custom Metrics - User Timing API
performance.mark(“startTask”);
//Some stuff you want to measure happ...
© AKAMAI - EDGE 2016
What matters to a business executive?
$$$$$$$
● Business impact analysis
● Simplified/lean metrics
● ...
© AKAMAI - EDGE 2016
What matters to a business executive?
© AKAMAI - EDGE 2016
What matters to a business executive?
© AKAMAI - EDGE 2016
What matters to a business executive?
© AKAMAI - EDGE 2016
Don’t create unintended silos
Biz
Ops
Dev
Design
© AKAMAI - EDGE 2016
Don’t create unintended silos
The success of my
business is a
reflection of...
The digital experience...
© AKAMAI - EDGE 2016
“I just want to buy my Mom’s
birthday present.”
“I just want to know if it will
rain today.”
“I just ...
© AKAMAI - EDGE 2016
How do I collect it?
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
Real users are not normal
What do I do with it?
© AKAMAI - EDGE 2016
What do I do with it?
© AKAMAI - EDGE 2016
2.9s - p50 (median)
4.4s - Average
10.4s - p95
What do I do with it?
© AKAMAI - EDGE 2016
What should I throw out?
© AKAMAI - EDGE 2016
Load Time =
3500 years
© AKAMAI - EDGE 2016
Load Time =
-(10years)
© AKAMAI - EDGE 2016
What should I keep?
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
© AKAMAI - EDGE 2016
Other opportunities for bots...
1. How does performance impact bot behavior?
2. Are bots skewing the ...
© AKAMAI - EDGE 2016
Summary
● We have amazing tools at our disposal - your users are
wired for sound!
● Measuring what ma...
Upcoming SlideShare
Loading in …5
×

Measuring what matters

473 views

Published on

Metrics are everywhere! We’ve done a great job of keeping pace with measuring the output of our applications, but how are we doing with measuring what really matters? This talk will explore the various metrics available to application owners today, highlight what’s coming tomorrow and level-set on the relative importance as it relates to the user experience.

Published in: Technology
  • Be the first to comment

Measuring what matters

  1. 1. Measuring What Matters
  2. 2. © AKAMAI - EDGE 2016 Introductions Cliff Crocker Akamai Buddy Brewer SOASTA
  3. 3. © AKAMAI - EDGE 2016 Three eras of modern performance measurement 2001-2006 Basically one browser 2007-2010 Transitional period 2011-2016 WTF just happened
  4. 4. © AKAMAI - EDGE 2016 2001-2006: Internet Explorer 6 2001 - 2006
  5. 5. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 2009 2010
  6. 6. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 IE7 iPhone 3G Firefox 3 Android Chrome 1 2009 2010
  7. 7. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 IE7 iPhone 3G Firefox 3 Android Chrome 1 2009 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 2010
  8. 8. © AKAMAI - EDGE 2016 2007-2010: Transitional period 2007 IE7 iPhone Firefox 2 2008 IE7 iPhone 3G Firefox 3 Android Chrome 1 2009 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 2010 IE8 iPhone 3GS Firefox 3 Android 2.1 Chrome 1 iPad Nav Timing
  9. 9. © AKAMAI - EDGE 2016 2011-2016: WTF just happened ? How many browsers in a typical Thanksgiving weekend?
  10. 10. © AKAMAI - EDGE 2016 2011-2016: WTF just happened 572 How many browsers in a typical Thanksgiving weekend?
  11. 11. © AKAMAI - EDGE 2016 Meanwhile...
  12. 12. © AKAMAI - EDGE 2016
  13. 13. © AKAMAI - EDGE 2016 So what can I measure?
  14. 14. © AKAMAI - EDGE 2016 What is available today - Synthetic Tooling Pros CPU metrics Test environments & other places without real users Access to everything including headers Cons Gaps in: Geos, browsers, pages Impossible to precisely match user demo- & technographics No way to realistically measure browser cache impact
  15. 15. © AKAMAI - EDGE 2016 What is available today - RUM Pros Near perfect coverage of browsers, geos, pages Ability to understand impact to performance Realistic measurement of browser cache impact Cons Gaps in: Geos, browsers, pages Impossible to precisely match user demo- & technographics No access to HTTP header data
  16. 16. © AKAMAI - EDGE 2016 Engagement Revenue Coverage Caching CPU Pre-prod Availability Page details Resources Headers SyntheticReal
  17. 17. © AKAMAI - EDGE 2016
  18. 18. © AKAMAI - EDGE 2016 https://www.w3.org/wiki/Web_Performance/Publications W3C - Performance Working Group
  19. 19. © AKAMAI - EDGE 2016 Navigation Timing API
  20. 20. © AKAMAI - EDGE 2016 Resource Timing API
  21. 21. © AKAMAI - EDGE 2016 Coming soon to a browser near you! Measure the size of assets in addition to performance. Resource Timing (2.0) - Transfer Size
  22. 22. © AKAMAI - EDGE 2016 ● Measure performance using high-precision timestamps ● Consists of “marks” and “measures” ● PerformanceMark: Timestamp ● PerformanceMeasure: Duration between two given marks Browser Support - User Timing API
  23. 23. © AKAMAI - EDGE 2016 Browser Support - Navigation Timing API
  24. 24. © AKAMAI - EDGE 2016 Browser Support - Resource Timing API
  25. 25. © AKAMAI - EDGE 2016 Browser Support - User Timing API
  26. 26. © AKAMAI - EDGE 2016 So what should I measure?
  27. 27. © AKAMAI - EDGE 2016
  28. 28. © AKAMAI - EDGE 2016 What matters depends on you
  29. 29. © AKAMAI - EDGE 2016 What matters to an ops engineer? ● Preference for binary metrics ● Looks for indicators of system performance ● Affinity to network and back-end metrics MTTR
  30. 30. © AKAMAI - EDGE 2016 What matters to an ops engineer? Network Server
  31. 31. © AKAMAI - EDGE 2016 What matters to an ops engineer? Network Server function getPerf() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart, connect: timing.connectEnd - timing.connectStart }; } ttfb: timing.responseStart - timing.connectEnd};
  32. 32. © AKAMAI - EDGE 2016 Help answers questions like... ● Do I have an issue with an upstream DNS resolver? ● Am I having Tier1 connectivity issues? ● How fast is my application server responding?
  33. 33. © AKAMAI - EDGE 2016 What matters to an ops engineer? Don’t be misled by “super fast” responses ● DNS Caching ● Prefetching ● Connection reuse ● Pages served from browser cache
  34. 34. © AKAMAI - EDGE 2016 What matters to an ops engineer? Object Level Timing ● Resource timing API ● Binary metric for testing infrastructure ● Measure performance of a third-party ● CDN healthcheck
  35. 35. © AKAMAI - EDGE 2016 What matters to an ops engineer? Object Timing var rUrl = ‘https://edge.akamai.com/ec/us/multimed ia/images/edge/akamai-edge-conference-s peaker-buddy-brewer.jpg’; var me = performance.getEntriesByName(rUrl)[0]; var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart }
  36. 36. © AKAMAI - EDGE 2016 What matters to a front-end developer? Optimized Load Times ● “Do no harm” ● Looks for indicators of application performance ● Looks to measure a “fully loaded” page ● Affinity for front-end metrics
  37. 37. © AKAMAI - EDGE 2016 What matters to a front-end developer? Client
  38. 38. © AKAMAI - EDGE 2016 What matters to a front-end developer?
  39. 39. © AKAMAI - EDGE 2016 What matters to a front-end developer? function getPerf() { var timing = window.performance.timing; return { Basepage: timing.responseEnd - timing.responseStart, DLoading: timing.domLoading – timing.navigationStart, DInt: timing.domIneractive – timing.navigationStart, DContLoaded: timing.domContentLoadedEventEnd – timing.navigationStart, DContLoadTime: timing.domContentLoadedEventEnd – timing.domContentLoadedEventStart, DComplete: timing.domComplete - timing.navigationStart, PLoad: timing.loadEventStart - timing.navigationStart }; } Client
  40. 40. © AKAMAI - EDGE 2016 Be wary of onload... ~1s OnLoad ~3.5s visibly complete
  41. 41. © AKAMAI - EDGE 2016 What matters to a designer? UX ● Performance by design (D4P) ● Looks for indicators of human perception ● How quickly the page renders ● Affinity for custom metrics and render metrics
  42. 42. © AKAMAI - EDGE 2016 RUM first paint and start render Chrome - “First Paint” Visual Start Render
  43. 43. © AKAMAI - EDGE 2016 User perception Courtesy of the filament group: https://www.filamentgroup.com/lab/weight-wait.html
  44. 44. © AKAMAI - EDGE 2016 Custom Metrics
  45. 45. © AKAMAI - EDGE 2016 Custom Metrics - User Timing API performance.mark(“startTask”); //Some stuff you want to measure happens here performance.mark(“stopTask”); //Measure the duration between the two marks performance.measure(“taskDuration”,“startTask”,“stopTask”); //Measure the load of an image <img src=”image1.gif” onload=”performance.mark(‘image1’)”>
  46. 46. © AKAMAI - EDGE 2016 What matters to a business executive? $$$$$$$ ● Business impact analysis ● Simplified/lean metrics ● Actionable data
  47. 47. © AKAMAI - EDGE 2016 What matters to a business executive?
  48. 48. © AKAMAI - EDGE 2016 What matters to a business executive?
  49. 49. © AKAMAI - EDGE 2016 What matters to a business executive?
  50. 50. © AKAMAI - EDGE 2016 Don’t create unintended silos Biz Ops Dev Design
  51. 51. © AKAMAI - EDGE 2016 Don’t create unintended silos The success of my business is a reflection of... The digital experience I provide, which depends on... how quickly I can render content in the browser, which is impacted by... the performance and stability of my infrastructure. 1 2 3 4 Biz Design Ops Dev
  52. 52. © AKAMAI - EDGE 2016 “I just want to buy my Mom’s birthday present.” “I just want to know if it will rain today.” “I just want to share pictures of my vacation”
  53. 53. © AKAMAI - EDGE 2016 How do I collect it?
  54. 54. © AKAMAI - EDGE 2016
  55. 55. © AKAMAI - EDGE 2016 Real users are not normal What do I do with it?
  56. 56. © AKAMAI - EDGE 2016 What do I do with it?
  57. 57. © AKAMAI - EDGE 2016 2.9s - p50 (median) 4.4s - Average 10.4s - p95 What do I do with it?
  58. 58. © AKAMAI - EDGE 2016 What should I throw out?
  59. 59. © AKAMAI - EDGE 2016 Load Time = 3500 years
  60. 60. © AKAMAI - EDGE 2016 Load Time = -(10years)
  61. 61. © AKAMAI - EDGE 2016 What should I keep?
  62. 62. © AKAMAI - EDGE 2016
  63. 63. © AKAMAI - EDGE 2016
  64. 64. © AKAMAI - EDGE 2016 Other opportunities for bots... 1. How does performance impact bot behavior? 2. Are bots skewing the numbers I’m reporting? 3. Am I providing an optimal experience for search bots? 4. Does any of this impact SEO?
  65. 65. © AKAMAI - EDGE 2016 Summary ● We have amazing tools at our disposal - your users are wired for sound! ● Measuring what matters is situation independent ● Important to look at all metrics from the same context ● Use robust statistics when looking at your data

×