@jeroentjepkemaPerformance   ::   Analytics   ::   Optimization
a Web Performance Dashboard,         up & running in 90 minutes   Key concepts for measuring and presenting performance da...
Dashboard design   principles
Dashboard design   principles                       Collecting                   Performance data
Dashboard design   principles                       Collecting                   Performance data                         ...
Dashboard, FTW!
500       0                 1000                           1500                                         2000              ...
500       0                 1000                           1500                                            2000           ...
# Payment orders   Login processPayment process    Savings
January 25 2007, 9:05am              # Payment orders   Login process             Payment process      Savings
January 25 2007, 9:05am              # Payment orders   Login process             Payment process      Savings
http://www.nrc.nl/apps/bigboard/
Part 1: Five Design principles for a Web Performance Dashboard
#1 - Data for all, not for everything
Your goal is to be understood...
How technical is your audience?Your goal is to be understood...
How technical is your audience?Your goal is to be understood...        How will they use it?
How technical is your audience?Your goal is to be understood...        How will they use it?                              ...
3 levels of performance data...
Trending
Trending  Future Big pictureActionable CapacityOptimization
Service Levels   Trending                   Future                  Big picture                 Actionable                ...
Service Levels       Trending     Yesterday          FutureHow did we perform?    Big picture  Versus targets?     Actiona...
Real Time    Service Levels       Trending                 Yesterday          Future            How did we perform?    Big...
Real Time                           Service Levels               Trending         Now                                Yeste...
Real Time                           Service Levels               Trending          Now                               Yeste...
#2 - Focus on End User Experience
Why? Customer loyalty!
Just a1-second  delay
Just a1-second         7%           loss in conversion  delay                                @jeroentjepkema, MeasureWorks
Just a                 7%              11%1-second   loss in conversion                                  fewer  delay     ...
Just a                 7%              11%1-second                          fewer           16% decrease           loss in...
Study conducted by MeasureWorks & idr1, Emerce, 2012    Dutch eRetail100   Dutch eTravel30   Dutch eFinance50             ...
Speed        @jeroentjepkema, MeasureWorks
Globe     Neckermann     SunwebD-reizen                    Arke                    Shopping for a holiday...              ...
www.vrijuit.nl                                        Page size: 1201Kb      Experience fast? Try... http://bit.ly/SHYZpRh...
Reliability              @jeroentjepkema, MeasureWorks
Functional errors                    @jeroentjepkema, MeasureWorks
Technical errors@jeroentjepkema, MeasureWorks
@jeroentjepkema, MeasureWorks
And there’s marketing...
Goodcampaign
Good           gonecampaign
Good           gone   BAD!campaign
#3 - Provide context
What does it mean?
What does it mean?Why is it important?
What does it mean?Why is it important? Next best action?
#4 - Functionality first
What do you sell to the end users?
Creates common language between         business and IT...
Metric: Availability99%         versus           99min                ...with understandable targets
#5 - Make it visually attractive
Group items
Use colours to display status
Make it move...
Design with the end user in mind
And the dashboard is...
Real TimeAnd the dashboard is...
Real Time                          Focused on End UserAnd the dashboard is...
Real Time                          Focused on End User                           Providing contextAnd the dashboard is...
Real Time                          Focused on End User                           Providing contextAnd the dashboard is... ...
Real Time                          Focused on End User                           Providing contextAnd the dashboard is... ...
All that matters... Show them where it hurts
Part 2: Collecting Performance Data
Technology:Responsiveness: How End Users    End User Experience               How we build,                               ...
Complexity of a typical web transactionSource: Compuware Gomez                         @jeroentjepkema, MeasureWorks
(CLOUD) DATA CENTER                       INTERNAL USERS               INTERNET                    CUSTOMERS              ...
Measuring End User Experience is noteasy...                                @jeroentjepkema, MeasureWorks
Measuring End User Experience is noteasy...        You need diagnostic details for things you can                         ...
Measuring End User Experience is noteasy...        You need diagnostic details for things you can                         ...
Measure End User Experience? Outside-in,from the browser perspective...                               @jeroentjepkema, Mea...
Performance Measurement Toolkit
Lot’s of different user scenario’s                @jeroentjepkema, MeasureWorks
@jeroentjepkema, MeasureWorks
Changing bandwith: DSL, Mobile, WiFi, etc.
Real Devices
Real usage...
Consistency in measurements            @jeroentjepkema, MeasureWorks
Correlating data for root cause analysis                     @jeroentjepkema, MeasureWorks
Performance Data collection   @jeroentjepkema, MeasureWorks
Synthethic      Real User Benchmark          MonitoringPerformance Data collection   @jeroentjepkema, MeasureWorks
Simulate business transactions
Via multiple devices & browsers
From where multiple locations...
Used for error detection & Root Cause Analysis
Used for error detection & Root Cause Analysis
Used for error detection & Root Cause Analysis
Used for error detection & Root Cause Analysis
Synthetic monitoring  Metrics collected                             Cons  Easy collection of data                       No...
Synthetic monitoring   Vendors             When selecting:   Alertsite           Ease of use with recording scripts   Gome...
Synthetic monitoring   Vendors                       When selecting:   Alertsite                     Ease of use with reco...
Ultimately, synthetic monitoring shows youif your site’s working or not...
But, synthetic isn’t enough...
Synthetic heartbeat  Real UsersBut, synthetic isn’t enough...
Real User      Real User Benchmark          MonitoringPerformance Data collection   @jeroentjepkema, MeasureWorks
Navigation timing2 ways of measuring...                                                    Browser RUM             Disclai...
h"p://www.w3.org/TR/naviga2on32ming/5                    Some background info:Navigation timing                    http://...
http://caniuse.com/nav-timing
1                   2                        3                    4                                                   As p...
Gomez
Gomez
LognormalSoasta
Torbit
Relies on navigation timing API, custom variables can be addedGoogle Analytics
New Relic
Real User monitoring  Metrics collected                                 Cons  Measure ALL pages from ALL users            ...
Real User monitoring   Vendors             When selecting:   Lognormal           Do I need to build my own datastorage?   ...
Real User monitoring   Vendors                          When selecting:   Lognormal                        Do I need to bu...
Ultimately, Real User Monitoring shows you how manyusers are affected by bad performance...
Real User       benchmarkingPerformance Data collection   @jeroentjepkema, MeasureWorks
First view 11.349s                    Real usage             Repeat view               4.357s
Average page load time per bandwith (seconds) for Dutch eTravel30       8,8                                               ...
Average page load time per Browser (seconds) Dutch eTravel30                                                              ...
Benchmark competitors                                                        arke.nl    5,50 sec                          ...
Webpagetest.org
Object level                             !!        Optimization tips   Webpagetest.org    !
Webpagetest.org
Mobitest / Akamai
Real User benchmarking  Metrics collected                                 Cons  Variety of real browsers and real devices ...
Ultimately, Real User benchmarking gives you periodicinsight in real usage scenario’s...
Synthethic monitoring   Real User Monitoring   Real User benchmarkingPerformance Data collection                          ...
Synthethic monitoring           Real User Monitoring   Real User benchmarking             Used for...  Heartbeat, runs wit...
Synthethic monitoring                Real User Monitoring     Real User benchmarking             Used for...              ...
Synthethic monitoring                Real User Monitoring        Real User benchmarking             Used for...           ...
Synthetic   Browser RUM   Competition
Synthetic vs. RUM Synthetic vs. RUM                     Synthetic   Browser RUM   Competition
Synthetic vs. RUM Synthetic vs. RUM                                  You vs. competition                     Synthetic   B...
Raw data
Part 3: Building the dashboard
Start with your web analytics tool...
Every website has goals                          http://www.flickr.com/photos/itsgreg/446061432/
Organic Search    Campaigns     Ad Network                                                       Transactional site       ...
Select your top customer journeys
Map the customer journeys with your ownweb application delivery chain...
Journey 1                    Journey 2                      Journey 3Synthetic transactions              Tier1      Webser...
Journey 1                    Journey 2                      Journey 3Synthetic transactions              Tier1      Webser...
Journey 1                    Journey 2                      Journey 3Synthetic transactions              Tier1      Webser...
Journey 1                    Journey 2                      Journey 3Synthetic transactions              Tier1      Webser...
Don’t forget third parties
Defining service levels & thresholds...
First thing is to establishing a baseline:
First thing is to establishing a baseline:                          A pre-defined set of metrics
First thing is to establishing a baseline:                           A pre-defined set of metrics                          ...
First thing is to establishing a baseline:                           A pre-defined set of metrics                          ...
First thing is to establishing a baseline:                             A pre-defined set of metrics                        ...
Example...
Purchasing a book,                      Customer journeymust be completed (speed),               Metric: Speedwhere every ...
Repeat this for every customer journey defined...
Next, group the content...
KISS
Layered approachOnline brand   Bol.com
Layered approachOnline brand    Bol.com                Books                Music     Products    Toys                 etc.
Layered approachOnline brand              Bol.com                           Books                           Music     Prod...
Layered approachOnline brand              Bol.com     Overall performance                           Books                 ...
Layered approachOnline brand              Bol.com        Overall performance                           Books              ...
Layered approachOnline brand              Bol.com        Overall performance                           Books              ...
Layered approach    Big PictureOnline brand              Bol.com          Overall performance                           Bo...
Mixing it all together...
Displayed in dashboard   Metrics   Group information
Displayed in dashboard   Metrics   Group information      Functionality
Displayed in dashboard   Metrics   Group information      Functionality                   Service Levels
Displayed in dashboard            Metrics               Group information                                 Threshholds     ...
Displayed in dashboard            Metrics               Group information                                 Threshholds     ...
Displayed in dashboard            Metrics               Group information                                 Threshholds     ...
Displayed in dashboard            Metrics               Group information                                 Threshholds     ...
Displayed in dashboard             Metrics               Group information                                  Threshholds   ...
Displayed in dashboard             Metrics               Group information                                  Threshholds   ...
Displayed in dashboard             Metrics               Group information                                  Threshholds   ...
Designing the infrastructure...
Custom reports                             Mobile               Performance                     (SLA/Trending)            ...
Synthethic   RUM
Synthethic   RUM
Synthethic   RUM
Data model RUM
It all looks the same...
...but still so different
...but still so different                            Synthethic monitoring
...but still so different                            http://bit.ly/MW-namingconvention
Authentication layerhttp://github.com/symfony/symfonyhttp://www.symfony-project.org
Visualshttps://www.html5rocks.com/en/tutorials/canvas/performance                               https://www.highcharts.com...
Demo       Demo time
Designing the interface...
Technical metricsNavigation
Technical metricsNavigation                        Business metrics
Technical metrics             This is the part that movesNavigation                            Business metrics
Demo:URL: https://app.measureworks.nlUID: demo@measureworks.nlPassword: performance
Metric: Page speedDatacollection: RUMTreshold: Historically based or fixedTimeframe: Average 5min
Metric: UptimeDatacollection: Synthethic monitoringTreshold: Page errors (4xx, 5xx)Timeframe: Average 5min
Metric: Third Party uptimeDatacollection: Synthethic monitoringTreshold: Error objects/domain (4xx, 5xx)Timeframe: Average...
Metric: Application errorDatacollection: RUM & Synthethic monitoringTreshold: # object errors per funnel (4xx, 5xx)Timefra...
Metric: Content errorDatacollection: Synthethic monitoringTreshold: Object size, Context match, User transaction failureTi...
Metric: Pageviews impactedDatacollection: RUMTreshold: % of pageviews affected vs. total pageviewsTimeframe: Average 5min
Metric: Users affectedDatacollection: RUM & Navigation timingTreshold: % of browser session vs. bouncerateTimeframe: Avera...
Metric: Competition rankingDatacollection: WebpagetestTreshold: % competitors faster based oncurrent average page speedTim...
URL: https://app.measureworks.nl UID: velocity@measureworks.nl         PW: performance
Become a dashboard rockstar...
1. Look into your webanalyticsBecome a dashboard rockstar...
1. Look into your webanalytics                                 2. Define a performance baselineBecome a dashboard rockstar...
1. Look into your webanalytics                                 2. Define a performance baseline                            ...
1. Look into your webanalytics                                 2. Define a performance baseline                            ...
1. Look into your webanalytics                                   2. Define a performance baseline                          ...
1. Look into your webanalytics                                   2. Define a performance baseline                          ...
1. Look into your webanalytics                                   2. Define a performance baseline                          ...
One more things...Couple
 of
Start with a functional design
OrganisatieCreate a good team                     @jeroentjepkema, MeasureWorks
Design with your end users in mind
Collect feedback as early as possible
The art of deleting
Be careful with using API’s
Combining datasources... WTF?
Recommended...
Data Visualization:      Noah Iliinsky/Julie Steel: Desiging Data Visualizations - http://oreil.ly/SryQyV                 ...
Questions?
Thanks! More questions?M: jtjepkema@measureworks.nlT: @jeroentjepkemaW: www.measureworks.nl
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final
Upcoming SlideShare
Loading in …5
×

MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final

5,329 views

Published on

For the Velocity Conference Europe 2012 workshop day this presentation is about the essentials for creation and building a Web Performance dashboard. This with ultimate goal of providing the audience a framework for designing and building a web performance dashboard. The session will cover the following 3 items:
Design guidelines: What defines a web performance dashboard? How to make sure it’s actionable and for people to actually use it on day to day basis?
Data collection: Why performance data? The various ways there are to collect data (e.g. synthetic versus RUM data, Webpagetest, Mobile) and how to correlate the different types of data and tools
Building the dashboard: How to build the actual dashboard, providing an overview of the tools/techniques used

At the end of the workshop you will be able to design and build your own dashboard based on the framework provided, or to optimize the current dashboards within your organization.

Published in: Technology

MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard final

  1. 1. @jeroentjepkemaPerformance :: Analytics :: Optimization
  2. 2. a Web Performance Dashboard, up & running in 90 minutes Key concepts for measuring and presenting performance data Velocity Conference Europe October 2, 2012
  3. 3. Dashboard design principles
  4. 4. Dashboard design principles Collecting Performance data
  5. 5. Dashboard design principles Collecting Performance data Building the Web Performance dashboard
  6. 6. Dashboard, FTW!
  7. 7. 500 0 1000 1500 2000 2500 300000:00:0001:00:0002:00:0003:00:0004:00:0005:00:0006:00:0007:00:0008:00:0009:00:0010:00:0011:00:0012:00:0013:00:0014:00:0015:00:0016:00:0017:00:0018:00:0019:00:0020:00:0021:00:0022:00:0023:00:00 Visits Users
  8. 8. 500 0 1000 1500 2000 2500 300000:00:0001:00:0002:00:0003:00:0004:00:0005:00:0006:00:0007:00:0008:00:0009:00:0010:00:00 Alert status11:00:0012:00:0013:00:0014:00:0015:00:0016:00:0017:00:0018:00:0019:00:0020:00:00 Be Careful21:00:0022:00:0023:00:00 Visits Users
  9. 9. # Payment orders Login processPayment process Savings
  10. 10. January 25 2007, 9:05am # Payment orders Login process Payment process Savings
  11. 11. January 25 2007, 9:05am # Payment orders Login process Payment process Savings
  12. 12. http://www.nrc.nl/apps/bigboard/
  13. 13. Part 1: Five Design principles for a Web Performance Dashboard
  14. 14. #1 - Data for all, not for everything
  15. 15. Your goal is to be understood...
  16. 16. How technical is your audience?Your goal is to be understood...
  17. 17. How technical is your audience?Your goal is to be understood... How will they use it?
  18. 18. How technical is your audience?Your goal is to be understood... How will they use it? How fast can they execute?
  19. 19. 3 levels of performance data...
  20. 20. Trending
  21. 21. Trending Future Big pictureActionable CapacityOptimization
  22. 22. Service Levels Trending Future Big picture Actionable Capacity Optimization
  23. 23. Service Levels Trending Yesterday FutureHow did we perform? Big picture Versus targets? Actionable Historical context Capacity Optimization
  24. 24. Real Time Service Levels Trending Yesterday Future How did we perform? Big picture Versus targets? Actionable Historical context Capacity Optimization
  25. 25. Real Time Service Levels Trending Now Yesterday Future Everything working? How did we perform? Big picture Is it me or the internet? Versus targets? Actionable How to solve it? Historical context Capacity Simple Optimization0hours 24hours 7days 1Quarter
  26. 26. Real Time Service Levels Trending Now Yesterday Future Everything working? How did we perform? Big picture Is it me or the internet? Versus targets? Actionable How to solve it? Historical context Capacity Simple Data for dashboard Optimization0hours 24hours 7days 1Quarter
  27. 27. #2 - Focus on End User Experience
  28. 28. Why? Customer loyalty!
  29. 29. Just a1-second delay
  30. 30. Just a1-second 7% loss in conversion delay @jeroentjepkema, MeasureWorks
  31. 31. Just a 7% 11%1-second loss in conversion fewer delay page views @jeroentjepkema, MeasureWorks
  32. 32. Just a 7% 11%1-second fewer 16% decrease loss in conversion in customer satisfaction delay page views @jeroentjepkema, MeasureWorks
  33. 33. Study conducted by MeasureWorks & idr1, Emerce, 2012 Dutch eRetail100 Dutch eTravel30 Dutch eFinance50 4% 6% 12%Social Media mentions = Performance KPI
  34. 34. Speed @jeroentjepkema, MeasureWorks
  35. 35. Globe Neckermann SunwebD-reizen Arke Shopping for a holiday... @jeroentjepkema, MeasureWorks
  36. 36. www.vrijuit.nl Page size: 1201Kb Experience fast? Try... http://bit.ly/SHYZpRhttp://blog.dynatrace.com/2012/09/26/why-page-size-matters-even-more-for-mobile-web-apps/ @jeroentjepkema, MeasureWorks
  37. 37. Reliability @jeroentjepkema, MeasureWorks
  38. 38. Functional errors @jeroentjepkema, MeasureWorks
  39. 39. Technical errors@jeroentjepkema, MeasureWorks
  40. 40. @jeroentjepkema, MeasureWorks
  41. 41. And there’s marketing...
  42. 42. Goodcampaign
  43. 43. Good gonecampaign
  44. 44. Good gone BAD!campaign
  45. 45. #3 - Provide context
  46. 46. What does it mean?
  47. 47. What does it mean?Why is it important?
  48. 48. What does it mean?Why is it important? Next best action?
  49. 49. #4 - Functionality first
  50. 50. What do you sell to the end users?
  51. 51. Creates common language between business and IT...
  52. 52. Metric: Availability99% versus 99min ...with understandable targets
  53. 53. #5 - Make it visually attractive
  54. 54. Group items
  55. 55. Use colours to display status
  56. 56. Make it move...
  57. 57. Design with the end user in mind
  58. 58. And the dashboard is...
  59. 59. Real TimeAnd the dashboard is...
  60. 60. Real Time Focused on End UserAnd the dashboard is...
  61. 61. Real Time Focused on End User Providing contextAnd the dashboard is...
  62. 62. Real Time Focused on End User Providing contextAnd the dashboard is... Functionality first
  63. 63. Real Time Focused on End User Providing contextAnd the dashboard is... Functionality first Visually attractive
  64. 64. All that matters... Show them where it hurts
  65. 65. Part 2: Collecting Performance Data
  66. 66. Technology:Responsiveness: How End Users End User Experience How we build, measure & optimize perceive (front-end) performance performance @jeroentjepkema, MeasureWorks
  67. 67. Complexity of a typical web transactionSource: Compuware Gomez @jeroentjepkema, MeasureWorks
  68. 68. (CLOUD) DATA CENTER INTERNAL USERS INTERNET CUSTOMERS Third-party/ Cloud Services Storage DB Servers Web Servers Major Local This is what you control... Network What you’re blamed for.. ISP Content ISP Delivery Mobile Load Networks Carriers Middleware App BalancersMainframe Servers Servers Web Performance Delivery Chain @jeroentjepkema, MeasureWorks
  69. 69. Measuring End User Experience is noteasy... @jeroentjepkema, MeasureWorks
  70. 70. Measuring End User Experience is noteasy... You need diagnostic details for things you can control and/or change @jeroentjepkema, MeasureWorks
  71. 71. Measuring End User Experience is noteasy... You need diagnostic details for things you can control and/or change You need insights in the things you can control, but do impact your bottom line @jeroentjepkema, MeasureWorks
  72. 72. Measure End User Experience? Outside-in,from the browser perspective... @jeroentjepkema, MeasureWorks
  73. 73. Performance Measurement Toolkit
  74. 74. Lot’s of different user scenario’s @jeroentjepkema, MeasureWorks
  75. 75. @jeroentjepkema, MeasureWorks
  76. 76. Changing bandwith: DSL, Mobile, WiFi, etc.
  77. 77. Real Devices
  78. 78. Real usage...
  79. 79. Consistency in measurements @jeroentjepkema, MeasureWorks
  80. 80. Correlating data for root cause analysis @jeroentjepkema, MeasureWorks
  81. 81. Performance Data collection @jeroentjepkema, MeasureWorks
  82. 82. Synthethic Real User Benchmark MonitoringPerformance Data collection @jeroentjepkema, MeasureWorks
  83. 83. Simulate business transactions
  84. 84. Via multiple devices & browsers
  85. 85. From where multiple locations...
  86. 86. Used for error detection & Root Cause Analysis
  87. 87. Used for error detection & Root Cause Analysis
  88. 88. Used for error detection & Root Cause Analysis
  89. 89. Used for error detection & Root Cause Analysis
  90. 90. Synthetic monitoring Metrics collected Cons Easy collection of data No real user data, unlimited bandwith while testing Heartbeat collection of data over time Combination with CDN Testing without actual visitor traffic Detects macro outages, not user Test from multiple locations events Object level details Detects only what you script Detailed alerting via Text, Mail or SNMP No measurement of traffic volume Root cause analysis data, with error codes, Places load on the site under test screenshots, source code and waterfall data Maintenance of tests takes time
  91. 91. Synthetic monitoring Vendors When selecting: Alertsite Ease of use with recording scripts Gomez Check the API functionalities Keynote Level of detail with alert messages Catchpoint Global versus local coverage Siteconfidence IP Label Reliability of sending alert messages Websitepulse Pingdom Etc...
  92. 92. Synthetic monitoring Vendors When selecting: Alertsite Ease of use with recording scripts Gomez Check the API functionalities Keynote Level of detail with alert messages Catchpoint Global versus local coverage Siteconfidence IP Label Reliability of sending alert messages Websitepulse For Native Apps: Pingdom Gomez Keynote Etc... ...
  93. 93. Ultimately, synthetic monitoring shows youif your site’s working or not...
  94. 94. But, synthetic isn’t enough...
  95. 95. Synthetic heartbeat Real UsersBut, synthetic isn’t enough...
  96. 96. Real User Real User Benchmark MonitoringPerformance Data collection @jeroentjepkema, MeasureWorks
  97. 97. Navigation timing2 ways of measuring... Browser RUM Disclaimer: There’s also third category Datacenter RUM, this will not be covered in this out of the presentation. Contact me if your want details
  98. 98. h"p://www.w3.org/TR/naviga2on32ming/5 Some background info:Navigation timing http://66.7percentangel.com/2011/12/breaking-down-onload-event-performance-bookmarklet/ http://www.html5rocks.com/en/tutorials/webperformance/basics/ http://www.w3.org/TR/2011/CR-navigation-timing-20110315/#nt-dom-content-event-start
  99. 99. http://caniuse.com/nav-timing
  100. 100. 1 2 3 4 As pages execute, After onload tag sendInsert tag (.js file) into Pages are requested tag collects detailed report for (mobile) web pages from browser/device performance metrics further analysis tag.js tag.js tag.js tag.js Browser RUM
  101. 101. Gomez
  102. 102. Gomez
  103. 103. LognormalSoasta
  104. 104. Torbit
  105. 105. Relies on navigation timing API, custom variables can be addedGoogle Analytics
  106. 106. New Relic
  107. 107. Real User monitoring Metrics collected Cons Measure ALL pages from ALL users No traffic is no data Measures traffic as well as performance Needs change to application code Quantative performance data to analyse user May require physical installation of satisfaction data storage or data reporter Data can be directly correlated with Web Sample rate Analytics Great for trending and creating the big picture
  108. 108. Real User monitoring Vendors When selecting: Lognormal Do I need to build my own datastorage? Gomez Check the API functionalities Keynote How long is raw data stored Torbit Interface Google Analytics Mobile support Boomerang.js Oracle Etc...
  109. 109. Real User monitoring Vendors When selecting: Lognormal Do I need to build my own datastorage? Gomez Check the API functionalities Keynote How long is raw data stored Torbit Interface Google Analytics Mobile support Boomerang.js Oracle For Native Apps: Google Analytics Etc... Gomez Localytics
  110. 110. Ultimately, Real User Monitoring shows you how manyusers are affected by bad performance...
  111. 111. Real User benchmarkingPerformance Data collection @jeroentjepkema, MeasureWorks
  112. 112. First view 11.349s Real usage Repeat view 4.357s
  113. 113. Average page load time per bandwith (seconds) for Dutch eTravel30 8,8 1,5 mbps 10 mbps 20 mbps 3,9 3,4 56% 13% Bandwith
  114. 114. Average page load time per Browser (seconds) Dutch eTravel30 IE7 6,5 IE8 3,9 IE9 3,6 40% 8% Devices used
  115. 115. Benchmark competitors arke.nl 5,50 sec sunweb.nl 3,91 sec vakantiexperts.nl 2,15 sec dreizen.nl 6,13 sec thomascook.nl 1,79 secSource: Webpagetest.org, IE9 10Mb up/ 2 Mb down
  116. 116. Webpagetest.org
  117. 117. Object level !! Optimization tips Webpagetest.org !
  118. 118. Webpagetest.org
  119. 119. Mobitest / Akamai
  120. 120. Real User benchmarking Metrics collected Cons Variety of real browsers and real devices Can be difficult to setup available for testing Requires physical installation Repetitive collection of real usage scenario’s Scripting (is difficult) Collect optimization metrics, waterfall and page speed score Great for trending and creating the big picture
  121. 121. Ultimately, Real User benchmarking gives you periodicinsight in real usage scenario’s...
  122. 122. Synthethic monitoring Real User Monitoring Real User benchmarkingPerformance Data collection @jeroentjepkema, MeasureWorks
  123. 123. Synthethic monitoring Real User Monitoring Real User benchmarking Used for... Heartbeat, runs without traffic Test specific customer journeys Object level detail Collect detailed alerts, including root cause analysis Desktop/Mobile Site Gomez Keynote Watchmouse Alertsite Mobile Apps Gomez KeynotePerformance Data collection @jeroentjepkema, MeasureWorks
  124. 124. Synthethic monitoring Real User Monitoring Real User benchmarking Used for... Used for... Heartbeat, runs without traffic Real usage information from Test specific customer journeys all users!! Object level detail Trending/Optimization Collect detailed alerts, including Business impact root cause analysis Desktop/Mobile Site Desktop/Mobile Site Gomez Gomez Keynote LogNormal Watchmouse Torbit Alertsite Google Analytics Mobile Apps Mobile Apps Gomez Gomez Keynote Localytics Google AnalyticsPerformance Data collection @jeroentjepkema, MeasureWorks
  125. 125. Synthethic monitoring Real User Monitoring Real User benchmarking Used for... Used for... Used for... Heartbeat, runs without traffic Real usage information from Periodic testing of user scenario’s Test specific customer journeys all users!! with real devices and bandwith Object level detail Trending/Optimization Optimization details Collect detailed alerts, including Business impact Competitive scan root cause analysis Desktop/Mobile Site Desktop/Mobile Site Desktop/Mobile Site Gomez Gomez Webpagetest Keynote LogNormal Watchmouse Torbit Alertsite Google Analytics Mobile Apps Mobile Apps Mobile Apps Gomez Gomez Perfecto Mobile Keynote Localytics Device Anywhere Google AnalyticsPerformance Data collection @jeroentjepkema, MeasureWorks
  126. 126. Synthetic Browser RUM Competition
  127. 127. Synthetic vs. RUM Synthetic vs. RUM Synthetic Browser RUM Competition
  128. 128. Synthetic vs. RUM Synthetic vs. RUM You vs. competition Synthetic Browser RUM Competition
  129. 129. Raw data
  130. 130. Part 3: Building the dashboard
  131. 131. Start with your web analytics tool...
  132. 132. Every website has goals http://www.flickr.com/photos/itsgreg/446061432/
  133. 133. Organic Search Campaigns Ad Network Transactional site Visitor Offer €" Abondenment) Upselling Reach Purchase step 1 €" Purchase step 2 €" Mailing, alerts, €" promotions Conversion €" Disengagement) Enrolment Impact)on)site) €" Negative €" PositiveGoals = Customer journeys
  134. 134. Select your top customer journeys
  135. 135. Map the customer journeys with your ownweb application delivery chain...
  136. 136. Journey 1 Journey 2 Journey 3Synthetic transactions Tier1 Webserver Webserver Webserver Webserver Application Application Tier2 server server Database Tier3 Backend Backend Backend Preferably one per backend used
  137. 137. Journey 1 Journey 2 Journey 3Synthetic transactions Tier1 Webserver Webserver Webserver Webserver Application Application Tier2 server server Database Tier3 Backend Backend Backend Preferably one per backend used
  138. 138. Journey 1 Journey 2 Journey 3Synthetic transactions Tier1 Webserver Webserver Webserver Webserver Application Application Tier2 server server Database Tier3 Backend Backend Backend Preferably one per backend used
  139. 139. Journey 1 Journey 2 Journey 3Synthetic transactions Tier1 Webserver Webserver Webserver Webserver For proper alerting, eliminate doubles Tier2 Application server Application server Database Tier3 Backend Backend Backend Preferably one per backend used
  140. 140. Don’t forget third parties
  141. 141. Defining service levels & thresholds...
  142. 142. First thing is to establishing a baseline:
  143. 143. First thing is to establishing a baseline: A pre-defined set of metrics
  144. 144. First thing is to establishing a baseline: A pre-defined set of metrics that describes normal behavior
  145. 145. First thing is to establishing a baseline: A pre-defined set of metrics that describes normal behavior in order to detect variancies
  146. 146. First thing is to establishing a baseline: A pre-defined set of metrics that describes normal behavior in order to detect variancies and to be comparable within historic context
  147. 147. Example...
  148. 148. Purchasing a book, Customer journeymust be completed (speed), Metric: Speedwhere every page loads under 4 sec., Target: Secusing IE8 and higher, User scenariofrom any location in the Netherlands, User locationsfor 95% of all users, Percentileevery day between 6am and 12pm, Windowmeasured with Real User Monitoring. Collection type Source: Metrics 101, Velocityconf 2010
  149. 149. Repeat this for every customer journey defined...
  150. 150. Next, group the content...
  151. 151. KISS
  152. 152. Layered approachOnline brand Bol.com
  153. 153. Layered approachOnline brand Bol.com Books Music Products Toys etc.
  154. 154. Layered approachOnline brand Bol.com Books Music Products Toys etc. Search Services per product Reviews Purchase Discounts
  155. 155. Layered approachOnline brand Bol.com Overall performance Books Music Products Toys etc. Search Services per product Reviews Purchase Discounts
  156. 156. Layered approachOnline brand Bol.com Overall performance Books Music Products Toys Detailed performance etc. Search Services per product Reviews Purchase Discounts
  157. 157. Layered approachOnline brand Bol.com Overall performance Books Music Products Toys Detailed performance etc. Search Services per product Reviews Root cause Purchase Discounts
  158. 158. Layered approach Big PictureOnline brand Bol.com Overall performance Books Music Products Toys Detailed performance etc. Search Services per product Reviews Root cause Purchase Discounts Detailed level of performance
  159. 159. Mixing it all together...
  160. 160. Displayed in dashboard Metrics Group information
  161. 161. Displayed in dashboard Metrics Group information Functionality
  162. 162. Displayed in dashboard Metrics Group information Functionality Service Levels
  163. 163. Displayed in dashboard Metrics Group information Threshholds Functionality Service Levels Real Users versus Synthethic
  164. 164. Displayed in dashboard Metrics Group information Threshholds Functionality Service Levels Real Users versus Synthethic Business context
  165. 165. Displayed in dashboard Metrics Group information Threshholds Functionality Service Levels Real Users versus Synthethic Business context Customer Journey
  166. 166. Displayed in dashboard Metrics Group information Threshholds Functionality Service Levels Real Users versus Synthethic Page views Bouncerate Business context Customer Journey Affected users sessions Competition
  167. 167. Displayed in dashboard Metrics Group information Threshholds Functionality Service Levels Real Users versus Synthethic Page views Bouncerate Business context Customer Journey Affected users sessions Competition Real Time Performance
  168. 168. Displayed in dashboard Metrics Group information Threshholds Functionality Service Levels Real Users versus Synthethic Page views Bouncerate Business context Customer Journey Affected users sessions Competition Real Time Performance Delivery chain
  169. 169. Displayed in dashboard Metrics Group information Threshholds Functionality Real Users versus Synthethic Service Levels Page views Bouncerate Business context Customer Journey Affected users sessions Competition Speed Application errors Real Time Performance Content errors Delivery chain Third party errors Downtime
  170. 170. Designing the infrastructure...
  171. 171. Custom reports Mobile Performance (SLA/Trending) application dashboard 4 4 5 Authentication layer Vendor API used for both real time1 information as raw data download API for data upload to mobile app / 22 dashboard 33 Import HTML alert mail Alert Datawarehouse 3 message Authentication mechanism for both4 login as selective data transfer 1 1 1 1 Remote access to Datawarehouse for SLA5 or Trend reporting, based on same data as real time dashboard WPT Synthethic RUM Analytics
  172. 172. Synthethic RUM
  173. 173. Synthethic RUM
  174. 174. Synthethic RUM
  175. 175. Data model RUM
  176. 176. It all looks the same...
  177. 177. ...but still so different
  178. 178. ...but still so different Synthethic monitoring
  179. 179. ...but still so different http://bit.ly/MW-namingconvention
  180. 180. Authentication layerhttp://github.com/symfony/symfonyhttp://www.symfony-project.org
  181. 181. Visualshttps://www.html5rocks.com/en/tutorials/canvas/performance https://www.highcharts.com http://zqi.me/vizd3
  182. 182. Demo Demo time
  183. 183. Designing the interface...
  184. 184. Technical metricsNavigation
  185. 185. Technical metricsNavigation Business metrics
  186. 186. Technical metrics This is the part that movesNavigation Business metrics
  187. 187. Demo:URL: https://app.measureworks.nlUID: demo@measureworks.nlPassword: performance
  188. 188. Metric: Page speedDatacollection: RUMTreshold: Historically based or fixedTimeframe: Average 5min
  189. 189. Metric: UptimeDatacollection: Synthethic monitoringTreshold: Page errors (4xx, 5xx)Timeframe: Average 5min
  190. 190. Metric: Third Party uptimeDatacollection: Synthethic monitoringTreshold: Error objects/domain (4xx, 5xx)Timeframe: Average 5min
  191. 191. Metric: Application errorDatacollection: RUM & Synthethic monitoringTreshold: # object errors per funnel (4xx, 5xx)Timeframe: Average 5min
  192. 192. Metric: Content errorDatacollection: Synthethic monitoringTreshold: Object size, Context match, User transaction failureTimeframe: Average 5min
  193. 193. Metric: Pageviews impactedDatacollection: RUMTreshold: % of pageviews affected vs. total pageviewsTimeframe: Average 5min
  194. 194. Metric: Users affectedDatacollection: RUM & Navigation timingTreshold: % of browser session vs. bouncerateTimeframe: Average 5min
  195. 195. Metric: Competition rankingDatacollection: WebpagetestTreshold: % competitors faster based oncurrent average page speedTimeframe: Average 60min
  196. 196. URL: https://app.measureworks.nl UID: velocity@measureworks.nl PW: performance
  197. 197. Become a dashboard rockstar...
  198. 198. 1. Look into your webanalyticsBecome a dashboard rockstar...
  199. 199. 1. Look into your webanalytics 2. Define a performance baselineBecome a dashboard rockstar...
  200. 200. 1. Look into your webanalytics 2. Define a performance baseline 3. Start with synthethic monitoringBecome a dashboard rockstar...
  201. 201. 1. Look into your webanalytics 2. Define a performance baseline 3. Start with synthethic monitoring 4. Design your dashboardBecome a dashboard rockstar...
  202. 202. 1. Look into your webanalytics 2. Define a performance baseline 3. Start with synthethic monitoring 4. Design your dashboard 5. Build a report based on your design and collect feedbackBecome a dashboard rockstar...
  203. 203. 1. Look into your webanalytics 2. Define a performance baseline 3. Start with synthethic monitoring 4. Design your dashboard 5. Build a report based on your design and collect feedback 6. Build a dashboardBecome a dashboard rockstar...
  204. 204. 1. Look into your webanalytics 2. Define a performance baseline 3. Start with synthethic monitoring 4. Design your dashboard 5. Build a report based on your design and collect feedback 6. Build a dashboard 7. Add rum and other datasourcesBecome a dashboard rockstar...
  205. 205. One more things...Couple
  206. 206.  of
  207. 207. Start with a functional design
  208. 208. OrganisatieCreate a good team @jeroentjepkema, MeasureWorks
  209. 209. Design with your end users in mind
  210. 210. Collect feedback as early as possible
  211. 211. The art of deleting
  212. 212. Be careful with using API’s
  213. 213. Combining datasources... WTF?
  214. 214. Recommended...
  215. 215. Data Visualization: Noah Iliinsky/Julie Steel: Desiging Data Visualizations - http://oreil.ly/SryQyV Real User Monitoring:Wednesday Oct 3, 11.20am - The 3.5s dash for attention and other stuff we found in RUM WebPage test: Thursday Oct 4, 15.30pm - WebPage test, beyond the basics Join a Web Performance meetup: Go to www.meetup.com and search for web performance. For Netherlands go to http://www.meetup.com/Dutch-Web-Operations-Meetup/
  216. 216. Questions?
  217. 217. Thanks! More questions?M: jtjepkema@measureworks.nlT: @jeroentjepkemaW: www.measureworks.nl

×