Rolling out      Real User Monitoring      at Telefónica Germany      Frank Thelen, Telefónica      Oliver Ochs, Holistico...
Who we are             2
Telefónica             3
Holisticon AGManagement and IT Consulting                                   Holisticon AG is a management                 ...
Agenda     Our Website.                  SLAs - Service Level01                            04   Agreements.     RUM – Real...
01Chapter titleOur Website.Arial 41 pt
7
Our WebsiteTimeline2008       2011   2012   Today                   8
Our Website „Primordial Soup“http://www.alice-dsl.deExisting Alice-PortalStep-by-step approachto a new Web platformManual ...
Our Website„Do the Move“            http://dsl.o2online.de            Change from Telecom Italia            to Telefónica ...
Our WebsitePublic WebPagetest                     http://dsl.o2online.de                     Public WebPagetest instance  ...
Our WebsitePrivate WebPagetest + RUM                                   http://dsl.o2online.de                             ...
02RUM –Chapter titleArial 41User Monitoring.  Real pt
RUMConcept                             Synthetic    Real User                             Monitoring   Monitoring      Sce...
02   RUM     Navigation Timing API                             15
02   RUM     Implementation, simple measurement, logging via AJAX     <script>                 function () {              ...
02   RUM     Make or Buy, Sensoring simple, analysis not so simple                                       17
RUMHow many samples are needed?      o2 Portal – mobile products        DSL Portal – broadband products      2.1 M PageVie...
03KPIs –Chapter titleArial 41 pt  Key Performance Indicators.
03   KPIs     Timestamps and Intervals              Synthetic         Real User              Monitoring        Monitoring ...
03   KPIs     Perception of our website                                      Most important: fold                         ...
03   KPIs     Understand your Browser and your Website!                       Perceived Load Time                         ...
03   KPIs     Relative Frequency        Perceived Load Time / 24 hours / all pages                                        ...
03   KPIs     Trend                                          90% of page page page                                        ...
04SLAs –Chapter titleArial 41 pt Level Agreements.  Service
04   SLAs     Our Requirements – We are experimenting       Perceived Load Time / Trend / Ratio                           ...
04   SLAs     Be specific        Perceived Load Time / Segmented by Page Type                                           27
05Chapter titleWhat we learned.Arial 41 pt
05   What we learned     Whats going on over the weekend?        Perceived Load Time / Trend                              ...
05   What we learned     Whats going on over the weekend?        Bandwidth / Trend                                     30
05   What we learned     What about Mobile?       Perceived Load Time / Segmented by Network                              ...
05   What we learned     We have a Problem!       Time to First Byte / Trend                       Release      Release   ...
06Chapter titleWhat we changed.Arial 41 pt
06   What we changed                       • “Performance Team” – dedicated group      Organisational   • Process for perf...
35
36
Upcoming SlideShare
Loading in …5
×

Rolling out Real User Monitoring at Telefónica Germany

1,327 views

Published on

Telefónica Germany, best known for its consumer brand o2 (www.o2online.de), operate several Web portals for prospects, customers, service agents, and business partners. With 3 million PageViews a day, their Web portals are an integral part of nearly all major business processes.

Performance is a critical factor that directly influences conversion rates and revenue of a Web portal. Thus performance is a major requirement that adds specific effort to the entire Web development process. Telefónica Germany use a variety of tools to ensure and verify that performance requirements are met. In 2012 we rolled out “Real User Monitoring” based on the recent “Navigation Timing API” adding a new aspect to Web page monitoring that no other approach provides.

“Real User Monitoring” takes place right in your users’ browsers – measuring your users’ true experience when accessing your Web pages. This captures all factors, e.g., bandwidth, network latency, hardware, protocol, banners, tracking pixel, mashups, browser extensions – allowing new findings, e.g.,

How many users actually load a particular page in up to 2.5 seconds?
How many users does our portal actually reach (coverage), i.e., how many users do we abandon due to their limited bandwidth?
Does my portal actually meet service level agreements (SLAs)?
This session addresses the following questions:

Navigation Timing API – What does it do?
How to choose the right parameters
How to implement the sensor
How to send the data to the server
How much data is needed
Which pages should be measured?
Which key performance indicators (KPIs) make sense?
Which service level agreements (SLAs) make sense?
What did we learn from real users’ performance data?
This session reports on the introduction of “Real User Monitoring” at Telefónica Germany. It provides experience and learnings, practical tips, dos and don’ts. Attendees will walk away with a clear unterstanding of “Real User Monitoring” based on the “Navigation Timing API”, its benefits and limitations, and how it can be implemented.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,327
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Rolling out Real User Monitoring at Telefónica Germany

  1. 1. Rolling out Real User Monitoring at Telefónica Germany Frank Thelen, Telefónica Oliver Ochs, Holisticon AGLondon, October 3rd 2012
  2. 2. Who we are 2
  3. 3. Telefónica 3
  4. 4. Holisticon AGManagement and IT Consulting Holisticon AG is a management and IT consulting company from Hamburg/Germany. Based on a holistic consulting approach we are assisting our customers in development projects on all technical, tactical as well as strategic levels. 4
  5. 5. Agenda Our Website. SLAs - Service Level01 04 Agreements. RUM – Real User What we learned.02 Monitoring. 05 KPIs - Key Performance What we changed.03 Indicators. 06
  6. 6. 01Chapter titleOur Website.Arial 41 pt
  7. 7. 7
  8. 8. Our WebsiteTimeline2008 2011 2012 Today 8
  9. 9. Our Website „Primordial Soup“http://www.alice-dsl.deExisting Alice-PortalStep-by-step approachto a new Web platformManual optimization with YSlow,Google Pagespeed andWebPagetest“Performance Task Force“ 2008 2011 2012 Today 9
  10. 10. Our Website„Do the Move“ http://dsl.o2online.de Change from Telecom Italia to Telefónica / Merger with O2 Germany Redesign „Do the Move“ Build process for JS/CSS/Img: Unit testing with Specs, Linting of JS & CSS, packaging, minification, automatic image optimization2008 2011 2012 Today 10
  11. 11. Our WebsitePublic WebPagetest http://dsl.o2online.de Public WebPagetest instance Purely synthetic monitoring2008 2011 2012 Today 11
  12. 12. Our WebsitePrivate WebPagetest + RUM http://dsl.o2online.de Private WebPagetest instance Internal QS systems Runs hourly against PROD Daily against VAL Real User Monitoring2008 2011 2012 Today 12
  13. 13. 02RUM –Chapter titleArial 41User Monitoring. Real pt
  14. 14. RUMConcept Synthetic Real User Monitoring Monitoring Scenarios as planned most diverse Symptoms fair good Analysis / Diagnosis good bad Scope pre/post post release release 14
  15. 15. 02 RUM Navigation Timing API 15
  16. 16. 02 RUM Implementation, simple measurement, logging via AJAX <script> function () { if(typeof window.performance !== "undefined") { var msg={}, b = window.performance.timing; msg.navigationStart = b.navigationStart; msg.fetchStart = b.fetchStart; msg.requestStart = b.requestStart; msg.responseStart = b.responseStart; msg.responseEnd = b.responseEnd; msg.domLoading = b.domLoading; msg.domInteractive = b.domInteractive; msg.domComplete = b.domComplete; msg.loadEventEnd = b.loadEventEnd; unica.log(msg); jQuery.log(msg); } } </script> 16
  17. 17. 02 RUM Make or Buy, Sensoring simple, analysis not so simple 17
  18. 18. RUMHow many samples are needed? o2 Portal – mobile products DSL Portal – broadband products 2.1 M PageViews/Day 1.9 M PageViews/Day including new customer portal 160 k PageViews/Day 18
  19. 19. 03KPIs –Chapter titleArial 41 pt Key Performance Indicators.
  20. 20. 03 KPIs Timestamps and Intervals Synthetic Real User Monitoring Monitoring dns domainLookupEnd – domainLookupStart connect connectEnd – connectStart ttfb responseStart – connectEnd page responseEnd – responseStart 20
  21. 21. 03 KPIs Perception of our website Most important: fold Items below the Visual Complete slow (availability) Sometimes very Navigation Content downloaded via AJAX Product description domInteractive Enhancement after Order button, shopping cart 21
  22. 22. 03 KPIs Understand your Browser and your Website! Perceived Load Time Receiving response from server (or from Time to First Byte DOM Processing I + II application caches, or from local resources) Parsing jQuery response $(document).ready is fired! JSs running. Loading sub- resources (CSS, JS, images, fonts) 22
  23. 23. 03 KPIs Relative Frequency Perceived Load Time / 24 hours / all pages 23
  24. 24. 03 KPIs Trend 90% of page page page 50% of 80% of viewsviewsin 5.0inload in 2.9 load load 1.2 views seconds or less lessor less seconds or seconds Perceived Load Time / Trend 24
  25. 25. 04SLAs –Chapter titleArial 41 pt Level Agreements. Service
  26. 26. 04 SLAs Our Requirements – We are experimenting Perceived Load Time / Trend / Ratio 85% SLA 75% SLA 26
  27. 27. 04 SLAs Be specific Perceived Load Time / Segmented by Page Type 27
  28. 28. 05Chapter titleWhat we learned.Arial 41 pt
  29. 29. 05 What we learned Whats going on over the weekend? Perceived Load Time / Trend 29
  30. 30. 05 What we learned Whats going on over the weekend? Bandwidth / Trend 30
  31. 31. 05 What we learned What about Mobile? Perceived Load Time / Segmented by Network 31
  32. 32. 05 What we learned We have a Problem! Time to First Byte / Trend Release Release Release 29 Patch 1 29 Patch 2 29 Patch 3 32
  33. 33. 06Chapter titleWhat we changed.Arial 41 pt
  34. 34. 06 What we changed • “Performance Team” – dedicated group Organisational • Process for performance issues • Identifying changes in trend lines Mission • Analysing, reporting and tracking • Monitoring content changes (“content bugs”) Quality • Performance moved from ad-hoc activities in development to a continuously monitored aspect of Management quality Portal • Asking questions • How is the site perceived by mobile users? Management 34
  35. 35. 35
  36. 36. 36

×