Who cares about customer
         experience?
         A Complete
         Web Monitoring
         perspective




Wednesd...
“Hard” data

             Analytics                  Usability     Performability
           (what did they            (ho...
What did they do?
         Web analytics




Wednesday, November 11, 2009
http://www.flickr.com/photos/diegocupolo/3511117614/
Wednesday, November 11, 2009
Accounting                                     Optimization


                                                            ...
How did they do it?
         Web Interaction Analytics




Wednesday, November 11, 2009
http://www.flickr.com/photos/trekkyandy/189717616/

Wednesday, November 11, 2009
Why did they do it?
         Voice of the Customer




Wednesday, November 11, 2009
http://www.flickr.com/photos/karola/3623768629/
Wednesday, November 11, 2009
Could they do it?
         Performance & availability




Wednesday, November 11, 2009
Wednesday, November 11, 2009
What were they saying?
         Community monitoring




Wednesday, November 11, 2009
!
Wednesday, November 11, 2009
What were they up to?
         Competitive analysis




Wednesday, November 11, 2009
http://www.flickr.com/photos/31690139@N02/2965956581/
Wednesday, November 11, 2009
Why here? Why now?
         Online business is becoming the default.




Wednesday, November 11, 2009
!8B                                         • In 2006 !3B online retail
                                 !1.1
            ...
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Online marketers want to
                  maximize their revenues.



Wednesday, November 11, 2009
http://www.davidross.com.mx/admin/wideimagerepository/sergioZymanAmp.jpg
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
http://www.goal-posts.net/media/fbl-181_cat_07_160_x_160.jpg
Wednesday, November 11, 2009
http://www.johnsoncontrols.com/publish/etc/medialib/jci/ps/Media_Kit.Par.25973.File.dat/complex_machinery_HIRES.JPG
Wednes...
http://www.infovisual.info/02/img_en/078%20Different%20land%20animals%201.jpg

Wednesday, November 11, 2009
http://upload.wikimedia.org/wikipedia/commons/0/0e/Soccer_Youth_Goal_Keeper.jpg
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Some sites have ungoals.
         Things you really don’t want visitors to do.




Wednesday, November 11, 2009
You really don’t want web
                    users to call you.
          US$15


          US$12


            US$9


  ...
Websites
    have a dirty
    little secret


   http://todaystatus.files.wordpress.com/2009/04/ww11-secret.jpg
Wednesday, ...
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
http://www.inquisitr.com/2097/site-meter-causing-internet-explorer-failure/



Wednesday, November 11, 2009
http://www.flickr.com/photos/aleermakers/3455786409/
Wednesday, November 11, 2009
Wednesday, November 11, 2009
http://www.octulipfestival.com/images/Picture%20144.jpg
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Figure 3          Interactive user productivity versus computer response time for human-intensive
                        ...
10 s
                                1s

                               100 ms
                               10 ms
      ...
Wednesday, November 11, 2009
Wednesday, November 11, 2009
http://www.flickr.com/photos/spunter/393793587   http://www.flickr.com/photos/laurenclose/2217307446




Wednesday, November...
!"#$%&'()*+(,+()&#$-."#)/
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Shopzilla had another angle

         •      Big, high-traffic site          •   16 month re-engineering
         •      10...
5-12% increase in
                               revenue.


Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Tying web latency to
                     business outcomes.


Wednesday, November 11, 2009
http://www.flickr.com/photos/spunter/393793587   http://www.flickr.com/photos/laurenclose/2217307446




                   ...
Wednesday, November 11, 2009
http://www.flickr.com/photos/mrmoorey/160654236




Wednesday, November 11, 2009
Wednesday, November 11, 2009
ATTENTION                              ENGAGEMENT     CONVERSION

                                 NEW
                   ...
Wednesday, November 11, 2009
VISITOR         STRANGELOOP                WEB
                                               ACCELERATOR               SE...
What we learned:



Wednesday, November 11, 2009
Traffic levels
                             9,000
    Total number of visits




                             6,750



    ...
Bounce rate
                           20



                           15
     Visits that bounced




                  ...
% visits marked “new”
 % of visits that had no returning cookie



                                            14



     ...
That means...
                           9000
  Value Number of visits




                           6750



            ...
Average time on site
                             31
    Time on site (minutes)




                             23



   ...
Pages per visit
                         16



                         12
    Average pages seen




                    ...
Conversion rate
                                       20
                                            and order value
    ...
This is just one case
              LOTS


            # OF
          VISITS

                               OPTIMIZED
   ...
With one outcome
              LOTS


            # OF
          VISITS
                               21.58%
            ...
With one outcome
              LOTS

                               Best 5%            Worst 5%
            # OF
         ...
Lots of different results
              LOTS             24%

                                     18%
            $ PER  ...
You have your own curve
              LOTS


            $ PER
              DAY


                  0
                   ...
“What’s a latency?”




Wednesday, November 11, 2009
!"#$%&'(')*$+#,$%&)-.)/01-
     Browser                                                                      Data center

...
Analytics site

                                              Server


                                                  D...
21--#/01-.$%1$*1)3$4'%/5&&#-%
       !"##$%&"#'(')'***+,-.%/00$#.+%"1'234+5(6+36+758
       !"##$%&"#'7')'***+,-.%/00$#.+%...
6")%$/15*3$!"##$%&'$71$+&1-78
       1#$2.$34)5#                            D"4(@4G(@(&%
       • 6789+J-#B'.C$'=-.$      ...
Wednesday, November 11, 2009
How do we measure it?




Wednesday, November 11, 2009
Test                  Testing
        config                   node

                                         Data center
 ...
Synthetic isn’t enough




Wednesday, November 11, 2009
Browser                                    Load      Web
                               Network      balancer   server
   ...
Justifying an investment in
                       performance

          Current
                    *
                  ...
Justifying an investment in
                       performance

      $10,000 *                (   0.1607 + 0.0551   )   =...
Conclusions
         • Links between performance and business KPIs
                are undeniable

         • By talking t...
Putting it all together
         The Big Picture




Wednesday, November 11, 2009
“Hard” data

             Analytics                  Usability     Performability
           (what did they            (ho...
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Wednesday, November 11, 2009
Thanks!
     alistair@bitcurrent.com               sean@httpd.org
     @acroll                               @seanpower
  ...
Upcoming SlideShare
Loading in …5
×

Web Performance: Who Cares About Customer Experience? : Sean Power and Alistair Croll

2,163 views

Published on

This was presented on the MeasureWorks event in Baarn on 12 November 2009 by Sean Power and Alistair Croll. More Event Info:
www.measureworks-event.nl and www.measureworks.nl

Published in: Business, Technology, Design
  • Be the first to comment

Web Performance: Who Cares About Customer Experience? : Sean Power and Alistair Croll

  1. 1. Who cares about customer experience? A Complete Web Monitoring perspective Wednesday, November 11, 2009
  2. 2. “Hard” data Analytics Usability Performability (what did they (how did they (could they do do on the interact with what they site?) it?) wanted to?) Complete Web Monitoring Community VoC Competition (what were (what were (what are they they saying?) their up to?) motivations?) “Soft” data Wednesday, November 11, 2009
  3. 3. What did they do? Web analytics Wednesday, November 11, 2009
  4. 4. http://www.flickr.com/photos/diegocupolo/3511117614/ Wednesday, November 11, 2009
  5. 5. Accounting Optimization http://www.flickr.com/photos/sanchom/2963072255/ http://www.flickr.com/photos/thomasclaveirole/538819881/ Wednesday, November 11, 2009
  6. 6. How did they do it? Web Interaction Analytics Wednesday, November 11, 2009
  7. 7. http://www.flickr.com/photos/trekkyandy/189717616/ Wednesday, November 11, 2009
  8. 8. Why did they do it? Voice of the Customer Wednesday, November 11, 2009
  9. 9. http://www.flickr.com/photos/karola/3623768629/ Wednesday, November 11, 2009
  10. 10. Could they do it? Performance & availability Wednesday, November 11, 2009
  11. 11. Wednesday, November 11, 2009
  12. 12. What were they saying? Community monitoring Wednesday, November 11, 2009
  13. 13. ! Wednesday, November 11, 2009
  14. 14. What were they up to? Competitive analysis Wednesday, November 11, 2009
  15. 15. http://www.flickr.com/photos/31690139@N02/2965956581/ Wednesday, November 11, 2009
  16. 16. Why here? Why now? Online business is becoming the default. Wednesday, November 11, 2009
  17. 17. !8B • In 2006 !3B online retail !1.1 sales in the Netherlands !6B !1.7 • In 2011 !12B in total online sales in Benelux, with more than half of !1.9 !4B the Dutch adult population shopping online !2B !3.3 • In 2014 Forrester estimates over !8B in !0B online retail sales in the Netherlands Other Leisure flights Consumer electronics Hotels Wednesday, November 11, 2009
  18. 18. Wednesday, November 11, 2009
  19. 19. Wednesday, November 11, 2009
  20. 20. Wednesday, November 11, 2009
  21. 21. Online marketers want to maximize their revenues. Wednesday, November 11, 2009
  22. 22. http://www.davidross.com.mx/admin/wideimagerepository/sergioZymanAmp.jpg Wednesday, November 11, 2009
  23. 23. Wednesday, November 11, 2009
  24. 24. Wednesday, November 11, 2009
  25. 25. Wednesday, November 11, 2009
  26. 26. http://www.goal-posts.net/media/fbl-181_cat_07_160_x_160.jpg Wednesday, November 11, 2009
  27. 27. http://www.johnsoncontrols.com/publish/etc/medialib/jci/ps/Media_Kit.Par.25973.File.dat/complex_machinery_HIRES.JPG Wednesday, November 11, 2009
  28. 28. http://www.infovisual.info/02/img_en/078%20Different%20land%20animals%201.jpg Wednesday, November 11, 2009
  29. 29. http://upload.wikimedia.org/wikipedia/commons/0/0e/Soccer_Youth_Goal_Keeper.jpg Wednesday, November 11, 2009
  30. 30. Wednesday, November 11, 2009
  31. 31. Wednesday, November 11, 2009
  32. 32. Wednesday, November 11, 2009
  33. 33. Wednesday, November 11, 2009
  34. 34. Wednesday, November 11, 2009
  35. 35. Wednesday, November 11, 2009
  36. 36. Wednesday, November 11, 2009
  37. 37. Some sites have ungoals. Things you really don’t want visitors to do. Wednesday, November 11, 2009
  38. 38. You really don’t want web users to call you. US$15 US$12 US$9 US$6 $5.50 $3.00 US$3 $0.24 $0.45 US$0 Web self-service IVR Email Live phone Cost estimates BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and Improves Customer Service” Low Average High Wednesday, November 11, 2009
  39. 39. Websites have a dirty little secret http://todaystatus.files.wordpress.com/2009/04/ww11-secret.jpg Wednesday, November 11, 2009
  40. 40. Wednesday, November 11, 2009
  41. 41. Wednesday, November 11, 2009
  42. 42. Wednesday, November 11, 2009
  43. 43. http://www.inquisitr.com/2097/site-meter-causing-internet-explorer-failure/ Wednesday, November 11, 2009
  44. 44. http://www.flickr.com/photos/aleermakers/3455786409/ Wednesday, November 11, 2009
  45. 45. Wednesday, November 11, 2009
  46. 46. http://www.octulipfestival.com/images/Picture%20144.jpg Wednesday, November 11, 2009
  47. 47. Wednesday, November 11, 2009
  48. 48. Figure 3 Interactive user productivity versus computer response time for human-intensive interactions for system A E 600 - 3 T -" INTERACTIVE USER PRODUCTIVITY (IUP) w -HUMAN-INTENSIVE COMPONENT OF IUP 7 MEASURED DATA (HUMAN-INTENSIVE E 500 - A z " COMPONENT) U E - w E 0 > - > - - 400 3 n F 2 0 0 300 - 200 - 100 - 0 0- I 1 I I I 0 1 2 3 4 5 COMPUTER RESPONSE TIME (SI (1981) A. J. Thadhani, IBM Systems Journal, Volume 20, number 4 Wednesday, November 11, 2009
  49. 49. 10 s 1s 100 ms 10 ms ! Zzz Wednesday, November 11, 2009
  50. 50. Wednesday, November 11, 2009
  51. 51. Wednesday, November 11, 2009
  52. 52. http://www.flickr.com/photos/spunter/393793587 http://www.flickr.com/photos/laurenclose/2217307446 Wednesday, November 11, 2009
  53. 53. !"#$%&'()*+(,+()&#$-."#)/ Wednesday, November 11, 2009
  54. 54. Wednesday, November 11, 2009
  55. 55. Shopzilla had another angle • Big, high-traffic site • 16 month re-engineering • 100M impressions a day • Page load from 6 seconds to 1.2 • 8,000 searches a second • 20-29M unique visitors a • Uptime from 99.65% to month 99.97% • 100M products • 10% of previous hardware needs http://en.oreilly.com/velocity2009/public/schedule/detail/7709 Wednesday, November 11, 2009
  56. 56. 5-12% increase in revenue. Wednesday, November 11, 2009
  57. 57. Wednesday, November 11, 2009
  58. 58. Wednesday, November 11, 2009
  59. 59. Tying web latency to business outcomes. Wednesday, November 11, 2009
  60. 60. http://www.flickr.com/photos/spunter/393793587 http://www.flickr.com/photos/laurenclose/2217307446 KPIs Wednesday, November 11, 2009
  61. 61. Wednesday, November 11, 2009
  62. 62. http://www.flickr.com/photos/mrmoorey/160654236 Wednesday, November 11, 2009
  63. 63. Wednesday, November 11, 2009
  64. 64. ATTENTION ENGAGEMENT CONVERSION NEW VISITORS SEARCHES GROWTH CONVERSION PAGES TIME RATE TWEETS NUMBER OF VISITS PER ON 0 MENTIONS VISIT SITE ORDER ADS SEEN LOSS VALUE BOUNCE RATE Wednesday, November 11, 2009
  65. 65. Wednesday, November 11, 2009
  66. 66. VISITOR STRANGELOOP WEB ACCELERATOR SERVER Decide whether to optimize Normal Receive Accelerated content page Insert Process scripts Optimize? segment marker Send analytics Unaccelerated GOOGLE ANALYTICS Wednesday, November 11, 2009
  67. 67. What we learned: Wednesday, November 11, 2009
  68. 68. Traffic levels 9,000 Total number of visits 6,750 4,500 8,505 2,250 4,740 0 Optimized Unoptimized Visitor experience Wednesday, November 11, 2009
  69. 69. Bounce rate 20 15 Visits that bounced 10 13.38% 14.35% 5 0 Optimized Unoptimized Visitor experience Wednesday, November 11, 2009
  70. 70. % visits marked “new” % of visits that had no returning cookie 14 11 7 13.61% 10.85% 4 0 Optimized Unoptimized Visitor experience Wednesday, November 11, 2009
  71. 71. That means... 9000 Value Number of visits 6750 4500 7,582 4,095 2250 923 645 0 Optimized Unoptimized Wednesday, November 11, 2009
  72. 72. Average time on site 31 Time on site (minutes) 23 16 30.17 23.83 8 0 Optimized Unoptimized Visitor experience Wednesday, November 11, 2009
  73. 73. Pages per visit 16 12 Average pages seen 8 15.64 11.04 4 0 Optimized Unoptimized Visitor experience Wednesday, November 11, 2009
  74. 74. Conversion rate 20 and order value Difference due to optimization 15 10 16.07 5 5.51 0 Conversion rate Order value Wednesday, November 11, 2009
  75. 75. This is just one case LOTS # OF VISITS OPTIMIZED 0 0 VISITOR LATENCY 10,000 Different visitors experienced different performance levels. Wednesday, November 11, 2009
  76. 76. With one outcome LOTS # OF VISITS 21.58% 0 BETTER 0 VISITOR LATENCY 10,000 Right now we have a single experiment, and a single resulting business impact. Wednesday, November 11, 2009
  77. 77. With one outcome LOTS Best 5% Worst 5% # OF VISITS 21.58% 0 BETTER 0 VISITOR LATENCY 10,000 Visitors who were optimized fall into a range – the 5th to 95th percentile. Wednesday, November 11, 2009
  78. 78. Lots of different results LOTS 24% 18% $ PER 14% 12% DAY 9.5% 0 0 VISITOR LATENCY 10,000 If we have several experiments, we can understand the relationship better. Wednesday, November 11, 2009
  79. 79. You have your own curve LOTS $ PER DAY 0 0 VISITOR LATENCY 10,000 Every web business has a curve like this hidden inside it. Wednesday, November 11, 2009
  80. 80. “What’s a latency?” Wednesday, November 11, 2009
  81. 81. !"#$%&'(')*$+#,$%&)-.)/01- Browser Data center Server TCP SYN (“let’s talk”) TCP SYN ACK (“Agreed: let’s talk”) TCP ACK (“OK, we’re talking) SSL (“Someone might be listening!”) SSL (“Here’s a decoder ring”) HTTP GET / (“Can I have your home page?”) HTTP 200 OK (“Sure!”) (thinks [index.html] (“Here it is!”) a bit) (Renders furiously) Bump, bump. [img js css] (“Have this too!”) TCP FIN (“Thanks! I’m done now.”) TCP FIN ACK (“You’re welcome. Have a nice day.”) Wednesday, November 11, 2009
  82. 82. Analytics site Server Data center Browser Server Snurken Mashup site Server Wednesday, November 11, 2009
  83. 83. 21--#/01-.$%1$*1)3$4'%/5&&#-% !"##$%&"#'(')'***+,-.%/00$#.+%"1'234+5(6+36+758 !"##$%&"#'7')'***+,-.%/00$#.+%"1'234+5(6+36+758 !"##$%&"#'5')'9:-#;-.$+9:+-<$0%$<&"#=+%"1'239+7>+47+4(8' !"##$%&"#'?')'=.@&%+=A-B$=C@0$+#$.'233+779+9D+598 !"##$%&"#'9')'=.@&%+=A-B$=C@0$+#$.'233+779+9D+598 !"##$%&"#'6')'***+E$$B,/0#$0+%"1'233+76(+D3+75?8 !"##$%&"#'3')'=.@&%+F$.%A-%GH+%"1'25(9+7?+>+7>8' !"##$%&"#'4')'%$.0G+%"1'25(>+34+7>?+7((8 !"##$%&"#'>')'-#+F$.%A-%GH+%"1'25(9+7?+>+7>8 !"##$%&"#'D')'%0@IH$FF+%"1'25(>+34+7>(+5?38 !"##$%&"#'7(')'***+F""FA$)@#@AH&%=+%"1'245+79+55?+7948' !"##$%&"#'77')'***+@<./0$+%"1'234+7D5+93+7D8 !"##$%&"#'75')'=.@&%+@<./0$+%"1'234+7D5+93+568 !"##$%&"#'7?')'=+%A-%G.@A$+#$.'233+779+9D+598' !"##$%&"#'79')'***+%A-%G.@A$+#$.'246+756+>5+4(8 Wednesday, November 11, 2009
  84. 84. 6")%$/15*3$!"##$%&'$71$+&1-78 1#$2.$34)5# D"4(@4G(@(&% • 6789+J-#B'.C$'=-.$ • <@(#)&9+U@B'0$:/$=.= • :19+K"/.$'.C$'<@%G$.= • 8#$"#$9+!@<@%-.HS'@;@-A@,-A-.HS'VW • ;<19+L=.@,A-=C'@'%"##$%&"# • 7#&-.$A9+R@%G$.'A"==S'X!R'&1$"/.=S' • <679+K$B-0$%.M";$0A@H'.C$'.0@N% .0@N%'=C@<-#F • 88=9+O$F"&@.$'$#%0H<&"# • DHH@(54I.)9+Y"F-%S'B$<$#B$#%-$= • >;;19+K$:/$=.'%"#.$#. • 74"(*4I.)9+Y""<-#FS'=$==-"#'&1$"/.= • >.,&+?#@4%9+P$#$0@.$'.C$'%"#.$#. • <.)&#)&9+U0"G$#S'/#0$#B$0@,A$ • 7#&-.$A+?#@4%9+Q$A-;$0'.C$'%"#.$#. • C:D9+RA/F-#S'1$1"0HS'VW • 145A#&+@.,,9+K$%";$0'E0"1'$00"0= • B$.-,#$9+R@0=$'.C$'<@F$S'F$.'1"0$' %"#.$#. • C:D,9+LT$%/.$'%"B$'"#'.C$'%A-$#. • B$.-,#$E+F89+K$#B$0'.C$'<@F$ Wednesday, November 11, 2009
  85. 85. Wednesday, November 11, 2009
  86. 86. How do we measure it? Wednesday, November 11, 2009
  87. 87. Test Testing config node Data center Testing node Website Reporting service Testing node Wednesday, November 11, 2009
  88. 88. Synthetic isn’t enough Wednesday, November 11, 2009
  89. 89. Browser Load Web Network balancer server tap User A User B User C Visit Aggregate Alerts history reports ! P1 P2 P3 Wednesday, November 11, 2009
  90. 90. Justifying an investment in performance Current * Increased + (Increased daily orders conversions order value ) ROI = (days) Cost of performance enhancement Wednesday, November 11, 2009
  91. 91. Justifying an investment in performance $10,000 * ( 0.1607 + 0.0551 ) = $2,158 23.17 = days $50,000 $50,000 Caveats: Your mileage will vary This is just how to think about it Wednesday, November 11, 2009
  92. 92. Conclusions • Links between performance and business KPIs are undeniable • By talking the same language, IT and marketing can finally agree on what to do about it • Changing from “X times faster” to “$Y more money” makes the business care • More research is needed Wednesday, November 11, 2009
  93. 93. Putting it all together The Big Picture Wednesday, November 11, 2009
  94. 94. “Hard” data Analytics Usability Performability (what did they (how did they (could they do on the interact with do what they site?) it?) wanted to?) Complete Web Monitoring Community VoC Competition (what were (what were (what are they they saying?) their up to?) motivations?) “Soft” data Wednesday, November 11, 2009
  95. 95. Wednesday, November 11, 2009
  96. 96. Wednesday, November 11, 2009
  97. 97. Wednesday, November 11, 2009
  98. 98. Thanks! alistair@bitcurrent.com sean@httpd.org @acroll @seanpower linkedin/in/alistaircroll linkedin/in/slpower bitcurrent.com www.watchingwebsites.com rednod.com www.watchingwebsites.com http://www.amazon.com/Complete-Web-Monitoring- Performance-Communities/dp/0596155131/ Wednesday, November 11, 2009

×