ONLINE DIALOGUEGuiding you to digital maturity   A/B-testing Workshop Web Analytics Congress    2013/03/13, Utrecht, the N...
A/B-testing                                  1.  M e t h o d o l o g y                                        2.  To o l s...
1995@tonwesseling@onlinedialogue     The first graphical web browsers
#measure@tonwesseling@onlinedialogue               Logfile analyses
Why?@tonwesseling@onlinedialogue           To o p t i m i z e
To improve@tonwesseling@onlinedialogue                 To g r o w
We as ked            Where     can we improve?                  We took small steps                 So we could act now...
@tonwesseling    @onlinedialogue       They did something else…
Knowlegde	                                             People	      @tonwesseling    @onlinedialogue                      ...
@tonwesseling    @onlinedialogue       Out of reach
1999@tonwesseling@onlinedialogue     Overstock.com design change
2002@tonwesseling@onlinedialogue     Overstock.com design change
2005@tonwesseling@onlinedialogue     Overstock.com design change
2010@tonwesseling@onlinedialogue     Overstock.com design change
2013@tonwesseling@onlinedialogue     Overstock.com design change
What happ end to:            Where     can we improve?                  We took small steps                 So we could...
W hat’s the d ifference?    @tonwesseling    @onlinedialogue   With the owner of the local bakery store?
Company	  goals	                                                                                                          ...
H ow to communicate    @tonwesseling    @onlinedialogue   With the brains of your visitors
Be like B ill    @tonwesseling    @onlinedialogue                In Groundhog day
Be like B ill    @tonwesseling    @onlinedialogue              And convince the brain
Do you know this man?    @tonwesseling    @onlinedialogue   W. E d w a r d s D e m i n g
Deming Cycl e     @tonwesseling    @onlinedialogue              Quality management
Focus on q ua lity         results of work efforts    quality = ___________________            total cost    @tonwesseling...
Focus on q ua lity                                                                            “When you focus just on cost...
Obser vation  +O               PDCA    Like            the Toyota production system @tonwesseling @onlinedialogue       ...
The Toyota way   Challenge                (We   form a long-term vision, meeting challenges with                   coura...
Kaizen                   改善@tonwesseling@onlinedialogue      continuous improvement process
“ Change for the better ”@tonwesseling@onlinedialogue                Kaizen – Good change
How to perform experiments      using the scientific method@tonwesseling@onlinedialogue        Kaizen – Good change
Hypothes es                           Hypot                                 hesis"                                        ...
A/B-testing                                              Default"                              Variation                  ...
A /B-te sting maturity                                                                                                    ...
SAME N mod el    @tonwesseling    @onlinedialogue              L e t ’s d o i t t o g e t h e r
SAME N mod el                    S"           A"        M"          E"                   N"               Start reason"   ...
“Without an analyzed reason                      to start, it does not make                       any sense to start at al...
“Do youunderstand whatis happening?”!    @tonwesseling    @onlinedialogue       Analyze
“You don’t make perfect                       designs in 1 time.!                       !                       Step by st...
“Not only the                           conversion,                           look for                           insights!...
“If someone can not    reproduce your learnings… !    Then, there were no    learnings...”!    @tonwesseling    @onlinedia...
Tools!    @tonwesseling    @onlinedialogue     Everything is already here as SaaS
Tools!    @tonwesseling    @onlinedialogue       The tools we use these days
Start reasonGebruikers Onderzoek (SurveyMonkey), Gebruikers wensen (Uservoice),   Continue gebruikers feedback (GetSatisfa...
Start reason@tonwesseling@onlinedialogue                   Surveymonkey
Start reason@tonwesseling@onlinedialogue             Getsatisfaction.com
Start reason@tonwesseling@onlinedialogue                   Kampyle.com
Start reason@tonwesseling@onlinedialogue          Tr e s h o l d s ! ( G o o g l e A n a l y t i c s )
Start reason@tonwesseling@onlinedialogue           Tr e n d s p o t ( B e n c h m a r k i n g )
Start reason@tonwesseling@onlinedialogue       Mozenda.com: competitve information
Start reason@tonwesseling@onlinedialogue           Momentum (social insights)
Start reason@tonwesseling@onlinedialogue          I n t e r n a l i n s i g h t s ( Ya m m e r. c o m )
AnalyzeAnalytics pakket, Mousemaps en Screen recordings (Clicktale), Clickmaps(Crazyegg, Clicktale), Survey (SurveyMonkey)...
Analyze@tonwesseling@onlinedialogue   Get your tag management right! (Qubit open tag)
Analyze@tonwesseling@onlinedialogue              Clicktale.com
Analyze@tonwesseling@onlinedialogue              Qualaroo.com
Analyze@tonwesseling@onlinedialogue              Usertesting.com
Analyze@tonwesseling@onlinedialogue              Notableapp.com
MakeWire-Frames (Mockflow.com), Concept testing (Verifyapp.com), Clickabledemo’s / usability (Soldifyapp.com), Usability fe...
Make@tonwesseling@onlinedialogue           M o c k fl o w. c o m
Make@tonwesseling@onlinedialogue           Ve r i f y a p p . c o m
Make@tonwesseling@onlinedialogue           Soldifyapp.com
Make@tonwesseling@onlinedialogue    V i s u a l w e b s i t e o p t i m i z e r. c o m ( t e s t i n g )
Make@tonwesseling@onlinedialogue       Personyze.com (targeting)
Make@tonwesseling@onlinedialogue           Conductrics.com (RTD)
E valuateAnalytics pakket (segmenten), Mousemaps en Screen recordings (Clicktale),Clickmaps (Crazyegg, Clicktale), Survey ...
E valuate@tonwesseling@onlinedialogue                Dive in that data!
E valuate@tonwesseling@onlinedialogue            Don’t forget the why!
E valuate@tonwesseling@onlinedialogue       Measure lifetime value indicators
NotifyLearnings bord, Learnings boekje, Powerpoint single slide learning (voormail, maar ook voor op TV’s), Gedrags person...
Notify                                               Publish & Promote                     your insights!@tonwesseling@onl...
Test tips                                               “ Not every test              will improve conversion ”@tonwesseli...
Test tips                                                “ You will learn                   you should target ”@tonwesseli...
Psychology    @tonwesseling    @onlinedialogue            Wheel of persuasion
Test tips@tonwesseling@onlinedialogue         Abnamro.nl: Landing pages
Test tips@tonwesseling@onlinedialogue     Californiaclosets.com: Landing pages
Test tips@tonwesseling@onlinedialogue    Perrysport.nl – landing page messaging
Test tips@tonwesseling@onlinedialogue      Alphaecological.com: Form location
Test tips@tonwesseling@onlinedialogue            Jules.fr: Shop photos
Test tips@tonwesseling@onlinedialogue                Kras.nl: Footer
Test tips@tonwesseling@onlinedialogue            Wehkamp.nl: Princing
Test tips@tonwesseling@onlinedialogue           Kickz.com: price focus?
Test tips@tonwesseling@onlinedialogue    D e s k a w a y. c o m – “ u g l y b r o t h e r ” p r o d u c t
Test tips@tonwesseling@onlinedialogue       CDFnetworks.com – Form lay-out
Test tips@tonwesseling@onlinedialogue        G s m s t u n t e r. n l – i n t e r n a l s e a r c h
Test tips@tonwesseling@onlinedialogue         Bijenkorf.nl: point-of-actions
Test tips@tonwesseling@onlinedialogue      A c t i e v a n d e d a g . n l : C TA m o b i l e p r o o f
Test tips@tonwesseling@onlinedialogue        Basecamphq.com – next action
Test tips@tonwesseling@onlinedialogue            Kras.nl - checkmarks
Test tips@tonwesseling@onlinedialogue            VGZ.nl - checkmarks
Test tips@tonwesseling@onlinedialogue        Hotelschiphol.nl - checkmarks
Test tips@tonwesseling@onlinedialogue         T h e c o r k s c r e w. i e - d i s c o u n t
Test tips@tonwesseling@onlinedialogue          Bol.com – like positioning
Test tips@tonwesseling@onlinedialogue         Inshared.nl – Sticky content
To d o:               Optimize                       your dialogue                 Focus on quality             Spread ...
Upcoming SlideShare
Loading in …5
×

Web Analytics Congress 2013 a/b-test workshop

1,710 views

Published on

Presentation by Ton Wesseling (90 minute workshop) at #wac2013.

Published in: Business

Web Analytics Congress 2013 a/b-test workshop

  1. 1. ONLINE DIALOGUEGuiding you to digital maturity A/B-testing Workshop Web Analytics Congress 2013/03/13, Utrecht, the Netherlands Slides: http://ondi.me/wac2013
  2. 2. A/B-testing 1.  M e t h o d o l o g y 2.  To o l s 3.  Te s t i n g t i p s To n W e s s e l i n g Chief Conversion Officer Online DialogueSlides: http://ondi.me/wac2013
  3. 3. 1995@tonwesseling@onlinedialogue The first graphical web browsers
  4. 4. #measure@tonwesseling@onlinedialogue Logfile analyses
  5. 5. Why?@tonwesseling@onlinedialogue To o p t i m i z e
  6. 6. To improve@tonwesseling@onlinedialogue To g r o w
  7. 7. We as ked   Where can we improve?   We took small steps   So we could act now!@tonwesseling@onlinedialogue …and it was fun to do!
  8. 8. @tonwesseling @onlinedialogue They did something else…
  9. 9. Knowlegde   People   @tonwesseling @onlinedialogue Amazing
  10. 10. @tonwesseling @onlinedialogue Out of reach
  11. 11. 1999@tonwesseling@onlinedialogue Overstock.com design change
  12. 12. 2002@tonwesseling@onlinedialogue Overstock.com design change
  13. 13. 2005@tonwesseling@onlinedialogue Overstock.com design change
  14. 14. 2010@tonwesseling@onlinedialogue Overstock.com design change
  15. 15. 2013@tonwesseling@onlinedialogue Overstock.com design change
  16. 16. What happ end to:   Where can we improve?   We took small steps   So we could act now!@tonwesseling@onlinedialogue …and it was fun to do!
  17. 17. W hat’s the d ifference? @tonwesseling @onlinedialogue With the owner of the local bakery store?
  18. 18. Company  goals   €€€                     Customer  goals   Company   Prospect  /  client   Time  &  Money   Knowledge   Behavior   Behavior   Knowledge   ACtude   Mul<   Mul<   Mul<-­‐   Personality   Tools   purpose     phase   channel   Internal     PlaHorm   Profile   influencers   Internal   Internal   Mo<va<on   influencers   Context   influencers   External  influencers   Compe<tors   Media   Social  environment   @tonwesseling @onlinedialogue Dialogue model – improve dialogue insights
  19. 19. H ow to communicate @tonwesseling @onlinedialogue With the brains of your visitors
  20. 20. Be like B ill @tonwesseling @onlinedialogue In Groundhog day
  21. 21. Be like B ill @tonwesseling @onlinedialogue And convince the brain
  22. 22. Do you know this man? @tonwesseling @onlinedialogue W. E d w a r d s D e m i n g
  23. 23. Deming Cycl e @tonwesseling @onlinedialogue Quality management
  24. 24. Focus on q ua lity results of work efforts quality = ___________________ total cost @tonwesseling @onlinedialogue When it is defined by this ratio it leads to results
  25. 25. Focus on q ua lity “When you focus just on costs they will rise”@tonwesseling@onlinedialogue W. E d w a r d s D e m i n g
  26. 26. Obser vation  +O PDCA   Like the Toyota production system @tonwesseling @onlinedialogue Japan started to learn
  27. 27. The Toyota way   Challenge  (We form a long-term vision, meeting challenges with courage and creativity to realize our dreams.)   Kaizen  (We improve our business operations continuously, always driving for innovation and evolution.)   Genchi Genbutsu  (Go to the source to find the facts to make correct decisions.)@tonwesseling@onlinedialogue Continuous improvement
  28. 28. Kaizen 改善@tonwesseling@onlinedialogue continuous improvement process
  29. 29. “ Change for the better ”@tonwesseling@onlinedialogue Kaizen – Good change
  30. 30. How to perform experiments using the scientific method@tonwesseling@onlinedialogue Kaizen – Good change
  31. 31. Hypothes es Hypot hesis" Alternative" @tonwesseling @onlinedialogue Statistical hypothesis testing
  32. 32. A/B-testing Default" Variation A" Variation B" Winner!" @tonwesseling @onlinedialogue Small steps to learn how to improve the dialogue
  33. 33. A /B-te sting maturity Fase 5" Fase 4" Gesegmenteerde dialogen" Fase 3" Gespreksontwerp toepassen" Fase 2" Welke informatie zoekt de gebruiker?" Hoe halen we Niet dezelfde Fase 1" drempels weg?" De stap van dialoog voor Yeah, we kunnen pagina elke bezoeker" testen!" Willen begrijpen optimalisatie hoe je de naar user flow bezoeker beter optimalisatie" Waar kunnen we nu direct het kunt servicen" Maak niet uit Personaliseren   wat we testen verschil maken?" als we maar Mo<veren   testen!" Leren   Direct  resultaat   Laaghangend  fruit   @tonwesseling @onlinedialogue Yo u s h o u l d s t a r t a t l e v e l 1 b u t m o v e f a s t t o l e v e l 3
  34. 34. SAME N mod el @tonwesseling @onlinedialogue L e t ’s d o i t t o g e t h e r
  35. 35. SAME N mod el S" A" M" E" N" Start reason" Analyze" Make" Evaluate" Notify" @tonwesseling @onlinedialogue Te l l t h e s t o r y “ n a v e r t e l l e n ”
  36. 36. “Without an analyzed reason to start, it does not make any sense to start at all.”! @tonwesseling @onlinedialogue Start reason
  37. 37. “Do youunderstand whatis happening?”! @tonwesseling @onlinedialogue Analyze
  38. 38. “You don’t make perfect designs in 1 time.! ! Step by step we’ll grow to the final result.”! ! Hypotheses based design!! @tonwesseling @onlinedialogue Make
  39. 39. “Not only the conversion, look for insights!”! @tonwesseling @onlinedialogue Evaluate
  40. 40. “If someone can not reproduce your learnings… ! Then, there were no learnings...”! @tonwesseling @onlinedialogue Notify (Navertellen)
  41. 41. Tools! @tonwesseling @onlinedialogue Everything is already here as SaaS
  42. 42. Tools! @tonwesseling @onlinedialogue The tools we use these days
  43. 43. Start reasonGebruikers Onderzoek (SurveyMonkey), Gebruikers wensen (Uservoice), Continue gebruikers feedback (GetSatisfaction, Kampyle), Analytics trendbreuk notificaties (analytics pakket), Benchmark vergelijken (Trendspot), Markpositie veranderd (Mozenda / Yahoo Pipes), Social (Finchline, Momentum), Intern (Basecamp / Yammer). "@tonwesseling@onlinedialogue To o l s
  44. 44. Start reason@tonwesseling@onlinedialogue Surveymonkey
  45. 45. Start reason@tonwesseling@onlinedialogue Getsatisfaction.com
  46. 46. Start reason@tonwesseling@onlinedialogue Kampyle.com
  47. 47. Start reason@tonwesseling@onlinedialogue Tr e s h o l d s ! ( G o o g l e A n a l y t i c s )
  48. 48. Start reason@tonwesseling@onlinedialogue Tr e n d s p o t ( B e n c h m a r k i n g )
  49. 49. Start reason@tonwesseling@onlinedialogue Mozenda.com: competitve information
  50. 50. Start reason@tonwesseling@onlinedialogue Momentum (social insights)
  51. 51. Start reason@tonwesseling@onlinedialogue I n t e r n a l i n s i g h t s ( Ya m m e r. c o m )
  52. 52. AnalyzeAnalytics pakket, Mousemaps en Screen recordings (Clicktale), Clickmaps(Crazyegg, Clicktale), Survey (SurveyMonkey), tijdelijke gebruikersfeedback(Qualaroo, 4Q en eigen tool (met Google Analytics)), Usability feedback(Feedbackarmy.com, Usertesting.com, Conceptfeedback.com), Internefeedback (notableapp.com)." @tonwesseling @onlinedialogue To o l s
  53. 53. Analyze@tonwesseling@onlinedialogue Get your tag management right! (Qubit open tag)
  54. 54. Analyze@tonwesseling@onlinedialogue Clicktale.com
  55. 55. Analyze@tonwesseling@onlinedialogue Qualaroo.com
  56. 56. Analyze@tonwesseling@onlinedialogue Usertesting.com
  57. 57. Analyze@tonwesseling@onlinedialogue Notableapp.com
  58. 58. MakeWire-Frames (Mockflow.com), Concept testing (Verifyapp.com), Clickabledemo’s / usability (Soldifyapp.com), Usability feedback (FeedbackarmyUsertesting, Conceptfeedback, usability test), Interne feedback(Notableapp.com), Test bouwen / uitvoeren (via CMS & script of VWO ofOptimizely), Targeting (Personyze, Conductrics)." @tonwesseling @onlinedialogue To o l s
  59. 59. Make@tonwesseling@onlinedialogue M o c k fl o w. c o m
  60. 60. Make@tonwesseling@onlinedialogue Ve r i f y a p p . c o m
  61. 61. Make@tonwesseling@onlinedialogue Soldifyapp.com
  62. 62. Make@tonwesseling@onlinedialogue V i s u a l w e b s i t e o p t i m i z e r. c o m ( t e s t i n g )
  63. 63. Make@tonwesseling@onlinedialogue Personyze.com (targeting)
  64. 64. Make@tonwesseling@onlinedialogue Conductrics.com (RTD)
  65. 65. E valuateAnalytics pakket (segmenten), Mousemaps en Screen recordings (Clicktale),Clickmaps (Crazyegg, Clicktale), Survey voor NPS / CES (SurveyMonkey)." @tonwesseling @onlinedialogue To o l s
  66. 66. E valuate@tonwesseling@onlinedialogue Dive in that data!
  67. 67. E valuate@tonwesseling@onlinedialogue Don’t forget the why!
  68. 68. E valuate@tonwesseling@onlinedialogue Measure lifetime value indicators
  69. 69. NotifyLearnings bord, Learnings boekje, Powerpoint single slide learning (voormail, maar ook voor op TV’s), Gedrags persona’s / Huisstijl update, Test /learnings archief (spreadsheet), intern (intranet, yammer, basecamp) enexterne publicaties." @tonwesseling @onlinedialogue To o l s
  70. 70. Notify Publish & Promote your insights!@tonwesseling@onlinedialogue Spread knowledge
  71. 71. Test tips “ Not every test will improve conversion ”@tonwesseling@onlinedialogue But you will learn from every test!
  72. 72. Test tips “ You will learn you should target ”@tonwesseling@onlinedialogue Move on and you’ll see
  73. 73. Psychology @tonwesseling @onlinedialogue Wheel of persuasion
  74. 74. Test tips@tonwesseling@onlinedialogue Abnamro.nl: Landing pages
  75. 75. Test tips@tonwesseling@onlinedialogue Californiaclosets.com: Landing pages
  76. 76. Test tips@tonwesseling@onlinedialogue Perrysport.nl – landing page messaging
  77. 77. Test tips@tonwesseling@onlinedialogue Alphaecological.com: Form location
  78. 78. Test tips@tonwesseling@onlinedialogue Jules.fr: Shop photos
  79. 79. Test tips@tonwesseling@onlinedialogue Kras.nl: Footer
  80. 80. Test tips@tonwesseling@onlinedialogue Wehkamp.nl: Princing
  81. 81. Test tips@tonwesseling@onlinedialogue Kickz.com: price focus?
  82. 82. Test tips@tonwesseling@onlinedialogue D e s k a w a y. c o m – “ u g l y b r o t h e r ” p r o d u c t
  83. 83. Test tips@tonwesseling@onlinedialogue CDFnetworks.com – Form lay-out
  84. 84. Test tips@tonwesseling@onlinedialogue G s m s t u n t e r. n l – i n t e r n a l s e a r c h
  85. 85. Test tips@tonwesseling@onlinedialogue Bijenkorf.nl: point-of-actions
  86. 86. Test tips@tonwesseling@onlinedialogue A c t i e v a n d e d a g . n l : C TA m o b i l e p r o o f
  87. 87. Test tips@tonwesseling@onlinedialogue Basecamphq.com – next action
  88. 88. Test tips@tonwesseling@onlinedialogue Kras.nl - checkmarks
  89. 89. Test tips@tonwesseling@onlinedialogue VGZ.nl - checkmarks
  90. 90. Test tips@tonwesseling@onlinedialogue Hotelschiphol.nl - checkmarks
  91. 91. Test tips@tonwesseling@onlinedialogue T h e c o r k s c r e w. i e - d i s c o u n t
  92. 92. Test tips@tonwesseling@onlinedialogue Bol.com – like positioning
  93. 93. Test tips@tonwesseling@onlinedialogue Inshared.nl – Sticky content
  94. 94. To d o:   Optimize your dialogue   Focus on quality   Spread your knowledge@tonwesseling@onlinedialogue Thank you! Slides: http://ondi.me/wac2013

×