Presentation by Ton Wesseling (90 minute workshop) at #wac2013.

1. ONLINE DIALOGUE
Guiding you to digital maturity
A/B-testing
Workshop Web Analytics Congress
2013/03/13, Utrecht, the Netherlands
Slides: http://ondi.me/wac2013

2. A/B-testing
1. M e t h o d o l o g y
2. To o l s
3. Te s t i n g t i p s
To n W e s s e l i n g
Chief Conversion Officer
Online Dialogue
Slides: http://ondi.me/wac2013

3. 1995
@tonwesseling
@onlinedialogue
The first graphical web browsers

4. #measure
@tonwesseling
@onlinedialogue
Logfile analyses

5. Why?
@tonwesseling
@onlinedialogue
To o p t i m i z e

6. To improve
@tonwesseling
@onlinedialogue
To g r o w

7. We as ked
 Where can we improve?
 We took small steps
 So we could act now!
@tonwesseling
@onlinedialogue
…and it was fun to do!

8. @tonwesseling
@onlinedialogue
They did something else…

9. Knowlegde
People
@tonwesseling
@onlinedialogue
Amazing

10. @tonwesseling
@onlinedialogue
Out of reach

11. 1999
@tonwesseling
@onlinedialogue
Overstock.com design change

12. 2002
@tonwesseling
@onlinedialogue
Overstock.com design change

13. 2005
@tonwesseling
@onlinedialogue
Overstock.com design change

14. 2010
@tonwesseling
@onlinedialogue
Overstock.com design change

15. 2013
@tonwesseling
@onlinedialogue
Overstock.com design change

16. What happ end to:
 Where can we improve?
 We took small steps
 So we could act now!
@tonwesseling
@onlinedialogue
…and it was fun to do!

17. W hat’s the d ifference?
@tonwesseling
@onlinedialogue
With the owner of the local bakery store?

18. Company
goals
€€€

Customer
goals
Company
Prospect
/
client
Time
&
Money
Knowledge
Behavior
Behavior
Knowledge
ACtude
Mul<
Mul<
Mul<-­‐
Personality
Tools
purpose
phase
channel
Internal
PlaHorm
Profile
influencers
Internal
Internal
Mo<va<on
influencers
Context
influencers
External
influencers
Compe<tors
Media
Social
environment
@tonwesseling
@onlinedialogue
Dialogue model – improve dialogue insights

19. H ow to communicate
@tonwesseling
@onlinedialogue
With the brains of your visitors

20. Be like B ill
@tonwesseling
@onlinedialogue
In Groundhog day

21. Be like B ill
@tonwesseling
@onlinedialogue
And convince the brain

22. Do you know this man?
@tonwesseling
@onlinedialogue
W. E d w a r d s D e m i n g

23. Deming Cycl e
@tonwesseling
@onlinedialogue
Quality management

24. Focus on q ua lity
results of work efforts
quality =
___________________
total cost
@tonwesseling
@onlinedialogue
When it is defined by this ratio it leads to results

25. Focus on q ua lity
“When you focus just on costs they will rise”
@tonwesseling
@onlinedialogue
W. E d w a r d s D e m i n g

26. Obser vation
 +O PDCA
 Like the Toyota production system
@tonwesseling
@onlinedialogue
Japan started to learn

27. The Toyota way
 Challenge
 (We form a long-term vision, meeting challenges with
courage and creativity to realize our dreams.)
 Kaizen
 (We improve our business operations continuously,
always driving for innovation and evolution.)
 Genchi Genbutsu
 (Go to the source to find the facts to make correct
decisions.)
@tonwesseling
@onlinedialogue
Continuous improvement

28. Kaizen
改善
@tonwesseling
@onlinedialogue
continuous improvement process

29. “ Change for the better ”
@tonwesseling
@onlinedialogue
Kaizen – Good change

30. How to perform experiments
using the scientific method
@tonwesseling
@onlinedialogue
Kaizen – Good change

31. Hypothes es
Hypot
hesis"
Alternative"
@tonwesseling
@onlinedialogue
Statistical hypothesis testing

32. A/B-testing
Default"
Variation
A"
Variation
B"
Winner!"
@tonwesseling
@onlinedialogue
Small steps to learn how to improve the dialogue

33. A /B-te sting maturity
Fase 5"
Fase 4" Gesegmenteerde
dialogen"
Fase 3" Gespreksontwerp
toepassen"
Fase 2" Welke informatie
zoekt de gebruiker?"
Hoe halen we
Niet dezelfde
Fase 1"
drempels weg?" De stap van dialoog voor
Yeah, we kunnen
pagina elke bezoeker"
testen!" Willen begrijpen optimalisatie
hoe je de naar user flow
bezoeker beter optimalisatie"
Waar kunnen
we nu direct het kunt servicen"
Maak niet uit Personaliseren
wat we testen verschil maken?"
als we maar Mo<veren
testen!"
Leren
Direct
resultaat
Laaghangend
fruit
@tonwesseling
@onlinedialogue
Yo u s h o u l d s t a r t a t l e v e l 1 b u t m o v e f a s t t o l e v e l 3

34. SAME N mod el
@tonwesseling
@onlinedialogue
L e t ’s d o i t t o g e t h e r

35. SAME N mod el
S" A" M" E" N"
Start reason" Analyze" Make" Evaluate" Notify"
@tonwesseling
@onlinedialogue
Te l l t h e s t o r y “ n a v e r t e l l e n ”

36. “Without an analyzed reason
to start, it does not make
any sense to start at all.”!
@tonwesseling
@onlinedialogue
Start reason

37. “Do you
understand what
is happening?”!
@tonwesseling
@onlinedialogue
Analyze

38. “You don’t make perfect
designs in 1 time.!
!
Step by step we’ll grow
to the final result.”!
!
Hypotheses based design!!
@tonwesseling
@onlinedialogue
Make

39. “Not only the
conversion,
look for
insights!”!
@tonwesseling
@onlinedialogue
Evaluate

40. “If someone can not
reproduce your learnings… !
Then, there were no
learnings...”!
@tonwesseling
@onlinedialogue
Notify (Navertellen)

41. Tools!
@tonwesseling
@onlinedialogue
Everything is already here as SaaS

42. Tools!
@tonwesseling
@onlinedialogue
The tools we use these days

43. Start reason
Gebruikers Onderzoek (SurveyMonkey), Gebruikers wensen (Uservoice),
Continue gebruikers feedback (GetSatisfaction, Kampyle), Analytics
trendbreuk notificaties (analytics pakket), Benchmark vergelijken
(Trendspot), Markpositie veranderd (Mozenda / Yahoo Pipes), Social
(Finchline, Momentum), Intern (Basecamp / Yammer). "
@tonwesseling
@onlinedialogue
To o l s

44. Start reason
@tonwesseling
@onlinedialogue
Surveymonkey

45. Start reason
@tonwesseling
@onlinedialogue
Getsatisfaction.com

46. Start reason
@tonwesseling
@onlinedialogue
Kampyle.com

47. Start reason
@tonwesseling
@onlinedialogue
Tr e s h o l d s ! ( G o o g l e A n a l y t i c s )

48. Start reason
@tonwesseling
@onlinedialogue
Tr e n d s p o t ( B e n c h m a r k i n g )

49. Start reason
@tonwesseling
@onlinedialogue
Mozenda.com: competitve information

50. Start reason
@tonwesseling
@onlinedialogue
Momentum (social insights)

51. Start reason
@tonwesseling
@onlinedialogue
I n t e r n a l i n s i g h t s ( Ya m m e r. c o m )

52. Analyze
Analytics pakket, Mousemaps en Screen recordings (Clicktale), Clickmaps
(Crazyegg, Clicktale), Survey (SurveyMonkey), tijdelijke gebruikersfeedback
(Qualaroo, 4Q en eigen tool (met Google Analytics)), Usability feedback
(Feedbackarmy.com, Usertesting.com, Conceptfeedback.com), Interne
feedback (notableapp.com)."
@tonwesseling
@onlinedialogue
To o l s

53. Analyze
@tonwesseling
@onlinedialogue
Get your tag management right! (Qubit open tag)

54. Analyze
@tonwesseling
@onlinedialogue
Clicktale.com

55. Analyze
@tonwesseling
@onlinedialogue
Qualaroo.com

56. Analyze
@tonwesseling
@onlinedialogue
Usertesting.com

57. Analyze
@tonwesseling
@onlinedialogue
Notableapp.com

58. Make
Wire-Frames (Mockflow.com), Concept testing (Verifyapp.com), Clickable
demo’s / usability (Soldifyapp.com), Usability feedback (Feedbackarmy
Usertesting, Conceptfeedback, usability test), Interne feedback
(Notableapp.com), Test bouwen / uitvoeren (via CMS & script of VWO of
Optimizely), Targeting (Personyze, Conductrics)."
@tonwesseling
@onlinedialogue
To o l s

59. Make
@tonwesseling
@onlinedialogue
M o c k fl o w. c o m

60. Make
@tonwesseling
@onlinedialogue
Ve r i f y a p p . c o m

61. Make
@tonwesseling
@onlinedialogue
Soldifyapp.com

62. Make
@tonwesseling
@onlinedialogue
V i s u a l w e b s i t e o p t i m i z e r. c o m ( t e s t i n g )

63. Make
@tonwesseling
@onlinedialogue
Personyze.com (targeting)

64. Make
@tonwesseling
@onlinedialogue
Conductrics.com (RTD)

65. E valuate
Analytics pakket (segmenten), Mousemaps en Screen recordings (Clicktale),
Clickmaps (Crazyegg, Clicktale), Survey voor NPS / CES (SurveyMonkey)."
@tonwesseling
@onlinedialogue
To o l s

66. E valuate
@tonwesseling
@onlinedialogue
Dive in that data!

67. E valuate
@tonwesseling
@onlinedialogue
Don’t forget the why!

68. E valuate
@tonwesseling
@onlinedialogue
Measure lifetime value indicators

69. Notify
Learnings bord, Learnings boekje, Powerpoint single slide learning (voor
mail, maar ook voor op TV’s), Gedrags persona’s / Huisstijl update, Test /
learnings archief (spreadsheet), intern (intranet, yammer, basecamp) en
externe publicaties."
@tonwesseling
@onlinedialogue
To o l s

70. Notify
Publish & Promote
your insights!
@tonwesseling
@onlinedialogue
Spread knowledge

71. Test tips
“ Not every test
will improve conversion ”
@tonwesseling
@onlinedialogue
But you will learn from every test!

72. Test tips
“ You will learn
you should target ”
@tonwesseling
@onlinedialogue
Move on and you’ll see

73. Psychology
@tonwesseling
@onlinedialogue
Wheel of persuasion

74. Test tips
@tonwesseling
@onlinedialogue
Abnamro.nl: Landing pages

75. Test tips
@tonwesseling
@onlinedialogue
Californiaclosets.com: Landing pages

76. Test tips
@tonwesseling
@onlinedialogue
Perrysport.nl – landing page messaging

77. Test tips
@tonwesseling
@onlinedialogue
Alphaecological.com: Form location

78. Test tips
@tonwesseling
@onlinedialogue
Jules.fr: Shop photos

79. Test tips
@tonwesseling
@onlinedialogue
Kras.nl: Footer

80. Test tips
@tonwesseling
@onlinedialogue
Wehkamp.nl: Princing

81. Test tips
@tonwesseling
@onlinedialogue
Kickz.com: price focus?

82. Test tips
@tonwesseling
@onlinedialogue
D e s k a w a y. c o m – “ u g l y b r o t h e r ” p r o d u c t

83. Test tips
@tonwesseling
@onlinedialogue
CDFnetworks.com – Form lay-out

84. Test tips
@tonwesseling
@onlinedialogue
G s m s t u n t e r. n l – i n t e r n a l s e a r c h

85. Test tips
@tonwesseling
@onlinedialogue
Bijenkorf.nl: point-of-actions

86. Test tips
@tonwesseling
@onlinedialogue
A c t i e v a n d e d a g . n l : C TA m o b i l e p r o o f

87. Test tips
@tonwesseling
@onlinedialogue
Basecamphq.com – next action

88. Test tips
@tonwesseling
@onlinedialogue
Kras.nl - checkmarks

89. Test tips
@tonwesseling
@onlinedialogue
VGZ.nl - checkmarks

90. Test tips
@tonwesseling
@onlinedialogue
Hotelschiphol.nl - checkmarks

91. Test tips
@tonwesseling
@onlinedialogue
T h e c o r k s c r e w. i e - d i s c o u n t

92. Test tips
@tonwesseling
@onlinedialogue
Bol.com – like positioning

93. Test tips
@tonwesseling
@onlinedialogue
Inshared.nl – Sticky content

94. To d o:
 Optimize
your dialogue
 Focus on quality
 Spread your knowledge
@tonwesseling
@onlinedialogue
Thank you! Slides: http://ondi.me/wac2013