Keynote by Ton Wesseling - CEO / Founder Testing.Agency at Ilive 2015 Latvia, Riga: Run more experiments! Don't start to soon with A/B-testing, but once you can: scale up! Use your full test capacity and make sure experimenting becomes part of your company DNA. It's high energy, it's fun. In this presentation the ROAR model is explained - the model you can use to know when to run which and how many experiments.

1. i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# i L i v e 2 0 1 5 - @ To n W
Run more experiments

2. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Thank you for having me

3. Ton Wesseling | @ TonW
Run more experiments #ilive2015
On this stage

4. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Visiting Riga

5. Ton Wesseling | @ TonW
Run more experiments #ilive2015
On november 11th

6. Ton Wesseling | @ TonW
Run more experiments #ilive2015
http://testing.agency

7. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Meeting you all

8. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Conversion people

9. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Always fun

10. Ton Wesseling | @ TonW
Run more experiments #ilive2015
When we start cooking

11. Ton Wesseling | @ TonW
Run more experiments #ilive2015
High energy & fun

12. Ton Wesseling | @ TonW
Run more experiments #ilive2015
We are
Evidence Based

13. Ton Wesseling | @ TonW
Run more experiments #ilive2015
And
LEAN

14. Ton Wesseling | @ TonW
Run more experiments #ilive2015
It has been a long day though

15. Ton Wesseling | @ TonW
Run more experiments #ilive2015
What about your energy?

16. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Value for your time and money

17. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Reboost your energy

18. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Please stand up

19. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Raise your hands

20. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Raise your hands

21. Ton Wesseling | @ TonW
Run more experiments #ilive2015
High energy conversion fun

22. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Calm down and focus

23. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Why this song?
ROAR

24. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Because of Katy Perry?

25. Ton Wesseling | @ TonW
Run more experiments #ilive2015
No, it’s the name of the song!

26. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Today
the
ROAR
model

27. Ton Wesseling | @ TonW
Run more experiments #ilive2015
20 years ago

28. Ton Wesseling | @ TonW
Run more experiments #ilive2015
I learned about him

29. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Quality circle – the start of LEAN

30. Ton Wesseling | @ TonW
Run more experiments #ilive2015
LEAN

31. Ton Wesseling | @ TonW
Run more experiments #ilive2015
So I looked at data

32. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Made some changes

33. Ton Wesseling | @ TonW
Run more experiments #ilive2015
And measured if it was successful

34. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Why can this lead to failures?

35. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Average conversion rate looks stable

36. Ton Wesseling | @ TonW
Run more experiments #ilive2015
But in fact

37. Ton Wesseling | @ TonW
Run more experiments #ilive2015
There is a big weekly change!

38. Ton Wesseling | @ TonW
Run more experiments #ilive2015
I only did small improvements

39. Ton Wesseling | @ TonW
Run more experiments #ilive2015
So I made many wrong decisions
26 out of 44 weeks: over 5% change!

40. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Then I learned about this story…

41. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Bill Murray in Groundhog Day

42. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Trying to get the girl

43. Ton Wesseling | @ TonW
Run more experiments #ilive2015
He has a stable testing environment
A/B-Testing

44. Ton Wesseling | @ TonW
Run more experiments #ilive2015

45. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Methods used to improve conversion

46. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Methods companies plan to use

47. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A/B-testing is
HOT

48. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Everyone is pushing to test more

49. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Like Amazon, Like Zalando
GermanEcommercemarket,thankyouAndréMorys

50. Ton Wesseling | @ TonW
Run more experiments #ilive2015
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon

51. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Never satisfied with conversion rates

52. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Start testing or test more

53. Ton Wesseling | @ TonW
Run more experiments #ilive2015
NO
Growth problems will disappear…

54. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A/B-testing is not a solution
It’s a methodology

55. Ton Wesseling | @ TonW
Run more experiments #ilive2015
It’s part of a:
Continuous
Optimization
Program

56. Ton Wesseling | @ TonW
Run more experiments #ilive2015
It’s a way of working
It’s company DNA

57. Ton Wesseling | @ TonW
Run more experiments #ilive2015
You need to know:
When
How
What

58. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Growth curve – how it was teached

59. Ton Wesseling | @ TonW
Run more experiments #ilive2015
the
ROAR
model

60. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!

61. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!

62. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!

63. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk!

64. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization!

65. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation!

66. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!

67. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!

68. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!

69. Ton Wesseling | @ TonW
Run more experiments #ilive2015
How much impact do you need?
http://ondi.me/size!

70. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!
15% impact
needed!
5% impact!
needed!
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!

71. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!

72. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Phase 1: bridge the gap – Risk

73. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Phase 1: bridge the gap – Risk

74. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Get out of the office

75. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Risk

76. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Yes, I’ve got those 1000+ conversions
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

77. Ton Wesseling | @ TonW
Run more experiments #ilive2015
With 1000+ conversions a month
Your test capacity is max.
20+ tests a year
with a statistical power of 80%
on predicted uplifts of 15%

78. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests with measurable impact
On average
1 out of 3
Simple truth: you’re just not always able to create a winner with enough impact

79. Ton Wesseling | @ TonW
Run more experiments #ilive2015
So in this case, you’ve got impact
Every 7 or 8 weeks

80. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Slower? Energy will run out!
And your continous optimization program will die

81. Ton Wesseling | @ TonW
Run more experiments #ilive2015
What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?!

82. Ton Wesseling | @ TonW
Run more experiments #ilive2015
What to test? Business cases!
Service charge, do
it or not? Include in
price or not?!

83. Ton Wesseling | @ TonW
Run more experiments #ilive2015
What to test? Big design changes

84. Ton Wesseling | @ TonW
Run more experiments #ilive2015
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

85. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Finally you can test details…
Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!

86. Ton Wesseling | @ TonW
Run more experiments #ilive2015
You should grow to
Customer Buying Reasons

87. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Reasons behind the customer journey

88. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A continuous optimization team
Analyst
Psychologyst
Designer
Developer
Team Lead

89. Ton Wesseling | @ TonW
Run more experiments #ilive2015
They keep on asking:
Why are these users here?

90. Ton Wesseling | @ TonW
Run more experiments #ilive2015
And they want to know
How they can
help their users’ brains
to fullfil their needs?

91. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Data & Psychology
Digital Data
Persuasion Psychology
Analyze
&
Experiment

92. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A lack of resources

93. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Testing takes resources
Use it to LEARN

94. Ton Wesseling | @ TonW
Run more experiments #ilive2015
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find

95. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A n d t h e n s t a r t : FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Find
Your optimization team needs a good analyst / researcher!

96. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Many sources out there to Find

97. Ton Wesseling | @ TonW
Run more experiments #ilive2015
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Analyze
Really a good analyst / researcher!!

98. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Use your analytics data

99. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Create heatmaps

100. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Create screen recordings

101. Ton Wesseling | @ TonW
Run more experiments #ilive2015
To find out: The critical spots

102. Ton Wesseling | @ TonW
Run more experiments #ilive2015
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer!

103. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Source: Psychology

104. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Two systems

105. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Two systems that intervene

106. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Use that persuasion knowledge
Wheel-of-Persuasion.com
Resource & courses

107. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests that can be done

108. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests that can be done

109. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests that can be done

110. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests that can be done

111. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests that can be done

112. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Tests that can be done

113. Ton Wesseling | @ TonW
Run more experiments #ilive2015
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Test
And your team needs a good front-end developer!

114. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Get your A/B-solution in place

115. Ton Wesseling | @ TonW
Run more experiments #ilive2015
But you need Front-end resources!

116. Ton Wesseling | @ TonW
Run more experiments #ilive2015
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Analyze
Your analyst will analyze the results!

117. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Look at details

118. Ton Wesseling | @ TonW
Run more experiments #ilive2015
FA C T & A C T
&
Tell
Combine
Analyze
Test
Create
Analyze
Find
Tell
Conclude
Your psychologyst should conclude and your conversion lead should tell!

119. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Store your learnings

120. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Push your behavioral insights

121. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Everyone should use your knowledge!
Data Insights Action

122. Ton Wesseling | @ TonW
Run more experiments #ilive2015
+ Optimization! + Automation!
Back to the ROAR model
Time span!
Conversionspermonth!
Risk! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

123. Ton Wesseling | @ TonW
Run more experiments #ilive2015
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

124. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Now you’re testing
On your way to
4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!

125. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Test capacity
maximum number of
experiments per year
that your site or app
can handle

126. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Regret
amount of money
you are losing per year by
not using your full test capacity

127. Ton Wesseling | @ TonW
Run more experiments #ilive2015
ROAR – Automation
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

128. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Automation

129. Ton Wesseling | @ TonW
Run more experiments #ilive2015
You tested which locations are critical

130. Ton Wesseling | @ TonW
Run more experiments #ilive2015
You know which dialogues are critical

131. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Automation

132. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!

133. Ton Wesseling | @ TonW
Run more experiments #ilive2015
+ Optimization! + Automation!
ROAR – Automation
Time span!
Conversionspermonth!
Risk! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

134. Ton Wesseling | @ TonW
Run more experiments #ilive2015
ROAR – You don’t stop optimizing
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

135. Ton Wesseling | @ TonW
Run more experiments #ilive2015
ROAR – Re-think
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

136. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A/B-testing is the ultimate resource

137. Ton Wesseling | @ TonW
Run more experiments #ilive2015
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon

138. Ton Wesseling | @ TonW
Run more experiments #ilive2015
CRO brings user knowledge

139. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Store your learnings

140. Ton Wesseling | @ TonW
Run more experiments #ilive2015
You will make big impact on Discover

141. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Now, that’s
HOT

142. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Done testing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!

143. Ton Wesseling | @ TonW
Run more experiments #ilive2015
A/B-testing
Minimize regret

144. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Test as much as you can
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!
15% impact
needed!
5% impact!
needed!
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!

145. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Like Amazon, Like Zalando
GermanEcommercemarket,thankyouAndréMorys

146. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Enjoy the high energy fun!

147. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Experimenting is a way of working
It’s company DNA

148. Ton Wesseling | @ TonW
Run more experiments #ilive2015
Run more experiments

149. i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# i L i v e 2 0 1 5 - @ To n W
Run more experiments