Ton Wesseling presentation at Digital Marketing Conference – iLive 2015, Riga, Latvia iLive conference is the place to be for actionable sessions on SEO, Google analytics, social media, community building and brand development in the online environment. Follow us on Facebook & Twitter & Instagram: Facebook: http://www.facebook.com/iLivelv Twitter: http://www.twitter.com/iLivelv Instagram: http://www.instagram.com/iliveconference

1. i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# i L i v e 2 0 1 5 - @ To n W
Run more experiments

2. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Thank you for having me

3. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
On this stage

4. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Visiting Riga

5. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
On november 11th

6. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
http://testing.agency

7. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Meeting you all

8. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Conversion people

9. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Always fun

10. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
When we start cooking

11. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Open minded, outgoing

12. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
High energy & fun

13. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
We are
Evidence Based

14. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And
LEAN

15. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
It has been a long day

16. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What about your energy?

17. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Value for your time and money

18. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Reboost your energy

19. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Please stand up

20. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Raise your hands

21. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Raise your hands

22. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
High energy conversion fun

23. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Calm down and focus

24. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Why this song?
ROAR

25. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Because of Katy Perry?

26. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
No, it’s the name of the song!

27. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Today
the
ROAR
model

28. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
20 years ago

29. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I learned about him

30. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Quality circle – the start of LEAN

31. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
LEAN

32. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
So I looked at data

33. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Made some changes

34. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And measured if it was successful

35. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And measured if it was successful

36. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Why can this lead to failures?

37. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Average conversion rate looks stable

38. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
But in fact

39. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
There is a big weekly change!

40. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I only did small improvements

41. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
So I made many wrong decisions
26 out of 44 weeks: over 5% change!

42. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Then I learned about this story…

43. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Bill Murray

44. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Trying to get the girl

45. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Constant testing environment
A/B-Testing

46. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5

47. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Methods used to improve conversion

48. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Methods companies plan to use

49. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-testing is
HOT

50. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-testing is
HOT

51. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Everyone is pushing to test more

52. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Like Amazon, Like ZalandoGermanEcommercemarket,thankyouAndréMorys

53. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon

54. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Never satisfied with conversion rates

55. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Start testing or test more

56. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
NO
Growth problems will disappear…

57. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-testing is not a solution
It’s a methodology

58. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
It’s part of a:
Continuous
Optimization
Program

59. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
It’s a way of working
It’s company DNA

60. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You need to know:
When
How
What

61. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Growth curve – how it was teached

62. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
the
ROAR
model

63. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth

64. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth

65. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth

66. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk

67. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization

68. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization + Automation

69. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think

70. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month

71. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
10.000 conversions
per month
1.000 conversions
per month

72. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
How much impact do you need?
http://ondi.me/size

73. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
10.000 conversions
per month
1.000 conversions
per month
15% impact
needed
5% impact
needed
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%

74. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimization phases - ROAR
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month

75. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Phase 1: bridge the gap – Risk

76. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Phase 1: bridge the gap – Risk

77. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Get out of the office

78. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Risk

79. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Yes, I’ve got those 1000+ conversions
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month

80. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
With 1000+ conversions a month
Your test capacity is max.
20+ tests a year
with a statistical power of 80%
on predicted uplifts of 15%

81. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests with measurable impact
On average
1 out of 3
Simple truth: you’re just not always able to create a winner with enough impact

82. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
So in this case, you’ve got impact
Every 7 or 8 weeks

83. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Slower? Energy will run out!
And your continous optimization program will die

84. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?

85. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What to test? Business cases!
Service charge, do
it or not? Include in
price or not?

86. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What to test? Big design changes

87. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Optimization – moving up
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month

88. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Finally you can test details…
Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!

89. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You should grow to
Customer Buying Reasons

90. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Reasons behind the customer journey

91. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A continuous optimization team
Analyst Psychologyst
Designer Developer
Team Lead

92. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
They keep on asking:
Why are these users here?

93. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And they want to know
How they can
help their users’ brains
to fullfil their needs?

94. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Data & Psychology
Digital Data Persuasion Psychology
Analyze
&
Experiment

95. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A lack of resources

96. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Testing takes resources
Use it to LEARN

97. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind

98. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A n d t h e n s t a r t : FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFindFind
Your optimization team needs a good analyst / researcher

99. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Many sources out there to Find

100. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind Analyze
Really a good analyst / researcher!

101. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Use your analytics data

102. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Create heatmaps

103. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Create screen recordings

104. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
To find out: The critical spots

105. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer

106. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Source: Psychology

107. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Two systems

108. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Two systems that intervene

109. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
System 1 taking over

110. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And the automated respons rules

111. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Even when it’s wrong

112. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Use that persuasion knowledge

113. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

114. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

115. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

116. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

117. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

118. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

119. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

120. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

121. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

122. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

123. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done

124. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind Test
And your team needs a good front-end developer

125. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Get your A/B-solution in place

126. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You need Front end development

127. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind
Analyze
Your analyst will analyze the results

128. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Look at details

129. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellCombineAnalyze
TestCreateAnalyzeFind
TellConclude
Your psychologyst should conclude and your conversion lead should tell

130. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Store your learnings

131. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Push your behavioral insights

132. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Everyone should use your knowledge!
Data Insights Action

133. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
+ Optimization + Automation
Back to the ROAR model
Time span
Conversionspermonth
Risk Re-think
1.000 conversions
per month
10.000 conversions
per month

134. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Optimization – moving up
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month

135. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now you’re testing
On your way to
4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst

136. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Test capacity
maximum number of
experiments per year
that your site or app
can handle

137. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Regret
amount of money
you are losing per year by
not using your full test capacity

138. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Automation
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month

139. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Automation

140. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You tested which locations are critical

141. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You know which dialogues are critical

142. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Automation

143. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?

144. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
+ Optimization + Automation
ROAR – Automation
Time span
Conversionspermonth
Risk Re-think
1.000 conversions
per month
10.000 conversions
per month

145. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – You don’t stop optimizing
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month

146. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Re-think
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month

147. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-testing is the ultimate resource

148. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon

149. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
CRO brings user knowledge

150. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Store your learnings

151. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You will make big impact on Discover

152. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now, that’s
HOT

153. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now, that’s
HOT

154. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Done testing?

155. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Done testing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge

156. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-testing
Minimize regret

157. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Test as much as you can
Time span
Conversionspermonth
Risk + Optimization + Automation Re-think
10.000 conversions
per month
1.000 conversions
per month
15% impact
needed
5% impact
needed
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%

158. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Like Amazon, Like ZalandoGermanEcommercemarket,thankyouAndréMorys

159. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Experimenting is a way of working
It’s company DNA

160. T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Run more experiments

161. i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# i L i v e 2 0 1 5 - @ To n W
Run more experiments