Slides of my ROAR keynote at #elitecamp2015 from Peep Laja / ConversionXL in Estonia. I spoke about Risk, +Optimization, +Automation and Re-invent and explained how http://testing.agency can help you getting things done.

1. D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# E l i t e C a m p - @ To n W
When to run more experiments?

2. Ton Wesseling | @ TonW
Run more experiments #TonW
http://testing.agency

3. Ton Wesseling | @ TonW
Run more experiments #TonW
Today
the
ROAR
model

4. Ton Wesseling | @ TonW
Run more experiments #TonW
Methods used to improve conversion

5. Ton Wesseling | @ TonW
Run more experiments #TonW
Methods companies plan to use

6. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing is
HOT

7. Ton Wesseling | @ TonW
Run more experiments #TonW
Everyone is pushing to test more

8. Ton Wesseling | @ TonW
Run more experiments #TonW
Never satisfied with conversion rates

9. Ton Wesseling | @ TonW
Run more experiments #TonW
Start testing or test more

10. Ton Wesseling | @ TonW
Run more experiments #TonW
NO
Growth problems will dissapear…

11. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing is not a solution
It’s a methodology

12. Ton Wesseling | @ TonW
Run more experiments #TonW
It’s part of a:
Continuous
Optimization
Program

13. Ton Wesseling | @ TonW
Run more experiments #TonW
And you need to know:
When
How
What

14. Ton Wesseling | @ TonW
Run more experiments #TonW
Growth curve – how it was teached

15. Ton Wesseling | @ TonW
Run more experiments #TonW
the
ROAR
model

16. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!

17. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!

18. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!

19. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk!

20. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization!

21. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation!

22. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!

23. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!

24. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!

25. Ton Wesseling | @ TonW
Run more experiments #TonW
Statistics

26. Ton Wesseling | @ TonW
Run more experiments #TonW
Optimization phases - ROAR
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
10.000 conversions
per month!
1.000 conversions!
per month!
15% impact
needed!
5% impact!
needed!
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!

27. Ton Wesseling | @ TonW
Run more experiments #TonW
Low number testing?
Why can it hurt you?

28. Ton Wesseling | @ TonW
Run more experiments #TonW
Warning: low significance levels
ü Big chance the winner is not a real winner

29. Ton Wesseling | @ TonW
Run more experiments #TonW
Warning: low power levels
ü Big chance a real winner is not recognized

30. Ton Wesseling | @ TonW
Run more experiments #TonW
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
ü …with an average uplift of 10%

31. Ton Wesseling | @ TonW
Run more experiments #TonW
Average: 75% significance & 40% power

32. Ton Wesseling | @ TonW
Run more experiments #TonW
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized(40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+

33. Ton Wesseling | @ TonW
Run more experiments #TonW
Yes, I’ve got those 1000+ conversions
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

34. Ton Wesseling | @ TonW
Run more experiments #TonW
With 1000+ conversions a month
Your test capacity is max.
20+ tests a year
with a statistical power of 80%
on predicted uplifts of 15%

35. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests with measureable impact
On average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact

36. Ton Wesseling | @ TonW
Run more experiments #TonW
So in this case, you’ve got impact
Every 7 or 8 weeks

37. Ton Wesseling | @ TonW
Run more experiments #TonW
Slower? Energy will run out!
And your continous optimization program will die

38. Ton Wesseling | @ TonW
Run more experiments #TonW
What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?!

39. Ton Wesseling | @ TonW
Run more experiments #TonW
What to test? Business cases!
Service charge, do
it or not? Include in
price or not?!

40. Ton Wesseling | @ TonW
Run more experiments #TonW
What to test? Big design changes
Yes this can be done with an A/B-test tool, you only need A/B-test coders!

41. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

42. Ton Wesseling | @ TonW
Run more experiments #TonW
Finally you can test details…
Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!

43. Ton Wesseling | @ TonW
Run more experiments #TonW
You should grow to
Customer Buying Reasons

44. Ton Wesseling | @ TonW
Run more experiments #TonW
Reasons behind the customer journey

45. Ton Wesseling | @ TonW
Run more experiments #TonW
A continuous optimization team
Analyst
Psychologyst
Designer
Developer
Team Lead

46. Ton Wesseling | @ TonW
Run more experiments #TonW
They keep on asking:
Why are these users here?

47. Ton Wesseling | @ TonW
Run more experiments #TonW
And they want to know
How they can
help these users’ brains
to fullfil their needs?

48. Ton Wesseling | @ TonW
Run more experiments #TonW
Data & Psychology
Digital Data
Persuasion Psychology
Analyze
&
Experiment

49. Ton Wesseling | @ TonW
Run more experiments #TonW
A lack of resources

50. Ton Wesseling | @ TonW
Run more experiments #TonW
Testing takes resources
Use it to LEARN

51. Ton Wesseling | @ TonW
Run more experiments #TonW
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find

52. Ton Wesseling | @ TonW
Run more experiments #TonW
Know the company
Mission, Vision,
Stategy & Goals

53. Ton Wesseling | @ TonW
Run more experiments #TonW
Know your user (journeys)

54. Ton Wesseling | @ TonW
Run more experiments #TonW
Based on: asking – why are you here?

55. Ton Wesseling | @ TonW
Run more experiments #TonW
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer!

56. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation

57. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
Need help with coding A/B-test variations like this one? – http://testing.agency!

58. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation

59. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation

60. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation
Need help with coding A/B-test variations like this one? – http://testing.agency!

61. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A & B variation

62. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation

63. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation

64. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: A variation

65. Ton Wesseling | @ TonW
Run more experiments #TonW
Tests that can be done: B variation

66. Ton Wesseling | @ TonW
Run more experiments #TonW
FA C T & A C T
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
Analyze
Your analyst will analyze the results!

67. Ton Wesseling | @ TonW
Run more experiments #TonW
We measure in Analytics software

68. Ton Wesseling | @ TonW
Run more experiments #TonW
Calculate results: ABTestGuide.com/calc

69. Ton Wesseling | @ TonW
Run more experiments #TonW
Not significant
ü It’s not a winner, stop
ü But, it does not mean it’s a losing one!
ü Average: 33% of your tests will have winners
If you are optimizing dialogues!

70. Ton Wesseling | @ TonW
Run more experiments #TonW
Choose what to analyze up front
ü You just go and dig until you find something
(you will always find something that seems true BUT is not!)

71. Ton Wesseling | @ TonW
Run more experiments #TonW
Don’t stop the test once it’s significant!

72. Ton Wesseling | @ TonW
Run more experiments #TonW
You just let it run until it’s significant

73. Ton Wesseling | @ TonW
Run more experiments #TonW
You just let it run until it’s significant
Your test has a fixed lenght!
It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!

74. Ton Wesseling | @ TonW
Run more experiments #TonW
Be aware: Your test takes too long
ü Cookie deletion
(you will lose 10% in two weeks)
They will re-enter the and pollute your samples!

75. Ton Wesseling | @ TonW
Run more experiments #TonW
Test length: ABTestGuide.com/calc

76. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: Don’t test broken stuff
ü Always check browser compatibility!
ü Don’t break dynamic stuff!

77. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: Slow down the control down
ü Add the same code to the control

78. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: leave out s o m e visitors
ü Only test on those who can be changed
(if you test a sales promotion – leave out your current clients who just logon)

79. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: Only test on fresh people
ü Start with control for a full purchase cycle
(and then start sending traffic to your variations – setting the control to 0%)

80. Ton Wesseling | @ TonW
Run more experiments #TonW
Tip: give them time to buy!
ü Stop getting fresh visitors after your test time
(send new traffic to the control, but give you tested people time to finish converting)

81. Ton Wesseling | @ TonW
Run more experiments #TonW
+ Optimization! + Automation!
Back to the ROAR model
Time span!
Conversionspermonth!
Risk! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

82. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

83. Ton Wesseling | @ TonW
Run more experiments #TonW
Test capacity
maximum number of
experiments per year
that your site or app
can handle

84. Ton Wesseling | @ TonW
Run more experiments #TonW
Regret
amount of money
you are losing per year by
not using your full test capacity

85. Ton Wesseling | @ TonW
Run more experiments #TonW
Calculate your test capicity
http://ondi.me/size!

86. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Automation
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

87. Ton Wesseling | @ TonW
Run more experiments #TonW
Automation

88. Ton Wesseling | @ TonW
Run more experiments #TonW
You tested which locations are critical

89. Ton Wesseling | @ TonW
Run more experiments #TonW
You know which dialogues are critical

90. Ton Wesseling | @ TonW
Run more experiments #TonW
Automation
Contextual bandits
If you are ready – give me a call!!

91. Ton Wesseling | @ TonW
Run more experiments #TonW
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!

92. Ton Wesseling | @ TonW
Run more experiments #TonW
+ Optimization! + Automation!
ROAR – Automation
Time span!
Conversionspermonth!
Risk! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

93. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – You don’t stop optimizing
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

94. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Maybe you’ll never automate
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

95. Ton Wesseling | @ TonW
Run more experiments #TonW
ROAR – Re-think
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!

96. Ton Wesseling | @ TonW
Run more experiments #TonW
The ultimate Re-think resource

97. Ton Wesseling | @ TonW
Run more experiments #TonW
CRO brings user knowledge

98. Ton Wesseling | @ TonW
Run more experiments #TonW
You will make big impact on Discover

99. Ton Wesseling | @ TonW
Run more experiments #TonW
Customer knowledge center
Making money from conversion improvements
Creating more and more customer insights
Which can be monetized
on even a larger scale

100. Ton Wesseling | @ TonW
Run more experiments #TonW
Now, that’s
HOT

101. Ton Wesseling | @ TonW
Run more experiments #TonW
Done testing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!

102. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing
Don’t start to soon

103. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing
Test the right way

104. Ton Wesseling | @ TonW
Run more experiments #TonW
A/B-testing
Minimize regret
Run more experiments, maximize your capacity!

105. Ton Wesseling | @ TonW
Run more experiments #TonW
http://testing.agency

106. D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# E l i t e C a m p - @ To n W
When to run more experiments?