[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

SUBTITLE BELOW
Optimize for €€€
HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

@AM_Klaassen / @tomvdberg1
How we got here…

The quest for female speakers…

Female speakers…
?

Our lovely colleagues

Conversion rate optimization
Analytics Psychology

Our clients

Adding
direct value
Learning
user behavior

Van der Valk
 Unique website for each hotel
 1 centralized team
 Paid by % of their turnover

Optimization of Valk
 Increase revenue through CRO
 Reduce costs per experiment

“We’re not in the business of
science, we’re in the business
of making money”

Jorrin Quest
GTM testing
http://gtmtesting.com

GTM testing
 Easy setup
 Free
 Preview mode

Focus on finding proof

Example test: Order flow

Example test: Set completion
A B

Test result

• No effect on conversion
 Don’t implement
 Re-test with higher volumes
Conclusion

What’s the alternative?
Frequentist
statistics
Bayesian
statistics

Bayesian Test evaluation

Focus on risk assessment
88,4%
A test result is the probability that B outperforms A:
ranging from 0% - 100%

IMPLEMENT B PROBABILITY * EFFECT ON REVENU
Expected risk 11,6% - € 69.791
Expected uplift 88,4% € 213.530
Contribution € 180.733
* Based on 6 months and an average order value of € 175
Make a risk assessment

Implement B
B

abtestguide.com/bayesian/
Roy Schieving
Annemarie Klaassen

Conclusion Bayesian
 Easier to understand and
communicate
 Better suits the business
 Don’t throw away good ideas
(indicatively significant)
 Higher implementation rate and
revenue
€ 0
€ 1.000.000
€ 2.000.000
€ 3.000.000
€ 4.000.000
€ 5.000.000
€ 6.000.000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Expected impact on revenue in 1 year
€ Frequentist € Bayesian

50%
50%
A/B-test

Traffic allocation over time
A
B
Traffic
Time
Profit or loss?

What’s the alternative?
A/B-test Bandits

If you have multiple options:
which bandit
will pay-out the
most?

Explore Exploit
?
Explore / Exploit dilemma

Bandit
100%

Bandit
80%
20%
10%
90%

Traffic allocation over time
A
B
Traffic
Time
More Profit!

Limited time frame Automation
When use Bandits?
Earn over learn

Example 1

Example 1
 Older target group
 Comparing price on
other sites
 Behavior of others

Example 1
Control No message
Variation B Copy default
Variaton C
Variation D

Example 1 – Effect on revenue
A
B
Traffic
Time
+ € 6.637

Example 2
Control New control
Variation B Copy control
Variation C
Variation D

Example 2 – Effect on revenue
A
B
Traffic
Time
+ € 19.862

Conclusion Bandits
 Less regret during testing
 No implementation costs
 Optimizing per user segment

1. Free test software
2. Bayesian statistics
3. Bandit testing

TÄNAN VÄGA!
Bayesian calculator: abtestguide.com/bayesian/
Slide deck: ondi.me/elitecamp2016
@AM_Klaassen
annemarie@onlinedialogue.com
nl.linkedin.com/in/amklaassen
@tomvdberg1
tom@onlinedialogue.com
nl.linkedin.com/in/vandenbergtom

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

More Related Content

What's hot

Viewers also liked

Similar to [Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

More from CXL

Recently uploaded

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths