Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

4,774 views

Published on

A/B testing is a great way to really learn from user behavior, but this is mostly not the main objective of an optimization program. It needs to improve revenue! If you're only implementing significant results, you're leaving a lot of money on the table. We will explain how we run optimization at the biggest hotel chain in the Netherlands.

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

[Elite Camp 2016] Annemarie Klaassen & Tom van de Berg - Moving Beyond Testing for Absolute Truths

  1. 1. SUBTITLE BELOW Optimize for €€€ HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS
  2. 2. @AM_Klaassen / @tomvdberg1 How we got here…
  3. 3. @AM_Klaassen / @tomvdberg1 The quest for female speakers…
  4. 4. @AM_Klaassen / @tomvdberg1 Female speakers… ?
  5. 5. @AM_Klaassen / @tomvdberg1 Our lovely colleagues
  6. 6. What we do…
  7. 7. @AM_Klaassen / @tomvdberg1 Conversion rate optimization Analytics Psychology
  8. 8. Lots of A/B-tests
  9. 9. @AM_Klaassen / @tomvdberg1 Our clients
  10. 10. @AM_Klaassen / @tomvdberg1 Adding direct value Learning user behavior
  11. 11. @AM_Klaassen / @tomvdberg1
  12. 12. @AM_Klaassen / @tomvdberg1
  13. 13. @AM_Klaassen / @tomvdberg1 Van der Valk  Unique website for each hotel  1 centralized team  Paid by % of their turnover
  14. 14. @AM_Klaassen / @tomvdberg1 Optimization of Valk  Increase revenue through CRO  Reduce costs per experiment
  15. 15. “We’re not in the business of science, we’re in the business of making money”
  16. 16. 3 Tactics
  17. 17. 1. Free test software
  18. 18. @AM_Klaassen / @tomvdberg1 Jorrin Quest GTM testing http://gtmtesting.com
  19. 19. @AM_Klaassen / @tomvdberg1 GTM testing  Easy setup  Free  Preview mode
  20. 20. 2. Change of statistics
  21. 21. @AM_Klaassen / @tomvdberg1 Focus on finding proof
  22. 22. @AM_Klaassen / @tomvdberg1 Example test: Order flow
  23. 23. @AM_Klaassen / @tomvdberg1 Example test: Set completion A B
  24. 24. @AM_Klaassen / @tomvdberg1 Test result
  25. 25. @AM_Klaassen / @tomvdberg1 • No effect on conversion  Don’t implement  Re-test with higher volumes Conclusion
  26. 26. @AM_Klaassen / @tomvdberg1
  27. 27. @AM_Klaassen / @tomvdberg1 What’s the alternative? Frequentist statistics Bayesian statistics
  28. 28. @AM_Klaassen / @tomvdberg1 Example test: Set completion A B
  29. 29. @AM_Klaassen / @tomvdberg1 Bayesian Test evaluation
  30. 30. @AM_Klaassen / @tomvdberg1 Focus on risk assessment 88,4% A test result is the probability that B outperforms A: ranging from 0% - 100%
  31. 31. @AM_Klaassen / @tomvdberg1 IMPLEMENT B PROBABILITY * EFFECT ON REVENU Expected risk 11,6% - € 69.791 Expected uplift 88,4% € 213.530 Contribution € 180.733 * Based on 6 months and an average order value of € 175 Make a risk assessment
  32. 32. @AM_Klaassen / @tomvdberg1 Implement B B
  33. 33. @AM_Klaassen / @tomvdberg1 abtestguide.com/bayesian/ Roy Schieving Annemarie Klaassen
  34. 34. @AM_Klaassen / @tomvdberg1 Conclusion Bayesian  Easier to understand and communicate  Better suits the business  Don’t throw away good ideas (indicatively significant)  Higher implementation rate and revenue € 0 € 1.000.000 € 2.000.000 € 3.000.000 € 4.000.000 € 5.000.000 € 6.000.000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 Expected impact on revenue in 1 year € Frequentist € Bayesian
  35. 35. 3. Change of test method
  36. 36. @AM_Klaassen / @tomvdberg1 50% 50% A/B-test
  37. 37. @AM_Klaassen / @tomvdberg1 Traffic allocation over time A B Traffic Time Profit or loss?
  38. 38. @AM_Klaassen / @tomvdberg1 What’s the alternative? A/B-test Bandits
  39. 39. If you have multiple options: which bandit will pay-out the most?
  40. 40. @AM_Klaassen / @tomvdberg1 Explore Exploit ? Explore / Exploit dilemma
  41. 41. @AM_Klaassen / @tomvdberg1 Bandit 100%
  42. 42. @AM_Klaassen / @tomvdberg1 Bandit 80% 20% 10% 90%
  43. 43. @AM_Klaassen / @tomvdberg1 Traffic allocation over time A B Traffic Time More Profit!
  44. 44. @AM_Klaassen / @tomvdberg1 Limited time frame Automation When use Bandits? Earn over learn
  45. 45. How?
  46. 46. @AM_Klaassen / @tomvdberg1 Example 1
  47. 47. @AM_Klaassen / @tomvdberg1 Example 1  Older target group  Comparing price on other sites  Behavior of others
  48. 48. @AM_Klaassen / @tomvdberg1 Example 1 Control No message Variation B Copy default Variaton C Variation D
  49. 49. 11,1% 10,6% 10,4% 10,5%
  50. 50. @AM_Klaassen / @tomvdberg1 Example 1 – Effect on revenue A B Traffic Time + € 6.637
  51. 51. @AM_Klaassen / @tomvdberg1 Example 2 Control New control Variation B Copy control Variation C Variation D
  52. 52. 9,2% 8,9% 7,8% 8,4%
  53. 53. @AM_Klaassen / @tomvdberg1 Example 2 – Effect on revenue A B Traffic Time + € 19.862
  54. 54. @AM_Klaassen / @tomvdberg1 Conclusion Bandits  Less regret during testing  No implementation costs  Optimizing per user segment
  55. 55. 1. Free test software 2. Bayesian statistics 3. Bandit testing
  56. 56. revenue Maximize
  57. 57. @AM_Klaassen / @tomvdberg1
  58. 58. “We’re not in the business of science, we’re in the business of making money”
  59. 59. TÄNAN VÄGA! Bayesian calculator: abtestguide.com/bayesian/ Slide deck: ondi.me/elitecamp2016 @AM_Klaassen annemarie@onlinedialogue.com nl.linkedin.com/in/amklaassen @tomvdberg1 tom@onlinedialogue.com nl.linkedin.com/in/vandenbergtom

×