Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Test for business growth - analyzing A/B-test with a Bayesian approach

Keynote from Annemarie Klaassen of Online Dialogue at SuperWeek Jamaica, May 10th 2016

  • Login to see the comments

Test for business growth - analyzing A/B-test with a Bayesian approach

  1. 1. SUBTITLE BELOW
  2. 2. @AM_Klaassen A bit about me…
  3. 3. @AM_Klaassen
  4. 4. What I do…
  5. 5. @AM_Klaassen My lovely colleagues
  6. 6. @AM_Klaassen Conversion rate optimization Analytics Psychology
  7. 7. @AM_Klaassen
  8. 8. Lots of A/B-tests
  9. 9. @AM_Klaassen Adding direct value Learning user behavior
  10. 10. The challenge of a successful A/B-test program
  11. 11. You need at least 1 winner in 2 weeks
  12. 12. Low energy in test team
  13. 13. Low visibility in the organization
  14. 14. A/B-test program dies
  15. 15. We have 1 in 4 significant winners
  16. 16. Most only 1 in 8
  17. 17. So you need 4 tests per week
  18. 18. You need high traffic volumes
  19. 19. So, we need more winners!
  20. 20. 2 Solutions
  21. 21. 1. Improve your test program
  22. 22. @AM_Klaassen Improve your implementation rate
  23. 23. @AM_Klaassen Improve your implementation rate
  24. 24. @AM_Klaassen Improve your implementation rate
  25. 25. 2. Redefine your winners
  26. 26. Challenges of Frequentist statistics
  27. 27. @AM_Klaassen Challenge 1: Hard to Understand In a frequentist test you state a null hypothesis: H0 = Variation A and B have the same conversion rate
  28. 28. @AM_Klaassen Say for example you did an experiment and the p-value of that test was 0.01. Challenge 1: Hard to Understand http://onlinedialogue.com/abtest-visualization-excel/
  29. 29. @AM_Klaassen Which statement about the p-value (p=0.01) is true? a) You have absolutely disproved the null hypothesis: that is, there is no difference between the variations b) There is a 1% chance of observing a difference as large as you observed even if the two means are identical c) There’s a 99% chance that B is better than A Challenge 1: Hard to Understand
  30. 30. @AM_Klaassen Which statement about the p-value is true? a) You have absolutely disproved the null hypothesis: that is, there is no difference between the variations b) There is a 1% chance of observing a difference as large as you observed even if the two means are identical c) There’s a 99% chance that B is better than A Challenge 1: Hard to Understand
  31. 31. @AM_Klaassen Which statement about the p-value is true? a) You have absolutely disproved the null hypothesis: that is, there is no difference between the variations b) There is a 1% chance of observing a difference as large as you observed even if the two means are identical c) There’s a 99% chance that B is better than A Challenge 1: Hard to Understand
  32. 32. @AM_Klaassen So the p-value only tells you: How unlikely is it that you found this result, given that the null hypothesis is true (that there is no difference between the conversion rates) Challenge 1: Hard to Understand
  33. 33. @AM_Klaassen Confused HiPPO
  34. 34. @AM_Klaassen Challenge 2: Focus on finding proof
  35. 35. @AM_Klaassen Challenge 2: Focus on finding proof
  36. 36. @AM_Klaassen Challenge 2: Focus on finding proof
  37. 37. @AM_Klaassen What’s the alternative? Frequentist statistics Bayesian statistics
  38. 38. Advantages of Bayesian statistics
  39. 39. @AM_Klaassen 1. No statistical terminology involved 2. Answers the question directly: ‘what is the probability that variation B is better than A’ Advantage 1: Easy to understand
  40. 40. @AM_Klaassen Advantage 1: Easy to understand
  41. 41. @AM_Klaassen Remember…?
  42. 42. @AM_Klaassen Advantage 1: Easy to understand
  43. 43. Happy HiPPO
  44. 44. @AM_Klaassen A test result is the probability that B outperforms A: ranging from 0% - 100% Adv 2: Focus on risk assessment
  45. 45. @AM_Klaassen Adv 2: Focus on risk assessment 11% 89% Download PDF: ondi.me/change
  46. 46. Depends on the cost
  47. 47. @AM_Klaassen Take the cost into account
  48. 48. @AM_Klaassen Take the cost into account
  49. 49. @AM_Klaassen Ondi.me/bayes/
  50. 50. @AM_Klaassen Make a risk assessment IMPLEMENT B PROBABILITY Expected risk 10.9% Expected uplift 89.1% Contribution
  51. 51. @AM_Klaassen Make a risk assessment
  52. 52. @AM_Klaassen Make a risk assessment IMPLEMENT B PROBABILITY AVERAGE DROP/UPLIFT Expected risk 10.9% -1.85% Expected uplift 89.1% 5.92% Contribution
  53. 53. @AM_Klaassen Make a risk assessment IMPLEMENT B PROBABILITY AVERAGE DROP/UPLIFT * EFFECT ON REVENU Expected risk 10.9% -1.85% - $ 115,220 Expected uplift 89.1% 5.92% $ 370,700 Contribution $ 317,936 * Based on 6 months and an average order value of € 100
  54. 54. @AM_Klaassen Calculate the ROI IMPLEMENT B BUSINESS CASE Contribution $ 317,936
  55. 55. @AM_Klaassen Calculate the ROI IMPLEMENT B BUSINESS CASE Contribution $ 317,936 Margin (20%) $ 63,587 Cost of implementation $ 15,000
  56. 56. @AM_Klaassen Calculate the ROI IMPLEMENT B BUSINESS CASE Contribution $ 317,936 Margin (20%) $ 63,587 Cost of implementation $ 15,000 ROI 424%
  57. 57. @AM_Klaassen Or the payback period IMPLEMENT B BUSINESS CASE Average CR change 5.00%
  58. 58. @AM_Klaassen Or the payback period IMPLEMENT B BUSINESS CASE Average CR change 5.00% Extra margin per week $ 2,400 Cost of implementation $ 15,000
  59. 59. @AM_Klaassen Or the payback period IMPLEMENT B BUSINESS CASE Average CR change 5.00% Extra margin per week $ 2,400 Cost of implementation $ 15,000 Payback period 6.25 weeks
  60. 60. We still need the scientist
  61. 61. @AM_Klaassen Adding direct value Learning user behavior
  62. 62. @AM_Klaassen The cut-off probability for implementation is not the same as the cut-off probability for a learning CHANCE LEARNING? < 70 % No learning 70 – 85 % Indication – need retest to confirm 85 – 95 % Strong indication – need follow-up test to confirm > 95 % Learning We still need the scientist!
  63. 63. Comparison
  64. 64. @AM_Klaassen Comparison both methods • 50 A/B-tests, • 50.000 visitors per variation, • conversion rate of 2%, • average order value of $100, • minimum contribution of $150,000 in 6 months time • (equivalent to $30,000 extra margin : ROI of 200%)
  65. 65. @AM_Klaassen Comparison both methods
  66. 66. @AM_Klaassen Comparison both methods
  67. 67. @AM_Klaassen Comparison both methods FREQUENTIST BAYESIAN Implementations 10 29
  68. 68. @AM_Klaassen Comparison both methods FREQUENTIST BAYESIAN Implementations 10 29 Expected uplift $ 4,682,600 $11,068,800 Expected risk $ 234,130 $ 2,984,800
  69. 69. @AM_Klaassen Comparison both methods FREQUENTIST BAYESIAN Implementations 10 29 Expected uplift $ 4,682,600 $11,068,800 Expected risk $ 234,130 $ 2,984,800 Risk % 5% 27% Contribution $4,448,470 $9,757,489 Margin (20%) $ 889,974 $1,951,498
  70. 70. @AM_Klaassen Maximize margin $
  71. 71. Implementation rate Higher
  72. 72. Revenue and margin Maximize
  73. 73. Happy HiPPO
  74. 74. Higher in test teamenergy
  75. 75. Higher in the organization visibility
  76. 76. A/B-test programSuccessful
  77. 77. THANK YOU! Download PDF: ondi.me/change Bayesian calculator: ondi.me/bayes Slide deck: ondi.me/annemarie @AM_Klaassen annemarie@onlinedialogue.com nl.linkedin.com/in/amklaassen

×