Download as PDF, PPTX
Uploaded byTom Capper
Conversion Conference Berlin
This document discusses best practices for interpreting A/B test results and making decisions based on those results. It cautions against relying solely on significance values and p-values, noting that these only indicate the likelihood of the data given the null hypothesis, not the likelihood of the null hypothesis being true. It emphasizes considering confidence intervals, choosing the right metrics like profit or revenue, and weighing the risks of false positives versus missed opportunities when deciding whether to roll out a change. The key takeaways are to incorporate risk into test design, focus on the real goal of increasing profit, and recognize that the stakes are generally lower for online experiments than medical trials.
Conversion Conference Berlin
- 1.
- 2. A Good A/BTest Result: “10% Uplift, With 95% Significance”
- 3. ● What doesthis mean? ● Is this correct? “10% Uplift, With 95% Significance”
- 4.
- 5.
- 7.
- 8. Marketer: “Roll itout!” Statistician (me): *sobs*
- 9.
- 11.
- 12. You will learntoday: ● The most common serious errors in A/B testing ● How to avoid them ● How to interpret your result ● Whether to roll it out
- 13. How to Runan A/B Test
- 14. 1. Test design 2.Results interpretation 3. Decision
- 15.
- 16. Jargon: Null Hypothesis ●The hypothesis that your variant and original are functionally equivalent
- 17. e.g. an A/ATest vs. A A
- 18.
- 19. Jargon: P-Value ● Thechance of a result this extreme if the null hypothesis is true ● E.g. 0.05 for 95% significance
- 20.
- 21. Jargon: Critical Value ●What you compare your p-value with when deciding whether to reject the null hypothesis
- 22.
- 23.
- 24. A B C DE F Multivariate Testing Landing Page: Product Pages:
- 25. Multivariate Testing A B CD Landing Page: Product Page:
- 26.
- 27.
- 28. Multivariate Testing A C D BA CD BA C D B
- 29. Multivariate Testing A C D BA C D BA C D BA C D B
- 30. Multivariate Testing A BLandingPage: A: 5% B: 7.5%
- 31. Multivariate Testing C D C:5% D: 7.5% Product Page:
- 32. Multivariate Testing A C D BA C D BA C D BA C D B
- 33. A C Multivariate Testing D B A CD BA C D BA C D B AC: 0% BD: 5% BC: 10% AD: 10%
- 34.
- 36.
- 37. Multiple Testing A BC D E F
- 38.
- 39. False Positives Test: Healthy Test:Ill False Negatives
- 40. False Positives Test: Healthy Test:Ill False Positives False Negatives
- 41. Multiple Testing 1 A/Atest: 5% chance of achieving 95% significance.
- 42. Multiple Testing 1 A/ATest: 5% chance
- 43. Multiple Testing 1 A/ATest: 2 A/A Tests: 5% chance 9.75% chance
- 44. Multiple Testing 1 A/ATest: 2 A/A Tests: 3 A/A Tests: 5% chance 9.75% chance 14.26% chance
- 45. Multiple Testing 1 A/ATest: 2 A/A Tests: 3 A/A Tests: 4 A/A Tests: 5% chance 9.75% chance 14.26% chance 18.55% chance
- 46. Multiple Testing 1 A/ATest: 2 A/A Tests: 3 A/A Tests: 4 A/A Tests: n A/A Tests: 5% chance 9.75% chance 14.26% chance 18.55% chance 1-0.95^n
- 47. Multiple Testing Solutions: 1. Acceptrisk of false positives
- 48. Multiple Testing Solutions: 1. Acceptrisk of false positives 2. Bonferroni correction
- 49. Bonferroni Approximation Standard: P-valuevs………..…. 0.05
- 50. Bonferroni Approximation Standard: P-valuevs………..…. Approximation: P-value vs…... 0.05 0.05/N
- 51. Bonferroni Correction Standard: P-valuevs………..…. Bonferroni: P-value vs………. 0.05 1-(1-0.05)^(1/N)
- 52. Multiple Testing Solutions: 1. Acceptrisk of false positives 2. Bonferroni correction 3. Holm-Bonferroni correction
- 53. Choosing the RightMetric
- 54. Choosing the RightMetric Conversion Rate vs. Average Session Value
- 55. Choosing the RightMetric Conversion Rate vs. Average Session Value Profit?
- 56.
- 57. Stopping Rules Common: Whenmy test reaches significance.
- 58. “Significance so far”varies over time.
- 59. Stopping Rules Y YY Y Y N N N N N
- 60. Stopping Rules Y YY Y Y Y YN N N
- 61.
- 62.
- 63.
- 64. Stopping Rules Solutions: 1. Sequentialtesting - e.g. Optimizely 2. Bayesian testing - e.g. VWO 3. Predetermined sample size
- 65.
- 66. Sample Size forAverage Session Value Testing
- 67.
- 68.
- 69.
- 70.
- 72. 1. Test design 2.Results interpretation 3. Decision
- 73.
- 74.
- 75. Interpreting the P-value 1test reaches 95% significance: 5% chance of data this extreme if variants functionally equivalent.
- 76.
- 77.
- 78. Analogy Question: How likelyis it that my analytics or site are broken?
- 79. Analogy Question: How likelyis it that my analytics or site are broken? Non-Answer: We only go a whole day with no conversions once every 2 months.
- 80. Analytics is brokenwith probability 1 or 0.
- 81. Interpreting the P-value Question:How likely is it that this variation actually does nothing? Non-Answer: We’d only see a difference this big 5% of the time.
- 82. Meanwhile in IndustryTools: ● “Chance to beat baseline” ● “We are 95% certain that the changes in test “B” will improve your conversion rate”
- 83.
- 84. Unanswered Questions Question: Howlikely is it that the increase will be less than predicted?
- 85. Unanswered Questions Question: Howlikely is it that the increase will be negative?
- 86. One Mistake Probability ofOutcome given Data vs. Probability of Data given Null
- 87. Unanswered Questions Question: Howlikely is it that these results are a fluke?
- 88.
- 89.
- 90.
- 91. Everything Else StillApplies
- 92. Choosing the RightMetric
- 93.
- 94.
- 95. 1. Test design 2.Results interpretation 3. Decision
- 96.
- 97. A Good A/BTest Result: “10% Uplift, With 95% Significance”
- 98. But what aboutthis? “10% Uplift, With 60% Significance”
- 99. Jargon: P-Value ● Thechance of a result this extreme if the null hypothesis is true ● E.g. 0.05 for 95% significance
- 100. “10% Uplift, With60% Significance” ● 40% chance of data at least this extreme if variation functionally identical
- 101. “10% Uplift, With60% Significance” ● 40% chance of data at least this extreme if variation functionally identical ● The variation is probably better than the baseline
- 102.
- 103. Are You OKWith False Positives?
- 106.
- 107. Data is Expensive: ●Opportunity Cost ● Exploration vs. Exploitation
- 108.
- 109. Hang on… Why ShouldI Care About Significance?
- 110. 1. Ignoring Significance Doesn’tAllow You to Ignore Statistics
- 111.
- 112. Risk Factors: ● Agility ●Business attitudes ● What’s the worst that could happen?
- 113.
- 114.
- 115. 1. Think aboutsignificance and risk during test design
- 116. 2. Remember yourreal KPI: Profit
- 117. 3. You’re nottesting medicines
- 118. @THCapper Takeaways: 1. Think aboutsignificance and risk during test design 2. Remember your real KPI: Profit 3. You’re not testing medicines