Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The math-behind-ab-testing

The math-behind-ab-testing

  1. 1. The math behind A/B testing     How  to  perform  a  non-­‐biased  test
  2. 2. A/B testingNot a replacement for common senseIt only gives you a level of confidenceHelps you achieve only local maxima
  3. 3. AB experiment: Toss a coin    Heads  =  successful  conversion.  Tails  =  no  conversion.   Hypothesis:  Wearing  a  Red  color  t-­‐shirt  will  increase  conversion
  4. 4. 46 heads outof 10054 heads outof 100
  5. 5. Conversion increased by 17% Changing  your  t-­‐shirt  to  red  increases  conversion
  6. 6. Whats wrong Conversion  is  never  a  single  number.  Its  a  range.Probability µ Variance/Noise
  7. 7. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  =  0.46 ∞  coin  tosses
  8. 8. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  ==  0.46 µ    0.46 ∞  coin  tosses 100  coin  tosses Sample  mean  ≠  population  mean
  9. 9. The role of chance Red Blue Comparison  between  two  noisy  samples
  10. 10. Statistical significance Red Blue Standard Error (SE) = Square root of (p * (1-p) / n) p = conversion rate, n = sample size How much deviation from average conversion rate (p) can be expected if this experiment is repeated multiple times.
  11. 11. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
  12. 12. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
  13. 13. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE h3p://visualwebsiteop=mizer.com/ab-­‐split-­‐significance-­‐calculator/
  14. 14. Sample size Standard Error (SE) = Square root of (p * (1-p) / n)
  15. 15. Sample size Standard Error (SE) = Square root of (p * (1-p) / n) Min.  sample  size  to  calculate  the  statistical  signiLicance Statistical  conLidence Existing  conversion  rate  of  website Difference  in  conversion  rate  you  want  to  detect Number  of  variations  you  want  to  test h3p://www.testsignificance.com/
  16. 16. Ideal testDetermine  the  sample  sizeCheck  the  results  only  once  you  have  reached  the  sample  sizeDetermine  the  statistical  signiLicancePick  based  on  long  term  plan  if  no  clear  winner
  17. 17. Thanks

×