The math-behind-ab-testing

3,715
-1

Published on

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,715
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

The math-behind-ab-testing

  1. 1. The math behind A/B testing     How  to  perform  a  non-­‐biased  test
  2. 2. A/B testingNot a replacement for common senseIt only gives you a level of confidenceHelps you achieve only local maxima
  3. 3. AB experiment: Toss a coin    Heads  =  successful  conversion.  Tails  =  no  conversion.   Hypothesis:  Wearing  a  Red  color  t-­‐shirt  will  increase  conversion
  4. 4. 46 heads outof 10054 heads outof 100
  5. 5. Conversion increased by 17% Changing  your  t-­‐shirt  to  red  increases  conversion
  6. 6. Whats wrong Conversion  is  never  a  single  number.  Its  a  range.Probability µ Variance/Noise
  7. 7. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  =  0.46 ∞  coin  tosses
  8. 8. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  ==  0.46 µ    0.46 ∞  coin  tosses 100  coin  tosses Sample  mean  ≠  population  mean
  9. 9. The role of chance Red Blue Comparison  between  two  noisy  samples
  10. 10. Statistical significance Red Blue Standard Error (SE) = Square root of (p * (1-p) / n) p = conversion rate, n = sample size How much deviation from average conversion rate (p) can be expected if this experiment is repeated multiple times.
  11. 11. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
  12. 12. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
  13. 13. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE h3p://visualwebsiteop=mizer.com/ab-­‐split-­‐significance-­‐calculator/
  14. 14. Sample size Standard Error (SE) = Square root of (p * (1-p) / n)
  15. 15. Sample size Standard Error (SE) = Square root of (p * (1-p) / n) Min.  sample  size  to  calculate  the  statistical  signiLicance Statistical  conLidence Existing  conversion  rate  of  website Difference  in  conversion  rate  you  want  to  detect Number  of  variations  you  want  to  test h3p://www.testsignificance.com/
  16. 16. Ideal testDetermine  the  sample  sizeCheck  the  results  only  once  you  have  reached  the  sample  sizeDetermine  the  statistical  signiLicancePick  based  on  long  term  plan  if  no  clear  winner
  17. 17. Thanks

×