Successfully reported this slideshow.
Upcoming SlideShare
×

of

17

Share

# The math-behind-ab-testing

See all

See all

### The math-behind-ab-testing

1. 1. The math behind A/B testing     How  to  perform  a  non-­‐biased  test
2. 2. A/B testingNot a replacement for common senseIt only gives you a level of conﬁdenceHelps you achieve only local maxima
3. 3. AB experiment: Toss a coin    Heads  =  successful  conversion.  Tails  =  no  conversion.   Hypothesis:  Wearing  a  Red  color  t-­‐shirt  will  increase  conversion
5. 5. Conversion increased by 17% Changing  your  t-­‐shirt  to  red  increases  conversion
6. 6. Whats wrong Conversion  is  never  a  single  number.  Its  a  range.Probability µ Variance/Noise
7. 7. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  =  0.46 ∞  coin  tosses
8. 8. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  ==  0.46 µ    0.46 ∞  coin  tosses 100  coin  tosses Sample  mean  ≠  population  mean
9. 9. The role of chance Red Blue Comparison  between  two  noisy  samples
10. 10. Statistical signiﬁcance Red Blue Standard Error (SE) = Square root of (p * (1-p) / n) p = conversion rate, n = sample size How much deviation from average conversion rate (p) can be expected if this experiment is repeated multiple times.
11. 11. Statistical signiﬁcance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
12. 12. Statistical signiﬁcance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
13. 13. Statistical signiﬁcance 95% confidence: True conversion rate lies within this range: p ± 2 * SE h3p://visualwebsiteop=mizer.com/ab-­‐split-­‐signiﬁcance-­‐calculator/
14. 14. Sample size Standard Error (SE) = Square root of (p * (1-p) / n)
15. 15. Sample size Standard Error (SE) = Square root of (p * (1-p) / n) Min.  sample  size  to  calculate  the  statistical  signiLicance Statistical  conLidence Existing  conversion  rate  of  website Difference  in  conversion  rate  you  want  to  detect Number  of  variations  you  want  to  test h3p://www.testsigniﬁcance.com/
16. 16. Ideal testDetermine  the  sample  sizeCheck  the  results  only  once  you  have  reached  the  sample  sizeDetermine  the  statistical  signiLicancePick  based  on  long  term  plan  if  no  clear  winner
17. 17. Thanks
• #### DingxinXu

Sep. 9, 2019

May. 11, 2019
• #### tweety69711

Oct. 28, 2017
• #### VidyaSagarM4

Sep. 15, 2017
• #### tonycao

Dec. 27, 2016

Nov. 7, 2016
• #### seufagner

Sep. 29, 2016
• #### azambuja

Jul. 18, 2016
• #### anton1987

Apr. 29, 2016

Mar. 26, 2016
• #### harshaxv

Aug. 10, 2015

Jul. 8, 2015
• #### darkseed

Dec. 23, 2014
• #### Raina_Yin

Dec. 21, 2014
• #### rajendrak1

Sep. 24, 2014
• #### ssssss55

Jun. 23, 2014
• #### vinodkunwar1

Mar. 16, 2013

Total views

11,980

On Slideshare

0

From embeds

0

Number of embeds

41

0

Shares

0