Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

17

Share

The math-behind-ab-testing

Related Audiobooks

Free with a 30 day trial from Scribd

See all

The math-behind-ab-testing

  1. 1. The math behind A/B testing     How  to  perform  a  non-­‐biased  test
  2. 2. A/B testingNot a replacement for common senseIt only gives you a level of confidenceHelps you achieve only local maxima
  3. 3. AB experiment: Toss a coin    Heads  =  successful  conversion.  Tails  =  no  conversion.   Hypothesis:  Wearing  a  Red  color  t-­‐shirt  will  increase  conversion
  4. 4. 46 heads outof 10054 heads outof 100
  5. 5. Conversion increased by 17% Changing  your  t-­‐shirt  to  red  increases  conversion
  6. 6. Whats wrong Conversion  is  never  a  single  number.  Its  a  range.Probability µ Variance/Noise
  7. 7. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  =  0.46 ∞  coin  tosses
  8. 8. Whats wrong Sample  =  100  tosses. µ  =  0.5 µ  ==  0.46 µ    0.46 ∞  coin  tosses 100  coin  tosses Sample  mean  ≠  population  mean
  9. 9. The role of chance Red Blue Comparison  between  two  noisy  samples
  10. 10. Statistical significance Red Blue Standard Error (SE) = Square root of (p * (1-p) / n) p = conversion rate, n = sample size How much deviation from average conversion rate (p) can be expected if this experiment is repeated multiple times.
  11. 11. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
  12. 12. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE
  13. 13. Statistical significance 95% confidence: True conversion rate lies within this range: p ± 2 * SE h3p://visualwebsiteop=mizer.com/ab-­‐split-­‐significance-­‐calculator/
  14. 14. Sample size Standard Error (SE) = Square root of (p * (1-p) / n)
  15. 15. Sample size Standard Error (SE) = Square root of (p * (1-p) / n) Min.  sample  size  to  calculate  the  statistical  signiLicance Statistical  conLidence Existing  conversion  rate  of  website Difference  in  conversion  rate  you  want  to  detect Number  of  variations  you  want  to  test h3p://www.testsignificance.com/
  16. 16. Ideal testDetermine  the  sample  sizeCheck  the  results  only  once  you  have  reached  the  sample  sizeDetermine  the  statistical  signiLicancePick  based  on  long  term  plan  if  no  clear  winner
  17. 17. Thanks
  • DingxinXu

    Sep. 9, 2019
  • fradyali

    May. 11, 2019
  • tweety69711

    Oct. 28, 2017
  • VidyaSagarM4

    Sep. 15, 2017
  • tonycao

    Dec. 27, 2016
  • MakotoKIMURA8

    Nov. 7, 2016
  • seufagner

    Sep. 29, 2016
  • azambuja

    Jul. 18, 2016
  • anton1987

    Apr. 29, 2016
  • jainadit

    Mar. 26, 2016
  • harshaxv

    Aug. 10, 2015
  • YvonneAdams2

    Jul. 8, 2015
  • darkseed

    Dec. 23, 2014
  • Raina_Yin

    Dec. 21, 2014
  • rajendrak1

    Sep. 24, 2014
  • ssssss55

    Jun. 23, 2014
  • vinodkunwar1

    Mar. 16, 2013

Views

Total views

11,980

On Slideshare

0

From embeds

0

Number of embeds

41

Actions

Downloads

0

Shares

0

Comments

0

Likes

17

×