How many visitors do you need for your A/B test?

How many visitors do you need
for your A/B test?

testing simplified!

© Wingify Software Pvt. Ltd.

Disclaimer
• We can never say that we need X number of
visitors for an A/B test
– Theoretically, an A/B test can take infinite number
of visitors before producing statistically significant
result
• Instead, what we can say is that we need to at
least test X number of visitors
– With Y probability (usually 80%) of detecting a
statistical difference in results (if there is any)


Phew! Explanations please..


First, let’s define statistical significance
• Conversion rate is never an exact number, it is
always a range. That is, we can never say
conversion rate is 10%, although we always say it is
10% ± 1%

– This is because as we collect data, we are estimating
what real conversion rate is (in statistical terms, we are
estimating population mean from sample mean)

– Initially, our guesses are raw (as we only have few data
points) but as we test more visitors, the error range
decreases and we have better estimates of conversion
rate


But, still, conversion rates are always
estimates…
• So, we now have two conversion rate estimates for control
and variation, say following:

• Observe how these ranges are overlapping, and so even if
conversion of variation appears to be worse, we cannot say for sure
(until these ranges are non overlapping, as following)


So, how many visitors to A/B test?
• There are two scenarios in A/B test:
– Variation is performing better (or worse) as
compared to control
• Difference in conversion rate is statistically significant

– Variation is performing similarly as compared to
control
• Difference in conversion rate is not statistically
significant


So, how many visitors to A/B test?
• Aim of A/B test calculations is to make sure we test
enough visitors in order to know with certain
confidence whether there is any statistical difference in
control and variation conversion rate

• As stressed earlier, we can never be 100% sure that
after testing X number of visitors we will know if test
has a statistically better or worse performing variation
– If we test even more visitors, there are even better
chances of finding a statistical difference if it really is there


Factors important in calculations
• Suppose we want to calculate X, which is the number
of visitors we need to test in order to find out whether
statistical significance is there

• There are various factors which help us calculate X:
– Statistical Power (usually 80%): it is the probability with
which you expect to find out statistical significance after
testing X visitors. (There are 80% chances that after testing
X visitors, we will find statistical significant results if they
are there)
• As statistical power increases, your chances of finding a statistical
difference gets better (but of course you need to test more
visitors)


Factors important in calculations
– Statistical confidence (normally 95%): once statistical difference is found, it is
the confidence we have in that difference. (There is 5% chance that the
difference in conversion rate is not real and is due to randomness)
• If you need higher statistical confidence, we need to test more visitors

– Existing conversion rate of website: for lower conversion rates websites (say
ones with 1% conversion), we need to test many more visitors as compared to
situation if average conversion rate is higher (say 10%)

– Difference in conversion rate you want to detect: if you want to detect even a
small difference in conversion rate (say you want to know if variation differs from
control by even 0.1%), you need to test many more visitors. If you are only
concerned with detecting a large differences (say only >10%), you need to test
lesser number of visitors

– Number of variations you are testing: obviously, if you are testing 4 variations
you need twice the number of visitors as compared to situation when you are
just testing 2 variations.


You need a thumb rule?

Sorry, there is no thumb rule to find
out how many visitors you need to
test :(


Not all is lost, though…
• You don’t have to be a statistician in order to do
these calculations, you can use an online calculator
to find out number of visitors to test:

http://visualwebsiteoptimizer.com/ab-split-test-duration/


Questions?

Paras Chopra, CEO, Wingify
paras@wingify.com


How many visitors do you need for your A/B test?

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

How many visitors do you need for your A/B test?