Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How many visitors do you need for your A/B test?


Published on

This presentation will help you understand which factors are important while calculate number of visitors needed for an A/B test.

Published in: Technology, Business
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here

How many visitors do you need for your A/B test?

  1. 1. How many visitors do you need for your A/B test? testing simplified!© Wingify Software Pvt. Ltd.
  2. 2. Disclaimer • We can never say that we need X number of visitors for an A/B test – Theoretically, an A/B test can take infinite number of visitors before producing statistically significant result • Instead, what we can say is that we need to at least test X number of visitors – With Y probability (usually 80%) of detecting a statistical difference in results (if there is any)© Wingify Software Pvt. Ltd.
  3. 3. Phew! Explanations please..© Wingify Software Pvt. Ltd.
  4. 4. First, let’s define statistical significance • Conversion rate is never an exact number, it is always a range. That is, we can never say conversion rate is 10%, although we always say it is 10% ± 1% – This is because as we collect data, we are estimating what real conversion rate is (in statistical terms, we are estimating population mean from sample mean) – Initially, our guesses are raw (as we only have few data points) but as we test more visitors, the error range decreases and we have better estimates of conversion rate© Wingify Software Pvt. Ltd.
  5. 5. But, still, conversion rates are always estimates… • So, we now have two conversion rate estimates for control and variation, say following: • Observe how these ranges are overlapping, and so even if conversion of variation appears to be worse, we cannot say for sure (until these ranges are non overlapping, as following)© Wingify Software Pvt. Ltd.
  6. 6. So, how many visitors to A/B test? • There are two scenarios in A/B test: – Variation is performing better (or worse) as compared to control • Difference in conversion rate is statistically significant – Variation is performing similarly as compared to control • Difference in conversion rate is not statistically significant© Wingify Software Pvt. Ltd.
  7. 7. So, how many visitors to A/B test? • Aim of A/B test calculations is to make sure we test enough visitors in order to know with certain confidence whether there is any statistical difference in control and variation conversion rate • As stressed earlier, we can never be 100% sure that after testing X number of visitors we will know if test has a statistically better or worse performing variation – If we test even more visitors, there are even better chances of finding a statistical difference if it really is there© Wingify Software Pvt. Ltd.
  8. 8. Factors important in calculations • Suppose we want to calculate X, which is the number of visitors we need to test in order to find out whether statistical significance is there • There are various factors which help us calculate X: – Statistical Power (usually 80%): it is the probability with which you expect to find out statistical significance after testing X visitors. (There are 80% chances that after testing X visitors, we will find statistical significant results if they are there) • As statistical power increases, your chances of finding a statistical difference gets better (but of course you need to test more visitors)© Wingify Software Pvt. Ltd.
  9. 9. Factors important in calculations – Statistical confidence (normally 95%): once statistical difference is found, it is the confidence we have in that difference. (There is 5% chance that the difference in conversion rate is not real and is due to randomness) • If you need higher statistical confidence, we need to test more visitors – Existing conversion rate of website: for lower conversion rates websites (say ones with 1% conversion), we need to test many more visitors as compared to situation if average conversion rate is higher (say 10%) – Difference in conversion rate you want to detect: if you want to detect even a small difference in conversion rate (say you want to know if variation differs from control by even 0.1%), you need to test many more visitors. If you are only concerned with detecting a large differences (say only >10%), you need to test lesser number of visitors – Number of variations you are testing: obviously, if you are testing 4 variations you need twice the number of visitors as compared to situation when you are just testing 2 variations.© Wingify Software Pvt. Ltd.
  10. 10. You need a thumb rule? Sorry, there is no thumb rule to find out how many visitors you need to test :(© Wingify Software Pvt. Ltd.
  11. 11. Not all is lost, though… • You don’t have to be a statistician in order to do these calculations, you can use an online calculator to find out number of visitors to test:© Wingify Software Pvt. Ltd.
  12. 12. Questions? Paras Chopra, CEO, Wingify© Wingify Software Pvt. Ltd.