Be the first to like this
When running an A/B test with a small experimental cell, one needs to decide whether to use all remaining traffic as the control group or to use a control group of the same size as the experimental cell. Here I construct a minimal theoretical model to show that test results can depend on this choice of control group. This comes from a combination of two effects: (1) unconverted (anonymous) visitors coming back to a website with an identity that cannot be linked to their first visit (e.g. on a new device or with cookies turned off) and (2) return rates and/or second-visit conversion rates varying between control and experiment experiences.
So what do we do? In the model I found that test results will usually be most accurate when we use equal-size experimental and control cells, so I recommend using equal-size cells with a holdout cell whenever a 50/50 split is not appropriate. However, I found that even in this case results will not in general agree with what we would measure if we could track visitors perfectly. This is reminder that A/B test results on anonymous web traffic must be taken with a grain of salt.