The document discusses the issues associated with A/B testing, particularly the reliance on statistical assumptions that can lead to misleading results. It highlights the discrepancies between expected and actual outcomes in various tests, emphasizing the need for careful analysis and replication. Additionally, the document touches on the importance of hypothesis formation and the potential pitfalls of multivariate testing and multiple comparisons.

1. A/B testing and problems with
statistics
Web Analytics Wednesday Singapore
Nikolay Novozhilov, Wego.com
www.novozhilov.co

2. Is there a problem with A/B testing?

3. Imaginary uplifts
100 tests done, 10 successful, 10% uplift each…
…expect 159% growth!
Expectation Reality

4. Why?
… and what to do about it

5. Lies, damned lies, and statistics
All different! All based on assumptions!!!
Tool Test used
Optimizely Two-tailed sequential likelihood ratio test
with false discovery rate controls
Google Analytics Bayes estimate with uniform beta prior
VWO Intersection of confidence intervals for
binominal distribution
Leanplum Confidence intervals at p=5%, unknown
statistic
Usereffect Chi-square statistics
Commerce Sciences Welch's t-test

6. What is p-value and why it is 5%?
All tests are
based on
assumptions!
Assumption #1:
You don’t look at
the data upfront

7. What happens if you look?
I played Monte Carlo in
Excel
And here is the result:
• 5% p-value
• 1000 “users” in each
sample
• CR of 2%
• A wins over A 29% of
the times!

8. What do you do about it?
Don’t look! (just kidding)
Google “O'Brien & Fleming interim
analysis” (no, still kidding )
Keep calm, more stuff coming!

9. “My test on Buy button showed
interesting results…”
Buy Now! Buy Now! Buy Now! Buy Now!
Buy Now! Buy Now! Buy Now! Buy Now!
Buy Now! Buy Now! Buy Now! Buy Now!
-3% -23% +6% -9%
-2% +22% -11% -14%
-1% +9% -12% -1%
10000 users in each variant, base CR=1%

10. But in reality all colors were
the same…
Buy Now! Buy Now! Buy Now! Buy Now!
Buy Now! Buy Now! Buy Now! Buy Now!
Buy Now! Buy Now! Buy Now! Buy Now!
-3% -23% +6% -9%
-2% +22% -11% -14%
-1% +9% -12% -1%
10000 users in each variant, base CR=1%

11. The real problem!
Multivariate testing
Multiple comparisons

12. Be smart or be Google
Sample
size
Significa
nce
Effect
size
Power

13. Start with a good hypothesis!
But people are good in finding plausible
explanations for data!

14. Replication
Do your dirty
business
Register Replicate
This might work!

15. Stop math, I’m a web designer!

16. Visual way of doing it

17. Has some stat meaning!
ReplicationsVariance observation