This document discusses the challenges and pitfalls of split testing experiments. It outlines four threats to data validation, including selection bias, multiple comparisons, outliers, and low statistical power. It emphasizes the importance of running tests long enough to reach statistical significance and avoiding premature conclusions. Finally, it recommends being disciplined by using test grids that vary one element at a time and establishing criteria for stopping tests.
3. Split Testing: Web Technology Evolution
March 20082006
Oracle 2015
2010
2010
2010
20042003
Omniture 2007
Adobe 2009
Harder to Use Easier to Use
4. Split vs. Multivariate Testing
Control
Variation A
Variation B
Variation C
25%
25%
25%
25%
Incoming Traffic
Randomly Split
15.0%
Conversion
18.5%
Conversion
17.3%
Conversion
14.4%
Conversion
5. Split vs. Multivariate Testing
Item 1
Item 2
Item 3
drumph again make donald drumph again make donald drumph
again make donald drumph again make donald drumph again make
donald drumph again make donald drumph again make donald
drumph again make donald drumph again make donald drumph
again make donald drumph again make donald drumph again make
donald drumph again make donald drumph again make donald
drumph again make donald drumph again make donald drumph
again make donald drumph again
drumph again make donald drumph again make donald drumph
again make donald drumph again make donald drumph again make
donald drumph again make donald drumph again make donald
drumph again make donald drumph again make donald drumph
again make donald drumph again make donald drumph again make
donald drumph again make donald drumph again make donald
Variations of Item 1
Variations of Item 2
Variations
of Item 3
Multivariate
Testing