Slides from my talk at the Netflix Experimentation Meetup held on 02/07/2018.

1. One weird trick
to increase the power
of your A/B tests,
lose the weight,
regrow the hair,
and change your life.
Statisticians hate it!

2. John Clevenger

3. Chart has been cropped. Full version from statista
millions

4. 118 million members
=
~ $11 billion yearly revenue
x
$8 monthly fee

5. 0.1% change in retention rate
x
$11 billion yearly revenue
=
$12 million revenue change

6. Switching from independent-samples test
analysis to covariate-adjusted analysis can
massively increase the power of your tests.

7. before after

8. Getting more power.

9. difference in means
sd / sqrt(N)

10. Smaller effects are harder to detect.
difference in means
sd / sqrt(N)

11. difference in means
sd / sqrt(N)
Shrink variance without changing N.

12. Control StrangeFlix
Fake experiment.

13. Streaming correlated with country.

14. Control
StrangeFlix

15. Control
StrangeFlix

16. between-condition variability
within-condition variability

17. No covariates:
streaming ~ condition
One covariate:
streaming ~ condition + country
Multiple covariates:
streaming ~ condition + country + tenure + device

18. Leveraging covariate-adjustment to
increase power at Netflix.

19. Past
streaming
Tenure
Country

21. 36% power 80% power

22. One not so weird trick
to increase the power
of your A/B tests