Slides from our talk at Indeed about how standard A/B testing methodology biases treatment estimates.

1. Your A/B Tests are
Lying to You.

2. Proprietary and confidential
John Clevenger
Data Scientist
John McDonnell
Data Science Manager

3. Proprietary and confidential
Stitch Fix

4. Proprietary and confidential
How does Stitch Fix work?

5. Proprietary and confidential
How does Stitch Fix work?

6. Proprietary and confidential
How does Stitch Fix work?

7. Proprietary and confidential
How does Stitch Fix work?

8. Proprietary and confidential
How does Stitch Fix work?

9. Proprietary and confidential
How does Stitch Fix work?

10. Proprietary and confidential
Hard problems: inferring latent size
Blog post on latent size

11. Proprietary and confidential
Hard problems: inferring latent style
Blog post on latent style

12. Proprietary and confidential 12
We hope to convince you:

13. Proprietary and confidential 13
We hope to convince you:
● Standard A/B testing methods lead you to overestimate
the business impact of rolled out features.

14. Proprietary and confidential 14
We hope to convince you:
● Standard A/B testing methods lead you to overestimate
the business impact of rolled out features.
● The only way around this is to massively increase
sample size, run replications/holdouts, or to model out
the bias.

15. Proprietary and confidential 15
We hope to convince you:
● Standard A/B testing methods lead you to overestimate
the business impact of rolled out features.
● The only way around this is to massively increase
sample size, run replications/holdouts, or to model out
the bias.
● You should think hard about your goal: Is it just
hypothesis testing or is it estimation too?

16. Proprietary and confidential
Airbnb’s great quarter
Example from Airbnb [blogpost]

17. Proprietary and confidential
Example from Airbnb [blogpost]
Airbnb’s great quarter

18. Proprietary and confidential
Example from Airbnb [blogpost]
Airbnb’s great quarter

19. Proprietary and confidential
Example from Airbnb [blogpost]
Airbnb’s great quarter

20. Proprietary and confidential
Something doesn’t add up
Example from Airbnb [blogpost]

21. Proprietary and confidential 21
What’s going
on here?

22. Proprietary and confidential
Survivorship Bias
● Treating samples that passed some selection
threshold as though they were randomly selected.

23. Proprietary and confidential
Factory line at the mint
● Your job: determine if each coin is fair.

24. Proprietary and confidential
You flip each coin 10 times

25. Proprietary and confidential
Your first day

26. Proprietary and confidential
30% of coins are bad?

27. Proprietary and confidential
Hm

28. Proprietary and confidential
● Our measurements will be biased when:
What’s going on?

29. Proprietary and confidential
● Our measurements will be biased when:
○ We expect variability in our measurements.
What’s going on?

30. Proprietary and confidential
● Our measurements will be biased when:
○ We expect variability in our measurements.
and
○ We condition on only those measurements that
pass some evidence threshold.
What’s going on?

31. Proprietary and confidential
Goal of research
● To generalize from some sample to a population:
○ Inferences about the population are only as good as the sample.

32. Proprietary and confidential 32
How does this
affect A/B testing?

33. Proprietary and confidential
2019 Stitch Fix website

34. Proprietary and confidential
2011 Stitch Fix website

35. Proprietary and confidential
Get on the list for “a Stitch Fix”

36. Proprietary and confidential
Let’s go back in time
TreatmentControl

37. Proprietary and confidential
2019 website is 1% better than 2011
TreatmentControl

38. Proprietary and confidential 38
Simulated
A/B tests

39. Proprietary and confidential
Simulated A/B tests
● Control:
○ Draw 50k samples from a binomial distribution where
probability of success = 50%.

40. Proprietary and confidential
Simulated A/B tests
● Control:
○ Draw 50k samples from a binomial distribution where
probability of success = 50%.
● Treatment:
○ Draw 50k samples from a binomial distribution where
probability of success = 50% * 1.01 = 50.5%.

41. Proprietary and confidential
Simulated A/B tests
● Control:
○ Draw 50k samples from a binomial distribution where
probability of success = 50%.
● Treatment:
○ Draw 50k samples from a binomial distribution where
probability of success = 50% * 1.01 = 50.5%.
● Run statistical test
○ Roll out 2019 if it’s better than 2011 and p < 0.05.

42. Proprietary and confidential
Simulated A/B tests
● Control:
○ Draw 50k samples from a binomial distribution where
probability of success = 50%.
● Treatment:
○ Draw 50k samples from a binomial distribution where
probability of success = 50% * 1.01 = 50.5%.
● Run statistical test
○ Roll out 2019 if it’s better than 2011 and p < 0.05.
● Repeat 10k times

43. Proprietary and confidential 43
Results

44. Proprietary and confidential
10k treatment estimates

45. Proprietary and confidential
The mean of all tests is 1%

46. Proprietary and confidential
Unbiased estimate
The mean of all tests is 1%

47. Proprietary and confidential
p < 0.05 as cutoff

48. Proprietary and confidential
p < 0.05 as cutoff

49. Proprietary and confidential
70% too big!
p < 0.05 as cutoff

50. Proprietary and confidential
100% too big!
p < 0.01 as cutoff
100% too big!

51. Proprietary and confidential
p < 0.001 as cutoff
130% too big!

52. Proprietary and confidential
Lesson
If your power < 1 then using any evidence threshold to
determine winners inflates treatment estimates.

53. Proprietary and confidential 53
2 goals
of experimentation

54. Proprietary and confidential
2 goals of experimentation
Hypothesis Testing
● Goal: Know if treatment is different from control.
● Output: Probability of observed data given null.

55. Proprietary and confidential
2 goals of experimentation
Hypothesis Testing
● Goal: Know if treatment is different from control.
● Output: Probability of observed data given null.
Treatment Effect Estimation
● Goal: Know how different treatment is from control.
● Output: distribution of treatment effect sizes compatible with observed
data.

56. Proprietary and confidential
2 goals of experimentation
Hypothesis Testing -> Inflated Treatment Estimates

57. Proprietary and confidential 57
What can we do?

58. Proprietary and confidential
Solutions
Increase Sample Size
● Pro: no inflation when power = 1.
● Cons: time consuming, opportunity costs.

59. Proprietary and confidential
Solutions
Increase Sample Size
● Pro: no inflation when power = 1.
● Cons: time consuming, opportunity costs.
Holdouts/Replications
● Pro: no inflation.
● Cons: resource intensive, time consuming, opportunity costs.

60. Proprietary and confidential
Solutions
Increase Sample Size
● Pro: no inflation when power = 1.
● Cons: time consuming, opportunity costs.
Holdouts/Replications
● Pro: no inflation.
● Cons: resource intensive, time consuming, opportunity costs.
Estimate and Remove Bias
● Pro: easy.
● Cons: relies on assumptions, based on long run expectation.

61. Proprietary and confidential 61
Estimate and
remove bias

62. Proprietary and confidential
Model the experimentation process

63. Proprietary and confidential
Simulated A/B tests
1%

64. Proprietary and confidential
1%
50k samples from binomial
with probability of success
= 50%.
Control
Simulated A/B tests

65. Proprietary and confidential
1%
50k samples from binomial
with probability of success
= 50%.
50k samples from binomial
with probability of success
= 50% * 1.01 = 50.5%.
Control Treatment
Simulated A/B tests

66. Proprietary and confidential
Control Treatment
Repeat 50k times.
Simulated A/B tests

67. Proprietary and confidential 67
Results

68. Proprietary and confidential
All 50k tests

69. Proprietary and confidential
Only rollouts

70. Proprietary and confidential
Inflation is a function of power

71. Proprietary and confidential
Inflation is a function of power
Problem
● We never know the true power of any test.

72. Proprietary and confidential
Inflation is a function of power
Problem
● We never know the true power of any test.
Solution
● Infer power from observed difference.

73. Proprietary and confidential
Infer power from observed difference

74. Proprietary and confidential
Once we infer power, we can estimate inflation.
Infer power from observed difference

75. Proprietary and confidential
Infer inflation from observed difference

76. Proprietary and confidential
Once we infer expected inflation, we can shrink the
observed difference appropriately.
Infer inflation from observed difference

77. Proprietary and confidential
Correct for inflation

78. Proprietary and confidential 78
We hope we convinced you:

79. Proprietary and confidential 79
● Standard A/B testing methods lead you to overestimate
the business impact of rolled out features.
We hope we convinced you:

80. Proprietary and confidential 80
We hope we convinced you:
● Standard A/B testing methods lead you to overestimate
the business impact of rolled out features.
● The only way around this is to massively increase
sample size, run replications/holdouts, or to model out
the bias.

81. Proprietary and confidential 81
We hope we convinced you:
● Standard A/B testing methods lead you to overestimate
the business impact of rolled out features.
● The only way around this is to massively increase
sample size, run replications/holdouts, or to model out
the bias.
● You should think hard about your goal: Is it just
hypothesis testing or is it estimation too?

82. Proprietary and confidential 82
Fin

83. Proprietary and confidential 83
Appendix

84. Proprietary and confidential
Early stopping/sequential testing
● Efficient A/B testing where you peek at the data more than once.
○ Controls Type I error rate by “spending” alpha incrementally at each
peak.

85. Proprietary and confidential
Early stopping/sequential testing
● Efficient A/B testing where you peek at the data more than once.
○ Controls Type I error rate by “spending” alpha incrementally at each
peak.
○ Example:
■ Collect data for 1 week, run test, stop test if t-statistic is more
extreme than t-statistic at that peek.

86. Proprietary and confidential
Early stopping/sequential testing
● Efficient A/B testing where you peek at the data more than once.
○ Controls Type I error rate by “spending” alpha incrementally at each
peak.
○ Example:
■ Collect data for 1 week, run test, stop test if t-statistic is more
extreme than t-statistic at that peek.
● Problem: leads to even more bias in treatment estimates!

87. Proprietary and confidential
Early stopping/sequential testing
● Efficient A/B testing where you peek at the data more than once.
○ Controls Type I error rate by “spending” alpha incrementally at each
peak.
○ Example:
■ Collect data for 1 week, run test, stop test if t-statistic is more
extreme than t-statistic at that peek.
● Problem: leads to even more bias in treatment estimates.

88. Proprietary and confidential
2 peeks

89. Proprietary and confidential
10 peeks