Experimentation abounds, but how do we test our tests? I’ll share some ways we at Etsy proved our experimentation methods broken, and the approach we took to fixing them. I’ll discuss multiple ways of running A/A tests (as opposed to A/B tests), and a statistical method called bootstrapping, which we used to remedy our experiment analysis.

1. Introduction to the
(Bag of Little) Bootstrap(s)

2. Agenda
1 2 3
4 5
Identification of the
Problem
Classic Bootstrap Bag of Little Bootstraps
Evaluation of Success
(or Failure)
Next Steps
2

3. Identification of the Problem
SECTION 1
3

4. A/B Testing
4

5. CONTROL VARIANT
No pugs. No change. Etsy with PUG!
5
Standard A/B Experiment

6. CONTROLS VARIANTS
6
Browser # ABJ463MZ2
OFF
to pug or not to pug?

7. CONTROLS VARIANTS
7

8. CONTROLS VARIANTS
8

9. 9

10. 10
standard error:
accounting for variance
and sample sizes

11. confidence intervals
11
0 0
? → +

12. A/A Testing
12

13. CONTROL VARIANT
No pugs. No change. No pugs. No change.
13
Standard A/A Experiment

14. 14
→ +
What the?

15. Statistical Assumptions
for Parametric Testing
15

16. i.i.d.
independent & identically distributed
16
A
A
B
C
D
E

17. normally distributed
17

18. 18
narrow confidence intervals

19. Classic Bootstrap
SECTION 2
19

20. 20
...
sample boots:
WITH replacement
(do this many, many times)

21. take the 95%
confidence interval
21
2.5th 97.5th
distribution of t-statistics
boot 1
boot 3
boot 2
boot 4
boot 1 results
boot 2 results
boot 3 results
boot 4 results

22. 22
BOOTSTRAP BY USER
instead of by "visit" (one user can have many "visits")
user id visit id $
A 45 1
A 23 1
A 85 0
B 37 0
C 12 1
C 72 0
A
B
C
}
}
}
closer to i.i.d. (independent & identically distributed)

23. Bag of Little Bootstraps
SECTION 3
23

24. 24

25. 25
A B C D E Fusers in our
original experiment
fetch a bag:
resample withOUT replacementbag 1
A
E
F
B
25

26. 26
A B C D E Fusers in our
original experiment
bag 1
F
A
E
B
E
B
F
A
F
B
monte carlo subsamples:
resample WITH replacement (from our bag)
*TO SIZE OF ORIGINAL DATA SET*
boot 1
26

27. 27
t-statistic =
difference in populations
standard error
[per monte carlo'd subsample]

28. bootstrapped
confidence interval
28
2.5th 97.5th
t-statistics
E
B
F
A
F
B
boot 1
E
B
F
A
B
boot 2
A
E
F
B
boot 3
E
A
A
E
F
A
F
B
boot 4
B

29. A B C D E Fusers in our
original experiment
confidence interval
from bag 1
2.5th 97.5th
t-statistics
E
B
F
A
B
boot 2
E
B
F
A
F
B
boot 1
E
F
B
boot 3
E
F
A
F
B
boot 4
A
E
A
A
B
bag 1
F
A
E
B
29

30. average the
confidence intervals
30
avg. avg.
2.5th 97.5th
averaged t-statistics
bag 1
bag 3
bag 2
bag 4
bag 1 results
bag 2 results
bag 3 results
bag 4 results

31. Fixes i.i.d. & distribution
Suitable for distributed systems
Faster, less memory
WINS
31

32. Evaluation of Success
(or Failure)
SECTION 4
32

33. Reshuffled Data
(Simulated A/As)
33

34. Generated Data
(Simulated A/Bs)
34

35. Existing Experiments
35

36. Next Steps
SECTION 5
36

37. Hyperparameter optimization,
Poisson Bootstrap
37

38. Support for larger experiments
38

39. Power calculations
& complicated metrics
39

40. Resources
Kleiner, A., Talwalkar, A., Sarkar, P., and Jordan, M. I. A scalable bootstrap for massive
data. arXiv preprint arXiv:1112.5016v2, 2012. URL http://arxiv.org/abs/1112.5016v2
Bakshy, E., Eckles, D. Uncertainty in Online Experiments with Dependent Data: An
Evaluation of Bootstrap Methods. arXiv preprint arXiv:1304.7406v3, 2013. URL
https://arxiv.org/pdf/1304.7406v3.pdf
Idea for Bootstrapping @ Etsy: @hpster (Hilary Parker)
40

41. QUESTIONS, COMMENTS, REFUTATIONS?
Contact me: Emily Sommer (esommer@etsy.com)
Thanks!
41