Experimental Meta-analyses
•
0 likes•156 views
Dominic Coey at LinkedIn Experimentation Meetup on Dec 13, 2018
Recommended
More Related Content
Similar to Experimental Meta-analyses
Similar to Experimental Meta-analyses (20)
Recently uploaded
Recently uploaded (20)
Experimental Meta-analyses
- 1. Experimental Meta-analyses Dominic Coey, coey@fb.com Tom Cunningham, tomcunningham@fb.com Facebook
- 2. Why Meta-analysis? Part I. Suppose you have two metrics, A and B. • Metric A is significant at the 5% level in 5% of all experiments. • Metric B is significant at the 5% level in 40% of all experiments. All else equal, which should you trust more?
- 3. Why Meta-analysis? Part I. Metric A suggests no experiment ever has any effect. All noise! Our best guess of the true effect = zero. Metric B suggests some experiments have an effect. Our best guess of the true effect = estimated effect x 0.82 (if everything is normal and mean zero).
- 4. Why Meta-analysis? Part II. Imagine this is the histogram of estimated effect sizes from historical experiments. You see a 2% lift in your new experiment. What should you infer?
- 5. Why Meta-analysis? Part II. Each observed effect, y, is the sum of • the true treatment effect, t • sampling error, e If y is very large, likely in part due to a large draw of e. So should adjust y downwards ("shrink") to offset this, and get a better estimate of t.
- 6. Why Meta-analysis? Part III. Consider a test which improves metric A ("comments") but degrades a related metric B ("posts"). What, if anything, can we conclude from this? Can the movement in B give us more information about the movement in A?
- 7. Why Meta-analysis? Part III. Silly example: • the true lift in each experiment is t ~iid F, equal for both metrics • observed values are yA = t + eA, yB = t + eB, for independent eA, eB • metric B's contains extra information about metric A More generally, might have some joint distribution of (tA, tB, yA, yB), estimated on past experiments.
- 8. Conclusion Tech companies run lots of experiments, but they often fall into a small number of experiment types. Ignoring the information in past, highly related experiments is leaving a lot on the table! Our paper on experiment splitting develops some of these issues.
- 9. Appendix Where does the 0.82 number come from? Consider the model where • the true effect t ~ N(0, vt) • the sampling error e ~ N(0, ve) • the observed outcome is y = t + e Can show • E(t | y) = y x vt/(vt + ve). • If the fraction of rejections at the 5% level is p, then vt/(vt + ve) = 1 - (Φ-1(p/2)/1.96)2.