Recommended
More Related Content
What's hot
What's hot (20)
Similar to [PositConf 2023] How Data Scientists Broke A/B Testing (and How We Can Fix It)
Similar to [PositConf 2023] How Data Scientists Broke A/B Testing (and How We Can Fix It) (20)
Recently uploaded
Recently uploaded (20)
[PositConf 2023] How Data Scientists Broke A/B Testing (and How We Can Fix It)
- 5. Launch on Neutral (But thanks anyway)
- 7. A real PM “If it’s something we really believe in, I’ll launch on a flat result … if it’s part of a broader strategy.” “My features are hard as shit to build, but easy to tweak, so I’m not always worried about statistical significance.” Another real PM
- 8. NotjustNHST Features aren’t IID Path dependencies in feature roadmaps We develop experiences by building up features over time and it’s helpful to launch them incrementally MDE is basically zero Feature costs are nearly all sunk before the test Any lift pays off
- 9. NotjustNHST Risk is mismeasured Decision makers don’t think about Type I and II error rates, per se They just want to make more money than they lose
- 13. Non-inferioritydesigns Let’s try not to wreck the place Superiority Non-Inferiority
- 14. Non-inferioritydesigns Let’s try not to wreck the place • Inferiority margins ( ) prompt us to ask: • How much do we believe in this feature? • How quickly will we improve on it? • Stakeholders can give meaningful answers to these questions • Compare to MDE/minimal lift, which is often made up • Avoid meaningless minimum e ff ect estimates • Can power against a “no e ff ect” alternative Δ
- 16. What’s the rush?
- 17. Thecostsoflongexperiments Time is money, folks • Opportunity cost of time: • Experimental features live on a roadmap, waiting for launch decisions delays development of subsequent features • Opportunity cost of sampling: • As long as the experiment runs, many users aren’t getting the best variant • Maintenance costs: • More experiments running means more complexity in the codebase, more e ff ort, etc.
- 19. Whenisdataworthit? Good things are worth waiting for •Waiting is costly, but data is valuable. •We should keep going as long as the value of more data exceeds the cost of more time •Quantify our impatience as part of test design
- 20. ExpectedValuevs.CostofData $0 $20,000 $40,000 $60,000 $80,000 Test Length 0 15 30 45 60 Exp. Value Cost Net Exp. Value
- 21. Whyisdatavaluable? How dumb am I, in dollars? • Before we have data, our range of potential lifts is wide • Our best guess could be way o ff ; we could make a big mistake • Observing data narrows the range, even if our new guess is wrong, it won’t be wrong by as much. • If the value of being less wrong (in expectation) exceeds the cost of waiting for the data, LFG!
- 28. Sequentialtestingdecisions Don’t stop ’til you get enough • We can do this again after collecting some data • This changes the core decision from: “is B > A?” to “should I stop or continue testing?” • Good fi t for A/B tests, where we collect data passively just by waiting • Once more data isn’t worth it, launch the best observed variant, the inference problem is irrelevant (Claxton ’96) • This is our best information, and it’s not worth getting more
- 29. Lessons
- 30. What’stheProblem? Going back to basics There’s no silver bullet You may have other problems; you’ll need other solutions Misuse of tools should prompt us to rethink the problem What are we actually trying to solve? What are the costs, benefits, and risks?
- 31. What’stheProblem? Going back to basics Are we solving the problem, or treating symptoms? Launch-on-neutral, run-til-significant, peeking, etc. are symptoms, not the root problem Lots of advanced techniques speed up tests, but don’t actually address reasons for impatience
- 32. Here,there,andeverywhere You’re soaking in it This isn’t just about A/B testing But it’s a domain where we have very familiar tools close at hand
- 33. Whatareweherefor? People who solve problems for people are the luckiest people in the world This is the fun stuff This is where we add value as data scientists These problems aren’t solved Try new stuff!
- 34. Carl Vogel Principal Data Scientist carl.vogel@babylist.com Thanks!