5. Life B.D. (Before DataCamp)
➔ Worked on 60+
experiments with search
➔ 8+ year history of
➔ 500+ experiments per year
6. Life B.D. (Before DataCamp)
➔ 5 data engineers working on the experimentation platform
➔ Over a thousand metrics computed for each experiment
➔ Fancy UI
From How Etsy Handles Peeking in A/B Testing by Callie McRee and Kelly Shen
7. First weeks at DataCamp
➔ No system for planning,
analyzing, or presenting
➔ And no data engineers to
10. Who here has had a “first this then that” question?
➔ Who tried X then did Y?
➔ What percent of people who did X then did Y?
➔ What was the last thing people did before doing Y?
➔ What are all the things people did after doing X?
11. Questions I might answer about an A/B test
➔ What percent of people in the treatment vs. control registered?
➔ What were the ad clicks that had a course start within 2 days?
23. Things that have happened …
➔ People are put in both control and treatment
➔ People in the experiment have no page views
➔ People have multiple experiment starts in the same group
➔ There aren’t the same number of people in control and treatment
➔ Experiment starts didn’t have cookies (so we couldn’t track user)
32. Leveling up …
➔ Common request: What % increase can we detect in a 2 week test?
➔ Can I make a tool so people can answer this themselves without code?
➔ Delivering information -> discovering information