Downloaded 96 times
More Related Content
Similar to Building an A/B Testing Analytics System with R and Shiny
Building an A/B Testing Analytics System with R and Shiny
- 1. Building an A/BTesting Analytics System with R and ShinyEmily Robinson @robinson_es
- 2. About Me ➔ DataScientist at DataCamp ➔ R user ~7 years ➔ Enjoy talking about: ◆ Building and finding data science community ◆ Diversity in STEM ◆ R
- 3.
- 4. What is A/BTesting?
- 5. Life B.D. (BeforeDataCamp) ➔ Worked on 60+ experiments with search team ➔ 8+ year history of experimentation ➔ 500+ experiments per year
- 6. Life B.D. (BeforeDataCamp) ➔ 5 data engineers working on the experimentation platform ➔ Over a thousand metrics computed for each experiment ➔ Fancy UI From How Etsy Handles Peeking in A/B Testing by Callie McRee and Kelly Shen
- 7. First weeks atDataCamp ➔ No system for planning, analyzing, or presenting experiment results ➔ And no data engineers to build it
- 8.
- 9. 1. Build toolsto save yourself time
- 10. Who here hashad a “first this then that” question? ➔ Who tried X then did Y? ➔ What percent of people who did X then did Y? ➔ What was the last thing people did before doing Y? ➔ What are all the things people did after doing X?
- 11. Questions I mightanswer about an A/B test ➔ What percent of people in the treatment vs. control registered? ➔ What were the ad clicks that had a course start within 2 days?
- 12. Lengthy, repetitive code ➔Lots of copying and pasting ➔ Hard to switch between types of funnels
- 13. And when you’redoing repetitive tasks ... Package
- 14. Unfortunately … Me andwriting packages
- 15. Fortunately … I hadDavid Robinson Sorry, this David Robinson
- 16.
- 17. Structure 1. Table 1 2.Table 2 3. User column name(s) 4. Time column name(s) 5. Type of afterjoin 6. Type of join
- 18. Example: first-any ➔ Whatare all the courses people started after visiting the homepage for the first time?
- 19. Example: first-firstafter ➔ Whatpercent of people saw the pricing page and then subscribed?
- 20. Example: max-gap argument ➔What percent of people saw the pricing page and then subscribed within four days?
- 21. ➔ Many funnel-types: ➔Lastbefore-firstafter, any-any, first-any … ➔ Supports all types of dplyr joins: ➔ Inner, left, right, full, semi, and anti ➔ Works on remote tables ➔ Bug fixes, pull requests, feature requests welcome ➔ Try it yourself! Funneljoin: github.com/datacamp/funneljoin
- 22. 2. Everything thatcan go wrong, will go wrong
- 23. Things that havehappened … ➔ People are put in both control and treatment ➔ People in the experiment have no page views ➔ People have multiple experiment starts in the same group ➔ There aren’t the same number of people in control and treatment ➔ Experiment starts didn’t have cookies (so we couldn’t track user)
- 24. You need tocheck your assumptions
- 25.
- 26. As a famousdata scientist once said … When you’ve run the same process three times, make a dashboard
- 27. 3. Build toolsthat empower others
- 28. Health Metrics Dashboard *These are fake numbers
- 29. By metric view *These are fake numbers
- 30. By metric view *These are fake numbers
- 31. Individual experiments view *These are fake numbers
- 32. Leveling up … ➔Common request: What % increase can we detect in a 2 week test? ➔ Can I make a tool so people can answer this themselves without code? ➔ Delivering information -> discovering information
- 33.
- 34.
- 35. 4. Make iteasy to do the right thing
- 36. ➔ Clarifies decision-making ➔Can have additional “guardrail” metrics that you don’t want to negatively impact Best Practice 1: Have one key metric per experiment
- 37.
- 38. Best practice 2:Run your experiment for length you’re planned on ➔ Otherwise, you may quadruple your false positive rate!
- 39. Show start andend date in dashboard
- 40.
- 41. Recap 1. Build toolsto save yourself time 2. Everything that can go wrong will go wrong 3. Build tools that empower others 4. Make it easy to do the right thing
- 42. Many thanks to… ➔ The growth and data science teams at DataCamp ➔ Anthony Baker & David Robinson, co-authors of funneljoin ➔ Analytics & Data Engineering team at Etsy
- 43.