Recommended
More Related Content
What's hot
What's hot (20)
Similar to 6 Guidelines for A/B Testing
Similar to 6 Guidelines for A/B Testing (20)
Recently uploaded
Recently uploaded (20)
6 Guidelines for A/B Testing
- 1. 6 Guidelines for A/B Testing Emily Robinson @robinson_es
- 2. Experimentation Process Prioritizing Creating Analyzing Decision- Making
- 3. Experimentation Process Prioritizing Creating Analyzing Decision- Making
- 4. Three tests idea to increase registration Test Idea 3: Revamp the course pages Test Idea 1: Preload sign-up modal on our learning platform Test Idea 2: Add recommended courses to top of search page
- 5. One way to pick …
- 6. Prioritizing: Estimate number of new conversion per day What’s the population of your test idea? What’s the current conversion rate? What’s your estimated increase? Then: Multiply to get new daily conversions
- 7. Gathering numbers Page Visitors per Day Learning Platform 70,000 Search page 10,000 Course page 40,000
- 8. Gathering numbers Page Visitors per Day Conversion Rate Learning Platform 70,000 2% Search page 10,000 21% Course page 40,000 10%
- 9. Gathering numbers Page Visitors per Day Conversion Rate Estimated Impact Learning Platform 70,000 2% 5% Search page 10,000 21% 2% Course page 40,000 10% 3%
- 10. Gathering numbers Page Visitors per Day Conversion Rate Estimated Impact Learning Platform 70,000 2% 5% Search page 10,000 21% 2% Course page 40,000 10% 3% VD x CR EIx = NCD
- 11. Calculating Impact Page Visitors per Day Conversion Rate Estimated Impact New Conversions per Day Learning Platform 70,000 2% 5% 70 Search page 10,000 21% 2% 42 Course page 40,000 10% 3% 120 VD x CR EIx = NCD
- 12. Picking one test Page Visitors per Day Conversion Rate Estimated Impact New Conversions per Day Homepage 70,000 2% 5% 70 Pricing page 10,000 21% 2% 42 Course page 40,000 10% 3% 120 VD x CR EIx = NCD
- 13. Experimentation Process Prioritizing Creating Analyzing Decision- Making
- 14. Creating: Run a Power Analysis • 80% Power: if there’s an effect of this size, there’s an 80% chance you will detect it • How long will you need to run your experiment for to get 80% power?
- 16. Deciding on changes to try What could improve registration on the course page? Add registration form on page? Add user testimonials? Preload registration modal? Add company logos? Add video? Why not all of them!
- 17. … Two Months Later Oh no, we spent two months working and registrations didn’t go up! Why didn’t it work? Data Scientist:
- 18. Creating: Work on Smaller, Incremental Tests
- 19. Creating: Work on Smaller, Incremental Tests
- 20. Creating: Work on Smaller, Incremental Tests
- 21. Picking one change add registration form on page
- 22. Run it and dump the data on the data scientist? We finished the experiment! Can you analyze it for us? “To consult the statistician after an experiment is finished is often merely to ask [them] to conduct a post mortem examination. [They] can perhaps say what the experiment died of.” - Sir R.A. Fisher
- 23. Creating: Have a Data Scientist Involved • Can make sure you’re tracking the right metrics • Can help with the power calculation • Can help iterate on ideas
- 24. Experimentation Process Prioritizing Speccing Analyzing Decision- Making
- 25. Things that have happened ➔ People are put in both control and treatment ➔ People in the experiment have no page views ➔ People have multiple experiment starts in the same group ➔ Experiment starts didn’t have cookies (so we couldn’t track user) ➔ People weren’t split evenly between control and treatment
- 26. Analyzing: Have Health Checks
- 27. Analyzing: Have Health Checks * These are fake numbers
- 28. Analyzing: Have Health Checks
- 29. Experimentation Process Prioritizing Speccing Analyzing Decision- Making
- 30. Decision-Making: Be careful of launching on “neutral”
- 31. Decision-Making: Be careful of launching on “neutral” • What is the technical complexity and debt you’re adding? • Is it a foundational feature? • Could be a negative impact too small to detect • Decide beforehand whether you’ll launch it on “neutral”
- 32. Conclusion
- 33. Experimentation Process Prioritizing Creating Analyzing Decision- Making • Calculate potential increase • Run a power calculation • Work on smaller, incremental tests • Have a Data Scientist involved • Monitor health metrics • Be careful of launching on “neutral”
- 34. Research papers ➔ Controlled experiments on the web: survey and practical guide (2008) ➔ Seven rules of thumb for web site experiments (2014) ➔ A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments (2017) ➔ Democratizing online controlled experiments at Booking.com (2017)
- 35. Blog posts and presentations ➔ Design for Continuous Experimentation by Dan McKinley ➔ Scaling Airbnb’s Experimentation Platform by Jonathan Parks ➔ Please, please don’t A/B test that by Tal Raviv ➔ How Etsy handles peeking in A/B Testing by Callie McRee and Kelly Shen