Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
6 Guidelines for A/B Testing
Emily Robinson
@robinson_es
Experimentation Process
Prioritizing Creating Analyzing
Decision-
Making
Experimentation Process
Prioritizing Creating Analyzing
Decision-
Making
Three tests idea to increase registration
Test Idea 3: Revamp the course
pages
Test Idea 1: Preload sign-up
modal on our l...
One way to pick …
Prioritizing: Estimate number of new conversion per day
What’s the population of your test idea?
What’s the current conver...
Gathering numbers
Page Visitors per Day
Learning Platform 70,000
Search page 10,000
Course page 40,000
Gathering numbers
Page Visitors per Day Conversion Rate
Learning Platform 70,000 2%
Search page 10,000 21%
Course page 40,...
Gathering numbers
Page Visitors per Day Conversion Rate Estimated Impact
Learning Platform 70,000 2% 5%
Search page 10,000...
Gathering numbers
Page Visitors per Day Conversion Rate Estimated Impact
Learning Platform 70,000 2% 5%
Search page 10,000...
Calculating Impact
Page Visitors per Day Conversion Rate Estimated Impact New Conversions
per Day
Learning Platform 70,000...
Picking one test
Page Visitors per Day Conversion Rate Estimated Impact New Conversions
per Day
Homepage 70,000 2% 5% 70
P...
Experimentation Process
Prioritizing Creating Analyzing
Decision-
Making
Creating: Run a Power Analysis
• 80% Power: if there’s an effect of this size, there’s an 80% chance
you will detect it
• ...
Power Calculator
bookingcom.github.io/powercalculator
Deciding on changes to try
What could
improve
registration on
the course page?
Add registration form on page?
Add user tes...
… Two Months Later
Oh no, we spent two
months working and
registrations didn’t go
up! Why didn’t it
work?
Data Scientist:
Creating: Work on Smaller, Incremental Tests
Creating: Work on Smaller, Incremental Tests
Creating: Work on Smaller, Incremental Tests
Picking one change
add registration form on page
Run it and dump the data on the data scientist?
We finished the
experiment! Can you
analyze it for us?
“To consult the sta...
Creating: Have a Data Scientist Involved
• Can make sure you’re tracking the right metrics
• Can help with the power calcu...
Experimentation Process
Prioritizing Speccing Analyzing
Decision-
Making
Things that have happened
➔ People are put in both control and treatment
➔ People in the experiment have no page views
➔ P...
Analyzing: Have Health Checks
Analyzing: Have Health Checks
* These are fake numbers
Analyzing: Have Health Checks
Experimentation Process
Prioritizing Speccing Analyzing
Decision-
Making
Decision-Making: Be careful of launching on “neutral”
Decision-Making: Be careful of launching on “neutral”
• What is the technical complexity and debt you’re adding?
• Is it a...
Conclusion
Experimentation Process
Prioritizing
Creating
Analyzing
Decision-
Making
• Calculate potential increase
• Run a power calc...
Research papers
➔ Controlled experiments on the web: survey and practical guide (2008)
➔ Seven rules of thumb for web site...
Blog posts and presentations
➔ Design for Continuous Experimentation by Dan McKinley
➔ Scaling Airbnb’s Experimentation Pl...
Thank you!
hookedondata.org
@robinson_es
Upcoming SlideShare
Loading in …5
×

6 Guidelines for A/B Testing

59 views

Published on

Given at CXL Live 2019.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

6 Guidelines for A/B Testing

  1. 1. 6 Guidelines for A/B Testing Emily Robinson @robinson_es
  2. 2. Experimentation Process Prioritizing Creating Analyzing Decision- Making
  3. 3. Experimentation Process Prioritizing Creating Analyzing Decision- Making
  4. 4. Three tests idea to increase registration Test Idea 3: Revamp the course pages Test Idea 1: Preload sign-up modal on our learning platform Test Idea 2: Add recommended courses to top of search page
  5. 5. One way to pick …
  6. 6. Prioritizing: Estimate number of new conversion per day What’s the population of your test idea? What’s the current conversion rate? What’s your estimated increase? Then: Multiply to get new daily conversions
  7. 7. Gathering numbers Page Visitors per Day Learning Platform 70,000 Search page 10,000 Course page 40,000
  8. 8. Gathering numbers Page Visitors per Day Conversion Rate Learning Platform 70,000 2% Search page 10,000 21% Course page 40,000 10%
  9. 9. Gathering numbers Page Visitors per Day Conversion Rate Estimated Impact Learning Platform 70,000 2% 5% Search page 10,000 21% 2% Course page 40,000 10% 3%
  10. 10. Gathering numbers Page Visitors per Day Conversion Rate Estimated Impact Learning Platform 70,000 2% 5% Search page 10,000 21% 2% Course page 40,000 10% 3% VD x CR EIx = NCD
  11. 11. Calculating Impact Page Visitors per Day Conversion Rate Estimated Impact New Conversions per Day Learning Platform 70,000 2% 5% 70 Search page 10,000 21% 2% 42 Course page 40,000 10% 3% 120 VD x CR EIx = NCD
  12. 12. Picking one test Page Visitors per Day Conversion Rate Estimated Impact New Conversions per Day Homepage 70,000 2% 5% 70 Pricing page 10,000 21% 2% 42 Course page 40,000 10% 3% 120 VD x CR EIx = NCD
  13. 13. Experimentation Process Prioritizing Creating Analyzing Decision- Making
  14. 14. Creating: Run a Power Analysis • 80% Power: if there’s an effect of this size, there’s an 80% chance you will detect it • How long will you need to run your experiment for to get 80% power?
  15. 15. Power Calculator bookingcom.github.io/powercalculator
  16. 16. Deciding on changes to try What could improve registration on the course page? Add registration form on page? Add user testimonials? Preload registration modal? Add company logos? Add video? Why not all of them!
  17. 17. … Two Months Later Oh no, we spent two months working and registrations didn’t go up! Why didn’t it work? Data Scientist:
  18. 18. Creating: Work on Smaller, Incremental Tests
  19. 19. Creating: Work on Smaller, Incremental Tests
  20. 20. Creating: Work on Smaller, Incremental Tests
  21. 21. Picking one change add registration form on page
  22. 22. Run it and dump the data on the data scientist? We finished the experiment! Can you analyze it for us? “To consult the statistician after an experiment is finished is often merely to ask [them] to conduct a post mortem examination. [They] can perhaps say what the experiment died of.” - Sir R.A. Fisher
  23. 23. Creating: Have a Data Scientist Involved • Can make sure you’re tracking the right metrics • Can help with the power calculation • Can help iterate on ideas
  24. 24. Experimentation Process Prioritizing Speccing Analyzing Decision- Making
  25. 25. Things that have happened ➔ People are put in both control and treatment ➔ People in the experiment have no page views ➔ People have multiple experiment starts in the same group ➔ Experiment starts didn’t have cookies (so we couldn’t track user) ➔ People weren’t split evenly between control and treatment
  26. 26. Analyzing: Have Health Checks
  27. 27. Analyzing: Have Health Checks * These are fake numbers
  28. 28. Analyzing: Have Health Checks
  29. 29. Experimentation Process Prioritizing Speccing Analyzing Decision- Making
  30. 30. Decision-Making: Be careful of launching on “neutral”
  31. 31. Decision-Making: Be careful of launching on “neutral” • What is the technical complexity and debt you’re adding? • Is it a foundational feature? • Could be a negative impact too small to detect • Decide beforehand whether you’ll launch it on “neutral”
  32. 32. Conclusion
  33. 33. Experimentation Process Prioritizing Creating Analyzing Decision- Making • Calculate potential increase • Run a power calculation • Work on smaller, incremental tests • Have a Data Scientist involved • Monitor health metrics • Be careful of launching on “neutral”
  34. 34. Research papers ➔ Controlled experiments on the web: survey and practical guide (2008) ➔ Seven rules of thumb for web site experiments (2014) ➔ A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments (2017) ➔ Democratizing online controlled experiments at Booking.com (2017)
  35. 35. Blog posts and presentations ➔ Design for Continuous Experimentation by Dan McKinley ➔ Scaling Airbnb’s Experimentation Platform by Jonathan Parks ➔ Please, please don’t A/B test that by Tal Raviv ➔ How Etsy handles peeking in A/B Testing by Callie McRee and Kelly Shen
  36. 36. Thank you! hookedondata.org @robinson_es

×