[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test


Published on

[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test

Published in: Internet
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test

  1. 1. Measuring For Impact: Knowing What, and How to A/B Test @mike_greenfield CEO/Co-Founder, Laserlike 2014-08-07 @mike_greenfield
  2. 2. You know you should A/B test. @mike_greenfield
  3. 3. You also know you should exercise more eat less sugar spend less on coffee wear sunscreen etc., etc. @mike_greenfield
  4. 4. (Don’t worry, I’m not going to say anything else about sugar or sunscreen.) @mike_greenfield
  5. 5. So, how do you create a culture in which people will constructively A/B test? Do six things. @mike_greenfield
  6. 6. 1. Embrace “I don’t know” We have 2+ ideas. I don’t know which one will be more effective. @mike_greenfield
  7. 7. @mike_greenfield
  8. 8. 2. Have Data, Choose Metrics To test, you need: • People using your product • (Approximate) agreement on the metrics that matter @mike_greenfield
  9. 9. Not Many Users? Don’t A/B test! • Laserlike, has ~60 users and has never run an A/B test • We will run many, many tests when we have enough users • A test should have at least a few hundred instances (and a lot more if effect sizes are likely to be small) • Test iff you can have “business significance” @mike_greenfield
  10. 10. Know What You Want to Optimize • If it’s important, you should be running tests to improve it • If it’s not important, spend time on other things • Most tests should be aimed at improving 1-2 specific variables @mike_greenfield
  11. 11. 3. Have Clear Process, Tech for Testing @mike_greenfield
  12. 12. A/B Testing Process • New feature: if possible, roll out to a small test subset first (10s or 100s of thousands) • Version change: always test things that could (cumulatively) have business impact • Everyone on the product team should be running and resolving tests @mike_greenfield
  13. 13. A/B Testing Tech • Using a third party testing service is akin to building your site on Wordpress: great at some scales/competency levels • No matter how you’re testing, a new test should be at most a few lines of code • It should be easy to see how each side of a test compares across many variables @mike_greenfield
  14. 14. 4. Understand the Math of What to Test @mike_greenfield
  15. 15. Process: Same vs. New Tweak • What’s the probability your tweak will have a positive effect? • What kind of effect might that have, and how might that effect change the company’s prospects? • Will you be able to measure the change? • Optimize on one variable, but look at others @mike_greenfield
  16. 16. Process: Same vs. Big Change • What’s the probability that your change will have a negative impact? • How big an impact might there be? • Will you be able to measure the change? • Holistic approach @mike_greenfield
  17. 17. A/B Test for Quality • Circle of Moms: test “warning” users when questions seemed short, low quality • Resulting questions were graded for quality, without grader knowing test bucket • End result: warning yielded ~5% fewer questions, but much higher quality @mike_greenfield
  18. 18. 5. Understand the Math of Picking Winners @mike_greenfield
  19. 19. Resolving Too Soon vs. Resolving Too Late • How big is the potential audience for this test? • Example 1: end of year “most popular baby names” email that will never be sent again • Example 2: Facebook signup flow @mike_greenfield
  20. 20. Longitudinal Tests vs. Immediate Tests • Longitudinal: change home page, email frequency, product framing • Need to examine effect over a long period • Immediate: change button color, email subject • Likely that long-term effects will be minimal @mike_greenfield
  21. 21. Automatically Resolve Tests? • Longitudinal tests should not be automatically resolved • Example: new home page design • Immediate tests can be automatically resolved when speed is important and there is one clear objective function • Example: Circle of Moms email subject optimization @mike_greenfield
  22. 22. Choose robust statistics • Bad: # of page views • Good: % of users viewing at least [5, 25, 100] pages • Potentially bad: # of sales (when small) • Potentially good: # of people getting through the second step of a sales funnel @mike_greenfield
  23. 23. 6. Celebrate A/B Testing Successes @mike_greenfield
  24. 24. @mike_greenfield
  25. 25. Thanks. mike@laserlike.com @mike_greenfield @mike_greenfield