Your SlideShare is downloading. ×
0
Measuring For Impact: Knowing
What, and How to A/B Test
@mike_greenfield
CEO/Co-Founder, Laserlike
2014-08-07
@mike_greenf...
You know you should A/B test.
@mike_greenfield
You also know you should
exercise more
eat less sugar
spend less on coffee
wear sunscreen
etc., etc.
@mike_greenfield
(Don’t worry, I’m not going to
say anything else about sugar
or sunscreen.)
@mike_greenfield
So, how do you create a culture
in which people will
constructively A/B test?
Do six things.
@mike_greenfield
1. Embrace “I don’t know”
We have 2+ ideas.
I don’t know which one will be more
effective.
@mike_greenfield
@mike_greenfield
2. Have Data, Choose
Metrics
To test, you need:
• People using your product
• (Approximate) agreement on the metrics
that ...
Not Many Users? Don’t A/B test!
• Laserlike, has ~60 users and has never
run an A/B test
• We will run many, many tests wh...
Know What You Want to Optimize
• If it’s important, you should be running
tests to improve it
• If it’s not important, spe...
3. Have Clear Process, Tech
for Testing
@mike_greenfield
A/B Testing Process
• New feature: if possible, roll out to a small
test subset first (10s or 100s of thousands)
• Version...
A/B Testing Tech
• Using a third party testing service is akin to
building your site on Wordpress: great at
some scales/co...
4. Understand the Math of
What to Test
@mike_greenfield
Process: Same vs. New Tweak
• What’s the probability your tweak will have
a positive effect?
• What kind of effect might t...
Process: Same vs. Big Change
• What’s the probability that your change will
have a negative impact?
• How big an impact mi...
A/B Test for Quality
• Circle of Moms: test “warning” users when
questions seemed short, low quality
• Resulting questions...
5. Understand the Math of
Picking Winners
@mike_greenfield
Resolving Too Soon vs. Resolving
Too Late
• How big is the potential audience for this
test?
• Example 1: end of year “mos...
Longitudinal Tests vs. Immediate
Tests
• Longitudinal: change home page, email
frequency, product framing
• Need to examin...
Automatically Resolve Tests?
• Longitudinal tests should not be
automatically resolved
• Example: new home page design
• I...
Choose robust statistics
• Bad: # of page views
• Good: % of users viewing at least [5, 25,
100] pages
• Potentially bad: ...
6. Celebrate A/B Testing
Successes
@mike_greenfield
@mike_greenfield
Thanks.
mike@laserlike.com
@mike_greenfield
@mike_greenfield
Upcoming SlideShare
Loading in...5
×

[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test

1,549

Published on

[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test

Published in: Internet

Transcript of "[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test"

  1. 1. Measuring For Impact: Knowing What, and How to A/B Test @mike_greenfield CEO/Co-Founder, Laserlike 2014-08-07 @mike_greenfield
  2. 2. You know you should A/B test. @mike_greenfield
  3. 3. You also know you should exercise more eat less sugar spend less on coffee wear sunscreen etc., etc. @mike_greenfield
  4. 4. (Don’t worry, I’m not going to say anything else about sugar or sunscreen.) @mike_greenfield
  5. 5. So, how do you create a culture in which people will constructively A/B test? Do six things. @mike_greenfield
  6. 6. 1. Embrace “I don’t know” We have 2+ ideas. I don’t know which one will be more effective. @mike_greenfield
  7. 7. @mike_greenfield
  8. 8. 2. Have Data, Choose Metrics To test, you need: • People using your product • (Approximate) agreement on the metrics that matter @mike_greenfield
  9. 9. Not Many Users? Don’t A/B test! • Laserlike, has ~60 users and has never run an A/B test • We will run many, many tests when we have enough users • A test should have at least a few hundred instances (and a lot more if effect sizes are likely to be small) • Test iff you can have “business significance” @mike_greenfield
  10. 10. Know What You Want to Optimize • If it’s important, you should be running tests to improve it • If it’s not important, spend time on other things • Most tests should be aimed at improving 1-2 specific variables @mike_greenfield
  11. 11. 3. Have Clear Process, Tech for Testing @mike_greenfield
  12. 12. A/B Testing Process • New feature: if possible, roll out to a small test subset first (10s or 100s of thousands) • Version change: always test things that could (cumulatively) have business impact • Everyone on the product team should be running and resolving tests @mike_greenfield
  13. 13. A/B Testing Tech • Using a third party testing service is akin to building your site on Wordpress: great at some scales/competency levels • No matter how you’re testing, a new test should be at most a few lines of code • It should be easy to see how each side of a test compares across many variables @mike_greenfield
  14. 14. 4. Understand the Math of What to Test @mike_greenfield
  15. 15. Process: Same vs. New Tweak • What’s the probability your tweak will have a positive effect? • What kind of effect might that have, and how might that effect change the company’s prospects? • Will you be able to measure the change? • Optimize on one variable, but look at others @mike_greenfield
  16. 16. Process: Same vs. Big Change • What’s the probability that your change will have a negative impact? • How big an impact might there be? • Will you be able to measure the change? • Holistic approach @mike_greenfield
  17. 17. A/B Test for Quality • Circle of Moms: test “warning” users when questions seemed short, low quality • Resulting questions were graded for quality, without grader knowing test bucket • End result: warning yielded ~5% fewer questions, but much higher quality @mike_greenfield
  18. 18. 5. Understand the Math of Picking Winners @mike_greenfield
  19. 19. Resolving Too Soon vs. Resolving Too Late • How big is the potential audience for this test? • Example 1: end of year “most popular baby names” email that will never be sent again • Example 2: Facebook signup flow @mike_greenfield
  20. 20. Longitudinal Tests vs. Immediate Tests • Longitudinal: change home page, email frequency, product framing • Need to examine effect over a long period • Immediate: change button color, email subject • Likely that long-term effects will be minimal @mike_greenfield
  21. 21. Automatically Resolve Tests? • Longitudinal tests should not be automatically resolved • Example: new home page design • Immediate tests can be automatically resolved when speed is important and there is one clear objective function • Example: Circle of Moms email subject optimization @mike_greenfield
  22. 22. Choose robust statistics • Bad: # of page views • Good: % of users viewing at least [5, 25, 100] pages • Potentially bad: # of sales (when small) • Potentially good: # of people getting through the second step of a sales funnel @mike_greenfield
  23. 23. 6. Celebrate A/B Testing Successes @mike_greenfield
  24. 24. @mike_greenfield
  25. 25. Thanks. mike@laserlike.com @mike_greenfield @mike_greenfield
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×