Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A/B Puzzle

143 views

Published on

This presentation is from Performance Marketing Summit 2017 (March 14, 2017 in Austin, TX). Session description: How A/B tests lie to us, and how to drive genuine improvement.

Published in: Marketing
  • Be the first to comment

  • Be the first to like this

A/B Puzzle

  1. 1. A/B PUZZLE How A/B tests lie to us, and how to drive genuine improvement Jason Cohen, founder & CTO @asmartbear http://wpengine.com
  2. 2. The puzzle
  3. 3. 758 out of 24,483 visits 85% confidence
  4. 4. What do stats really tell us?
  5. 5. 758 out of 24,483 visits 85% confidence
  6. 6. Finding the Cheaters 105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.
  7. 7. Finding the Cheaters IN REALITY Clean (100) Juicing (5) TEST RESULT (95% accuracy) Clean Juicing 5 095 5
  8. 8. Insight: Rare events are indistinguishable from random noise
  9. 9. 758 out of 24,483 visits 85% confidence
  10. 10. Easy significance testing with hamsters Trust in the Hamster. bit.ly/abhamster
  11. 11. Easy significance testing N = A+B = total # of conversions.
 D = (A–B)/2 = half the difference.
 Significant if D2 > N. bit.ly/abhamster N = 758 + 685 = 1443
 D = ( 758 — 685 ) / 2 = 37
 1369 > 1443?
 No! Not significant.
  12. 12. Final Result: As predicted Wasted another month
  13. 13. 30 40 50 60 70 12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11 OtherInbox Splash Page
  14. 14. Trust the Hamster
  15. 15. Beware the Rare
  16. 16. Defeat rare by
 seeking only huge results
  17. 17. Can you A/B test your way to huge results?
  18. 18. 1,116 250 4,100 1,151 Individuals: 67% off Group: 3% off
  19. 19. Wisdom of the Crowd “I’m as clueless as you, but in the equal and opposite direction.”
  20. 20. What’s the funniest joke?
  21. 21. And the winner is…
  22. 22. Jokes, Pizza, Design, Features, Voice, Values, Culture
  23. 23. Crowds find objective truth but kill creativity
  24. 24. Beware the rare. Seek huge results. Don’t let data kill creativity. Jason Cohen, founder & CTO @asmartbear http://wpengine.com

×