Your SlideShare is downloading. ×

A/B Testing with Mike Greenfield


Published on

Everything you need to know to perform a kickass A/B test – start testing today!

Everything you need to know to perform a kickass A/B test – start testing today!

Published in: Technology, Business
1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. The Art of A/B Testing@mike_greenfieldnumeratechoir.com500 Startups, 2013-05-08
  • 2. What’s an A/B Test?Definition: An A/B Test is a means by whicha product’s users are randomly givenone of two or more experiences.Usage: Companies use A/B Tests todiscover which experience is mosteffective.
  • 3. A/B Testing OverviewWhy You Should A/B TestWhat You Should A/B TestWhen You Should A/B TestHow You Should A/B TestWhat I’ve Learned from A/B Testing
  • 4. Why You Should A/B Test
  • 5. Humans’ quantitative intuition is poorBeliefs:1) “I have great product intuition”2) “This is business, not science”3) “It leads to a local maximum”Reality:At Internet scale, testing and measuringproperly has a huge return on investment.
  • 6. “I have great product intuition”• Improved signup flow: often a disaster• Sharing on Facebook: can be good orbad• Home page changes: usually a mixedbag…but product changes != progress
  • 7. “This is business, not science”• Requires oversight and sleuthing• Only works for big improvements• Requires other factors to stay the same…but change-then-measure is flawed
  • 8. Test Test Type GoalChange ButtonColorOptimization Increased clickthroughOld Site Designvs. New SiteDesignHolistic (aka don’tshoot yourself inthe foot)Make an informeddecision on old vs. new“It leads to a local maximum”…so test holistically
  • 9. What You Should A/B Test
  • 10. Test an almost viral flow1000signups300invitingfriends3000invitations1500invitationclicks900signupsK-factor = 0.9☑ A/B Test this flow!
  • 11. Don’t test a non-viral flow1000signups100invitingfriends600invitations200invitationclicks80signupsK-factor = 0.08Spend time elsewhere
  • 12. Test emailsubject linesSubject: The Craziest ThingMy Child Has DoneSubject: 5 Embarrassing KidMomentsvs.
  • 13. Test emailsubject linesSubject: The Craziest ThingMy Child Has DoneSubject: 5 Embarrassing KidMomentsvs.+153%
  • 14. Improve purchase funnelGet 2011 PicksGet Bracketbrainsvs.
  • 15. Improve purchase funnelGet 2011 PicksGet Bracketbrainsvs.+30%
  • 16. Validate your new design** if you have scaleScale What to Do ReasoningEarly stage Just change it. Nothing to lose, no data totest.Something to lose A/B test it. The existing product isprobably more effectivethan you think.Understand theconsequences of an“upgrade.”
  • 17. When You Should A/B Test
  • 18. Test only if you can get tostatistical significance(Google: “split test calculator”)Big: 200,000emails with a 3%CTR option and a4% CTR option-------------------------6000 vs. 8000Small: 200 emailswith a 3% CTRoption and a 4%CTR option--------------------------6 vs. 8
  • 19. Test only if you can get to businesssignificance• Only things that can cumulativelyhave a meaningful impact on yourbusiness• For emails, a small list means smallimprovements: don’t test• For virality, small changes matter if andonly if you’re close to K=1
  • 20. Rule of thumb: test every user-facing change that will be seenby 10,000-100,000 people
  • 21. How You Should A/B Test
  • 22. Okay Choice: Use CommercialTools• MixPanel• Unbounce• Optimizely• Google Analytics
  • 23. Best Choice: Build Your OwnFramework• Yep, it’s work with no immediatepayoff• Your mom won’t care• Your users won’t careBut…• There are simple ways to get started• It gives you tons of flexibility
  • 24. Why Build Your Own• Incorporate tests in many places(page ordering, new designs, emailcontent, email strategy, mobile)• Look at results holistically• Go back and see how any testinfluences anything, not just the statsyou’re tracking
  • 25. CodeIt needs to be super simple to create a test{ab_test_if test=“signup_reason”option=“awesome” user=$viewer}because it’s awesome!{/ab_test_if}{ab_test_if test=“signup_reason” option=“free”user=$viewer}because it’s free!{/ab_test_if}
  • 26. Data StructureAB_TESTS (id, name, time_created)AB_TEST_OPTIONS(id, ab_test_id, weight, name)USER_AB_TEST_OPTIONS(id, user_id/visitor_id, ab_test_option_id, time_created)
  • 28. Scaling• An A/B testing system can yield a lot ofDB writes• Reporting means many long-runningSQL queries• Need to batch several aspects
  • 29. Run Some A/A Tests• Test two versions of the same thing• If results are wildly off, something’swrong with the testing system• Deciding too early is a major issue: it’susually best to be conservative beforechoosing a winner
  • 30. More framework implementationdetails:
  • 31. What I’ve learned from A/B testing
  • 32. Focus on 1 item in emails• Clear subject focused on that item (Whythe Giants will win the World Series)• Body of text focused on that item(peripheral content is okay on theperiphery)• Clear, big clickthrough action in theemail body (See why the Giants will win)• Require clickthrough to get the full story
  • 33. In signup, minimize distraction• Provide context/messaging of whatthe product is, but don’t make itclickable• Clear “next” or “continue” steps toguide user through the process• Remove unnecessary navigation
  • 34. Highlight friends, not your product• Most effective: your friends are doingsomething; you should join them• Unless you’re Apple, no one caresabout your new feature or new design• People probably don’t care aboutyour fancy new social network• “Join my circle because I trust you”beats “check out this great product”
  • 35. When possible,TELL USERS WHAT TO DO
  • 36. 3 Things to Remember
  • 37. A/B testing = good culture• Data trump opinions• Iterate quickly but intelligently• Everyone gets better at predictingproduct success
  • 38. Test changes if they’re likely tohave both statistical and businesssignificance
  • 39. Validate the Big StuffA/B Test Holistically; testingisn’t a substitute for productvision.Optimize the Small StuffThe details matter more than youthink.
  • 40.