Successfully reported this slideshow.

A/B Testing with Mike Greenfield

9,136 views

Published on

Everything you need to know to perform a kickass A/B test – start testing today!

Published in: Technology, Business

A/B Testing with Mike Greenfield

  1. The Art of A/B Testing@mike_greenfieldnumeratechoir.com500 Startups, 2013-05-08
  2. What’s an A/B Test?Definition: An A/B Test is a means by whicha product’s users are randomly givenone of two or more experiences.Usage: Companies use A/B Tests todiscover which experience is mosteffective.
  3. A/B Testing OverviewWhy You Should A/B TestWhat You Should A/B TestWhen You Should A/B TestHow You Should A/B TestWhat I’ve Learned from A/B Testing
  4. Why You Should A/B Test
  5. Humans’ quantitative intuition is poorBeliefs:1) “I have great product intuition”2) “This is business, not science”3) “It leads to a local maximum”Reality:At Internet scale, testing and measuringproperly has a huge return on investment.
  6. “I have great product intuition”• Improved signup flow: often a disaster• Sharing on Facebook: can be good orbad• Home page changes: usually a mixedbag…but product changes != progress
  7. “This is business, not science”• Requires oversight and sleuthing• Only works for big improvements• Requires other factors to stay the same…but change-then-measure is flawed
  8. Test Test Type GoalChange ButtonColorOptimization Increased clickthroughOld Site Designvs. New SiteDesignHolistic (aka don’tshoot yourself inthe foot)Make an informeddecision on old vs. new“It leads to a local maximum”…so test holistically
  9. What You Should A/B Test
  10. Test an almost viral flow1000signups300invitingfriends3000invitations1500invitationclicks900signupsK-factor = 0.9☑ A/B Test this flow!
  11. Don’t test a non-viral flow1000signups100invitingfriends600invitations200invitationclicks80signupsK-factor = 0.08Spend time elsewhere
  12. Test emailsubject linesSubject: The Craziest ThingMy Child Has DoneSubject: 5 Embarrassing KidMomentsvs.
  13. Test emailsubject linesSubject: The Craziest ThingMy Child Has DoneSubject: 5 Embarrassing KidMomentsvs.+153%
  14. Improve purchase funnelGet 2011 PicksGet Bracketbrainsvs.
  15. Improve purchase funnelGet 2011 PicksGet Bracketbrainsvs.+30%
  16. Validate your new design** if you have scaleScale What to Do ReasoningEarly stage Just change it. Nothing to lose, no data totest.Something to lose A/B test it. The existing product isprobably more effectivethan you think.Understand theconsequences of an“upgrade.”
  17. When You Should A/B Test
  18. Test only if you can get tostatistical significance(Google: “split test calculator”)Big: 200,000emails with a 3%CTR option and a4% CTR option-------------------------6000 vs. 8000Small: 200 emailswith a 3% CTRoption and a 4%CTR option--------------------------6 vs. 8
  19. Test only if you can get to businesssignificance• Only things that can cumulativelyhave a meaningful impact on yourbusiness• For emails, a small list means smallimprovements: don’t test• For virality, small changes matter if andonly if you’re close to K=1
  20. Rule of thumb: test every user-facing change that will be seenby 10,000-100,000 people
  21. How You Should A/B Test
  22. Okay Choice: Use CommercialTools• MixPanel• Unbounce• Optimizely• Google Analytics
  23. Best Choice: Build Your OwnFramework• Yep, it’s work with no immediatepayoff• Your mom won’t care• Your users won’t careBut…• There are simple ways to get started• It gives you tons of flexibility
  24. Why Build Your Own• Incorporate tests in many places(page ordering, new designs, emailcontent, email strategy, mobile)• Look at results holistically• Go back and see how any testinfluences anything, not just the statsyou’re tracking
  25. CodeIt needs to be super simple to create a test{ab_test_if test=“signup_reason”option=“awesome” user=$viewer}because it’s awesome!{/ab_test_if}{ab_test_if test=“signup_reason” option=“free”user=$viewer}because it’s free!{/ab_test_if}
  26. Data StructureAB_TESTS (id, name, time_created)AB_TEST_OPTIONS(id, ab_test_id, weight, name)USER_AB_TEST_OPTIONS(id, user_id/visitor_id, ab_test_option_id, time_created)
  27. ReportingselectAB_TEST_OPTION_ID,ACTIVITY_ID,count(distinct USER_ID) USERS_DOING_ACTIVITYfrom USER_ACTIVITY a, USER_AB_TEST_OPTIONS bwhere a.USER_ID=b.USER_IDand AB_TEST_OPTION_ID in (…)and ACTIVITY_ID in (…)and a.time_created>b.time_createdGROUP BY AB_TEST_OPTION_ID, ACTIVITY_ID;
  28. Scaling• An A/B testing system can yield a lot ofDB writes• Reporting means many long-runningSQL queries• Need to batch several aspects
  29. Run Some A/A Tests• Test two versions of the same thing• If results are wildly off, something’swrong with the testing system• Deciding too early is a major issue: it’susually best to be conservative beforechoosing a winner
  30. More framework implementationdetails: bit.ly/artofabtesting
  31. What I’ve learned from A/B testing
  32. Focus on 1 item in emails• Clear subject focused on that item (Whythe Giants will win the World Series)• Body of text focused on that item(peripheral content is okay on theperiphery)• Clear, big clickthrough action in theemail body (See why the Giants will win)• Require clickthrough to get the full story
  33. In signup, minimize distraction• Provide context/messaging of whatthe product is, but don’t make itclickable• Clear “next” or “continue” steps toguide user through the process• Remove unnecessary navigation
  34. Highlight friends, not your product• Most effective: your friends are doingsomething; you should join them• Unless you’re Apple, no one caresabout your new feature or new design• People probably don’t care aboutyour fancy new social network• “Join my circle because I trust you”beats “check out this great product”
  35. When possible,TELL USERS WHAT TO DO
  36. 3 Things to Remember
  37. A/B testing = good culture• Data trump opinions• Iterate quickly but intelligently• Everyone gets better at predictingproduct success
  38. Test changes if they’re likely tohave both statistical and businesssignificance
  39. Validate the Big StuffA/B Test Holistically; testingisn’t a substitute for productvision.Optimize the Small StuffThe details matter more than youthink.
  40. Thanks.mike@mikegreenfield.com@mike_greenfield

×