Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
The Essentials of Community Building by Mack Fogelson


A/B Testing with Mike Greenfield

Everything you need to know to perform a kickass A/B test – start testing today!

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

A/B Testing with Mike Greenfield

  1. The Art of A/B Testing@mike_greenfieldnumeratechoir.com500 Startups, 2013-05-08
  2. What’s an A/B Test?Definition: An A/B Test is a means by whicha product’s users are randomly givenone of two or more experiences.Usage: Companies use A/B Tests todiscover which experience is mosteffective.
  3. A/B Testing OverviewWhy You Should A/B TestWhat You Should A/B TestWhen You Should A/B TestHow You Should A/B TestWhat I’ve Learned from A/B Testing
  4. Why You Should A/B Test
  5. Humans’ quantitative intuition is poorBeliefs:1) “I have great product intuition”2) “This is business, not science”3) “It leads to a local maximum”Reality:At Internet scale, testing and measuringproperly has a huge return on investment.
  6. “I have great product intuition”• Improved signup flow: often a disaster• Sharing on Facebook: can be good orbad• Home page changes: usually a mixedbag…but product changes != progress
  7. “This is business, not science”• Requires oversight and sleuthing• Only works for big improvements• Requires other factors to stay the same…but change-then-measure is flawed
  8. Test Test Type GoalChange ButtonColorOptimization Increased clickthroughOld Site Designvs. New SiteDesignHolistic (aka don’tshoot yourself inthe foot)Make an informeddecision on old vs. new“It leads to a local maximum”…so test holistically
  9. What You Should A/B Test
  10. Test an almost viral flow1000signups300invitingfriends3000invitations1500invitationclicks900signupsK-factor = 0.9☑ A/B Test this flow!
  11. Don’t test a non-viral flow1000signups100invitingfriends600invitations200invitationclicks80signupsK-factor = 0.08Spend time elsewhere
  12. Test emailsubject linesSubject: The Craziest ThingMy Child Has DoneSubject: 5 Embarrassing KidMomentsvs.
  13. Test emailsubject linesSubject: The Craziest ThingMy Child Has DoneSubject: 5 Embarrassing KidMomentsvs.+153%
  14. Improve purchase funnelGet 2011 PicksGet Bracketbrainsvs.
  15. Improve purchase funnelGet 2011 PicksGet Bracketbrainsvs.+30%
  16. Validate your new design** if you have scaleScale What to Do ReasoningEarly stage Just change it. Nothing to lose, no data totest.Something to lose A/B test it. The existing product isprobably more effectivethan you think.Understand theconsequences of an“upgrade.”
  17. When You Should A/B Test
  18. Test only if you can get tostatistical significance(Google: “split test calculator”)Big: 200,000emails with a 3%CTR option and a4% CTR option-------------------------6000 vs. 8000Small: 200 emailswith a 3% CTRoption and a 4%CTR option--------------------------6 vs. 8
  19. Test only if you can get to businesssignificance• Only things that can cumulativelyhave a meaningful impact on yourbusiness• For emails, a small list means smallimprovements: don’t test• For virality, small changes matter if andonly if you’re close to K=1
  20. Rule of thumb: test every user-facing change that will be seenby 10,000-100,000 people
  21. How You Should A/B Test
  22. Okay Choice: Use CommercialTools• MixPanel• Unbounce• Optimizely• Google Analytics
  23. Best Choice: Build Your OwnFramework• Yep, it’s work with no immediatepayoff• Your mom won’t care• Your users won’t careBut…• There are simple ways to get started• It gives you tons of flexibility
  24. Why Build Your Own• Incorporate tests in many places(page ordering, new designs, emailcontent, email strategy, mobile)• Look at results holistically• Go back and see how any testinfluences anything, not just the statsyou’re tracking
  25. CodeIt needs to be super simple to create a test{ab_test_if test=“signup_reason”option=“awesome” user=$viewer}because it’s awesome!{/ab_test_if}{ab_test_if test=“signup_reason” option=“free”user=$viewer}because it’s free!{/ab_test_if}
  26. Data StructureAB_TESTS (id, name, time_created)AB_TEST_OPTIONS(id, ab_test_id, weight, name)USER_AB_TEST_OPTIONS(id, user_id/visitor_id, ab_test_option_id, time_created)
  28. Scaling• An A/B testing system can yield a lot ofDB writes• Reporting means many long-runningSQL queries• Need to batch several aspects
  29. Run Some A/A Tests• Test two versions of the same thing• If results are wildly off, something’swrong with the testing system• Deciding too early is a major issue: it’susually best to be conservative beforechoosing a winner
  30. More framework implementationdetails:
  31. What I’ve learned from A/B testing
  32. Focus on 1 item in emails• Clear subject focused on that item (Whythe Giants will win the World Series)• Body of text focused on that item(peripheral content is okay on theperiphery)• Clear, big clickthrough action in theemail body (See why the Giants will win)• Require clickthrough to get the full story
  33. In signup, minimize distraction• Provide context/messaging of whatthe product is, but don’t make itclickable• Clear “next” or “continue” steps toguide user through the process• Remove unnecessary navigation
  34. Highlight friends, not your product• Most effective: your friends are doingsomething; you should join them• Unless you’re Apple, no one caresabout your new feature or new design• People probably don’t care aboutyour fancy new social network• “Join my circle because I trust you”beats “check out this great product”
  35. When possible,TELL USERS WHAT TO DO
  36. 3 Things to Remember
  37. A/B testing = good culture• Data trump opinions• Iterate quickly but intelligently• Everyone gets better at predictingproduct success
  38. Test changes if they’re likely tohave both statistical and businesssignificance
  39. Validate the Big StuffA/B Test Holistically; testingisn’t a substitute for productvision.Optimize the Small StuffThe details matter more than youthink.
  • basaam07079

    May. 1, 2021
  • chidubemaosuji

    Nov. 10, 2020
  • IlonaKovalova1

    Mar. 12, 2020
  • BullOxy

    Jan. 26, 2020
  • RiverPlacid

    Jul. 27, 2019
  • amjadaghzaf

    Jun. 21, 2019
  • BenyaminNajafi

    Dec. 16, 2018

    Jun. 1, 2018
  • AlexsanderLuizdeQuei

    May. 10, 2018
  • LouisBauer1

    Feb. 7, 2018
  • ssuser88997a

    Nov. 16, 2017
  • olagoketobiwhiki

    Sep. 19, 2017
  • lexwuang

    Dec. 25, 2016
  • rayonhung1

    Nov. 17, 2016
  • DeepakSingh105

    Apr. 20, 2015
  • aelamin

    Feb. 9, 2015
  • tantrieuf31

    Aug. 11, 2014
  • fariborzn1

    Apr. 13, 2014
  • NadezhdaShilova1

    Mar. 5, 2014
  • luvinurie

    Feb. 5, 2014

Everything you need to know to perform a kickass A/B test – start testing today!


Total views


On Slideshare


From embeds


Number of embeds