Optimizing website conversions through A/B split testing

832 views

Published on

In this presentation Javid Jamae CTO/Co-founder of Camperoo (www.camperoo.com) will teach you how to define, run, and measure A/B tests and other experiments in your Lean Startup. This was presented at the Houston Lean Startup Circle on June 18, 2013.

Published in: Business, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
832
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
28
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Optimizing website conversions through A/B split testing

  1. 1. OPTIMIZINGCONVERSIONSTHROUGHA/B SPLIT TESTINGJavid JamaeCTO / Co-founder CamperooPresented at the Houston Lean Startup CircleJune 18, 2013Special thanks to - EmilianoVillareal
  2. 2. ‘‘The goal of a Lean Startup is to inform ourriskiest business model assumptions throughempirical testing with customers – notrhetorical reasoning on a white board.~ ASH MAURYA
  3. 3. AirBNBZappos
  4. 4. Users coming in through a Zappos-style homepage will be 10% more likelyto go from the homepage into aproduct search results pageEXPERIMENTDESIGNFALSIFIABLE HYPOTHESISEXPERIMENT SCOPE40 man-hours to design and code750 unique visitors per variant
  5. 5. UNIQUE VISITORS RANDOMIZER ANALYTICSCONTROLVARIANTEXPERIMENTIMPLEMENTATION50%50%
  6. 6. THEWINNERWAS?5.86% ??Percentage of people going from the homepageto a product search results pageControl Variant
  7. 7. THEWINNERWAS?5.86% 6.84%16.89% increase!Percentage of people going from the homepageto a product search results pageControl Variant
  8. 8. BUTWHY?Was it the:- Filters- Category browsing- Different slider- The combination?
  9. 9. In Lean Startups, success is definedby learning, not by test resultsXA/BTOOL √ValidatedLearningLoop√Variant
  10. 10. +
  11. 11. Users coming into an AirBNB-stylehomepage with categories will be 5%more likely to go into a product searchresults page than without categoriesEXPERIMENTDESIGNFALSIFIABLE HYPOTHESISEXPERIMENT SCOPE1 man-hours to design and code750 unique visitors per variant
  12. 12. THEWINNERWAS?5.90% ??Percentage of people going from the homepageto a product search results pageControl Variant
  13. 13. THEWINNERWAS?5.90% 5.98%Percentage of people going from the homepageto a product search results pageControl Variant1.35% increase
  14. 14. THELEANPROCESSIDEAHYPOTHESISEXPERIMENT DESIGNEXPERIMENTATIONMEASUREMENTScientific= Process
  15. 15. IDEAHYPOTHESISEXPERIMENT DESIGNEXPERIMENTATIONMEASUREMENTPrioritizeDefineDesignImplementTrackHow do I...
  16. 16. PRIORITIZINGEXPERIMENTS
  17. 17. ‘‘If you measure something and it’s notattached to a goal, in turn changing yourbehavior, you’re wasting your time.ALISTAIR CROLL & BENJAMIN YOSKOVITZBOOK: LEAN ANALYTICS
  18. 18. GOALORIENTEDPLANNINGLanding PageChangesSEORankingsGoal: IncreaseAcquisitionInboundChannels
  19. 19. AcquisitionActivationReferralRevenueRetentionPIRATEMETRICS
  20. 20. OneMetricThatMatters
  21. 21. Lean Analytics (Lean Series). O’Reilly Media.
  22. 22. StageRISKSACROSSSTAGESProductCustomerMarketProblem/SolutionIdentifyCompetition/PositioningEarly / InceptionMVPNarrow/OutboundPricingProductScaleInboundCost StructureScaleRisksQualitative,Activation / RetentionQuantitative,Revenue / Growth
  23. 23. Favor experiments thatmake sense for yourcurrent stage & goals
  24. 24. Favor experiments thatare innovative
  25. 25. Favor experiments thathave high risk
  26. 26. Favor experiments thatyou can measure
  27. 27. Favor experiments thatengaged customers seemto need or are asking for
  28. 28. Theory of ConstraintsA chain is no strongerthan its weakest link.The Goal - Eliyahu M. Goldratt
  29. 29. DEFININGHYPOTHESES
  30. 30. Is ManchesterUnited causingglobal warming?
  31. 31. http://theconversation.com/how-david-beckham-caused-global-warming-the-man-u-climate-model-4548
  32. 32. ‘‘If you torture the data long enough, it willconfess to anything.~UNKNOWN
  33. 33. Don’t takeLeaps of Faith
  34. 34. Leap of faith: A Zappos-style front page will increasebrowsability on our websiteFalsifiable hypothesis: Users coming in through aZappos-style homepage will be 10% more likely to gofrom the homepage into a product search results page(vs. the AirBNB-style homepage)
  35. 35. Formula:[Repeatable Action] will [Expected Result]Writing a blog post about Houston activitieswill result in 25 new accountsUsers entering Variation B of the store pagewill be 2% more likely to purchasePutting an item on sale will increase thechances that it will be added to a cart by 10%
  36. 36. Draw a linein the sand......before youstart gathering data.
  37. 37. DESIGNINGEXPERIMENTS
  38. 38. Are you measuringimpact on aGoal or a Metric
  39. 39. LEARNINGFROMDATAA/B TestingCohortAnalysisAnalysisTestingLongitudinal Cross-SectionalMultivariateTestingSegmentation(Conditional)FeatureRollout
  40. 40. Decide on aexit criteriaup front
  41. 41. CostIt will take 5-10 hours to implement, ifit takes longer, we will abandon.ScopeWe need 1,000 unique visitors per variantbefore the data has any meaning to us.TimeboxIf the test doesn’t end by July 2, then wewill abandon.Follow upIf the test fails, then try experiment C. Ifthe test passes, then experiment B.
  42. 42. Pick the typeof analysis
  43. 43. TYPESOFANALYSISJanuaryFebruaryMarchAprilMaysegmentation(cross-sectional)cohortanalysis(longitudinal)
  44. 44. Pick the typeof test
  45. 45. UNIQUE VISITORS RANDOMIZERCONTROLVARIANTVersion AVersion BA/B
  46. 46. UNIQUE VISITORS RANDOMIZERCONTROLVARIANT 1A/B/NVARIANT 2VARIANT 3
  47. 47. MULTIVARIATETESTINGEnroll Sign Up Join NowColor:Text:Size:27 Combinations
  48. 48. MULTIVARIATETESTINGEnroll Sign Up Join NowEnroll Sign Up Join NowEnroll Sign Up Join NowEnroll Sign Up Join NowEnroll Sign Up Join NowEnroll Sign Up Join NowEnrollEnrollEnrollSign UpSign UpSign UpJoin NowJoin NowJoin Now
  49. 49. IMPLEMENTINGEXPERIMENTS
  50. 50. Amazon A/B Testing Service (beta)FREETOOLSGoogle Analytics - Experiments APIGoogle Analytics - Content Experiments
  51. 51. WYSIWYGTOOLS
  52. 52. ENTERPRISETOOLS
  53. 53. http://www.roidna.com/tools/ab-testing-tool/
  54. 54. THECAMPEROOWAY?github.com/andrew/splitSPLIT + MIXPANELOpen Source,RubyFreemium,SAASmixpanel.com
  55. 55. TRACKINGEXPERIMENTS
  56. 56. Goal Build Measure LearnIncreaseAcquisitionZapposRedesignAirBNBwith categorynavigationSearch enginedisplay ofcustomer ratingsIncreaseActivationDraconianmodal dialogboxKANBANBOARD
  57. 57. VALIDATEDLEARNINGBOARDhttp://www.ashmaurya.com/2012/06/the-lean-stack/
  58. 58. EXPERIMENTREPORTINGhttp://spark59.s3.amazonaws.com/presentations/All%20About%20Experiments.pdf
  59. 59. WEEKLYPLANNINGWeekly meeting:Review experiment resultsChallenge interpretationKill or restart overdue experimentsDecide next experiments
  60. 60. THEEND...to be continued?

×