Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Rebekah Radice - How to Scale Successful Omni-Channel marketing Campaigns
Download to read offline and view in fullscreen.


4 Steps Toward Scientific A/B Testing

Download to read offline

To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.

Related Books

Free with a 30 day trial from Scribd

See all

4 Steps Toward Scientific A/B Testing

  1. 1. What is A/B Testing?
  3. 3. #ScienceOfTesting A/B testing is not... Validation of guesswork Consumer psychology gimmicks “Meek Tweaking” Images: Hubspot, Conversion Rate Experts
  4. 4. #ScienceOfTesting It’s also not... A waste of time Impossible to get right Beyond the scope of your job
  5. 5. A/B Testing: Defined Conducting experiments to optimize your customer experience. What is A/B testing? OR
  6. 6. 4 Steps of Scientific A/B Testing
  7. 7. #ScienceOfTesting The 4 Steps of A/B Testing Step 1 Analyze data Step 2 Form a hypothesis Step 3 Construct an experiment Step 4 Interpret results
  9. 9. Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting
  10. 10. #ScienceOfTesting Use quantitative & qualitative data Quantitative data tells you where to test Qualitative data gives you an idea of what should be tested
  11. 11. #ScienceOfTesting Quantitative datasets •  Web traffic •  Email marketing •  Order history •  CRM interactions •  Support tickets …and more!
  12. 12. #ScienceOfTesting Run high-impact tests
  13. 13. Don’t choose tests randomly Access this spreadsheet in this blog post: how-to-use-data-to-choose-your-next-ab-test/
  14. 14. #ScienceOfTesting Qualitative data •  User testing •  Survey data •  Heat mapping •  Your sales & account teams
  16. 16. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” •  The element that is modified •  Isolate one variable for an A/B test •  Call to action, visual media, forms
  17. 17. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” •  The predicted outcome •  Use data to determine the size of effect •  More email sign-ups, clicks on a CTA
  18. 18. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” •  Demonstrate your customer knowledge •  What assumption will be proven wrong if the experiment is a draw or loses?
  19. 19. #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the call-to-action is shorter, the conversion rate will increase. Strong Hypothesis If the call-to-action text is changed to “Complete My Order,” the conversion rates in the checkout will increase, because the copy is more specific and personalized.
  20. 20. #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the checkout funnel is shortened to fewer pages, the checkout completion rate will increase. Strong Hypothesis If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our traffic drop out of the funnel by clicking on these links.
  22. 22. A/B Testing: Defined Every test has 3 parts DESIGNTECH CONTENT
  23. 23. #ScienceOfTesting Content: What are you saying? VS.
  24. 24. #ScienceOfTesting Design: How does it look? VS.
  25. 25. #ScienceOfTesting Tech: How does it work? VS.
  26. 26. The most effective tests often combine all 3 elements: content, design, tech #ScienceOfTesting
  28. 28. #ScienceOfTesting What are we looking for? •  How confident am I that the observed difference from my experiment was not due to chance? •  95% Statistical Significance = 5% probability that the observed difference was due to chance.
  29. 29. #ScienceOfTesting Confidence intervals High statistical confidence Lower risk of implementing a test that won by chance
  30. 30. #ScienceOfTesting Sample size calculator
  31. 31. #ScienceOfTesting Once you reach significance: •  Variation wins: Launch the variation or update your website. •  Original wins: Learn why hypothesis was incorrect. •  In either case: Think about what to test next.
  32. 32. Examples!
  33. 33. A/B Testing: Defined A simple test
  34. 34. A/B Testing: Defined Iterative testing on a core hypothesis A solid test
  35. 35. A/B Testing: Defined Cohort analysis + website changes + biz process changes A more complicated test A B
  36. 36. #ScienceOfTesting Step 1: Data collection
  37. 37. #ScienceOfTesting Step 2: Hypothesis “If [Variable], then [Result], because [Rationale].” If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.
  38. 38. #ScienceOfTesting Step 3: Experiment •  Changes to heading text •  Custom fields in •  Business process changes for sales reps •  Custom analysis in RJMetrics based on offline conversion event
  39. 39. #ScienceOfTesting Step 4: Results TBD A B
  40. 40. Arm Your Organization
  41. 41. Marketing Increase the impact of your tests by bringing more team members into the process #ScienceOfTesting Product Sales Engineering
  42. 42. Document your test results in a central repository. #ScienceOfTesting Heat maps Optimizely results Hypothesis What we learned Variations
  43. 43. #ScienceOfTesting Other tried and true tactics •  Build excitement by sharing your wins with the company •  Hold a competition for the biggest winning variation •  Votes on variations to see who has the highest accuracy of predicting winners
  44. 44. Thanks!
  • PaytonMaurer

    Jan. 2, 2020
  • tarunsachdeva1654

    Sep. 16, 2019
  • EvaYifanGong

    Jan. 15, 2018
  • rincon.rise

    Sep. 20, 2017
  • ethenliu

    Aug. 3, 2017
  • MosBar

    Jun. 8, 2017
  • AaronChow1

    Apr. 12, 2017
  • sjmonk5

    Jan. 26, 2017
  • SallyTung

    Oct. 31, 2015
  • sallysixlol

    Jun. 19, 2015
  • BobBrooijmans

    May. 22, 2015
  • Crislol3

    Feb. 6, 2015
  • BasHennephof

    Jan. 30, 2015
  • lofipunk

    Jan. 27, 2015
  • rachelstuppy1

    Dec. 12, 2014
  • robertyu1

    Oct. 3, 2014
  • chops

    Jul. 25, 2014

To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.


Total views


On Slideshare


From embeds


Number of embeds