4 Steps Toward Scientific A/B Testing

3,267 views
2,889 views

Published on

To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.

Published in: Marketing
1 Comment
9 Likes
Statistics
Notes
  • We will help you perform a proper A/B testing, generate hypotheses and ideas for you, install all needed tools and run tests. Learn more at http://abtst.com/agency?lang=en&utm_source=slideshare&utm_medium=comment&utm_campaign=comments
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
3,267
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
196
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide

4 Steps Toward Scientific A/B Testing

  1. 1. What is A/B Testing?
  2. 2. MYTH BUSTING
  3. 3. #ScienceOfTesting A/B testing is not... Validation of guesswork Consumer psychology gimmicks “Meek Tweaking” Images: Hubspot, Conversion Rate Experts
  4. 4. #ScienceOfTesting It’s also not... A waste of time Impossible to get right Beyond the scope of your job
  5. 5. A/B Testing: Defined Conducting experiments to optimize your customer experience. What is A/B testing? OR
  6. 6. 4 Steps of Scientific A/B Testing
  7. 7. #ScienceOfTesting The 4 Steps of A/B Testing Step 1 Analyze data Step 2 Form a hypothesis Step 3 Construct an experiment Step 4 Interpret results
  8. 8. STEP 1 | ANALYZE DATA
  9. 9. Asking the right questions is hard. Arm yourself with data. #ScienceOfTesting
  10. 10. #ScienceOfTesting Use quantitative & qualitative data Quantitative data tells you where to test Qualitative data gives you an idea of what should be tested
  11. 11. #ScienceOfTesting Quantitative datasets •  Web traffic •  Email marketing •  Order history •  CRM interactions •  Support tickets …and more!
  12. 12. #ScienceOfTesting Run high-impact tests
  13. 13. Don’t choose tests randomly Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/ how-to-use-data-to-choose-your-next-ab-test/
  14. 14. #ScienceOfTesting Qualitative data •  User testing •  Survey data •  Heat mapping •  Your sales & account teams
  15. 15. STEP 2 | FORM A HYPOTHESIS
  16. 16. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” •  The element that is modified •  Isolate one variable for an A/B test •  Call to action, visual media, forms
  17. 17. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” •  The predicted outcome •  Use data to determine the size of effect •  More email sign-ups, clicks on a CTA
  18. 18. #ScienceOfTesting Parts of a hypothesis “If [Variable], then [Result], because [Rationale].” •  Demonstrate your customer knowledge •  What assumption will be proven wrong if the experiment is a draw or loses?
  19. 19. #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the call-to-action is shorter, the conversion rate will increase. Strong Hypothesis If the call-to-action text is changed to “Complete My Order,” the conversion rates in the checkout will increase, because the copy is more specific and personalized.
  20. 20. #ScienceOfTesting All hypotheses are not created equal Weak Hypothesis If the checkout funnel is shortened to fewer pages, the checkout completion rate will increase. Strong Hypothesis If the navigation is removed from checkout pages, the conversion rate on each step will increase because our website analytics shows portions of our traffic drop out of the funnel by clicking on these links.
  21. 21. STEP 3 | CONSTRUCT AN EXPERIMENT
  22. 22. A/B Testing: Defined Every test has 3 parts DESIGNTECH CONTENT
  23. 23. #ScienceOfTesting Content: What are you saying? VS.
  24. 24. #ScienceOfTesting Design: How does it look? VS.
  25. 25. #ScienceOfTesting Tech: How does it work? VS.
  26. 26. The most effective tests often combine all 3 elements: content, design, tech #ScienceOfTesting
  27. 27. STEP 4 | EVALUATE RESULTS
  28. 28. #ScienceOfTesting What are we looking for? •  How confident am I that the observed difference from my experiment was not due to chance? •  95% Statistical Significance = 5% probability that the observed difference was due to chance.
  29. 29. #ScienceOfTesting Confidence intervals High statistical confidence Lower risk of implementing a test that won by chance
  30. 30. #ScienceOfTesting Sample size calculator http://optimize.ly/StatCalculator
  31. 31. #ScienceOfTesting Once you reach significance: •  Variation wins: Launch the variation or update your website. •  Original wins: Learn why hypothesis was incorrect. •  In either case: Think about what to test next.
  32. 32. Examples!
  33. 33. A/B Testing: Defined A simple test
  34. 34. A/B Testing: Defined Iterative testing on a core hypothesis A solid test
  35. 35. A/B Testing: Defined Cohort analysis + website changes + biz process changes A more complicated test A B
  36. 36. #ScienceOfTesting Step 1: Data collection
  37. 37. #ScienceOfTesting Step 2: Hypothesis “If [Variable], then [Result], because [Rationale].” If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able to increase prospect to trial conversion rate. Talking to sales will ensure all their questions get answered, improving their overall experience and increasing willingness to take the next step with RJMetrics.
  38. 38. #ScienceOfTesting Step 3: Experiment •  Changes to heading text •  Custom fields in Salesforce.com •  Business process changes for sales reps •  Custom analysis in RJMetrics based on offline conversion event
  39. 39. #ScienceOfTesting Step 4: Results TBD A B
  40. 40. Arm Your Organization
  41. 41. Marketing Increase the impact of your tests by bringing more team members into the process #ScienceOfTesting Product Sales Engineering
  42. 42. Document your test results in a central repository. #ScienceOfTesting Heat maps Optimizely results Hypothesis What we learned Variations
  43. 43. #ScienceOfTesting Other tried and true tactics •  Build excitement by sharing your wins with the company •  Hold a competition for the biggest winning variation •  Votes on variations to see who has the highest accuracy of predicting winners
  44. 44. Thanks!

×