To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.

2. What is
A/B Testing?

3. MYTH BUSTING

4. #ScienceOfTesting
A/B testing is not...
Validation of
guesswork
Consumer psychology
gimmicks
“Meek Tweaking”
Images: Hubspot, Conversion Rate Experts

5. #ScienceOfTesting
It’s also not...
A waste of time
Impossible to
get right
Beyond the scope of
your job

6. A/B Testing: Defined
Conducting experiments to optimize
your customer experience.
What is A/B testing?
OR

7. 4 Steps of
Scientific A/B Testing

8. #ScienceOfTesting
The 4 Steps of A/B Testing
Step 1
Analyze data
Step 2
Form a
hypothesis
Step 3
Construct an
experiment
Step 4
Interpret
results

9. STEP 1 | ANALYZE DATA

10. Asking the right questions is hard.
Arm yourself with data.
#ScienceOfTesting

11. #ScienceOfTesting
Use quantitative & qualitative data
Quantitative data
tells you
where to test
Qualitative data
gives you an idea of
what should be tested

12. #ScienceOfTesting
Quantitative datasets
• Web traffic
• Email marketing
• Order history
• CRM interactions
• Support tickets
…and more!

13. #ScienceOfTesting
Run high-impact tests

14. Don’t choose tests randomly
Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/
how-to-use-data-to-choose-your-next-ab-test/

15. #ScienceOfTesting
Qualitative data
• User testing
• Survey data
• Heat mapping
• Your sales & account teams

16. STEP 2 | FORM A HYPOTHESIS

17. #ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The element that is modified
• Isolate one variable for an A/B test
• Call to action, visual media, forms

18. #ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The predicted outcome
• Use data to determine the size of effect
• More email sign-ups, clicks on a CTA

19. #ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• Demonstrate your customer knowledge
• What assumption will be proven wrong if
the experiment is a draw or loses?

20. #ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the call-to-action is shorter, the
conversion rate will increase.
Strong Hypothesis
If the call-to-action text is changed to
“Complete My Order,” the conversion
rates in the checkout will increase,
because the copy is more specific and
personalized.

21. #ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the checkout funnel is shortened to
fewer pages, the checkout completion
rate will increase.
Strong Hypothesis
If the navigation is removed from
checkout pages, the conversion rate
on each step will increase because our
website analytics shows portions of
our traffic drop out of the funnel by
clicking on these links.

22. STEP 3 | CONSTRUCT AN EXPERIMENT

23. A/B Testing: Defined
Every test
has 3 parts
DESIGNTECH
CONTENT

24. #ScienceOfTesting
Content: What are you saying?
VS.

25. #ScienceOfTesting
Design: How does it look?
VS.

26. #ScienceOfTesting
Tech: How does it work?
VS.

27. The most effective tests often combine all 3
elements: content, design, tech
#ScienceOfTesting

28. STEP 4 | EVALUATE RESULTS

29. #ScienceOfTesting
What are we looking for?
• How confident am I that the observed difference
from my experiment was not due to chance?
• 95% Statistical Significance = 5% probability that
the observed difference was due to chance.

30. #ScienceOfTesting
Confidence intervals
High statistical
confidence
Lower risk of
implementing a test
that won by chance

31. #ScienceOfTesting
Sample size calculator
http://optimize.ly/StatCalculator

32. #ScienceOfTesting
Once you reach significance:
• Variation wins: Launch the variation or update
your website.
• Original wins: Learn why hypothesis was
incorrect.
• In either case: Think about what to test next.

33. Examples!

34. A/B Testing: Defined
A simple test

35. A/B Testing: Defined
Iterative testing on a core hypothesis
A solid test

36. A/B Testing: Defined
Cohort analysis + website changes
+ biz process changes
A more complicated test
A
B

37. #ScienceOfTesting
Step 1: Data collection

38. #ScienceOfTesting
Step 2: Hypothesis
“If [Variable], then [Result], because [Rationale].”
If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able
to increase prospect to trial conversion rate.
Talking to sales will ensure all their questions get answered, improving their overall
experience and increasing willingness to take the next step with RJMetrics.

39. #ScienceOfTesting
Step 3: Experiment
• Changes to heading text
• Custom fields in Salesforce.com
• Business process changes for
sales reps
• Custom analysis in RJMetrics
based on offline conversion event

40. #ScienceOfTesting
Step 4: Results
TBD
A
B

41. Arm Your
Organization

42. Marketing
Increase the
impact of your
tests by bringing
more team
members into the
process
#ScienceOfTesting
Product
Sales
Engineering

43. Document your test results in a central repository.
#ScienceOfTesting
Heat maps
Optimizely
results
Hypothesis
What we
learned
Variations

44. #ScienceOfTesting
Other tried and true tactics
• Build excitement by sharing your wins with the company
• Hold a competition for the biggest winning variation
• Votes on variations to see who has the highest accuracy
of predicting winners

45. Thanks!