A/B testing presentation for Comm4Dev.

1. A/B TESTING
[A.K.A., BUCKET TESTS AND SPLIT TESTING ]
WHAT IT IS, ISN’T, CAN DO FOR YOU
+ HOW TO DESIGN YOUR TESTS

2. What isn’t A/B
testing?

3. A/B testing is not
• A communications strategy
• An objective unto itself
• Based on observational data or
opinion
• Something that should be
examined in isolation of other
information

4. What is A/B
testing?

5. A/B testing is a tool to
• examine user experience by testing
2 variants to inform decision
making
• use quantitative analysis to shape
design and copy
• gather objective information on the
impact of communication
approaches

6. Examples
• button placement
• headline wording
• color in graphics or text
• link naming
• photos vs. iconography
• e-newsletter subject lines
• banner ad placement

7. What Can A/B
Testing Do for
You?

8. A/B testing can
• make communications efforts data-driven
• help you craft a better user experience
• aid you in achieving communications
goals
• inform your strategies and actions
• provide quantitative data to inform or
justify your decision making

9. How to Design
Them

10. By putting on our
lab coats.

11. [Insert image of
fav scientists
here.]

12. Scientific method
1. Ask a question
2. Do research or intuit a scenario
3. Create a hypothesis (If/then statement)
4. Design your experiment using control and
variable groups
5. Run experiment and collect data
6. Analyze your data and draw conclusions
7. Capture conclusions and share them forward
8. Rerun experiment or adjust variable for new
experiment

13. Tools to Use
• Paid services like Optimizely.com
• Free services with other paid services like
e-newsletter campaign vendors for testing
subject lines
• Free services like Google Analytics
(Content Experiments)
• Data that you’re already collecting Google
Analytics (page visits, time on page),
Google Adwords, e-newsletter click-thru and
open rates, donations, PDF downloads

14. What A/B Tests
Aren’t

15. A way to break out into
something completely new. It’s
incrementalism, not game
changing.
A substitute for a vision.
Useful without enough data.