This document provides an overview of AB testing. It explains that AB testing involves forming a hypothesis about which of two variants will perform better, running a test to compare the variants, and analyzing the results statistically to determine if one variant is significantly better. The document recommends testing things like subject lines, content, site architecture, and ads to improve metrics like open rates, click rates, conversions. It emphasizes testing iteratively with small changes, having a clear goal in mind, and not letting testing dictate all design decisions.

1. The ABC
of AB testing
jamie woolley
jme.woolley@gmail.com
jamiewoolley.org

2. Why test at all?
● Find out what works...
● ...and what doesn’t
● Make decisions based on
evidence
● Avoid arguments

3. AB testing is iterative

4. Take baby steps

5. Multivariate tests

6. What should you test?
Almost anything...
● Subject lines
● Email content
● Site architecture
● Site usability
● Donation forms
● Landing pages
● Facebook ads
● Google Adwords
● Images

7. Know your goal
Decide what you’re trying to improve
● Open rates
● Click rates
● Form completion
● Traffic

8. How to test
1. Form your hypothesis — what do you predict will
happen?
2. Create your variants
3. Decide sample size and test duration
4. Test!
5. Interpret results

10. Statistics! Run!

11. Easy signifance testing tools are online, such as http://butisitstatisticallysignificant.com/

12. Mind your p-values
● Check for statistical significance, a measure of
confidence in your results
● The lower the p-value, the more confident you
can be that your results are due to your test
variables
● The standard measure is 0.05 or lower, so you
can be at least 95% confident your hypothesis is
correct

13. p-value is less than
0.05 so you can
accept your
hypothesis - hooray!

14. p-value is greater
than 0.05 so results
could be due to other
factors - boo

16. Testing does a lot… but not everything
● AB testing can’t tell you why — but you can
make educated guesses
● It’s not suitable for large redesigns as it won’t be
clear what influenced the results
● Testing won’t tell you if you’re addressing the
right problem, so clear goals are important
● Don’t let testing dictate creative design
decisions

17. Thank you
jamie woolley
jme.woolley@gmail.com
jamiewoolley.org