Presentation delivered at Netroots Nation 2013 by CCAH's Damien Shirley and PFAW's Ben Betz.

1. The Online Laboratory:
Perfecting Your Outreach
Through Email Testing
#NNOnlineLab

2. WHO ARE YOU?
• Ben Betz
– Online Strategy Manager, People For the
American Way
• Damien Shirley
– Account Executive, Chapman Cubine
Adams + Hussey

3. WHAT IS AN EMAIL TEST?
• Send multiple versions of an email
• Identical except for a single test variable
• To a random, representative sample of your
email list
• Measuring which version performs better
• Goal: Try a new approach and see if it works
VS.

4. WHY SHOULD I TEST?
• Every email list is different
• There are best practices, but they don’t
always apply to every email list.
• You may know how your supporters
respond to your emails, but not why
• If you’re not testing, you’re guessing
• You might be leaving something on the
table
• Bottom line: Test to optimize your email
engagement.

5. WHAT DO I NEED TO GET
STARTED?
• Email list
• Email platform (CRM)
– Many have built-in testing modules
• Email messaging program
• Confidence calculator

6. WHAT SHOULD I TEST?
• Anything
• Test approaches that could improve
your email performance
– Let your data guide you
• Make your tests scalable
• Think outside the inbox

7. WHAT SHOULD I TEST: TESTING
IDEAS
• Sender
• Send time
• Send date
• Subject line
• Personalization
• Callout box position
• Callout box image
• Salutation
• Signer
• Length
• Tone
• Landing page
• Landing page
layout

8. HOW DO I TEST?
• Components:
– Control and test email versions
• Only difference: your test variable
• Only one test variable at a time
– Control and test audiences
• Random, representative samples of your list
– Bottom line: the only difference between
email versions should be the variable
you’re testing

9. HOW DO I TEST: TEST TYPES
• A/B split test
– One control email variant & one test email
variant
– Most common
– Easiest to set up
Control Test

10. • Multivariate Test
– One control email variant & multiple test
email variants
– Still only one test variable
– Better for larger lists
Control Test 1 Test 2
HOW DO I TEST: TEST TYPES

11. • Pre-Launch Test
– Send your control and test variant(s) to
small segments of your email list
– Launch the winning version to the rest of
your list
– Takes more time to set up
Control Test
To Full List
HOW DO I TEST: TEST TYPES

12. HOW OFTEN SHOULD I TEST?
• Always
• Or, as often as you can
• Continue until you get significant
results

13. HOW DO I KNOW WHAT WON?
• Check your data
• Use a confidence calculator
–Determine statistical significance
–Calculate your confidence interval

14. HOW DO I KNOW WHAT WON?
• What if neither variant wins?
–No significant difference: go with
your preference
–Mixed results: prioritize your metrics

15. WHAT NOW?
• Apply your results
• Plan your next test
• Build a testing library
• Embrace failure
• You’re not done yet: retesting
– No set schedule
– Let your data guide you

16. WHAT TO TEST: MEMBER DRIVE
• Test idea: Test including a member
drive versus standard renewal
messaging.
• Test type: A/B split test
• Control: Standard renewal messages.
• Test: Messages included the member
drive goal for new or renewed
members.

17. Standard Renewal Member Drive
WHAT TO TEST: MEMBER DRIVE

18. • Results:
– The member drive test had a higher
clickthrough rate.
– The member drive also test generated more
revenue per donor.
– Using a confidence calculator, we
determined that both of these results were
statistically significant.
• Recommendation:
– Include member drives in future renewal
campaigns.
WHAT TO TEST: MEMBER DRIVE

19. WHAT TO TEST: EMAIL LAYOUT
Personalized Member Invoice
vs. Standard Email Format
Invoice was the winner

20. WHAT TO TEST: CALLOUT BOX
IMAGE
Member Card
had a higher
clickthrough rate
Thermometer
had a higher response
rate (winner)

21. WHAT TO TEST: SENDER
• Testing campaign-y names as sender
versus names of staff
– Stop GOP Witch Hunts vs. NAME, PFAW
– Overturning Citizens United vs. NAME,
PFAW
– PFAW Membership Director vs. NAME,
PFAW

22. WHAT TO TEST: MESSAGE
• Message A vs. Message B
• Roll out a message to smaller segment
of the email list to test its strength prior
to full-list launch

23. QUESTIONS?