Smart testing is all about taking calculated risks that are designed from the start to render insights, whether they succeed or fail. Kristin and Amanda share their regimen for determining what to test and risk, getting management buy-in, testing into success, trying concepts and not just content, and tracking results over time.

1. Test to Learn
Kristin Bond (Girl Scouts of the USA) &
Amanda Soehnlen (Highmark Health)
@asoehnlen@emailsnarketing

2. ALL
Tests
are
risks.
@asoehnlen@emailsnarketing

3. GOOD RISKS:
● Inform about audience
● Establish best practices
● Increase credibility
● Neutral to positive impact
BAD RISKS:
● Impact list health
● Hurt brand
● Break laws
● Lose lots of money
@asoehnlen@emailsnarketing

4. How to Come up
with what to test.
@asoehnlen@emailsnarketing

5. What do you want to know about your Subscribers?
● What is your goal? (Hint: it’s probably not opens or clicks - it’s probably
conversions)
● You should be optimizing for the cta.
Measure “winners” by:
Which version made the subscriber do the thing?
@asoehnlen@emailsnarketing

6. Your test May have inconclusive results. That’s okay.
@asoehnlen@emailsnarketing

7. Test Concepts
Rather Than
Content.
@asoehnlen@emailsnarketing

8. Test Concepts Rather than Content!
Instead of…
“I’m going to write two
subject lines I like and see
which performs better.
OMG Look at me, I’m A/B
testing like a REAL
marketer!”
Try…
“I’m going test the
concept of short vs long
subject lines and see
which performs better.
OMG Look at me, I’m A/B
testing like a REAL
marketer!”
@asoehnlen@emailsnarketing

9. “I’m going to test subject lines”
isn’t enough.
@asoehnlen@emailsnarketing

10. By testing concepts, you give
yourself opportunity to apply
results to future emails.
@asoehnlen@emailsnarketing

11. Ideas for Concepts to Test
● Funny vs. Serious
● Short vs. Long
● Images vs. Text
● Direct vs. Indirect
● Dollar Discount vs. Percentage
● Urgency vs. Discount
● Company’s name vs. a person’s
● Weekends vs. Weekdays
● Free shipping vs. Discount
● Text links vs. Buttons
● Early in the day vs Late
● Afternoon vs Evening
● Emojis vs text
● Gifs vs Jpegs
...Or whatever else you want to learn about your subscribers
@asoehnlen@emailsnarketing

12. I told you so. Or, Wow, you were right. Huh.
“If you’re right, yay, you win, and if
you’re wrong - you learn something.”
-Amanda Soehnlen
@asoehnlen@emailsnarketing

13. You ran a test.
You have a
winner.
What now?
@asoehnlen@emailsnarketing

14. What
About
The
“Losers”?
@asoehnlen@emailsnarketing

15. Rule of thumb:
If your results are worse than your control, use your
control (and record the test details)
(…. Your test has a control, right?)
@asoehnlen@emailsnarketing

16. If you’re running
a binary test…
(Do emojis work for your audience?)
@asoehnlen@emailsnarketing

17. Do emojis work for your audience?
@asoehnlen@emailsnarketing
Find your result, use it for everyone:
I love emojis! Oh. This sucks.
I love emojis! Yay! Text is great!
Find your result, segment the users:
60% 40%
100%

18. If you’re running A
OPEN-ENDED TEST…
(What’s the best send time for my users?)
@asoehnlen@emailsnarketing

19. When should I send my emails?
@asoehnlen@emailsnarketing
Find your result, use it for everyone:
I like emails at 7pm! Oh. This sucks.
I like emails at 7PM! 3pm!
Find your result, Continue testing variants:
2am!
55% 45%
80% 20%

20. If you’re testing a
concept, you have to
test more than
once.
@asoehnlen@emailsnarketing

21. Pop Quiz:
If this email had been a
subject line test, no
matter how it performed,
the results would be
inconclusive.
Any guesses why?
@asoehnlen@emailsnarketing

22. Pop Quiz:
Any guesses why?
Hint: Check the date.
@asoehnlen@emailsnarketing

23. The Reveal:
@asoehnlen@emailsnarketing

24. You may think you have a winner.
But you could be wrong.
Test the concept again. A lot.
@asoehnlen@emailsnarketing

25. Let’s talk about:
BLOG POSTS THAT
FEATURE
“THE BEST ”
@asoehnlen@emailsnarketing

26. IF IT’S NOT AN INDUSTRY BEST PRACTICE, THAT BLOG POST
DOESN’T KNOW “WHAT’S BEST.”
EVERY BLOG THAT DOESN’T USE THE CAVEAT “FOR MY
AUDIENCE…”
QUESTION. THE. RESULTS.
IT MAY NOT BE APPLICABLE TO YOU OR YOUR AUDIENCE.
@asoehnlen@emailsnarketing

27. Everything you
know could be
WRONG.
@asoehnlen@emailsnarketing

28. That’s why you
test.
@asoehnlen@emailsnarketing