Delivered by Kath Pay, CEO of Holistic Email Marketing, at Figaro Digital's Email Marketing & CRM Seminar on Thursday 9 February 2017. Aimed at marketing professionals looking to push the power of their email marketing, this session reveals how to leverage the testing ability of your emails to improve the performance of your other marketing acquisition and conversion channels. You will learn how to build a hypothesis into your emails, to drive the actions that provide the answers you're looking for. As well as how to use a push channel such as email to inform and improve pull channel performance. Plus, tactics to test for long-term results as well as short-term results.

1. Realise the Benefits of
Holistic Testing
Kath Pay,
Holistic Email Marketing

3. Kath Pay,
Founder & CEO
Holistic Email Marketing
@kathpay
kath@holisticemail.com

5. US Interactive Marketing Spend 2016 Orders by Marketing Channel
Forrester Custora

6. Using the actual numbers, let’s look
at them in some pretty graphs
Spend
Social 6.52%
Email 3.22%
Display 36.02%
Search 43.49%
Orders
Social 1.50%
Email 18.40%
Display 1.10%
Search 41.80%
Forester Custora

7. Difference
Email 15.18%
Search -1.69%
Social -5.02%
Display -34.92%

8. And that’s
with the
attribution
being
inaccurate!
Source: DMA UK
57% save the email for later and 48% bear the
information in mind for later. 47% go to the
website via another route

10. What if?
• Image of girl with ???? Or person with surprised (happy) face

12. Why test?
You want to find out what resonates with your prospects and
customers, so they’re more likely to action your objective.
Simple!

14. Adhoc testing
Definition:
Random, occasional test of a factor with the
aim of determining which version delivers the
best result. No methodology or hypothesis is
used in this process.

15. Let’s test the subject line!
Control: Our sale is on now!
Get 15% off today!
Variant: Our sale is on today!
Get 15% off now!

16. Holistic testing
Definition:
This is a regularly-performed test based on a
hypothesis, with the two-fold aim of not only
finding out which version delivers the best
result, but also aims to discover insights about
the audience.

17. What makes my audience tick?
Hypothesis:
Loss aversion copy is a stronger motivator towards
conversions than benefit-led copy because people
hate losing out more than they enjoy benefiting.
Control: Our sale is on now!
Get 15% off today!
Variant: 15% off today only.
Don’t miss out!

18. Who makes up your email database?
Your Target Market!

19. Your database is your target market!
Use #email as your primary testing
tool & roll out findings to other
channels @kathpay @Figaro_Digital
Tweet this

20. Identify your objective1

21. Adhoc Testing Objectives
1. Gain an immediate uplift in an
email campaign
Holistic Testing Objectives
1. Gain an immediate uplift in
conversions of an email campaign
2. Gain insights into your customers
that can be used within your other
channels & increase revenue
across the board
3. Make the email channel the main
channel for testing & bring
additional budget & resources to it

22. Have a desire to
learn about your
audience
2

23. Test strategic issues such as a pain-
point or business challenge. Ask
yourself “What makes my audience
tick?” @kathpay @Figaro_Digital
Tweet this

24. Identify what
question are you
trying to answer?
3

25. Create a Hypothesis4

26. It needs to centre around 3 main
things
1. Presumed problem
2. Proposed solution
3. Anticipated result (and reason why)

27. A Hypothesis can be proved or
disproved
A hypothesis pretty much says, “I think by making this change,
it will cause this effect.” So, based on your results, you should
be able to say “this is true” or “this is false.”….
BECAUSE.......

28. Let’s look at some hypotheses to use
in email
Hypothesis Factor tested Channels to roll it out to
Emotional question will generate more sales
than a directive statement
Subject line, CTA,
title, copy
Website, landing page,
banner ads, PPC ads, social
media, retargeting ads
Double Loyalty points will generate more sales
than 15% loyalty points
Subject line, CTA,
title, copy
Website, landing page,
retargeting ads, social
media
Emotive image of person smiling & wearing
outfit will generate more sales than image
displaying outfit laid out.
image Website, landing page,
banner ads, Social media,
retargeting ads

29. Select the factor to test5

30. When you use a hypothesis to
test #email you’re not
restricted to testing the same
element. The hypothesis
stands @kathpay
@Figaro_Digital
Tweet this

31. Identify your success
metric6

32. Focus on & measure the correct
metric!
A: Increased total orders by 21%
And revenue by 35%, even though A had a 10.5% lower open
rate than B.
Sent 50/50 split test over 4 weeks to abandoners
Source: Native Remedies
A Reminder: Items Saved in Your Cart – Shop Now for an
Extra 10% off
B Come back and Save an Extra 10%

33. Analyse and learn7

34. Create a test plan

35. Metrics & Fields to include
• Total Sent
• Open Rate
• Click to Open Rate
• Products purchased
• Conversion Rate
• AOV
• Statistical Confidence
• Conclusion
• Recommendation

36. Statistical Validity
http://abtestguide.com/calc/

37. Test hypothesis multiple times
• Because you’re performing hypothesis-led tests you can:
• Use different wording
• Potentially use multiple factors

38. Roll results out to other
channels8

39. Perform tests in email

40. https://www.behave.org/test/call-action-stress-urgency-important-information/
Hypothesis: Informational subject line will deliver more conversions because a mortgage is a
serious and lengthy decision-making process
Urgency Informational
Winner!
49.2% uplift
99% confidence

41. https://www.behave.org/test/whats-the-best-content/
Winner!
63% uplift
95% confidence
Hypothesis: Video content will deliver better results because video content is very consumable
Video ContentDatasheet Content

42. Roll these insights out to
other channels &
continually optimise

43. Aggregation of marginal gains
The BG cycling team’s winning philosophy is ‘aggregation of
marginal gains’: Get a lot of small things right, put them
together, and they add up to a significant gain
www.guardian.co.uk/sport/2008/aug/14/olympics2008.olypmicscycling1
http://www.teamsky.com/article/0,27290,17547_5792058,00.html
“It means taking the 1% from everything
you do; finding a 1% margin for
improvement in everything you do.”

44. Bring more ££ & value
into email marketing!9

45. Tests which would have
been quicker, cheaper and
easier to test in email

46. https://www.behave.org/test/copy-converts-better/
Winner!
Hypothesis: “Get A Quote” would perform best. Because the water treatment services are
expensive, the idea of a “quote” would establish more trust – as it suggests a personal
experience with a representative, allowing you to learn more about the company’s water
treatment options.
Non-personalPersonal
104% uplift
96% confidence

47. https://www.behave.org/test/without-or-with-zeros
Winner!
Hypothesis: Shortening the price would indicate the item costs less. So, visitors would be most
receptive to the truncated price, without the zeros
LongShort
9.3% increase in add-to-cart clicks
29% rise in visits/order
47% lift in RPV
99% confidence

48. It’s a cycle of
continuous
learning
Identify your
objective
Have a desire to
know your customers
Identify a question or
pain
Create your
hypothesis
Select the factor to
test
Test in email
Identify your success
metric
Analyze & Learn
Apply to appropriate
channels
Bring value & ££ into
email

49. www.holisticemail.com
hello@holisticemail.com
@holisticemail