What you're missing in A/B email testing

What you’re missing in
A/B email testing
@jonathanpay | @HolisticEmail | #LitmusLive

Jonathan Pay
Consultant
Holistic Email Marketing
@jonathanpay
jonathan@holisticemail.com

Testing in email

US Interactive Marketing Spend 2016 Orders by Marketing Channel
Forrester
Custora

Spend vs Orders by Email Channel
Spend Orders
3% spend 18% orders

-5%
15%
-35%
-2%
-40%
-30%
-20%
-10%
0%
10%
20%
Comparison
Social
Email
Display
Search

And that’s with the
attribution being
inaccurate!
57% save the email for
later
48% bear the information
in mind for later.
47% go to the website via
another route
DMA UK

The 2017 Email Benchmarking Report produced by Digital Doughnut & Dotmailer

Results
Resources
Recognition

Why test?
You want to find out what resonates with your
prospects and customers, so they’re more likely to
action your objective. Simple!

A/B testing
Multivariate testing

A/B Example
A/B Control SL: Our sale is on now! Get 15% off today!
A/B Variant SL: Our sale is on today! Get 15% off now!

Multivariate Example
M/V Control SL: Our sale is on now! Get 15% off
today!
M/V Variant SL: Our sale is on today! Get 15% off
now!
M/V Control CTA: Blue
M/V Variant CTA: Red
M/V Control Image 1
M/V Variant Image 2

A
B
A
B
A
B

A definition of Holistic testing
This is a regularly-performed test based on a
hypothesis, with the two-fold aim of not only finding
out which version delivers the best result, but also
aims to discover insights about the audience.

It needs to center around 3 main
things
1. Presumed problem
2. Proposed solution
3. Anticipated result (and reason why)

A hypothesis can be
proved or disproved

Holistic testing example
Hypothesis: Loss aversion copy is a stronger
motivator towards conversions than benefit-led copy
because people hate losing out more than they enjoy
benefiting.
Control SL: Our sale is on now! Get 15% off today!
Control CTA: Shop now
Variant SL: 15% off today only. Don’t miss out!
Variant CTA: Don’t miss out – shop now

Most commonly tested design
factors on websites
colour (changing the colour of elements on a website)
+0.0% uplift
buttons (modifying website buttons) -0.2% uplift
calls to action (changing the wording on a website to
be more suggestive) -0.3% uplift
The meta-analysis of 6,700 ecommerce experiments
qubit.com/research/what-works-in-ecommerce

For greater scope and results, test
motivations
scarcity (stock pointers) +2.9% uplift
social proof (informing users of others’ behaviour)
+2.3% uplift
urgency (countdown timers) +1.5% uplift
The meta-analysis of 6,700 ecommerce experiments
qubit.com/research/what-works-in-ecommerce

Who makes up your email
database?
Your target market

1.
Identify your objective

2.
Have a desire to learn about
your audience

3.
Identify the question you’re
trying to answer

4.
Create a hypothesis

Let’s look at some hypotheses used
in email
Hypothesis:
Emotional question will generate more sales than a directive statement.
“Double Loyalty points” will generate more sales than “2X loyalty points”.
Emotive image of person smiling & wearing outfit will generate more sales than
displaying outfit laid out.

5.
Select the factors to test

Let’s look at some factors to test
your hypotheses
Hypothesis: Factors tested:
Emotional question will generate more sales than a
directive statement.
Subject line, CTA, title, copy
“Double Loyalty points” will generate more sales than
“2X loyalty points”.
Subject line, CTA, title, copy
Emotive image of person smiling & wearing outfit will
generate more sales than displaying outfit laid out.
Image

6.
Identify your success
metric

Focus on & measure the correct
metric!
A: Increased total orders by 21%
And revenue by 35%, even though A had a 10.5%
lower open rate than B.
Sent 50/50 split test over 4 weeks to abandoners
A Reminder: Items Saved in Your Cart – Shop Now for an
Extra 10% off
B Come back and Save an Extra 10%
Native Remedies

7.
Analyse and learn

Aggregation of marginal gains
The GB cycling team’s winning philosophy is
‘aggregation of marginal gains’: Get a lot of small
things right, put them together, and they add up to a
significant gain. “It means taking the 1% from
everything you do; finding a 1% margin for
improvement in everything you do.”

Create a test plan

Metrics & Fields to include
Total Sent
Open Rate
Click to Open Rate
Products purchased
Conversion Rate
AOV
Statistical Confidence
Conclusion
Recommendation

Test the hypothesis multiple times
To ensure it wasn’t an anomaly. Because you’re
performing hypothesis-led tests you can:
• Use different wording
• Potentially use multiple factors
As long as it supports the hypothesis

Statistical Validity
abtestguide.com/calc/

8.
Roll results out to other
channels

Let’s look at some channels to share
your insights with
Hypothesis: Factors tested: Can share with:
Emotional question will generate more
sales than a directive statement.
Subject line, CTA, title,
copy
Website, landing
pages, banner ads,
PPC ads, social media,
retargeting ads
“Double Loyalty points” will generate
more sales than “2X loyalty points”.
Subject line, CTA, title,
copy
Website, landing
pages, retargeting
ads, social media
Emotive image of person smiling &
wearing outfit will generate more sales
than displaying outfit laid out.
Image
Website, landing
pages, banner ads,
social media,
retargeting ads

Perform tests in email

Roll these insights out to
other channels &
continually optimise

A A
B
?
Website
Landing pages
PPC
Social
Banner ads
Retargeting
B
A
B

9.
Bring more ££ & value into
email marketing!

www.holisticemail.com
hello@holisticemail.com
@holisticemail

What you're missing in A/B email testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to What you're missing in A/B email testing

Similar to What you're missing in A/B email testing (20)

Recently uploaded

Recently uploaded (20)

What you're missing in A/B email testing