Why Guess When You Can Test? Testing Techniques from the Experts

Your Presenter:
www.highroadsolution.com
Jenny Lassi has been affiliated with
HighRoad Solution for 8 years first as a
client and then as an employee.With
experience handling deployments and
deliverability for a wide range of clients
with optimized recipient engagement top-
of-mind, Jenny routinely trouble shoots
deliverability issues from ISP level filtering
to email client level filtering (email
Forensics as she refers to them). She also
consults with clients on authentication,
content, content strategy, email preference
center management & other deliverability
optimization tactics.

Today’s Content
• What is AB Testing?
• What is Multi-VariatTesting?
• The 5 C’s & How They Apply to Testing
• Starting Simply
• Testing the Complex
• Best practices
• Resources takeaways

What is AB Testing?

AB Testing Defined
A/B testing is a term commonly used in web development, online
marketing, and other forms of advertising to describe simple
randomized experiments with two variants,A and B, which are the
control and treatment in the controlled experiment.
The formal or scientific name used for this process and other
related processes is hypothesis testing. Other names include
randomized controlled experiments, online controlled experiments,
and split testing.

What is Multivariate Testing?

Multivariate Testing Defined
In internet marketing, multivariate testing is a process by which
more than one component of a website may be tested in a live
environment. It can be thought of in simple terms as numerous A/B
tests performed on one page at the same time.A/B tests are usually
performed to determine the better of two content variations;
multivariate testing can theoretically test the effectiveness of
limitless combinations.
Note: Be cautious of testing too many variables as it can
be difficult to understand what is behind the lift in
response.

eMail Strategy

The 5 C’s of Email
Marketing Strategy
•Configuration (Clean House!)
•Content
•Create
•Criticize
•Continue

The 5 C’s: Configuration
• IP Address (Dedicated or
Shared)
• DKIM (Domain Keys)
• Sender ID (SPF Record)

The HighRoad Solution
Deliverability Cops – Dedicated IP
• Sender Rejected
• Message Filtered
• Unknown Users
• Infrastructure
• Volume
• SpamTraps
• Blacklists
• Complaints
• FBL Compaints (Routed to your
Opt-Outs)

The HighRoad Solution
Deliverability Cops – Shared IP
• Spam Traps
• Blacklists
• Complaints

Authentication
High Level View: DKIM/Sender ID (it’s not a scary term or complicated at all… I promise)
• You tell us what domain(s) you send from
• We supply you with 4 snippets of text that you add to your domain’s DNS and instructions
• You add the txt records to your DNS and tell us when you’re done
• We test/validate the keys and enable on your account

Why not just blast?
Invalid email Addresses
+
Excessive email Bounces
+
Emails Filtered to Junk/SPAM
Folders
=
Wasted $$

eMail Filtering
Email
Filtering
Layers

Do you know if your eMails
are reaching an inbox?
When a SureSend
Domain Delivery
Report is enabled
when sending a Live
message, you will see
the Domain Delivery
tab on the email stats

Do you know if your eMails
are reaching an inbox?

Cleaning House
Do whatever is in your control to
reach an inbox!
• Data Hygiene (Invalids/Bounces)
• DKIM/Sender ID Authentication
• Address any issues with filtering
that you can
THEN… and only then should
you AB Test

The 5 C’s: Content
Common ContentTests:
• Subject Lines
• Lead Story
• TOC
• More text vs. read more

The 5 C’s: Create
• Test standard html vs. responsive
template
• Call-To-Action Button Color &
Placement
• Layout
• Images

Case Study Workshop
Can you trust your gut?

Case Study Workshop
Can being more
verbose get a higher
response?

Version A

Version B

Case Study Workshop
What do you think?
Version A?
Version B?

Case Study Workshop
Version B won an
astounding 247% more
clicks

Case Study Workshop
Does a larger lead
story get people to
click?

Case Study Workshop
Version B, the one
without bigger featured
content, increased
clicks by 75% and
online registrations by a
stunning 400%

Case Study Workshop
Does more or less
white space increase
engagement?

Case Study Workshop
Version A Increased
clickthroughs by 5.6%,
and heatmaps showed
that there was a higher
engagement as a
whole across the entire
email.

Case Study Workshop
Subject line straight
shooter or a bit playful?

Case Study Workshop
Version B – the playful version –
increased opens for French-speaking
Canadians.
However, Version A – the
plainspoken version – increased
opens for English-speaking
Canadians.

Case Study Workshop
Which local time works
better for B2B email
offers 9am or 1pm?

Version A – 9am

Version B – 1pm

Case Study Workshop
Version A, the email sent at 9 am local time at four
U.S. time zones increased lifts in all time zones –
here are the specifics:
• EST – 12.5% lift in opens
• CST – 100% lift in opens
• MDT – 35% lift in opens
• PST – 4,090% STUNNING lift in opens

The 5 C’s: Criticize
• Analyze delivery metrics (bounces, opens &
clicks)
• Use bounce logs to address deliverability issues
24-48 hours post deployment (not a month later)
• Isolate subscribers who are not engaged with
your emails (no opens/click throughs)
• Create an AB Test Win-Back campaign for non-
engaged subscribers with an incentive to respond
• Analyze how many subscribers opt-down or
unsubscribe from particular messages (maybe the
perceived value of content is low)

The 5 C’s: Continue
Use what you have
learned and apply it to
future deployments

Best PracticesTakeaways
• Be more strategic with your testing with the 5-C’s.
• You can learn from other AB Tests, but testing against your own
audience is the only way to see what increases engagement.
• Send to known engaged subscribers when testing or your results
may vary based on inbox placement.

Questions?

Shortcuts
Control what you CAN control about deliverability:
• DKIM/Sender ID Authentication
• List Hygiene
• Having an Email Preference Center
• Deliver content members or registrants want, when they want it,
how frequently they want it and looks good where they read it
Analyze and learn from AB Split testing to adjust how you send
future emails:
• Use A/B split testing to learn about what has the best
engagement
• Review creative/content that had the best engagement and do
more of that
• Review bounce logs daily to see if any domains may be blocking
you and reach out to postmaster@ and abuse@ those domains
to ask them to whitelist you

Thank you!
Other questions for Jenny?
eMail = jenny@highroadsolution.com
Twitter = @highroadjenny

References & Resources
https://en.wikipedia.org/wiki/Ab_testing
http://en.wikipedia.org/wiki/Multivariate_testing
http://whichtestwon.com/

Why Guess When You Can Test? Testing Techniques from the Experts

Recommended

Recommended

More Related Content

Similar to Why Guess When You Can Test? Testing Techniques from the Experts

Similar to Why Guess When You Can Test? Testing Techniques from the Experts (20)

More from HighRoad Solution

More from HighRoad Solution (20)

Recently uploaded

Recently uploaded (20)

Why Guess When You Can Test? Testing Techniques from the Experts