Beyond A/B Testing: How to Take Your Split Tests to the Next Level

800-920-7227 | www.pinpointe.com
Presented by:
Michael Pelosi

A Leader in Business-to-Business Email Marketing
Brought to you by
Pinpointe

Digital Media Strategist at Pinpointe On-Demand
www.pinpointe.com
Phone: 617-461-1209 | mpelosi@pinpointe.com
Twitter / Facebook: @Pinpointe
Featured Speaker
Michael Pelosi
Eryn Branham
Pinpointe On-Demand, Inc.
ebranham@pinpointe.com
800-920-7227
Moderator

BEYOND A | B
TESTING
A S K A Q U E S T I O N , F I N D A N A N S W E R , A N D
P L A N F O R T H E F U T U R E

WHAT IS A | B TESTING?
A/B Testing, or split testing, tests one component of your marketing
campaign (subject line, call-to-action, etc.) against another within a
certain time frame and across a fixed amount of users.

TEST EVERYTHING (ALMOST)
• The preferences and tastes of your customers change as you offer more products and
services.
• Don’t just throw darts at the dartboard and hope they stick.A/B testing gives a framework for
picking winners.
• Consistent A/B testing builds a more data driven email and digital marketing culture.

REASONS NOT TO A/B TEST
• Your campaign is performing well. If it’s not broken, don’t fix it. Or you just may
break it.
• More data is does not lead to better decisions. It only leads to more analysis.
• If something just isn’t working, don’t spend the time testing. Spend the time
fixing.A/B testing is all about providing a lift for a metric of interest.

A TOUCH OF RANDOM IN THE AIR
• A/B testing is powerful, but it’s not a crystal ball.
• Customer behavior can be unpredictable and sometimes random.
• There’s always an unseen error in A/B test results.
• The results from an A/B test today may not be the same as the results from the same test
tomorrow.

HOW WE THINK PEOPLE READ THEIR
EMAILS

Can you go to the
grocery store on your
way home?

4 SIMPLE STEPS TO TEST
Select A and B

Select how many subscribers will receive A and how
many will receive B.Then select for how long the test
will run for.

Conduct an A/A test to establish a baseline.

Develop the hypothesis, wait for the result, and
analyze the data.

#1. SELECT A & SELECT B
•Test small differences to start.
•Have a good business reason for selecting
B.

EXAMPLE ONE: EMAIL SUBJECT LINE
CONTROL SUBJECT LINE (A)
VARIATION SUBJECT LINE (B)
We Could Really UseYour Input on This Problem
The 7 Strategies We Used to Solve Our Problem

MORE SUBJECT LINES
CONTROL SUBJECT LINE (A)
VARIATION SUBJECT LINE (B)
“This is just a service I can’t live without.”
“7 services curated just for you”

MAKING ONLY ONE CHANGE AT A TIME
If you’re testing a subject line, everything after the subject line
has to stay the same.The same goes for email layout, landing
pages, etc.

SUBJECT LINE A
“Our biggest summer sale: 25% off select brushes”

SUBJECT LINE B
“Brush Bonanza is Here! 25% Off Premium Brushes”

EMAIL A

SUBJECT LINE TESTING IDEAS
1. Subject line length (character differences, short versus long subject lines).
2. Using numbers in the subject line vs. no numbers in the subject line.
3. Using “testimonials” or “quotes.”
4. Using emojis.
5. Using urgent words versus a relaxed words.
6. Using personalized lines versus generic lines.

MORE EXAMPLES

EXAMPLE 2: PLAIN TEXT VERSUS
IMAGE BASED EMAILS

EXAMPLE 3: CALL TO ACTION
BUTTONS

THE CTA RESEARCH
• Best Color: Blue
• Best Size for Mobile: 44 x 44 pixels
• Not Much Other Research: Perfect Place forYour Case
Studies

EMAIL CTA: BUTTON PANEL

CTA PLAIN TEXT
•Because plain text emails offer larger lifts,
A/B test image CTA versus link CTA.
•Use actionable words that are clearly links.

CTA LINK EXAMPLES PLAIN TEXT
• Buy now
• Buy today
• Claim your spot
• Claim your spot today
• 50% off your purchase
• Happy Birthday! Enjoy our gift to you.
• Download our whitepaper
• Tell us you’ll be there!
• Give us a follow on Facebook to stay in touch

PLAIN TEXT CTA IN EMAILS
Multiple CTAs, One Email

MORE PLAIN TEXT EXAMPLES
Multiple CTAs, One Email

#2. SELECT A SAMPLE SIZE AND TIME
FRAME
• Your split testing experiment has to have enough people.
• The experiment needs to go on long enough to get valid
results.

HOW MANY PEOPLE WILL I NEED?
• For email testing, your sample size needs to be your open rate.
• Landing pages, CTAs on your website, etc., it’s based on traffic.
• In a typical A/B test for email, 50% of subscribers get A, 50% get
B.

GENERAL SAMPLE SIZE TABLE

HOW TO CONDUCT AN EMAIL A/B
TEST
1. Select variation A and variation B.
2. Select how many subscribers you will need.
3. Send out the emails randomly to the subscribers in a 50/50 split.
4. Wait some time period until everyone has had a chance to open the
email and select a winner.

FATAL A/B TESTING FLAW
Never stop an A/B test early. For landing pages
and website based tests, 2 to 4 weeks is ideal.
Once you develop a time frame, don’t look until
the test is over.

DAYS 1-9. STOP HERE AND A WINS

AT DAY 14, B WINS

AT DAY 30, NO REAL RECENT WINNER

#3. BEFORE A/B, A/A TEST THE ASSET
• What if it wasn’t a change in subject line or layout that
resulted in a change?
• What if it was the behavior of your customers instead?
• In an A/A test, you perform a 50% / 50% split within your
subscribers, but they both get the A variation.

WHY?
A/B testers started to become suspicious of high
conversion rates given by split testing software (300%
conversion just by changing the color of a button!).
When an A/A test was performed, the
detected lift was still pretty high.

EMAIL CTA BUTTON TEST

EXAMPLE SPLIT TEST RESULTS
Blue Conversion Rate = 60%
Red Conversion Rate = 50%
LIFT FROM BLUE = 10%

EMAIL CTA BUTTON TEST: A/A

A/A TEST RESULTS
Red Button Conversion: 47.75%
Red Button Conversion: 63.03%
Lift: 16%

BLUE IS NOT THE WINNER

THE RESULTS FROM AN A/A TEST SHOULD
SHOW NO DIFFERENCE BETWEEN A AND A.

#4. DEVELOP A HYPOTHESIS, ANALYZE THE
DATA
• Your hypothesis is your assumption as to how the test will
go.Typically, you’ll state: “There is no difference between A
and B.”
• Set a desired lift in the desired metric for analyzing the
data.“I want to see a 20% lift, + or – 5% for this test to be
successful.”

THE UNCOMFORTABLE TRUTH
There are literally millions of variations you could make that would
improve or ruin your desired metric.You’re tests are only as good as
your assumptions as to what should be tested. Don’t get caught up in
testing. Get caught up in gathering data and intelligence to make your
email / website / landing page / subject lines better for your audience.

SPLIT TEST TO CREATE A BETTER
EXPERIENCE
The ultimate goal of split testing is to not
have to split test anymore until there is a
valid reason for doing so.

PRACTICAL SPLIT TESTING CASES

NETFLIX LANDING PAGE A
Chrome Browser landing page. Netflix giving me, based on past data, what I want.

NETFLIX LANDING PAGE B
Firefox Browser Landing Page Result. Netflix B variation is a scroll through of their original
programming.Variation A indulged my current interests.Variation B is about future value.

MORE SENSIBLE SPLIT TESTING
CONTROL EMAIL
Same offer, two different email addresses, eight hours apart. From Macy’s.

BUTTON TESTING

WHAT LARGER TESTS LOOK LIKE

BIGGER CHANGES IN EMAIL
12: 52 PM
Gmail

1:56 PM
Business
Email

LANDING PAGE TEST SEQUENCE
PROBLEM:After clicking a button in our email,
the subscriber arrives at a landing page.We
want to increase conversions on that landing
page by 10%.

VARIATION A

THE METHOD
• Sampling Method: With 5,000 subscribers, we must
randomly sample 726 customers in our database. 363
receive A, and 363 receive B in a 50/50 split.
• Length of Time: Check results five days from the time the
email is sent out.

VARIATION B

THE RESULTS ARE IN

WINNER

BUT WAIT….WE NEED AN EXTRA STEP
Always review the quality of the data to make sure everything was fair
in the experiment. If not, you may be using bad data.

THE IDEAL SCENARIO

BUT DATA CAN BE MESSY

PROBLEMS AND FIXES
• Different open rates lower the validity of your data.
• You can’t use the results even though there was a clear
winner.
• You can re-run the test with a new sample 50/50 split.

EASY SPLIT TESTING IN PINPOINTE

RECAP
1. 4 Simple Steps to SplitTesting.
2. Have a business justification for the best testing results (you’re only as good as your
assumptions).
3. Always establish a baseline metric with an A/A test.
4. Don’t create data for the sake of creating data.
5. Don’t spend all your time testing. Spend all your time improving. Split testing is only a tool.

Questions
Ask Michael…
Digital Media Strategist
Pinpointe On-Demand
mpelosi@pinpointe.com
Twitter/Facebook: @Pinpointe
www.pinpointe.com

We Thank You
We appreciate you taking the time to watch this today!

Check Out Our Blog
www.pinpointe.com/blog
Get more great marketing content.
Easily create automated campaign sequences.
www.pinpointe.com/get-started - Select program & enter code PPTWEBNR
FREE MONTH of Pinpointe Service (limit 100K emails)
Try Pinpointe’s Email Marketing Solution for FREE

Follow
for marketing tips & info
@pinpointe
www.facebook.com/Pinpointe
www.linkedin.com/company/pinpointe
Pinpointe

Beyond A/B Testing: How to Take Your Split Tests to the Next Level

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Beyond A/B Testing: How to Take Your Split Tests to the Next Level

Similar to Beyond A/B Testing: How to Take Your Split Tests to the Next Level (20)

More from Pinpointe On-Demand

More from Pinpointe On-Demand (20)

Recently uploaded

Recently uploaded (20)

Beyond A/B Testing: How to Take Your Split Tests to the Next Level