DM 201 - Avalon Consulting - Testing Creative

1
Avalon Consulting Group, Inc.
All rights reserved, 2014
DIRECT MARKETING 201
The Strategy and Metrics of Creative Testing
Avalon Consulting Group
May 14, 2014
www.avalonconsulting.net

2
Testing QUIZ!
Which Test Won?

3

4

5

6

7

8
And the Winners Are…

9
In case you’ve forgotten,
your opinion doesn’t
matter…

10
Creative Strategy & Testing

11
TESTING STRATEGY
Testing is essential to have a healthy, productive
direct marketing program.
1. Each organization is different and not all best practices are universal.
2. Used to determine what works for your organization.
3. Test making incremental or large changes, seeing their impact, while minimizing
the overall risk.
4. Help find ways to improve performance, can save money and improve the bottom-
line.
Things to remember about testing:
 Test one element at a time
 Test to save cost (formats, components) or boost response rate or average gift
(offer, teasers, treatments)
 Ensure the statistical significance of your test panel.

12
CASE STUDY: A Snapshot of the
NPCA Acquisition Testing History

13
NPCA Acquisition Control Packages

14
DM ACQUISITION CONTROL PERFORMANCE
($30.00)
($25.00)
($20.00)
($15.00)
($10.00)
($5.00)
$0.00
July2011
Aug2011
Sept2011
Oct2011
Nov2011
Dec2011
Jan2012
Feb2012
March2012
April2012
May2012
Survey net/donor
Member card net/donor
• The FY12 overall results of
the acquisition controls have
shown a weakening of the
member card, with the
survey more cost effectively
acquiring donors.
• However, Avalon analysis of
the member card showed a
stronger ROI, so both
packages continue to be
mailed as co-controls in
FY13.

15
Test Within Your Acquisition Control Packages
Premium (cost savings)
CONTROL: Calendar and bucket hat
TEST: Calendar only
WINNER: Control - Calendar bucket
hat
Ask String (boost avg. gift)
CONTROL: $15 minimum gift to
receive premium
TEST: $25 upgrade ask test
WINNER: There was no statistical
winner in the original test, but we
mailed a panel of this package every
month. Backend analysis shows ROI
of $25 ask is stronger.
Combine elements of co-controls
CONTROL: Member card package or
Survey package
TEST: Member card in Survey package
WINNER: Test – Member card in the
Survey package. Re-test confirmed
results.
Combine tone of Survey and Member Card
CONTROL: Member card package or
Survey package
TEST: Combine the urgency of the
member card with the emotion of the
survey letter.
WINNER: There was no statistical
winner. Re-testing to get significant
results.
Phrase-based formatting (boost response rate)
CONTROL: Control acquisition package
TEST: Use ReadSmart to make subtle
changes in the arrangement of the text
on the page
WINNER: There is no statistical
winner.
Removal of insert (cost savings)
CONTROL: Control acquisition package
TEST: Remove glossy color insert
WINNER: Control with the insert.

16
Test New Acquisition Packages
Map package
Included a foldout wall map and personalized teaser with a national park in the prospect’s
state.
The test has a 0.91% response rate – comparable to the control’s 0.92%, and a higher average
gift than the control ($23.13 vs. $22.73), but the costs were too high.
Patriotic package
Used a stirring, patriotic tone to highlight the bond Americans feel with their national parks,
and included a lift note with a quote from Ken Burns.
Avg. gifts are only $0.30 apart, the strong control had the higher response rate (0.99% vs. the
test’s 0.83%) and revenue/M - $234 vs. the test’s $196.

17
ACQUISITION TESTS – NEW PACKAGES
Postcard Package (Merge K) This package was developed to try and incorporate more park images into
the acquisition with the theory that if someone was reminded of the beauty they were helping to
protect, they would be more responsive. The “Wish you were here!” package included a 4-color insert
with four postcards showcasing the grandeur of the parks. The letter invited members to enjoy our
parks now, and support them for generations to come, by joining NPCA.
The postcard test had a significantly weaker revenue/M, compared to the member card and the survey.

18
ACQUISITION TESTS – NEW PACKAGES
Magazine Package The magazine subscription package included a 4-color sample of the National Parks
magazine, and emphasized the magazine as NPCA’s premier membership benefit.
In addition to a weaker average gift, the package currently has a 10% weaker RR than the survey
package and a 30% weaker RR than the member card, as well as a weaker revenue/M.

19
Review Acquisition Results by Premium
$0.00
$5.00
$10.00
$15.00
$20.00
$25.00
Anorak Bison Blanket Bucket hatTote bag
Survey
Survey
Review back-end premium
costs to acquire a donor.
Include front-end as well as
back-end costs.
The anorak, blanket and bucket
hat all have comparable results
in the member card package
and can be mailed
interchangeably, but in the
survey, the blanket, tote bag
and bucket hat are strongest
because their costs to acquire a
donor are lower.
$10.20
$10.40
$10.60
$10.80
$11.00
$11.20
$11.40
$11.60
$11.80
Anorak Bison Blanket Bucket hat Tote bag
Member card
Member
card

20
Online Activist Conversion Results
The Sept pre-email was the first contact in the series. Both Group A
and Group C had a 0.05% response rate.
The Oct DM acquisition was sent to groups A and B – group A had
received the pre-email and group B did not. Group A had a 1.03%
response rate, while group B had a 0.98% response rate. By
comparison, the overall mailing had a 0.86% response rate.
The Nov TM acquisition made 4,723 contacts and 286 members joined
with an average gift of $27.83. Groups A and B had nearly identical
results on the phone.
The Dec post-email was the last contact in the series. Group B had a
0.10% response rate, while Group C had 0.08%. The modeled groups
had an average gift of $31.30. In addition, the non-modeled group had
a $26.75 average gift.

21
Testing for Impact

22
Testing Carriers:
Getting them into the envelope

23
Image vs. No Image

24
Plain Carrier
Test
Tested against 4-
color control
carrier
10
Plain vs. 4-Color Carrier

25
9
Teaser and Photo Testing

26
Testing Offers:
Compelling them to Respond

27
Discount Offer
Test
Added discount
to carrier teaser/
changed PS
discount
language from $3
to 10% discount
9
Discount offer

28
Ask string testing
JUNE: Control asking string was tested against descending asking string and asking
string with only three asks.
 In both cases, control is maintained with higher response rate and higher
gross/thousand.
Control
Test 1
Test 2

29
C3 vs. C4 DM & EM SURVEY APPEAL
JANUARY mailed annual survey in
official-looking carrier with companion e-
appeal. Messaging focused on state of
our democracy.
TESTS: 501c3 version of appeal was
tested.
 DM: C3 test version was winner at
85% confidence with higher response
rate (4.85% vs. 4.75%) and $1.46
higher average gift.
 EM: C4 control had higher response
rate (0.28% vs. ).24%), but $17.91
lower average gift. There was no
statistical winner.

30
Testing Engagement Devices:
Engaging Beyond Giving

31
9
A Petition as a Call to Action

32
The NPCA Mining Appeal exceeded
budgeted revenue goal by 17% and net
by 33%.
The drive to web test, with the URL
listed on the reply and in the P.S of
the letter earned more revenue
due to a higher response rate and
beat the control at 84% confidence.
Drive to Web Test

33
Testing Messaging

34
PILOT CAMPAIGN
Package #1: From the Field/Mike Fay

35
PILOT CAMPAIGN
Package #2: RSVP/Invitation

36
Testing Tone and Language: Aggressive

37
Testing Tone and Language: Cooperative

38
Testing to Settle an Argument

39
Testing a Long-Standing Control

40
Matching Gift Appeal Test

41
Matching Gift Appeal Test

42
Logo Testing

43
2 p vs. 4 p

44
And Don’t Forget:
Testing to Reduce Costs

45
Some of our Favorite Cost-Saving Tests:
Test 1st vs. 3rd Class postage
Test a modified or standard size or format
Test eliminating an insert
Test an E-R0 effort to invite people to renew
online and not receive a DM renewal

46
The Analytics of Testing

47
• There are two phases to validating and analyzing direct marketing
testing. The first is ensuring that the test sample size is valid.
• A sample size that is too small will not provide reliable results.
Part 1: Sample Size Calculator
Go to:
http://www.avalonconsulting.net
/Sample-Size-Calculator.html
Recommendation: Perform this
calculation before the campaign is
conducted to guide the size of the
test and to ensure testing is
worthwhile.

48
• Step 1: How much do I need to mail?
• Enter proposed volume for control or test
panels.
• Step 2: Is there a valid percent return?
• Enter expected response (pre-campaign,
use a conservative estimate.)
• Step 3: Is my confidence above 80%?
• If not, remove either quantity or response to
determine what values are needed.
Repeat with actual campaign results
if needed and if fluctuating from
proposed.
Part 1: Sample Size Calculator

49
Part 2: Testing Confidence Calculator
• After a valid sample size is achieved, the next step is to determine
if the results of the test are significant.
• Important: Just because there is a difference in performance
does not mean the performance can be replicated. Often, it is
within the margin of error – a flip of the coin.
• Only make decisions on valid results.
• Goal: Determine if the difference in net per thousand mailed
(other metrics can be used) is significant or within the margin
of error. Typically, a confidence level of 80% is a starting point,
with 95% being ideal.

50
Testing Confidence Calculator – Process
• Step 1:
• Gather control and test panel quantities, gifts, revenue, and roll out costs.
• Remove outlier gifts from quantity, gifts, and revenue. A typical rule of thumb is
$1,000 gifts or more.
• Step 2:
• Calculate “upper bound” and “lower bound” for the response rate, revenue per
thousand, and/or net per thousand. It is typical to start with the standard normal
deviate of 1.96.
• Step 3:
• If there is no overlap between the upper bound of one panel and the lower bound
of another, a confidence level will be found.
Upper Lower
Net/M % # Bound Bound
Test Group Net/M Winner Conf. Conf. Net/M Net/M
Control $1,477 - $1,611 $1,343
Test $1,826 WINNER $1,982 $1,670
95% 1.96

51
Thank you!
Allison Porter
President
Avalon Consulting Group
2030 M Street, NW Suite 700
Washington, DC 20036
202-429-6080 x102
Allisonp@avalonconsulting.net
Follow Avalon on LinkedIn Today!

DM 201 - Avalon Consulting - Testing Creative

Recommended

Recommended

More Related Content

Similar to DM 201 - Avalon Consulting - Testing Creative

Similar to DM 201 - Avalon Consulting - Testing Creative (20)

More from Avalon Consulting

More from Avalon Consulting (20)

Recently uploaded

Recently uploaded (20)

DM 201 - Avalon Consulting - Testing Creative