This document summarizes the results of various direct marketing tests conducted by Avalon Consulting Group to determine the most effective strategies and creative elements. Some key findings include: a red/gold carrier outperformed a plain carrier in response rates and revenue; including a photo or teaser increased response rates and revenue by 21%; tests of different messaging tones found that a cooperative tone performed better than an aggressive one; and tests of offers like matching gifts found substantial response and revenue lifts compared to non-matching gift campaigns. The document emphasizes that testing should have clear goals, change one element at a time, and use statistical methods to determine significant results.

1. 1
Cover Page
Direct Marketing 201
The Strategy and Metrics of Creative Testing
Presented March 29, 2017
Avalon Consulting Group
avalonconsulting.net ©2017 Avalon Consulting Group.
All Rights Reserved.
avalonconsulting.net

2. 2
Testing QUIZ!
Which Test Won?

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8
And the Winners Are…

9. 9
LESSON:
Your opinion doesn’t matter.

10. 10
Testing can have many goals:
• Get people to open your solicitation
• Increase response/revenue
• Include engagement
• Settle an argument
• Determine what messaging resonates
• Save costs

11. 11
Testing Carriers:
Getting them into the envelope

12. 12
Image vs. No Image
 The red/gold
carrier had a higher
response rate,
average gift, and
16% higher REV/M.

13. 13
Teaser and Photo Testing
 21% increase in
revenue/m –
response rate and
avg. gift stronger

14. 14
Testing a Deadline
 Deadline won with a 26%
higher Rev/M due to a
higher average gift and
flat response rate

15. 15
Control
Test
Plain vs. Image Carrier
Front
 Plain carrier had a
26% higher
response rate and
higher average gift

16. 16
And a reminder to check your test
panels…
INVALID

17. 17
Testing Offers:
Compelling Donors to Respond

18. 18
To boost response: Include call to action
 30% increase in
response rate, and
20% lift in average
gift!

19. 19
To boost response: Include matching gifts
$0
$500
$1,000
$1,500
$2,000
$2,500
$3,000
$3,500
Campaign 1 Campaign 2
Matching Gift No Matching Gift
76%
higher!
94%
higher!

20. 20
Discount offer
 Brought in 10%
more members,
discount offer
won at 90%
confidence.

21. 21
ROI after 1 year
Ask string testing

22. 22
Premium vs. No Premium
-$11,445 -$11,710
No
Premium
test joins
had the
same
long-term
net
revenue.
With a
premium
at $15.
$9.18
$11.47
The No
Premium
test joins
had a
higher
donor
value.

23. 23
Testing Engagement Devices:
Engaging Beyond Giving

24. 24
Member Card Test

25. 25
Mission-Relevant Engagement Test
Placemat
Place Cards

26. 26
Petition to Congress Test

27. 27
Testing Messaging

28. 28
Testing Tone and Language:
Aggressive vs. Cooperative

29. 29
Part I: Pilot Acquisition Conversion
Campaign
Target Launch: October 2013
Testing tone and language:
Story vs. traditional RSVP invitation
Concept #1: Letter from the Field Concept #2: RSVP/Invitation

30. 30
Concept: Member Card
Testing tone and language:
RSVP vs. Member Card

31. 31
Testing tone and language:
Testing across channels

32. 32
Testing Tone and
Language
Test
Control

33. 33
Testing to Settle an Argument

34. 34
Testing a Long-Standing Control

35. 35
Logo Testing

36. 36
2 page vs. 4 page

37. 37
And Don’t Forget:
Testing to Reduce Costs

38. 38
Some of our Favorite Cost-Saving Tests:
Test 1st vs. 3rd Class postage.
Test a modified or standard size or format.
Test eliminating an insert.
Test an E-R0 effort to invite people to renew online and not receive a DM renewal.
Machine vs. actual handwriting.
Use less color for more impact.
Reduce personalization and resulting matching costs
Use a drop-cut format rather than a separate reply form
Use a window carrier instead of a closed face envelope.

39. 39
The Analytics of Testing

40. 40
How to set up your test
To ensure valid testing results, it is important to
follow the guidelines below:
Ensure the
statistical
significance of
your test – use a
sample size
calculator to
accurately project
your panels.
Ensure the goal
of the test is
clear – what will
be the measure
of success?
• 20% lift in response
rate? 10% lift in
average gift? 15%
reduction in cost?
Test one
element at a
time to isolate
the impact of
the variable
change.
• Carrier? Copy? Ask
string?
Choose tests
where there is
a “reason to
believe” the
change will
improve the
campaign.
• What information
informs the testing
decision?

41. 41
How to measure your test
It is important to accurately read the results of your
test, beyond just “eye-balling”:
A difference in results may be insignificant and
within the margin of error.
Use a calculator designed to identify significant
differences before determining a winning test.
Use roll-out costs to extrapolate the true impact of
the test at full quantities.

42. 42
Thank you!
Mina Stanard
Senior Vice President
Avalon Consulting Group
805 15th Street, NW Suite 700
Washington, DC 20005
202-627-6538
minas@avalonconsulting.net
Connect with Avalon Today!