A presentation on multivariable testing in Direct Marketing and Direct Mail. Marketing Day 2010.

1. Multivariable Testing in Direct Mail
Efficient tests with limited resources
Gordon H. Bell
President
LucidView

2. Analytics & Testing: 2 sources of insight
Testing
– Front-end planning
– Prove cause-and-effect
– Ask new questions
– Design the test to give the specific
data you want
Analytics
– Back-end analysis
– Look for correlations
– Answer questions, but cannot ask any
– General data

3. Testing has always been important
• “Almost any question can be answered,
cheaply, quickly and finally, by a test campaign.”
– Claude Hopkins, Scientific Advertising, 1923
• “Testing is still the best way to find true breakthroughs.”
– Stone & Jacobs, Successful Direct Marketing Methods, 2008

4. is now more efficient
Testing has always been important
Many questions
• “Almost any question can be answered,
cheaply, quickly and finally, by a test campaign.”
^
multivariable

5. What is Multivariable Testing?
• Many different test designs and statistical
techniques developed to test variables more
efficiently
• Change many variables at once – in an organized
way – so you can separate out the impact of each
– Number of test “recipes” (test versions)
– Each with a unique combination of all elements
– Analyzing all recipes together to quantify effects
• Test design =
collection of related test recipes

6. 4 Key Benefits
Benefit Scientific Multivariable Testing Split-run Testing
Efficiency 2 - 35 variables in one test 1 variable = 1 test
Speed and Constant sample size Increasing sample size
Sample size (no matter how many variables)
Depth of Accurate, robust, and comparative One main effect
insights main effects and interactions (conditional, with no interactions)
Flexibility Wide range of test designs One choice

7. Getting Started
Important steps include how to:
1. Define the scope
2. Select a good design
3. Determine the right sample size
4. Create multivariable test recipes
5. Analyze and interpret results

8. #1 Define the Scope
Where is multivariable testing valuable and
manageable?
• High potential return-on-investment
• Clear metrics
• Fast results
• Flexibility to create many versions of the
mailing
• Insights can be applied long-term

9. Financial Times: Price Test
(from the DMA07 Conference)
• Focus on price testing
• Measure response rate and revenue
• Test in one direct mail campaign
• All variables lasered in black ink
(easy and inexpensive to change)
Test Elements (-) Control (+) New Idea
A Price $49 for 26 weeks $99 for 12 months
B Longer term (+25%) No Yes (32 weeks / 15 months)
C Shipping & handling fee No Yes, $4.95
D Streamline price box No Yes, redesign price box and
(3 lines vs 5) reply slip

10. FT “control” mail package

11. #2 Select a Good Test Design
Many choices of multivariable test designs
• Which “handful” of combinations should be tested to
clearly see results?
• Want to have
– Few test recipes (versions of the mailing), but…
– Lots of data for better insights
– Clear, actionable results
– A test design that the marketing team can
understand and believe

12. Financial Times:
Streamline price box
Streamline price box
Longer term (+25%)
Longer term (+25%)
S&H (delivery) fee
S&H (delivery) fee
test design and recipes
Price
Price
Recipe A B C D Mailed Recipe A B C D AB AC BC Mailed
1 – – – – 25,000 1 – – – – + + + 25,000
2 + – – + 25,000 2 + – – + – – + 25,000
3 – + – + 25,000 3 – + – + – + – 25,000
4 + + – – 25,000 4 + + – – + – – 25,000
5 – – + + 25,000 5 – – + + + – – 25,000
6 + – + – 25,000 6 + – + – – + – 25,000
7 – + + – 25,000 7 – + + – – – + 25,000
8 + + + + 25,000 8 + + + + + + + 25,000

13. Financial Times:
Streamline price box
Longer term (+25%)
S&H (delivery) fee
Price
test design and recipes
B: Longer term D: Streamline
A: Price C: S&H fee
Recipe A B C D Mailed (+25%) price box
Recipe 1 No
1 – – – – 25,000 (control)
$49
($49 for 26 weeks)
No No
No Yes, redesign
2 + – – + 25,000 Recipe 2 $99
($99 for 12 months)
No
price box
Yes Yes, redesign
3 – + – + 25,000 Recipe 3 $49
($49 for 32 weeks)
No
price box
4 + + – – 25,000 Recipe 4 $99
Yes
($99 for 15 months)
No No
5 – – + + 25,000 Recipe 5 $49
No
($49 for 26 weeks)
Yes, $4.95
Yes, redesign
price box
6 + – + – 25,000 Recipe 6 $99
No
($99 for 12 months)
Yes, $4.95 No
7 – + + – 25,000 Recipe 7 $49
Yes
($49 for 32 weeks)
Yes, $4.95 No
Yes Yes, redesign
8 + + + + 25,000 Recipe 8 $99
($99 for 15 months)
Yes, $4.95
price box

14. #3 Find the Right Sample Size
The best sample size depends upon
– Response rate (or variation in sales)
– How small a change you want to see
Lower response rate, or the need to see a
smaller change, requires a larger sample size.

15. Sample Size (N) example
• For Financial Times…
– Say we usually have a 1.5% response rate (R)
– And we want to see if any test elements affect
response by 10% or more (0.15%)
– Then…
31.38 ● R ● (1-R)
N=
(smallest change) 2
31.38 ● 1.50% ● 98.5%
N= 2
= 206,062
(0.15%)

16. #4 Create the multivariable recipes
• The science determines which “handful”
of combinations should be tested
• The art of testing is making sure…
– Elements remain bold
(in the difference between “–” and “+” levels)
– Execution is clear and consistent

17. Creatives: control (recipe 1) and “all plus” recipe 8
D+
A+ C+
B+

18. #5 Analyze and Interpret Results
• A well-designed test is relatively easy to analyze
– Analysis of variance, regression, or…
pencil and calculator
(with some more in-depth analysis)
• Three main steps in the analysis are:
1. Analyze main effects (each column in the design)
2. Analyze interactions
3. Consider alternative solutions
and additional metrics

19. FT price test: main effects & interactions
FT Price Test: Gross response rate
optimal = 32.7% lift
B: Longer term
(yes, +25%)
18.9% increase in response versus the control +
… unless both are set
C: Shipping &
at the best combination
17.2% drop -
handling fee (no)
BC interaction 13.8% further increase in response if... -
BC Interaction
D: Streamline
-
price box
A: Price -
AC interaction + Significant effects
(beyond line)
AB interaction +
B-: Current B+: Longer term (+25%)
0.000% 0.010% 0.020% 0.030% 0.040% 0.050% 0.060% / 12 mo)
(26 wk 0.070%
Term and S&H do not Subscription Term
affect response rate…
C-: No S&H fee C+: Add $4.95 S&H fee

20. Financial Times: summary
Multivariable testing offered a way to
• Test easy, high-impact changes
• Learn more with a small sample size
• Increase insights
– More accurate main effects
– Clear interactions
• In contrast, with split-run testing
– We would have needed 3x the sample for equal confidence
– At this sample size, not one effect would be significant
– Interactions would be impossible to see

21. Hearst Launch of the
Food Network Magazine
(presented at the DMA09 conference)
New magazine for a new market =
Need for speed
• Learn fast  ramp-up quickly
• Test a lot of ideas to see what works
• Increase confidence in test results for a
rapid roll-out
• Optimize price/offer along with creatives

22. Food Network Magazine Creative Test
(13 Creative Elements)
Changes to the outer envelope
A Window on OE H Reply By date
B Change logo on OE and form J Add chef photos on form
C Return address on OE K Change description
D Form layout L Brochure
E Change "Charter Discount" M Chit
F Audience Development copy N Lift note
G Savings presentation
Additional inserts

23. Elements on the Envelope
B: Logo C: Return Address A: Window

24. B: Change logo
E: Change “Charter Discount”
F: Audience Development copy
G: Savings presentation
H: Reply-By date
J: Add chef photos
K: Change description
D: form
layout

25. Change logo on OE and form
Audience Development copy
Change "Charter Discount"
16-recipe
Add chef photos on form
Return address on OE
Savings presentation
Change description
Creative Test
Window on OE
Reply By date
Design
Form layout
Brochure
Lift note
Chit
Net
Recipe A B C D E F G H J K L M N Mailed
response
1 – – – – – – – – – – – – – 25,000 0%
2 + + – – – – + + – – + + + 25,000 -21%
3 – – – + – + + + + – – + – 25,000 -8%
4 + + + – + – – – + – – + – 25,000 -6%
5 + + + + + + + + – – – – – 25,000 -11%
6 – + – + + – + – – + – + + 25,000 -25%
7 – – + – + – + + + – + – + 25,000 -20%
8 + – + – – + – + – + – + + 25,000 -10%
9 + – + + – – + – + + – – + 25,000 -11%
10 + – – + + – – + – + + – – 25,000 -42%
11 – + – – + + – + + + – – + 25,000 -20%
12 + + – + – + – – + – + – + 25,000 -10%
13 – + + + – – – + + + + + – 25,000 -31%
14 – – + + + + – – – – + + + 25,000 -26%
15 + – – – + + + – + + + + – 25,000 -37%
16 – + + – – + + – – + + – – 25,000 -38%
400,000

26. Test Elements (on the 1-page form)
Control (recipe #1) Recipe #11
B+
E+
F+
J+
K+
H+

27. FNM: Net Response Rate
Net Response Rate
optimal = 9.75% lift
Test element (optimal setting)
L-: Brochure (yes) Response drops 16.8% when brochure is removed
K-: Change description (control) New description reduces response 14.3%
E-: Change Charter Discount (no) "Research Discount" hurts 7.3%
J+: Add chef photos on form (yes) 3.8% lift
N+: Lift note (yes) 3.5% lift
G-: Savings presentation (control) -3.1%
A+: No window on OE (closed-faced) +2.5%
D-: Form layout
H-: Reply By date
M-: Chit
C+: Return address on OE
Significant effects
B-: Change logo on OE and form
beyond line
F-: Audience Development copy
0.00% 0.25% 0.50% 0.75% 1.00% 1.25%

28. Food Network Magazine Results
Multivariable "refining" tests
(July 09 test results)
8.8% increase in NRR
17.3% increase in Profit
Multivariable "screening" tests
(Mar09 tests - Jul09 confirmation)
8.4% increase in NRR 35% jump in profit
17.6% increase in Profit over 2 campaigns
Original control (versus 2 years with A/B splits)
(March 2009)
(sample size of 600,000 vs.
4.7 MM for split-run testing)

29. Whirlpool Contact Strategy Test
(presented at DM Days – New York, 2007)
Focus = Annual renewal of the extended service contract
Test Elements
A Timing
B Early mail drop
C Expiration mail drop
D Late mail drop
E Outer envelope
F Lift note
G Phone Call in Week 1

30. Contact Strategy Test
G+
A+ B+ E+
X C+ E+ D+
Control B+ E+
X C+ E+ D+
Week: 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7
Expiration Date

31. Expiration mail drop
Whirlpool
Outer envelope
Early mail drop
Late mail drop
Test Design
Lift note
Timing
Phone
Recipe A B C D E F G Response Week 1 Week 2 Week 3 Week 4
1 - - - - - - - customers customers customers customer
2 + - - - + - +
3 - + - - + + -
4 + + - - - + +
5 - - + - + + +
6 + - + - - + -
7 - + + - - - +
8 + + + - + - -
9 - - - + - + +
10 + - - + + + -
11 - + - + + - +
12 + + - + - - -
13 - - + + + - -
14 + - + + - - +
15 - + + + - + -
16 + + + + + + +

32. Test Results
Effects: Total Response Rate
A: Timing (spread out) -4.67%
D: Late mail drop (yes) +3.00%
G: Phone call (yes) -2.68%
AB (add early drop) -2.43%
BD +2.22%
ABD +1.92%
B: Early mail drop (yes) +1.61%
F: Lift note +1.41%
AG -0.83%
AD +0.79%
C: Expiration mail drop +0.63%
E: Outer envelope +0.61%
Significant effects
AF +0.27% (beyond line)
AC -0.11%
AE +0.02%
0.0% 0.5% 1.0% 1.5% 2.0% 2.5%
(effects in percentage points on bottom; as % of control on end)

33. Interaction Effect
AB Interaction (total response rate)
49%
But it does help if
contacts are kept
48%
spread out
47%
On average,
the early mail drop 46%
(B) has no impact
45%
A-: control (9,4,1,-4) A+: condensed (6,3,1,-3)
A: Timing of contacts
B-: No early mail drop B+: Yes (+14 or +9)

34. Whirlpool Test: summary
• Response rate increased 7.81% (7.3% predicted)
• 8.3% increase in revenue (worth $ millions)
• 6.76% increase in annual profit
• With split-run tests, would have…
– Needed 2½ years of testing (vs. 4 months)
– Completely missed the interaction
– Seen nothing significant in 4 months
(due to 3x greater error)

35. “Testing is still the best way
to find true breakthroughs.”
– Bob Stone and Ron Jacobs,
Successful Direct Marketing Methods, 2008
“ Like trading in your bicycle for a
helicopter, multivariable testing gives you the
speed,
power, and flexibility to get farther, faster.”
– Gordon Bell,
Marketing Day, 23 March 2010