The document summarizes an accelerated testing approach used by Hearst Magazines to test retention and customer relationship management (CRM) programs. It describes challenges with traditional testing given long testing cycles and limited volumes. A "mosaic" test design was used testing 17 elements across 11 creative changes and 6 offer/contact strategy elements. This allowed testing many ideas simultaneously in a shorter time period to gain insights into interactions. Results showed significant effects from elements like magazine branding and guarantee language. The analytical plan involved segmenting samples into groups testing different element combinations.

1. Accelerated Testing
for Retention and CRM Programs
Brian Fichtel
Senior Director, Customer Retention
Hearst Magazines
Gordon Bell
President, LucidView
Managing Partner, Artestry
gbell@lucidview.com
1

2. Overview of Retention / CRM programs
• Multiple effort subscription renewal series utilizing a mix
of direct mail, e-mail, telephone and on-magazine efforts
• Generally segment by original acquisition source
(e.g. Direct Mail, Internet) and subscriber tenure
• Series runs for as long as 9 months
• Retention efforts are highly responsive and highly
profitable but have small quantities compared to
Acquisition Direct Mail
• All of the above makes testing a long, challenging
process with limited testing opportunities
2

3. Control Renewals / Retention Program
4 months 1 month after
before expiration expiration
- 3 months + 2 months
- 2 months + 3 months
- 1 month + 4 months
Subscription
expiration
3

4. CRM Testing: Real-World Challenges
4 months
before expiration
- 3 months
Subscription Direct mail Internet
Prior
- 2 months segments Renewals
- 1 month
Town & Country
Car and Driver
Country Living
Housekeeping
Woman’s Day
Food Network
Road & Track
Cosmopolitan
Marie Clair
Seventeen
Mechanics
Magazine
Magazine
20 U.S.
Redbook
Veranda
Popular
Esquire
Oprah
ELLE
Good
Subscription Magazines
Expiration
Monthly
Month Month Month Month
+ 2 months expire 1 2 3 4
groups
+ 3 months
+ 4 months
4

5. “Business Case” for Multivariate Testing
• Many creative, offer and segmentation test ideas but
small retention quantities that limit testing opportunities
• Started with Envelope size
Postage
Stronger copy pushing payment
Laser magazine title on all letters
41 ideas to test… Return slip position
Branding
More aggressive in promotion of RAB discount
Increase urgency around expire
Format Attached mail
Offer and payment info Same form with all efforts
Urgency Postcard with drive-to-web
Magazine benefits copy Offer expression - $0.97 vs. rounded up (9.97 vs. 10)
Guarantee No discount for late effort High segments
Payment chit Incentive - extra issues vs. pricing
Highlight discount Credit card renewal option
Change price 2- and 3-year renewal option
Incentive for early renewal Hearst Rewards (earn points)
Late-series incentive Renew now for digital copy/app
Multi-year offer Special offer upon cancel request
Compress contact stream Add a chit - gift/cross-sell subscription
More e-mail Premium with 2- or 3-year term
Start renewal cycle earlier Shorter term offer (6 issues) offer
Add telemarketing efforts Sweepstakes entry for early renewal
E-mail beforehand Magnitude of price change (small/medium) 5
Decrease time between efforts

6. “Business Case” for Multivariate Testing
• Multivariate testing provided opportunity to test many
ideas in a short period of time
– Enabling results in 9 months instead of 3 years
• Multivariate testing also provided an opportunity to
gain insights about test element interactions
that’s not possible with traditional A/B testing
6

7. Test Design: Universe of Possibilities
2-level
Test elements
Selected designs A/B split-run
tests
Orthogonal test designs
• Full factorial Taguchi designs
• Fractional factorial • Use many levels
• Bold assumption that
•Plackett-Burman interactions don’t exist
Optimal designs
• Complete freedom
• Cost = complexity
and error if assumptions
•Centerpoints
are wrong • CCD (etc.)
7

8. “Mosaic” Test Design
• Two test designs
1. 11 creative changes
2. 6 elements of the offer and contact strategy
• Why?...
a) Number of test elements
b) Magnitude of impact (creative changes tend to have a smaller effect)
c) Value of interactions – mix of contacts and offer elements tend to have
larger, more valuable interactions
• Choices
1. 12-recipe test for 11 creative elements ( = fewer package versions)
2. 16-recipe test for 6 offer and creative elements ( = more interactions)
8

9. CRM Testing: Real-World Challenges
4 months
before expiration
- 3 months Subscription Direct mail Internet
Prior
segments Renewals
- 2 months
Housekeeping
Country Living
Woman’s Day
Food Network
Cosmopolitan
Road & Track
Marie Clair
Seventeen
Mechanics
Magazine
Magazine
20 U.S.
Redbook
Veranda
Car and
Country
Popular
Esquire
Town &
Oprah
Driver
Good
ELLE
- 1 month Magazines
Subscription Monthly
Month Month Month Month
Expiration expire 1 2 3 4
groups
+ 2 months 11 creative 6 elements of
4
Test ideas segmentation elements the offer and
models (2,048 possible contact strategy
+ 3 months combinations) (96 combos)
+ 4 months
9

10. Test Elements
Brainstormed 41 new ideas  selected 17 test elements (split into 2 test designs)
Test Elements - Creatives (-) Control (+) New idea
A Envelope size Large Small
B Postage Metered Indicia
C Return slip position Top of form Bottom of form
D Branding Magazine branding Hearst corporation branding
E Format Letter (with signature) Statement (no signature)
F Offer and payment info On right side Move to left side
G Urgency Current urgency copy Different urgency copy
H Magazine benefits copy Control Stronger benefits
J Guarantee None Add guarantee copy
K Payment chit No Add chit
L Highlight discount No Yes
Test Elements - Offer and Strategy (-) Control (+) New idea
A Change price Control +/- 20%
B Incentive for early renewal No Yes, offer incentive in 2nd effort
C Late-series incentive Discount Premium
D Multi-year offer Only once Add multi-year option to additional efforts
E Compress series timing Control (every 4 weeks) Compress (every 2-3 weeks) and add efforts at end
F More e-mail efforts Control More e-mail following DM efforts
10

11. “Mosaic” Test Designs
Creative Test Offer & Strategy Test
Incentive for early renewal
Compress series timing
Magazine benefits copy
Highlight discount (later
Offer and payment info
Late-series incentive
More e-mail efforts
Return slip position
Guarantee copy
Multi-year offer
Envelope size
Urgency copy
Change price
Payment chit
Branding
Postage
Format
Recipe A B C D E F G H J K L Sample Recipe A B C D E F Sample
1 – – – – – – – – – – – 20563 1 – – – – – – 20,437
2 + + + – + + – + – – – 20588 2 + + – – – – 19,880
3 + – + + – + – – – + + 20480 3 – – – + – + 20,585
4 – – – + + + – + + – + 20471 4 + + + – + – 19,945
5 + + – + + – + – – – + 20466 5 + + + + + + 20,081
6 + – – – + + + – + + – 20457 6 – + – + + – 20,467
7 + – + – – – + + + – + 20581 7 – – + – + – 20,390
8 – + + – + – – – + + + 20633 8 + – + – – + 19,979
9 – – + + + – + + – + – 20569 9 + – + + – – 20,083
10 – + – – – + + + – + + 20455 10 + – – + + – 20,090
11 – + + + – + + – + – – 20516 11 – + – – + + 20,190
12 + + – + – – – + + + – 20528 12 + + – + – + 19,988
13 – + + + – – 20,516
14 – – + + + + 20,557
15 + – – – + + 19,951 11
16 – + + – – + 20,170

12. Creative Test (control)
12

13. Creative Test (2 versions)
E+
H+
G+
D+
C+
F+
13

14. Mosaic Creative Test – Results
Hearst Magazines CreativeTest: Renewal Rate
|Effects| as % change versus the control
Elements: sign of effect (optimal setting)
D-: Branding (magazine) -3.59%
J-: Guarantee (no) -1.48%
A-: Envelope size (large) -1.22%
L-: Highlight discount (no) -1.22%
G-: Urgency -0.81%
E-: Format -0.73% Four significant effects:
K+: Payment Chit +0.58% D-: Brand letters with the magazine title
(not Hearst Magazines)
F+: Of f er and Payment Inf o +0.39%
C-: Return slip position -0.23% J- and L-: Skip the guarantee language
and highlighting the discount
H-: Magazine benef its copy -0.16%
A-: Keep the larger envelope
B-: Postage -0.10%
0.0% 1.0% 2.0% 3.0%
Effect (avg RR of test as % change vs. control)
14

15. Incentive for early renewal
Mosaic Offer & Strategy
Compress series timing
Late-series incentive
Test – Design
More e-mail efforts
Multi-year offer
Change price
Recipe A B C D E F Sample
3-levels of price means that each A+
1 – – – – – – 20,437
2 + + – – – – 19,880
Recipe is divided into two groups:
3 – – – + – + 20,585 Half with a 20% drop in price
4 + + + – + – 19,945 Half with a 20% price increase
5 + + + + + + 20,081
6 – + – + + – 20,467
7 – – + – + – 20,390
8 + – + – – + 19,979
9 + – + + – – 20,083
10 + – – + + – 20,090
11 – + – – + + 20,190
12 + + – + – + 19,988
13 – + + + – – 20,516
14 – – + + + + 20,557
15 + – – – + + 19,951
16 – + + – – + 20,170
15

16. Mosaic Offer & Strategy Test – analytical plan
Incentive for early renewal
Compress series timing
Late-series incentive
More e-mail efforts
Multi-year offer
Change price
Total Sample Size
Segment 1 Segment 2 Segment 3
Recipe A B C D E F Model 1 Model 2 Model 3 Model 4 Model 1 Model 2 Model 3 Model 4 Model 1 Model 2 Model 3 Model 4
1 – – – – – – 20,437
2 + + – – – – 19,880
3 – – – + – + 20,585
4 + + + – + – 19,945
5 + + + + + + 20,081
6 – + – + + – 20,467
7 – – + – + – 20,390
8 + – + – – + 19,979
9 + – + + – – 20,083
10 + – – + + – 20,090
11 – + – – + + 20,190
12 + + – + – + 19,988
13 – + + + – – 20,516
14 – – + + + + 20,557
15 + – – – + + 19,951
16 – + + – – + 20,170
16

17. - 4 months
Offer & Strategy Test –
$ $
Execution: recipes 1 vs. 5
- 3 months
$ $
- 2 months
$ $
- 1 month $ $
$ $
Expiration
$ $
+ 1 month
$ $
+ 2 months
$ $
+ 3 months $ $
+ 4 months $ $
+ 5 months 17

18. Mosaic Offer Test – Results
Hearst Magazines Billing & Renewals Test: Renewal Rate
|Effects| as % change versus the control (optimal = 6.7% lift)
Elements: sign of effect (optimal setting)
C-: Late-series incentive (discount) -5.86%
B+: Incentive f or early renewal (yes) +3.04%
D-: Multi-year of f er (no) -2.43%
A+: Change price (yes) +1.56%
F+: More e-mail (yes) +1.22%
E+: Compress contact stream (yes) +0.86%
CF 0.67%
AF 0.65% All 6 elements are significant
AD 0.36%
(but no interactions)
AB 0.35%
C-, B+: Keep the later discount,
CD 0.25% but add and early-renewal incentive
ACD 0.17% E+, F+: Add more contacts – more e-mails
AC 0.12% and compressed timing
D-: Skip the multi-year offer
ACF 0.11%
A+: Change price (but how?)
AE 0.07%
0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0%
Effect (avg RR of test as % change vs. control)
18

19. Mosaic Offer Test – Effect of Price…
Effect (A): Price change
(change in Revenue versus control) To maximize revenue
250%
% change in revenue vs. control
• For magazine #1,
200% increase price 20%
(Revenue increases 19%
150% but response is 9% lower)
100%
• Increase price 20%
Magazine #1 for magazine #2
50%
Magazine #2 (Revenue increases 23%
but response is 22% lower)
0% Magazine #3
-50% • For magazine #3, remain
-20% Control +20% at the control price
A: Price
19

20. … Depends upon segmentation
Effect (A): Price change Differences by
(change in Revenue by Segment) Segmentation Model
250%
• “Loyal” customers
% change in revenue vs. control
200% (defined by a combination
of statistical models)
150%
#1 - Cancel risk  Respond well
#1 - Loyal to higher prices
100% #2 - Cancel risk
• Subscribers who are a
#2 - Loyal
50% “Cancel risk” are better
#3 - Cancel risk
served at the control or
0% #3 - Loyal lower price…
-50%
-20% Control +20% Because… Need to consider response
(not shown) along with revenue
A: Price
 Renewal rate drops up to 26%
for this high-risk segment
 And we need to deliver promised rate base
 Plus renewals are lower cost
than new subscribers 20

21. Summary of Mosaic Tests
• Tested 17 marketing-mix elements at once
• Quantified 10 significant elements
– Analyzing main effects, interactions, and “curvature”
in the price effect
• Pinpointed differences by segment, title, and
statistical model
• Tested in 9 months what would require more
than 3½ years with A/B splits
21

22. Conclusions & Insights
• Benefits of multivariate testing for Hearst retention marketing
- Identified creative, offer, segmentation elements with most leverage
with some rolled out immediately, others fine-tuned via more tests
- Low-leverage test elements removed from future test consideration
- Additional tests to “fine-tune” high-leverage elements led to roll outs
 New control series that outperformed existing control series of the
last 7 years (response lifts as high as 15%)
 New model-based offer and segmentation strategy based on a
subscriber’s likelihood to renew or cancel
22

23. Risks & Recommendations
• Risks of multivariate testing for Retention efforts
- Testing more elements increases risk of negative impact
- Normally low risk test elements could be high risk when mixed with
other test elements to form counter-intuitive recipes needed for test
- Control creative may not accommodate variety of recipes required
for testing, which may force additional creative changes and
response risk
• Recommendations on when to use multivariate testing
- Statistically significant A/B test volumes can’t accommodate
all test elements
- There’s a lot to test and results are needed quickly
- The organization supports bold, high risk/reward testing
23

24. Accelerated Testing
for Retention and CRM Programs
Brian Fichtel
Senior Director, Customer Retention
Hearst Magazines
Gordon Bell
President, LucidView
Managing Partner, Artestry
gbell@lucidview.com
24