Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multivariable testing in Direct Mail

1,938 views

Published on

A presentation on multivariable testing in Direct Marketing and Direct Mail. Marketing Day 2010.

Published in: Business
  • Be the first to like this

Multivariable testing in Direct Mail

  1. 1. Multivariable Testing in Direct Mail Efficient tests with limited resources Gordon H. Bell President LucidView
  2. 2. Analytics & Testing: 2 sources of insight Testing – Front-end planning – Prove cause-and-effect – Ask new questions – Design the test to give the specific data you want Analytics – Back-end analysis – Look for correlations – Answer questions, but cannot ask any – General data
  3. 3. Testing has always been important • “Almost any question can be answered, cheaply, quickly and finally, by a test campaign.” – Claude Hopkins, Scientific Advertising, 1923 • “Testing is still the best way to find true breakthroughs.” – Stone & Jacobs, Successful Direct Marketing Methods, 2008
  4. 4. is now more efficient Testing has always been important Many questions • “Almost any question can be answered, cheaply, quickly and finally, by a test campaign.” ^ multivariable
  5. 5. What is Multivariable Testing? • Many different test designs and statistical techniques developed to test variables more efficiently • Change many variables at once – in an organized way – so you can separate out the impact of each – Number of test “recipes” (test versions) – Each with a unique combination of all elements – Analyzing all recipes together to quantify effects • Test design = collection of related test recipes
  6. 6. 4 Key Benefits Benefit Scientific Multivariable Testing Split-run Testing Efficiency 2 - 35 variables in one test 1 variable = 1 test Speed and Constant sample size Increasing sample size Sample size (no matter how many variables) Depth of Accurate, robust, and comparative One main effect insights main effects and interactions (conditional, with no interactions) Flexibility Wide range of test designs One choice
  7. 7. Getting Started Important steps include how to: 1. Define the scope 2. Select a good design 3. Determine the right sample size 4. Create multivariable test recipes 5. Analyze and interpret results
  8. 8. #1 Define the Scope Where is multivariable testing valuable and manageable? • High potential return-on-investment • Clear metrics • Fast results • Flexibility to create many versions of the mailing • Insights can be applied long-term
  9. 9. Financial Times: Price Test (from the DMA07 Conference) • Focus on price testing • Measure response rate and revenue • Test in one direct mail campaign • All variables lasered in black ink (easy and inexpensive to change) Test Elements (-) Control (+) New Idea A Price $49 for 26 weeks $99 for 12 months B Longer term (+25%) No Yes (32 weeks / 15 months) C Shipping & handling fee No Yes, $4.95 D Streamline price box No Yes, redesign price box and (3 lines vs 5) reply slip
  10. 10. FT “control” mail package
  11. 11. #2 Select a Good Test Design Many choices of multivariable test designs • Which “handful” of combinations should be tested to clearly see results? • Want to have – Few test recipes (versions of the mailing), but… – Lots of data for better insights – Clear, actionable results – A test design that the marketing team can understand and believe
  12. 12. Financial Times: Streamline price box Streamline price box Longer term (+25%) Longer term (+25%) S&H (delivery) fee S&H (delivery) fee test design and recipes Price Price Recipe A B C D Mailed Recipe A B C D AB AC BC Mailed 1 – – – – 25,000 1 – – – – + + + 25,000 2 + – – + 25,000 2 + – – + – – + 25,000 3 – + – + 25,000 3 – + – + – + – 25,000 4 + + – – 25,000 4 + + – – + – – 25,000 5 – – + + 25,000 5 – – + + + – – 25,000 6 + – + – 25,000 6 + – + – – + – 25,000 7 – + + – 25,000 7 – + + – – – + 25,000 8 + + + + 25,000 8 + + + + + + + 25,000
  13. 13. Financial Times: Streamline price box Longer term (+25%) S&H (delivery) fee Price test design and recipes B: Longer term D: Streamline A: Price C: S&H fee Recipe A B C D Mailed (+25%) price box Recipe 1 No 1 – – – – 25,000 (control) $49 ($49 for 26 weeks) No No No Yes, redesign 2 + – – + 25,000 Recipe 2 $99 ($99 for 12 months) No price box Yes Yes, redesign 3 – + – + 25,000 Recipe 3 $49 ($49 for 32 weeks) No price box 4 + + – – 25,000 Recipe 4 $99 Yes ($99 for 15 months) No No 5 – – + + 25,000 Recipe 5 $49 No ($49 for 26 weeks) Yes, $4.95 Yes, redesign price box 6 + – + – 25,000 Recipe 6 $99 No ($99 for 12 months) Yes, $4.95 No 7 – + + – 25,000 Recipe 7 $49 Yes ($49 for 32 weeks) Yes, $4.95 No Yes Yes, redesign 8 + + + + 25,000 Recipe 8 $99 ($99 for 15 months) Yes, $4.95 price box
  14. 14. #3 Find the Right Sample Size The best sample size depends upon – Response rate (or variation in sales) – How small a change you want to see Lower response rate, or the need to see a smaller change, requires a larger sample size.
  15. 15. Sample Size (N) example • For Financial Times… – Say we usually have a 1.5% response rate (R) – And we want to see if any test elements affect response by 10% or more (0.15%) – Then… 31.38 ● R ● (1-R) N= (smallest change) 2 31.38 ● 1.50% ● 98.5% N= 2 = 206,062 (0.15%)
  16. 16. #4 Create the multivariable recipes • The science determines which “handful” of combinations should be tested • The art of testing is making sure… – Elements remain bold (in the difference between “–” and “+” levels) – Execution is clear and consistent
  17. 17. Creatives: control (recipe 1) and “all plus” recipe 8 D+ A+ C+ B+
  18. 18. #5 Analyze and Interpret Results • A well-designed test is relatively easy to analyze – Analysis of variance, regression, or… pencil and calculator (with some more in-depth analysis) • Three main steps in the analysis are: 1. Analyze main effects (each column in the design) 2. Analyze interactions 3. Consider alternative solutions and additional metrics
  19. 19. FT price test: main effects & interactions FT Price Test: Gross response rate optimal = 32.7% lift B: Longer term (yes, +25%) 18.9% increase in response versus the control + … unless both are set C: Shipping & at the best combination 17.2% drop - handling fee (no) BC interaction 13.8% further increase in response if... - BC Interaction D: Streamline - price box A: Price - AC interaction + Significant effects (beyond line) AB interaction + B-: Current B+: Longer term (+25%) 0.000% 0.010% 0.020% 0.030% 0.040% 0.050% 0.060% / 12 mo) (26 wk 0.070% Term and S&H do not Subscription Term affect response rate… C-: No S&H fee C+: Add $4.95 S&H fee
  20. 20. Financial Times: summary Multivariable testing offered a way to • Test easy, high-impact changes • Learn more with a small sample size • Increase insights – More accurate main effects – Clear interactions • In contrast, with split-run testing – We would have needed 3x the sample for equal confidence – At this sample size, not one effect would be significant – Interactions would be impossible to see
  21. 21. Hearst Launch of the Food Network Magazine (presented at the DMA09 conference) New magazine for a new market = Need for speed • Learn fast  ramp-up quickly • Test a lot of ideas to see what works • Increase confidence in test results for a rapid roll-out • Optimize price/offer along with creatives
  22. 22. Food Network Magazine Creative Test (13 Creative Elements) Changes to the outer envelope A Window on OE H Reply By date B Change logo on OE and form J Add chef photos on form C Return address on OE K Change description D Form layout L Brochure E Change "Charter Discount" M Chit F Audience Development copy N Lift note G Savings presentation Additional inserts
  23. 23. Elements on the Envelope B: Logo C: Return Address A: Window
  24. 24. B: Change logo E: Change “Charter Discount” F: Audience Development copy G: Savings presentation H: Reply-By date J: Add chef photos K: Change description D: form layout
  25. 25. Change logo on OE and form Audience Development copy Change "Charter Discount" 16-recipe Add chef photos on form Return address on OE Savings presentation Change description Creative Test Window on OE Reply By date Design Form layout Brochure Lift note Chit Net Recipe A B C D E F G H J K L M N Mailed response 1 – – – – – – – – – – – – – 25,000 0% 2 + + – – – – + + – – + + + 25,000 -21% 3 – – – + – + + + + – – + – 25,000 -8% 4 + + + – + – – – + – – + – 25,000 -6% 5 + + + + + + + + – – – – – 25,000 -11% 6 – + – + + – + – – + – + + 25,000 -25% 7 – – + – + – + + + – + – + 25,000 -20% 8 + – + – – + – + – + – + + 25,000 -10% 9 + – + + – – + – + + – – + 25,000 -11% 10 + – – + + – – + – + + – – 25,000 -42% 11 – + – – + + – + + + – – + 25,000 -20% 12 + + – + – + – – + – + – + 25,000 -10% 13 – + + + – – – + + + + + – 25,000 -31% 14 – – + + + + – – – – + + + 25,000 -26% 15 + – – – + + + – + + + + – 25,000 -37% 16 – + + – – + + – – + + – – 25,000 -38% 400,000
  26. 26. Test Elements (on the 1-page form) Control (recipe #1) Recipe #11 B+ E+ F+ J+ K+ H+
  27. 27. FNM: Net Response Rate Net Response Rate optimal = 9.75% lift Test element (optimal setting) L-: Brochure (yes) Response drops 16.8% when brochure is removed K-: Change description (control) New description reduces response 14.3% E-: Change Charter Discount (no) "Research Discount" hurts 7.3% J+: Add chef photos on form (yes) 3.8% lift N+: Lift note (yes) 3.5% lift G-: Savings presentation (control) -3.1% A+: No window on OE (closed-faced) +2.5% D-: Form layout H-: Reply By date M-: Chit C+: Return address on OE Significant effects B-: Change logo on OE and form beyond line F-: Audience Development copy 0.00% 0.25% 0.50% 0.75% 1.00% 1.25%
  28. 28. Food Network Magazine Results Multivariable "refining" tests (July 09 test results) 8.8% increase in NRR 17.3% increase in Profit Multivariable "screening" tests (Mar09 tests - Jul09 confirmation) 8.4% increase in NRR 35% jump in profit 17.6% increase in Profit over 2 campaigns Original control (versus 2 years with A/B splits) (March 2009) (sample size of 600,000 vs. 4.7 MM for split-run testing)
  29. 29. Whirlpool Contact Strategy Test (presented at DM Days – New York, 2007) Focus = Annual renewal of the extended service contract Test Elements A Timing B Early mail drop C Expiration mail drop D Late mail drop E Outer envelope F Lift note G Phone Call in Week 1
  30. 30. Contact Strategy Test G+ A+ B+ E+ X C+ E+ D+ Control B+ E+ X C+ E+ D+ Week: 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 Expiration Date
  31. 31. Expiration mail drop Whirlpool Outer envelope Early mail drop Late mail drop Test Design Lift note Timing Phone Recipe A B C D E F G Response Week 1 Week 2 Week 3 Week 4 1 - - - - - - - customers customers customers customer 2 + - - - + - + 3 - + - - + + - 4 + + - - - + + 5 - - + - + + + 6 + - + - - + - 7 - + + - - - + 8 + + + - + - - 9 - - - + - + + 10 + - - + + + - 11 - + - + + - + 12 + + - + - - - 13 - - + + + - - 14 + - + + - - + 15 - + + + - + - 16 + + + + + + +
  32. 32. Test Results Effects: Total Response Rate A: Timing (spread out) -4.67% D: Late mail drop (yes) +3.00% G: Phone call (yes) -2.68% AB (add early drop) -2.43% BD +2.22% ABD +1.92% B: Early mail drop (yes) +1.61% F: Lift note +1.41% AG -0.83% AD +0.79% C: Expiration mail drop +0.63% E: Outer envelope +0.61% Significant effects AF +0.27% (beyond line) AC -0.11% AE +0.02% 0.0% 0.5% 1.0% 1.5% 2.0% 2.5% (effects in percentage points on bottom; as % of control on end)
  33. 33. Interaction Effect AB Interaction (total response rate) 49% But it does help if contacts are kept 48% spread out 47% On average, the early mail drop 46% (B) has no impact 45% A-: control (9,4,1,-4) A+: condensed (6,3,1,-3) A: Timing of contacts B-: No early mail drop B+: Yes (+14 or +9)
  34. 34. Whirlpool Test: summary • Response rate increased 7.81% (7.3% predicted) • 8.3% increase in revenue (worth $ millions) • 6.76% increase in annual profit • With split-run tests, would have… – Needed 2½ years of testing (vs. 4 months) – Completely missed the interaction – Seen nothing significant in 4 months (due to 3x greater error)
  35. 35. “Testing is still the best way to find true breakthroughs.” – Bob Stone and Ron Jacobs, Successful Direct Marketing Methods, 2008 “ Like trading in your bicycle for a helicopter, multivariable testing gives you the speed, power, and flexibility to get farther, faster.” – Gordon Bell, Marketing Day, 23 March 2010

×