The document discusses best practices for designing multi-channel marketing campaign tests to optimize budgets. It recommends establishing hypotheses, developing test strategies segmented by customer characteristics, and creating a measurement plan with control groups and milestones to evaluate impact. Specific tests are proposed to determine the best frequency, number of channels, and frequency levels to increase customer engagement like conversions or opt-ins. Micro tests are also suggested to refine email subject lines, calls to action and improve key metrics like opens and downloads. The overall goal is to learn how to maximize results through rigorous experimentation and analysis of multi-channel campaign components.

1. How Do You Know…
Designing Multi-Channel
Campaign Tests That
Optimize Your Budget
Jim DeLash
November 18, 2014

2. One version had a Conversion Rate 24.6% higher
than the other one
Page layout

3. Moving registration to the right side form
increased the site’s Conversion Rate by 24.6%
Page layout

5. How do you know?

6. The right frequency

7. The best subject line

8. The optimal channel mix

10. Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

11. Establishing hypotheses
Statements to be proven
All tests must support a hypothesis
Extrapolating the results to other brands

12. Low unaided recall scores for
advertising creative will require
additional frequency….
Establishing hypotheses

13. Low unaided recall scores for
advertising creative will require
additional frequency….
Establishing hypotheses

14. Low unaided recall scores for
advertising creative will require
additional frequency….
BUT HOW MUCH MORE?
Establishing hypotheses

16. TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses

17. TEST…
More direct mail or e-mail
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses

18. TEST…
More direct mail or e-mail
More direct mail and e-mail
Different levels of frequency
Establishing hypotheses

19. TEST…
More direct mail or e-mail
More direct mail and e-mail
10 15 20
Establishing hypotheses

20. TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses

24. The best design is not always the best approach
Do not let design conflate the objective of an email or message
Every unnecessary piece of content is waste and reduces your
chances of getting a click

25. Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

26. Developing Test Strategies by
Segment
Specialist or Generalist
Current User or Prospect
Decile

27. Awareness & Trial Offer
Developing Test Strategies by
Segment
Multiple
Brands Cadence

31. What will we learn from this test….
1. Best frequency by Specialty and Quintile
2. One channel vs. Two channels (Segments 5,7,9)
3. Total frequency: 10 vs. 12 vs. 14

33. Version B, the one with the ‘Save Now’ button, increased opt-ins
for logged-in users by 333% at a 99.9% confidence level.

34. Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

35. Control Groups
Measurement Milestones
Test Analysis
Creating a Measurement Plan

36. There are only two ways to get
fired from Harrah’s:
stealing from the company,
or
failing to include a proper
control group in your business
experiment

37. Control Groups…
Not a ‘holdout’ group
Must include all relevant variables
Large enough to be statistically valid
Creating a Measurement Plan

38. Measurement Milestones…
Track progress against goal
Forecast test impact early
Connect KPIs with Rx impact
Creating a Measurement Plan

39. Test Analysis…
Evaluate overall campaign impact
Both TRx and NRx
Creating a Measurement Plan

40. How can an e-mail test be
successful with a low open rate?

43. MICRO TESTS

44. MICRO TESTS

46. Version B, the subject line broken into three consumable
sections, increased opens by 73.7%.

48. Version B, the ‘Get My Free Paper’ Call to Action,
increased downloads by 25% at a 95% confidence level.

50. Version B, the ambiguous subject line, increased
opens by 112% at a 95% confidence rate.

51. Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan

52. What is the optimal number of
samples to offer in a multi-channel
campaign?

53. So, how will you know?