DonorVoice presents:Innovation Webinar SeriesA Proven Way to Get Better Returnsfrom your Direct Mail Testing Program
The Problem Realities of The New World • Cost per donor to acquire goes up • Massive competition Current Testing Process • Biased (too much reliance on conventional wisdom) • Slow & Costly (A/B testing to infinity) • Inefficient (Throwing out baby with bath water)
The Solution A New Way to Test • Feedback from target audience • Deceptively simple – let’s ask ‘em (how ask matters, a lot).
Battle Tested Should bottle be red, white, blue, green, Product development yellow? What should brand name be? • Member packages What about size/shape of bottle? How much does it cost? • Consumer package goods What marketing messages do we put • Direct mail On bottle?
Why Test? Beat the Control. Period. End of Story.
How Much Cost to BeatControl? Outer Envelope Type 1 Type 2 Type 3 This is 27 tests OE Color goldenrod brownkraft gray with all Annual Look combinations OE Teaser Free Gift! Appeal Inside! (3x3x3). #10 with OE Type 6x9 #10 Plain window 27 Tests Old Method $88,500 (production costs, opportunity cost) New Method $24,000 (includes tool, 3 live tests) 269% difference
But We Don’t Do 27 Tests If you do 7 or more tests a year, this is financially better deal on first project + Odds of finding control beating product sooner are much higher For NO TEST organizations – Think your control can’t be beat? How much money is left on the table with a sub-optimum product?
Case Study #1 • Client needed to identify new family as “face” Possible Test Appeals of organization. • Wet test results performed exactly as predicted producing a test/family that did beat the control. • If conventional wisdom had been relied upon the best options would not have been mailed. Test A, #1 in DV Tool Control, #2 in DV Tool Test B, #3 in DV Control family Tool
Case Study #1 Financial Benefit: • Production & postage savings from NOT mailing all test packages $46,000 • Anticipated annual revenue increase with rollout of better performing control $72,000 $118,000
Case Study #2 Color of Outer Envelope Personalization Labels Calendar Grid Backend Premium
Case Study #2 Financial Benefit: • Savings from not producing & mailing predicted losing test packages $45,000 • Increase in net revenue by doubling panel volume on predicted winners $20,400 $65,400
Okay, But… “Our best donors are not online…” “People don’t do what they say...”
“Process” for Deciding Whatto Test Purple it is! What do we test?
A/B Testing to Infinity(and Beyond!) 6 variations, 216 possible combinations. 10 tests a year, 22 years Can’t analyze across tests + • Test a did poorly • Test b did okay • Test c did well Unanswered Question: If we combine b and c is that better, great, the same, or worse?
Baby in Bath Water Test package that changes everything. It loses. Now what?
Step Two Identify “Ingredients” to be Tested Reply Form Ingredient Size Ingredient Johnson Box Ingredient Message Ingredient Teaser Ingredient
Step Three Finalize Grid/Artwork ATTRIBUTE Level 1 Level 2 Level 3 Level 4 L2 (Annual Appeal Labels L1 (LYH labels) - Nov version) none n/a Outside OE2 (double dop OE3 (love heart envelope OE1 (control) cut) teaser) OE4 (4 color image) BS2( double drop Buckslip black and white BS3( Double drop cut insert BS1 (control version) image) with 4 color image) n/a C1 (certificate of C2 (certificate of appreciation - Annual appreciation - LYH Certificate appeal) version) none n/a RS2 (double drop RS3 (double drop cut Reply Slip & RS1 (Control reply cut no image reply with 4 color image RS4 (love heart reply Letter and letter) and letter) reply and letter) and letter) LN2 (lift note black Lift Note none and white) LN3 (yellow and blue) n/a
Step Four Design Experiment Our tool determines what combinations to show each respondent.
Step Five Conduct the Survey Renewal • Client provides email sample Acquisition • Develop filter questions • Match any profile
Step Six Provide Results Informational Insert in Control “Baby” significantly outperforms control Buckslip
Step Six Provide Results What to Mail (real examples) • Run live test with lift note in addition to and in place of certificate • Live test (2012) alternative copy against “Year End” • Don’t bother with test colors on OE • Don’t bother with new design on calendar • Flower labels better than anchor for men and women • Tote better than duffle as back-end premium with renewal Who To Mail • Formal education is big discriminator for package preference • Append to file for renewal selection • Look at applying in acquisition (skews lower HH income, in South)
Contact Information Kevin Schulman DonorVoice, CEO (202) email@example.com Twitter: @kschulmanDV Pretest Tool: Get to rollout faster.