1. Getting Testing Right:
A Practical Guide to Testing in
Direct Marketing
IoF DM and Fundraising - March 2013
2. About me
Richard Hughes
Marketing Data Team Manager
Previously:
Data Planner at Bluefrog
Data Analyst at Good Agency / Cascaid
Database Administrator at Crusaid (AIDS/HIV charity)
3. Objectives
• Talk through some of the finer points of testing
• the strategy side and the data techie side
• Both are really important!
• A brain dump of everything I’ve learnt about testing
• Advice on best practice
• Show that it can be exciting
• Inspire you to think about testing you can do
4. Why talk about testing?
– Not enough people talk about how to do it
– There is little info on the web for DM
– I’ve seen it go wrong
– Used well, testing can be very powerful, requires
some thought and planning!
5. Definitions
Split testing, or A / B testing, is when an
audience is split into two or more groups and
given different treatments in order to determine
the most effective treatment
7. Why Test Anyway?
How many
communications
How much should should we send
we ask our throughout the
supporters to year?
donate?
Which creative
should we
choose?
Which Email
Subject gets the
best open rate?
8. Why Test Anyway?
What stationery
types perform the
What’s the best? More money
best time to on more expensive
send a pack? packs?
Who is the
best
signatory?
How do cash
appeals affect
regular giving
attrition rates?
11. Gut Reaction Versus Evidence
• Sometimes as experienced marketeers we
know intuitively know the answers to some of
these questions
• But we want to move to situation with where
we make evidence based decisions
13. Concepts
We are trying to find out if one approach is
more likely to get better results than another
•Testing is affected by probability
–This means there is no guarantee that an approach
will always “win”
–We can say that it is more likely to win and we can
say how confident we are
14. Sampling Distribution
• When we test we …
– measure a sample of our audience and use that to
generalise about the rest of the database
15. The results can be put into a bell curve
If we
sample data
from our
database
many times
and treat in
certain way
we get a
normal
distribution
17. Stats Summary
• The response rate for each test is a normally
distributed
• We want to measure the difference in
performance between a given treatment and
the control.
• The difference itself is a normally distributed
random variable.
19. Annual Testing Strategy
• Good testing starts with careful thinking
• Document what you want to find out
• Check and reflect on your questions
• Ensure that tests will deliver actionable results
20. Annual Testing Strategy
•Build scenarios to understand where you
are going to get the best value
•Prioritise – focus on the best outcomes
For UNICEF, this means focusing Select tests that will have most
on the outcome that brings the impact, e.g. in mail packs, focus
best result for children on outers rather than copy buried
inside.
21. Cautionary Tales 1
• Testing can be expensive
– Paying for different creative
– Paying for different stationery to be printed
– Ring fencing certain supporters from different
comms is all expensive
• This is an important consideration when
thinking about the value of the test
22. Designing Tests: Sample Sizes
• Think about volume for your test
– You need sufficient quantity in your test
• The sample needs have enough volume to
be able to generalise about the population
23. Calculating Sample Sizes
Deep Dark Statistics:
• www.lucidview.com/sample_size.htm
• The most useful online resource that has
quite a technical explanation
24. Calculating Sample Sizes
• Two things determine sample size
– Existing Response Rate
• Low number of responders means we need a bigger
sample
– Uplift of test
• Small uplift means we need a bigger sample to see if
there is a difference
25. Sample Sizes – Worked Example
Take from http://www.testsignificance.com/
26. Testing more than one thing at once
• Need to be careful, but can split by more
than one test
Treatment A Treatment B Totals X & Y
Treatment X Segment 1 Segment 2
Treatment Y Segment 3 Segment 4
Totals for A & B
27. Cautionary Tales 2
• Be careful about testing too
many things in one campaign
– They can be difficult to manage
– Cause confusion evaluating
28. Selected the Data
• Once you you’ve decided on the volumes
the next task is to make sure you split the
data fairly
– This means selecting two or more samples,
ordering by factors that are important and
selecting alternate rows
– Do not take top / bottom half of spreadsheet
29. Coding
• This might be a no brainer but ensuring
the coding of A and B is set up right is
important
30. Evaluating
• We need to determin if two different results
are significant
• This means showing that we are 95%
confident there is a significant difference
• Quite a few websites that can help
31. Evaluating
• If we are testing prompt amounts in packs
we also need to test to see if the average
gift is significantly different
• We can use a T-test for this
32. Cautionary Tales 3
• Testing sometimes don’t tell us anything
interesting
• This is a lesson in setting expectations
• Don’t say “we’re going to find out
which is better”
• Instead say “We’re going to find out if
there is any difference”
33. Don’t forget to focus on Net Income
Mailed Cost Response RR Income Net Average RoI
10000 £7,500 800 8% £14,400 £6,900 £18 1.92
10000 £11,000 1100 11% £19,800 £8,800 £18 1.80
35. Final Thoughts
• Testing is about making incremental
improvements
• If you need more dramatic change then think
about your overall fundraising strategy
• Make sure you do lots of planning
36. Summary
• What are your marketing questions?
Testing Strategy • What are your priorities?
• Calculate testing volume
Test Design • Split data fairly, Code data appropriately
Execute • Mail, email, phone
Evaluate • Evaluate significance of results
Build Insight • Update documentation on your audience insights
