Getting Testing Right:A Practical Guide to Testing inDirect MarketingIoF DM and Fundraising - March 2013
About meRichard HughesMarketing Data Team ManagerPreviously:Data Planner at BluefrogData Analyst at Good Agency / CascaidDatabase Administrator at Crusaid (AIDS/HIV charity)
Objectives • Talk through some of the finer points of testing • the strategy side and the data techie side • Both are really important! • A brain dump of everything I’ve learnt about testing • Advice on best practice • Show that it can be exciting • Inspire you to think about testing you can do
Why talk about testing? – Not enough people talk about how to do it – There is little info on the web for DM – I’ve seen it go wrong – Used well, testing can be very powerful, requires some thought and planning!
Definitions Split testing, or A / B testing, is when an audience is split into two or more groups andgiven different treatments in order to determine the most effective treatment
Why Test Anyway? How many communicationsHow much should should we send we ask our throughout the supporters to year? donate? Which creative should we choose?Which EmailSubject gets thebest open rate?
Why Test Anyway? What stationery types perform the What’s the best? More money best time to on more expensive send a pack? packs? Who is the best signatory?How do cashappeals affectregular givingattrition rates?
Marketing Triangle Audience Elements that affect results Timing Message
Gut Reaction Versus Evidence• Sometimes as experienced marketeers we know intuitively know the answers to some of these questions• But we want to move to situation with where we make evidence based decisions
ConceptsWe are trying to find out if one approach ismore likely to get better results than another•Testing is affected by probability –This means there is no guarantee that an approach will always “win” –We can say that it is more likely to win and we can say how confident we are
Sampling Distribution• When we test we … – measure a sample of our audience and use that to generalise about the rest of the database
The results can be put into a bell curveIf wesample datafrom ourdatabasemany timesand treat incertain waywe get anormaldistribution
Two CurvesMathematicalproperties ofthe curvemeans wecan use statsto determinehow likelytest a and Bare different
Stats Summary• The response rate for each test is a normally distributed• We want to measure the difference in performance between a given treatment and the control.• The difference itself is a normally distributed random variable.
Structured Approach: Testing Life Cycle Testing Strategy Design Test Insight (Tactics) Evaluate Execute
Annual Testing Strategy• Good testing starts with careful thinking• Document what you want to find out • Check and reflect on your questions • Ensure that tests will deliver actionable results
Annual Testing Strategy •Build scenarios to understand where you are going to get the best value •Prioritise – focus on the best outcomesFor UNICEF, this means focusing Select tests that will have moston the outcome that brings the impact, e.g. in mail packs, focusbest result for children on outers rather than copy buried inside.
Cautionary Tales 1• Testing can be expensive – Paying for different creative – Paying for different stationery to be printed – Ring fencing certain supporters from different comms is all expensive• This is an important consideration when thinking about the value of the test
Designing Tests: Sample Sizes• Think about volume for your test – You need sufficient quantity in your test• The sample needs have enough volume to be able to generalise about the population
Calculating Sample SizesDeep Dark Statistics:• www.lucidview.com/sample_size.htm• The most useful online resource that has quite a technical explanation
Calculating Sample Sizes• Two things determine sample size – Existing Response Rate • Low number of responders means we need a bigger sample – Uplift of test • Small uplift means we need a bigger sample to see if there is a difference
Sample Sizes – Worked ExampleTake from http://www.testsignificance.com/
Testing more than one thing at once• Need to be careful, but can split by more than one test Treatment A Treatment B Totals X & YTreatment X Segment 1 Segment 2Treatment Y Segment 3 Segment 4Totals for A & B
Cautionary Tales 2• Be careful about testing too many things in one campaign – They can be difficult to manage – Cause confusion evaluating
Selected the Data• Once you you’ve decided on the volumes the next task is to make sure you split the data fairly – This means selecting two or more samples, ordering by factors that are important and selecting alternate rows – Do not take top / bottom half of spreadsheet
Coding• This might be a no brainer but ensuring the coding of A and B is set up right is important
Evaluating• We need to determin if two different results are significant• This means showing that we are 95% confident there is a significant difference• Quite a few websites that can help
Evaluating• If we are testing prompt amounts in packs we also need to test to see if the average gift is significantly different• We can use a T-test for this
Cautionary Tales 3• Testing sometimes don’t tell us anything interesting• This is a lesson in setting expectations • Don’t say “we’re going to find out which is better” • Instead say “We’re going to find out if there is any difference”
Don’t forget to focus on Net Income Mailed Cost Response RR Income Net Average RoI 10000 £7,500 800 8% £14,400 £6,900 £18 1.92 10000 £11,000 1100 11% £19,800 £8,800 £18 1.80
Building Insight• Understanding what your tests means for your programme• Updating your strategy
Final Thoughts• Testing is about making incremental improvements• If you need more dramatic change then think about your overall fundraising strategy• Make sure you do lots of planning
Summary • What are your marketing questions?Testing Strategy • What are your priorities? • Calculate testing volume Test Design • Split data fairly, Code data appropriately Execute • Mail, email, phone Evaluate • Evaluate significance of results Build Insight • Update documentation on your audience insights