Getting testing right


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Getting testing right

  1. 1. Getting Testing Right:A Practical Guide to Testing inDirect MarketingIoF DM and Fundraising - March 2013
  2. 2. About meRichard HughesMarketing Data Team ManagerPreviously:Data Planner at BluefrogData Analyst at Good Agency / CascaidDatabase Administrator at Crusaid (AIDS/HIV charity)
  3. 3. Objectives • Talk through some of the finer points of testing • the strategy side and the data techie side • Both are really important! • A brain dump of everything I’ve learnt about testing • Advice on best practice • Show that it can be exciting • Inspire you to think about testing you can do
  4. 4. Why talk about testing? – Not enough people talk about how to do it – There is little info on the web for DM – I’ve seen it go wrong – Used well, testing can be very powerful, requires some thought and planning!
  5. 5. Definitions Split testing, or A / B testing, is when an audience is split into two or more groups andgiven different treatments in order to determine the most effective treatment
  6. 6. Why Test Anyway?
  7. 7. Why Test Anyway? How many communicationsHow much should should we send we ask our throughout the supporters to year? donate? Which creative should we choose?Which EmailSubject gets thebest open rate?
  8. 8. Why Test Anyway? What stationery types perform the What’s the best? More money best time to on more expensive send a pack? packs? Who is the best signatory?How do cashappeals affectregular givingattrition rates?
  9. 9. Marketing Triangle Audience Elements that affect results Timing Message
  10. 10. The Data Pyramid
  11. 11. Gut Reaction Versus Evidence• Sometimes as experienced marketeers we know intuitively know the answers to some of these questions• But we want to move to situation with where we make evidence based decisions
  12. 12. Concepts
  13. 13. ConceptsWe are trying to find out if one approach ismore likely to get better results than another•Testing is affected by probability –This means there is no guarantee that an approach will always “win” –We can say that it is more likely to win and we can say how confident we are
  14. 14. Sampling Distribution• When we test we … – measure a sample of our audience and use that to generalise about the rest of the database
  15. 15. The results can be put into a bell curveIf wesample datafrom ourdatabasemany timesand treat incertain waywe get anormaldistribution
  16. 16. Two CurvesMathematicalproperties ofthe curvemeans wecan use statsto determinehow likelytest a and Bare different
  17. 17. Stats Summary• The response rate for each test is a normally distributed• We want to measure the difference in performance between a given treatment and the control.• The difference itself is a normally distributed random variable.
  18. 18. Structured Approach: Testing Life Cycle Testing Strategy Design Test Insight (Tactics) Evaluate Execute
  19. 19. Annual Testing Strategy• Good testing starts with careful thinking• Document what you want to find out • Check and reflect on your questions • Ensure that tests will deliver actionable results
  20. 20. Annual Testing Strategy •Build scenarios to understand where you are going to get the best value •Prioritise – focus on the best outcomesFor UNICEF, this means focusing Select tests that will have moston the outcome that brings the impact, e.g. in mail packs, focusbest result for children on outers rather than copy buried inside.
  21. 21. Cautionary Tales 1• Testing can be expensive – Paying for different creative – Paying for different stationery to be printed – Ring fencing certain supporters from different comms is all expensive• This is an important consideration when thinking about the value of the test
  22. 22. Designing Tests: Sample Sizes• Think about volume for your test – You need sufficient quantity in your test• The sample needs have enough volume to be able to generalise about the population
  23. 23. Calculating Sample SizesDeep Dark Statistics:•• The most useful online resource that has quite a technical explanation
  24. 24. Calculating Sample Sizes• Two things determine sample size – Existing Response Rate • Low number of responders means we need a bigger sample – Uplift of test • Small uplift means we need a bigger sample to see if there is a difference
  25. 25. Sample Sizes – Worked ExampleTake from
  26. 26. Testing more than one thing at once• Need to be careful, but can split by more than one test Treatment A Treatment B Totals X & YTreatment X Segment 1 Segment 2Treatment Y Segment 3 Segment 4Totals for A & B
  27. 27. Cautionary Tales 2• Be careful about testing too many things in one campaign – They can be difficult to manage – Cause confusion evaluating
  28. 28. Selected the Data• Once you you’ve decided on the volumes the next task is to make sure you split the data fairly – This means selecting two or more samples, ordering by factors that are important and selecting alternate rows – Do not take top / bottom half of spreadsheet
  29. 29. Coding• This might be a no brainer but ensuring the coding of A and B is set up right is important
  30. 30. Evaluating• We need to determin if two different results are significant• This means showing that we are 95% confident there is a significant difference• Quite a few websites that can help
  31. 31. Evaluating• If we are testing prompt amounts in packs we also need to test to see if the average gift is significantly different• We can use a T-test for this
  32. 32. Cautionary Tales 3• Testing sometimes don’t tell us anything interesting• This is a lesson in setting expectations • Don’t say “we’re going to find out which is better” • Instead say “We’re going to find out if there is any difference”
  33. 33. Don’t forget to focus on Net Income Mailed Cost Response RR Income Net Average RoI 10000 £7,500 800 8% £14,400 £6,900 £18 1.92 10000 £11,000 1100 11% £19,800 £8,800 £18 1.80
  34. 34. Building Insight• Understanding what your tests means for your programme• Updating your strategy
  35. 35. Final Thoughts• Testing is about making incremental improvements• If you need more dramatic change then think about your overall fundraising strategy• Make sure you do lots of planning
  36. 36. Summary • What are your marketing questions?Testing Strategy • What are your priorities? • Calculate testing volume Test Design • Split data fairly, Code data appropriately Execute • Mail, email, phone Evaluate • Evaluate significance of results Build Insight • Update documentation on your audience insights
  37. 37. Any questions?
  38. 38. Thank you• Richard Hughes•