Social Marketing Conference:Changing Behaviour Through Communications 30 November 2011 www.charitycomms.org.uk www.twitter.com/CharityComms www.facebook.com/CharityComms
Evaluating social marketing Patrick Ladbury www.thensmc.com
Evaluation - Simples!• What behaviour do you want people to do?• Which people do you want to do that behaviour?• By when?• How can we measure it?
An evaluation planQuestions to ask • What are we measuring? – Changes in knowledge, attitudes (intention, priority) and behaviour • What are we measuring against? – What are our baselines or controls? • Who is doing the measuring? – Do we need any independent input? • When do we do the measuring? • Can we turn the outcome into an economic ROI?
An evaluation planDifferent types of evaluationTYPE DefinitionFORMATIVE Done throughout the development of a project to pre-test (or ‘sound out’) aspects with the audience or stakeholders. Keeps you on track in terms of what will work with the target segment.PROCESS A review of how the project was managed – usually involving key internal and external stakeholders.ECONOMIC Evaluating cost effectiveness , return on investment and value for moneyIMPACT (or Measures the changes that have happened directly as a result of our intervention.Output) These are likely to be changes in knowledge, awareness and, most importantly of all, behaviourOUTCOME Seeks to link the short term actions (impacts) with the longer term benefits (outcomes) The changes in behaviour will have been encouraged in order to meet a wider, or longer term benefit.
An evaluation planTemplateWhat are we measuring? How are we going to What needs to Who by? When measure it be done? Resource needed by? (method)?IMPACT - KnowledgeIMPACT - AttitudeIMPACT – BehaviourOUTCOME –
A step by step approachPlanning a behaviour change intervention usingSocial marketing is a step by step processUsing the process helps to ensure more efficient andeffective interventions and better use of resources.
Getting started: exampleA number of young people under 16 were regularlydrinking alcohol on the streets of North Tyneside atthe weekend.The number of ASBOs handed out by police wasincreasing as were the number of complaintsreceived by residents who were feeling less safe onthe streets..
Scoping: exampleResearch led to other areas that impacted on theissue and that needed evaluating.• Alcohol sales to young people• Attendance and activities at youth clubs• Young people’s drinking levels
Smoking cessation services Smokers Smokers Smokers Smokers All who quit who want who try who smokers using to quit to quit succeed support 70%* 45% 23% 2.7% 100% Driving Triggering Making quitting motivation action more effective*NB figures relate to all smokers.
Intervention Mix – Tobacco Control SUPPORT DESIGN Give people the means to change Change physical context “Pop Up” cessation services in targeted Better NHS Stop Smoking Services areas e.g. 360 Asda supermarkets “Quit Kit” for cold turkey - quit plan, Over 200,000 people signed up to online information about medication, details for communities their local stop smoking services. INFORM/EDUCATE CONTROL/COMPEL Provide information Incentivise/Disincentivise Normalising NHS Stop Smoking - designed Legislation to end tobacco displays in “for people like me” shops Focus on fact that their children 3 times Using tax to maintain high price more likely to start Plain packaging of tobacco products
The Results• 480,000 Quit Kits ordered• 95% new contacts for the CRM database• 3 million smokers attempted to quit –08/09• Nearly 220,000 stayed quit 1 year later• Budget of £21,115,194 (down 22%)• 3 year payback of £73.5 million
Past evaluation work: example #1Measurement Method TimingFormative Qualitative DevelopmentOutcome Questionnaire – pre and post Development & Knowledge, attitudes, behaviour (intended evaluation and actual)Process Benchmark criteria Development, Horizon scanning implementation & Stakeholder interviews evaluation Emerging key insights, opportunities and threatsEvent monitoring Number of attendees Implementation & Impact on knowledge/awareness & evaluation behaviourMedia analysis Content analysis – qualitative Implementation & Impact – n evaluation Web analysis
Past evaluation work: example #2Measurement Method TimingAwareness and Quantitative survey: Development &experience of quitting Awareness and experiences evaluationProcess Benchmark criteria Development, Horizon scanning implementation & Stakeholder interviews evaluation Emerging key insights, opportunities and threatsService delivery Qualitative Development &evaluation evaluationCustomer journey Observation, including Development &mapping assessment of sign-posting evaluationTelephone helpline Quantitative survey Development &monitoring evaluation
New HIV diagnoses in 2006African Americansmake up 13% of thetotal population 18
What behaviour?Get an HIV test within3 months of havingunprotected sex 19
Key Insights• Fear of results & stigma are main barriers• Strong family and church ties• Everyday burdens outweigh concerns about HIV (e.g., children, work, bills, etc.)• Concerns about partner’s sexual behaviour are significant 20
Key Message• Look out for yourself – Take Charge. Take the Test• Emphasized: – Danger from their partners risky behaviour – Empowerment that comes with getting tested – Availability and convenience of testing – The “norm” that others were getting tested
The Marketing Mix• Targeted advertising – radio, direct mail, outdoor• Drive people to free HIV testing locations• Toll free line & website• Community outreach targeted key influencers (beauty salon owners etc)• Community “rapid HIV testing” events• Focus on “normalizing” testing 22
The Results• Significant increase in awareness and intention• Increased testing rates by nearly 70%• Calls to the HIV hot line rose 290%• Set new records for the number of HIV tests conducted in one day• No significant impact on use of existing HIV testing services 23
Key Lessons• Rapid testing events in the community were the key to success• They helped to reduce the “Intention – Action” barrier• Greater support for cities to expand the number, reach and impact of these events 24
Value for Money• VfM is about getting the best health possible for the resources available• Short term vs. long term impact
Measuring Health Gain, Costsand Savings• Health Gain is measured as QALYs – 1 QALY = £25,000• Costs to all stakeholders: Local Authorities, Government, NHS, Employer• Savings from long term reductions in costs to the stakeholders• Health Gain of behaviour change is taken from WHO Global Health Risk report – NICE used this data, in conjunction with their evidence base
Using the Tool: Data Needed• Baseline data for behavioural goal• Cost of intervention• Number of people in target segment/audience• Post intervention data – E.g. number of quitters – Age• % of target audience with high levels of disadvantage
Conclusion• Continuous evaluation• Measuring behaviour which counts• Who needs to know• How to sing!