Where does the research fit it in? http://thensmc.com/
Get your evaluation in earlyProject/programme cycle Implementatio Planning Evaluation nIntervention cycle Intervention Insight Implementatio Best developmen Evaluation research n practise tEvaluation FORMATIVE EVALUATION: PROCESS OUTCOME or IMPACT - Insight research EVALUATION EVALUATION - Baseline + - Pre testing DOCUMENTATION OF LEARNING/BEST - Design of process and PRACTICE impact evaluation cycle
School of thought that questions how useful market research is…“Whether it is company executivesseeking to define their corporatestrategy or politicians wanting tounderstand the electorate, theidea that questions answered on aquestionnaire or discussed in afocus group can provide usefulinsights on which to base businessdecisions is the cause of productfailures, political blunders andwasted billions”Philip Graves, Consumer.ology
Results from our global study on government intervention in 24 countries What, if anything, do you think government should do? Average over all four policy areas % Strongly support/tend to support intervention Provide information 92% Force of Provide incentives 87%Make behaviour more 69% expensive/ difficult Ban behaviour 62% Make companies act 88% against behaviour Source: Ipsos Global @dvisor Base: c.500 - 1,000 residents aged 16-64 (18-64 in the US and Canada) in each country, November 2010
People want financial incentives to save for their pension, and 7 in 10 support being forced to save Next, thinking about how people plan for retirement. What, if anything, do you think government should do? % Strongly support/tend to support Provide information 92% Provide incentives 90% Make pension scheme enrolment automatic 79% Make pension scheme enrolment mandatory 69%Make employers contribute to 87% pension schemes Base: c.500 - 1,000 residents aged 16-64 (18-64 in the US and Canada) Source: Ipsos Global @dvisor in each country, November 2010
But we also gave people theopportunity to say whether thegovernment should not get involvedin their behaviour
Half still have a negative gut reaction to the “nanny state” What, if anything, do you think government should do? Average over all four policy areas % Strongly support/tend to support Not get involved in what people choose to eat Not get involved in what people choose to save for retirementNot get involved in whether or notpeople choose to live sustainably Not get involved in how people make decisions about smoking Not get involved (average) Base: c.500 - 1,000 residents aged 16-64 (18-64 in the US and Canada) Source: Ipsos Global @dvisor in each country, November 2010
Inconsistent views? 53% agreed that “government should not get involved in what people choose to save for retirement” 69% agreed that “government should change the law so that everyone has to enrol in a pension scheme” AND 36% agreed with both statements!
What people say and how they act is different but… “A measure which does not have public support is, in general, less likely to succeed” House of Lords report on Behaviour Change
Putting the customer at the centre isnot the same as asking them everytime Source: National Social Marketing Centre
The danger of not asking first… “Doesn’t bother me, it’s shit here.” Blackburn resident
Knowing about something does notmean you’ll do it
What is relevant to measure? Q Have you eaten any fruit and vegetables in the last 24 hours?20050029 5%1 portion 7%2 portions 10% 10% Awareness of 5-a-day 11% recommendation3 portions 2005 2009 12% % %4 portions 13% 5 portions a day 68 78 14% 1-4 portions a day 14 8 55% 6+ portions a day 3 15+ portions 49% Don’t know 14 10Not stated 6% 8% Not stated 2 3 Think about what you are trying to Base: achieve All respondents living in core wards (1,732). (2 February 27-March 2009)
Just because it is new and creativedoes not mean it is good A methodology needs to be fit for purpose rather than just innovative
A couple of case studies#1 Cervical Screening in TowerHamlets
Our research approach Secondary research 11 pilot interviews with 9 key stakeholder women in Tower interviews Hamlets Peer research: 15 peer researchers speaking to 82 women in their social networks Ethnographic interviews: 6 detailed video interviews Primary research
What is peer research? a participatory qualitative approach based on training members of a community (peer researchers) drawn from hard-to- reach groups, to carry out in-depth conversational interviews with individuals from within their own social networks by tapping into established relationships of trust peer research generates rich narratives about people’s lives quickly, providing a depth of insight into how people view their world, conceptualise their behaviour and experiences, and make decision on key issues. it also builds channels through which the voices of frequently excluded groups can be heard, and enables these groups to enter into dialogue with programmes, implementers and decision makers
Our approach to peer research We recruited 15 peer researchers (through nurseries, schools, housing associations & community organisations) - 7 Bangladeshi women, 6 white British women and 2 Somali women All women were invited to two half days of training The peer researchers were given 2-3 weeks to undertake at least 5 interviews with their friends and family. The women were given quotas they were asked to fill They were given a data collection sheet to record the findings of each interview as well as NHS leaflets on the facts of cervical screening to hand out after the interviews After each interview they completed, they were asked to phone us to feedback the findings Findings feedback workshops were held after each group of peer researchers had completed their interviews
Competition analysis – cervicalscreeningAim is to understand what factors compete for the time and attention of the audience, and includes internal (e.g. psychological factors) and external (e.g. people and contextual influences) competitionInternal factors: External factors: – Women’s perceptions of – Administrative and process screening as painful, failures (incomplete addresses, embarrassing or inconvenient reminders not sent) – The fear of detection of cancer – System failure (formal and informal opt-outs, no mechanism – Lack of awareness of the tests’ for chasing non-attenders) indications and benefits – Inconvenient clinic times – Lack of awareness of the screening procedure (e.g. due to – Unavailability of a female absence of screening in home screener country) – Lack of information in – Considering oneself not to be at appropriate language risk of developing cervical cancer – Social and cultural norms (which – Linguistic difficulties contradict health advice)
#2 Increasing brief interventions foralcohol in A&E
Aims of the project To increase the number of patients seen in A&E who receive a brief intervention for alcohol; To effectively engage with health professionals working at A&E departments in Tower Hamlets To increase the number of health professionals who believe that delivering the brief intervention for alcohol is effective; To increase the number of referrals from A&E to relevant local services. To establish how best to integrate and utilise two new specialist alcohol nurses in A&E
Our research approachScoping phase Review of secondary Secondary literature, including research previous work in this area and with this audience.Primary 11 Interviews with a rangeresearch of A&E staff to understand Stakeholder the initial issues around the interviews delivery of brief interventions. Shadowing of healthcare professionals during their shifts in A&E to observe Ethnography: barriers and observations in observation practice, cross-checking early findings.
Access all staff – 11 shifts Ambulance workers We followed Police in the department one staffShadowed Nurses at all levels member member each but of staff Doctors at all levels accessed 10-20 GP streamers people each Receptionist shiftWe accessed A&E at different times of the day and night and weekdays and weekends
Pattern drinking is perceived as binary not on a continuum The attitude in A&E seems to place drunks in two categories: ‘Problem drinker’ and ‘A few too many’
The danger of ‘Us vs. Them’ – two models Referr Vs.A& Referr A& al al teamsE teams E It was very clear that for referrals to work the relationship must be a partnership
Referral system needs to be easy to use and become second nature Fast Clear Easy to use Need limited involvement Not need further paperwork The key to the success of the nurses is that they are seen to decrease rather than increase workload
Senior staff really set the toneSenior staff must be aware how their perception of alcohol affects that of everyone else
Interesting to revisit this in light of thefindings…
Existing Examples (1): Defra’s Environmental SegmentationModelPlotting the Segments Ability to act High High potential andSegment willingness and willingability 1: Positive greens 2: Waste watchers I think it’s important that I do as ‘Waste not, want not’ that’s much as I can to limit my impact important, you should live life 3: Concerned on the environment. thinking about what you are consumers 18% doing and using. I think I do more than a lot of 12% people. Still, going away is important, I’d find that hard to give 7: Honestly up..well I wouldn’t, so carbon off- disengaged setting would make me feel Maybe there’ll be an better. Willing environmental disaster, maybe not. Makes no difference to me, 14% to Act 5: Cautious participants I’m just living life the way I want I do a couple of things to help High to.Low the environment. I’d really like 18% to do more, well as long as I saw others were. 14% 6: Stalled starters 4: Sideline supporters I don’t know much about I think climate change is a big climate change. I can’t afford problem for us. I know I don’t a car so I use public think much about how much water transport.. I’d like a car or electricity I use, and I forget to though. turn things off..I’d like to do a bit 10% more. 14%Low potential andunwilling Low
Key principles for segmentationSegmentation is… 1. A practical tool Key question: What do you want the model to do? (eg. tightly predict specific behaviours, or show differences/similarities between different audience groups?) (eg. will you be conducting further research with the segments, or ‘just’ targeting them?) If a behaviour change tool: segment on the behaviour(s) in question, OR the most proximal determinants of those behaviours 2. An iterative process Make this as transparent as possible but keep it flexible 3. As much an art as a science Focus on practical purposes of the model for diverse stakeholders: more heads are better than one (steering group, advisory group etc)
Attributes of an effective segmentationmodel1. Usability - Is the model easily understandable, and memorable – can people assimilate and use it? - Can the segments be reached in the ‘real world’?2. Replicability - Can the segments be easily found in subsequent qualitative and quantitative samples?3. Stability - Will the segments be relatively stable over time?
A 5 stage process for segmentation 1. Scoping Stage / Survey Design 2. Survey Fieldwork, Topline Findings 3. Factor/Cluster Analysis 4. Profiling the Segments 5. Embedding and Replication
Again do look to see what has alreadybeen done
Spending money on the researchversus the interventionResearch Interventio n Difficult to justify research sometimes when conducting it will take money away from interventions
What can you do to save money? Find out what has been done already – Move away from every population is different – Contact research companies, government departments, academics – This means you are filling in the gaps rather than starting afresh Is there routinely available data out there? – Possibly very useful for evaluation Who can you team up with to pool resources? – Shared aims and resources Be sure what you need – The sharper the brief the more you will get back
Developing a theory of change The model sets out the links between the chosen focus of this initiative and the activities, short and longer-term outcomes whose effectiveness is being evaluated. Impacts (longer Outputs: Outcomes (short term): Inputs: • What will/has and medium • What is the long term • What resources have been made happened/been done as a term): reason for the programme – available for the result of the inputs? what is it designed to bring •What are the interim ways programme? about to measure whether the impacts are likely to occur
Some examples of when it didn’t workout as expected The teenage pregnancy intervention that had a rather different outcome from the one intended Introducing sterilising tablets into prisons to clean needles which had an unfortunate side effect The initiative to reduce the level of mugging in Deptford which proved unpopular with their neighbours Unintended consequences show the importance of research
So…Start earlyBe clear where you want to get toAll this is not as new as it seemsDesign a methodology fit for purpose but flexible!Think about what data you already haveLook at what has already been doneLook to innovation when appropriateAlways think about unintended consequencesGood luck!
Thank youFor further information contact:email@example.com 020 7347 3000
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.