v
UNDERSTANDING, MEASURING AND COMMUNICATING
YOUR IMPACT
Eibhlin Ni Ogain
Bristol, 21st June, 2013
AGENDA
Concepts and definitions
Understanding your impact
and what to measure
Assessing the strength of
evidence you need
...
NPC: TRANSFORMING THE CHARITY SECTOR
3
NPC works at the
nexus between
charities and
funders
Charity
SectorFunder
Increasin...
WHAT IS IMPACT MEASUREMENT ANYWAY?
4 4
• What is impact?
– The overall difference an organisation,
programme or interventi...
IMPACT: POSITIVE, NEGATIVE, INTENDED,
UNINTENDED COMPONENTS
5
Positive Negative
Intended
School programme that
improves ed...
UNDERSTANDING YOUR
IMPACT
NPC – Measuring your impact.: Introductory Workshop for Charities 7
7
Tracker 1 Tracker 2 Tracker 3 Tracker 4
WHERE TO BEG...
8
Enable people with mental illness or epilepsy to
live and work successfully in their own communities.”
“Make sure the be...
COMPONENTS OF NPC’S APPROACH TO
MEASURING SOCIAL IMPACT
9
Strategic vision / goals A well-developed
Theory of
Change
Exist...
1010
HOW TO LINK ACTIVITIES WITH LONG TERM
VISION
101010
• NPC approach: Use theory of change to identify
specifically wha...
THEORY OF CHANGE
• Links activities intermediate outcomes  final outcomes
– clarifies what the activities aim to achieve...
CASE STUDY
Scenario
• Intensive drama
programme with
young people.
• Programme lasts a
year and frequency
and duration of
...
TIPS FOR DEVELOPING A THEORY OF
CHANGE
13
• Get the views of lots of people—do you have the right people in your
workshop?...
COMMON PITFALLS
1. Including non-outcomes
2. Overcomplicating models
3. Over-claiming outcomes
4. Not following a logical ...
ASSESSING THE
STRENGTH OF EVIDENCE
YOU NEED
HOW ROBUST DOES EVIDENCE OF YOUR
IMPACT NEED TO BE?
• What do your target
stakeholders (funders?) think?
• What is possibl...
LEVELS OF EVIDENCE
17
Randomised
control trial
Anecdotes /
quotes
Before and
after survey
Self-reported
change
Case
studie...
LEVELS OF EVIDENCE: EXAMPLE
18
The Brandon Centre provides
counselling and psychotherapy to
young people between the ages ...
LEVELS OF EVIDENCE: EXAMPLE
19
The Brandon Centre provides
counselling and psychotherapy to
young people between the ages ...
20
• Before and after assessments and young person’s offending history recorded
• At two year follow up, MST group showed ...
MEASURING YOUR
IMPACT
QUALITATIVE OR QUANTITATIVE DATA OR
BOTH
To be informed by…
• the outcomes you want to measure
AND
• How accessible are yo...
NO MAGIC RULE FOR SELECTING
APPROPRIATE TOOLS AND MEASURES
It depends on:
– ‘Fit’ between outcome and measure – will the d...
TYPES OF QUANTITATIVE TOOLS & THEIR
USES
24
Type of tool
Measures
progress for
individuals
Aggregate
to show
change
Robust...
TYPES OF QUALITATIVE TOOLS AND THEIR
USES
25
Type of tool
Measures
progress for
individuals
Aggregate to
show change
Robus...
CASE STUDY
Scenario
• Intensive drama
programme with
young people.
• Programme lasts a
year and frequency
and duration of
...
27
The charity prioritised improved communication skills, confidence and
behaviour as outcomes to measure
Recommendation:
...
ARTICULATING YOUR
IMPACT
71
COMMUNICATING YOUR RESULTS
Knowing your audiences
• Commissioners
• Trusts & foundations
• Major donors
• Beneficiaries...
WHAT DO YOU NEED TO COMMUNICATE
Five key questions:
1. What’s the problem you’re trying to tackle?
2. What are you doing t...
3131
RESOURCES
• Website dedicated to Theory of Change: http://www.theoryofchange.org/
• NPC’s report on Theory of Change:...
RESOURCES
• Survey question bank: A website where you can research widely used surveys and
single questions by theme or us...
Upcoming SlideShare
Loading in …5
×

Understanding, measuring and communicating your impact

840 views

Published on

Eibhlin Ni Ogain, New Philanthropy Capital
South West Regional Group: understanding, measuring and communicating impact
www.charitycomms.org.uk/events

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Understanding, measuring and communicating your impact

  1. 1. v UNDERSTANDING, MEASURING AND COMMUNICATING YOUR IMPACT Eibhlin Ni Ogain Bristol, 21st June, 2013
  2. 2. AGENDA Concepts and definitions Understanding your impact and what to measure Assessing the strength of evidence you need Articulating your impact
  3. 3. NPC: TRANSFORMING THE CHARITY SECTOR 3 NPC works at the nexus between charities and funders Charity SectorFunder Increasing the impact of charities eg, impact-focused theories of change Strengthening the partnership Eg, collaboration towards shared goals Increasing the impact of funders eg, effective commissioning Consultancy Think tank
  4. 4. WHAT IS IMPACT MEASUREMENT ANYWAY? 4 4 • What is impact? – The overall difference an organisation, programme or intervention makes • What is impact measurement? – Planning, managing, tracking and reviewing how much difference you are making
  5. 5. IMPACT: POSITIVE, NEGATIVE, INTENDED, UNINTENDED COMPONENTS 5 Positive Negative Intended School programme that improves educational attainment Campaigns against multi-national corporations hurts shareholder profits Unintended School programme improves parents’ integration in UK Welfare payments create poverty traps / dependency
  6. 6. UNDERSTANDING YOUR IMPACT
  7. 7. NPC – Measuring your impact.: Introductory Workshop for Charities 7 7 Tracker 1 Tracker 2 Tracker 3 Tracker 4 WHERE TO BEGIN Why are we measuring? Alice: "Would you tell me, please, which way I ought to go from here?" Cheshire Cat: "That depends a good deal on where you want to get to." Alice: "I don’t much care where." Cheshire Cat: "Then it doesn’t matter which way you go."
  8. 8. 8 Enable people with mental illness or epilepsy to live and work successfully in their own communities.” “Make sure the best of the past is kept to enrich our lives today and in the future.” “Relieve distress, mobilise personal resources and facilitate growth in adolescents towards responsibility and self-fulfilment.” “…to help youth and their families to live, work, and study with dignity, hope, and joy.” “Transform the charity sector” MISSIONS
  9. 9. COMPONENTS OF NPC’S APPROACH TO MEASURING SOCIAL IMPACT 9 Strategic vision / goals A well-developed Theory of Change Existing evidence Appropriate measurement tools
  10. 10. 1010 HOW TO LINK ACTIVITIES WITH LONG TERM VISION 101010 • NPC approach: Use theory of change to identify specifically what a charity hopes to achieve through its activities, and come up with appropriate measures • Takes more time than listing measures • Common approach: list desired outcomes and put appropriate measures against them • Ignores distinction between a charity’s direct and indirect influence • Risks being a wish list that a charity finds too hard to achieve
  11. 11. THEORY OF CHANGE • Links activities intermediate outcomes  final outcomes – clarifies what the activities aim to achieve and how – provides a structure for identifying what can be measured – provides the case for why achieving intermediate outcomes is important 11 A conceptual map of how activities lead to outcomes
  12. 12. CASE STUDY Scenario • Intensive drama programme with young people. • Programme lasts a year and frequency and duration of contact is high. • New and untested approach but charity believes it to be very effective. • Want to generate evidence to illustrate this effectiveness and expand the programme to other locations. Young people take part in drama workshops focusing on role play Young people have improved communication skills Young people have more confidence to seek help Young people have improved behaviour Young people learn about different’ viewpoints Improved communication with caregivers and teachers Young people get the help they need Young people learn a new skill Young people feel better about themselves Programme using drama and theatre to improve communication skills and behaviour of at risk youth
  13. 13. TIPS FOR DEVELOPING A THEORY OF CHANGE 13 • Get the views of lots of people—do you have the right people in your workshop? • Do intermediate and final outcomes reflect user experience? • End goal—How will you know if it has been achieved? • Backwards mapping—what has to happen for this to happen? • Outcomes—are they actually outcomes? Are they all on there? • Assumptions—are your causal links based on evidence or not? • Enabling factors—what would derail your intervention?
  14. 14. COMMON PITFALLS 1. Including non-outcomes 2. Overcomplicating models 3. Over-claiming outcomes 4. Not following a logical flow 5. More than one outcome in the same box 6. Similar outcomes occur more than once 7. Outcomes not specific enough 14
  15. 15. ASSESSING THE STRENGTH OF EVIDENCE YOU NEED
  16. 16. HOW ROBUST DOES EVIDENCE OF YOUR IMPACT NEED TO BE? • What do your target stakeholders (funders?) think? • What is possible, given resources etc? 16
  17. 17. LEVELS OF EVIDENCE 17 Randomised control trial Anecdotes / quotes Before and after survey Self-reported change Case studies Control groups Credibility Basic Advanced Nesta level 1 Nesta level 2 Nesta levels 3, 4, 5
  18. 18. LEVELS OF EVIDENCE: EXAMPLE 18 The Brandon Centre provides counselling and psychotherapy to young people between the ages of 12 and 21 The Brandon Centre provides counselling and psychotherapy to young people between the ages of 12 and 21 Level two • The centre routinely carried before and after assessments Moving to level five • Results showed poor outcomes and early drop out for young people with behavioural problems • Reviewed evidence on effective interventions and found MST Needed to show that: • MST is more effective than current YOT services at reducing behavioural problems and youth offending
  19. 19. LEVELS OF EVIDENCE: EXAMPLE 19 The Brandon Centre provides counselling and psychotherapy to young people between the ages of 12 and 21 Level five • A randomised control trial was needed to answer these questions • Recruited young offenders on a referral or supervision order • Young people randomly allocated to two groups: MST or usual YOT services Young people YOT usual service MST Random allocation
  20. 20. 20 • Before and after assessments and young person’s offending history recorded • At two year follow up, MST group showed significantly greater reduction in offending and problem behaviours • Expansion of MST to other LA’s as a commissioned service LEVELS OF EVIDENCE: EXAMPLE 20 The Brandon Centre provides counselling and psychotherapy to young people between the ages of 12 and 21 Before measure After measure MST Before measure After measure YOT usual service
  21. 21. MEASURING YOUR IMPACT
  22. 22. QUALITATIVE OR QUANTITATIVE DATA OR BOTH To be informed by… • the outcomes you want to measure AND • How accessible are your users/beneficiaries/stakeholders? – How easy is it to get a representative sample? (for quant data) – How easy is it for respondents to take part in research? • How much time and resources do you have? 22
  23. 23. NO MAGIC RULE FOR SELECTING APPROPRIATE TOOLS AND MEASURES It depends on: – ‘Fit’ between outcome and measure – will the data be meaningful? – What is widely accepted? – Cost / effort to collect, use, analyse data – Cost/ effort from others to provide data – Intensity and duration of programme – Your appetite for rigour • NPC preference for standardised measures that have been tested to robust standards 23
  24. 24. TYPES OF QUANTITATIVE TOOLS & THEIR USES 24 Type of tool Measures progress for individuals Aggregate to show change Robust measure of change Explain why change has happened Admin. data ? ? ?  Case work tool (e.g. Outcome Star)   ?  Clinical tools     Scales (developed by researchers) ?    Single questions ?  ?  Hard outcomes (e.g. statutory data)    
  25. 25. TYPES OF QUALITATIVE TOOLS AND THEIR USES 25 Type of tool Measures progress for individuals Aggregate to show change Robust measure of change Helps explain why change has happened Interviews  ? ?  Observation  ? ?  Focus groups     Ad hoc feedback ?   ?
  26. 26. CASE STUDY Scenario • Intensive drama programme with young people. • Programme lasts a year and frequency and duration of contact is high. • New and untested approach but charity believes it to be very effective. • Want to generate evidence to illustrate this effectiveness and expand the programme to other locations. Young people take part in drama workshops focusing on role play Young people have improved communication skills Young people have more confidence to seek help Young people have improved behaviour Young people learn about different’ viewpoints Improved communication with caregivers and teachers Young people get the help they need Young people learn a new skill Young people feel better about themselves Programme using drama and theatre to improve communication skills and behaviour of at risk youth
  27. 27. 27 The charity prioritised improved communication skills, confidence and behaviour as outcomes to measure Recommendation: • Use short standard scales to measure these outcomes, pre and post participation in the programme. Follow up one year after they finish the programme. • Use academic research to evidence the links between improvements in these outcomes and reduced risk of anti-social behaviour. • Use these pre and post questionnaires with a random sample of 50 young people as it has limited resources to measure everyone it works with. CASE STUDY
  28. 28. ARTICULATING YOUR IMPACT
  29. 29. 71 COMMUNICATING YOUR RESULTS Knowing your audiences • Commissioners • Trusts & foundations • Major donors • Beneficiaries • Volunteers • Media etc. • Regulators • Trustees • Management team • Staff Funders StakeholdersOrganisation
  30. 30. WHAT DO YOU NEED TO COMMUNICATE Five key questions: 1. What’s the problem you’re trying to tackle? 2. What are you doing to address it? (your activities) 3. What does that achieve? (your outcomes) 4. How do you know? (your evidence) 5. How are you learning & improving? 30
  31. 31. 3131 RESOURCES • Website dedicated to Theory of Change: http://www.theoryofchange.org/ • NPC’s report on Theory of Change: http://www.thinknpc.org/publications/theory-of- change/ • A good website to look for measures and indicators: http://wilderdom.com/tools/ToolsSummaries.html • http://www.performwell.org/ A website where you can find outcomes and standardised scales relevant to a range of human welfare services • The innovation network’s logic model workbook: http://www.innonet.org/client_docs/File/logic_model_workbook.pdf • The Centre for What works (http://www.whatworks.org/) has an outcome portal where you can browse different outcomes and their corresponding indicators. • TRASI (http://trasi.foundationcenter.org/) has a database where you can search different approaches to impact assessment. • The resource page of the Inspiring Impact programme (coordinated by NPC in partnership with 7 other organisations), which includes a list of measurement tools and systems http://inspiringimpact.org/resources/
  32. 32. RESOURCES • Survey question bank: A website where you can research widely used surveys and single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/ • A list of the most common psychological scales and questionnaires: http://www.ull.ac.uk/subjects/psychology/psycscales.shtml • Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science Students and Researchers. Sage Publications Ltd • How to communicate your results: Hedley, S et al (2010) Talking about results. New Philanthropy Capital. http://www.thinknpc.org/publications/talking-about-results/ • A Journey to Greater Impact: NPC report profiling six charities who radically improved their approach to impact measurement: http://www.thinknpc.org/publications/a-journey- to-greater-impact/ • NPC’s mapping of outcomes for social investment: a number of report mapping outcomes in different charity sectors and outlining appropriate measurement tools: http://www.thinknpc.org/publications/mapping-outcomes-for-social-investment/ • Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design. Jossey-Bass: San-Francisco. 32 3232

×