Iona Joy, Head of charity team at NPC, presented on NPC's approach to measurement at the CSV conference in March 2015. NPC works at the nexus between charities and funders to increase the impact of both. NPC uses a four pillar approach to measuring impact that involves mapping a theory of change, prioritizing outcomes to measure, choosing an appropriate level of evidence, and selecting data sources and tools. An effective measurement framework is developed through this process to help charities strategically plan, deliver, assess, and review their impact.

1. vNPC’S APPROACH TO MEASUREMENT
Iona Joy, Head of charity team, NPC
CSV conference March 2015
#impactmatters

2. NPC: TRANSFORMING THE CHARITY SECTOR
2
NPC works at the
nexus between
charities and
funders
Charity
SectorFunder
Increasing the
impact of charities
eg, impact-focused
theories of change
Strengthening the
partnership
Eg, collaboration
towards shared
goals
Increasing the
impact of funders
eg, effective
commissioning
Consultancy
Think tank
#impactmatters

3. OUR STRATEGY PROCESS
3
Strategy development sits within a wider process
Development ImplementationPreparation
Leadership
Stakeholders
Culture/
values
Plan
Deliver
Assess
ReviewPu
rp
os
e
Purpose
Strategy
ExternalResources
Leadership, governance and culture
focused on continuous learning
Development ImplementationPreparation
#impactmatters

4. IMPACT CYCLE
4
Plan
for impact
Deliver
with impact
Assess
impact
Review
for future impact
• Short term service design:
rapid response to data
• Long term strategy design:
adaptation of strategy
#impactmatters

5. WHY USE IMPACT DATA IN YOUR STRATEGY?
INCENTIVES
5
Internal
(improving
services)
External
(pressure
from funders)
• Internal pressure to
improve service delivery
25% use measurement to
improve services* NPC hopes
more will take up cause!
*Pritchard, D., Ni Ogain, E., Lumley, T. 2013. Making an Impact. New Philanthropy Capital.
• External pressure from
funders
Half of charities* driven by
funders to measure
#impactmatters

6. THE FOUR PILLARS APPROACH TO
MEASURING IMPACT
6
Map your
theory of
change
Prioritise
what you
measure
Choose
your level
of evidence
Select your
sources and
tools
Effective measurement
framework developed
Strategic vision
Leadership
Case for impact measurement
#impactmatters

7. DEVELOPING AND USING
A THEORY OF CHANGE
#impactmatters

8. WHAT IS A THEORY OF CHANGE?
Links activities intermediate outcomes  final outcomes
A description of how activities lead to outcomes
8
- Clarifies what the activities aim to
achieve and how
- Provides the case for why achieving
intermediate outcomes is important
- Provides a structure for identifying
what can be measured
#impactmatters

9. HOW TO REPRESENT A THEORY OF CHANGE
9
Planning Triangle Logic Model Outcomes Chain
However you represent your theory of change, it should be supported by a written description.
#impactmatters

10. THEORY OF CHANGE FOR A PARENTING
CHARITY: OUTCOMES CHAIN EXAMPLE
Improved life
chances for
children from
disadvantaged
backgrounds
Practical 1-
1 support
to parents
Increased
parental
commitment to
school Improved
attainment of
children from
disadvantaged
backgrounds
Improvement in
children’s
abilities and
temperament
Improved
attendance and
behaviour in
school
Address specific
problems
stopping children
from attending
school
10
Parents better
support
children’s
learning
Improved
parenting skills
Parenting
classes
School-based
activities to help
children from
disadvantaged
backgrounds
Activity
Enabling
factor
Intermediate
outcome
Long-term
goal
#impactmatters

11. PRIORITISING OUTCOMES
#impactmatters

12. PRIORITISE OUTCOMES THAT:
• Are directly influenced (rather than indirectly supported)
• are important / material to the mission
• are not too costly to measure
• will produce credible data
12
#impactmatters

13. CHOOSING THE RIGHT
LEVEL OF EVIDENCE
#impactmatters

14. THREE AUDIENCES FOR EVALUATION
14
1) Funders/commissioners:
- Does the intervention make a difference?
- What’s your track record?
- What are the costs/benefits?
3) The evidence-base:
- What have you learned?
- How can we all achieve more?
2) Your own organisation
- Do we really make a difference?
- How can we improve?
#impactmatters

15. COUNTERFACTUAL
• Comparing the world with the
service in it with what the world
would be like without it.
• Control group
• Attribution
15
#impactmatters

16. LEVELS OF EVIDENCE
16
Randomised
control trial
Anecdotes /
quotes
Before and
after survey
Self-reported
change
Case
studies
Control
groups
Credibility
Basic Advanced
#impactmatters

17. ALTERNATIVES
• Waiting lists
• Dosage: users who’ve received a smaller ‘dose’ of a
service
• Type of service: comparing users who’ve received one
type of service compared to another
• Use existing data sources to create control groups (quasi-
experimental approaches)
• Ask users whether it’s made a difference
17
Comparison groups
No comparison group
#impactmatters

18. SELECTING DATA
SOURCES AND TOOLS
#impactmatters

19. QUESTIONS TO ANSWER (ITERATIVELY)
19
• Who?
• When?
• How?
Which
beneficiaries?
Stakeholders?
Staff?
Everyone or
sample?
Before and
after?
How
frequently?
Quantitative
and/or
qualitative?
How credible
does the data
need to be?
#impactmatters

20. • Quantitative data (numbers)
• Statistical estimates
• Prevalence of views,
attitudes and experiences
• Admin data or questionnaires
(paper, web, etc.)
• Qualitative data (words)
• Detailed understanding
• In-depth interviewing
(telephone or face-to-face)
• Observation
• Stakeholders’ views
Proportion of
beneficiaries whose
outcomes have
improved, and by how
much.
What did beneficiaries
think, did it make a
difference to them?
How? How could it have
been better?
DIFFERENT TYPES OF EVIDENCE
#impactmatters

21. A LOT OF IMPACT EVIDENCE ALREADY EXISTS
21
Improved life
chances for
children from
disadvantaged
backgrounds
Practical 1-1
support to
parents
Increased
parental
commitment to
school Improved
attainment of
children from
disadvantaged
backgrounds
Improvement in
children’s
abilities/
temperament
Improved
attendance and
behaviour in
school
Address specific
problems
stopping children
from attending
school
Parents better
support
children’s
learning
Improved
parenting skills
Parenting
classes
School-based
activities to help
children from
disadvantaged
backgrounds
✔
✔
✔
✔✔
✔
?
?
?
?
https://www.gov.uk/government/publications/national-evaluation-of-the-troubled-families-programme
http://www.local.gov.uk/c/document_library/get_file?uuid=1f145177-2552-47dc-978f-e46f6c65f4cd&groupId=10180
#impactmatters

22. EVALUATION TRAPS
22
Collect data that matters, and work together
Don’t force
squares into
circles & don’t
collect arbitrary
data
#impactmatters

23. INSPIRING IMPACT: MEASURING UP! AND
THE IMPACT HUB
Measuring Up!
• online, step-by-step self-assessment tool
• enables you to review and improve your
organisation’s impact practice
• looks at the way you plan, evidence, communicate
and learn from the difference your work makes
The Inspiring Impact Hub
• pulls together resources relevant to improving
impact practice
• enables users to search and filter results according
to their needs
23
#impactmatters

24. RESOURCES
24
http://www.thinknpc.org/publications/theory-of-change/
http://www.ces-vol.org.uk/Publications-Research/publications-
free-downloads/making-connections
NPC – Brief Introduction to Theory of Change
http://inspiringimpact.org/
http://www.thinknpc.org/publications/npcs-four-pillar-approach/
http://www.thinknpc.org/publications/creating-your-
theory-of-change/
#impactmatters

25. NPC’S STRATEGY TRIANGLE
25
Focus on what achieves impact
Core purpose
Resources and
capabilities
Strategy
External
environment
Mission, vision, values
Theory of change
Impact data into ToC
Needs mapping
PESTLE/STEEPL
5 forces
Policy
Organisational/
governance review
eg Little blue book
Options analysis
#impactmatters

26. v
THANK YOU
Iona.Joy@thinknpc.org
London, March 2015