4. The Building Change Trust is
a 10 year £10 million charitable fund
endowed by the Big Lottery Fund
to support change and transformation in the
Northern Ireland Community and Voluntary
sector
5. Our vision
“A strong, vibrant, independent
and relevant community and
voluntary sector in NI”
6. CENI - one of the 5 original Trust bid partners
along with CFNI, RCN, BITC and VNOW
CENI - significant work on outcome and impact
Measuring Up – identifying needs
Making Reporting more Effective
Use of Quality Standards
Measuring Change approach piloted with BIG, Belfast
City Council and Neighbourhood Renewal
7. Trust actively considering its further and future
role with respect to impact measurement
May appear an esoteric and abstract issue
but
8. Our vision of a strong, independent and
relevant sector that changes people and places
for the better requires that:
• More cvs organisations are helped to focus
on impact
• Cvs organisations understand and use
appropriate tools to set out achievements
9. Context is
• Evidence based policy making and outcomes
based commissioning
• Identification of and investment in what works
• Organisations are striving to make best use of
limited resources for maximum benefit
10. Anticipate continuing to work with CENI
• Exploring use of impact measurement
process with government
• Will also be working in partnership with NPC
and others to bring benefits of Inspiring
Impact to NI
11. Plan to invest some resources focusing on
impact in order to support change for positive
development in sector
Not something we can do alone and want to
work with government and the sector to take this
initiative forward
12. Keen to hear your views on what action needs to be
taken here.
In the first instance contact Trust Administrator, Nigel McKinney
info@buildingchangetrust.org
Visit our website
www.buildingchangetrust.org
Twitter
@changetrust
14. Challenges and issues: a UK perspective
from the Inspiring Impact programme
Tris Lumley, Head of Development, NPC
Collaborating on Outcomes: Funders and the Sector
Working Together
30th May 2012
15. What’s the context for impact
measurement?
• Charities and social enterprises under pressure to
demonstrate impact
• What should they do to improve?
• Driven by funders or for own benefit?
• What’s proportionate for them?
• Working in isolation, reinventing the wheel
• Funders may want to help
• What should they do to support improvement?
• Who should pay to increase charities’ capacity?
• How should funders think about their own impact?
17. If impact measurement is driven by funders…
…it will probably fail to become embedded and
really help charities learn and increase impact
We need structural solutions to structural barriers
18. Why am I here?
• NPC trying to help create structural solutions:
• Inspiring Impact—10 year collaborative programme
• Focusing on barriers, solutions and incentives
• At this conference:
• To share a UK perspective on impact
• To be part of discussion in NI context
19. What are the barriers to progress?
Providers Funders Commissioners
Incentives weak (with few Incentives highly variable Incentives skewed (towards
penalties for poor impact reporting). (and often inconsistent and poorly outputs and ‘cost to serve’. Focus on
communicated) accountability over evidence / learning.)
Cultural resistance (at
leadership and frontline) Cultural resistance (to Cultural resistance (cuts are
prioritise impact, to spend money on both a positive and a negative factor)
Insufficient resources (money, measurement)
expertise and capacity) Insufficient resources (money
Insufficient resources (money and capacity for them and provider)
Technical challenges (don’t and capacity for them and provider)
have skills and systems to measure; Technical challenges (skills
lack of consensus on indicators, Technical challenges (skills and systems to identify high impact
methods etc) and systems to identify high impact providers, and collect and synthesise
providers, and collect and synthesise evidence)
Practice (inability to attribute (or learning and evidence)
predict) an organisation’s contribution to Practice: (inability to use impact
outcomes with confidence) Practice: (challenge of building the data to drive cost reduction / budget
evidence base over the long term) reallocation)
Cross-cutting barriers
19
20. What are Inspiring Impact’s solutions?
• We’ve identified five key strands of work
• Leadership and culture
• Coordinating support
Charities & social enterprises
• Data, tools and systems
• Shared measurement
• Funder, commissioner & investor practice
22. Answers to key questions
What does good practice look like?
What’s proportionate?
How do others do it? How do we stack up?
What works in our field? How can we improve?
What approach is right for us?
Which tools/systems should we use?
23. Concrete deliverables
What does good practice look like?
Code of Good Impact Practice
What’s proportionate?
How do others do it? Impact
How do we stack up?
Shared measurement
What works in our field? Diagnostic
How can we improve?
What approach resources us?
Coordinated is right for
Marketplace for tools/systems?
Which tools/systems should we use?
25. Capturing different ‘types’ of funder
impact
Strategic Strategy for deploying resources to
impact achieve impact
Practice How behaviour as a funder impacts on
impact grantees
Funding The impact on beneficiaries and the
impact community achieved through funding
26. Plans with funders will emerge
dynamically
• Working group of foundations
• Identify key aspects
• Explore potential research, actions
• Scope out plans and campaigns for short and
medium term
28. We’re just getting started
• Programme launching June 2012
• Establishing links/partnerships across UK
• Building advisory groups for key projects
• Gathering intelligence on existing initiatives
• Want to hear from you about things we should be
aware of and opportunities to engage…
29. Get in touch…
Tris Lumley, Head of Development, NPC
tlumley@philanthropycapital.org
020 7620 4883
31. Brendan McDonnell
Director CENI
Collaborating on Outcomes
Introducing
‘Measuring Change’
32. Context
Climate of austerity and increasing social need - challenge for public
funders and projects to measure the outcome of their interventions.
Concordat and PAC report - need for collaboration between Govt & Sector
Recent CENI research shows that:
existing systems focus on counting activities and ensuring financial
probity - not designed to measure outcomes
approaches not standardised so difficult to aggregate diverse project
outcomes to show overall programme impact
prohibitive cost of comprehensive, robust outcome
measurement systems can ‘crowd out’ resources
.
33. Measuring Change
Practical, robust and cost-effective approach to capturing the
hard-to-measure qualitative outcomes of community-based
programmes.
What are the things you want to Change? Outcomes
Where are you starting from? Baseline
What difference has been made? Change
34. How it Works
Two distinct but connected components:
• Developing an overarching framework of programme-level
outcomes
• Applying an innovative data collection method to measure
project baseline and change against these outcomes.
Underpinned by involvement of both programme and project
stakeholders in a collaborative process facilitated by CENI.
35. Develop Outcomes Framework
Key Elements
Strategic Focus: Facilitated sessions with Funder to rationalise a
‘theory of change’ for the programme and articulate specific change
outcomes.
Structure/Themes: Identify key themes which give shape and
structure to an outcomes framework - common format to locate
diverse project-level outcomes.
Stakeholder Involvement: Process informed by input from both
projects and programme funder - fosters shared ownership of the
outcomes framework.
36. Sample Programme Outcomes Framework
Theme Change Outcomes
People • Enhanced Individual Capacity
• Educational Development
• Better Healthy lifestyle Choices
• Positive Mental Health
• Improved Family Cohesion
• Improved Personal safety
Community • Better Engagement with Hard to Reach Groups
• More Active Involvement in the Community
• Enhanced Volunteering
• Improved Access to Community Facilities
Organisation • Better Partnership working
• More Strategic influence
• Enhanced Practice development
Relationship • Better Bonding
• More Bridging
• Improved Linking
37. Data Collection: ‘Nominal Group Technique’
Multiple Perspectives: involves stakeholders (partners, management,
staff, volunteers, users) in a facilitated ‘Expert Panel’ - maximises
involvement and enables ‘triangulation’ of evidence.
Facilitation: CENI evaluator as a ‘critical friend’ ensures equality of
input across all stakeholders, challenging and testing their evidence.
Measurement Scale: uses the Rickter numerical scale (0 - 10) to rate
project baseline position and change. Scale helps neutralise contentious
debate and reach consensus.
Critical Debate: facilitates candid discussion and critical peer review -
challenges stakeholders to prioritise what outcomes are
important and be realistic about their contribution to change.
38. Putting a Metric on Qualitative Change
Low High
0 1 2 3 4 5 6 7 8 9 10
• Baseline – panels rate their project on each outcome at the start
• Change – panels rate their project’s progress on each outcome (after
a
set time)
Panel discussion is recorded to give context and rationale for the scores
Projects can also rate importance and difficulty of achieving
outcomes
39. Data Analysis
An effective way of extracting large amounts of data from diverse
perspectives within a project and distilling this into an agreed estimate.
Applied systematically across multiple projects it enables the generation
of robust evidence in a standardised format.
This can be analysed:
Horizontally: to baseline and measure the progress of individual
projects against the programme outcomes
Vertically: to indicate overall programme impact by aggregating
data from individual projects
40. Baseline and Change Scores for Funded Projects
10.0
9.0
8.0
7.0
6.0
Baseline
5.0
Change
4.0
3.0
2.0
1.0
0.0
A B C D E F G H I J
Projects
41. Case Studies
Measuring Change has been successfully piloted by CENI across a
number of funding programmes including:
Big Lottery Fund - support grant holders to baseline and measure
change.
Neighbourhood Renewal Programme - identify and baseline
Community Renewal outcomes.
Belfast City Council - inform review of strategic outcomes and help
support change management.
42. Conclusion
Still a work in progress – ‘Another Tool in the Kit’
Successfully piloted – case studies
Flexible – readily transferable to variety of situations (Age NI, IFA)
Provides mechanism to aid collaboration in both identifying and
measuring outcomes
Contribute to wider body of knowledge on evaluation and
outcomes measurement
Shift from Counting Activities to Measuring Change
43. Measuring Change Case Studies
Big Lottery Fund Norrie Breslin
Neighbourhood Renewal Alison Chambers
Belfast City Council Catherine Taggart
44. Supporting grant holders to
baseline and measure change
Norrie Breslin,
Head of Policy and Learning
Big Lottery Fund NI
45. Mission and Values
Our Mission
• Bringing improvements to communities and the
lives of people most in need.
Our Values
Making best use of lottery money
Using Knowledge and evidence
Being supportive and helpful
46. Funding approach
Strategic programmes
Outcomes funder
Partnerships
Working across outcomes
2 stage application process
Development grants
Self evaluation
5 year projects
47. Programmes
Live and Learn
17 Projects, totalling over £16.2million
Learning and well being
Safe and Well
18 Projects, totalling nearly £17million
Safety and well being
Wide range of projects and beneficiaries
Across geographic and specific communities
48. The Issue
Varied response to self evaluation
Support for grant holders
Demonstrate impact
Move from programme specific evaluation
49. The Journey
Change matrix
Communicating to grant holders
Communicating to staff
Encouraging participation
50. The Outcome
Participation 27/35
Impact on grant holders
Involving wider staff team
NI Committee
BIG UK wide
Next steps
52. Neighbourhood Renewal
Strategy launched 2003
36 Neighbourhoods
Over 102,000 households - around 280,000 people
Neighbourhood Partnerships established in each area
Representative of public/private/political and voluntary
and community sectors
Neighbourhood Action Plans developed detailing the
priority issues to be addressed in each area
61. Context
• BCC Investment programme
• Economic Environment
• BCC Community Development Framework
• RPA – planned transfer of Regeneration function
• Pilot – single CD fund for city
62. Community Grant Programme
• BCC more than just a grant administrator: both
financial and officer support
• Programme across 6 grant categories:
- Advice & Information
- Capacity Support
- Revenue Support for Community Buildings
- C&YP: summer schemes and Ur City 2
- Project grants
- Community Chest
• £2.6m across 412 grants last year
63. Service grant programme
Small grants:
community development projects
Grants for Summer schemes appropriate for children & young people aged
5-14 years
Community chest grant (in support of, for example, small scale seasonal
activity)
Large grants:
• Advice services: core funding to cover revenue and other operating costs;
supplementary funding (with DSD) for advice outreach costs
• Revenue (financial support towards running costs of community buildings)
• Capacity building (grant funding to support the building
of skills and knowledge of community organisations and
local people).
65. Community Development Framework
Community development activity is the main means by which we can
be better engaged with local people and support their involvement in
improving the city and its neighbourhoods. It enables people to come
together to:
• define needs, issues and solutions for their community; and
• influence or take decisions about issues that matter to them
and that affect their lives;
• take action to help themselves and make a difference.
It is a long-term, value-based process which targets
positive social change.’
66. Community Development Strategy
Align our resources to
achieve new vision and
strategy
• Core Community
Development work
• Engagement that works
• Building Effective
Partnerships
• Shared Service Design &
Delivery
66
67. CENI pilot (& beyond .........)
Capacity Building Grant elected to pilot the application of the
Measuring Change model within a funding programme
designed to build the assets of local communities
Ambition to review and agree the strategic outcomes of the
programme in light of the CD framework and
To develop an outcomes framework based on Social Assets
To inform the design of related assessment criteria (for a
future version of the programme)
To inform the design of a performance management
framework (compliance and change)
68. Capacity Review
1. Link supported grant activity to identified programme outcomes
2. Produce Standard Framework of Programme Outcomes;
3. Apply Standard Measurement Tool to:
• Baseline projects against programme outcomes
• Evidence individual contribution to programme
• Enable comparison or relative change across projects, themes
• Aggregate to demonstrate programme impact
4. Support Projects to Use Framework and Measurement Tool
68
69. Why ................
Funder Funded organisation
Demonstrate the services we Demonstrate potential to funders,
commission are having a positive board, user groups, recipients .....
change on individuals and society and
therefore strengthen the case for
renewed funding.
Monitor performance / compliance Know project / service is on track
Identify and disseminate practice Contribute to an evidence base of
learning practice
Communicate impact of investment Raise profile (communications)
Motivate staff, elected members, ... Motivate staff and volunteers
69
70. How...............
• CENI trawl of current applications to list outcomes
as stated by funded groups
• Group these outcomes into common themes,
using the original SA framework to help shape the
approach
• Staff consultation (NGT): clarify purpose and focus
and agree priorities
• Iterative process to produce refined strategic aim
and draft outcomes framework
71. What ............
Theme (4) Desired Outcome (11)
Core: Enhance Capacity of funded Support Group
Developing Support Group core Stronger Partnership
competency Increased leverage
Operational: Improved local infrastructure of groups
Developing Local Groups (to be capable, Improved group capacity
representative and resilient) Increased group resilience
Operational: Increased individual participation
Developing Individuals (to be more Improved individual skills
involved and better skilled) Enhance volunteering
Strategic: Improved Social Capital
Developing communities (Relationships, Improved Quality of Life
services and wellbeing)
72. Next steps
• Process of consultation and testing of the
framework
• Incorporate revised framework into
transition year of CDIP
• Assess framework (evaluate)
• Assess capacity of groups and design and
delivery support programme
72
73. Challenge
• The right strategy
• Engaging and supporting community groups and
staff team
• Practicality Test (simple, clear, proportional, transparent,
accountable, verifiable, affordable, etc)
o Design assessment model (criterion based)
o Design PMF (practicality test)
• Communication throughout the process
75. Roundtable discussions
What do you see as the main issues / challenges in relation
to funders and the sector collaborating to develop outcome
measures?
How do you think the Measuring Change approach
contributes to collaboration between funders and the sector
around outcomes?
How does Measuring Change complement other
approaches to collaboration on outcomes measurement
currently available to the sector?
Funders define lots of the incentives – encourage impact measurement and evaluation Their practice also has an impact Impact Summit gathered views of how we might work at the level of funders and their practice/incentives
Funders define lots of the incentives – encourage impact measurement and evaluation Their practice also has an impact Impact Summit gathered views of how we might work at the level of funders and their practice/incentives
Neighbourhood Renewal Strategy launched in June 2003 36 Neighbourhoods identified 102,000 households 280,000 people Range in size from 800 – 22,000 Average size 8,000 (10,000 in Belfast and Derry) Will be consistent with the principles of policy evaluation as outlined in the ‘Magenta Book’ and will seek to determine the extent to which a Neighbourhood Renewal has met or is meeting its objectives and that those intended to benefit have done so. Will use a range of research methods to systematically investigate the effectiveness of Neighbourhood Renewal interventions, implementation and processes, and to determine their merit, worth, or value in terms of improving the social and economic conditions of different stakeholders. Will adopt a summative evaluation (sometimes called impact evaluation) approach which asks questions about the impact of a policy, programme or intervention on specific outcomes and for different groups of people. The review will ask if the goals of the programme are being achieved? Will review and consider the transfer of learning from evaluations of neighbourhood interventions from other regions of the UK. Will consider the implications of the review for policy and practice in this area. Will invite a peer review of the analysis and conclusions drawn. Community - To develop confident communities that are able and committed to improving the quality of life in their areas Economic - To develop economic activity in the most deprived neighbourhoods and connect them to the wider urban economy Social - To improve social conditions for the people who live in the most deprived neighbourhoods through better co-ordinated public services and the creation of safer environments Physical - To help create attractive, safe, sustainable environments in the most deprived areas
THIS IS THE DESIRED END RESULT FOR NEIGHBOURHOOD RENEWAL – HEALTHY, VIBRANT COMMUNITIES WHO ARE PROUD TO LIVE IN THEIR AREA WHO RECEIVE GOOD PUBLIC SERVICES WHERE CHILDREN ARE FREE TO PLAY WHERE PEOPLE FEEL SUFFICIENTLY SKILLED TO TAKE ADVANTAGE OF EMPLOYMENT OPPORTUNITIES OR HAVE THE CONFIDENCE TO PERSUE EDUCTIONAL/VOCATIONAL TRAINING COMMUNITIES WHO ARE OPTIMISTIC ABOUT THEIR FUTURE AND ABOUT SHARING IT WITH OTHERS
Community Renewal – Community Development Economic Renewal – reducing worklessness Social Renewal – Education, Health and Crime Physical Renewal – improving facilities and the physical environment Measured outcomes in line with indicators from the National Indicator Set as recommended by the Department for Communities and Local Government in consultation on a regeneration framework.