Final final collaborating on outcomes 30 5 12


Published on

Published in: Health & Medicine, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Funders define lots of the incentives – encourage impact measurement and evaluation Their practice also has an impact Impact Summit gathered views of how we might work at the level of funders and their practice/incentives
  • Funders define lots of the incentives – encourage impact measurement and evaluation Their practice also has an impact Impact Summit gathered views of how we might work at the level of funders and their practice/incentives
  • Neighbourhood Renewal Strategy launched in June 2003 36 Neighbourhoods identified 102,000 households 280,000 people Range in size from 800 – 22,000 Average size 8,000 (10,000 in Belfast and Derry) Will be consistent with the principles of policy evaluation as outlined in the ‘Magenta Book’ and will seek to determine the extent to which a Neighbourhood Renewal has met or is meeting its objectives and that those intended to benefit have done so. Will use a range of research methods to systematically investigate the effectiveness of Neighbourhood Renewal interventions, implementation and processes, and to determine their merit, worth, or value in terms of improving the social and economic conditions of different stakeholders. Will adopt a summative evaluation (sometimes called impact evaluation) approach which asks questions about the impact of a policy, programme or intervention on specific outcomes and for different groups of people. The review will ask if the goals of the programme are being achieved? Will review and consider the transfer of learning from evaluations of neighbourhood interventions from other regions of the UK. Will consider the implications of the review for policy and practice in this area. Will invite a peer review of the analysis and conclusions drawn. Community - To develop confident communities that are able and committed to improving the quality of life in their areas Economic - To develop economic activity in the most deprived neighbourhoods and connect them to the wider urban economy Social - To improve social conditions for the people who live in the most deprived neighbourhoods through better co-ordinated public services and the creation of safer environments Physical - To help create attractive, safe, sustainable environments in the most deprived areas
  • Community Renewal – Community Development Economic Renewal – reducing worklessness Social Renewal – Education, Health and Crime Physical Renewal – improving facilities and the physical environment Measured outcomes in line with indicators from the National Indicator Set as recommended by the Department for Communities and Local Government in consultation on a regeneration framework.
  • Final final collaborating on outcomes 30 5 12

    1. 1. 30 May 2012 Belfast Castle
    2. 2. Philip McDonagh Chair
    3. 3. Bill OsborneBuilding Change Trust
    4. 4. The Building Change Trust is a 10 year £10 million charitable fund endowed by the Big Lottery Fund to support change and transformation in the Northern Ireland Community and Voluntary sector
    5. 5. Our vision“A strong, vibrant, independent and relevant community and voluntary sector in NI”
    6. 6. CENI - one of the 5 original Trust bid partnersalong with CFNI, RCN, BITC and VNOWCENI - significant work on outcome and impact Measuring Up – identifying needs Making Reporting more Effective Use of Quality Standards Measuring Change approach piloted with BIG, Belfast City Council and Neighbourhood Renewal
    7. 7. Trust actively considering its further and futurerole with respect to impact measurementMay appear an esoteric and abstract issuebut
    8. 8. Our vision of a strong, independent andrelevant sector that changes people and placesfor the better requires that:• More cvs organisations are helped to focus on impact• Cvs organisations understand and use appropriate tools to set out achievements
    9. 9. Context is• Evidence based policy making and outcomes based commissioning• Identification of and investment in what works• Organisations are striving to make best use of limited resources for maximum benefit
    10. 10. Anticipate continuing to work with CENI• Exploring use of impact measurement process with government• Will also be working in partnership with NPC and others to bring benefits of Inspiring Impact to NI
    11. 11. Plan to invest some resources focusing onimpact in order to support change for positivedevelopment in sectorNot something we can do alone and want towork with government and the sector to take thisinitiative forward
    12. 12. Keen to hear your views on what action needs to betaken here.In the first instance contact Trust Administrator, Nigel McKinneyinfo@buildingchangetrust.orgVisit our websitewww.buildingchangetrust.orgTwitter@changetrust
    13. 13. Will HairePermanent Secretary, DSD
    14. 14. Challenges and issues: a UK perspectivefrom the Inspiring Impact programmeTris Lumley, Head of Development, NPCCollaborating on Outcomes: Funders and the SectorWorking Together30th May 2012
    15. 15. What’s the context for impactmeasurement?• Charities and social enterprises under pressure to demonstrate impact • What should they do to improve? • Driven by funders or for own benefit? • What’s proportionate for them? • Working in isolation, reinventing the wheel• Funders may want to help • What should they do to support improvement? • Who should pay to increase charities’ capacity? • How should funders think about their own impact?
    16. 16. And impact’s not just aboutmeasurement…
    17. 17. If impact measurement is driven by funders… …it will probably fail to become embedded and really help charities learn and increase impactWe need structural solutions to structural barriers
    18. 18. Why am I here?• NPC trying to help create structural solutions: • Inspiring Impact—10 year collaborative programme • Focusing on barriers, solutions and incentives• At this conference: • To share a UK perspective on impact • To be part of discussion in NI context
    19. 19. What are the barriers to progress? Providers Funders Commissioners Incentives weak (with few  Incentives highly variable  Incentives skewed (towards penalties for poor impact reporting). (and often inconsistent and poorly outputs and ‘cost to serve’. Focus on communicated) accountability over evidence / learning.) Cultural resistance (at leadership and frontline)  Cultural resistance (to  Cultural resistance (cuts are prioritise impact, to spend money on both a positive and a negative factor) Insufficient resources (money, measurement) expertise and capacity)  Insufficient resources (money  Insufficient resources (money and capacity for them and provider) Technical challenges (don’t and capacity for them and provider) have skills and systems to measure;  Technical challenges (skills lack of consensus on indicators,  Technical challenges (skills and systems to identify high impact methods etc) and systems to identify high impact providers, and collect and synthesise providers, and collect and synthesise evidence) Practice (inability to attribute (or learning and evidence) predict) an organisation’s contribution to  Practice: (inability to use impact outcomes with confidence)  Practice: (challenge of building the data to drive cost reduction / budget evidence base over the long term) reallocation) Cross-cutting barriers 19
    20. 20. What are Inspiring Impact’s solutions?• We’ve identified five key strands of work • Leadership and culture • Coordinating support Charities & social enterprises • Data, tools and systems • Shared measurement • Funder, commissioner & investor practice
    21. 21. Our initial focus for charities andsocial enterprises…
    22. 22. Answers to key questions What does good practice look like? What’s proportionate? How do others do it? How do we stack up?What works in our field? How can we improve? What approach is right for us? Which tools/systems should we use?
    23. 23. Concrete deliverables What does good practice look like? Code of Good Impact Practice What’s proportionate? How do others do it? Impact How do we stack up?Shared measurementWhat works in our field? Diagnostic How can we improve? What approach resources us? Coordinated is right for Marketplace for tools/systems? Which tools/systems should we use?
    24. 24. And for funders themselves…
    25. 25. Capturing different ‘types’ of funderimpact Strategic Strategy for deploying resources to impact achieve impact Practice How behaviour as a funder impacts on impact grantees Funding The impact on beneficiaries and the impact community achieved through funding
    26. 26. Plans with funders will emergedynamically• Working group of foundations • Identify key aspects • Explore potential research, actions • Scope out plans and campaigns for short and medium term
    27. 27. What’s next for Inspiring Impact?
    28. 28. We’re just getting started• Programme launching June 2012• Establishing links/partnerships across UK• Building advisory groups for key projects• Gathering intelligence on existing initiatives• Want to hear from you about things we should be aware of and opportunities to engage…
    29. 29. Get in touch…Tris Lumley, Head of Development, NPCtlumley@philanthropycapital.org020 7620 4883
    30. 30. Questions for Speakers
    31. 31. Brendan McDonnell Director CENICollaborating on Outcomes Introducing ‘Measuring Change’
    32. 32. ContextClimate of austerity and increasing social need - challenge for publicfunders and projects to measure the outcome of their interventions.Concordat and PAC report - need for collaboration between Govt & SectorRecent CENI research shows that: existing systems focus on counting activities and ensuring financial probity - not designed to measure outcomes approaches not standardised so difficult to aggregate diverse project outcomes to show overall programme impact prohibitive cost of comprehensive, robust outcome measurement systems can ‘crowd out’ resources .
    33. 33. Measuring ChangePractical, robust and cost-effective approach to capturing thehard-to-measure qualitative outcomes of community-basedprogrammes. What are the things you want to Change? Outcomes Where are you starting from? Baseline What difference has been made? Change
    34. 34. How it WorksTwo distinct but connected components:• Developing an overarching framework of programme-level outcomes• Applying an innovative data collection method to measure project baseline and change against these outcomes.Underpinned by involvement of both programme and projectstakeholders in a collaborative process facilitated by CENI.
    35. 35. Develop Outcomes FrameworkKey Elements Strategic Focus: Facilitated sessions with Funder to rationalise a ‘theory of change’ for the programme and articulate specific change outcomes. Structure/Themes: Identify key themes which give shape and structure to an outcomes framework - common format to locate diverse project-level outcomes. Stakeholder Involvement: Process informed by input from both projects and programme funder - fosters shared ownership of the outcomes framework.
    36. 36. Sample Programme Outcomes FrameworkTheme Change OutcomesPeople • Enhanced Individual Capacity • Educational Development • Better Healthy lifestyle Choices • Positive Mental Health • Improved Family Cohesion • Improved Personal safetyCommunity • Better Engagement with Hard to Reach Groups • More Active Involvement in the Community • Enhanced Volunteering • Improved Access to Community FacilitiesOrganisation • Better Partnership working • More Strategic influence • Enhanced Practice developmentRelationship • Better Bonding • More Bridging • Improved Linking
    37. 37. Data Collection: ‘Nominal Group Technique’ Multiple Perspectives: involves stakeholders (partners, management, staff, volunteers, users) in a facilitated ‘Expert Panel’ - maximises involvement and enables ‘triangulation’ of evidence. Facilitation: CENI evaluator as a ‘critical friend’ ensures equality of input across all stakeholders, challenging and testing their evidence. Measurement Scale: uses the Rickter numerical scale (0 - 10) to rate project baseline position and change. Scale helps neutralise contentious debate and reach consensus. Critical Debate: facilitates candid discussion and critical peer review - challenges stakeholders to prioritise what outcomes are important and be realistic about their contribution to change.
    38. 38. Putting a Metric on Qualitative Change Low High 0 1 2 3 4 5 6 7 8 9 10• Baseline – panels rate their project on each outcome at the start• Change – panels rate their project’s progress on each outcome (after a set time)Panel discussion is recorded to give context and rationale for the scoresProjects can also rate importance and difficulty of achievingoutcomes
    39. 39. Data AnalysisAn effective way of extracting large amounts of data from diverseperspectives within a project and distilling this into an agreed estimate.Applied systematically across multiple projects it enables the generationof robust evidence in a standardised format.This can be analysed:Horizontally: to baseline and measure the progress of individual projects against the programme outcomesVertically: to indicate overall programme impact by aggregating data from individual projects
    40. 40. Baseline and Change Scores for Funded Projects 10.0 9.0 8.0 7.0 6.0 Baseline 5.0 Change 4.0 3.0 2.0 1.0 0.0 A B C D E F G H I J Projects
    41. 41. Case StudiesMeasuring Change has been successfully piloted by CENI across anumber of funding programmes including: Big Lottery Fund - support grant holders to baseline and measure change. Neighbourhood Renewal Programme - identify and baseline Community Renewal outcomes. Belfast City Council - inform review of strategic outcomes and help support change management.
    42. 42. Conclusion Still a work in progress – ‘Another Tool in the Kit’ Successfully piloted – case studies Flexible – readily transferable to variety of situations (Age NI, IFA) Provides mechanism to aid collaboration in both identifying and measuring outcomes Contribute to wider body of knowledge on evaluation and outcomes measurement Shift from Counting Activities to Measuring Change
    43. 43. Measuring Change Case Studies Big Lottery Fund Norrie Breslin Neighbourhood Renewal Alison Chambers Belfast City Council Catherine Taggart
    44. 44. Supporting grant holders tobaseline and measure changeNorrie Breslin,Head of Policy and LearningBig Lottery Fund NI
    45. 45. Mission and ValuesOur Mission• Bringing improvements to communities and the lives of people most in need.Our Values  Making best use of lottery money  Using Knowledge and evidence  Being supportive and helpful
    46. 46. Funding approachStrategic programmes Outcomes funder Partnerships Working across outcomes 2 stage application process Development grants Self evaluation 5 year projects
    47. 47. ProgrammesLive and Learn 17 Projects, totalling over £16.2million Learning and well beingSafe and Well 18 Projects, totalling nearly £17million Safety and well being Wide range of projects and beneficiaries Across geographic and specific communities
    48. 48. The Issue Varied response to self evaluation Support for grant holders Demonstrate impact Move from programme specific evaluation
    49. 49. The Journey Change matrix Communicating to grant holders Communicating to staff Encouraging participation
    50. 50. The Outcome Participation 27/35 Impact on grant holders Involving wider staff team NI Committee BIG UK wide Next steps
    51. 51. Alison ChambersNeighbourhood Renewal Identify and BaselineCommunity Renewal Outcomes
    52. 52. Neighbourhood Renewal Strategy launched 2003 36 Neighbourhoods Over 102,000 households - around 280,000 people Neighbourhood Partnerships established in each area Representative of public/private/political and voluntary and community sectors Neighbourhood Action Plans developed detailing the priority issues to be addressed in each area
    53. 53. NEIGHBOURHOOD RENEWAL –STRATEGIC OBJECTIVES• Community Renewal developing confident communities• Economic Renewal developing economic activity• Social Renewal improving social conditions• Physical Renewal creating attractive, safe environments
    54. 54. Key Outcome Areas Community Development Worklessness Demography Education Health Crime Physical
    55. 55. Community Renewal Mid Term Review findings Lack of Definition and Baseline Position Inputs, activities, outputs no outcomes
    56. 56. Community Renewal CENI – Social Assets Research Outcomes Framework Measuring Change
    57. 57. Community Renewal 14 outcomes – 5 themes 3 Areas Expert Panels
    58. 58. Community Renewal Programme Evaluation Profile Areas Next Steps
    59. 59. Capacity Grant: reviewing outcomes Cate Taggart Community Development Manager
    60. 60. Context• BCC Investment programme• Economic Environment• BCC Community Development Framework• RPA – planned transfer of Regeneration function• Pilot – single CD fund for city
    61. 61. Community Grant Programme• BCC more than just a grant administrator: both financial and officer support• Programme across 6 grant categories: - Advice & Information - Capacity Support - Revenue Support for Community Buildings - C&YP: summer schemes and Ur City 2 - Project grants - Community Chest• £2.6m across 412 grants last year
    62. 62. Service grant programmeSmall grants: community development projects Grants for Summer schemes appropriate for children & young people aged 5-14 years Community chest grant (in support of, for example, small scale seasonal activity)Large grants:• Advice services: core funding to cover revenue and other operating costs; supplementary funding (with DSD) for advice outreach costs• Revenue (financial support towards running costs of community buildings)• Capacity building (grant funding to support the building of skills and knowledge of community organisations and local people).
    63. 63. Capacity Building Grant recipientswith Neighbourhood Renewal areas
    64. 64. Community Development FrameworkCommunity development activity is the main means by which we canbe better engaged with local people and support their involvement inimproving the city and its neighbourhoods. It enables people to cometogether to:• define needs, issues and solutions for their community; and• influence or take decisions about issues that matter to them and that affect their lives;• take action to help themselves and make a difference. It is a long-term, value-based process which targets positive social change.’
    65. 65. Community Development Strategy Align our resources to achieve new vision and strategy• Core Community Development work• Engagement that works• Building Effective Partnerships• Shared Service Design & Delivery 66
    66. 66. CENI pilot (& beyond .........)Capacity Building Grant elected to pilot the application of theMeasuring Change model within a funding programmedesigned to build the assets of local communitiesAmbition to review and agree the strategic outcomes of theprogramme in light of the CD framework andTo develop an outcomes framework based on Social AssetsTo inform the design of related assessment criteria (for afuture version of the programme)To inform the design of a performance managementframework (compliance and change)
    67. 67. Capacity Review1. Link supported grant activity to identified programme outcomes2. Produce Standard Framework of Programme Outcomes;3. Apply Standard Measurement Tool to: • Baseline projects against programme outcomes • Evidence individual contribution to programme • Enable comparison or relative change across projects, themes • Aggregate to demonstrate programme impact4. Support Projects to Use Framework and Measurement Tool 68
    68. 68. Why ................Funder Funded organisationDemonstrate the services we Demonstrate potential to funders,commission are having a positive board, user groups, recipients .....change on individuals and society andtherefore strengthen the case forrenewed funding.Monitor performance / compliance Know project / service is on trackIdentify and disseminate practice Contribute to an evidence base oflearning practiceCommunicate impact of investment Raise profile (communications)Motivate staff, elected members, ... Motivate staff and volunteers 69
    69. 69. How...............• CENI trawl of current applications to list outcomes as stated by funded groups• Group these outcomes into common themes, using the original SA framework to help shape the approach• Staff consultation (NGT): clarify purpose and focus and agree priorities• Iterative process to produce refined strategic aim and draft outcomes framework
    70. 70. What ............Theme (4) Desired Outcome (11)Core: Enhance Capacity of funded Support GroupDeveloping Support Group core Stronger Partnershipcompetency Increased leverageOperational: Improved local infrastructure of groupsDeveloping Local Groups (to be capable, Improved group capacityrepresentative and resilient) Increased group resilienceOperational: Increased individual participationDeveloping Individuals (to be more Improved individual skillsinvolved and better skilled) Enhance volunteeringStrategic: Improved Social CapitalDeveloping communities (Relationships, Improved Quality of Lifeservices and wellbeing)
    71. 71. Next steps• Process of consultation and testing of the framework• Incorporate revised framework into transition year of CDIP• Assess framework (evaluate)• Assess capacity of groups and design and delivery support programme 72
    72. 72. Challenge• The right strategy• Engaging and supporting community groups and staff team• Practicality Test (simple, clear, proportional, transparent, accountable, verifiable, affordable, etc) o Design assessment model (criterion based) o Design PMF (practicality test)• Communication throughout the process
    73. 73. THANKS
    74. 74. Roundtable discussions What do you see as the main issues / challenges in relation to funders and the sector collaborating to develop outcome measures? How do you think the Measuring Change approach contributes to collaboration between funders and the sector around outcomes? How does Measuring Change complement other approaches to collaboration on outcomes measurement currently available to the sector?
    75. 75. Panel Discussion/Questions Moving on: Next Steps