Methodologies and approaches to evaluation in third sector organisations / Jean Ellis


Published on

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Good for users: An intervention in itself, Keep staff focused on outcomes, Opportunity to shape services Respond to external pressures: Charity commission (Hallmarks of an effective charity’, SORP, Summary Information Return) , Funding lever, General drive to accountability, Payment by results Good for staff: Feedback on performance, Involvement in assessment and shaping services, Can challenge/ confirm views, Role clarity, Save time in long run Make your organisation better: Better information systems, Effective use of resources, Demonstrates results, Transparency , Responsive to needs Tension between accountability and learning. Jean’s research 2008 found that much monitoring and self-evaluation done for accountability. Accountability paradox, from accountability and learning: ‘ The collection of data for compliance purposes alone, even with a focus on benefits for users , may not in itself provide a means of improving value.’ ‘ Paradoxically, in the context of an overall effectiveness agenda, the external reporting focus limits the use of findings for improved management and wider learning – to increase effectiveness.’
  • Some of you may have seen before – this is a planning tool – won’t have time to do today
  • 2.50 Mention hard and soft outcomes Outcome areas Example outcomes Organisational strength More secure funding; Improved management structures Service provision Greater lobbying voice; New organisations established to deliver services Policy Change in funding policy; Change in national policy Sharing good practice More joint ventures between member organisations
  • The Age UK befriending scheme
  • Impact arts wanted to set up this new type of project. Fab pad offers arts, design and practical skills training to vulnerable young people, when they get a new tenancy, to help them turn their house into a home. Their funders considered it risky, and didn’t quite understand the role of arts in social change Impact wanted to demonstrate the different made in terms their funders would understand As a result of their SROI, Fab Pad have gained follow-on funding support for the programme, developed a higher profile with their main partners and funders, and have begun applying SROI to other aspects of their business as a strategic management tool. Are planning to do SROIs on their different Fab Pad projects and compare them Remember the ratio doesn’t represent an actual social return; it is a sum of how much all the stakeholders value the service outcomes What we need is a standard approach to measuring and combining proxies that have been calculated differently, rather than measure everything in same way
  • Reporting and dissemination should be part of the plan from the outset – to increased engagement and utilisation Know your audience: Before you start your evaluation, answer the following questions: Who are your primary evaluation users? What do they need to know? What do they not need to know? When do they need to know it? Why do they need to know it? What methods will they think are credible?   What does this mean for your evaluation? Range of methods to capture different audiences Analysis continuum: Raw data: Familiarise, edit, filter Description: Develop a framework, code and describe Interpretation: What do the findings mean? Recommendations: What are the practical consequences? Findings should answer Evaluation Qs – use these to keep you focused. Recommendations must link to findings – common mistake
  • Here are some challenges we can discuss at the end One that might be worrying you is funding ie, cost of external vs self-evaluation How about also better understanding needed or the role of evaluation in organisational learning, not just accountability Also an issue of reconciling and choosing between different tools and approaches
  • Methodologies and approaches to evaluation in third sector organisations / Jean Ellis

    1. 1. Methodologies and approaches to evaluation in the third sector International conference on evaluation, Barcelona 3 November 2011 Dr Jean Ellis Charities Evaluation Services
    2. 2. The policy waves <ul><li>Early 1990s – more business-like sector and efficiency focus </li></ul><ul><li>Mid 1990s – focus on results and effectiveness </li></ul><ul><li>From 2004 – emphasis on performance and value for money </li></ul><ul><li>Current situation – emphasis on savings and social value </li></ul>
    3. 3. Some definitions Monitoring The routine, systematic collection of information for the purpose of checking progress against project plans    Evaluation Using monitoring and other information to make judgements and give value
    4. 4. Evaluation <ul><li>Evaluation aims to answer agreed questions and to make a judgement against specific criteria. Like other research, for a good evaluation, data must be collected and analysed systematically, and its interpretation considered carefully . </li></ul><ul><ul><ul><ul><ul><li>CES (2009) Practical Monitoring and Evaluation </li></ul></ul></ul></ul></ul>
    5. 5. The first policy wave <ul><li>Early 1990s – drive for a more business-like sector and efficiency focus </li></ul>
    6. 6. The purposes of evaluation <ul><li>Evaluation for accountability – to demonstrate achievement and compliance </li></ul><ul><li>Evaluation for learning – to share findings, improve, replicate and innovate </li></ul>
    7. 7. <ul><li>Monitoring and evaluation: </li></ul><ul><li>is good for you </li></ul><ul><li>is good for users </li></ul><ul><li>meets external demands </li></ul><ul><li>can make your organisation better </li></ul>Why do it?
    8. 8. Steps to self-evaluation 1- Clarify aims and outcomes, objectives and outputs 2- Set performance indicators; choose tools 5-Use findings 3- Collect information 4- Analyse and interpret FIVE STEP PLAN
    9. 9. CES planning triangle © Overall aim Objectives This is what we do Specific aims This is why we do it Outputs Outcomes
    10. 10. The Family Centre To provide opportunities for play, drama and dancing To provide workshops, information and advice on parenting To organise outings for children and their parents To improve the lives of the children who use the centre To increase children’s self-esteem To enable parents to give each other practical support Objectives Specific aims Overall aim To run a drop-in centre with a play area To improve the parenting knowledge and skills of the parents who use the centre
    11. 11. Second policy wave <ul><li>Mid-1990s – focus on results and effectiveness </li></ul>
    12. 12. More definitions Outputs are an organisation’s detailed activities, services and products – what you do Outcomes are all the changes, benefits, learning or other effects that happen as a result of an organisation’s activities – what you change
    13. 13. Aims, outcomes and indicators <ul><li>Aims – the broad changes you plan to achieve </li></ul><ul><li> </li></ul><ul><li>Outcomes – the specific changes for individuals, families, communities, organisations and policy </li></ul><ul><li> </li></ul><ul><li>Indicators – well-defined and measurable information that will show change </li></ul>
    14. 14. Intermediate outcomes There are often smaller changes that need to happen before the final, desired outcome can be reached. These steps along the way are known as intermediate outcomes. Families engage with befriending service  Greater awareness within the family of later life conditions  Families feel supported, less isolated  Increased communication and coping skills  Improved family relationships and dynamics
    15. 15. The Outcomes Star © Triangle Consulting 2011
    16. 16. Underpinned by a model of change © Triangle Consulting 2011
    17. 17. Third policy wave <ul><li>From 2004 – emphasis on performance and value for money </li></ul>
    18. 18. Drowning in data <ul><li>Too much time collecting and reporting information </li></ul><ul><li>Data not always analysed or used </li></ul><ul><li>Costly in time and resources </li></ul>But monitoring and evaluation IT systems could help greatly.
    19. 19. Current policy wave <ul><li>The current situation – emphasis on savings to the public purse and social value . </li></ul>
    20. 20. Using Social Return on Investment <ul><li>Fab Pad supports young homeless people to sustain new tenancies. The SROI evaluation carried out on Fab Pad revealed that for every £1 invested by the government in support, £8.38 of social return was derived in reduced health care costs, reduced welfare benefits expenditure and reduced costs of repeat homelessness. </li></ul>
    21. 21. Evaluation for learning <ul><li>What worked? </li></ul><ul><li>What parts worked? </li></ul><ul><li>Who did they work for? </li></ul><ul><li>In what contexts did they work? </li></ul><ul><li>What else happened? </li></ul><ul><li>Were there any negative consequences? </li></ul><ul><li>Did the results last? </li></ul>
    22. 22. Accountability or Learning? <ul><li>Closed enquiry </li></ul><ul><li>Proving success </li></ul><ul><li>Showing cause and effect </li></ul><ul><li>Used for reporting and marketing </li></ul><ul><li>Open enquiry </li></ul><ul><li>Valuing failure as well </li></ul><ul><li>Understanding context </li></ul><ul><li>Used for improvement and change. </li></ul>
    23. 23. Theory of change <ul><li>It makes clear all the assumptions about how change will happen. </li></ul>Activities Short-term change Steps along the way Longer-term change
    24. 24. Challenges and opportunities Managing data Getting skilled Finding meaning Methods Innovation Using findings New technology
    25. 25. A winning charity <ul><li>The Leeds Survivor Led Crisis Service: </li></ul><ul><li>We are passionate about monitoring and evaluation as we believe it is the absolute right of our visitors and callers to know that we scrutinise our work in order to ensure that we are providing an excellent service which meets their needs. </li></ul>
    26. 26. <ul><li>CES contacts: </li></ul><ul><li> </li></ul><ul><li>020 7713 5722 </li></ul><ul><li>[email_address] </li></ul><ul><li>Jean Ellis : </li></ul><ul><li>[email_address] </li></ul><ul><li>02070789386 </li></ul>