Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Methodologies and approaches to evaluation in third sector organisations / Jean Ellis


Published on

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

Methodologies and approaches to evaluation in third sector organisations / Jean Ellis

  1. 1. Methodologies and approaches to evaluation in the third sector International conference on evaluation, Barcelona 3 November 2011 Dr Jean Ellis Charities Evaluation Services
  2. 2. The policy waves <ul><li>Early 1990s – more business-like sector and efficiency focus </li></ul><ul><li>Mid 1990s – focus on results and effectiveness </li></ul><ul><li>From 2004 – emphasis on performance and value for money </li></ul><ul><li>Current situation – emphasis on savings and social value </li></ul>
  3. 3. Some definitions Monitoring The routine, systematic collection of information for the purpose of checking progress against project plans    Evaluation Using monitoring and other information to make judgements and give value
  4. 4. Evaluation <ul><li>Evaluation aims to answer agreed questions and to make a judgement against specific criteria. Like other research, for a good evaluation, data must be collected and analysed systematically, and its interpretation considered carefully . </li></ul><ul><ul><ul><ul><ul><li>CES (2009) Practical Monitoring and Evaluation </li></ul></ul></ul></ul></ul>
  5. 5. The first policy wave <ul><li>Early 1990s – drive for a more business-like sector and efficiency focus </li></ul>
  6. 6. The purposes of evaluation <ul><li>Evaluation for accountability – to demonstrate achievement and compliance </li></ul><ul><li>Evaluation for learning – to share findings, improve, replicate and innovate </li></ul>
  7. 7. <ul><li>Monitoring and evaluation: </li></ul><ul><li>is good for you </li></ul><ul><li>is good for users </li></ul><ul><li>meets external demands </li></ul><ul><li>can make your organisation better </li></ul>Why do it?
  8. 8. Steps to self-evaluation 1- Clarify aims and outcomes, objectives and outputs 2- Set performance indicators; choose tools 5-Use findings 3- Collect information 4- Analyse and interpret FIVE STEP PLAN
  9. 9. CES planning triangle © Overall aim Objectives This is what we do Specific aims This is why we do it Outputs Outcomes
  10. 10. The Family Centre To provide opportunities for play, drama and dancing To provide workshops, information and advice on parenting To organise outings for children and their parents To improve the lives of the children who use the centre To increase children’s self-esteem To enable parents to give each other practical support Objectives Specific aims Overall aim To run a drop-in centre with a play area To improve the parenting knowledge and skills of the parents who use the centre
  11. 11. Second policy wave <ul><li>Mid-1990s – focus on results and effectiveness </li></ul>
  12. 12. More definitions Outputs are an organisation’s detailed activities, services and products – what you do Outcomes are all the changes, benefits, learning or other effects that happen as a result of an organisation’s activities – what you change
  13. 13. Aims, outcomes and indicators <ul><li>Aims – the broad changes you plan to achieve </li></ul><ul><li> </li></ul><ul><li>Outcomes – the specific changes for individuals, families, communities, organisations and policy </li></ul><ul><li> </li></ul><ul><li>Indicators – well-defined and measurable information that will show change </li></ul>
  14. 14. Intermediate outcomes There are often smaller changes that need to happen before the final, desired outcome can be reached. These steps along the way are known as intermediate outcomes. Families engage with befriending service  Greater awareness within the family of later life conditions  Families feel supported, less isolated  Increased communication and coping skills  Improved family relationships and dynamics
  15. 15. The Outcomes Star © Triangle Consulting 2011
  16. 16. Underpinned by a model of change © Triangle Consulting 2011
  17. 17. Third policy wave <ul><li>From 2004 – emphasis on performance and value for money </li></ul>
  18. 18. Drowning in data <ul><li>Too much time collecting and reporting information </li></ul><ul><li>Data not always analysed or used </li></ul><ul><li>Costly in time and resources </li></ul>But monitoring and evaluation IT systems could help greatly.
  19. 19. Current policy wave <ul><li>The current situation – emphasis on savings to the public purse and social value . </li></ul>
  20. 20. Using Social Return on Investment <ul><li>Fab Pad supports young homeless people to sustain new tenancies. The SROI evaluation carried out on Fab Pad revealed that for every £1 invested by the government in support, £8.38 of social return was derived in reduced health care costs, reduced welfare benefits expenditure and reduced costs of repeat homelessness. </li></ul>
  21. 21. Evaluation for learning <ul><li>What worked? </li></ul><ul><li>What parts worked? </li></ul><ul><li>Who did they work for? </li></ul><ul><li>In what contexts did they work? </li></ul><ul><li>What else happened? </li></ul><ul><li>Were there any negative consequences? </li></ul><ul><li>Did the results last? </li></ul>
  22. 22. Accountability or Learning? <ul><li>Closed enquiry </li></ul><ul><li>Proving success </li></ul><ul><li>Showing cause and effect </li></ul><ul><li>Used for reporting and marketing </li></ul><ul><li>Open enquiry </li></ul><ul><li>Valuing failure as well </li></ul><ul><li>Understanding context </li></ul><ul><li>Used for improvement and change. </li></ul>
  23. 23. Theory of change <ul><li>It makes clear all the assumptions about how change will happen. </li></ul>Activities Short-term change Steps along the way Longer-term change
  24. 24. Challenges and opportunities Managing data Getting skilled Finding meaning Methods Innovation Using findings New technology
  25. 25. A winning charity <ul><li>The Leeds Survivor Led Crisis Service: </li></ul><ul><li>We are passionate about monitoring and evaluation as we believe it is the absolute right of our visitors and callers to know that we scrutinise our work in order to ensure that we are providing an excellent service which meets their needs. </li></ul>
  26. 26. <ul><li>CES contacts: </li></ul><ul><li> </li></ul><ul><li>020 7713 5722 </li></ul><ul><li>[email_address] </li></ul><ul><li>Jean Ellis : </li></ul><ul><li>[email_address] </li></ul><ul><li>02070789386 </li></ul>