Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluation Workshop

363 views

Published on

Evaluation Workshop

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Evaluation Workshop

  1. 1. Using evaluation to measure impact Jean Taylor, Strategy and Development Manager, Lambeth Council
  2. 2. Session coverage and learning objective This session will: • introduce evaluation and its benefits; • offer guidance on how to choose the right evaluation approach; • describe the main stages of evaluation; and, • key issues you need to consider at each of these. Learning objective: • Delegates will gain an understanding of when to use evaluation, how to choose an appropriate evaluation approach and key issues they should consider when undertaking an evaluation
  3. 3. What is evaluation? ‘Evaluation is an objective process of understanding how a policy or other intervention was implemented, what effects it had, for whom, how and why’. The Magenta Book: Guidance for Evaluation, The Cabinet Office (2011)
  4. 4. What are the benefits of evaluation? • Development/improvement of practice • Support reinvestment in future activity • Help save money in future (by helping people focus on what works) • Provides accountability: how funding has been spent, what has been achieved, assessing return on investment
  5. 5. Where evaluation occurs in policy cycle Rationale Objectives Appraisal and planning Monitoring Evaluation Feedback
  6. 6. Factors influencing evaluation approach • The nature of the policy, the scale of its objectives, complexity, innovation, form of implementation and future direction; • The objectives of the evaluation and the types of questions it would ideally answer; • The timing of key policy decisions and the information on which they need to be based; • The types of impact which are expected, on what timescales, and the availability of information and data on them; • The time and resources available for the evaluation. Adapted from the Magenta Book
  7. 7. Three main types of evaluation • Process: whether policy being implemented as intended, what works or not • Impact: objective test of the changes that have occurred and the extent these are attributable to the policy • Economic: compare benefits of the policy with the costs
  8. 8. Impact evaluation • What were the policy outcomes, what change was observed and how much, how much could be said to have been caused by the policy as opposed to other factors? • Did the policy achieve its stated objectives? • How did any changes vary across different individuals, stakeholders, groups etc and how did this compare with what was anticipated? • Did any outcomes occur which were not originally intended and how significant were they?
  9. 9. Types of data • Existing data – Monitoring data/performance data – Service level surveys – Spontaneous feedback (e.g. via social media) • New data – Surveys (e.g. to measure scale of change) – In-depth interviews (e.g. to understand impacts and factors underpinning impact) – Focus groups – Case studies (e.g. to understand impact in context) – Other modes: visual evidence
  10. 10. Main stages/components of evaluation 1. Defining the policy objectives and intended outcomes 2. Defining the audience for evaluation 3. Identifying the evaluation objectives and research questions 4. Selecting the evaluation approach 5. Identifying the data requirements 6. Identifying the necessary resources and governance arrangements 7. Conducting the evaluation 8. Using and disseminating the evaluation findings Source: The Magenta Book, 2011
  11. 11. Main stages/components of evaluation: NPC’s four pillar approach 1. Map your theory of change – Provides clarity, reveals causal links, offers coherent framework 2. Prioritise what you measure – What is most important (not what is most convenient) 3. Choose your level of evidence – Depends on your needs, capabilities, resources (and those of your audience) 4. Select your sources and tools – Existing tools or data sources, bespoke ones
  12. 12. Logic and theory of change models • Way of thinking through: – the issue you are trying to address – the change you want to achieve – the steps you need to take to get there • Not a new way of thinking – It is a way of formalising your thinking • Distinction between logic model and ToC = latter surfaces assumptions about interdependencies
  13. 13. Key components • Range of different approaches to ToC but all include following components: – Definition of final outcome/goal(s) – ‘Backwards mapping’ – mapping of intermediate outcomes that need to be achieved on way to final outcome – Definition of activities that will make change happen • In contrast to logic models, theory of change surfaces assumptions about what will make change happen
  14. 14. Example Theory of Change: A Charities Evaluation Service Planning Triangle© for a supported housing project
  15. 15. Example of Theory of change: logic model Source: NPC
  16. 16. PRACTICE SESSION
  17. 17. Deciding your approach: example projects • Read the brief project description and then as a group develop a proposed evaluation approach. Please answer these questions: – What is the evaluation for and who are the audience? – What are the key outcomes to be measured? – What should the evaluation focus on (priority measures)? – What level/quality of evidence is needed? – What sources of information and data will you draw on and what tools will you use for data collection? – When you would undertake the data collection? – What other information would you ideally have about these projects?
  18. 18. Further resources The Magenta Book: Guidance on Evaluation, The Cabinet Office (2011) https://www.gov.uk/government/uploads/system/uploads/attachment_d ata/file/220542/magenta_book_combined.pdf Building Your Measurement Framework: NPC’s Four Pillar Approach (2014) http://www.thinknpc.org/publications/npcs-four-pillar-approach/ Creating your theory of change: NPC’s practical guide (2014) http://www.thinknpc.org/publications/creating-your-theory-of-change/
  19. 19. Check out the Insight Hub What we know about trends in our services What we know about who we support What we know about our customer service transactions What we know about issues reported to us How Lambeth compares What are the unit costs & benefits of services http://bit.ly/insighthublambeth http://bit.ly/insighthublambethgov
  20. 20. Come to our next sessions Sign up at http://bit.ly/localinsightprog Theme Date Calculating social impact 17 May Community Action Network Understanding what matters to local residents 24 May Lambeth Understanding the strengths & needs of local communities 1 June Young Foundation Understanding local demographics 7 June LGA Using research to inform policy 14 June Newham Council

×