Download to read offline
Uploaded byRobin Beveridge
Presentation4 feb 15
This document outlines an agenda and content for a workshop on monitoring and evaluation techniques. The all-day workshop covers key concepts, logic chains, quantitative and qualitative data collection methods, quantitative evaluation techniques like sampling and extrapolation, using case studies and qualitative stories, and calculating value for money. Participants will work in groups to develop logic chains for their own projects, discuss best practices for gathering beneficiary feedback, and begin drafting an evaluation plan. The goal is to provide practical skills and strategies for organizations to effectively demonstrate their impact.
Presentation4 feb 15
- 1. Proving Your Impact: monitoring& evaluation techniques 4th February 2015 Robin Beveridge and Tony Sacco
- 2. • Domestics • Who’swho • What’s in store today • Ground rules • Networking and learning from each other Introduction
- 3. Structure of theday: AM 9.30 Introductions and warm up 9.45 Key concepts and logic chains Break 10.45 Capturing information from beneficiaries – what works? 11.30 Lunch Quantitative evaluation techniques
- 4. Structure of theday: PM 14.10 15.00 Workshop feedback and close Evaluation in practice Wowing the world 13.00 15:30 ‘Telling the story’ using case studies 13.30 Value for money
- 5. Getting started • Overthe next 5 minutes… Write down 3 things you hope to get out of today on ‘post-its’ Speak with someone you have never met before today Introduce yourselves and swap one of your ‘hopes’ Repeat this with two more people • Then… …post your ‘post-its’ up on the wall.
- 6. Setting the groundrules…
- 7. Setting the groundrules…
- 8. Key Concepts • Monitoring •Evaluation • Outputs • Outcomes • Impact • Aims & Objectives • Quantitative • Qualitative • Beneficiaries • Stakeholders • Sample/Sampling • Extrapolation • Value for Money • Additionality • Social Return • Gross and Net
- 9. Logic chain example1: This workshop Caused by Output Need Activity Impact Outcome Limited ability to demonstrate impact Workshops and peer learning People trainedHigher level of skills and knowledge Better demonstration of impact Lack of Monitoring and evaluation knowledge Rationale
- 10. Logic chain example2: Streetwise Caused by Output Need Activity Impact Outcome High level of teenage pregnancy Providing C-cards Young people accessing condoms Less un-protected sex Lower level of teenage pregnancy Poor access to contraceptives
- 11. Working in pairs •Using templates provided, develop a ‘Logic Chain’ for a project you are involved with • Share that with someone on your table • Feed back to the workshop: – What was easy, what was hard? – Does it make sense?
- 12.
- 13. Capturing information frombeneficiaries – What works? • In groups consider… –What are the different ways of capturing information from participants / beneficiaries? –Identify some ‘pros and cons’ of each approach.
- 14. Capturing information frombeneficiaries – Some thoughts Questionnaires • Wide coverage • Tick box / narrative approach? • Accessibility • Self-completed or ‘managed’ • ‘Survey Monkey’ • Before / after / distance travelled • Confidentiality • Response rate
- 15. Capturing information frombeneficiaries – Some thoughts
- 17. Capturing information frombeneficiaries – some thoughts Feedback forms • Immediate feedback from ‘captive audience’? • More of / less of • Most useful / least useful • Improvements • Can be superficial and ‘too immediate’ • Follow up again later?
- 18. Capturing information frombeneficiaries – some thoughts In-depth interview • Richer content • Particular insights • Range of views / perspectives • Resource intensive
- 19. Capturing information frombeneficiaries – some thoughts Focus Groups • A group with similar characteristics (e.g. single parents) • ‘Managing’ the group • Recording the outcomes Round Table • Different perspectives
- 20. Capturing information frombeneficiaries – some thoughts Comments / concerns / compliments • Forms / Box? • Managing the process • The feedback loop
- 23. Timing issues… • Captiveaudience • Getting buy-in from the outset • ‘When they’re gone they’re gone!’ • Bias? • Incentives?
- 25. Storage issues… • Paperbased • Spreadsheets • Databases • Bespoke approaches?
- 26. Questionnaire Design Considerations •Questionnaire – Introduction / opening questions – Meaningful order / clear sections – Thank you / return information • Questions – Numbered – Clear instructions – Routing – Layout • Testing – Peer review; Cognitive interview; – Focus group; pilot/field test
- 27. Don’ts… • Try toavoid…
- 28. …and do’s • Tryto avoid…
- 29. What more creativeways can you come up with to gather information from beneficiaries? Outside the box…
- 30.
- 31. • Why numbersmatter • What numbers do we need? • How to get useful numbers • Cross-tabulation • Sampling and extrapolating • How to present data Quantitative techniques 0 10 20 30 40 50 % Reaction to the word “Spreadsheet” Men Women n.b. fictional data!
- 32. Key numbers The projecthas achieved significant positive outcomes, with 447 young people supported to become enterprise ready up to December 2011. To date, roughly 1 in 4 of these beneficiaries have gone on to establish a business, and around 30% have moved into education, training or employment.” “An independent evaluation found that the service costed £145,000 over three years, and had resulted in significant cost savings to statutory services: conservatively estimated at £1,211,630.” • How many beneficiaries? • Of what sort? • What happened as a result? • What did it cost? • How much did it save? • Was it good value for money?
- 33.
- 34. • Filters • Count •Pivot Tables • Cross Tabulations Getting the numbers: 1) Excel % of beneficiaries Under 25 26-50 Over 50 Total Northumberland 0 5 15 20 Tyne & Wear 10 20 10 40 County Durham 15 10 0 25 Tees Valley 0 15 0 15 TOTAL 25 50 25 100 Count of Completed Column Labels Row Labels Jan Feb Mar Apr May Jun Jul Aug Sep Oct Grand Total f 12 3 3 6 6 3 3 6 9 51 m 3 3 6 9 6 6 3 3 39 Grand Total 12 6 3 9 6 15 9 9 9 12 90
- 35. CDCF/ESF Data –Radar Charts 0 1 2 3 4 5 6 7 8 9 10 Motivation Confidence Forms CV Interview IT Average Before After 0 1 2 3 4 5 6 7 8 9 10 Motivation Confidence Forms CV Interview IT Adriana Before After
- 36. Getting the numbers:2) Other • Survey Monkey • Star Online • Databases: – Access – Off the shelf systems – Bespoke design
- 37. Sampling and extrapolation •Most evaluation is based on a sample • How representative is your sample? • How diverse is your ‘population’? • Evaluation is NOT a science • Be aware of potential ‘bias’ in the sample • Use caveats in extrapolating
- 38. Sampling example 0 20 40 60 80 100 120 1 23 4-6 7-12 12+ Months from completion Length of time to find work We found that 100% of respondents had found work within 3 months of completing the course. If this is true for all beneficiaries, we will have helped 300 people to find work”
- 39. Sampling thoughts Try toget at least 10% of each part of the population: • By geography • By gender • By age • By time/stage • Other characteristics… To address bias in your sample: • Use a variety of methods • Ask searching/critical questions • Triangulate with stakeholder views • Pay attention to outlier views
- 40. Presenting Data –some tips • Not too much information • Order bars from high to low • Use a variety of formats, but not too many • Keep a consistent colour scheme • Remember titles, axes and legends • Keep it as clean as possible
- 41. For example 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Ethicity 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% White: BritishWhite: Other Other Ethnicity Ethnicity Other Ethnicity Mixed: White and Black Caribbean Mixed: White and Black African Mixed: White and Asian Mixed: Other Mixed Asian or Asian British: Indian Asian or Asian British: Pakistani Asian or Asian British: Bangladeshi Asian or Asian British: Other Asian Black or Black British: Black Caribbean Black or Black British: Black African Black or Black British: Other Black Chinese or Other Ethnic Group: Chinese Chinese or Other Ethnic Group: Other
- 42. Presenting Data –some tips Use free tools, eg: • Google Fusion Tables • Tableau public • Plot.ly • Any others?
- 43. That’s quite enoughabout data
- 44.
- 45.
- 46. Structure of theday: PM 14.10 15.00 Workshop feedback and close Evaluation in practice Wowing the World 13.00 15:30 ‘Telling the story’ using case studies 13.30 Value for money
- 47. Telling the Story UsingCase Studies Spend a couple of minutes discussing with one or two others… Things to consider when producing an ESF Community Grants Case Study
- 48. Case Studies: ESFApproach About the person: • Details of the client’s background, education/training and employment history and current situation. Training/support: • Details of what activity took place during the project • What the client feels s/he achieved in regard to soft skills (e.g. more confidence) or hard outcomes (e.g. a qualification, employment, or volunteering).
- 49. Case Studies: ESFApproach Plans for the future • How does the client hope to progress their employability, training or job searching? • Have they been referred to another organisation for ongoing support – if so which and what type ? Quote • A short quote from the client on how the project has made a difference. Signature
- 50. Telling the Story UsingCase Studies • The power of the story • The power of the actual words… • … and a picture paints 1,000 of them
- 51. The Power ofthe Story
- 52. The Power ofthe Actual Words • ‘If I didn’t have this I would have reoffended. I would have ended up getting drunk, fighting with someone and be back in prison.’ • ‘I felt miserable, I had no money and I was on benefits but now I’m independent. I feel good about working for my money.’
- 53.
- 54. Case Studies: AQuick Checklist • Before you start • Setting up the interview • On the day • After the event • Other thoughts
- 55.
- 56. Value for Money •What is the additional outcome impact? • What is the cost per outcome? • What is the return to society?
- 57. Gross No of peopleinto employmentPeople into employment who wouldn’t otherwise Would have got a job anyway (deadweight) Eligible people into employment who wouldn’t otherwise Would have got a job anyway (deadweight) Ineligiblepeople (leakage) Additional spend locally (multiplier) Additional eligible people into employment who wouldn’t otherwise Net/additional impact Ineligiblepeople (leakage) Would have got a job anyway (deadweight)Otherpeoplewhodon’t getorlosejobs (displacement) Additional eligible people into employment who wouldn’t otherwise Net additional impact Additional eligible people into employment who wouldn’t otherwise
- 58. Net additional impact Workthough part 1 of the worksheet
- 59. Net additional impactResults Gross Outcomes 250 Deadweight effect (those who’d have got a job anyway) 50 Gross Outcomes less Deadweight 200 Leakage effect (ineligible beneficiaries) 10 Gross Outcomes less Deadweight and Leakage 190 Displacement effect (other people losing jobs as a result) 0 Gross Outcomes less Deadweight, Leakage and Displacement 190 Multiplier effect 19 Gross Outcomes less Deadweight, Leakage and Displacement, plus Multiplier effect (“Net Additional Outcomes”) 209
- 60. Cost per outcome •Simple concept: TOTAL COST NO. of OUTCOMES • BUT: Gross or Net Outcomes? • Total cost or Total Funding? • See Part 2 of Workbook
- 61. Cost per outcomeResults Total Cost per Outcome (Gross) £3,000 ESF per Outcome (Gross) £1,350 Total Cost per Net Outcome £6,000 ESF per Net Outcome £2,700
- 62. Social Return • Thefinancial value to society of the outcomes • e.g. moving someone off benefit into work: – Benefits saved – Taxes contributed – Local spend – Reduced risk of costs arising from ill health, crime, anti-social behaviour
- 63. Social Return • Dependson lots of assumptions (rate of pay, length of employment, behaviour of the individual, age, personal circumstances etc) • Can calculate/estimate on a case-by-case – difficult, lengthy and expensive • Or use ready-made estimates e.g. Global Value Exchange
- 64. Global Value Exchange •Free to join and use • Open source estimates of the financial value of a huge range of outcomes www.globalvaluexchange.org
- 65. Social Return Results Valueto individuals of securing work £323,010 Value of volunteering £2,600 Value of increased earnings potential (excluding those who have secured work) £131,040 Value of fiscal savings to DWP £277,020 Total Social Return of the project £733,670
- 66. Social Return continued… •Can then divide Total Social Return by Total Cost to get return on investment e.g. if Project Z cost £200,000, SROI is £733,670 / £200,000 = £3.67 • Could have included other factors, e.g. – Value of volunteering – Increased earnings potential – Improved health
- 67. SROI warnings • Resultdepends on the factors included • Also depends on assumptions made • Similar projects making different assumptions will generate very different SROI scores • Dealing with time properly is complex • Doing SROI properly is expensive • Most projects can demonstrate high positive SROI, so even an high positive result may not unlock funding
- 68. Developing an EvaluationPlan for your project
- 69. Developing an EvaluationPlan: Some things to think about • Scope and Purpose – What? – Level – e.g. Project, Programme, Organisation – Formative / Summative – Audience?
- 70. Developing an EvaluationPlan: Some things to think about • Timing and Phasing – When? – Single phase or multiphase? • How do they fit together? – Previous evaluations? • The same approach or different? – Key deadlines? • ‘Doomed to success’
- 71. Developing an EvaluationPlan: Some things to think about • Collecting Data – What is required? • Back to the logic chain – Where can you get the data – What have you got already? – Storage and analysis – Any barriers to using data?
- 72. Developing an EvaluationPlan: Some things to think about • Who does it? – Internal / External? – Buy-in? – Objectivity – Budget
- 73. Developing an EvaluationPlan: Some things to think about • Collecting views – Options for beneficiaries? – Key stakeholders? – Options for Stakeholders
- 74. Developing an EvaluationPlan: Some things to think about • Managing the evaluation process – Who is responsible? – Sign off? • Dissemination – Who and how?
- 75. Developing an EvaluationPlan for your project
- 76. Wowing the World •Why are we doing this again? – Because we have to? – To improve our services to clients? – To attract new clients? – To demonstrate to ourselves and / or others that what we’re doing is effective? – To show Value for Money? – To convince people to give us some more?
- 77. Wowing the World •Whatever the reason we need to ‘broadcast it’ in some way • What approaches have you used that have been particularly effective in broadcasting what you’ve achieved and the impact that it has made?
- 78.
- 79. The word onthe street • Evaluation Reports • Summary Documents – Selected highlights • Newspapers / Newsletters • Annual Reports
- 80. The word onthe street
- 81. The word onthe street
- 82. The writing onthe wall…
- 83.
- 84.
- 85. Your Evaluation! • Anyimmediate thoughts? • Evaluation sheet – Free prize draw! • A more detailed conversation? • Peer support network?
Editor's Notes
- #39 Sample 40 35 24 100 Pop 1 120 105 75 300 Pop 2 40 45 50 55 60 50 300