D6 e6 outcome based measurement from theory to implementation
Outcome-Based MeasurementFrom Theory to ImplementationGhebray Consulting647firstname.lastname@example.org
Workshop Objectives1. Examine the process of developing & implementing outcome-based measurement2. Reflect on the value of embedding Theory of Change in developing a program theory or logic model3. Explore how the process of developing & implementing outcome-based measurement can be a vehicle for building evaluative culture
Agenda1. Presentation: An overview of measurement & evaluation (15 minutes)2. Discussion (10 minutes)3. Presentation: walking through the program planning cycle – embedding theory of change (15 minutes)4. Break (10 minutes)5. Small group discussion (50 minutes)6. Presentation: Measurement plan (15 minutes)7. Facilitated discussion: Building evaluative culture (25 minutes)8. Final thoughts & reflections (10 minutes)
Measurement•Measurement is the process of systematicallyand regularly collecting data on program quality,outputs, and outcomes for program participants(Not intended to establish causal relationship)Evaluation•Evaluation is a form of research method –in-depth and rigorous effort to measure programimpact and uses scientific research methods tocompare outcomes with what would happen inabsence of strategies or intervention(s)
INPUTS STRATEGIES OUTCOMESWhat was Which What was Which Didthe resources the strategies participantquantity & were most quantity were most s change?quality of important & quality important If so, howresources for of the for much andused to providing strategies achieving in whatimplement “high provided? the desired ways?strategies? quality” outcomes? strategy?
INPUTS OUTPUTS OUTCOMES IMPACTMONITORING PERFORMANCE OUTCOME IMPACT STUDY MEASUREMENT MEASUREMENTWhat is the What is the What are the What arequantity & quantity results for thequality of and quality participants impactsresources we of the and other that can behave invested program or stakeholders? attributedin this services we to theprogram? Are have program orthey aligned delivered? services?with programgoals?
Monitoring Questions• How many clients are in the program?• What are the socio-demographic characteristic of those in the program?• Are program participants part of the intended or target population?• How are program funds being used?
Performance Measurement Questions• How well was the program marketed or promoted?• Did the program offer adequate and high quality strategy or intervention?• How do clients perceive the program?• Are clients satisfied with the service and their encounters with service providers?
Outcome Measurement Questions•Did clients attain the intended outcomes?•What did clients learn?Impact Study Questions•Have clients’ quality of life improved followingprogram experience? Is the program responsiblefor their improved quality of life?•Is client participation in the program“responsible” for their improvement?
INPUTS OUTPUTS OUTCOMES IMPACTMONITORING PERFORMANCE OUTCOME IMPACT STUDY MEASUREMENT MEASUREMENT DOCUMENTING DOCUMENTING RESEARCHTRACKING INVESTMENT PROGRESS CAUSALITY SOME EVALUATION EXPERIENCE & HIGH EVALUATION EXPERTISE REQUIRED EXPERIENCE & EXPERTISE REQUIRED LOW/REASONABLE COST HIGH COST SHORTER TIME LONGER TIME
Before Implementing Outcome-basedMeasurement…1. Thorough program description• Program rationale, definition of the problem &feasibility of strategies or interventions• Clarity of causal assumptions - between resourcesstrategies or interventions and expected outcomesas well as performance measures or indicators2. Capacity & experience of program leaders tofacilitate and act on findings & lessons learned
Community Design Analysis, Design M & E PlanAsset & Needs Program Assessment Develop Engage Key Decision Data Collection Stakeholders Making Tools Share Data Collect ImplementFindings & Analysis Data Program Lessons Learned
CONTEXTUAL ASSUMPTIONS INPUTS STRATEGIES OUTPUTS OUTCOMES ANALYSISThe The Resources What Direct BenefitsReason implicit dedicated program products for clientsbehind theory to the does of during &the or belief program with program after theneed for why the inputs strategies programthe programprogram will work
Employment Program Example CONTEXTUAL STRATEGIES OUTPUTS ASSUMPTIONS INPUTS -# of classes ANALYSIS -Weekly soft -The absence -2 F.T.E Staff & mentorship-People in skills classes of a job history sessions our community: perpetuates -Program & -Match clients & offered Unemployment counseling-Have few job mentors space -# & type ofskills, bad or no -Job search skills jobs & limited - 8 months of clients training helps -Programjob histories supervised job served with job supplies & training & Readiness equipment-Have few job mentoringtraining & job sessions -Job mentorshipplacement can enhance on IMPACT OUTCOMEopportunities the job learning -Clients find -Clients learn soft jobs, remain skills & job strategies -Job placement employed & helps establish establish job -Clients establish supportive job history histories professional relationships with mentors
Using Program Theory to Plan and implement Outcome-based Measurement
What does “success” look like? According to Mark Friedman(2005), three basic questions can help us assess program success1.How much did we do?2.How well did we do it?3.Is anyone better off?
Outcome Indicator (Is anyone better off?)• Outcome indicators measure how much progress was made towards achieving the outcome - For an outcome “Newcomers are job ready to compete in Canadian job market”Outcome indicators include:• Rate of successful training & job placement completion (i.e. ability to prepare resume, positive feedback on mock interview & job placement• Maintaining professional relationship with mentors 6 month after end of program
Process Indicators (How well did we do it?)• Process indicators measure ways in which program services are delivered – For “Provide high quality service”Process indicators include:• Qualification & experience of mentors• Suitability of client/mentor match & job placement• Rate of client satisfaction
Output Indicators (How much did we do?)• Output indicators measure the quantity or the volume of services produced - For “Provide adequate support to targeted population”Output indicators include:- # & types services offered (i.e. learning sessions, employment counseling, mentorship etc.)- # and characteristics of clients served- # of mentors recruited & matched
Measuring Program “Success” # %1. Meet outreach target 150 1002. Attendance during orientation, intake & level of 75 50registration in program (# & type of clients served)3. High attendance, participation, & rate of satisfaction 65 434. Knowledge of job search strategies, Canadian labor 50 33market & work place culture5. Successful completion of program (including job 45 30placement)7. #s employed in profession related at ($17.00/hour) 26 178. Favorable Job assessment (tracked quarterly) 20 139. Job retention (for at least one year) 15 10
The Measurement & Evaluation Plan Program Indicators Data Data Who When Who Needs the Component Sources Collection Collects to Findings & to be Methods Data Collect Lessons Learned Measured Data did we do? How much Scope & Reach we do it it? How well did Program or Service Qualitybetter off?Is anyone Outcome for Participants
Building Evaluative Culture1. How do you create a shared understanding of measurement and evaluation? How do you make measurement & evaluation everyone’s business?2. What does measurement & evaluation mean to your board, staff etc?3. How do you balance between learning and accountability? How do you manage the conversation about measurement & evaluation with funders/staff? How do you negotiate what can and cannot be delivered?
References• Crawford, P. & Bryce, P. (2002). Project monitoring and evaluation: A method for enhancing the efficiency and effectiveness of aid project implementation• Friedman, M. (2005). Trying hard is not good enough: How to produce measurable improvements for customers and communities.• Penna, R. & Phillips, W. (2004). Outcome Frameworks: An overview for practitioners• Smith, M. (2010). Handbook of program evaluation for social work and health professionals• www.tccgrp.com