Labor Markets Core Course 2013: Monitoring and evaluation


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Labor Markets Core Course 2013: Monitoring and evaluation

  1. 1. 1Monitoring and EvaluationLaura B. RawlingsLead Social Protection SpecialistHuman Development NetworkWorld BankLabor Market Core CourseMay 2013
  2. 2. 2Objectives of this session1. Monitoring and Evaluation – the Foundationfor Results2. Using a RESULTS Chain3. Measuring Results in PublicWorks Programs4. Evaluationso Processo Targetingo Impact5. Conclusion - Moving Forward
  3. 3. Why are Monitoring and EvaluationImportant?Credible evidence is the foundation for: Results-based management• Governments and managers are being judged by their programs’performance, not their control of inputs• Shift in focus from inputs to outcomes from threats to tools Accountability and transparency between government and civil society between programs and beneficiaries Knowledge generation on developmenteffectiveness• Applied research, notably through impact evaluations, to determine whetherprograms are reaching their intended outcomes3
  4. 4. 4EVALUATION MONITORINGFrequency: Periodic Regular, ContinuousCoverage: Selected programs, All programsaspectsData: Sample based UniversalDepth of Tailored, often Tracks implementation,Information: to performance looks at WHATand impact/WHYCost: Can be high Cost spread outUtility: Major program Continuous programdecisions improvement, management
  5. 5. 5Objectives of this session1. Monitoring and Evaluation – the Foundationfor Results2. Using a RESULTS Chain3. Measuring Results in PublicWorks Programs4. Evaluationso Processo Targetingo Impact5. Conclusion - Moving Forward
  6. 6. 6Using a Results ChainResults chains are a simple approach to mappingthe causal logic/theory of change underpinning aprogram best used as a participatory tool, during project design basis for constructing a M&E approach which will test thevalidity of the theory of changeA results chain answers 3 questions What are the intended results of the program? How will we achieve the intended results? How will we know we have achieved the intendedresults?
  7. 7. What is a results chain?Inputs• Financial,human, andother resourcesmobilized tosupport activities• Budgets, staffing,other availableresourcesActivities• Actions taken orwork performedto convert inputsinto specificoutputs• Series of activitiesundertaken toproduce goods andservicesOutputs• Productsresulting fromconverting inputsinto tangibleoutputs• Goods and servicesproduced anddelivered , under thecontrol of theimplementingagencyOutcomes• Changesresulting fromuse of outputs bytargetedpopulation• Not fully underthe control ofimplementingagencyFinalOutcomes• The finalobjective of theprogram• Long-termgoals• Changes inoutcomes withmultiple drivers7Implementation (SUPPLY SIDE) Results (DEMAND + SUPPLY)
  8. 8. 8Objectives of this session1. Monitoring and Evaluation – the Foundationfor Results2. Using a RESULTS Chain3. Measuring Results in PublicWorks Programs4. Evaluationso Processo Targetingo Impact5. Conclusion - Moving Forward
  9. 9. Public Works ProgramResults Chain ExampleInputs• Budget for PWProgram• Ministry of Laborstaff• Staff fromparticipatingmunicipalitiesActivities• Setting of sub-minimum wage• Informationcampaign• Development andapplication ofenrollmentprocess• Selection of sites,contracting andtraining of PWoperatorsOutputs(Annual)• 50,000 jobsprovided by PWprogram• $1,000,000 inwages transferred• > 75% of programcosts transferredas wages• 2,000 PWsubprojectsproducedOutcomes• Net incometransfer tohouseholds• Skills acquired• Utility,maintenance ofPWsFinalOutcomes•Future income,employment• Among beneficiaryhouseholds:-income, assets-health, nutrition- education• Aggregateunemployment,poverty9Implementation (SUPPLY SIDE) Results (DEMAND + SUPPLY)
  10. 10. Goals depend on type of PW program• Safety net -- temporary or seasonal incomesupport in response to covariate shocks such asseasonal droughts, macro-economic crises• Poverty Alleviation – income transfer to poor• Employment/Insurance -- employment guarantee• Education/Training -- skills acquisition with longerterm employment/earnings objectives• Public goods – from subprojects created these inform the content of the results chainMeasuring PublicWorks Program Goals10
  11. 11. Multiple, often iterative goals are commonAcross beneficiaries:• Direct beneficiaries• Direct beneficiaries’ households• Communities benefitting from PW subprojects• Region, nation benefitting from PW programOver time:• During period of employment – short term outputs,outcomes• After period of employment -- longer-term outcomes,sustainability• Relative to crisis period – speed, appropriateness ofresponseMeasuring the Goals of PublicWorksPrograms11
  12. 12. Direct BeneficiariesOutputs• Targeting• Errors of inclusion• Errors of exclusion• Gender balance amongbeneficiaries• Income transfer received• Days/man-hoursemployedOutcomes• Net wage gain• Net employmentLonger-termOutcomes• Skills acquisition• Future employment andearningsBeneficiaries’HouseholdsOutcomes• Net income, consumptionduring PW period• Labor market status ofhousehold members• Nutritional status/foodsecurityLonger-Term Outcomes• Net income, consumptionover longer time period• Savings, assets• Education, health, nutrition• Reliance other types ofpublic transfers• Risk management in futureshocks12Typical PublicWorks IndicatorsPublic Works SubprojectsOutputs• Quality of goods andservices producedOutcomes• Sustainability, utility andimpact of subprojectsPublic Works ProgramOutputs• Labor intensity• Component costs• Wage rate relative tomarket, minimumOutcomes• Regional, national aggregateimpact on poverty,employment• Risk management in futurecrises
  13. 13. 13Objectives of this session1. Monitoring and Evaluation – the Foundationfor Results2. Using a RESULTS Chain3. Measuring Results in PublicWorks Programs4. Evaluationso Processo Targetingo Impact5. Conclusion - Moving Forward
  14. 14. Depend on the type of question being asked (Imas and Rist, 2009) Descriptive Seeks to determine what is taking place Often describes aspects of a process, condition, or set of views Includes process evaluations, beneficiary assessments Normative Compares what is taking place to what should be taking place Often used to assess targeting, completion of activities Measures inputs, activities and outputs in results chain Includes assessments of targeting/benefit incidence, coverage, adequacy Cause and Effect or Impact Assesses causal results and looks at outcomes; Addresses questions of attribution and implies a comparison of performance Uses before and after and with and without comparisons Impact evaluations of program level outcomes, operational options14Types of Evaluation
  15. 15. Process Evaluations – Decriptive - Assesses whether a program isbeing implemented as planned Tailored to program’s institutional arrangements and components Often include quantitative and qualitative approaches Particularly useful at early stages of program implementationTargeting/Incidence Analysis – Normative - Determines whetherthe program is reaching its intended beneficiaries Can be applied at the geographical and household levels Includes errors of inclusion and exclusion Needs a reference from national measures of poverty (usually director proxy measures of income or consumption) against which tobenchmark program performance Can use national surveys with ID of program beneficiaries, andoversampling if needed and/or regular program registration process15Types of Evaluations (examples)
  16. 16. Impact Evaluations – Cause and Effect - An assessment of thecausal effect of a project , program or policy on beneficiaries Uses a counterfactual obtained from a control or comparison group toestimate the state of the beneficiaries in the absence of the program Relies on baseline and follow-up data on treatment and comparisongroupsUseful for:-- Determining intermediate or final outcomes attributable to theintervention• Often used to examine questions with less clear answers such as changes inbehavior or outcomes with a range of drivers-- Testing program design options• For example, different outreach strategies or the relative effectiveness ofdifferent benefit packages16Types of Evaluations (examples)
  17. 17. Evaluations are derived from the question posedand should be tailored accordinglyEvaluations benefit from… Combining quantitative and qualitative data Cost – benefit analysis Ensuring timeliness of measuring results, producinginformation to inform key decisions Early planning! Keep an eye on costs and take advantage ofavailable data, national surveys17Structuring Evaluations
  18. 18. All impact evaluations estimate the counterfactual, usingcontrol or comparison groups: What would thetreatment group be like in the absence of the program?1. Experimental/Randomized Assignment- uses randomized assignment to determine who gets program treatment(s) and whois control among eligible beneficiaries- can be used ethically in cases where program cannot reach all potential beneficiariesat once; or to test program alternatives- random assignment creates statistically equivalent groups (treatment and control)which allows a valid estimate of the counterfactual2. Quasi-Experimental- mimics experimental designs- methods to create comparison groups include:Regression DiscontinuityDifferences in DifferencesInstrumentalVariablesStatistical Matching Choice of method depends on context. Rules of program operation are keybecause they determine eligibility for the program! Use them to IDcomparison group.18Impact Evaluation Methods
  19. 19. In many cases, the program cannot reach all potentialbeneficiaries at the same time – use these opportunities! Universe of eligible beneficiaries > # beneficiaries-- Use random selection/lottery to select who is offered benefits, againstthose not selected (controls)-- Fair, transparent and ethical way to assign benefits to equally deservingpopulations Oversubscription:-- Give each eligible unit the same chance of receiving treatment-- Compare those offered treatment with those not offered treatment(controls) Phase in:-- Give each eligible unit the same chance of receiving treatment first,second, third….randomly select order of phase in-- Compare those offered treatment first, with those offered treatmentlater (controls)19Experimental Design/RandomizedAssignment
  20. 20. Example - Assessing Impact on NetIncome Key question in PW projects:What is the netincome gain to participating workers’households? The income gain of the program does not equalthe gross wage rate (Ravallion, 2009) Depends on the behavioral responses of the mainrecipient. Poor people “cannot afford to be idle”. Needto forego other forms of income to join PW Depends on the behavioral response of otherhousehold members (take up displaced activities?) Depends on the conditions of the local labor market atthe time of the introduction of the program
  21. 21. What is the net income gain toparticipating workers’ households? Clearly a question that needs a counterfactual analysis: what wouldhave happened in the absence of the program? Idea of impact evaluation is to estimate change in outcomes that isattributable to the program Net Gains = Gross Wage – Foregone Income Estimate of foregone income essential to provide cost-benefit analysis(Murgai, Ravallion 2005) Use control or comparison groups to estimate foregone income (ie thecounterfactual – what the income the beneficiaries would have had inthe absence of the program) Higher foregone income relative to gross wage = less effective
  22. 22. What results are needed to assess the performance ofyour program? Use results chains to define a shared view of the project’s theoryof change, causal pathways to reaching the main projectobjectives Establish the M&E framework early on, get good baseline data Monitor activities, outputs, key performance indicators Collect cost data for cost-benefit analysis Do you need to establish that the program caused observedchanges in outcomes? If so, use an impact evaluation Impact evaluations need good counterfactuals, which depend onthe validity of comparison groups  clear rules of programoperation allow you identify valid comparison groups22Conclusion