Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Basics of Impact Evaluation (Module 2)


Published on

The goal of this course is to provide policy analysts and project managers with the tools for evaluating the impact of a project, program or policy. This course provides information on the methods that can be used to measure the impact of a project, program or policy on the well-being of individuals and households. The course addresses the ways in which the results of an impact evaluation may be put to use – such as, to improve the design of projects and programs, as an input into cost-benefit analysis, and as a basis for policy decisions.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Basics of Impact Evaluation (Module 2)

  2. 2. IE Answers 3 Questions 1. Why is evaluation valuable? 2.What makes a good impact evaluation? 3. How to implement evaluation?
  3. 3. Why Evaluate? Need evidence on what works o Limited budget force choices o Bad policies could hurt Improve program/policy implementation o Design: eligibility, benefits o Operations: efficiency & targeting Information key to sustainability o Budget negotiations o Informing government, donors, and public
  4. 4. Allocate Limited Resources? Benefit-Cost analysis o Comparison of choices o Highest return investment Benefit: o Change in outcome indicators o Measured through impact evaluation Cost: o Additional cost of providing benefit o Not accounting cost
  5. 5. IE versus “Traditional” M & E Monitoring o Outcome trends over time o E.g. poverty, school enrollment, mortality “Process” Evaluation o Implementation o Efficiency o Targeting Administrative Data Management Information Systems
  6. 6. But IE Answers… What is the effect of a program on outcomes of interest? How much better off are beneficiaries because of the intervention? How would outcomes change under alternative program designs? Does the program impact people differently (e.g. females, poor, minorities) Is the program cost-effective? Traditional M&E cannot answer these questions
  7. 7. Use Impact Evaluation To…. Scale-up pilot-interventions/programs Discontinue programs Adjust program benefits Inform stakeholders (i.e. donors) Learning for others E.g., Female stipend project in Bangladesh From pilot phase to cover entire country IE pursues governments in other countries to introduce
  8. 8. What Makes a Good Impact Evaluation? Assessing impact is, however, difficult: Examples: o By how much does an anti-poverty program lower poverty? o What is a beneficiary’s income with the program compared to without the program? Compare same individual with & without programs at same point in time Never observe same individual with and without program at same point in time
  9. 9. Solving the Evaluation Problem Counterfactual: what would have happened without the program Need to estimate counterfactual o I.e. find a control or comparison group Counterfactual Criteria o Treated & counterfactual groups have identical characteristics on average o Only reason for the difference in outcomes is due to the intervention
  10. 10. 2 “Counterfeit” Counterfactuals Before and after o Same Individual before the treatment Non-Participants o Those who choose not to enroll in program o Those who were not offered the program
  11. 11. Baseline study Participants Time Income Y2 Y1 Y0 Program Impact? Before and After Comparisons: Participants
  12. 12. Impact Participants Time Income Y4 Y2 Y0 Program Control Y1 Y3 Counterfactual With and Without Comparisons: Looking at Counterfactual
  13. 13. Before and After Compare Y before and after intervention Y2-Y0 = Estimated Impact Y0 = counterfactual Estimate Does not control for time varying factors Y1= True counterfactual Y2-Y1= True impact Y1-Y0 is over-estimate
  14. 14. With and Without Comparisons: The Case of Non-Participants…. Compare non-participants to participants Counterfactual: non-participant outcomes Problem: why did they not participate? Counterfeit program effect: (Y4 – Y3) Counterfactual program effect: (Y4 – Y2) So without right counterfactual, counterfeit comparison leads to underestimate program effect
  15. 15. What’s wrong? Selection bias: People choose to participate for specific reasons Many times reasons are directly related to the outcome of interest o Micro-credit: ability and earning; o Conditional cash transfer: Schooling status and schooling expenditures Cannot separately identify impact of the program from these other factors/reasons
  16. 16. Program Placement Bias Gov’t offers micro-credit program to villages with high poverty Compare poverty in villages offered program to poverty in villages not offered Program targeted based on poverty, so: o Treatments have high poverty o Counterfactuals have low poverty Cannot separately identify program impact from geographic targeting criteria
  17. 17. Need to know…. Why some get program and others not How beneficiaries get into treatment versus control group If reasons correlated w/ outcome o cannot identify/separate program impact from other explanations of differences in outcomes The process by which data is generated
  18. 18. Possible Solutions… Guarantee comparability of treatment and control groups ONLY remaining difference is intervention In this seminar, we will consider o Experimental design/randomization o Quasi-experiments o Propensity score matching o Double differences o Instrumental Variables o Regression discontinuity and pipe line method