• Like
Session 1   general overview
Upcoming SlideShare
Loading in...5
×
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
207
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Chris NicolettiActivity #267: Analysing the socio-economicimpact of the Water Hibah on beneficiaryhouseholds and communities (Stage 1)Impact EvaluationTraining CurriculumSession 1April 16, 2013
  • 2. This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledgeits use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: AncillaryMaterial, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and notnecessarily those of the World Bank.MEASURING IMPACTImpact Evaluation Methods for PolicyMakers
  • 3. 3• My name is Chris Nicoletti• From NORC• Senior Impact Evaluation Analyst.• Worked in Zambia, Ghana, Cape Verde, Philippines,Indonesia, Colombia, Burkina Faso, etc.• Live in Colorado– I like to ski, hike, climb, bike, etc.– Married and do not have any children• What is your name?• Let’s go around the room and do introductions…Introduction…Impact Evaluation Training Curriculum - Activity 267
  • 4. 4Tuesday - Session 1INTRODUCTION AND OVERVIEW1) Introduction2) Why is evaluation valuable?3) What makes a good evaluation?4) How to implement an evaluation?Wednesday - Session 2EVALUATION DESIGN5) Causal Inference6) Choosing your IE method/design7) Impact Evaluation ToolboxThursday - Session 3SAMPLE DESIGN AND DATA COLLECTION9) Sample Designs10) Types of Error and Biases11) Data Collection Plans12) Data Collection ManagementFriday - Session 4INDICATORS & QUESTIONNAIRE DESIGN1) Results chain/logic models2) SMART indicators3) Questionnaire DesignOutline: topics being covered
  • 5. 5Today, we will answer thesequestions…Impact Evaluation Training Curriculum - Activity 267Why is evaluation valuable?What makes a good impactevaluation?How to implement an impactevaluation?123Why is evaluation valuable?What makes a good impactevaluation?123
  • 6. 66Today, we will answer thesequestions…Impact Evaluation Training Curriculum - Activity 267Why is evaluation valuable?What makes a good impactevaluation?123 How to implement an impactevaluation?
  • 7. 77Why Evaluate?Impact Evaluation Training Curriculum - Activity 267Need evidence on what worksInformation key to sustainabilityImprove program/policy implementation123 Limited budget and bad policies could hurt Design (eligibility, benefits) Operations (efficiency & targeting) Budget negotiations Informing beliefs and the press Results agenda and Aid effectiveness
  • 8. 8Results-Based Managementis a global trendEstablishing links between monitoring andevaluation, policy formulation, and budgetsManagers are judged by their programs’performance, not their control of inputs:A shift in focus from inputs to outcomes.Critical to effective public sector managementWhat is new about results?Impact Evaluation Training Curriculum - Activity 267
  • 9. 9Monitoring vs. EvaluationMonitoring EvaluationFrequency Regular, Continuous PeriodicCoverage All programs Selected program, aspectsData Universal Sample basedDepth ofInformationTracks implementation,looks at WHATTailored, often to performanceand impact/ WHYCost Cost spread out Can be highUtilityContinuous programimprovement, managementMajor program decisionsImpact Evaluation Training Curriculum - Activity 267
  • 10. 10MonitoringA continuous process of collecting and analyzinginformation, to compare how well a project, program or policy isperforming against expected results, and to inform implementation and program management.Impact Evaluation Training Curriculum - Activity 267
  • 11. 1111Impact Evaluation AnswersImpact Evaluation Training Curriculum - Activity 267What was the effect of the program onoutcomes?How much better off are the beneficiariesbecause of the program/policy?How would outcomes change if theprogram design changes?Is the program cost-effective?
  • 12. 12EvaluationA systematic, objective assessment of an on-goingor completed project, program, or policy, its design,implementation and/or results, to determine the relevance and fulfillment of objectives,development efficiency, effectiveness, impact andsustainability, and to generate lessons learned to inform the decision makingprocess, tailored to key questions.Impact Evaluation Training Curriculum - Activity 267
  • 13. 13Impact EvaluationAn assessment of the causal effect of a project ,program or policy on beneficiaries. Uses acounterfactual… to estimate what the state of the beneficiaries would havebeen in the absence of the program (the control orcomparison group), compared to the observed state ofbeneficiaries (the treatment group), and to determine intermediate or final outcomes attributableto the intervention.Impact Evaluation Training Curriculum - Activity 267
  • 14. 1414Impact Evaluation AnswersImpact Evaluation Training Curriculum - Activity 267What is effect of a household (hh) waterconnection on hh water expenditure?Does contracting out primary health carelead to an increase in access?Does replacing dirt floors with cementreduce parasites & improve child health?Do improved roads increase access tolabor markets & raise income?
  • 15. 1515Answer these questionsImpact Evaluation Training Curriculum - Activity 267Why is evaluation valuable?How to implement an impactevaluation?What makes a good impactevaluation?123
  • 16. 1616How to asses impactImpact Evaluation Training Curriculum - Activity 267What is beneficiary’s test score with programcompared to without program?Compare same individual with & withoutprograms at same point in timeFormally, program impact is:α = (Y | P=1) - (Y | P=0)e.g. How much does an education programimprove test scores (learning)?
  • 17. 1717Solving the evaluation problemImpact Evaluation Training Curriculum - Activity 267Estimated impact is difference between treatedobservation and counterfactual.Counterfactual: what would have happenedwithout the program.Need to estimate counterfactual.Never observe same individual with and withoutprogram at same point in time.Counterfactual is key to impact evaluation.
  • 18. 1818Counterfactual CriteriaImpact Evaluation Training Curriculum - Activity 267Treated & Counterfactual(1) Have identical characteristics,(2) Except for benefiting from the intervention.No other reason for differences inoutcomes of treated and counterfactual.Only reason for the difference inoutcomes is due to the intervention.
  • 19. 19192 Counterfeit CounterfactualsImpact Evaluation Training Curriculum - Activity 267Before and AfterThose not enrolled Those who choose not toenroll in the program Those who were not offeredthe programSame individual before the treatment
  • 20. 20201. Before and After: ExamplesImpact Evaluation Training Curriculum - Activity 267You do not take into consideration thingsthat are changing over the interventionperiod.Agricultural assistance program Financial assistance to purchase inputs. Compare rice yields before and after. Before is normal rainfall, but after is drought. Find fall in rice yield. Did the program fail? Could not separate (identify) effect of financialassistance program from effect of rainfall.
  • 21. 21212.Those not enrolled: Example 1Impact Evaluation Training Curriculum - Activity 267Compare employment & earning of thosewho sign up to those who did notJob training program offeredWho signs up?Those who are most likely to benefit -i.e. those with moreability- would have higher earnings than non-participantswithout job trainingPoor estimate of counterfactual
  • 22. 22What’s wrong?Impact Evaluation Training Curriculum - Activity 267Selection bias: People choose to participatefor specific reasons123 Job Training: ability and earning Health Insurance: health status and medicalexpendituresMany times reasons are related to theoutcome of interestCannot separately identify impact of theprogram from these other factors/reasons
  • 23. 23Possible Solutions???Impact Evaluation Training Curriculum - Activity 267Need to guarantee comparability of treatmentand control groups.ONLY remaining difference is intervention.In this training we will consider: Experimental Designs Quasi-experiments (Regression Discontinuity, Doubledifferences) Non-experimental (or) Instrumental Variables.EXPERIMENTAL DESIGN!!!
  • 24. 2424Answer these questionsImpact Evaluation Training Curriculum - Activity 267Why is evaluation valuable?How to implement an impactevaluation?What makes a good impactevaluation?123
  • 25. 25When to use ImpactEvaluation?Evaluate impact when project is: Innovative Replicable/scalable Strategically relevant for reducingpoverty Evaluation will fill knowledge gap Substantial policy impactUse evaluation within a program to testalternatives and improve programsImpact Evaluation Training Curriculum - Activity 267
  • 26. 26Choosing what to evaluateCriteria Large budget share Affects many people Little existing evidence of impact fortarget population (IndII Examples?)No need to evaluate everythingSpend evaluation resources wisely
  • 27. 27IE for ongoing programDevelopmentAre there potential programadjustments that would benefit from acausal impact evaluation?Implementing parties have specificquestions they are concerned with.Are there parts of a program that maynot be working?
  • 28. 28How to make evaluationimpact policy focusedExample: Scale up pilot? (i.e., Water Hibah)Criteria: Need at least a X% average increase in beneficiaryoutcome over a given periodAddress policy-relevant questions What policy questions need to be answered? What outcomes answer those questions? What indicators measures outcomes? How much of a change in the outcomeswould determine success?
  • 29. 29Policy impact of evaluationWhat is the policy purpose?Provide evidence for pressing decisionsDesign evaluation with policy makersIndII Examples???
  • 30. 30 Decide what need tolearn. Experiment withalternatives. Measure and inform. Adopt better alternativesovertime.Policy impact of evaluationChange in incentives Rewards for changing programs. Rewards for generating knowledge. Separating job performance from knowledge generation.Cultural shiftFrom retrospective evaluationto prospective evaluation.Look back and judge
  • 31. 31• Choosing what to evaluate is something thatshould take time and careful consideration.• Impact evaluation is more expensive and oftenrequires third party consultation.• The questions that require an IE to answer shouldbe evident in your logic models and M&E plansfrom the beginning.• Remember, IE is an assessment of the causal effect ofa project, program or policy on beneficiaries.Choice should come from existinglogic models and M&E plans.
  • 32. CHOICE #1Retrospective Design or Prospective Design?
  • 33. 33Retrospective AnalysisRetrospective Analysis is necessary when wehave to work with a pre-assigned program(expanding an existing program) and existing data(baseline?)Examples: Regression Discontinuity: Education Project (Ghana) Difference in Differences: RPI (Zambia) Instrumental variables: Piso firme (México)
  • 34. 34• Use whatever is available – the data was not collected forthe purposes at hand.• The researcher gets to choose what variables to test, based onprevious knowledge and theory.• Subject to misspecification bias.• Theory is used instrumentally, as a way to provide astructure justifying the identifying assumptions.• Less money on data collection (sometimes), more moneyon analysis.• Does not really require “buy in” from implementers or fieldstaff.Retrospective Designs
  • 35. 35Prospective AnalysisIn Prospective Analysis, the evaluation isdesigned in parallel with the assignment of theprogram, and the baseline data can be gathered.Example: Progresa/Oportunidades (México)CDSG (Colombia)
  • 36. 36• Intentionally collect data for the purposes of the impactevaluation.• The variables collected in a prospective evaluation arecollected because they were considered potentialoutcome variables.• You should report on all of your outcome variables.• The evaluation itself may be a form of treatment.• It is the experimental design that is instrumental - givesmore power both to test the theory and to challenge it.• More money on data collection, less money on analysis.• Requires “buy in” from implementers and field staff.Prospective Designs
  • 37. 37Prospective DesignsUse opportunities to generate good controlsThe majority of programs cannot assignbenefits to all the entire eligible populationNot all eligible receive the programBudget limitations: Eligible beneficiaries that receive benefits are potential treatments Eligible beneficiaries that do not receive benefits are potentialcontrolsLogistical limitations: Those that go first are potential treatments Those that go later are potential controls
  • 38. 38• The decision to conduct an impact evaluation wasmade after the program began, and ex postcontrol households were identified.• We are now trying to use health data fromPuskesmas to “fill in the gaps” of the baseline.• This would be a retrospective design, becausethere was not an experimental design in place forthe roll out of the program.An example: Socio-econimpact of Endline Water Hibah
  • 39. CHOICE #2What type of Evaluation Design do youuse?
  • 40. 40Types of DesignsProspectiveRandomized AssignmentRandomized PromotionRegression DiscontinuityRetrospectiveRegression DiscontinuityDifferences in DifferencesMatchingModel-based / Instrumental Variable
  • 41. 41How to choose?Identification strategy depends onthe implementation of the programEvaluation strategy depends on therules of operations
  • 42. 42Who gets the program?Eligibility criteria Are benefits targeted? How are they targeted? Can we rank eligibles priority? Are measures good enough for fine rankings?Roll outEqual chance to go first, second, third?
  • 43. 43Rollout base on budget/administrative constraintsEthical ConsiderationsEqually deserving beneficiaries deserve an equalchance of going first Give everyone eligible an equal chance If rank based on some criteria, then criteriashould be quantitative and publicEquityTransparent & accountable methodDo not delay benefits
  • 44. 44The Method depends onthe rules of operationTargeted UniversalIn StagesWithoutcut-offo Randomizationo RandomizedRolloutWithcut-offo RD/DiDo Match/DiDo RD/DiDo Match/DiDImmediatelyWithoutcut-offo RandomizedPromotiono RandomizedPromotionWithcut-offo RD/DiDo Match/DiDo RandomizedPromotion
  • 45. 45• Provision of services to villages and households under the WaterHibah is not determined by randomization, but by assessment andWTP.• The dataset design exhibits some characteristics of a controlledexperiment with connected and unconnected, but connection decisionis not determined by randomization.• Household matching is not an efficient method with the potentialdiscrepancies we identified in the pilot test, and does not work verywell with the sample design that was chosen.• Village-level matching is not feasible because there are usuallyconnected and unconnected in a single village (locality).• The design we have chosen is: pretest-posttest-nonequivalent-control-group quasi-experimental design that will useregression-adjusted Difference-in-Difference impactestimators.An example: Socio-econimpact of Endline Water Hibah
  • 46. CHOICE #3What type of Sample Design do youuse?
  • 47. 47Types of DesignsRandom SamplingMulti-Stage SamplingSystematic SamplingStratified SamplingConvenience SamplingSnowball SamplingTypes of Sample DesignsPlus any combination of them!
  • 48. 48• It is important to note that sample design can beextremely complex.• A good summary is provided by Duflo (2006):• The power of the design is the probability that, for a given effect size and a givenstatistical significance level, we will be able to reject the hypothesis of zero effect.Sample sizes, as well as other (evaluation & sample) design choices, will affectthe power of an experiment.• There are lots of things to consider, such as:• The impact estimator to be used; The test parameters (power level, significancelevel); The minimum detectable effect; Characteristics of the sampled (target)population (population sizes for potential levels of sampling, means, standarddeviations, intra-unit correlation coefficients (if multistage sampling is used)); andthe sample design to be used for the sample surveyA good sample design requiresexpert knowledge
  • 49. 49The basic process is this…Level of PowerLevel ofHypothesisTestsCorrelations inoutcomeswithin groups(ICCS)Mean andVariance ofoutcomes &MDES
  • 50. 50• Most times, you do not have all of thisinformation.• Use existing studies; other data sources; assumptions.• Working backwards to fit a certain power size.• Working backwards b/c expected level of impactthat you want to test for.• You are working backwards to fit a certainbudget!• Build in marginal costs for each stage of sampling.• Decide whether or not to pursue project.The reality is…
  • 51. 51• Outcome indicators: we have simplified versions of themin the baseline, but they have been modified for endline Use baseline dataset to calculate ICCs.• Highest variation in outcome indicators was identified acrossvillages (localities)  primary sample unit is the village.• The # of households in the village was found to improve theefficiency of the design  stratify villages based on the # ofhouseholds• Marginal costs of village visit vs. household visit wereincluded.• The final sample design that was identified is referred toas: Stratified Multi-stage sampling with 250 villages and7-14 households per experimental group = 7,000 hhs.An example: Socio-econimpact of Endline Water Hibah
  • 52. What can IndII Do?Ensure your M&E systems are relevantand reliable…
  • 53. 53Data: Coordinate IE &Monitoring SystemsTypical content Lists of beneficiaries Distribution of benefits Expenditures Outcomes Ongoing process evaluationProjects/programs regularly collect data formanagement purposesInformation is needed for impact evaluation
  • 54. 54Manage M&E for results Tailor policy questions Precise unbiased estimates Use your resources wiselyBetter methodsCheaper dataTimely feedback and program changesImprove results on the groundProspective evaluations are easier andbetter with reliable M&E
  • 55. 55Evaluation uses information to:Verify who is beneficiaryWhen startedWhat benefits were actuallydeliveredNecessary condition for program tohave an impact: Benefits need toget to targeted beneficiaries.
  • 56. 56Overall MessagesEvaluation designImpact evaluationIs useful for: Validating program design Adjusting program structure Communicating to finance ministry & civil societyA good one requires estimating the counterfactual: What would have happened to beneficiaries if had notreceived the program Need to know all reasons why beneficiaries got program &others did not
  • 57. 57Other messagesGood M&E is crucial not only to effective project management but can be adriver for reformMonitoring and evaluation are separate, complementary functions, but both arekey to results-based managementHave a good M&E plan before you roll out your project and use it to inform thejourney!Design the timing and content of M&E results to further evidence-baseddialogueGood monitoring systems & administrative data can improve IE.Easiest to use prospective designs.Stakeholder buy-in is very important
  • 58. 58Tuesday - Session 1INTRODUCTION AND OVERVIEW1) Introduction2) Why is evaluation valuable?3) What makes a good evaluation?4) How to implement an evaluation?Wednesday - Session 2EVALUATION DESIGN5) Causal Inference6) Choosing your IE method/design7) Impact Evaluation ToolboxThursday - Session 3SAMPLE DESIGN AND DATA COLLECTION9) Sample Designs10) Types of Error and Biases11) Data Collection Plans12) Data Collection ManagementFriday - Session 4INDICATORS & QUESTIONNAIRE DESIGN1) Results chain/logic models2) SMART indicators3) Questionnaire DesignOutline: topics being covered
  • 59. Thank You!
  • 60. This material constitutes supporting material for the "Impact Evaluation in Practice" book. This additional material is made freely but please acknowledgeits use as follows: Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: AncillaryMaterial, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and notnecessarily those of the World Bank.MEASURING IMPACTImpact Evaluation Methods for PolicyMakers