How we can use impact evaluation to assure effective use of resources for development

9,661 views

Published on

Published in: Technology, Business
1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
9,661
On SlideShare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
44
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

How we can use impact evaluation to assure effective use of resources for development

  1. 1. How we can use impact evaluationto assure effective use of resources for development Maximo Torero, m.torero@cgiar.org Director Markets, Trade and Institutions Division (IFPRI) IFAD-IFPRI Partnership, January 31st. 2012
  2. 2. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  3. 3. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  4. 4. Need for impact evaluation Helps identify and measure the results Helps identify the causal link between intervention and results Provides a systematic and objective assessment of program impacts Helps determine if interventions are relevant and cost effective Promotes accountability, evidence-based policymaking, and learning.
  5. 5. Need for impact evaluation Over past decade, increased demand from governments, donor agencies and general public, for evidence of Impact of development policies.  Political tool: Brings accountability regarding the use of development money  Fiscal tool / budgetary tool: Allocate resources across different sectors or programs  Management tools: Understand how to better reach the objectives.
  6. 6. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  7. 7. Monitoring and Impact Evaluation:Monitoring A tool that provides regular information on:  How a project is being implemented  How a project is operating in the field  How a project is progressing relative to targets  What is the quality of service delivery (where applicable) Rationale for Monitoring:  Provides basis for corrective action  Holds implementers accountable for delivery of inputs  Provides assessment of continued relevance  Provides critical information for decision-making
  8. 8. Monitoring and Impact Evaluation:Evaluation Impact Evaluation:  Measures effectiveness and impact of programs or policies on outcomes of interest  Seeks to establish causality  Not all programs need to be evaluated; not all outcomes need to be measured in all evaluations
  9. 9. Indicators for Monitoring and Evaluation IMPACT Effect on living standards - better welfare impacts (e.g literacy, health) Evaluation - increase in participation, happiness OUTCOMES Access, usage and satisfaction of users - e.g. school attendance, vaccination rates, - food consumption, number of mobile phones OUTPUTS Goods and services generated - more local government services delivered Monitoring - e.g., textbooks, food delivered, roads built INPUTS Financial and physical resources - track resources used in the intervention -e.g. budget support for local service deliveryPage 9
  10. 10. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  11. 11. Guiding Principles of our IE Approach1. Generate information to influence decisions2. Specify which indicators and methods are most suitable for each type of projects3. Identify impact pathways4. Evaluation activities must be built into the project design4. Consider direct and indirect beneficiaries of projects5. Evaluation at different levels of aggregation: Individual, thematic, and overall program6. Incorporate complementarities and substitution among project impacts
  12. 12. Description of the ProjectStage 1: Stage 2: Stage 3:Consultation Development Feedback by Theme:IFAD’s: •Technology • Governments,• Objectives Identify: • Productivity • IFAD• Activities • Indicators • Market Access • Implementers• Information • Methods • Nutrition • CSO needs to by level of Aggregation: • Monitor performance • Individual projects • Theme • Evaluate Effectiveness • Agricultural Development • Asses Impact program Target PEOPLE and vulnerable GROUPS: • Poor and Women
  13. 13. Impact Evaluation: Impact Pathway The expected causal chain of events leading from project activities to outputs, to changes in the target population, and to the achievement of project objectives:  From INPUTS OUTCOMES IMPACTS Focus on the impact pathway allows to:  Understand how impacts are (or are not) achieved  Allows generalizability of findings  Provides key information for scaling up  Identification of indicators for each step along the impact pathway
  14. 14. Illustrative Impact Pathway, Indicators, Methods Example from: Science & Technology IMPACT PATHWAY INDICATORS METHODS Scholarships for plant breeders & Spending on scholarships grants for agronomic research & research grantsPROCESS  number and quality of varieties No. new varieties Internal program released Approved & released monitoring  availability and adoption of % male, female farmers improved crop varieties Using improved varieties Higher yields for farmers who Average yields among adopted improved varieties adopting farmersIMPACT Intra-HH surveys: Before/After, Income, expenditure, Beneficiary/  income,  poverty among farmer Well-being indicators Control (Diff in households among target groups Diff) (poor, women, etc.)
  15. 15. Applying the Methodology to specifictypes of interventions Technology  Example: Bio-fortification Productivity  Example: Grants to crop breeding programs Market Access  Example: Participation of small holders in the dairy value chain, “chilling plant hubs” Nutrition interventions  Example: Evaluation of specific interventions to improve nutrition of the most vulnerable
  16. 16. Bio-fortification Project (Science and Technology) Assumption: No price effect… IMPACT PATHWAY INDICATORS METHODSPROCESS Bio-fortification Spending on bio-fortification Internal program R&D monitoring Adoption of bio-fortified varieties No. of farmers and land adopting bio-fortified varieties. Greater yields for farmers who Average yields among adopted bio-fortified varieties adopting farmers Production of bio-fortified varieties Total production of bio- HH surveys fortified varieties •Beneficiary, control • Farmers, consumers Consumption of  consumption of No. of individuals and •DD estimator bio-fortified animal products, average consumption (by •Randomization varieties fruits, and vegetables type of individual))… •Panel: first round effect vs. secondIMPACT Reducing micronutrient round effects Change in micronutrient status malnutrition •Qualitative Improvements in health, work Morbidity, mortality, information: two-way performance, cognitive ability calling with the poor enrollment ratio in primary  income,  poverty among Income, expenditure, farmer households Well-being indicators
  17. 17. Chilling Plant Hubs (Market Access) IMPACT PATHWAY INDICATORS METHODSPROCESS Creation of farmer groups as dairy Internal program Number of DFBA created farmer business associations Monitoring and number of farmers (DFBA) participating (by gender) Qualitative Assessment: Number of plants and milk organizational Chilling plant construction capacity capacity Increase milk production of milk production of farmer member farmers members HH surveys Reduction in loss through Volume of loss due to •Beneficiary, control spoilage spoilage •DD estimator •Non-experimentalIMPACT design Sales to formal markets and Value of sales to formal •Qualitative traditional markets processors and to traditional information: two-way markets calling with the poor  income,  poverty among Income, expenditure, farmer members Well-being indicators
  18. 18. M&E at Different Levels of AggregationEvaluation strategy Indicators MethodsWhat needs to be Cross theme Meta analysis learned at the Program indicators: at the strategy strategy level? level poverty level Theme specific Meta analysis indicators: within themeWhat needs to be Theme  Market access learned at the level Database at  Productivity theme level? project level  Science and tech.  Data Analysis within themesWhat needs to be Project Quantitative learned at the Project level Indicators: project level?  Process indicators Qualitative  Outcome indicators analysis
  19. 19. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  20. 20. Impact Evaluation:Concepts Impact evaluation hinges on determining what would have happened if the program had not existed. Good practice involves a comparison of outcome before and after intervention with those with and without intervention Problem is identifying valid counterfactual
  21. 21. Impact Evaluation:MethodsQuantitative Methods cost reliability  Pre and post intervention, no control group  Pre and post intervention, with control group, but no randomization  Pre and post intervention, with control group and randomization Qualitative Methods - complementary – help: Interpret of quantitative results Identify unexpected impacts, or effects on groups that are not captured by quantitative surveys, etc.
  22. 22. CounterfactualIdeally: Observe the outcome variable for those in the program and For those same individuals had they NOT participated in the program (the counterfactual) So, constructing the counterfactual is the key issue that any empirical method must effectively handle.
  23. 23. Impact Evaluation: Finding a Counterfactual Before the program After the program A: “Treatment” Status Beneficiaries: E: Status before the program Real B: “ Non Treatment” Status Counterfactual C: “T reatment” StatusNon-Beneficiaries: F: Status before the program Estimated D: “ Non Treatment” Status Counterfactual Shaded boxes are Unshaded boxes are Observable situations Unobservable Concept: How is the outcome different than it would have been if the project had not been implemented? = A – B (but cannot be observed) So estimated impact is based on double difference: (A-E) – (D-F)
  24. 24. Supposed we observe an increase in outcome Y forbeneficiaries over time after an intervention (observed) Y1 Intervention Y0 baseline(t0) follow-up(t1)Page 24
  25. 25. To measure impact, we need to remove the counterfactualfrom the observed outcome (observed) Y1 Impact= Intervention Y1-Y1* Y1* (counterfactual) Comparison Y0 baseline(t0) follow-up(t1)Page 25
  26. 26. Treatment Effects: key obstacles Experimental vs. Non-Experimental Data  Experimental data rules out self-selection into the program (according to observables or unobservables) as a source of bias in measuring the treatment effect  So, this contribution of experimental data brings into high relief the two key obstacles that non- experimental data methods must overcome in order to avoid biased estimates of the average treatment effect:
  27. 27. Treatment Effects: Key obstacles (cont) 1. Self-selection into the program due to observables characteristics 2. Self-selection into the program due to unobservable characteristics Accounting for #1 is often difficult (or impossible) to accomplish. Even if #1 is accounted for in the method but # 2 is not, then bias in the result will inevitably occur.
  28. 28.  Similarly if the control and treatment groups are randomly selected from a population then there is no bias in the initial characteristics The impact of the procedure X can be attributed to the differences in the variable Y between the control and treatment group. Treatment Group Random (receives procedure Population selection X) Y Exp – Y Control Control group (not receives procedure X)
  29. 29.  Although normally experimental methods are not applied ¿Why we can not then apply a direct comparison between the control and treatment group? Because differences in characteristics of subjects, or what is called selection bias. NO random Treatment Group Population (receives procedure selection X) Because of initial differences between both groups, the effects of the treatment can not be identified by directly comparing the groups Quintile I Quintile II Quintile III Quintile IV QuintileV Control group (more poor) (more richer) (not receives procedure X)
  30. 30. Selection bias: “Graphically” Observed difference (G) Impact on the treated (ATT) = true effect of the program on its recipients Selection Bias (SB)Observed SB = 0 SB > 0 SB < 0 G G=ATT G>ATT G<ATT No selection bias Selection on Selection on “better-off” with “worse-off” with respect to the respect to the outcome outcome
  31. 31. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  32. 32. Overcoming selection bias Ex-ante  Experimental approach: the design of the program allows to introduce randomness in its allocation Ex-post  Natural experiment approaches: there are events that allow to simulate “exogeneity in the choice of treatment”  Control approaches: try to neutralize (reduce) as much as possible the selection bias
  33. 33. Experimental approaches Randomly allocate “Treatment” into a population.  Eliminates selection bias: E  YiC | T  = E  YiC | C   SB  0, G  ATT     Sometimes ethical critics  If the exclusion of some beneficiaries is only due to the evaluation, while benefits are well known  In reality, resource constraints are the limiting factor. Then, random selection can be considered a fair process (every potential beneficiary has same chance of being selected) Must be designed before the start of the program Remains the best approach.
  34. 34. How to randomize? Randomize program as a whole.  E.g. oversubscription: when there are limited supply and excess demand  select recipients by lotteries. Randomize phasing-in  Program cannot reach all intended beneficiaries the first year.  select first year recipients randomly Randomize encouragement.  Cannot randomize treatment for ethical or practical reasons.  Randomly allocate encouragement (e.g. vouchers). Only increases the probability that a treatment is received without changing it from zero to one  specific analytical challenges (partial (or imperfect) compliance).
  35. 35. Natural experiment approaches Use the fact that the program was allocated to some potential beneficiaries and not to others, for reasons that have nothing to do with the outcome itself.  Find variable that is strongly linked to participation (fully or partially) but not to outcome. Pipeline comparisons when administrative delays.  Compare current participants to prospective participants who also qualify. Regression discontinuity when program selection based on clear threshold on a given variable.  Compare people just before threshold to people just above. Instrumental variables  Use predicted participation as given by a variable linked to participation but not to outcome
  36. 36. Limitations of These Methodsof Impact Analysis Impact evaluation focuses on program benefits, ignoring costs. Measures one side of cost effectiveness. This limitation provides motivation for cost studies (Caldés, Coady and Maluccio, 2004) Methods provide estimates of average impact in a ‘black box’ form. Good for demonstrating impact, but limited for broader policy analysis (Ravallion, 2005)Page 36
  37. 37. Controls approaches Matching: compare people with similar ex-ante observable characteristics  Control for the effects of observable characteristics that may affect hh outcome.  Assumption: All components of selection bias are observable and measured (no omitted variables). Difference in difference: compare the evolution of the hh with treatment to the evolution of the hh without treatment  Neutralize time-invariant individual characteristics (observable and unobservable).  Neutralize effect due to other external events that may have affected outcome since the program started.  Assumption: absent the treatment, the outcomes in the two groups would have followed parallel trends Mixed: difference in difference on matched households
  38. 38. Summary Problem with “before / after” measure Welfare Difference could be driven by other events measure Problem with “with / without” measure 1’ 1 Difference could be driven by selection2 Double difference 1: “differences in evolution” Impact = (1) – (2). Controls for other events and self selection if the latent heterogeneity is additive and time invariant. Double difference 2: “differences in evolution” With Without With Without Impact = (1’) – (2). Before… After… Where initial differences are controlled for. E.g. matching and difference in difference If randomization or natural experiment approach, then original differences should not exist. Insuch cases, with/without measures can be sufficient
  39. 39. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  40. 40. Different Levels of Aggregation:A Common Evaluation Measure ∑ theme level + complementarities – Program level substitution (potential GE ERR, PRRR effects) ∑ project level + complementarities - Theme level ERR, PRRR substitution Careful evaluation at this level is the Poverty Project Level: Economic Reduction foundation of Rate of Rate of - Program Logic Diagram higher-level Return Return - Impact evaluation evaluations ERR PRRR - Cash flow analysis
  41. 41. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  42. 42. The concept of (stochastic) profit frontier to assure external validity This approach is based on a simple economic concept: the Milk production Production Possibility Frontier (PPF). Production All the possible production Possibility Frontier combinations are found within the PPF. Outside of the boundary are combinations which are not C achieveable under current Corn conditions production The efficient use of resources is along the boundary.
  43. 43. Accessibility AltitudeWater bodies RoadsLand use
  44. 44. Advantages of Micro-Region Typology Productive projects differentiated to Conditional Cash Transfers and meet local needs and problems Nutritional Programs The inclusion of socioeconomic characteristics and access in What are the principal differences the analysis allows for the between high and low efficiency identification of bottlenecks in households in the area? areas of high potential but low or medium efficiency Productive and Efficiency High potential and low Low potential and low potential based on market,Typology average efficiency socioeconomic, bio-physical average efficiency and access characteristics.Diagnosticfrom High poverty areas High poverty areasPovertymap 44
  45. 45. Advantages of a micro-region typology: classification Micro-Regions Poverty Potential EfficiencyCritical, lacking agricultural potential High Low High-Medium-LowMedium priority, no agricultural opportunities Medium Low High-Medium-LowLow priority Low Low High-Medium-LowHigh priority High Medium-High High-Medium-LowMedium priority, with agricultural opportunities Medium Medium-High Medium-LowLow priority, with agricultural opportunities Low Medium-High Medium-lowHigh performance Low Medium-High High
  46. 46. Estimation Methodology Estimation inputs INSUMOS PARA LA ESTIMACION ESTIMACION  Estimation Estimation output OUTPUT DE LA ESTIMACIÓN      Potential Potencial:   Precios de  Prices of products (P) and productos (P) y insumos (W),  Pesos asignados a los Step 1:PASO 1:  inputs (W), profits reported by Econometría  Weights assigned to ESTIMACION    beneficios reportados por el  Econometric Model insumos de acuerdo a Estimation household (π). Modelo de  inputs following (NIVEL DEL (Household hogar (π).  of the teoría económica y  fronteras  economic theory and  Level) HOGAR)  stochastic Efficiency Land, value of estocásticas de  empirical evidence evidencia empírica Eficiencia :  tierra, valor de  Profit frontier activities, socioeconomic beneficios   los activos, características  characteristics (Z), PASO 2:      biophysical conditions (G), socioeconómicas (Z),  Step 2: PREDICCIÓN  Prediction   market access (A). condiciones biofísicas (G),  (NIVEL  (Regional acceso a mercado (A). REGIONAL)  Level)   RESULTADO DE LA PREDICCION  INSUMOS PARA LA PREDICCION RESULTADO FINAL  Final result Prediction result Prediction inputs   Resultado de la estimación  Potencial productive a  Estimation results (weights) Productive potential at the (pesos)    nivel regional; eficiencia  regional level; Potencial  Boundary Product prices (P) and Frontera:  Precios de productos  Region level de acuerdo a las    productive y  Efficiency according to inputs (W) productive (P) y insumos (W).  características  socioeconomic potential and Pesos eficiencia a    characteristics, biophysical socioeconómicas,  Efficiency: Land, value of efficiency Eficiencia: tierra, valor de  nivel  conditions, market access activities, socioeconomic condiciones biofísicas,  within the area activos características  characteristics (Z), biophysical   regional  acceso a mercado dentro  conditions (G), market access socioeconomicas (Z),  (A).   del área  condiciones biofísicas (G),    acceso a mercado (A).     
  47. 47. Recap (1)… Targeting Criteria Data based on Efficiency Estimated Geo Layers cost of Market Access PPF: Input, Output, Profits Agricultural Profit Frontier Variable X Variable Z .4 1 Group 1 Available datasets: Group 2 Efficiency in .3 Land characteristics, Agricultural .9 Cumulative Density biophysical conditions, Density Group 1 .2 Profits Group 2 socioeconomic .8 characteristics, assets, .1 market access, etc. .7 0 4 6 8 10 12 0 2 4 6 8 10 values X Values Z
  48. 48. Recap (2)… MULTIPLE TARGETINGEfficiency Allocation DIMENSIONS Criteria Typology combines all these criteria Equity Allocation Criterion
  49. 49. Recap (3)…Recall the initial objective…. Micro-Regions Critical, lacking agricultural potential Low potential and low average efficiency Medium priority, no agricultural opportunities Low priority High priority High potential and low Medium priority, with agricultural opportunities average efficiency Low priority, with agricultural opportunities High performance 49
  50. 50. Recap (4): Groupingdiverse criteria into seven microregions…
  51. 51. Recap (5)… How does this translate into policies? High potential and low average efficiency What are the principal differences between high and low efficiency households in the area? Productive projects differentiated to meet local needs and problems
  52. 52. Recap (6)… How does this translate into policies? Low potential and low average efficiency Conditional Cash Transfers and Nutritional Programs
  53. 53. Can be applied to other settings? Guatemala Cost of Market Agricultural Profit Access Frontier Efficiency in Poverty Map Agricultural Profits
  54. 54. Guatemala: Seven-Class Typology With agricultural potential Without agricultural potential
  55. 55. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  56. 56. Some examples Extension services Market information Infrastructure in rural areas Property rights – land titling
  57. 57. Some examples Extension services Market information Infrastructure in rural areas Property rights – land titling
  58. 58. Diff-in-Diff, FE (Dercon et al 2008) Impact of road quality improvements and increased access to agricultural extension services on consumption and poverty in rural Ethiopia. Dependent variables: household is poor, consumption growth Treatment: receiving at least one extension visit, and access to all- weather roads (=1 if road to nearest town is all-weather road) Identification: IV model using GMM and controlling for household fixed effects Instrument for consumption in time t-p: fertile land holdings, number of adult equivalents and number of livestock units (all in logs) at time t- p. Receiving at least one extension visit reduces headcount poverty by 10 percentage points and increases consumption growth by 7 percent. Access to all-weather roads reduces poverty by 6.9 percentage points and increases consumption growth by 16.3 percent. Ex post Supply driven
  59. 59. Some examples Extension services Market information Infrastructure in rural areas Property rights – land titling
  60. 60. Page 60
  61. 61. Institutional arrangement for a simpleprice information systemSource: Hernanini (2007), World Bank
  62. 62. Flow of information and Institutionalagreements for virtual markets Source: Hernanini (2007), World Bank
  63. 63. Pseudo randomized IV - Mobile phones: the impact of Cellphones on grain Markets in Nigeria (Jenny C. Aker - 2008)  Due partly to costly information, price dispersion across markets is common in developed and developing countries  Between 2001 and 2006, cell phone service was phased in throughout Niger, providing an alternative and cheaper search technology to grain traders and other market actors  The author constructs a novel theoretical model of sequential search, in which traders engage in optimal search for the maximum sales price, net transport costs  The model predicts that cell phones will increase traders’ reservation sales prices and the number of markets over which they search, leading to a reduction in price dispersion across markets.  To test the predictions of the theoretical model, they use a unique market and trader dataset from Niger that combines data on prices, transport costs, rainfall and grain production with cell phone access and trader behavior. Page 63
  64. 64. Main results  The results provide evidence that cell phones reduce grain price dispersion across markets by a minimum of 6.4 percent and reduce intra-annual price variation by 10 percent  Cell phones have a greater impact on price dispersion for market pairs that are farther away, and for those with lower road quality. This effect becomes larger as a higher percentage of markets have cell phone coverage.  They provide empirical evidence in support of specific mechanisms that partially explain the impact of cell phones on market performance.  The primary mechanism by which cell phones affect market- level outcomes appears to be a reduction in search costs, as grain traders operating in markets with cell phone coverage search over a greater number of markets and sell in more markets.  The results suggest that cell phones improved consumer and trader welfare in Niger, perhaps averting an even worse outcome during the 2005 food crisis. Page 64
  65. 65. Pseudo randomized IV: Internet: Internet kiosks in India toprovide wholesale price information (Aparajita Goyal, 2008) Beginning in October 2000, it set up 1700 internet kiosks and 45 warehouses in Madhya Pradesh that provide wholesale price information and an alternative marketing channel to soybean farmers in the state Dependent variables: wholesale price of soybeans, sales in traditional markets, soybean cultivation Treatment: presence of internet kiosks and price warehouses Identification: variation in timing of the introduction of kiosks and warehouses Equivalent to randomization at the village level Ex post Demand driven
  66. 66. Main results  The estimates suggest an immediate and significant increase in the monthly wholesale market price of soybeans by 1-5 percent after the introduction of kiosks, lending support to the predictions of the theoretical model  While the presence of warehouses appears to have no effect on price, warehouses are associated with a dramatic reduction in the volume of sales in the traditional markets  Moreover, there is a significant increase in the area under soy cultivation. The estimates are robust to disaggregated measures of treatment and comparisons with alternative crops grown in the same season as soy  The results suggest that information can enhance the functioning of rural markets by making buyers more competitive. Page 66
  67. 67. Some examples Extension services Market information Infrastructure in rural areas Property rights – land titling
  68. 68. Pipeline comparisons when administrative delays (Torero2008) A B C D ETable 1. Timeline Secti Scheduled Start Scheduled End on Date Date The road to be improved was split A July 2008 June 2010 in the following segments: B Completed Completed C July 2008 December 2009 D October 2008 April 2010 E October 2008 February 2010 Table 2. Treatment and Control Groups Test Control Treatme Number Group nt Group Based on the geographic location and the 1 B A timelines, the following treatment-control groups are suggested: 2 B C 3 D C 4 E D
  69. 69. Pipeline comparisons when administrative delays
  70. 70. Pipeline comparisons when administrative delays.
  71. 71. Randomized- Barriers to connection in Ethiopia (Bernard andTorero 2009)  Connection fees range between USD 50 and USD 150 (drop down line and meter). Need to find ways to facilitate connection for the poorer.  Can CFL (energy-saving light bulb) positively influence energy use? How to promote the use of energy-saving light bulbs (consumes 4 times less, but costs 8 times more)?  What is best: 2 years loan or 5 years loan for connection fee?  Pilot study on 20 towns to assess optimal subsidies.  Experimental approach (randomize encouragement through distribution of vouchers).
  72. 72. This image cannot currently be display ed.
  73. 73. Public distribution
  74. 74. Random selection… Public distribution
  75. 75. Some examples Extension services Market information Infrastructure in rural areas Property rights – land titling
  76. 76. Matching, IV on cross-sectional data - Landproperty rights on productivity (- Markussen 2008) Dependent Variable: (log) value of output per hectare Treatment: The plot is held with a paper documenting ownership (titles, application receipts) Identification: IV mode of plot acquisition (dummies to indicate if the plot was given by the State, inherited, bought, donated, occupied for free) as instrument for the dummy “plot held with paper” Ex-post Demand driven: households and landholders apply for titles
  77. 77. Pseudo-Randomized - Land titling on ruralhouseholds (Torero and Field 2007) Dependent variables: household expenditure, change in rent/market value of dwelling, risk of expropriation, production, trade of land, collateral and credit markets, land ownership and tenancy, permanent ant transitory crops Treatment: to receive land title Identification: quasi random program implementation, kernel matching Ex post Demand driven, but few requirements and virtually free
  78. 78. The DatabaseThe survey covered 3204Peruvian rural households:521 from rural coast, 1622from rural highlands and1061 form rural jungle.The next map plots thetowns covered by the surveyand the valleys reached bythe PETT program. Fromthese 3204 households 1793match with at least oneprevious national survey.
  79. 79.  If control and treatment groups are randomly selected from a population then there is no bias in the initial characteristics The impact over income can be attributed to the access to title Treatment Group Random (receives procedure Population selection X) Y Exp – Y Control Control group (not receives procedure X)
  80. 80.  ¿Why we can not then apply a direct comparison between the control and treatment group? Because differences in characteristics of subjects, or what is called selection bias. Pseudo Treatment Group Population (receives procedure random selection X) Because of initial differences between both groups, the effects of the treatment can not be identified by directly comparing the groups Quintile I Quintile II Quintile III Quintile IV QuintileV Control group (more poor) (more richer) (not receives procedure X)
  81. 81.  Identify comparable pairs (with similar initial characteristics) and that differ only on the procedure We will use Propensity score matching. Find the pair to assure comparability Treatment Population pseudo random selection Impact of the procedure Control
  82. 82. Outline1. Need for impact evaluation2. Impact evaluation and monitoring3. Guiding principles for our impact evaluation approach4. Impact evaluation concepts5. Impact evaluation methods6. Aggregating impact evaluation results7. A methodology for external validity8. Some examples9. Final comments
  83. 83. Final comments The impact evaluation must be a part of the program design  It is very important to identify how to incorporate it now that the program already exists  For new programs it is necessary to invest in the design so that an impact evaluation is also part of it It is essential to identify the impact pathways, i.e. the expected causal chain of events leading from project activities to outputs, to changes in the target population, and to the achievement of project objectives Since the beginning of the program is necessary to specify the expected “outcomes” and the control group
  84. 84. Final comments It will be ideal to have an autonomous and external laboratory of impact evaluation Communications among all stakeholders is central Not all interventions need to be evaluated, it will be ideal to do it before scaling up so there is assurance that the intervention works Alignment of proper incentives – to contractors, evaluators, to implementers and to USAID country offices Finally, policy requires a causal model; “without it, we cannot understand the welfare consequences of a policy” (Deaton 2009)
  85. 85. Recommended readingsCaldés, Natalia, David Coady and John Maluccio. 2004. The Cost of Poverty Alleviation Transfer Programs: A Comparative Analysis of Three Programs in Latin America. IFPRI FCND Discussion Paper No. 174., Washington, DC.Duflo, Esther; Rachel Glennerster and Michael Kremer (2007):“Using Randomization in Development Economics Research: a Toolkit”CEPR discussion paper no. 6059Feder et al. 2004. Review of Agricultural Economics.Godtland et al. 2004. Economic Development and Cultural Change.Heckman, J.J., H. Ichimura, and P.E. Todd. 1997. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program.” Review of Economic Studies 64:605-654.Hirano and Imbens. 2004. The Propensity Score with Continuous Treatments. In Gelman & Meng, eds.Miguel and Kremer. 2004. Econometrica.Ravallion, Martin. 2005. Evaluating Anti-poverty Programs. World Bank Working Paper Series 3625, Washington, DC. Martin Ravallion (2003): “The Mystery of the Vanishing Benefits: An introduction to Impact Evaluation” The World Bank Economic Review, volume 15, no 1, pp115-140Page 85

×