Transcript of "Alessandra Garbero: Challenges of impact evaluation"
Challenges to impact evaluation: Solutions for IFAD Alessandra Garbero, PhD Econometrician SSD, IFAD
IFAD Commitments• Lift 80 m people out of poverty (2010-2015)• Reach 90 m beneficiaries (direct/indirect)• Challenge: how to measure quantitative impact?• Solution: Impact evaluation - Attribution 1. 30 rigorous evaluations - Feedback 2. Retrospective evaluations using non- experimental methods and IFAD/RIMS data - Accountability - Learning
IFAD’s projects portfolio: characteristics• Relatively small projects• Multiple interventions/components• Eligibility criteria not systematically related to IFAD 9R indicators• Multiple treatments (same beneficiaries for different projects)• Suboptimal baseline compliance• RIMS policy does not foresee control group and panel structure• RIMS does not collect income/expenditure dataNon-experimental methods
Methodological challenges for impactevaluation1. Enhance internal validity: the lack of comparison group prevents from causal inference between variables - can be eliminated ex-ante or dealt within the analysis ex-post2. Enhance external validity: extrapolate study’s results to other project areas - randomly selecting sites & within these sites randomly selecting treatment and comparison groups3. Purposive targeting of project beneficiaries: differences between participating and non-participating households (endogeneity)4. No income/expenditure data: use appropriate techniques with poverty proxies (small-area estimation methods or alternative poverty proxy methods)
Two empirical applications from IFAD’s projects• Focus on changes in expenditure-based poverty status• Can we measure these changes with the current data?• Shall we think about using poverty prediction methods? (regression models) Vietnam (DPPR): Decentralised Nicaragua (Prodesec): Programa Programme for Poverty de desarrollo económico de la Reduction región seca de Nicaragua Components: Capacity Building Components: Promote & Finance for Decentralization process; Business and Employment; Rural Production supports; Development Financial services; Strengthen Rural of rural villages’ small infrastructure Development Policy • Baseline 2006 • Baseline 2005 • Completion 2011 • Completion 2011 • LSMS 2002 – only available • LSMS 2005 – only available dataset at the time of the analysis dataset at the time of the analysis
From assets to expenditure-based poverty status• Regression-based method (OLS): A prediction model that estimates expenditure based on household characteristics (i.e. poverty explanatory factors i.e. “predictors”) using the LSMS• Poverty “predictors”: - Vietnam: HH size; education of HH head; sex head; age head; assets (vehicle, refrig., bike, moto radio, tv; toilet type and source of drinking water) - Nicaragua: HH size; gender of HH; electricity; toilet type; source of drinking water; farm HH; type of fuel; type of floor material.• Model selection: conditional on sig., R squared (0.60 for Vietnam vs. 0.40 for Nicaragua), presence of variables in both surveys• Limitations: - Vietnam: inferring poverty predictors based on 2002 LSMS relationships between expenditure and key predictors - Nicaragua: 2005 LSMS• Definition of poverty line: set at the 30th percentile of rural households (as in Minot 1998).
Results from poverty mapping: howaccurate is our model?• Comparison between predicted and actual poverty status using the OLS and the chosen poverty line - Vietnam: the model identifies 72% of the observed poor - Nicaragua: the model identifies 55% of the observed poor• The model performs better for Vietnam
Impact evaluation• IE Before after: compare changes in impact indicator before and after the project - the counterfactual is represented by the same group before they got the program• What are the potential problems with this? - Other factors contribute to change over time!• Other secondary datasets (2 points in time) needed to assess trends in the area (reconstruct counterfactuals)
Poverty reduction? A Naïve comparison Vietnam: RIMS Nicaragua RIMS• Model results: Poverty declined from • Model results: Poverty declined from 35% to 9% implies 25% of the 52% to 47% implies 5% of the sample lifted out of poverty sample lifted out of poverty Problems • Model based estimates of poverty based on limited available data • Naïve comparison: no impact attribution, no control group, no panel • Macroeconomic factors: • Poverty declined from 45% to 18% in the whole Vietnamese province. • Minor impact of project on relative wealth also based on assets (Nicaragua). Selection bias possible or longer term impacts
Way forwardRigorous ex-post evaluations need adequate secondary datasources for both poverty prediction and matching exercises toreconstruct the counterfactualTheory-based impact evaluation designs need to be mainstreamedwithin IFAD’s projects designsFuture data collection efforts: ensure adequate targeting - Piggy back on national surveys (LSMS/NSOs) if overlap. - Sample beneficiaries and non-beneficiaries within existing efforts. OversamplingIf no national surveys underway - Use an up to date sampling frame - Randomize! Treatment and control (control group larger than treatment) - Core questionnaire with poverty predictors
Way forwardRigorous impact evaluation require:• Commitment• Technical & analytical capacity• ResourcesWorth it?• Strategic relevance - Contribute to global understanding of agricultural pathways out of poverty• Increase evidence of well-functioning programs• Interventions that work for scaling up