Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How do impact evaluations improve
development operations and policy?
David K. Evans, World Bank
Africa Chief Economist’s O...
How do impact evaluations affect operations and policy?
Own project
• Clarify thinking about the theory of
change
• Improv...
I’m going to tell you some
stories (and show some
analysis).
But how do we measure impact?
2
Isn’t impact evaluation about...
Measuring the impact of impact evaluation
Second cousin to the event study
• Before-after is a poor evaluation
method
– Be...
Indicators for IE impact
3ie
• Closing knowledge or evidence
gaps?
• Changes in policy or programs?
• Key stakeholders and...
• Impact evaluation helps clarify thinking on exactly
how one gets from inputs to impacts
• By focusing on questions of
– ...
Own project
Improve quality of data systems
• Kenya Judicial Performance IE
– Impact of incentives for greater
efficiency
...
• Sample of 100 impact
evaluations, 1000+ projects
between 2005 and 2011
• Include time, sector, country,
and project lead...
• Notably, this increase in
disbursements happens after
impact evaluation begins
• Can’t completely control for
selection ...
Own project
Shifts in project design
Ethiopian Nile Irrigation and Drainage Project
• Baseline survey in Ethiopia
• Women ...
Own project
Influence scale-up of projects
Financial Literacy in Brazil
• Pilot in high schools
• Modest but positive, sig...
Other projects
Design of new projects
Projects reports that IEs influence design
• Education Modernization Project in
Kaza...
• Zambia Essential Drugs Public Pilot Program
• Consistent problem with stockouts of essential malaria drugs
• Randomized ...
Other projects
Culture of evidence
Mexico
• Progresa/Oportunidades 1997-2000
• 2000 Congress’ Degree: Annual
external eval...
Other projects
Contribute to a body of knowledge that changes how we do
development
Small questions or big questions?
• Sh...
3ie. “Policy Influence Plan.” 2012. http://evaluationvigie.fr/wp-content/uploads/2014/05/2012_PIP_template.docx
Ali, Danie...
i2i. Changing Policy through Research: DIME IE Portfolio in Kenya. n.d.
i2i. Selected Examples of DFID’s Contribution to E...
Upcoming SlideShare
Loading in …5
×

How do impact evaluations improve operations and policy? 9 ways and 15 examples

1,247 views

Published on

In this presentation, I explore 9 different ways that impact evaluation affects operations and policy, with examples.

This presentation was at a 3ie/IFAD conference on June 1, 2016, in Rome, Italy. References to the studies cited are on the last two slides.

Published in: Economy & Finance
  • Be the first to comment

How do impact evaluations improve operations and policy? 9 ways and 15 examples

  1. 1. How do impact evaluations improve development operations and policy? David K. Evans, World Bank Africa Chief Economist’s Office June 1, 2016 | 3ie/IFAD devans2@worldbank.org | @tukopamoja
  2. 2. How do impact evaluations affect operations and policy? Own project • Clarify thinking about the theory of change • Improve quality of data systems • Improve efficiency of projects • Shifts in project design • Influence scale-up of projects Other projects • Design of new projects • Choosing between designs • Culture of evidence • Contribute to a body of knowledge that changes how we do business 1 Evans|WorldBank|impactevaluations,operations&policy
  3. 3. I’m going to tell you some stories (and show some analysis). But how do we measure impact? 2 Isn’t impact evaluation about careful causal inference? Where’s the counterfactual? Image source: http://imperialcentrepaintingstudio.blogspot.com/2012/02/reading-comprehension-teach-your-child.html Evans|WorldBank|impactevaluations,operations&policy
  4. 4. Measuring the impact of impact evaluation Second cousin to the event study • Before-after is a poor evaluation method – Because usually many other things are changing • But useful when no other credible cause Embrace the ambiguity • The best impacts from impact evaluation can come when the resulting decisions are completely internalized by government (Goldstein 2016) • Short of an RCT of RCTs, embrace the ambiguity. 3 Jensen (2007) Evans|WorldBank|impactevaluations,operations&policy
  5. 5. Indicators for IE impact 3ie • Closing knowledge or evidence gaps? • Changes in policy or programs? • Key stakeholders and key influencers? • Plan to engage with them at each stage of the IE? • Cleary theory of change for policy influence? DIME (World Bank) • Capacity building – IE results discussed with client – Training provided for data analysis • Quality of data – Availability of data – Improved administrative data • Quality of policy decisions – Used to motivate scale-up/scale-down of policy – Adopted delivery mechanism based on IE results – Baseline informed design or implementation 4 Source: Legovini 2014Source: 3ie 2012 Evans|WorldBank|impactevaluations,operations&policy
  6. 6. • Impact evaluation helps clarify thinking on exactly how one gets from inputs to impacts • By focusing on questions of – Causality – Measurement • Impact evaluation improves theories of change Own project Clarify thinking about the theory of change 5 Figure sources: Imas & Rist 2009 Evans|WorldBank|impactevaluations,operations&policy
  7. 7. Own project Improve quality of data systems • Kenya Judicial Performance IE – Impact of incentives for greater efficiency – Co-developed tool for measuring efficiency of courts: How quickly do they get their cases out – Tool was adopted for all courts as official monitoring tool • Kenya Patient Safety IE – Impact of inspections of health facilities – Reform to regulatory framework of inspection system for minimum patient safety. – Design of quality of care measures 6 Sources: World Bank 2013; i2i n.d. (1); i2i n.d. (2) Evans|WorldBank|impactevaluations,operations&policy
  8. 8. • Sample of 100 impact evaluations, 1000+ projects between 2005 and 2011 • Include time, sector, country, and project leader effects • Significant increase in projects carrying out their activities Own project Improve efficiency of projects 7 Source: Legovini et al. 2015 Evans|WorldBank|impactevaluations,operations&policy
  9. 9. • Notably, this increase in disbursements happens after impact evaluation begins • Can’t completely control for selection into project • BUT suggestive that IEs help projects roll out and that they don’t slow projects down (as commonly claimed) Own project Improve efficiency of projects 8 Source: Legovini et al. 2015 Evans|WorldBank|impactevaluations,operations&policy
  10. 10. Own project Shifts in project design Ethiopian Nile Irrigation and Drainage Project • Baseline survey in Ethiopia • Women have – Lower access to extension – Greater time poverty – Much lower agricultural value-added for women than men • Result: Pilot strategies in Farmer’s Initiative Fund, designed for female farmers 9 Source: World Bank 2014 Evans|WorldBank|impactevaluations,operations&policy
  11. 11. Own project Influence scale-up of projects Financial Literacy in Brazil • Pilot in high schools • Modest but positive, significant impacts • Scale-up – High schools around the country – Elementary schools Other example: Land titling in Rwanda Conditional Cash Transfers • Mexico – Initial program showed positive results – Expansion to urban – Survival across administrations • PROGRESA  Oportunidades  Prospera • Tanzania – Pilot showed positive results – Key evidence informing scale-up 10Source: Dhaliwal & Tulloch 2012; Mkelemi & Evans 2016Source: Bruhn et al. 2013; Ali et al. 2011 Evans|WorldBank|impactevaluations,operations&policy
  12. 12. Other projects Design of new projects Projects reports that IEs influence design • Education Modernization Project in Kazakhstan – Influenced by synthesis of education impact evaluations • Adolescent Girls Empowerment Project in India – Influenced by IEs of programs to empower adolescent girls in Liberia and Uganda Citations of IEs in World Bank documents ↑ 6x 11 3,691 21,271 0 5,000 10,000 15,000 20,000 25,000 05 06 07 08 09 10 11 12 TotalIECitations Fiscal Year Source: Legovini 2014Source: Gender Innovation Lab 2015 Evans|WorldBank|impactevaluations,operations&policy
  13. 13. • Zambia Essential Drugs Public Pilot Program • Consistent problem with stockouts of essential malaria drugs • Randomized impact evaluation to test two different models • A: District planner • B: Direct orders from clinic to central medical stores Other example: Early child development in the Gambia Other projects Choosing between designs 12 Source for Zambia: Vledder et al. 2015; World Bank 2010 Evans|WorldBank|impactevaluations,operations&policy Source for the Gambia: Blimpo & Carneiro 2014
  14. 14. Other projects Culture of evidence Mexico • Progresa/Oportunidades 1997-2000 • 2000 Congress’ Degree: Annual external evaluations for projects • 2004 Social Development Law: National Council for Evaluation of Social Policy (CONEVAL) Tanzania • Two donor-initiated impact evaluations under TASAF-2 – Community-based conditional cash transfers – Vulnerable groups • Three client-initiated impact evaluations under TASAF-3 – Scaled-up cash transfers – Cash transfers on women’s empowerment – Cash transfers on transition to adulthood • False dichotomy: Donor-initiated vs client-initiated. But a shift along the spectrum. 13Source: Hernández Licona 2008 Evans|WorldBank|impactevaluations,operations&policy
  15. 15. Other projects Contribute to a body of knowledge that changes how we do development Small questions or big questions? • Should we give away insecticide treated nets or charge for them? • Higher prices – lower use – don’t improve targeting – don‘t increase use • Between 2000 and 2015: “450 million cases of malaria and 4 million deaths averted from ITN distribution” Many small studies add up to big answers Pricing experiments around the world 14Source: Dupas & Miguel 2016Source: Glennerster 2016 Evans|WorldBank|impactevaluations,operations&policy
  16. 16. 3ie. “Policy Influence Plan.” 2012. http://evaluationvigie.fr/wp-content/uploads/2014/05/2012_PIP_template.docx Ali, Daniel, Klaus Deininger, and Markus Goldstein. “Environmental and gender impacts of land tenure regularization in Africa: pilot evidence from Rwanda.” World Bank Policy Research Working Paper 5765. 2011. http://documents.worldbank.org/curated/en/2011/08/14833489/environmental-gender-impacts-land-tenure-regularization-africa- pilot-evidence-rwanda Blimpo, Moussa, and Pedro Carneiro. “Early childhood development: Some preliminary findings to guide future actions.” PowerPoint Presentation. 2014. Bruhn, Miriam, Luciana de Souza Leao, Arianna Legovini, Rogelio Marchetti, and Bilal Zia. “The impact of high school financial education: experimental evidence from Brazil.” World Bank Policy Research Working Paper 6723. 2013. http://documents.worldbank.org/curated/en/2013/12/18640673/impact-high-school-financial-education-experimental-evidence- brazil Dhaliwal, Iqbal, and Caitlin Tulloch. “From research to policy: using evidence from impact evaluations to inform development policy.” 2012. https://www.povertyactionlab.org/sites/default/files/publications/With%20JDE%20Revisions%20and%20Footnotes%202012.09. 03.pdf Dupas & Miguel. “Impacts and Determinants of Health Levels in Low-Income Countries.“ Prepared for the Handbook of Field Experiments. 2016. http://emiguel.econ.berkeley.edu/assets/miguel_research/73/Health_DupasMiguel_Handbook.pdf Gender Innovation Lab. Africa Gender Innovation Lab (GIL) Influence. November 2015. Glennerster, Rachel. “Not so small.” Running Randomized Evaluations blog. May 27, 2016. http://runningres.com/blog/2016/5/27/not-so-small Goldstein, Markus. Conversation with Dave Evans on a bus. Rome, Italy. May 31, 2016. Hernandez Licona, Gonzalo. “Impact evaluation in Mexico.” January 2008. http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and-Materials/449365- 1199828589096/GonzaloHernandezLicona.pdf References - 1 15 Evans|WorldBank|impactevaluations,operations&policy
  17. 17. i2i. Changing Policy through Research: DIME IE Portfolio in Kenya. n.d. i2i. Selected Examples of DFID’s Contribution to Evidence-Based Policy through i2i. n.d. Imas, Lnda G. Morris, and Ray C. Rist. The Road to Results: Designing and Conducting Effective Development Evaluations. World Bank. 2009. https://openknowledge.worldbank.org/bitstream/handle/10986/2699/52678.pdf Jensen, Robert. “The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector.” Quarterly Journal of Economics 122(3), August 2007. http://www.jstor.org/stable/pdf/25098864.pdf Legovini, Arianna. “The development impact of impact evaluation.” World Bank. 2014. http://www.worldbank.org/content/dam/Worldbank/Feature%20Story/japan/pdf/event/2014/120514_Arianna_Legovini.pdf Legovini, Arianna, Vincenzo di Maro, Caio Piza. “Impact evaluation helps deliver development projects.” World Bank Policy Res earch Working Paper 7157. 2015. http://documents.worldbank.org/curated/en/2015/01/23173058/impact-evaluation-helps-deliver- development-projects Mkelemi, Godfrey, and David K. Evans. “The Challenge and Success of Impact Evaluation for Tanzania Community-Based Conditional Cash Transfer Pilot.” Conference presentation, IFAD/3ie. May 31, 2016. http://www.slideshare.net/DavidEvans198/the-challenge-and- success-of-impact-evaluation-for-tanzania-communitybased-conditional-cash-transfer-pilot Prottas, Chris (@ChrisProttas). “I understand why, but real mental disconnect w anecdotes used to justify impact of impact evals.” May 27, 2016, 3:10am. Tweet. https://twitter.com/ChrisProttas/status/736137568954294272?lang=en Vledder, Monique, Jed Friedman, Mirja Sjoblom, Thomas Brown, and Prashant Yadiv. “Optimal Supply Chain Structure for Distributing Essential Drugs in Low Income Countries: Results from a Randomized Experiment.” Ross School of Business working paper 1269. March 2015. https://deepblue.lib.umich.edu/bitstream/handle/2027.42/110800/1269_Yadav.pdf World Bank. From Evidence to Policy: The Challenge of Ensuring Adequate Stocks of Essential Drugs in Rural Health Clinics. November 2010. World Bank. “Improving patient safety in Kenya.” November 6, 2013. PowerPoint Presentation. https://www.wbginvestmentclimate.org/results/upload/Kenya-PS-BBL_November-6-2013.pdf World Bank. Africa Impact Evaluation Update: Agriculture and Land. March 2014. References - 2 16 Evans|WorldBank|impactevaluations,operations&policy

×