This document discusses challenges in measuring the impact of projects by the International Fund for Agricultural Development (IFAD) and proposes solutions. Key challenges include attributing outcomes to IFAD projects given multiple interventions and lack of baseline data. The document proposes increasing rigorous impact evaluations, using retrospective non-experimental methods, and improving data collection. Two case studies from Vietnam and Nicaragua attempt to measure poverty reduction using existing data but find limitations without control groups or ability to fully attribute changes to projects. The document concludes future evaluations need better data and experimental designs to robustly measure impact.
Setting the scene – Trends in programming Research and Innovation for Impact Francois Stepman
6 April 2018. Rome. The SCAR Strategic Working Groups ARCH, AKIS and Food Systems organised jointly the Workshop: Programming Research and Innovation for Improved Impact
Presentation by Paul Winter
An overview of impact evaluation for organizations based on a program's Theory of Change, highlighting the need for a counterfactual and randomization (when possible) in order to convincingly demonstrate the effect of the program.
I gave this talk at a Nigeria Health Summit in March 2016. It was an introduction to impact evaluation: what it is, when it's a good idea, and some possible approaches.
Setting the scene – Trends in programming Research and Innovation for Impact Francois Stepman
6 April 2018. Rome. The SCAR Strategic Working Groups ARCH, AKIS and Food Systems organised jointly the Workshop: Programming Research and Innovation for Improved Impact
Presentation by Paul Winter
An overview of impact evaluation for organizations based on a program's Theory of Change, highlighting the need for a counterfactual and randomization (when possible) in order to convincingly demonstrate the effect of the program.
I gave this talk at a Nigeria Health Summit in March 2016. It was an introduction to impact evaluation: what it is, when it's a good idea, and some possible approaches.
Gender in monitoring, evaluation and impact assessmentILRI
Presentation by Jemimah Njuki at the FAO-ILRI Workshop on Integrating Gender in Livestock Projects and Programs, ILRI, Addis Ababa, 22-25 November 2011.
A presentation by Respichius Mitti as part of the Practicalities of Cohort and Longitudinal Research panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014
A presentation by Nalini Takeshwar as part of the Cohort Research for Programme and Policy panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014
Monitoring and Evaluation system for PFES: Key findings and policy recommenda...CIFOR-ICRAF
This presentation by Pham Thu Thuy, Karen Bennet, Vu Tan Phuong and Le Ngoc Dung shows the key findings for M&E for Environmental Service, Social Impact, PFES contract and financial flows.
Impact Evaluation Training with AERC: China Cash Transfer Programme Technical...The Transfer Project
A hypothetical technical proposal for China's conditional cash transfer programme from our impact evaluation training with AERC in Nairobi, Kenya in July 2019.
Webinar presentation by Susan Pietryzk. Access the webinar recording at http://www.measureevaluation.org/resources/webinars/measuring-impact-qualitatively
Planning the Evaluation
Impact models
Types of inference and choice of design
Defining the indicators and obtaining the data
Carrying out the evaluation
Disseminating evaluation findings
Working in large-scale evaluations
Two Examples of Program Planning, Monitoring and EvaluationMEASURE Evaluation
Presented by Laili Irani, Senior Policy Analyst for the Population Reference Bureau, as part of the Measuring Success Toolkit webinar in September 2012.
A series of modules on project cycle, planning and the logical framework, aimed at team leaders of international NGOs in developing countries.
Part 7 of 11.
There are two handouts to go with this module, Population Indicators, and a Logframe with blanks. http://www.slideshare.net/Makewa/population-indicators-handout and http://www.slideshare.net/Makewa/exercise-watsan-logframe-with-blanks
Build Your NGO: Monitoring & Evaluation Allie Hoffman
The presentation attached is designed for grassroots NGOs wanting to learn more about monitoring and evaluation.
The presentation is a mini 'how to', in addition to providing an overview of strategic planning
To learn more or with any direct questions, please visit our website: www.thepariproject.com
An 8-step guide to help not-for-profit organizations develop a Project Impact Pathway or Logic Model for their programs - The first presentation in our Project Evaluation Series. By DevMetrica
A presentation by Rachel Hinton as part of the Cohort Research for Programme and Policy panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014
Gender in monitoring, evaluation and impact assessmentILRI
Presentation by Jemimah Njuki at the FAO-ILRI Workshop on Integrating Gender in Livestock Projects and Programs, ILRI, Addis Ababa, 22-25 November 2011.
A presentation by Respichius Mitti as part of the Practicalities of Cohort and Longitudinal Research panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014
A presentation by Nalini Takeshwar as part of the Cohort Research for Programme and Policy panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014
Monitoring and Evaluation system for PFES: Key findings and policy recommenda...CIFOR-ICRAF
This presentation by Pham Thu Thuy, Karen Bennet, Vu Tan Phuong and Le Ngoc Dung shows the key findings for M&E for Environmental Service, Social Impact, PFES contract and financial flows.
Impact Evaluation Training with AERC: China Cash Transfer Programme Technical...The Transfer Project
A hypothetical technical proposal for China's conditional cash transfer programme from our impact evaluation training with AERC in Nairobi, Kenya in July 2019.
Webinar presentation by Susan Pietryzk. Access the webinar recording at http://www.measureevaluation.org/resources/webinars/measuring-impact-qualitatively
Planning the Evaluation
Impact models
Types of inference and choice of design
Defining the indicators and obtaining the data
Carrying out the evaluation
Disseminating evaluation findings
Working in large-scale evaluations
Two Examples of Program Planning, Monitoring and EvaluationMEASURE Evaluation
Presented by Laili Irani, Senior Policy Analyst for the Population Reference Bureau, as part of the Measuring Success Toolkit webinar in September 2012.
A series of modules on project cycle, planning and the logical framework, aimed at team leaders of international NGOs in developing countries.
Part 7 of 11.
There are two handouts to go with this module, Population Indicators, and a Logframe with blanks. http://www.slideshare.net/Makewa/population-indicators-handout and http://www.slideshare.net/Makewa/exercise-watsan-logframe-with-blanks
Build Your NGO: Monitoring & Evaluation Allie Hoffman
The presentation attached is designed for grassroots NGOs wanting to learn more about monitoring and evaluation.
The presentation is a mini 'how to', in addition to providing an overview of strategic planning
To learn more or with any direct questions, please visit our website: www.thepariproject.com
An 8-step guide to help not-for-profit organizations develop a Project Impact Pathway or Logic Model for their programs - The first presentation in our Project Evaluation Series. By DevMetrica
A presentation by Rachel Hinton as part of the Cohort Research for Programme and Policy panel discussion at the International Symposium on Cohort and Longitudinal Studies in Developing Contexts, UNICEF Office of Research - Innocenti, Florence, Italy 13-15 October 2014
Examples of metagenomics use cases for the Phylotastic! web tools. Presented a the Phylotastic hackathon, June 4-8 2012: http://www.evoio.org/wiki/Phylotastic
Science publications used to have the joint function of keeping the 'Minutes of Science' as well as transferring knowledge. The sheer amounts of material published (2 new articles in PubMed every minute of every day) make comprehensive knowledge transfer via reading of articles virtually impossible. When the literature is open, though, much of the essential knowledge it contains can be distilled and the big picture obtained without having to read all the articles, so that reading can then be reserved for those key articles that give insight in the reasoning and argumentation that leads to consensus. The result is a much more efficient knowledge transfer that doesn't have to compromise on comprehensiveness.
http://www.fao.org/save-food/info-resources/en/
FAO's 'Global Initiative on Food Loss and Waste Reduction' has taken a coordinating role, to enhance information exchange, collaboration, synergy and harmonization of strategies and methodologies.
Presentación de Germán A. Flores, PESA en Honduras en el marco del Primer Taller de Trabajo: Desafíos de la Cooperación Sur-Sur para fortalecimiento de la agricultura familiar algodonera, realizado los días 22, 23 y 24 de Octubre de 2013 en Lima, Perú.
El remplazo de alimentos por productos ultra-procesados: ¿que debemos hacer?FAO
Presentación de Jean Claude Moubarac durante la Consulta técnica "Una mirada integral a las políticas públicas de agricultura familiar, seguridad alimentaria, nutrición y salud pública: Acercando las agendas de trabajo en el seno del Sistema de Naciones Unidas", realizada en Lima, Perú, 19 y 20 de Mayo de 2014.
http://www.fao.org/americas/eventos/ver/es/c/231982/
http://www.fao.org/agora
This module is part of the AGORA Basic Course. The AGORA Basic Course highlights the baseline skills necessary to use the AGORA program effectively and efficiently. The AGORA programme (Access to Global Online Research in Agriculture), set up by the Food and Agriculture Organization of the UN (FAO) together with major publishers, enables developing countries to gain access to an outstanding digital library collection in the fields of food, agriculture, environmental science and related social sciences. Module 4 covers CAB Abstracts
AGORA Basic Course: Additional Resources. Tips for TrainersFAO
http://www.fao.org/agora
This module is part of the AGORA Basic Course. The AGORA Basic Course highlights the baseline skills necessary to use the AGORA program effectively and efficiently. The AGORA programme (Access to Global Online Research in Agriculture), set up by the Food and Agriculture Organization of the UN (FAO) together with major publishers, enables developing countries to gain access to an outstanding digital library collection in the fields of food, agriculture, environmental science and related social sciences. The module with Additional Resources includes tips for trainers presenting this course.
AGORA Basic Course: Module 7.3: E-journal, E-books and Internet Resources: Ot...FAO
http://www.fao.org/agora
This module is part of the AGORA Basic Course. The AGORA Basic Course highlights the baseline skills necessary to use the AGORA program effectively and efficiently. The AGORA programme (Access to Global Online Research in Agriculture), set up by the Food and Agriculture Organization of the UN (FAO) together with major publishers, enables developing countries to gain access to an outstanding digital library collection in the fields of food, agriculture, environmental science and related social sciences. Module 7.3 covers E-journal, E-books and Internet Resources (Agriculture Gateways/Portals)
http://www.fao.org/agora
This module is part of the AGORA Basic Course. The AGORA Basic Course highlights the baseline skills necessary to use the AGORA program effectively and efficiently. The AGORA programme (Access to Global Online Research in Agriculture), set up by the Food and Agriculture Organization of the UN (FAO) together with major publishers, enables developing countries to gain access to an outstanding digital library collection in the fields of food, agriculture, environmental science and related social sciences. Module 1 covers Background, partners, eligibility, use, copyright.
Using case-based methods to assess scalability and sustainability: Lessons fr...JSI
Overview of the SC4CCM project and end-line evaluation questions focused on scalability and sustainability. Methodological approaches including case selection strategies, mixed method approaches, within-case and cross-case analysis processes. (Sangeeta Mookherji, GWU)
Using case-based methods to assess scalability and sustainability: Lessons fr...Barb Knittel
Overview of the SC4CCM project and end-line evaluation questions focused on scalability and sustainability. Methodological approaches including case selection strategies, mixed method approaches, within-case and cross-case analysis processes. (Sangeeta Mookherji, GWU)
What Works: Active Labour Market Policies in Latin America and the CaribbeanVerónica Escudero
The findings from the project on active labour market policies in Latin America and the Caribbean were presented to students of the Summer School organized annually by the University of Geneva. Particular emphasis was placed on how impact evaluations of public policies are used as tools to inform policy recommendations in the work of the ILO.
What Works: Active Labour Market Policies in Latin America and the CaribbeanVerónica Escudero
This project presents the first systematic analysis of active labour market policies in Latin America and the Caribbean and assesses their effectiveness in improving labour market and social conditions
Transitioning from reach every district to reach every communityJSI
The presentation describes the expansion for routine immunization from district level to community level in Africa. Reaching remote communities is important to bring immunization to all children.
Presented by Pascale Schnitzer and Carlo Azzarri, IFPRI at the Africa RISING–CSISA Joint Monitoring and Evaluation Meeting, Addis Ababa, Ethiopia, 11-13 November 2013
Margarita beneke conditional cash transfers and rural development in latin am...UNDP Policy Centre
This presentation is part of the programme of the International Seminar "Social Protection, Entrepreneurship and Labour Market Activation: Evidence for Better Policies", organized by the International Policy Centre for Inclusive Growth (IPC-IG/UNDP) together with Canada’s International Development Research Centre (IDRC) and the Colombian Think Tank Fedesarrollo held on September 10-11 at the Ipea Auditorium in Brasilia.
Evaluation of Mozambique’s Child Grant: A cash + care intervention to reduce ...The Transfer Project
Juan Bonilla, Zlata Bruckauf, Rosa Castro-Zarzur & Amber Peterman
On behalf of the Mozambique Child Grant evaluation team
SVRI Forum – Cancun, Mexico – September 22nd, 2022
Ragui Assaad- University of Minnesota
Caroline Krafft- ST. Catherine University
ERF Training on Applied Micro-Econometrics and Public Policy Evaluation
Cairo, Egypt July 25-27, 2016
www.erf.org.eg
NAP Training Viet Nam - Designing and Monitoring Gender Indicators in Climate...UNDP Climate
This two-day workshop supported the Government of Viet Nam in building the necessary capacity to advance its National Adaptation Plan (NAP) process. The workshop closely focused on building National Adaptation Plans in the agricultural sector through multi-stakeholder collaboration, and increased knowledge and capacity on a number of topics including: prioritization of adaptation options, cost-benefit analysis, overview of the broad-based nature of climate change adaption impacts, analysis of challenges, and creation of an open discussion with key stakeholders on defining a road-map for the NAP process. The workshop was delivered using discussions and case studies to enhance interactive learning for participants, with supporting presentations by GiZ and SNV.
Webinar about the new book "Value Chain Development and The Poor: Promise, delivery, and opportunities for impact at scale" (eds. Jason Donovan, Dietmar Stoian, and Jon Hellin), recorded on June 17, 2021. For more information and video recording, visit https://bit.ly/3goPP5r
Similar to Alessandra Garbero: Challenges of impact evaluation (20)
Alessandra Garbero: Challenges of impact evaluation
1. Challenges to impact evaluation:
Solutions for IFAD
Alessandra Garbero, PhD
Econometrician
SSD, IFAD
2. IFAD Commitments
• Lift 80 m people out of poverty (2010-2015)
• Reach 90 m beneficiaries (direct/indirect)
• Challenge: how to measure quantitative impact?
• Solution: Impact evaluation
- Attribution 1. 30 rigorous evaluations
- Feedback 2. Retrospective evaluations using non-
experimental methods and IFAD/RIMS data
- Accountability
- Learning
3. IFAD’s projects portfolio: characteristics
• Relatively small projects
• Multiple interventions/components
• Eligibility criteria not systematically related to IFAD 9R
indicators
• Multiple treatments (same beneficiaries for different projects)
• Suboptimal baseline compliance
• RIMS policy does not foresee control group and panel
structure
• RIMS does not collect income/expenditure data
Non-experimental methods
4. Methodological challenges for impact
evaluation
1. Enhance internal validity: the lack of comparison group
prevents from causal inference between variables
- can be eliminated ex-ante or dealt within the analysis ex-post
2. Enhance external validity: extrapolate study’s results to
other project areas
- randomly selecting sites & within these sites randomly selecting
treatment and comparison groups
3. Purposive targeting of project beneficiaries: differences
between participating and non-participating households
(endogeneity)
4. No income/expenditure data: use appropriate techniques
with poverty proxies (small-area estimation methods or
alternative poverty proxy methods)
5. Two empirical applications from IFAD’s
projects
• Focus on changes in expenditure-based poverty status
• Can we measure these changes with the current data?
• Shall we think about using poverty prediction methods? (regression
models)
Vietnam (DPPR): Decentralised Nicaragua (Prodesec): Programa
Programme for Poverty de desarrollo económico de la
Reduction región seca de Nicaragua
Components: Capacity Building Components: Promote & Finance
for Decentralization process; Business and Employment; Rural
Production supports; Development
Financial services; Strengthen Rural
of rural villages’ small infrastructure
Development Policy
• Baseline 2006 • Baseline 2005
• Completion 2011 • Completion 2011
• LSMS 2002 – only available • LSMS 2005 – only available
dataset at the time of the analysis dataset at the time of the analysis
6. From assets to expenditure-based poverty
status
• Regression-based method (OLS): A prediction model that estimates expenditure
based on household characteristics (i.e. poverty explanatory factors i.e.
“predictors”) using the LSMS
• Poverty “predictors”:
- Vietnam: HH size; education of HH head; sex head; age head; assets (vehicle,
refrig., bike, moto radio, tv; toilet type and source of drinking water)
- Nicaragua: HH size; gender of HH; electricity; toilet type; source of drinking
water; farm HH; type of fuel; type of floor material.
• Model selection: conditional on sig., R squared (0.60 for Vietnam vs. 0.40 for
Nicaragua), presence of variables in both surveys
• Limitations:
- Vietnam: inferring poverty predictors based on 2002 LSMS relationships
between expenditure and key predictors
- Nicaragua: 2005 LSMS
• Definition of poverty line: set at the 30th percentile of rural households (as in
Minot 1998).
7. Results from poverty mapping: how
accurate is our model?
• Comparison between predicted and actual poverty
status using the OLS and the chosen poverty line
- Vietnam: the model identifies 72% of the observed
poor
- Nicaragua: the model identifies 55% of the observed
poor
• The model performs better for Vietnam
8. Impact evaluation
• IE Before after: compare changes in impact indicator
before and after the project
- the counterfactual is represented by the same group
before they got the program
• What are the potential problems with this?
- Other factors contribute to change over time!
• Other secondary datasets (2 points in time) needed to
assess trends in the area (reconstruct
counterfactuals)
9. Poverty reduction? A Naïve comparison
Vietnam: RIMS Nicaragua RIMS
• Model results: Poverty declined from • Model results: Poverty declined from
35% to 9% implies 25% of the 52% to 47% implies 5% of the
sample lifted out of poverty sample lifted out of poverty
Problems
• Model based estimates of poverty based on limited available data
• Naïve comparison: no impact attribution, no control group, no panel
• Macroeconomic factors:
• Poverty declined from 45% to 18% in the whole Vietnamese province.
• Minor impact of project on relative wealth also based on assets
(Nicaragua). Selection bias possible or longer term impacts
10. Way forward
Rigorous ex-post evaluations need adequate secondary data
sources for both poverty prediction and matching exercises to
reconstruct the counterfactual
Theory-based impact evaluation designs need to be mainstreamed
within IFAD’s projects designs
Future data collection efforts: ensure adequate targeting
- Piggy back on national surveys (LSMS/NSOs) if overlap.
- Sample beneficiaries and non-beneficiaries within existing efforts.
Oversampling
If no national surveys underway
- Use an up to date sampling frame
- Randomize! Treatment and control (control group larger than
treatment)
- Core questionnaire with poverty predictors
11. Way forward
Rigorous impact evaluation require:
• Commitment
• Technical & analytical capacity
• Resources
Worth it?
• Strategic relevance
- Contribute to global understanding of agricultural pathways out of
poverty
• Increase evidence of well-functioning programs
• Interventions that work for scaling up