Successfully reported this slideshow.
Your SlideShare is downloading. ×

Introduction of impact evaluation: What is it and how is it done?

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 15 Ad

Introduction of impact evaluation: What is it and how is it done?

Download to read offline

Presented by Colas Chervier (CIRAD) at "Workshop on impact evaluation methods and research collaboration kick-off", Samarinda, Indonesia, on 10 October 2022

Presented by Colas Chervier (CIRAD) at "Workshop on impact evaluation methods and research collaboration kick-off", Samarinda, Indonesia, on 10 October 2022

Advertisement
Advertisement

More Related Content

More from Center for International Forestry Research (CIFOR) (20)

Recently uploaded (20)

Advertisement

Introduction of impact evaluation: What is it and how is it done?

  1. 1. Counterfactual impact evaluation Introduction Colas Chervier CIRAD colas.chervier@cirad.fr 1
  2. 2. Content • Session 1. • What questions can these methods help address? • What are the main features of these methods and why use them? • Session 2. • What is a counterfactual? • What are the specific features of these methods? • How to implement them in practice (steps)? • Presentation of an example of an impact assessment study 2
  3. 3. The policy-relevant questions that these methods can help address. 3
  4. 4. Examples of previous research work (1/2) [Peru] In the 3 years between surveys, we observed a severe decline in forest revenue. However, by using a BACI study design and matching, we show that this decrease was not caused by the REDD+ interventions. Thus, REDD+ “did no harm” to local people, at least in terms of forest revenues (Solis et al, 2021) [Brazil] We find significant but small additional conservation effects from the implementation of the PES program. Notwithstanding, treatment effects are relatively larger in areas with higher deforestation pressure and higher potential agricultural income (Cisneros et al, 2022) [Brazil] We find that it is more effective to locate parks and payments away from each other, rather than in the same location or near each other. (Robalino et al. 2015) 4
  5. 5. Examples of previous research work (2/2) [Brazil] Results suggest that the accepted methodologies for quantifying carbon credits overstate impacts on avoided deforestation and climate change mitigation.(West et al. 2020) [Indonesia] Contrary to the objective of the program, community titles aimed at conservation did not decrease deforestation; if anything, they tended to increase forest loss. In contrast, community titles in zones aimed at timber production decreased deforestation, albeit from higher baseline forest loss rates. (Kraus et al. 2021) [Nepal] Our results indicate that CFM has, on average, contributed to significant net reductions in both poverty and deforestation across Nepal, and that CFM increases the likelihood of win–win outcomes. We also find that the estimated reduced deforestation impacts of community forests are lower where baseline poverty levels are high, and greater where community forests are larger and have existed longer (Oldekop 2021). 5
  6. 6. Example of questions applied to community forests (1/2) Directly related to the measure of impact… • Do community forests have an additional effect on deforestation and community incomes? • Does CF effectiveness depends on the context (e.g. in a REDD+ area vs. outside it), characteristics of the beneficiaries or the type of CF scheme? • Are community forestry schemes more effective when combined with other interventions? • Is it possible to achieve win-win outcomes through CF? • Etc. 6
  7. 7. Example of questions applied to community forests (2/2) … and beyond • Is CF worth replicating and if so, where? • By putting the measured impact into perspective with the costs incurred: is the CF model an efficient approach (for example compared to other management methods)? 7
  8. 8. Definition and justification of counterfactual impact evaluation 8
  9. 9. Overall definition • Impact evaluation is a quantitative assessment of the extent to which an intervention affects outcomes • The proper analysis of impact requires a counterfactual, i.e. a control group representing what those outcomes would have been in the absence of the intervention → By comparing counterfactual and intervention group, we can say how many hectares of forest have been saved or by how many times the income of a group of individuals has been increased by a program, and if this result if significant 9
  10. 10. Differences with more widespread methods • Monitoring tools • Following a number of indicators over time in program area • E.g. LTKL monitoring tool → • Different goal :does not aim to provide information about if an intervention/policy affects indicators measured • Carbon sequestration scenarios • Based on projections of historical trends in target areas • E.g. EK or any other forest carbon projects → • Biased: changes cannot be attributed to REDD+ interventions as lots of changes in the context can influence deforestation ERPD - FCPF 10
  11. 11. Biases associated with widespread methods Before/after comparison • Ex. carbon credits based on historical trends. • Time-varying conditions that influence the target outcome (e.g. change in the price of agricultural products) « Simple » with/without comparison • Selection bias, there are initial differences between control group and intervention (non- random placement) that influence the results Before/after PES Control Intervention (PES) 11
  12. 12. Introduction to quasi-experimental methods and data needs 12
  13. 13. Definition of quasi-experimental methods • Statistical methods that help identifies a control group (e.g. households or forest plots that did not receive the program) that represent a good counterfactual • Allows controlling for the main biases • Applied in cases where the target area of the intervention has already been selected (this decision cannot be influenced) • Specific data needs : baseline data (before the program starts) & in "non-project" areas 13
  14. 14. Statistical methods to get rid of these biases • Matching • Selects two "similar" groups based on observable characteristics (i.e. for which we have data) • Double difference • compares differences in outcomes over time between a population participating in a program and one not participating. • Often associated with matching • Synthetic control method • creates a control obtained as a weighted combination of control units. • Weights are assigned to control units so that their combination is as close as possible to the treated unit's and minimize differences in pre-treatment outcomes. 14
  15. 15. Conclusion day 1 • Provide answers to a wide range of questions relating to the effectiveness of policies with a view to their improvement/replication. • Because these methods identify robust control groups, allow to have a precise quantified measure free from bias. • Specific data needs, especially pre-intervention data. 15

×