Impact evaluation is a systematic and empirical assessment of the effects brought about by an intervention

1. Research Methodology:
An Overview of Impact Evaluation
BY:
IBITOLA OMOTAYO
23RD SEPTEMBER 2016

2. Knowledge Check
1. What is Impact Evaluation?
2. Is there a need for Impact Evaluation?
If yes, why? If no, why not?
3. What are the methods for conducting Impact
Evaluations?

3. Objectives
To understand the:
key concepts of Impact Evaluation.
aspects of Impact Evaluation.
3Ws of Impact Evaluation: why, what, and who
methods for conducting Impact Evaluation.
considerations for ensuring quality in Impact Evaluation

4. Impact Evaluation: what is it?
“IMPACT” refers to the positive and negative, intended and unintended, direct
and indirect, primary and secondary effects produced by an intervention.
(OECD)
•Impact evaluations measure the change in a development outcome that is
attributable to a defined intervention; (USAID)
•Impact evaluation is a systematic and empirical assessment of the effects
brought about by an intervention. (AusAID)

5. Aspects of Impact Evaluation
Impact
Evaluation
Formative
(ex-ante)
Summative

6. Formative Aspect of Impact Evaluation
Seeks to inform decisions with regards to making changes to a programme or
policy.
While many formative evaluations (i.e. baseline evaluation) focus on
processes, impact evaluations can be used formatively if an intervention is
ongoing. For example, the findings of an impact evaluation can be used to
improve implementation of a programme for the next intake of participants.

7. Summative Aspect of Impact Evaluation
Seeks to inform decisions about whether to continue, discontinue, replicate or
scale up an intervention.
Ideally, summative impact evaluation not only produces findings about ‘what
works’ but also provides information about what is needed to make the
intervention work for different groups in different settings, which can then be
used to inform decisions.

8. Impact Evaluation: Why should it be
done?
•To decide whether to fund an intervention– “ex-ante evaluation” is conducted before
an intervention is implemented to estimate its likely impacts and inform funding
decisions.
•To decide whether or not to continue or expand an intervention.
•To learn how to replicate scale up a pilot.
•To learn how to successfully adapt a successful intervention to suit another context.

9. Impact Evaluation: Why should it be
done?
•To show accountability:
reassure funders, including donors and taxpayers (upward accountability),
that money is being wisely invested.
 inform intended beneficiaries and communities (downward accountability)
about whether or not, and in what ways, a program is benefiting the
community

10. What Questions Does Impact Evaluation Seek to
Answer?
High-level questions directed at the overall purpose of the intervention and should
address-:
1. Overall impact
2. Nature of impacts and their distribution
3. Influence of other factors on the impacts
4. Mode of operation
5. Alignment of intended impacts to needs
***These are not the specific questions that are to be asked during interviews

11. Overall impact
•Did it work? Did the intervention produce the intended impacts in the short,
medium and long term?
•For whom, in what ways and in what circumstances did the intervention work?
•What unintended impacts (positive and negative) did the intervention
produce?

12. Nature of impacts and their distribution
•Are the impacts likely to be sustainable? •Did these impacts reach all intended
beneficiaries?
impact

13. Influence of other factors on the impacts
•How did [the intervention] work in
conjunction with other interventions,
programs or services to achieve outcomes?
•What helped or hindered [the intervention]
to achieve these impacts?

14. Mode Of Operation
•How did the intervention contribute to intended impacts?
•What were the particular features of the intervention that made a
difference?
•What variations were there in implementation?
•What has been the quality of implementation in different sites?
•To what extent are differences in impact explained by variations in
implementation?

15. Alignment of intended impact to needs
•To what extent did the impacts match the needs of the intended beneficiaries?

16. Who Should Conduct Impact Evaluation?
Impact evaluation can be undertaken by:
an external evaluator or evaluation team (e.g Preston –ENGINE)
an internal but separate unit of the implementing organization (e.g. M&E
dept within implementing organizations)
 those involved in an intervention (including community members);
a combined team of internal and external evaluators.

17. Choosing Methods for Impact Evaluation?
Evaluation
design
&
Estimation
methods
purpose of
the
evaluation
availability
of resources
nature of
the
intervention

18. Evaluation designs
http://www.socialresearchmethods.net

19. Evaluation designs
Experimental
• Independent
measures
• Repeated Measures
• Matched Pairs
Quasi- experimental
• Time series
• Separate sample
pre-test/post-test
• Separate sample
pre-test/post-test
control group etc….
Non-experimental
• single-variable
research
• correlational and
quasi-experimental
research
• qualitative
research
www.simplypsychology.org/experimental-designs.html

20. Estimation methods
The are usually based on the evaluation designs. Different designs require different estimation
methods to measure the counterfactual.
The four categories of estimation methods are:
◦ Comparison of means
◦ Multi-variate regression
◦ Instrumental variables methods
◦ Double difference or difference-in-difference.
*** Counterfactual: a comparison between what actually happened and what would have happened in
the absence of the intervention

21. Considerations for quality Impact
Evaluation
•UTILITY
•ACCURACY
•PROPRIETY (ETHICS)
•PRACTICALITY
•ACCOUNTABILITY

22. Considerations for quality Impact Evaluation
•UTILITY: the results of the evaluation must be useful to those who require the
information.
•ACCURACY: ensuring that findings are reported fairly, comprehensively and clearly.
•PROPRIETY (ETHICS):ethical issues of confidentiality and anonymity, as well as
potential harmful effects of being involved in the evaluation must be adequately
addressed.

23. Considerations for quality Impact Evaluation?
•PRACTICALITY: taking into account the available resources (time, money,
expertise and existing data) and when the results are needed to inform
decisions.
•ACCOUNTABILITY: refers to presenting clear evidence and criteria on which
conclusions have been drawn, and acknowledging their limitations.
Transparency about data sources is important.

24. QUESTIONS & COMMENTS

25. More Information
www.betterevaluation.org
interactive textbook at http://www.worldbank.org/pdt
https://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdf
http://www.interaction.org/impact-evaluation-notes.
https://dfat.gov.au/aid/how-we-measure-
performance/ode/Documents/impact-evaluation-discussion-paper.pdf

27. Post-test
1. What is impact evaluation?
2. Is there a need for impact evaluation?
If yes, why? If no, why not?
3. What are the methods for conducting Impact
evaluations?