(1) Impact evaluations should focus more on understanding dynamics of change and people's responses to incentives, rather than just registering outcomes. (2) Evaluation provides critical information on policy relevance and is a progressive learning exercise. (3) Novel approaches are needed to evaluate new development strategies.

1. (Em.) prof. dr. Ruerd Ruben
State of Art & New Challenges for
Development Impact Evaluation

2. Objectives & Outline
2
 Overview & Evolution of development effectiveness research
 Different frameworks for different purposes
 Novel approaches for generating robuust evidence
 Methods for increasing relevance towards policy uptake

3. Main take-aways
1. Impact analysis should focus more on the dynamics of
change (and less on the registration of outcomes).
2. Insights in impact is based on understanding of people’s
responses to different types of incentives.
3. Evaluation is a progressive learning exercise that provides
critical information on the relevance of policies.
4. New development strategies asks for
innovative approaches.
3

4. Evaluation Questions (ToR)
6W - What Works Where and When & Why and for Whom
Phases of Evaluation studies
1. M&E in project management cycle
2. Evidence-based policy making (MDGs, SDGs)
3. Impact studies & RCTs (ex-post)
4. Theory of change (ex-ante)
5. Systematic reviews
6. Causal inference
4
Finding too much spurious relations

5. Rise and Fall of Impact studies
5
Steep increase in number of impact
studies on development programs
Mixed positive/negative impact
scores (and some very limited
effects) – context matters

6. Difference-in-difference (DiD): causality & attribution
6
Major problems: How to deal with context
How to do scaling ?
What to do with contradictory outcomes ?
Basics:
• Comparing outcomes before & after
the intervention
• Constructing counterfactual: with &
without intervention
• Focus on causality & attribution

7. Pseudo experimental Approaches (panel data)
7
Focus on single interventions (laws, investment, training, etc)
Observable changes (income, knowledge, wealth, health)
Different Treatment Estimates: ATE, ATT, ATU, ITT

8. Field expertiments / RCTs
 Controlled experiments
 Behavioural responses to
interventions (Esther Duflo)
 Decision-making under risk,
stress or pressure
 Criticism:
● Ethics of exclusion
● From internal to
external validity
8

9. Different treatments:Women empowerment in Bangladesh (RCT)
31.0 29.7
26.2
30.4
36.8
29.5 30.6
48.6
50.9 50.3
46.6
56.7
49.0 50.3
0
10
20
30
40
50
60
T1 T2 T3 T4 T5 Control All
Impact of different types of services on women
empowerment (WEIA); baseline (BL) and endline (EL)
BL EL
T1: Nutrition BCC (male extension officers)
T2: Nutrition BCC (women groups)
T3: Agricultural prouction training (extension officers)
T4: Ag+Nut (AE)
T5: Ag+Nut+Gender Sensitication (AE+HKI)
Empowerment
↓
Asset Ownership
↓
Production & Income Decisions
↓
Nutrition

10. The role of Theories of Change (ToC)
10
ToC: simple analytical framework for
tracing how activities result in outputs &
outcomes and finally generate impact
Outcome harvesting: iterative process to
identify all types of factors that may influence
the final outcome and the effort to understand
how change is generated in a complex context
Theory (hypothesis)
Model specification
Empirical Testing

11. Common methodological challenges
Randomization: how to allocate treatments to subjects to
minimalize selection bias and to obtain valid estimates.
Power (sample & effect size): guarantee sufficient
number of valid observations to detect significant effects.
Endogeneity: control for variables that are correlated with
the independent variable (and with the error term).
Instruments: detect variables that influence treatment but
not the effect, to control for confounding factors.
11

12. Sample size (power) and Research Ethics
12
Sample size needs to increase for (a) small error
(correct prediction) & (b) capture small effect size
ETHICS of IMPACT STUDIES
• Literature review
• Knowledge Gaps
• Hypothesis to be tested
• Pre-registration
• Covid-19 protocol for data
collection & Analysis
• Pre-analysis Plan
• Peer review & publication
• Open data
Duflo et al (2011) : Being surveyed
changes behaviour

13. Multi-level (systems impact) analysis
Systems transformation: detection
of new patterns of interaction that
result from simultaneous change in:
 Structures - policy components of the
system.
 Flows — distribution of resources or
information.
 Rules — practices that influence how
the system is organized.
Identify intervention that change
the system dynamics (not only to
solve a problem)
13
Examples: Food Systems, Climate
Change, Pandemics, Financial Crises,
Landscape Transformation, etc.

14. Learning systems: using theory to inform practice
Linking multiple results levels
 Usually (too) much attention for
direct output: responses to
incentives (level 1 & 2)
 Emphasis given to measuring
intermediary effects on changing
practices and behaviour (level 3)
 Shift to system results: linking
intermediary effects to final system
performance through general theory
(level 4) 14

15. Meta analysis, Systematic Reviews & Evidence gap maps
15
Evidence
Gaps: Focus
the analysis on
the ‘known
unknowns’ and
only repeat
impact studies
in new
contexts
Complete overview of impact evidence of interventions (final outcomes)
 Focus on Knowledge GAPS + Requirements for field research (intermediary
outcomes)

16. Opportunities for mixed methods
Mixed methods: deliberate mix & sequence
of different analytical approaches, that lead to
converging insights about impact, through:
 Open problem exploration
 Quantitative & qualitative data collection
 Triangulation of information
 Statistical & content analysis
 Focus on interpretation
 Participation of clients
 Internal & external validity check
16

17. From causality …. To causal inference

18. Causal Inference; From Correlation to Causality
18
DAG of Maternal Health care (Vietnam) to understand
inequalities in child health of ethnic minorities
Rubin’s causal model (part
of Bayesian network)
Bad child health

19. Synthetic Control & Factorial design
19
Multiple Treatment/Multiple outcomes
& Separate Interactions
Aggregate control & synthetic control

20. Stratification & Clustering
20
Individual & joint testing
Changing size & composition of cohorts

21. Heterogeneous effects & Balancing
21
Different outcomes for different population segments
 Under which conditions are treatments more effective?
 Design policies to maximize their effectiveness
Balancing of baseline covariates to reduce
treatment bias after simple randomization
 Effect on power of explanation

22. Real-time evaluation & (on-the-job) learning
New opportunities for:
- Shifting from ex-ante/ex-post to ex-durante
evaluations
- Combining project design & learning: adaptive
management
- Rapid response: learning by doing in conflict
and crisis situations
- Hypothesis testing and development
- Short and longer term effects
22

23. Cost effectiveness & Benchmarking
Performance evaluation:
• Comparing costs with similar agencies
• Comparing outcomes in different settings
• Identifying standards for delivery
• Understanding critical success factors
23

24. New Opportunities: Machine learning, AI & Citizen science
Big data &: satellite information, drones, sensors, social &
Machine learning media messages, document review, biometrics
(Artificial Intelligence) BUT: selection bias; data quality, false algorithm
Phone surveys: rapid & large scale data collection & follow up
(Random digit dialing) (after Covid-19)
BUT: selection bias; declining response rate
Citizen science: systematic data collection for monitoring by
volunteers (e.g. rainfall; consumption diaries)
BUT: data quality, fall out
24

25. Relevance : use & uptake of evaluations
Impact of impact studies (Utilization-focus, Michael Quin Patton, 2012)
- Symbolic use: selective referencing to show transparency
- Instrumental (mis)use: for ex-ante appraisal and project selection
- Conceptual uptake: in agenda setting & exploration of policies
Levels of impact: Knowledge to Practices (KAP model)
- Reasonable insight in changes of knowledge
- Some understanding on change in practices
- Very limited work on changes in attitudes & behaviour
25

26. Good luck! KIT team
26