The document discusses the importance of context and external validity in impact evaluation, addressing how scientific findings can be generalized across different populations and settings. It outlines methods for testing external validity, including meta-analysis and structural modeling, while emphasizing the need to adapt policies based on contextual differences. Additionally, it introduces mechanism mapping as a framework for understanding the relationship between program inputs, activities, outputs, and final outcomes.

1. Context and
external validity in
impact evaluation

2. Michael-Paul James
Student led discussion on
Beyond ‘context matters’: Context and external validity in impact evaluation
Martin J. Williams
In service of PPOL 8630: Advanced Program Evaluation
Dr. Jacqueline Chattopadhyay
2

3. Threats
3
Internal Validity
✘ History
✘ Selection
✘ Maturation
✘ Statistical Regression
✘ Mortality
✘ Testing Effects
✘ Instrumentation
✘ Contamination
Different
People
Different
Places
Across
Different
Times
ExternalValidity?
ExternalValidity?

4. External Validity
The extent to which one can apply the findings of a
scientific study to the world at large.
4
Can sample be generalized to population?

5. 5
Context
Context

6. The Problem
✘ Constructability
○ What is a framework for understanding how context matters for policy
effectiveness?
✘ Generalizability
○ Will the impacts of a policy evaluated in a specific context be the same
in other, unspecified, contexts?
✘ Applicability
○ How do we apply evidence gathered from another context to our
specific context?
6
?

7. Existing Methods to Testing External Validity
✘ Various combinations of meta-analysis
○ A statistical process that synthesizes findings from many studies
✘ Extrapolation based on structural modelling
○ Estimating while assuming that existing trends will continue
✘ Exploiting selection and non-compliance within RCTs
○ Identifies a range of heterogeneous treatment effects.
○ Local Average Treatment Effects (LATE)
✘ Principal stratum analysis
○ Compute causal effects only within identified strata (race, gender, etc)
✘ Reweighting based on observable characteristics to make predictions
about a policy's effectiveness in other contexts
7

8. Local Average Treatment Effects (LATE)
the average potential outcome for comPliers when they do receive treatment
T T
8
T D D D D T
Always
Takers
Compliers
Never
Takers
Defiers
Assigned to Treatment* Assigned to Control
T T
D D
T T
D D
Compliers
Never
Takers¹
Complier/
Never Takers²
N/A
Average outcome in
control group²
Ave complier
outcome without
treatment
% of
Compliers
from ATT*
Ave Never Taker
outcome with no
treatment¹
% of Never
Takers
from ATT*
= x + x

9. Linear program theory model
9
Inputs Activities Outputs
Initial
Outcomes
intermediate
Outcomes
Final
Outcomes
Adapted from Donaldson, S. I. (2007). Program theory-driven evaluation science. New York, NY: Lawrence Erlbaum, p. 25.

10. Mechanism Mapping Method
1. Map out the policies intended Theory of Change
2. Map the contextual assumptions that must hold for the Theory of
Change to work
3. Map the actual characteristics of the context and compare them to
the assumptions
4. Adapt the policy to eliminate these mismatches between
assumptions and characteristics
10

11. Funds,
potential
workers
Procure food,
hire and train
workers
Gov’t provides
supplies, food,
and nutrition
advice to
mothers
Mother’s
nutrition
awareness
improves
Mothers
decide to use
food for selves
and infants
Mother and
infant
nutrition
improves
Funds will be
dedicated and
suitable
workers
available
Adequate
procurement,
logistics,
training
processes
Food and
advice
delivered to
target
mothers in
timely fashion
Mothers able
to attend
sessions, trust
advice
Mothers
control
household
food
allocation
More food will
improve
nutrition
Funds will be
dedicated and
suitable
workers
available
Adequate
procurement,
logistics,
training
processes
Food and
advice
delivered to
target
mothers in
timely fashion
Mothers able
to attend
sessions, trust
advice
Mothers do
not control
household
food
allocation
More food will
improve
nutrition
11
Inputs Activities Outputs
Intermediate
Outcomes 1
Intermediate
Outcomes 2
Final
Outcomes
Theoryof
Change
Contextual
Assumptions
Actual
Context
MechanismMappingExample Tamil Nadu Integrated Nutrition Programme v. Bangladesh Integrated Nutrition Programme

12. Funds,
potential
workers + extra
funds for home
visits
Procure food,
hire and train
workers +
Different types
of workers for
home visits
Gov’t provides
supplies, food,
and nutrition
advice to
mothers,
husbands, and
mothers in-law
through home
visits
Mother’s,
husband’s, and
mothers
in-law’s
nutrition
awareness
improves
Mothers,
husbands, and
mothers in-law
decide to use
food for selves
and infants
Mother and
infant nutrition
improves
Funds will be
dedicated and
suitable
workers
available +
Funds available
for home visit
cost
Adequate
procurement,
logistics,
training
processes +
Suitable home
visit workers
hired
Food and
advice
delivered to
target mothers
in timely
fashion +
Homes are
accessible for
visits
Mothers able
to attend
sessions +
Husbands &
Mothers in-law
Reached
through home
Visits, trust
advice
Better
awareness by
mothers,
husbands, and
mothers in-law
will improve
household food
allocation
More food will
improve
nutrition
12
Inputs Activities Outputs
Intermediate
Outcomes 1
Intermediate
Outcomes 2
Final
Outcomes
AdaptedTheoryof
Change
AdaptedContextual
Assumptions
MechanismMappingExample Tamil Nadu Integrated Nutrition Programme v. Bangladesh Integrated Nutrition Programme

13. 1.
Mechanism Mapping is arguably already informally
executed in the rational process by practitioners and
theorists. What is the value add of the additional step of
charting it?
13

14. 2.
Mechanism Mapping greatly oversimplifies behavior.
What are some of the risks in the process of
simplification?
14
Original
Target
Population
New
Target
Population
Total Population

15. 3.
Mechanism Mapping visualizes behavior through a linear
theory of change. What are some of the risks of thinking
about behavioral change linearly?
15
activities outputs outcomeinputs

16. 4.
Mechanism Mapping relies on judgement and
predispositions. What are some of the dangers or
potential downfalls of personal judgements in
anticipating inputs, activities, outputs, and outcomes?
16

17. 5.
Mechanism Mapping is an intuitive technique for
analyzing the differences between similar programs
exhibiting different results. Is there a value in mapping
when only a single study is available?
17

18. You are Amazing!
Ask me all the questions you desire. I will do my
best to answer honestly and strive to grasp
your intent and creativity.
18