Discussion of the questions of internal and external validity and how case-based approaches are relevant for informing replication and scale up. Case studies can help to extrapolate key facts regarding context dynamics, process mechanisms, implementation capability, and trajectories of change (Michael Woolcock, World Bank).

1. Using case studies to explore the generalizability
of ‘complex’ development interventions
Michael Woolcock
Development Research Group, World Bank
Kennedy School of Government, Harvard University
mwoolcock@worldbank.org
Exploring Complexity in the Health Sector:
A Panel on Case-Based Evaluation Approaches
John Snow, Inc, George Washington University School of Public Health,
in collaboration with the Institute of Medicine and the World Bank
December 4, 2014

2. [T]he bulk of the literature presently recommended
for policy decisions… cannot be used to identify ‘what
works here’. And this is not because it may fail to
deliver in some particular cases [; it] is not because
its advice fails to deliver what it can be expected to
deliver… The failing is rather that it is not designed to
deliver the bulk of the key facts required to conclude
that it will work here.
Nancy Cartwright and Jeremy Hardie (2012) Evidence-
Based Policy: A Practical Guide to Doing it Better (New
York: Oxford University Press, p. 137)
What ‘key facts’ do we need? How might we acquire them?

3. 10 mins, 3 points
• Using case studies to extrapolate from
– Present to future (time)
– Pilot to scale-up (size)
– ‘There’ to ‘here’ (place)
• In social science, even ‘rigorously’ identified impact
claims rarely provide warrant for such extrapolations
– Especially for ‘complex’ interventions
• Extrapolation requires ‘key facts’ regarding
– Context dynamics
– Process mechanisms
– Implementation capability
– Trajectories of change
Which careful analytic case studies can help to provide

4. Time
t = 0 t = 1
Net
Impact
Understanding impact trajectories

5. Time
t = 0 t = 1
Net
Impact
Understanding impact trajectories
“Same” impact
claim, but entirely
a function of when
the assessment
was done…

6. Time
t = 0 t = 1
Net
Impact
Understanding impact trajectories
A
B
C
If an evaluation
was done at ‘A’
or ‘B’, what
claims about
impact would be
made?

7. Time
t = 0 t = 1
Net
Impact
Understanding impact trajectories
A
B
C
?
D
t = 2

8. Extrapolation from ‘there’ to ‘here’:
Using RCTs to test generalizability of RCTs
• Bold, Sandefur et al (2013)
– Take a project (contract teachers) with a positive
impact from India, as determined by an RCT…
– …to Kenya; 192 schools randomly split into three
groups to receive a contract teacher:
• a control group
• through an NGO (World Vision)
• through the MoE
– Result?

9. Implementation matters (a lot)
Bold et al (2013)

10. Getting (some of the) ‘key facts’:
the role of case studies
• World Bank report on service delivery in Middle East North
Africa (MENA) region, 2015
– Using HH surveys to map, explore subnational variation in
quality of health care (breadth)…
– …complemented by case studies of ‘positive deviance’ to
explain how such outcomes occur (depth)
• Effectiveness of national community development program
in Indonesia
– (Barron, Diprose and Woolcock, 2011; Contesting
Development; Yale University Press)
– Explaining forms, sources of impact heterogeneity over time
and place
– Initial pilot results ‘modest’; but big success once scaled-up.
• How and why did this occur?

11. Primary source material
• Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) ‘Using
Mixed Methods in Monitoring and Evaluation: Experiences from
International Development’, in Abbas Tashakkori and Charles Teddlie (eds.)
Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage
Publications, pp. 613-641
• Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting
Development: Participatory Projects and Local Conflict Dynamics in
Indonesia New Haven: Yale University Press
• Pritchett, Lant, Salimah Samji and Jeffrey Hammer (2012) ‘It’s All About
MeE: Using Experiential Learning to Navigate the Design Space’ WIDER
Working Paper No. 2012/104
• Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project
Evaluation: A Contextualized Approach to Understanding Impact
Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14
• Woolcock, Michael (2013) ‘Using Case Studies to Explore the External
Validity of Complex Development Interventions’ Evaluation 19(3): 229-248