Discussion of the questions of internal and external validity and how case-based approaches are relevant for informing replication and scale up. Case studies can help to extrapolate key facts regarding context dynamics, process mechanisms, implementation capability, and trajectories of change (Michael Woolcock, World Bank).

1. Using case studies to explore the generalizability of ‘complex’ development interventions
Michael Woolcock
Development Research Group, World Bank
Kennedy School of Government, Harvard University
mwoolcock@worldbank.org
Exploring Complexity in the Health Sector:
A Panel on Case-Based Evaluation Approaches
John Snow, Inc, George Washington University School of Public Health, in collaboration with the Institute of Medicine and the World Bank
December 4, 2014

2. [T]he bulk of the literature presently recommended for policy decisions… cannot be used to identify ‘what works here’. And this is not because it may fail to deliver in some particular cases [; it] is not because its advice fails to deliver what it can be expected to deliver… The failing is rather that it is not designed to deliver the bulk of the key facts required to conclude that it will work here.
Nancy Cartwright and Jeremy Hardie (2012) Evidence- Based Policy: A Practical Guide to Doing it Better (New York: Oxford University Press, p. 137)
What ‘key facts’ do we need? How might we acquire them?

3. 10 mins, 3 points
•Using case studies to extrapolate from
–Present to future (time)
–Pilot to scale-up (size)
–‘There’ to ‘here’ (place)
•In social science, even ‘rigorously’ identified impact claims rarely provide warrant for such extrapolations
–Especially for ‘complex’ interventions
•Extrapolation requires ‘key facts’ regarding
–Context dynamics
–Process mechanisms
–Implementation capability
–Trajectories of change
Which careful analytic case studies can help to provide

4. Time
t = 0
t = 1
Net
Impact
Understanding impact trajectories

5. Time
t = 0
t = 1
Net Impact
Understanding impact trajectories
“Same” impact claim, but entirely a function of when the assessment was done…

6. Time
t = 0
t = 1
Net Impact
Understanding impact trajectories
A
B
C
If an evaluation was done at ‘A’ or ‘B’, what claims about impact would be made?

7. Time
t = 0
t = 1
Net
Impact
Understanding impact trajectories
A
B
C
?
D
t = 2

8. Extrapolation from ‘there’ to ‘here’: Using RCTs to test generalizability of RCTs
•Bold, Sandefur et al (2013)
–Take a project (contract teachers) with a positive impact from India, as determined by an RCT…
–…to Kenya; 192 schools randomly split into three groups to receive a contract teacher:
•a control group
•through an NGO (World Vision)
•through the MoE
–Result?

9. Implementation matters (a lot)
Bold et al (2013)

10. Getting (some of the) ‘key facts’: the role of case studies
•World Bank report on service delivery in Middle East North Africa (MENA) region, 2015
–Using HH surveys to map, explore subnational variation in quality of health care (breadth)…
–…complemented by case studies of ‘positive deviance’ to explain how such outcomes occur (depth)
•Effectiveness of national community development program in Indonesia
–(Barron, Diprose and Woolcock, 2011; Contesting Development; Yale University Press)
–Explaining forms, sources of impact heterogeneity over time and place
–Initial pilot results ‘modest’; but big success once scaled-up.
•How and why did this occur?

11. Primary source material
•Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) ‘Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development’, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641
•Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press
•Pritchett, Lant, Salimah Samji and Jeffrey Hammer (2012) ‘It’s All About MeE: Using Experiential Learning to Navigate the Design Space’ WIDER Working Paper No. 2012/104
•Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14
•Woolcock, Michael (2013) ‘Using Case Studies to Explore the External Validity of Complex Development Interventions’ Evaluation 19(3): 229-248