1. Implementing a Mixed-Methods Protocol
in Impact Evaluation:
Challenges and Opportunities
Manuela De Allegri

2. The 1st Challenge: The Task Ahead
• Impact evaluation of complex health intervention combining
Performance Based Financing (PBF) with Conditional Cash
Transfers (CCT) in rural Malawi
• Request to measures changes in both quantity and quality of
service provision
• Intervention facilities selected ex-ante by implementing
agency
• Possible scale up during impact evaluation phase

3. Starting with the Research Question
• What is the impact of a combination of supply-side
and demand-side interventions on quality of
maternal care services?
• At what cost are the benefits of the RBF
intervention produced?

4. The 1st Opportunity: Space for Creativity
Mixed-methods design relying on a quasi-experiment
(pre- and post-test design with independent controls)
and an explanatory qualitative component

5. Our Envisioned Mixed Methods Design
QUAN QUAN qual QUAN qual
Baseline Midterm Endline

6. Our Envisioned Mixed Methods Design
QUAN QUAN qual QUAN qual
Baseline Midterm Endline
Identify specific quan and qual questions

7. The 2nd Challenge & Opportunity: Large Study Team
• Till Baernighausen
• Stephan Brenner
• Jobiba Chinkhumba
• Christabel Kambala
• Albrecht Jahn
• Julia Lohmann
• Don Mathanga
• Jacob Mazalale
• Adamson Muula
• Malabika Sarker
• Bjarne Robberstad
• Danielle Wilhelm

8. The 3rd Challenge & Opportunity:
Agreeing on a Conceptual Model
• v PROCESS OUTCOME
Provision of care
Experience of care
INPUT

9. The 4th Challenge: What if Something Does Not
Quite Work as Planned?
• Unexpected quantitative
results at baseline
• Small samples for quan
measures of process quality
• Emerging pressing questions
on process elements

10. The 4th Opportunities: Change the Mixed
Methods Design
QUAN QUAN QUAL QUAN QUAL
Baseline Midterm Endline
+ +

11. Concluding Remarks
• Start with an explicit research question
• Exploit creativity & be daring
• Cherish large research teams
• Use theory as basis to build consensus in a team
• Allow co-existance of multiple mental models & make
adjustments to accommodate them
• Exploit the emerging nature of mixed methods designs

12. Thank you!
This study is made possible by the support of the American People through the
United States Agency for International Development (USAID). The findings of
this study are the sole responsibility of the research teams at the University of
Heidelberg and the College of Medicine Blantyre and do not necessarily reflect
the views of USAID or the United States Government.
This study was funded by the Norwegian Agency for Development Cooperation
(NORAD) and the Royal Norwegian Embassy in Lilongwe. The study is the
product of its authors, and responsibility for the accuracy of data included in this
report rests with the authors. The findings, interpretations and conclusions
presented do not necessarily reflect the views of NORAD or the Royal
Norwegian Embassy in Lilongwe or of NORAD.