A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. Author | Title (or subhead area)
Evaluating the impact of RBF
Programs: What do we know,
don’t know and what can we
do?
Damien de Walque | HRITF

2. What do we know?
• Well designed and well implemented RBF can have
positive impacts on population-level health outcomes, on
quality of care and on health providers’behaviors?
– Example from Plan Nacer in Argentina
• We also know from the first RBF experiment in Rwanda
(Basinga, Gertler et al. 2011) that this effect is driven by
the RBF mechanisms rather than just the additional
funding
• But, not all RBF experiments have had positive impacts.
Example from the Democratic Republic of Congo:
important lessons to be learned.

3. And … Learning fromImplementationstudies
Brazil
Ghana
Pakistan
Where are we learning?

4. Growth of impact evaluation portfolio
• 36 impact evaluations + 5 in pipeline
• 5 program assessments including 3 completed
• High increase in overall number = many at design stage
• Impact evaluations also progressing to final stage
• Endline surveys planned for 2014: Zambia, Zimbabwe
Number of impact evaluations
0
5
10
15
20
25
June 2011 June 2012 August 2013 March 2014
Design Baseline stage Follow-up stage Complete

5. How are we learning?
• Objective
– Capture a diversity of lessons and insights on RBF through rich set of evaluations
and analytical methods
• Learning on impact is absolutelynecessary
– Rigorous evidence remains thin: need impact measurement
• Learning on impact, however, is also not enough
– Important to understand why, how RBF works
– More to answer:
• contextual and implementation factors, process,costand sustainability
• Learning tools
– Impact evaluations, enhanced program assessments, program assessments
– Learning from RBF implementation case studies
– Analytical work on specific topics
• Learning methods
– Quantitative, qualitative, mixed

6. Diversity of themes
Intervention evaluated Countries
Supply-side RBF payments Afghanistan,Argentina,Armenia,Benin,BurkinaFaso,Burundi,
Brazil,Cambodia,Cameroon,CentralAfrican Republic,China,
Democratic RepublicofCongo,Djibouti,Ethiopia,Gambia,Haiti,
India,Kenya,Kyrgyz Republic,Lao PDR, Lesotho,Liberia, Mexico,
Nigeria,Pakistan, Philippines,Rwanda,Senegal,Sierra Leone,
Tajikistan,Turkey,Yemen,Zambia,Zimbabwe
Demand-side RBF payments Gambia,Lao PDR, Nigeria, Pakistan,Rwanda,Senegal,Yemen,
Zimbabwe
Community-Based RBF Gambia, India,Senegal,Rwanda
RBF for quality of care Afghanistan,Armenia,Argentina,Benin,Brazil,Cambodia,
Cameroon,CentralAfrican Republic,China,Haiti,Kyrgyz Republic,
Lao PDR, Nigeria, Senegal,Tajikistan,Turkey,Zambia,Zimbabwe
RBF in hospitals Afghanistan,Argentina,Burundi,China,India,Kyrgyz Republic,Lao
PDR, Liberia,Nigeria, Philippines,Senegal,Sierra Leone, Turkey
Additional financing Benin,Nigeria,Zambia,Zimbabwe
Differential incentive levels Central African Republic,China
Enhanced monitoring and
supervision
Cameroon,Kyrgyz Republic
RBF and training of providers Zimbabwe
Process vs. output Brazil
Negative Incentives (sanctions) Turkey

7. The HRITF Impact Evaluation Portfolio
• Strengths:
– Very broad IE portfolio (geographically and in terms of
content)
– Commitment to IE: no cherry picking of projects that
are likely to be successful.
– IE is well funded
• A lot on whether and how and why RBF has an impact?
• But what could we learn more?

8. What could we learn more?
(A few examples)
• More “horse races” betweenPBF and other interventions
aiming at improved service delivery or utilization,
examples:
– Demand-side incentives/CCTs
– Community/Citizen Score Cards and
Information/Benchmarking
– Quality control (check lists)
• How does the “logic of collective action” affects PBF
motivation? Incentives are per health facility not per
individual?Are they diluted when group size increases.
• Innovations in measurement of quality of care
• Sustainability

9. How can we work together?
Produce
rigorous,
high quality
evidence
Technical
Support
Tools and
resources
Community
of Practice
Capacity
Building
(i) Annual
Training
Workshops
(i) IE Clinics
(ii) Roster of Experts
(i) Funding
(ii) IE Toolkit
(iii) Cost-Analysis
(iv) Qualitative
9
Including YOU

10. Thank you!