This document provides preliminary findings from an impact evaluation of the RBF4MNH intervention in Malawi. The intervention included supply-side incentives for health facilities and demand-side cash transfers to women for facility deliveries. The study found: 1) No significant impact on utilization of antenatal care, delivery care, or postnatal care services, but more control area women chose to deliver in intervention facilities over time. 2) Mixed results for improvements in health facility quality, with some increases in staff training and equipment but no significant change in medicine availability. 3) Significant improvements in certain incentivized clinical tasks like risk assessment during delivery but no significant change in non-incentivized tasks.

1. RBF4MNH Impact Evaluation –
Preliminary Findings
Presenters: Manuela De Allegri & Stephan Brenner
Study team: Adamson Muula, Don Mathanga, Jacob Mazalale,
Christabel Kambala, Julia Lohmann, Danielle Wilhelm,
Jobiba Chinkhumba, Bjarne Robberstad,
Till Baernighausen, Tom Bossert
1

2. Background
• Evidence available on impact of
Performance-Based Financing (PBF)
interventions on health service utilization
• Lack of comparable evidence on impact of
PBF on quality of care
2

3. 3

4. The RBF4MNH intervention
• Funded by KfW and Norway
• Implemented by Malawi Ministry of Health with
technical support by Options
• Feasibility study completed by 2011
• Intervention launch 2012
• Active in 4 rural districts
• 17-28 Emergency Obstetric Care (EmOC) included
4

5. The RBF4MNH intervention
• Results-based financing intervention which
includes supply-side and demand-side incentives
• Preceded (and accompanied) by infrastructural
upgrade
• Supply-side: quantity and quality indicators
targeting labour and delivery services
• Demand-side: conditional cash transfers to
women upon delivering in health facility
5

6. Research question
What is the impact of a combination of supply-
side and demand-side PBF interventions on
quality of maternal care services?
Scope: ANC, delivery, early PNC, and early neonatal care
Focus: continuity of care around EmONC
Outcome: utilisation and quality indicators
6

7. vPROCESS OUTCOME
Provision of care
Experience of care
INPUT
MIXED METHODS APPROACH within framework of
CONTROLLED BEFORE & AFTER STUDY
Conceptual approach
7

8. Study design
• Controlled before and after study ...
• ... which evolved into a stepped wedge design
• Data from 33 EmOC facilities and their catchment
areas
• 18(23) intervention and 15(10) control
8

9. INPUT PROCESS OUTCOME
Infrastructural
assessment
Interviews with
healthcare
workers
Assessment
of clinical
records
Provider-
patient direct
observations
Exit interviews
Clinical records
Household
survey
Interviews with
healthcare
workers
Experience of care
FGD & in-depth interviews with all stakeholders
Provision of care
9

10. Mixed-methods in practice
10

11. Household survey: methods
• Strategy: repeated cross-sectional: 2013 (baseline – BL),
2014 (midline – ML), 2015 (endline – EL)
• Objective: assess impact on health service utilization
• Sample: 2 Enumeration Areas (EA) from each facility
catchment area & 25 women with delivery in past 24
months within each EA; 4 EAs from second-level facilities
• Analysis: DID regression model accounting for clustering
at facility level and for covariates (age, literacy, wealth,
gravidity, SES, switch)
11

12. Household survey: results (1)
Sample distribution
12

13. Household survey: results (2)
Sample distribution
13

14. Household survey: results (3)
Means & DID estimates
BL mean
control
BL mean
interventi
on
EL mean
control
EL mean
interventi
on
DID BL-EL
DID BL-EL
est.
Sig.
ANC
ANC
utilization
96,2% 96,9% 98,7% 98,9% -0,005 -0,002 0,892
ANC in
first
trimester
15,1% 15,1% 22,2% 21,8% -0,005 0,013 0,770
ANC 4+
visits
44,1% 44,0% 52,4% 50,6% -0,017 0,046 0,419
Delivery
Delivery at
facility
91,4% 91,1% 96,7% 94,9% -0,015 -0,006 0,819
PNC
PNC
utilization
84,9% 81,9% 82,6% 75,7% -0,040 -0,016 0,811
Timley
PNC within
7 days
45,8% 42,0% 38,0% 35,6% 0,013 0,031 0,715
Timely
PNC with
6 weeks
83,3% 79,3% 83,2% 76,1% -0,031 -0,004 0,948
14

15. Household survey: results (4)
Control women using delivery services in
intervention facilities
BL EL
% %
Non-movers 84.1 77.6
Movers 15.9 22.4
Total 100.0 100.0
Bonferroni multiple comparison test p-value = 0.019
15

16. Health facility assessment: methods
• Strategy: repeated cross-sectional:
• BL – 2013, ML - 2014, EL - 2015
• Objective: assess impact on service
readiness for facility-based delivery
• Sample: full sample of all 28 health clinics
in intervention and control arms
• Analysis: DID regression model, accounting
for covariates (switch)
16

17. Definition service readiness
• Based on WHO SARA domains for facility-based
delivery:
• staff & training (2 indicators)
• equipment (10 indicators)
• medicines & commodities (6 indicators)
• Percentage of meeting basic service readiness
requirements
– Domain scores: mean availability of items per domain for each
facility in percent
– Readiness index: mean of domain scores for each facility in
percent
• DID: comparison of average scores and indices across
facilities in each study arm
17

18. Health facility assessment: results (1)
BASELINE MIDLINE ENDLINE
N % N % N %
Total sample: 26 100 26 100 28 100
Distribution by district:
Balaka 7 26.9 7 26.9 7 25.0
Dedza 5 19.3 5 19.3 6 21.4
Mchinji 7 26.9 7 26.9 7 25.0
Ntcheu 7 26.9 7 26.9 8 28.6
Distribution by study arm:
Control 13 50.0 12 46.2 10 35.7
Intervention 13 50.0 14 53.8 18 64.3
Sample distribution

19. Health facility assessment: results (2)
Average percentage service readiness & DID estimates
19
BL mean
control
BL mean
interventi
on
EL mean
control
EL mean
interventi
on
DID BL-EL
DID BL-EL
est.
Sig.
Staff & Training 96.2 % 69.2 % 85.0 % 86.1 % 28.0 % 27.8 % 0.07
Equipment 72.3 % 76.9 % 75.0 % 88.3 % 8.7 % 4.6 % 0.63
Medicines &
commodities
79.5 % 88.5 % 86.7 % 87.0 % -8.6 % -14.7 % 0.09
Service Readiness
Index
82.7 % 78.2 % 82.2 % 87.2 % 9.4 % 5.9 % 0.44

20. Clinical performance: methods
• Strategy: repeated cross-sectional:
• BL – 2013, ML - 2014, EL - 2015
• Objective: assess impact on clinical service
provision during facility-based delivery
• Sample: convenience sample of all non-
complicated labor cases at health clinics
during 5-day observation period
• Analysis: DID regression model accounting for
clustering at birth assistant level and for
covariates (time period between initial case
presentation and entry of stage 2, switch)
20

21. Definition clinical performance
• Observed performance measured against clinical
standards (IMPAC, national QA guidelines):
• Complete assessment eclampsia risk (5)
• Correct partograph use (7)
• Correct performance infection prevention (7)
• Correct performance AMTSL (5)
• Percentage of meeting routine care standards
– Performance scores: mean of observed routine care
processes per domain for each observed labor case
• DID: comparison of average performance scores
across birth assistants in each study arm
21

22. Clinical performance: results (1)
Sample distribution
BASELINE MIDLINE ENDLINE
N % N % N %
Total sample: 45 100 102 100 87 100
Distribution by district:
Balaka 11 24.5 26 25.5 22 25.3
Dedza 179 20.0 24 23.5 14 16.1
Mchinji 15 33.3 33 32.4 28 32.2
Ntcheu 10 22.2 19 18.6 23 26.4
Distribution by study arm:
Control 16 35.6 51 50.0 38 43.7
Intervention 29 64.4 51 50.0 49 56.3

23. Clinical performance: results (2)
Average percentage clinical performance & DID estimates
23
BL mean
control
BL mean
interventi
on
EL mean
control
EL mean
interventi
on
DID BL-EL
DID BL-EL
est.
Sig.
Eclampsia risk
assessment
31.3 % 19.3 % 31.1 % 40.9 % 21.7 % 35.3 % 0.01
Correct partograph
use
40.0 % 24.4 % 44.5 % 57.6 % 28.7 % 21.9 0.13
Infection prevention 50.9 % 51.2 % 54.0 % 72.3 % 18.0 % 27.8 % 0.05
AMTSL 66.3 % 77.9 % 91.1 % 94.8 % -8.0 % 14.4 % 0.18

24. Perception of care: methods
• Strategy: repeated cross-sectional:
• BL – 2013, ML - 2014, EL - 2015
• Objective: assess impact on client’s
experience of care delivered during labor
• Sample: convenience sample of women
exiting health clinics after facility-based
delivery
• Analysis: DID regression model accounting for
clustering at facility level and for covariates
(age, literacy, wealth, gravidity, SES, switch)
24

25. Definition satisfaction with care
• Satisfaction with aspects of care received:
• Interpersonal interaction with birth assistant (13)
• Technical performance of birth assistant (7)
• Structural/organizational set-up of labour service (11)
• Average satisfaction for each dimension
– satisfaction scores: lowest-highest satisfaction on 10-
point scale for each dimension
• DID: comparison of average 10-point satisfaction
scores across clients in each study arm
25

26. Perception of care: results (1)
Sample distribution
BASELINE MIDLINE ENDLINE
N % N % N %
Total sample: 203 100 333 100 230 100
Distribution by district:
Balaka 36 17.7 52 15.6 45 19.6
Dedza 64 31.5 100 30.0 52 22.6
Mchinji 59 29.1 103 30.9 76 33.0
Ntcheu 44 21.7 78 23.4 57 24.8
Distribution by study arm:
Control 67 33.0 109 32.7 40 17.4
Intervention 136 67.0 224 67.3 190 82.6
Distribution by level of care:
BEmOC 121 59.6 209 62.8 147 63.9
CEmOC 82 40.4 124 37.2 83 36.1

27. Perception of care: results (2)
Average satisfaction with care received & DID estimates
27
BL mean
control
BL mean
interventi
on
EL mean
control
EL mean
interventi
on
DID BL-EL
DID BL-EL
est.
Sig.
Satisfaction with
interpersonal
aspects
9.3 9.2 9.2 8.9 -0.2 -0.1 0.86
Satisfaction with
clinical/technical
aspects
9.4 9.3 9.4 9.1 -0.2 -0.1 0.69
Satisfaction with
organizational/
structural aspects
9.2 9.3 9.1 8.8 -0.4 -0.2 0.57

28. Conclusions
• No measurable effect on utilization of delivery services,
but important effect in redirecting demand/women’s
choice
• No effect on ANC and PNC services – need for incentives
that address the continuum of care?
• Weak effects on service readiness in spite of additional
input upgrades
• Strong effects on incentivized clinical tasks – no
measurable effect on non-incentivized tasks
• No measureable effect on client satisfaction – high
satisfaction independent of intervention, satisfaction
measures not sufficiently specific?
28

29. Thank You
29