A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. LEARNING FROM IMPLEMENTATION
RONALD MUTASA
JED FRIEDMAN
WENCELAS NYAMAYARO
PATRON MAFAUNE
CHENJERAI SISIMAYI
EUBERT VUSHOMA
BERNARD MADZIMA
March 2014

2. Rationale for learning from implementation
 Fast changing implementation landscape
 Decentralized implementation arrangements
 Need to go beyond numbers – positivist paradigm (quantity or quality score indicators)
 Capture rich experiences and lessons from frontlines of PBF implementation & context
 Community engagement and support (HCC)
 Geographic influences (supervision aspect, cross-catchment area patient movement)
 Health facility management skills and dynamism
 Extent of mentorship and clinical supervision by district/province
 Striving for excellence by learning from and improving the design
 Supporting impact evaluation and not substituting it –better understanding of the our
intervention
2

3. Process & Impact Evaluation
Answering Policy and Operationally Relevant Questions
3
3/29/2014

4. Implementation & Learning Platforms
4
Baseline
(2011)
Midline Impact Evaluation
(Mar-Sept 2014)
Program Inception
Endline IE
(TBD)
Process Monitoring
and Evaluation (PME)
(November 2013)
Routine Performance
Review (Quarterly) –
Operational Data
Technical Review
(June 2012)
Mid-Term Review
(February 2013)
Technical Adjustments: Prices and Services
Technical Modifications –clinical quality,
streamlining verification, equity monitoring
2nd
PME Round
Planned for
November 2014

5. Analysis of Quality Sub-component Performance
0
20
40
60
80
100
Maternity
service
Medicine stock
management
Family and
child health
Referral
services
Community
services
environment
health
Quarter 2 - 2012
0
20
40
60
80
100
Maternity service
Medicine stock
management
Family and child
health
Referral services
Community services
environment health
Quarter 3 - 2013
• Selected component scores show improvement between the first and
latest quarters
5
3/29/2014

6. Geographical Spread of Performance
• General upward trend in the
performance of indicators
acrossthe districts
• Example: Shows all district
experiencea strong positive
trend in ANC4+ but 8 districts
demonstratingsignificantly
higher positive slopes of
change
6
3/29/2014

7. Moving beyond the ‘Black Box’
PBF Investments Program
Outcomes

8. Process Monitoring & Evaluation
 Help validate program model and the results
 A tool to avoid failure due to lack of “fidelity.”
 Evidence for mid-course adjustments to program design and
implementation
 Supplying rapid feedback about operations and outcomes that guide
program evolution
 Explain contextual factors that matter the most & account for variation in
provider performance
 Deepening policy and donor dialogue through in-depth understanding of
the intervention (1st & 2nd generation PBF issues)

9. PME Approach
 Sequential mixed method deployed
 Quantitative data was collected and analyzedto characterizethe performance
level of the health facilities.
 Health facilities fitting pre-identifiedcriteria (high, medium and low performers)
sampled to gather rich contextual qualitative data
 Wherever relevant and feasible, the quantitative data (on the trends of service
utilization, etc.) was used to triangulate with the qualitative data on the
implementation and performance of RBF.
 Key informants
 Health Center Committees, Health Workers, Head of Health Facilities, District Health
Executives (Management Team) & communities within catchment areas
3/29/2014
9

10. Moving beyond the ‘Black Box’ to the
‘Transparent Box’
PBF Investments Program
Outcomes

11. Validated Conceptual Framework
3/29/2014
11

12. Take Home Messages
 Learning from implementation pays huge dividends
in PBF (e.g. strategic purchasing; depth & focus of
supervision)
 Enables greater understanding of demand-
side/community dimensions
 Enhances realization of the full effect of PBF
 Context, context, context!!
3/29/2014
12

13. THANK YOU
13
3/29/2014