The Science of Delivery:
Impact Evaluation Results and
Lessons from RBF
Dinesh Nair, Benjamin Loevinsohn and Ifelayo Ojo
L...
The Why and How of
Top Ten Lessons
• Review of ongoing PBF operations suggested
some consistent challenges & useful lesson...
Lesson #1
“Show me the money!”
- Make timely payments
• Over 70% of projects have had issues with
making payments on time
3
Outpatient visits in Nigeria declined when
payments were interrupted
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Dec
Jan
Feb
Ma...
Lesson 1: “Show me the Money”
Make Timely Payments
Recommendations
Track time required for payments to reach facilities
...
Lesson #2
“Keep moving the goalposts!”
Continuous Quality Improvement
(CQI) implies Changing the
Quality Indicators
• Many...
0
20
40
60
80
100
Mwaro
Muramvya
Kirundo
Cibitoke
Buja-Rural
Kayanza
Ngozi
Makamba
Rutana
Bubanza
Bururi
Gitega
Karuzi
Muy...
Lesson 2: “Keep moving the goalposts!”
Continuous Quality Improvement (CQI)
implies Changing Quality Indicators
Recommenda...
Lesson #3
“To be discerning, We keep
on Learning”
• RBF as a tool allows for experimentation,
learning and course-correcti...
What did Zimbabwe do?
Quantitative analysis
Qualitative analysis
 Design demand and supply-side RBF (urban pilot)
 Targe...
Operational research studies should be budgeted
Carry out process evaluations and use results for
implementation dialogu...
Lesson #4
“Money is NOT the root of all
evil, Lack of Money is”
– Worry about Financial
Sustainability
• PBF should not be...
PBF sustainability requires wider health
system reforms
 Burundi PBF is
combined with FHC
 Government pays almost
half–1...
Where is the PBF money going?
14
35%
37%
39%
40%
45%
11%
10%
10%
9%
5%
16%
16%
12%
13%
10%
29%
31%
31%
33%
37%
8%
7%
7%
5%
3%
Q2 2012
Q3 2012
Q4 2012
Q1 20...
Recommendations
Track overall PBF expendituresand understand where
most PBF funds are going
It is easier to increase tar...
Lesson #5
“Math-Phobes of the World
Unite!”
– Use your data
• Data is vital but under-utilized, despite a lot
of effort in...
Internet applications with public front-
end displaying performance & financial
information
18
Burundi
Benin
Nigeria
How Zambia uses its data
• Quarterly analysis
• Extensive analysis by expert
o Diff-in-diff analysis of trends using HMIS ...
Lesson 5: Math-Phobes of the World
Unite! – Use your data
Recommendations
 Create reliable systems for data collection an...
Lesson #6
“Understand your
Customers”
-Demand side issues may be
under-appreciated
• There is need to understand root-caus...
Even with large improvements in utilization
since PBF was introduced, absolute
coverage levels remain low in Nigeria
Barri...
Lesson 6: Understand your customers –
Demand-side issues may be under-
appreciated
Recommendations
Consider demand-sideis...
Lesson #7
“Away with Flat-Liners”
- Identify poorly performing
regions & facilities. Do
something about them!
• Detailed u...
There are clearly facilities that are NOT making
progress i.e. “flat-liners”
Non-determinants
• Number of staff
• Remotene...
Lesson 7: Away with “Flat-Liners”
Recommendations
Use existing data to identify facilities that have not
improved or are ...
Lesson #8
“You can run but you can’t
hide”
– Worry about the politicaI
threats to PBF
• Even successful RBF schemes face p...
Zambia RBF Pilot
 RBF PIU not mainstreamed into MOH structure
 Institutional changes in MOH
 Key champions moved
28
Wat...
Recommendations
Political economy expertise should be engaged to
study factors important for scale-up and
sustainability ...
Lesson #9
“Even the Best Laid Plans
can go wrong”
– Analyze your Project Design
to Ensure Assumptions
Remain Valid
• Under...
Benin
 Bureaucracy makes it
difficult for facilities to
spend PBF incentives
 May reduce health worker
motivation to per...
Recommendations
Periodically review the progression of elements in your
theory of change, not just what you are trying to...
Lesson #10
“KISS – Keep Impact Studies
Straightforward”
- Avoid Impact Evaluation
Questions that are TOO Subtle
• It is ve...
0
600
1200
1800
2400
1* 2 3 4 1 2
2012 2013
Children immunized
Control
Treatment
The RBF intervention and additional
finan...
Experimental & Control groups in Nigeria
– try hard to be explicit on differences !!!!
35
Recommendations
Assess in advance if experimental variations are
likely to make a difference in outcomes.
Work with the ...
A ship in the harbor is safe
– but that is not
what ships are built for.-John A Shedd (1928)
Implement, learn and share
le...
Upcoming SlideShare
Loading in …5
×

Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF

158 views
103 views

Published on

A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

Published in: Healthcare
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
158
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Annual Results and Impact Evaluation Workshop for RBF - Day Four - The Science of Delivery - Impact Evaluation Results and Lessons from RBF

  1. 1. The Science of Delivery: Impact Evaluation Results and Lessons from RBF Dinesh Nair, Benjamin Loevinsohn and Ifelayo Ojo Learning from Implementation
  2. 2. The Why and How of Top Ten Lessons • Review of ongoing PBF operations suggested some consistent challenges & useful lessons • Focusing on the most important lessons will facilitate learning by other teams • Selection of the Top Ten was done by TTLs and RBF experts involved in a portfolio review of eight ongoing RBF schemes 2
  3. 3. Lesson #1 “Show me the money!” - Make timely payments • Over 70% of projects have had issues with making payments on time 3
  4. 4. Outpatient visits in Nigeria declined when payments were interrupted 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Adamawa Nasarawa Ondo Payments interrupted 4
  5. 5. Lesson 1: “Show me the Money” Make Timely Payments Recommendations Track time required for payments to reach facilities Establish explicit standards for maximum allowable delay (2 or 3 months) Provide resources up-front to health facilities: (i) gains their confidence; (ii) indicates PBF has started; and (iii) provides investment funds 5
  6. 6. Lesson #2 “Keep moving the goalposts!” Continuous Quality Improvement (CQI) implies Changing the Quality Indicators • Many facilities make rapid improvements in quality and then plateau 6
  7. 7. 0 20 40 60 80 100 Mwaro Muramvya Kirundo Cibitoke Buja-Rural Kayanza Ngozi Makamba Rutana Bubanza Bururi Gitega Karuzi Muyinga Ruyigi Cankuzo Buja-Mairie Qualityscores by province Average quality score plateaued after 3 quarters in Burundi 7
  8. 8. Lesson 2: “Keep moving the goalposts!” Continuous Quality Improvement (CQI) implies Changing Quality Indicators Recommendations Quantified quality checklists should emphasize process over structural indicators Revise checklist every 1–2 years to emphasize continuous improvement Include: 1) observation of patient-provider interaction; 2) records reviews (need standard records); 3) Vignettes; 4) Exit interviews, etc. Invest in training supervisors 8
  9. 9. Lesson #3 “To be discerning, We keep on Learning” • RBF as a tool allows for experimentation, learning and course-correction 9
  10. 10. What did Zimbabwe do? Quantitative analysis Qualitative analysis  Design demand and supply-side RBF (urban pilot)  Targeted urban maternal voucher (demand-side)  Pay-for-quality RBF  Reviewed pricing  Improved quality checklists 10
  11. 11. Operational research studies should be budgeted Carry out process evaluations and use results for implementation dialogue and decision-making. Participate in RBF portfolio reviews – look for new ideas Lesson 3: To be discerning - Keep on Learning 11
  12. 12. Lesson #4 “Money is NOT the root of all evil, Lack of Money is” – Worry about Financial Sustainability • PBF should not be made to substitute deficiencies in national health financing 12
  13. 13. PBF sustainability requires wider health system reforms  Burundi PBF is combined with FHC  Government pays almost half–1.4% of budget allocated  Common pool for development partners– two decreased contribution  Cumulative deficit of almost US$ 8M  Rwanda PBF health reforms were part of larger reforms  CBHI reform  HR reforms 13
  14. 14. Where is the PBF money going? 14
  15. 15. 35% 37% 39% 40% 45% 11% 10% 10% 9% 5% 16% 16% 12% 13% 10% 29% 31% 31% 33% 37% 8% 7% 7% 5% 3% Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Intervention HFs Intervention DMO Hospital Control 1 HFs Control 1 DMO Zambia: Aggregate consumption of PBF funds from Q2, 2012 – Q2, 2013 Where is the PBF money going? 15
  16. 16. Recommendations Track overall PBF expendituresand understand where most PBF funds are going It is easier to increase tariffs than decrease them. This suggests starting with relatively lowertariffs and increasing them Insist that Government(s) live within an envelope budget. Be careful about increasing expenditure untied to improved performance. Lesson 4: “Money is NOT the root of all evil, Lack of Money is” – Worry about Financial Sustainability 16
  17. 17. Lesson #5 “Math-Phobes of the World Unite!” – Use your data • Data is vital but under-utilized, despite a lot of effort invested into collecting, verifying and putting payment data on the web 17
  18. 18. Internet applications with public front- end displaying performance & financial information 18 Burundi Benin Nigeria
  19. 19. How Zambia uses its data • Quarterly analysis • Extensive analysis by expert o Diff-in-diff analysis of trends using HMIS to compare across the 3 study arms RBF vs. Additional financing Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783 p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024 RBF vs. Control Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761 p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011 Attendance outpatient total (calc) Immunised fully <1 year new Antenatal 1st visit before 20 weeks IPT 3rd dose to pregnant woman Postnatal care within 6 days Attendance Family Planning total (Calc) Delivery by skilled personnel 19
  20. 20. Lesson 5: Math-Phobes of the World Unite! – Use your data Recommendations  Create reliable systems for data collection and compilation  Ensure consistent flow of PBF data with minimal reporting burden on health workers  Make it somebody’s job to analyze data  Deploy software to help with data analysis  Crowd-source analysis!  Attempt to triangulate all available data sources to determine trends in utilization and outcomes 20
  21. 21. Lesson #6 “Understand your Customers” -Demand side issues may be under-appreciated • There is need to understand root-causes of the problem RBF is trying to solve. 21
  22. 22. Even with large improvements in utilization since PBF was introduced, absolute coverage levels remain low in Nigeria Barriers to service utilization in two Nigerian LGAs:  Transportation Challenges  Variable & unpredictable fees for Services and Drugs  Social and cultural Barriers Proposed solution: A voucher scheme to improve service uptake  Continuous cycle of learning and responding to help households overcome service utilization barriers and improve health outcomes. 22
  23. 23. Lesson 6: Understand your customers – Demand-side issues may be under- appreciated Recommendations Consider demand-sideissues if coverage levels remain low in spite of PBF (even if PBF has made a big difference). Understand barriers to access – cultural, social and financial barriers through household surveys; focus groups, key informant interviews, etc., Options for addressing demand side include: (i) closer work with community structures; (ii) CCTs; (iii) vouchers; (iv) BCC through facility staff or NGOs 23
  24. 24. Lesson #7 “Away with Flat-Liners” - Identify poorly performing regions & facilities. Do something about them! • Detailed understanding of predictors of success and failure is required 24
  25. 25. There are clearly facilities that are NOT making progress i.e. “flat-liners” Non-determinants • Number of staff • Remoteness • Qualification of in- charge • Business planning - 20 40 60 80 100 120 140 Positive deviants • Community engagement • Management capacity of in- charge Institutional Deliveryin Adamawa health facilities, normalized by 100,000 population 25
  26. 26. Lesson 7: Away with “Flat-Liners” Recommendations Use existing data to identify facilities that have not improved or are not performing well Carry out quantitative and qualitative studies (e.g. Nigeria and Zimbabwe) Consider the kind of technical support needed for the poorly performing facilities and re-examine what is currently available. 26
  27. 27. Lesson #8 “You can run but you can’t hide” – Worry about the politicaI threats to PBF • Even successful RBF schemes face political pressures that threaten their sustainability 27
  28. 28. Zambia RBF Pilot  RBF PIU not mainstreamed into MOH structure  Institutional changes in MOH  Key champions moved 28 Watch for winners and losers
  29. 29. Recommendations Political economy expertise should be engaged to study factors important for scale-up and sustainability of RBF, to figure out what can be done differently. Institutionalizing roles such as purchasing and project implementation within government structures may help improve buy in 29 Lesson 8: Worry about the political threats to PBF
  30. 30. Lesson #9 “Even the Best Laid Plans can go wrong” – Analyze your Project Design to Ensure Assumptions Remain Valid • Understand key-determinants of the PBF program and the linkages between them to produce results 30
  31. 31. Benin  Bureaucracy makes it difficult for facilities to spend PBF incentives  May reduce health worker motivation to perform  User-fees easier to spend  High fees may decrease healthcare utilization 31 Challenges with health facility autonomy in Benin and nested PBF in Afghanistan made implementation different Afghanistan  Unique environment of performance agreement with NGO  Ongoing security concerns  PBF overlaid on PPA
  32. 32. Recommendations Periodically review the progression of elements in your theory of change, not just what you are trying to achieve, but how you get there Useful analyses for every project include:  What is additional percentage of PBF to health workers' take-home pay? Are they motivated by this amount?  Balance of supervision versus autonomy at the health facility level  Resources available at the front-line for health facilities 32 Lesson 9: “Even the Best Laid Plans can go wrong” – analyze project design to ensure assumptions remain valid
  33. 33. Lesson #10 “KISS – Keep Impact Studies Straightforward” - Avoid Impact Evaluation Questions that are TOO Subtle • It is very difficult to assess differential impacts when there is minimal variation between experimental groups 33
  34. 34. 0 600 1200 1800 2400 1* 2 3 4 1 2 2012 2013 Children immunized Control Treatment The RBF intervention and additional financing group in Benin are looking the same  No clear difference is emerging  Another control group with no added finances has been included for counterfactual 34 0 2000 4000 6000 8000 1* 2 3 4 1 2 2012 2013 Assisted delivery Control Treatment
  35. 35. Experimental & Control groups in Nigeria – try hard to be explicit on differences !!!! 35
  36. 36. Recommendations Assess in advance if experimental variations are likely to make a difference in outcomes. Work with the IE team from the start Ensure counterparts fully understand research questions, and are convinced of utility. 36 Lesson 10: “KISS – Keep Impact Studies Straightforward” Avoid Impact Evaluation Questions that are TOO Subtle
  37. 37. A ship in the harbor is safe – but that is not what ships are built for.-John A Shedd (1928) Implement, learn and share lessons!

×