Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Final presentation zambia results

Zambia’s Results-Based Financing pilot project began in April 2012 in 11 rural districts, representing nine provinces out of a total of ten, 204 health facilities, and a total catchment population of nearly 1.7 million. The Zambian RBF model is one of the very few examples of “contracting in” with a view to build on and strengthen the existing public health system. The program was designed to help address various health system challenges including an insufficient and poorly motivated human-resource base; an erratic supply of essential medicines and medical supplies; limited autonomy in decision-making at decentralized levels of the health system; weak monitoring and evaluation systems; and poor quality of service delivery.
This presentation focuses on the results of the impact evaluation and provide an overview of some of the key messages and policy implications of the work.

  • Be the first to comment

Final presentation zambia results

  1. 1. Zambia Health Results Based Financing (RBF) Project Results from the Impact Evaluation Presenters Jed Friedman, Principal Investigator Jumana Qamruddin, Task Team Leader 1
  2. 2. Results based financing in Zambia 1. Provider-Purchaser split – Quantity and Quality data verification – Steering Committees (SCs) as Independent Verifiers – Periodic External Verification 2. “Performance based financing” through public health sector contracting. One of the very few examples of “contracting-in” – “Fee-for-service” on a set of Maternal and Child Health indicators 3. Managerial and financial autonomy of health facilities 4. Nine (9) health facility indicators targeting improvements in MCH 5. Health centre quality indicators in 10 areas 6. Performance package at District Medical Office 2
  3. 3. Impact Evaluation: Design Intervention (RBF) (10 Districts) Control 1 (10 Districts) Control 2 (10 Districts) RBF program (including incentives) +EmONC equipment Enhanced financing (equal to RBF incentives) + EmONC equipment Business-as-usual (status quo) 3 The IE seeks to determine the causal impact of HRBF on priority service provision and population health indicators
  4. 4. Three-arm experimental Study Design 30 districts matched in groups of three on key health systems and outcome indicators and randomly allocated to each arm: 10 districts per arm o 10 Intervention Districts (RBF) o 10 Enhanced (Input-Based) Financing Districts (C1) o 10 Business-as-usual (status quo) Districts (C2) • District triplets selected within each province by matching on: – geographical accessibility (i.e. rural and remoteness) – number and level of health facilities – average facility catchment population – proportion of staff in position – health services utilization rates • Difference-in-difference estimator between matched districts in treatment and control groups estimates program impact 4
  5. 5. District Selection 5
  6. 6. Implementation & Learning Platforms Baseline (Nov – Dec 2011) Endline (Nov – Dec 2014) Program Inception (April 2012) Process Evaluation (May – June 2013) Routine Performance Review (Quarterly) – Operational Data Program Ends (Oct 2014) 6
  7. 7. Questions investigated What is the causal effect of the RBF on targeted health indicators and other population outcomes of interest? – What are the effects on coverage of health services? – What are the effects on quality of care? – What are the effects on health system functionality? 7
  8. 8. • Population representative survey of health behavior and health outcomes • Baseline and endline data at community and household levels covering – 18 districts – 307 enumeration areas – 3064 households in BL and 3087 in EL • After full community listing, random sample of all households with a pregnancy related outcome in the two years before survey Data Source 1: Household Survey 8
  9. 9. • A comprehensive review of the structure, provision, and quality of care at facility level • 213 facilities in both baseline and follow up • Instruments – Facility checklist – Health worker tool (330 in BL; 402 in EL) – Exit interview tool – ANC (900 in BL; 1256 in EL) child illness (1064 in BL; 1273 in EL) • Data collected by independent contractor: University of Zambia Data Source 2: Health Facility Survey 9
  10. 10. A review of the completeness, accuracy and validity of reporting at facility level • 140 facilities: 105 in RBF districts and 35 in C2 districts • Instruments – Facility document review checklist – Client tracer tool • Data collected by independent contractor: Zambia Institute for Policy Analysis and Research (ZIPAR) Data Source 3: External Verification 10
  11. 11. To estimate the cost-effectiveness of RBF program in Zambia:  RBF versus status quo (C2)  RBF versus enhanced financing (C1)  C1 versus status quo (C2)  To assess the cost-effectiveness with and without adjustment for quality improvement over a broad number of MCH services Data Source 4: Cost-effectiveness analysis 11
  12. 12. Outline 1. Healthcare coverage 2. Quality of services 3. Health systems (incl. HRH, Finance) 4. Cost-effectiveness analysis Results 12
  13. 13. Healthcare coverage 13
  14. 14. Institutional & Skilled Deliveries 14 • Deliveries at the facility increased by 12.8 percentage points in RBF districts, and by 17.5 percentage points in enhanced financing districts RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate Impact estimate Impact estimate Facility delivery -0.049 0.128* 0.175* Skilled provider and facility -0.043 0.101 0.142* Note: Statistical significance determined by Fisher exact standard errors, * p < .12
  15. 15. Antenatal care coverage 15 • Many ANC indicators are already relatively well performing in Zambia before the RBF pilot period, and show little change as a result of the RBF program or enhanced financing • However one important exception: pregnant women present significantly earlier for their first ANC visit in RBF districts as compared to the controls RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate Impact estimate Impact estimate Any ANC -0.015 -0.015 0 4 or more ANC visits -0.004 -0.034 -0.029 Timing of first ANC visit -0.372* -0.476* -0.108 Note: Statistical significance determined by Fisher exact standard errors, * p < .12
  16. 16. Post-natal care coverage 16 RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate Impact estimate Impact estimate Any PNC -0.051 0.082 0.132* Note: Statistical significance determined by Fisher exact standard errors, * p < .12 • Both RBF and especially C1 increase coverage relative to C2
  17. 17. Family planning outcomes 17 RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate Impact estimate Impact estimate Any Contraception -0.024 -0.039 -0.017 Modern Contraception (of FP users) -0.045 0.002 0.046 Any FP outreach -0.078* 0.083* 0.159* Note: Statistical significance determined by Fisher exact standard errors, * p < .12 • Little effect on contraceptive take-up (although sample is not fully representative) • Increase in FP outreach in both RBF and especially C1 areas
  18. 18. Vaccination outcomes 18 • RBF performed better than C1 and C2 in fully vaccinated coverage but the impact estimates are not precise. • For some of the other measures of immunization, both the RBF and C1 performed better than C2 RBF vs. Control 1 RBF vs. Control 2 Control 1 vs. Control 2 Impact estimate Impact estimate Impact estimate Fully vaccinated 0.116 0.052 -0.046 Any vaccinations -0.066 0.015 0.081* BCG inject ever received 0.031 0.07* 0.028 DPT ever received -0.01 0.061* 0.056* Note: Statistical significance determined by Fisher exact standard errors, * p < .12
  19. 19. Quality of services 19
  20. 20. Structural Quality Facility infrastructure variables RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p- value Facility experiences no power outage -0.019 0.881 0.194 0.159 Facility experiences no water outage 0.041 0.688 0.051 0.476 Infrastructure index 0.195 0.470 0.483* 0.099 20 • Little change in individual measures of structural quality, however an aggregate index suggests gains in RBF compared with pure control districts • Gains in structural quality of care-specific indices RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Curative Care 0.39 0.204 0.28** 0.042 Family planning 0.15 0.578 0.08 0.546 Delivery Room 0.61** 0.010 0.57*** 0.000
  21. 21. Availability of drugs RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Iron tabs -0.03 0.722 -0.03 0.824 Folic acid tabs -0.09 0.455 0.13 0.259 Artemisinin-Based Combination Therapy (ACT) 0.04 0.693 0.27*** 0.008 Drug availability index -0.08 0.844 0.06 0.893 21 * p<0.1 ** p<0.05 *** p<0.01 • With the exception of ACT, little relative gain in drug availability for either RBF or enhanced financing
  22. 22. Availability of equipment RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Tape measure 0.15* 0.097 0.11 0.399 Baby scale (infant weighing scale) 0.05 0.643 0.22*** 0.007 Forceps, artery 0.08 0.406 0.16** 0.011 Needle holder -0.09 0.389 0.25*** 0.001 Equipment availability index 0.03 0.917 0.37* 0.088 22 * p<0.1 ** p<0.05 *** p<0.01 • Select equipment for delivery and neo-natal care more available in RBF districts
  23. 23. Quality of ANC (Source: Exit interviews) RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p- value Weighed -0.02 0.632 0.06 0.251 Blood pressure measured -0.03 0.809 0.08 0.452 Abdomen measured 0.07 0.152 0.09* 0.063 Abdomen palpated 0.00 0.987 0.12* 0.083 Advice on diet 0.14*** 0.009 0.02 0.850 Quality of ANC index 0.02 0.921 0.33 0.165 23 * p<0.1 ** p<0.05 *** p<0.01 • Process measures of ANC quality for a few measures are improved in RBF as compared to C1 and C2, but little gain in overall index
  24. 24. Quality of child health care (Source: Exit interviews) RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Asked age -0.01 0.880 0.02 0.776 Weighed child -0.07 0.378 0.06 0.498 Measured height -0.10 0.104 -0.02 0.577 Physically examined -0.09 0.327 -0.08 0.350 Quality of care index -0.09 0.669 0.14 0.565 24 * p<0.1 ** p<0.05 *** p<0.01 • No apparent gain in process quality of child health visit
  25. 25. Satisfaction on ANC (Source: Exit interviews) RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value The health worker spent a sufficient amount of time with the patient 0.08* 0.067 0.08* 0.081 You trust the health worker completely in this health facility 0.07* 0.066 0.03 0.569 Satisfaction index 0.04 0.826 0.12 0.574 25 * p<0.1 ** p<0.05 *** p<0.01 • Higher levels of patient satisfaction in selected dimensions of ANC (but not all) in RBF as compared to the two controls • Little apparent increase in overall satisfaction
  26. 26. Satisfaction on child health care (Source: Exit interviews) RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value The amount of time you spent waiting to be seen by a health provider was reasonable -0.02 0.823 -0.06 0.477 You trust the health worker completely in this health facility 0.11* 0.057 0.04 0.504 Satisfaction index 0.09 0.617 0.04 0.858 26 • Little apparent increase in overall satisfaction for child care
  27. 27. Health systems 27
  28. 28. Level of RBF revenue, RBF vs C1 28 38% 43% 78% 56% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% - 500,000 1,000,000 1,500,000 2,000,000 2,500,000 3,000,000 3,500,000 4,000,000 4,500,000 5,000,000 2012 2013 2014 Total AmountinUS$ RBF C1 Funds disbursed to C1 in propotion to RBF
  29. 29. Proportion of GRZ grant to RBF grant 29 230% 34% 171% -18% 26% 21% 13% 0% 5% 10% 15% 20% 25% 30% -50% 0% 50% 100% 150% 200% 250% 2012 2013 2014 Growth of RBF grant Growth of GRZ grant Proportion of GRZ grant to RBF grant
  30. 30. Use of RBF Funds, and Proportion of RBF staff incentives to Govt. staff salaries 30 0.1% 14% 14% 10% 0.7% 59% 51% 47% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 2012 2013 2014 Actual over Period Proportion of RBF staff incentives to GRZ staff salaries Proportion of RBF funds used for RBF staff incentives
  31. 31. Facility governance RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Number of Health Center Committee meetings held in the last 12 months 1.03 0.103 1.26* 0.093 Number of visits made by a district hospital representative for supervision 0.82* 0.065 0.66 0.542 Number of times performance of staff assessed internally 3.41*** 0.002 4.41*** 0.002 Number of times performance of staff assessed externally 1.40** 0.046 2.33*** 0.003 Number of times performance of the facility as a whole assessed externally 1.15 0.365 2.64** 0.022 31 • Increases in supervisory visits and performance assessments
  32. 32. Autonomy RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value able to allocate my facility budget 0.022 0.873 0.092 0.147 choice over who I allocate for what tasks. -0.067 0.565 0.035 0.556 choice over what services are provided in the facility. 0.036 0.797 0.127* 0.055 Autonomy index 0.06 0.880 0.26** 0.037 32 * p<0.1 ** p<0.05 *** p<0.01 • Gains in some measures of facility autonomy in RBF compared with C2, but not C1 districts
  33. 33. Job Satisfaction RBF vs. Control 1 RBF vs. Control 2 N β (s.e.) N β (s.e.) Work conditions 448 6.393 (5.121) 464 4.366* (2.183) Compensation 448 8.639** (4.081) 464 3.880* (1.994) Recognition 448 1.439 (2.842) 464 0.086 (1.324) Opportunities 448 4.686 (4.183) 464 3.641* (2.004) 33 * p<0.1 ** p<0.05 *** p<0.01 • Dimensions of job satisfaction generally higher in RBF districts, especially as compared with C2
  34. 34. Accuracy of reporting 34 Indicator proportion of control facilities under-reporting services Relative likelihood of RBF facilities underreporting P-value of relative likelihood Out-patient visit 0.643 0.049 0.634 Delivery 0.833 -0.154 0.100 Ante-natal care 0.654 0.029 0.787 HIV testing and counseling 0.655 -0.093 0.369 PMTCT 0.875 -0.156 0.082 • Most services in general are underreported – even in RBF districts! • For select services, RBF appears to improve accuracy of reported information
  35. 35. Cost-effectiveness analysis 35
  36. 36. Incremental cost effectiveness ratios 36 Comparison Cost/life saved (US$) Cost/QALY gained (US$) Mid-point (lower bound; upper bound) Mid-point (lower bound; upper bound) RBF vs C1 (unadjusted) 35,802 (17,143; 594,308) 1,513 (724; 24,544) RBF vs C1 (quality adjusted) 30,219 (14,882; 27,8582) 1,277 (628; 11,820) RBF vs C2 (unadjusted) 24,423 (11,452; 52,626) 1,031 (484; 2,223) RBF vs C2 (quality adjusted) 20,434 (10,233; 35,940) 863 (433; 1,518) C1 vs C2 (unadjusted) 14,786 (7,461; 144,906) 624 (182; 5,482) C1 vs C2 (quality adjusted) 12,280 (4,396; 310,513) 518 (168; 16,221)
  37. 37. Summary: Context, Coverage, Quality • One of the first 3-armed IE designs in the portfolio • Project was implemented during a period of several changes in GRZ leadership and ministry organization • RBF and C1 compared to C2 had considerable gains across a number of indicators • RBF vs C1 on health care coverage indicators were comparable • Structural quality: Results were mostly inconclusive but RBF better than C2 on the status of infrastructure and medical equipment; and both controls on quality of delivery rooms • Process quality: Minimal progress on process quality of maternal health care in RBF and C1 districts 37
  38. 38. Summary – HRH • Few gains in client satisfaction except: – Clients who visited RBF health facilities were more satisfied with the time that the health workers spent with them as compared to C1 and C2. – Clients trusted health workers in RBF facilities more than those in C1 facilities for both maternal and child health services • Job satisfaction and retention of health workers increased in both RBF and C1 but the gains were higher in RBF as compared to C1. 38
  39. 39. Summary: Cost Effectiveness Analysis • RBF delivered greater health gains, in terms of lives saved or QALYs gained, than C1 when compared with C2. • However these gains were supplied at a higher unit cost. In $/QALY, C1 is more cost-effective. • Both interventions can be considered cost-effective when compared with the annual per-capita income for Zambia. • However, cost-effectiveness analysis does not explicitly account for health system strengthening investments – certain dimensions of effectiveness with regards to RBF may have been missed by the analysis. 39
  40. 40. Summary: Incentives and RBF Grants • RBF incentives as a percentage of staff salaries: equal to 10% of staff salaries by end of implementation period – Intended ratio was higher but GRZ increased staff salaries for all civil servants ranging from 100% to 200% – Low powered incentives likely result in reduced ability of RBF to affect targeted outcomes • RBF grants as a percentage of operational expenditures: – At facility level, The RBF grant at facility level was spent more on operational activities as compared to staff incentives – Comparison to GRZ grants at facility level suggests that RBF grants may have played a substitutional role instead of being additional (as intended) 40
  41. 41. Summary: Provider Payment Mechanism • RBF was being implemented in a health system that already had relatively high coverage in some indicators – implications for efficiency of spending • Rather than fee-for-service paying for all services rendered, it may have been more effective to have used a target or coverage-based provider payment mechanism 41
  42. 42. Summary: Disbursement Mechanisms • By using two different mechanisms, the study was able to measure the success of each system in terms of overall level of RBF funding disbursed and used by facilities. • Disbursement of RBF performance grants directly to health facilities enabled fiscal decentralization and increased autonomy. • Results show that health facilities in the C1 districts did not receive the same amount as the RBF districts due to delayed retirement and low absorptive capacity. • By the end of the RBF program, the proportion of disbursement to C1 districts was only 56% of what the RBF districts had received. To note, health facilities in the RBF intervention group allocated 47% of the total RBF funds for staff incentives, and 53% for investment. 42
  43. 43. Conclusions • Both the RBF and the C1 arms contributed to some very important health gains as compared to business-as-usual “C2”, and C1 at even lower $/QALY • But the RBF observed relatively more gains in health systems governance, client perception, and health worker satisfaction • The health systems gains under the RBF may translate into population and health gains over a period longer than the 2 year measured under the pilot • Enhanced financing is not just money in a vacuum, involves signaling and direction. Better understanding the effectiveness of these mechanisms can inform policy and program development 43
  44. 44. Considerations and Implications -Focusing RBF mechanisms on improving quality as the primary focus -Design of National Health Insurance Scheme and other health sector priorities -Setting agenda for next generation of learning/operational research - Current IDA Lending operation: RBF component with a heavy process evaluation 44
  45. 45. Thank You! 45
  46. 46. Primary data collected from 18 districts Districts sampled for household survey for Phase I Questions Province Districts Central Chibombo; Kapiri Mposhi; Mumbwa Eastern Chadiza; Lundazi; Nyimba Luapula Kawambwa; Milenge; Mwense Northern Chinsali; Chilubi; Mporokoso North-Western Chavuma; Mufumbwe; Mwinilunga Southern Mazabuka; Namwala; Siavonga 46
  47. 47. Work Motivation RBF vs. Control 1 RBF vs. Control 2 N β (s.e.) N β (s.e.) Teamwork 447 0.385 (3.132) 462 0.925 (1.429) Autonomy 448 0.822 (4.311) 463 1.314 (1.768) Changes in facilities 448 -2.096 (2.664) 463 1.026 (1.240) Work environment 448 -1.788 (2.597) 463 1.257 (1.260) Self-concepts 448 -0.727 (1.866) 463 0.774 (1.075) Recognition 448 -0.380 (3.282) 461 -0.837 (1.330) Well-being 450 1.100 (2.981) 465 2.418* (1.236) Leadership of facility 431 -3.075 (4.885) 446 1.210 (2.613) 47 * p<0.1 ** p<0.05 *** p<0.01 • Work motivation largely unchanged
  48. 48. Distribution of RBF program costs 48

×