Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Setting a Path for Improved Health Outcomes RBF

Learning is a critical part of the HRITF RBF portfolio, with all programs benefiting from an embedded impact evaluation and in some cases, complemented by qualitative research components such as process evaluation studies. The presentation discusses the following topics:

1. Using RBF at the community-level to address demand side barriers
This presentation elaborates on the early evidence and the rationale for using RBF at the community level. It will share lessons learned from the implementation of community RBF at country level.

2. Using RBF to Strengthen Quality of Care: Early Lessons
This presentation discusses the broader policy implications of using RBF to strengthen the quality of care. It will explore how Measuring and Paying for the Quality of Care has been operationalized and will highlight the experience of Nigeria. Lastly, it will focus on measuring and Analyzing the Quality of Care from the Impact Evaluation perspective.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Setting a Path for Improved Health Outcomes RBF

  1. 1. Setting a Path for Improved Health Outcomes Results-Based Financing: the Evidence thus Far
  2. 2. Early evidence on Results-Based Financing: Demand and Community Based incentives
  3. 3. Evidence from a preliminary analysis of financial incentives for health 3  Financial incentives have worked, but… – Demand- and supply-side incentives work on different margins. Demand-side incentives encourage people to go to a facility, while supply-side incentives encourage health providers to deliver more and better care to people who have made it to the facility – Demand- and supply-side incentives are complements, and are best combined; – Community-based incentives, for example incentives to community health workers, could serve as “bridge” between supply and demand. – But few evaluations so far have looked at the combination of supply and demand side incentives and at the role of community-based incentives. – We need to learn more.
  4. 4. Conditional cash transfers and children health outcomes  Some health outcomes and behaviors might be easier to influence from the demand side (patients, population) rather than from the supply side (health care providers).  See example from Rwanda  (Conditional) cash transfers have been widely used and evaluated as a social protection mechanism.  When they are conditional, the conditions are linked to educational and/or health behaviors.  They usually have impacts on reducing poverty, but also on improving education and health outcomes. 4
  5. 5. 5 Gender and Conditionality: A Randomized Evaluation of Alternative Cash Transfer Delivery Mechanisms in Rural Burkina Faso
  6. 6. Cash Transfer Pilot Program Randomization Plan 6 75 villages (2775 households) _________ | _____________ | _ | ______|_____ | _____________ | ________ | 15 villages (540 households) Randomized CCT to Father 15 villages (540 households) Randomized CCT to Mother 15 villages (540 households) Randomized UCT to Father 15 villages (540 households) Randomized UCT to Mother 15 villages (615 households) Randomized to Control Group
  7. 7. Cash Transfers Overview  Transfer amount: – Ages 0-6: 4000 FCFA/year – Ages 7-10 (Grades 1-4): 8000 FCFA/year – Ages 11-15 (Grades 5+): 16000 FCFA/year  $1 USD = 500 FCFA  CCT: – Ages 0-6: Quarterly visits to health clinic for preventive care (growth monitoring) – Ages 7-15: School attendance rate>90%  UCT: – No requirements 7
  8. 8. Research Summary  Consider broad measure of welfare outcomes: education, health, livestock, agriculture, demographics, assets/infrastructure  For child education and health outcomes, conditional cash transfers outperform unconditional transfers  Giving cash to mothers does not lead to significantly better child education or health outcomes  Evidence that giving cash to fathers improves child health in bad rainfall years  Cash transfers to fathers yields more investment in livestock, cash crops, and improved housing 8
  9. 9. CCT and adolescent health outcomes including HIV prevention  Traditionally CCTs target education outcomes as well as mother/child health outcomes.  More recently they have also been tested as a way to influence adolescent/young adults health outcomes and behaviors, in particular for HIV prevention.  Those are behaviors and outcomes which are likely to be difficult to influence through a classic supply- side RBF program. 9
  10. 10. 10 STIs? HIV? Baird, Garfein, McIntosh and Özler, 2012
  11. 11. 11 STDs? HIV? + STIs? HIV?
  12. 12. 12 Study population (N=1,328) Control (N=827) Treatment (N=501) Unconditional Cash Transfer (N=265) Conditional Cash Transfer (N=236)
  13. 13. 13 Study population (N=1,328) Control (N=827) Treatment (N=501) Unconditional Cash Transfer (N=265) Conditional Cash Transfer (N=236) Relative risk (compared to control, adjusted) Pregnant now : 0.16 (p<0.05) Partner≥25 : 0.36 HIV : 0.47 HSV-2: 0.08 (p<0.05) Relative risk (compared to control, adjusted) Pregnant now : 1.17 Partner≥25 : 0.08 (p<0.05) HIV : 0.29 (p<0.05) HSV-2: 0.37 NB UCT significantly different than CCT only for “pregnant now” outcome
  14. 14. Impact Evaluation of the Rwanda Community Performance-Based Financing Program College of Medicine and Health Sciences School of Public Health
  15. 15. Background: Community PBF (Second Generation) 15 Since 2009, Community Health Workers (CHWs) were paid for reporting on health indicators in their communities Additional components were added through the Community Performance-Based Financing Program in order to promote targeted services This study evaluates the impact of 2 interventions that were added to the scheme: 1. Performance incentives for CHW cooperatives 2. Demand-side in-kind incentives
  16. 16. Background: organization of CHWs in Rwanda 16 Each village has 3 volunteers serving as Community Health Workers (CHWs). Multidisciplinary CHWs CHW in Charge of Maternal and Neonatal Health Criteria • Can read and write • Age 20-50 • Lives in the village • Elected by the village residents
  17. 17. Background: organization of CHWs in Rwanda 17 All the CHWs within the catchment area of a health center are organized in a CHW cooperative. Cooperative
  18. 18. Background: organization of CHWs in Rwanda 18 70% of payments received by a cooperative must be invested in income generating activities (IGAs). 30% of the payments and revenues from the IGAs are given to cooperative members. It is up to the cooperatives to determine distribution rules.
  19. 19. Intervention #1: Performance Incentives for CHW Cooperatives CHW cooperatives received financial rewards for: 1. Nutrition monitoring: # children 6-59 months monitored 2. Timely Antenatal Care: # of women accompanied/referred within first 4 months of pregnancy 3. In-Facility Delivery: # of women accompanied/referred for assisted delivery 4. New Family Planning users: # referred to health center 5. Regular Family Planning Users: # regular users at health center  4 indicators related to TB and HIV were added at a later stage and not evaluated 19
  20. 20. Intervention #2: Demand-Side In-Kind Transfers 20 Women received gifts for seeking care for the following services: * Women can only receive the gifts for one pregnancy every 3 years. Eligibility* Value (Ceiling) Suggested Package Initiation of Antenatal Care during first 4 months of pregnancy 5 USD Adult cloth and water treatment tablets OR baby cloth package and water treatment tablets Delivery in health center 6.67 USD Baby soap, baby shawl and baby bed sheets Initiation of Postnatal Care during the 10 days after delivery 3.33 USD An umbrella and water treatment tablets OR Adult cloths
  21. 21. Research Questions 1. Do the demand-side in-kind transfers and the performance incentives to CHW coops increase – Initiation of prenatal care within first 4 months of pregnancies? – Total prenatal care visit? – In-facility deliveries? – Rate of postnatal care within 10 days after delivery? 2. Is there a multiplicative effect when both interventions are implemented? 3. Do the performance incentives to CHW coops affect – Behavior and motivation of the CHWs? – Use of modern contraceptives? – Growth monitoring of children under 5 21
  22. 22. Study Design: RCT 22 198 sectors (sub-districts) were randomly allocated into 4 study arms: * Coops paid for reporting received the average amount received by the coops paid for performance Payments to CHW Coops For Reporting* For Performance Demand-Side Transfers No C S Yes D D+S
  23. 23. Timeline 23 2010 2011 2012 2013 2014 February- May 2010 • Baseline Survey November 2013- June 2014 • Follow-up Survey October 2010 • Interventions Introduced February 2013 • Last transfer of funds for in-kind transfers
  24. 24. Results: Maternal Health Services  Indicators: – Timely ANC – In-facility deliveries – Timely PNC  Sample of women with most recent birth in their village – Pregnancies resulting in a live birth 24
  25. 25. Results: ANC visit within first 4 months of pregnancy 25 50% 55% 60% 65% 70% 75% 80% 85% Control Demand Supply D + S Timely ANC 72% 82% 74% 80% Timely ANC • A positive and significant (at the 1% level) impact of the demand-side in-kind incentives of about 10 percentage points • The CHW incentives are not found to have a significant effect • No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms
  26. 26. Results: at least 4 ANC visits 26 25% 30% 35% 40% 45% 50% Control Demand Supply D + S 4 ANC 40% 46% 43% 45% Four or more ANC visits • Not targeted by the program! • Higher in the intervention sectors, but not statistically significant at the 10% level
  27. 27. Results: Skilled-attended in-facility delivery 27 70% 75% 80% 85% 90% 95% 100% Control Demand Supply D + S Deliv 94% 95% 96% 94% In-Facility Delivery • No statistically significant difference between the treatment arms • Rate has increased substantially in the duration of the study for other reasons
  28. 28. Results: PNC within 10 days after delivery 28 0% 5% 10% 15% 20% 25% Control Demand Supply D + S Timely PNC 13% 22% 11% 20% PNC within 10 days after delivery • A positive and significant (at the 5% level) impact of the demand-side in-kind incentives of about 7 percentage points • Not targeted by the CHW incentives intervention
  29. 29. Key Findings: Demand-Side In-Kind Incentives • The demand-side in-kind incentives caused an increase in timely ANC and PNC services • Although some challenges in procurement and frequent stock outs • Although some health centers independently implemented their own demand-side incentives strategies to promote utilization • Although funding ended before end-line data collection • Consistent with findings in other countries that implemented demand-side cash transfers 29
  30. 30. Key Findings: Performance Incentives to CHW Coops • No impact of incentives to CHW cooperative on targeted indicators, CHW behaviors and CHW motivation. • Potential reasons for lack of impact – Incentives were too low – Collective reward but individual effort – Pay-for-reporting could have already oriented the CHWs towards targeted indicators – Limited scope given the many supply-side programs targeting the same indicators 30
  31. 31. Research Team  Ministry of Health – Fidel Ngabo – Cathy Mugeni  University of Rwanda – Ina R. Kalisa – James Humuza – Jeanine Condo – Vedaste Ndahindwa  The World Bank – Gil Shapira – Netsanet W. Workie – Jeanette Walldorf 31 The study was funded by the Health Results Innovation Trust Fund (HRITF)
  32. 32. The Case of Community RBF
  33. 33. What is cRBF?  Community RBF: a set of different practices: – Based on the idea of contracting (cRBF) – Separation of functions (purchaser, provider, regulator and verifier) RBF is: “a cash payment or non-monetary transfer made to a national or subnational government, manager, provider, payer or consumer of health services after predefined results have been attained and verified. Payment is conditional on measurable actions being undertaken” (Musgrove, 2010) 33
  34. 34. Rationale for cRBF? 34 What is the objective? Provide services at the most peripheral and decentralized level Contracting of CHW Often attached to a health facility Stimulate the demand side Awareness meetings Contacts with the population Vouchers and incentives Achieve health related behavioral changes Part of all cRBF, sometimes stated more clearly (The Gambia and Congo) Health promotion / awareness [HP] Use of services [US] Health outcomes [HO]
  35. 35. Who is contracted in cRBF?  Who are the community actors contracted in cRBF? – Community Health Workers: in charge of providing specific services, often preventative care and awareness campaigns [in the spirit of the 1977 Alma-Ata conference] – Health Facility Committee members: co-managers of the health facilities, intermediaries between population and service providers [in the spirit of the 1988 Bamako conference] – Traditional healers –a large variety: traditional midwifes, herbalists, etc. – Other community actors: ▫ Village committee ▫ Community-based organizations  What is not included under cRBF? – Individuals directly: then closer to Conditional Cash Transfers (CCTs) 35
  36. 36. Lessons Learned from cRBF Operations
  37. 37. Country cRBF experiences  Contracting of Community (Health) Workers: – Benin – Cameroon – Republic of Congo – Rwanda  Contracting of Health Facility Cie. – DR Congo  Contracting community organizations The Gambia  Demand-side and voucher schemes (not discussed here) – The Gambia – Rwanda – Congo 37
  38. 38. Lesson 1: cRBF programs should be designed taking into consideration contextual factors 38
  39. 39. cRBF programs should be designed taking into consideration contextual factors (Cameroon) Example of Cameroon:  PBF Indicators started improving in HF but stagnated despite much efforts by health facilities  Reports of many drop outs concerning vaccinations, post natal consultations antenatal consultations and use of family planning among women.  Nutritional concerns of children were poorly addressed by program  Therefore something had to be done to re-stimulate demand for health services by the community  Reflection of the Government and partners led to identifying a Community PBF approach as a strategy worth trying  Experience of some health facilities sub-contracting with Health Committee Members had proven it’s worth in referrals and search for drop outs  Need therefore to contract Community Health Workers in a formal manner  a cPBF pilot was then started in July 2015 with a Community Monitoring component to strengthen the voice of the community in health care delivery 39
  40. 40. Lesson 1: Experience from RoC  Each Context is unique – Avoid Copy and paste  Context is essential to define the CPBF Model of RoC – Low coverage for some indicators – Absence of community networks  Objective: support households in the health seeking behaviors.  Interventions: – Put in place the community relays – Action plan signed with the household 40
  41. 41. Lesson 2: Existing community structures should be assessed and, where possible, strengthened using cRBF 41
  42. 42. Contracting Community Committees: The Gambia  Most communities in The Gambia have: – Village Development Committees (VDC) responsible for all development activities of the community; and ▫ Village Support Groups (VSG) comprising 4 women and 2 men who, with the VHW and TBA, are trained to promote optimal maternal, infant and young child feeding practices. They are an arm of the VDC.  During the design stage of the Maternal and Child Nutrition and Health Results Project, anchored on PHC, it was unanimously agreed that the VDC be contracted to implement the Demand side of the Project  This was strengthened by the type of indicators which could not be contracted to individuals: the demand side (cRBF) indicators focused on knowledge and practice  The verification of these indicators is done using a survey (LQAS) – therefore the entire community is contracted through the VDC  20% of the quarterly subsidy payment is given to the VSG as an incentive while the balance goes into the implementation of a community development project identified through a PRA 42
  43. 43. Experience of Benin (Similar to Cameroon)  Preexistence of community health workers: sensitize the population on health, refer patients to the health center  But fragmentation of package of services depending on sources of funds  cRBF relies on existing CHWs and train them on the complete package  Then, sub-contract between individuals and HF  The Health center: Coordination center to share good practices, to declare results, group monitoring, supervision and payment 43
  44. 44. Lesson 3: Broad participation in the selection of indicators is key… 44
  45. 45. Experience of Benin  Involve all actors in the process to prioritize indicators: – Central level MoH, Vertical programs, Donors, district level, local levels,…… (with focal persons at all steps)  Build ownership : – Good understanding by all stakeholders – Appropriate indicators for the implementation of the PIHI program – Coordinate and prioritize (All indicators cannot be part of the package) 45
  46. 46. Lesson 4: But it’s important to limit the number of indicators to ensure feasibility and quality 46
  47. 47. Variety and scope of indicators varies  Indicators can be at all levels  The number of indicators matters: – The Gambia (9): Health promotion – Benin (9): Referral system – DRC: Hybrid (functionality indicators, health promotion) – Cameroun (20): Referral system, service utilization – RoC: Health promotion 47
  48. 48. RoC: Advantages with few indicators  Better verification: – Good quality data: reliable  Better analysis of data collected: – Areas of weaknesses and strengths  Low cost of transactions for verification, high cost for individual indicators (Motivating for CHWs) 48
  49. 49. Lesson 5: Data collection tools should be simple 49
  50. 50. Make management tools simple 50 Current Challenges 1. Tools for community health workers and other community members are too complex 2. Tools are not effectively used because they are time consuming 3. Tools and processes are designed for the purchaser or the regulator rather than the users and community Recommendations 1. At the community level, tools should be simple and easy to use 2. Tools should be validated by the relevant community actors 3. Strengthen the community capacity for monitoring
  51. 51. Lesson 6: Systems are needed to monitor and maintain the quality of training at all levels of the health pyramid 51
  52. 52. Systems are needed to monitor and maintain the quality of training at all levels of the health pyramid 52 Current Challenges: 1. To decentralize, there is a need of training in cascade mode. 2. But, the cascade mode doesn’t ensure the quality of training at peripheral level A (100%)-- B (85%)--C ( 70%)-- D ( 45%) 3. The content of the training is losing some key information 4. During the implementation, new issues arise 5. Differentiated adaptation Recommendations: 1. Ensure quality of training at lower levels 2. More supervision and monitoring of the trained community actors 3. Benchmark the good practices of those who succeed to support the weak CHWs
  53. 53. Lesson 7 Payments to CHWs should be timely 53
  54. 54. Payments to CHWs should be timely (Cameroon) - During pilot period for cPBF payments from central level to health facilities were often delayed. At first facilities waited for PBF subsidies to arrive before paying CHWs, this led to long delays in paying CHWs, leading to demotivation and frustration of CHWs - To improve on the retention of the CHWs, the payment model was revised. - Now the quarterly facility contracts stipulate that the health facility should pay the CHWs monthly as soon as their verification is done; using facility resources (mix of cost recovery, PBF subsidies, etc.). - Difficult to convince all facilities to accept this approach, but by including it in the facility contract they, CVA was able to negotiate this payment mechanism. - After several months facilities have noted that it is possible to ensure timely payment of CHWs - CHW motivation and retention has improved. Model scaled up to other regions. 54
  55. 55. Lesson 8: ICT can be very useful but it should be built on solid systems and carefully tested 55
  56. 56. Use of Mobile Devices for Data Reporting and Verification: The Gambia Experience  The Gambia started with strengthening the already existing HMIS and incorporating RBF indicators – Data collection and reporting tools were reviewed and updated – The DHIS2 database updated to reflect the new information – PHC Circuits were re-demarcated to fit within health facility catchment areas  The country team is now considering the gradual introduction of the use of mobile devices starting with verification using tablets  Also considering the use of mobile money for the payment of CCTs to pregnant women 56
  57. 57. Lesson 9: There’s still a lot to learn! 57
  58. 58. Learning Opportunities  How best can community level data be used to inform activities?  How to ensure that CHWs only provide the services they are meant to provide?  How to appropriately share data with communities and promote community ownership of activities?  What is the impact of sub-projects funded through community incentives?  Why was there high CHW drop-out after initial training?  How best to do verification of community data? 58
  59. 59. What are we learning? Projects in the World Bank’s current portfolio of cRBF are in the process of answering some outstanding questions.  RoC and DRC are evaluating a strategy of paying health centers to conduct home visits jointly with community agents  Cameroon is assessing the impact on uptake of services of health centers subcontracting community health workers  In the Gambia, the impact on health behaviours and uptake of health services of performance payments to community organizations is being assessed 59
  60. 60. THANK YOU
  61. 61. Results-Based Financing & Quality of Care: Measuring and Paying for Quality Improvement
  62. 62. Session Outline: Measuring and Paying for Quality I. Existing Instruments and Methods II. Using Data for decision making III. Verifying Data Accuracy IV. Innovations in Measuring and Paying for Quality 62
  63. 63. 1. Existing Instruments and Methods 63
  64. 64. Measuring if the right inputs are in place 64
  65. 65. Liberia: Quality Assessment/ Monitoring Tools 1 Complicated and assisted delivery (including C-section) Any labor that is made more difficult or complex by a deviation from the normal procedure. Complicated delivery is defined as: assisted vaginal deliveries (vacuum extraction or forceps), C-section, episiotomy and other procedures. 17 2 Normal deliveries of at risk referrals High-risk pregnant women referred by health center to the hospital but delivered normally. A high-risk pregnancy is defined as: evidence of edema, mal presentation, increased BP, multi-parity, etc. 17 3 Counter referral slips returned to health facilities Hospital returns counter referrals letter with feedback on the referred patient to the referring health center. The counter referral letter is completed in triplicate, with one also given to the patient, and one retained by the hospital. 2.5 4 Newborn referred for emergency neonatal care treatment and treated Newborns referred for emergency neonatal care due to: perinatal complications, low birth weight, congenital malformation, asphyxia, etc. 5 6 Referred infants and under-fives with fever Any surgical procedure that does not involve anesthesia or respiratory assistance. 2.5 7 Minor surgical intervention Any surgery in which the patient must be put under general spinal/anesthesia and given respiratory assistance. Major surgery in the case of this package of services is defined as any of the following: Herniarraphy, Appendectomy, Myomectomy, Sleenectomy, Salpingectomy, Hysterectomy, Thyrodectomy, Mastectomy. 5 8 Major surgery (excluding CS, including major trauma) Patients transferred from a lower-level facility (health center or health clinic) to the hospital for emergency treatment. 18 9 Patients transported by ambulance 2.5 10 Number of training sessions held by faculty for nurses, midwifes and PA according to in-service curriculum and defined protocols. These indicators will incentivize the in-service training activities. 50 11 Number of nurses, midwifes and PAs that received specialized in-service training, relevant to benchmarks 10 Verified Total EarningsDefinition Six Hospitals Total Fee (USD)Indicators Claimed (c) Quantity Checklist Actual % Earned Points 1. Obstructed Labor 0.80 3.87 100% 33% 1.29 2. Hemorrhage 1.00 4.84 100% 71% 3.45 3. Maternal Sepsis 1.00 4.84 100% 50% 2.42 4. Eclampsia 0.70 3.39 100% 47% 1.59 5. Neonatal Asphyxia 1.00 4.84 100% 67% 3.23 6. Neonatal Sepsis 1.00 4.84 100% 54% 2.61 7. Prematurity 0.50 2.42 100% 47% 1.14 8. Maternal Newborn Best Practices 1.00 4.84 100% 54% 2.61 9. ETAT 1.00 4.84 100% 33% 1.61 10. Malaria 1.00 4.84 100% 71% 3.45 11. Pneumonia 1.00 4.84 100% 50% 2.42 12. Acute Diarrhea 0.80 3.87 100% 47% 1.82 13. Severe Acute Malnutrition 0.60 2.90 100% 67% 1.94 14. Surgical Safety 1.00 4.84 100% 54% 2.61 100% 60.00 100% 53% 32.20Total/Average Childbirth: Maternal-Newborn Pediatric (in-patient care) Surgical Care Quarter I III. Process of Care Detailed Score Checklists Weight (by importance) Point Allocation Max % (b) Process of Care Quality Checklists Score 1.GENERAL MANAGEMENT (30pt) 2. HUMAN RESOURCES FOR HEALTH (16pt) 3. HYGIENE AND MEDICAL WASTE DISPOSAL (27pt) 4. DRUGS MANAGEMENT (30 pt) 5. EQUIPMENT AND SUPPLIES (84pt) TOTAL % Date of Verfication TOTAL (187pt) REPUBLIC OF LIBERIA Ministry of Health and Social Welfare (MOHSW) Hospital Quarterly Quality Assessment Name of the Hospital Name of Team Leader of Quality Verification Verification Period Quarterly Quality Verification Score I. Management II. Structural (a) Management and Structural Checklist Indicators Max Points Actual Points Quarter I 1. General Management 30 2.6 2. Human Resources for Health 16 9 3. Hygiene and Medical Waste Disposal 27 0 4. Drugs Management 30 8 5. Equipment and Supplies 84 48 6. Aggregated Process of Care Score 60 32 Total 247 100 Total Percentage 100% 40% Total Quality Bonuses (USD) 159,678 64,517 PBF Bonus Calculation Tool Business/Operation Plan Health Worker Bonus Allocation LHSSP Indices Tool for Bonus Allocation to Individual Health Workers for Hospitals 1 200 50 30 300,000 0 6,944 2 200 70 30 420,000 0 9,722 3 150 80 30 360,000 0 8,333 4 - - - 5 - - - 6 - - - 7 - - - 8 - - - 9 - - - 10 - - - 11 - - - 12 - - - Quarter: Total PBF Incentives Earned % for Individual Bonus Attendance points [C] Hospital Name Total Individual Bonus Redemption Hospital July-Sept 2013 No Name of staff Staff category Monthly salary [A] Perfor- mance points [B] $50,000 50% $25,000 Total points = [A] x [B] x [C] Indices of the period PBF individual bonus Signature of receipt Min 50% Max 50% ~60% ~20% ~20% (1)Continuousmonitoring (d) Impact Evaluation Measuring processes and results 65
  66. 66. Liberia: Standards for Management Obstructed Labor: Illustrative Checklist Distilling Essential care Items (admission, labor) Chart review elements (see chart review guide for specific criteria) ; each element if recorded = 1 point Charts 1. Admission 1 2 3 4 5 1. Cervical dilation recorded at admission (# of cm) 2. Contraction frequency and duration charted at admission 3. Fetal presentation charted at admission 4. Partograph started when cervical dilation 4 cm or greater Admission Score (x/4) 2. Labor Monitoring (partograph) 1. Cervical dilation recorded at least every 4 hours 2. Frequency and duration contractions recorded at least every 30 minutes 3. Fetal HR recorded at least every 30 minutes Labor Monitoring Score (x/3) Each item has chart review guide that defines criteria Five patient charts reviewed: average score (% adherence best practices) links with bonus 66
  67. 67.  Record Reviews  Simulations of routine labor and delivery, postpartum hemorrhage and eclampsia using Mama Natalie  Simulation of newborn resuscitation using Neo Natalie  Simulation of surgical safety checklist use  Patient interviews by phone include basic quality tracers (access to sanitation facilities; recall health education messages; informal payments and general satisfaction using a Likert scale) Kyrgyzstan: multiple approaches to measuring quality 67
  68. 68. 2. Using Data for Decision Making 68
  69. 69. Nigeria: Institutional Deliveries increased from 20% to 44% during 2015(120% increase) 69 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15 Population Coverage for Institutional delivery – PBF districts National (PBF) Adamawa Nasarawa Ondo 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15 Population Coverage for Institutional Deliveries – DFF districts National (DFF) Adamawa Nasarawa Ondo
  70. 70. 70 Large variability in Institutional Deliveries across Health Centers Fufore District, Adamawa State Nigeria during 2013 - 20 40 60 80 100 120 Pariya HC Chigari HC Dasin Hausa HC Farang HC Ribadu HC Furore MCH HC Choli HC Gurin HC Malabu HC Karlahi HC Wuro Bokki HC Kabilo HC Saint Mary's Clinic HC Mayo-Ine HC
  71. 71. Burundi: Average total quality score for health centers, by province and time 71 0.0 20.0 40.0 60.0 80.0 100.0 Mwaro Muramvya Kirundo Cibitoke Buja-Rural Kayanza Ngozi Makamba Rutana Bubanza Bururi Gitega Karuzi Muyinga Ruyigi Cankuzo Buja-Mairie
  72. 72. Quthing District: average quality in health centers is the same after 12 months piloting of PBF due to autonomy problems 72 0 10 20 30 40 50 60 70 80 90 100 General_Management Child_Survival Environmental Health General_Consultations Reproductive_Health Essential_Drug_Management Tracer_Drugs Maternal_Health STI_HIV_TB Comm_Based_Services 2Q14 2Q15
  73. 73. 3. Verifying Data Accuracy 73
  74. 74. NIGERIA: Quality of Care at PHCs: Raising the Bar 0 10 20 30 40 50 60 70 80 90 100 December March June September December March June September December March June September December March 2011 2012 2013 2014 2015 PercentageQualityScore Adamawa Nasarawa Ondo National  Quality of care also improved significantly with emphasis on structural and process of care indicators (higher emphasis on process end 2013 leads to drop)  Overall patient perceptions on quality of care is relatively satisfactory  Counter-verification of the quality: relative large discrepancies 74
  75. 75. Concordance in 2015 and 2016 75 90% 97% 81% 66% 61% 66% 95% 96% 76% 76% 85% 92% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Fufore LGA Mayo Belwa LGA Wamba LGA Karu LGA Ile oluji / Okeigbo LGA Ondo East LGA 2015 Average Concordance 2016 Average Concordance 0-5% Concordance
  76. 76.  Ex-ante verification by district health team may be too gentle and not accurate: too close for comfort or still old fashioned ‘filling under the banana tree’?  Regular counter-verification with credible sanctions are an important requirement  Specifying incentives for district supervisors also seems a promising route (share of earnings; accreditation status; carrots and sticks)  Introduction of modern ICT such as tablet based checklists, which embed meta data (location; time; interviewer passcode) seem a promising approach too 76 Challenges to Measuring and Rewarding Quality Performance
  77. 77. 4. Innovations in Measuring and Paying for Quality 77
  78. 78. Virtual Patient presents with symptoms Provider cares for a variety of clinical cases Provider goes through the different clinical domains as when they see a patient Vignettes Provide a Standard Measure of Practice 78  Take History  Conduct a Physical Exam  Order Tests  Make a provisional diagnosis  Decide on treatment
  79. 79.  Tablets for quantified quality checklists (‘balanced score cards’) with automated uploads to a cloud based database and public dashboard. Offline data entry possible  (as above) Tablet based solution for Vignettes (under development)  Smart phone for community client interviews. Off line data entry possible. Automated uploads to a cloud based database and public dashboard. Results impact on performance payments  Web-based public dashboard for performance benchmarking 79 Technology Aids for Quality Measurements in PBF
  80. 80. 80
  81. 81. 1. Quality is poor and varied 2. Much improvements in access and structural elements of care 3. Improving clinical processes remains the big immediate challenge 4. Innovations are happening in the space of measuring clinical processes 5. Data from measurement needs to translate to decisions In Summary 81
  82. 82. Improving Quality of Care Using Measurement, Comparison, Validation
  83. 83. • If we can measure: • Target performance • Knowledge to perform • Capacity to perform • Performance • Then the gap between performance and targeted performance can be broken down into: • The know gap • The know-can gap • The can-do gap Target Performance Gap Know Gap Know-Can Gap Can-Do Gap Target Knowledge to perform Capacity to perform Performance Three Gap Model of Performance (Leonard et al., 2015) 83
  84. 84. The Three Gaps in Liberia from 2013 to 2015 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 30% 39% 28% 11% 13% 25%2% 2% 2% 57% 46% 45% performance can do gap know can gap know gap 2015 full 2015 partial 2013 partial The three samples include 10 hospitals in 2015 (2015 Full) and 4 hospitals observed in both 2013 (2013 partial) and 2015 (2015 partial) 84
  85. 85. What do we learn from these gaps?  This is not a pure impact evaluation: the biggest driver of changes in this data is the Ebola crises, not the RBF.  The biggest change from 2013 to 2015 is an increase in the can-do gap, which suggests a drop in motivation consistent with the crises.  The biggest gap is clearly the knowledge gap, but does this mean improved knowledge leads to improved performance? 85
  86. 86. Performance .2 .4 .6 .8 1 Competence to Perform bandwidth = .8 Examine the relationship between competence to perform and performance in the full sample. Does performance increase with competence? When health workers work in teams, performance can be high even if competence is low, but we can see evidence that increasing the competence of health workers at the lower end can improve performance. But at the upper end, improving competence does not improve performance, even though average performance is low. 86
  87. 87. How to measure?  Many tools are available to measure process quality. – Clinical Observations, Simulated Patients, Standardized Patients, Paper-Based Vignettes, Tablet-Based Vignettes, Video Vignettes, Patient Chart Audit…  Identify the key bottlenecks. – Observing relatively rare events is difficult and costly. – Consider simulations and vignettes.  Know your sample size. – Larger countries will require larger banks of vignettes or simulations. – These are costly to set up, but remember that rapid data means investing in these high startup costs.  Ken Leonard’s work in Tanzania shows that there are many ways of increasing attention span. 87
  88. 88. The Kyrgyz Performance Based Payments (PBP) Project: work jointly done with Aneesa Arur, Arsen Askerov, Jed Friedman, and Asel Sargaldakova  Kyrgyz Republic has had persistently high (for the region) maternal and neonatal mortality rates – Near-universal institutional deliveries (over 95%) and coverage of primary care services  Hypothesis is that poor quality of care is limiting improvements in MMR and NMR  Project aims to improve quality of care for Maternal and Neonatal Health (MNH) – 3 year pilot of Performance Based Payments (PBPs) focused on quality of MNH services at district hospitals – Quality to be assessed by peer evaluators every quarter using a Balanced Scorecard which includes structure, clinical care and process measures of quality (more on this later) – PBPs will be a dimension of Diagnosis Related Group (DRG) payments for MNH services; Hospital Directors have autonomy over use – In addition, hospitals expected to also receive performance feedback as part of the PBP intervention package 88
  89. 89. Measuring Quality of Maternal and Newborn Care  The study uses data from the baseline survey of the PBP Impact Evaluation  This survey was conducted in all 63 Rayon Territorial Hospitals and Centers of General Practice in the Kyrgyz Republic.  Instruments included: 1.Health facility assessments: Hospital assessment and ANC checklist 2.Simulated patients for post partum hemorrhage and neonatal asphyxia 3.Direct observations of deliveries and antenatal care visits 4.Clinical record audits for normal deliveries, complicated deliveries, stroke, AMI, neonatal asphyxia 5.Patient exit interviews  All components used structured (quantitative) questionnaires or checklists to collect data, and all field workers were trained clinicians 89
  90. 90. Direct Observation: Labor and Delivery 90 92% 78% 80% 45% 40% 87% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Palpates uterus 15 minutes after delivery of placenta Takes mother’s vital signs 15 minutes after birth Tasks for second and third stage of labor [4] Complications during previous pregnancies [3] Danger signs [2] General tasks for initial client assessment [1] [1] Checks clients card or asks client her age, length of pregnancy, and parity, Takes temperature, Takes pulse, Asks/notes amount of urine output, Performs general examination (e.g. for anemia, edema), Performs abdominal examination: checks fundal height with measuring tape, Performs abdominal examination: checks fetal presentation by palpation of abdomen, Performs abdominal examination: checks fetal heart rate with fetoscope/ultrasound, Performs vaginal examination (cervical dilation, fetal descent, position, membranes, meconium) [2] Fever, Foul smelling discharge, Headaches or blurred vision, Swollen Face or Hands, Convulsions or loss of consciousness, Shortness of breath, Vaginal bleeding [3] High blood pressure, Convulsions, Heavy bleeding during or after delivery / hemorrhage, previous c-section, Prior stillbirth, Prolonged labor, Prior neonatal death, Abortion, Prior assisted delivery [4] Supports perineum as baby's head is delivered, Assesses completeness of the placenta and membranes, Assesses for perineal and vaginal lacerations
  91. 91. Pre-eclampsia/eclampsia Knowledge Test 91 51% 78% 62% 74% 92% 68% 58% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Wrong: Actively Restrain Wrong: Give Intravenous Diazepam MeanActions To Take If Presented With Convulsion [2] Action to take: stabilize with Anti-Hypertensives Action to take: stabilize with Magnesium Sulfate Proper Diagnosis: Severe Pre-Eclampsia Mean Examination Actions [1] [1] Time Of Onset Of Present Symptoms, Level Of Consciousness, Any Convulsions, Check Vital Signs (Temp, Bp, Pulse, Respirations), Listen To / Assess Fetal Heart Tones, Fetal Movement, Check Urine Protein [2] Administer Oxygen At 4-6 L Per Minute If Available, Place In Side Lying Position, Protect From Injury, Give Magnesium Sulfate, Provide Anti-Hypertensives (Nifedipine Or Apresoline), Actions To Take If Presented With Convulsion: Mean
  92. 92. Comparing Patient Exit Interviews with Direct Observations 92 0.000 0.100 0.200 0.300 0.400 0.500 0.600 0.700 0.800 0.900 1.000 HIV status Blood pressure Urine test Augment Episiotomy Timing of Meds Dry Skin-to-skin Covered Initial Client Assessment Intermittent Observation of First Stage Labor Continuous Observation of Second and Third Stage Immediate Care Exit Interview: Unobserved Exit Interview: Observed Direct Observation
  93. 93. Measuring Quality of Maternal and Newborn Care  Administrative data from all 63 RTHs and CGPs on preventable maternal and neonatal complications that are targeted by Kyrgyz RBF pilot.  Extracted data on ICD-10 codes used for DRG payments on:  Perineal lacerations  Post-partum hemorrhage  Other obstetric trauma  Birth asphyxia  We calculate rates of delivery and neonatal complications for both types of hospitals and test the various measures of QoC from the survey data against these complications rates to see which measures are more predictive of complications rates. 93
  94. 94. Summary of Key Findings* 1.Instruments appear to be better suited to predicting complications rates for Territorial Hospitals rather than Centers of General Practice. 2.Criterion-based Clinical Audits do not appear to be predictive of hospital quality, particularly in Kyrgyzstan where meticulous documentation was not incentivized prior to the RBF pilot. 3.Direct Observations perform better in terms of having the expected sign on the correlation, but are often not significant predictors of QoC. 4.Simulations using the MamaNatalie anatomical model were more predictive of the administrative maternal and neonatal complications rates.  This finding is important from a policy perspective because training and evaluations of provider skill as well as IEs can use this relatively inexpensive tool. 94
  95. 95. Some Caveats 1.While we use data on the case mix treated by these hospitals, and consider preventable complications that are targeted by the Kyrgyz RBF pilot: a)Our results may be driven by the fact that complicated cases are systematically referred to some of these hospitals. b)However, the cadre of hospitals considered here is not the type that patients are referred to. c)Further, we attempted to select complications that were less likely to be screened through antenatal care. d)In addition, we account for hospital type in the analysis. 2.Further, unobservable third factors may lead to certain areas having less healthy populations 3.Certain complications may also be beyond the control of the hospital and may instead be a factor of the quality of ANC. 95
  96. 96. RBF and Quality of Care: What the impact evaluations are telling us
  97. 97. Evidence base for RBF and QoC is slim  Das et al. (2016) systematically review the published literature and find 8 studies that explore RBF impacts on QoC with methodological rigor  Wide variation in the studies – Burundi, DRC, Egypt, Philippines, and Rwanda – 3 RCTs, 4 dif-in-difs, 1 propensity matched case-control – 5 focused on PHCs, 2 on district hospitals, 1 on both – 3 directly incentivized limited set of quality indicators, 3 utilized composite quality index (BSC) – 3 directly paid health workers, 4 paid facilities – Incentives ranged from 5% to 275% of base salary – Measurement of quality includes hosuehold interview, patient exit interview, record review, direct observation, and vignette responses 97
  98. 98. Evidence base for RBF and QoC (II)  Wide variation in the findings: – Structural quality: very mixed findings ▫ Increase in number of qualified staff and drug availability in DRC 1 ▫ Increase in clinical knowledge in Philippines ▫ However majority of cases find little change – Process quality: some gains in ANC processes ▫ History taking, blood tests, urine tests increased in Egypt ▫ Summary process quality score improves by 0.2 SDs in Rwanda ▫ However no change in DRC, and no measurement in other studies – Quality outcomes: again, mixed findings ▫ Improved patient knowledge in Egypt and DRC ▫ Improved client satisfaction in DRC 1 and Burundi but not DRC 2 ▫ Little change in assessed health outcomes (nutritional status of U5s improves in Rwanda) 98
  99. 99. Evidence base for RBF and QoC (III)  Very difficult to generalize from current evidence base – Diversity of program design and involvement of QoC – Most evaluations not primarily concerned with QoC  Despite several programs granting autonomy and funds to enhance structural quality, evidence of improvement is minimal – Procurement and managerial bottlenecks?  Does increase in utilization negatively spillover onto QoC?  These mixed findings call for deeper investigation into – Design of RBF programs – Implementation of programs 99
  100. 100. Evidence base for RBF and QoC (IV)  RBF impact evaluation portfolio is expected to generate much more evidence (eventually over 30 country studies)  Let’s review in-depth results from two recently completed studies: – Zambia – Zimbabwe 100
  101. 101. Both Zambia and Zimbabwe saw gains in select targeted coverage measures  Delivery – In-facility deliveries increased 12.8 percentage points in Zambia – 13.4 pp increase in Zimbabwe  ANC and PNC – Concomitant gains in PNC in both countries – No gain in ANC coverage in either country  Family planning – No gains in Zambia – 12 pp increase in Zimbabwe, only among women with primary education or below  Child health – No improvements in vaccination coverage in Zimbabwe – 6-7 pp increase in select vaccination measures in Zambia – 4 pp reduction in extreme stunting in Zimbabwe 101
  102. 102. Zambia: Structural Quality • Little change in individual measures of structural quality, however an aggregate index suggests gains in RBF compared with pure control districts • Gains in structural quality of care-specific indices 102 RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p- value Facility experiences no power outage -0.019 0.881 0.194 0.159 Facility experiences no water outage 0.041 0.688 0.051 0.476 Infrastructure index 0.195 0.470 0.483* 0.099 RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Curative Care 0.39 0.204 0.28** 0.042 Family planning 0.15 0.578 0.08 0.546 Delivery Room 0.61** 0.010 0.57*** 0.000
  103. 103. Zambia: Quality of ANC • Process measures of ANC quality for a few measures are improved in RBF as compared to C1 and C2, but little gain in overall index • Household survey results suggest 3 percentage point increase in IPT coverage: a directly targeted process quality indicator 103 RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p- value Weighed -0.02 0.632 0.06 0.251 Blood pressure measured -0.03 0.809 0.08 0.452 Abdomen measured 0.07 0.152 0.09* 0.063 Abdomen palpated 0.00 0.987 0.12* 0.083 Advice on diet 0.14*** 0.009 0.02 0.850 Quality of ANC index 0.02 0.921 0.33 0.165
  104. 104. Zambia: Quality of child health care • No apparent gain in process quality of child health visit 104 RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value Asked age -0.01 0.880 0.02 0.776 Weighed child -0.07 0.378 0.06 0.498 Measured height -0.10 0.104 -0.02 0.577 Physically examined -0.09 0.327 -0.08 0.350 Quality of care index -0.09 0.669 0.14 0.565
  105. 105. Zambia: Satisfaction on ANC • Higher levels of patient satisfaction in selected dimensions of ANC (but not all) in RBF as compared to the two controls • Little apparent increase in overall satisfaction 105 RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value The health worker spent a sufficient amount of time with the patient 0.08* 0.067 0.08* 0.081 You trust the health worker completely in this health facility 0.07* 0.066 0.03 0.569 Satisfaction index 0.04 0.826 0.12 0.574
  106. 106. Zambia: Satisfaction on child health care • Little apparent increase in overall satisfaction for child care 106 RBF vs. Control 1 RBF vs. Control 2 Impact estimate p-value Impact estimate p-value The amount of time you spent waiting to be seen by a health provider was reasonable -0.02 0.823 -0.06 0.477 You trust the health worker completely in this health facility 0.11* 0.057 0.04 0.504 Satisfaction index 0.09 0.617 0.04 0.858
  107. 107. Zimbabwe: Structural Quality  Improvements in select measures of structural quality:  Higher incidence of biomedical waste disposal (16 % points; p = 0.027)  Increased availability of iron (16 pp), folic acid (21 pp), and urine dipsticks (42 pp)  Increased availability of select equipment  electric autoclave (29 pp) and refrigerator (27 pp)  However no gains in majority of measures 107
  108. 108. Structural Quality - Mapping of Checklist  Elements from the quality checklist were extracted from the facility survey instrument and assigned the same weight to calculate the indices. Process Quality – ANC (Household Survey) Zimbabwe: Structural and Process Quality 108 Impact estimate p-value Administration and planning 0.167 0.674 Medicines and sundries stock management 0.017 0.969 Out Patient Department 0.468 0.213 Family and Child Health 0.837** 0.021 Maternity Service 0.009 0.981 Referral services 0.182 0.667 Community services 0.049 0.866 Infection control and waste management 0.492 0.272 Impact estimate p-value Blood pressure measured 0.025 0.570 Urine sample taken 0.153** 0.027 Blood sample taken 0.084 0.129 Any tetanus injection 0.075* 0.056 Number of tetanus injections 0.312* 0.063 Any iron taken 0.003 0.951 Number of days iron taken -1.161 0.868 Anti-parasite drugs taken 0.031 0.117 Malaria prophylaxis taken 0.033 0.654
  109. 109.  Quality of service indicators recorded in the HMIS also show significant increases Zimbabwe: Process quality in the HMIS 109  Even for indicators that show no significant increase from patient recall data.
  110. 110. Main takeaways and priority questions  Systematic review and two country studies suggest – RBF is effective to improve process quality of ante natal care – Very mixed results on structural quality and client satisfaction – Little evidence in either direction on (a) quality of other processes, (b) long run health outcomes  Challenges with QoC improvements suggest need to revisit how QoC is measured and incentivized under RBF  Scope to revisit efficiency of RBF spending: reallocate funds away from coverage indicators where coverage is already high and towards quality indicators  Combine RBF with complementary investments in quality improvement (e.g. CQI) to amplify RBF impacts on quality?  Incentivize activities involved in the facility management of quality? 110
  111. 111. THANK YOU