Annual Results and Impact Evaluation Workshop for RBF - Day One - Strengthening Quality of Health Services in RBF Projects
Upcoming SlideShare
Loading in...5

Annual Results and Impact Evaluation Workshop for RBF - Day One - Strengthening Quality of Health Services in RBF Projects



A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.



Total Views
Slideshare-icon Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Annual Results and Impact Evaluation Workshop for RBF - Day One - Strengthening Quality of Health Services in RBF Projects Annual Results and Impact Evaluation Workshop for RBF - Day One - Strengthening Quality of Health Services in RBF Projects Presentation Transcript

    • Strengthening Quality of Heath Services in RBF Projects Dinesh Nair, Ronald Mutasa, Rianna MohammedPatron Mafaune, Kathleen Hill
    • Presentation Outline  Session objectives and overview  Integrating Quality into RBF projects  Quality in global health  Principles/approaches to improving quality  Integrating quality into RBF projects  Prioritizing health conditions  Selecting standards and defining quality measures  Measuring and verifying quality  Aligning RBF with QI & Health System Strengthening efforts  Country Examples: Liberia & Zimbabwe
    • Session Objectives  Understand importance of quality and gain familiarity with improvement approaches  Understand common operational needs and strategies of RBF projects that incentivize quality  Understand experience to date in RBF projects that incentivize quality
    • 4 The Issue of Quality in Health Care High quality care is…… Effective: Adherent with evidence-based standards Safe: does not harm patients Client centered: Respectful of patient needs, values & preferences Equitable: Does not vary in quality because of personal characteristics (gender, ethnicity, SES, etc) IOM, 2001, Crossingthe Quality Chasm
    • Preliminary Results from Service Delivery Indicator (SDI) Survey: Nigeria Sample of 1,172 health facilities in 6 states (one from each geo-political zone) end 2013 – not yet public  Diagnostic accuracy less than 40% (judged by vignettes) - less than half that observed in Kenya  Adherence to guidelines< 1/3rd (by vignettes)  Knowledge of maternal & neonatal complications< 20%, <40% of what it is in Kenya
    • Closing the Know-Do-Gap Between Proven Best Practices and Actual Practice in the Places Patients Receive Care 6
    • Conceptualizing Quality (Source: Donabedian) 7
    • Principles Underlying Improvement  Effective teamwork  Understanding how processes of care function within a system and critical bottlenecks  Use of data to continuously measure and track progress  Understanding and focus on patient needs  Regular shared learning for rapid dissemination best practices 8
    • Prioritizing What to Improve Deciding How to Measure if Care Is Improving 9  Choosing improvement Aims: “What are we trying to accomplish?”  Defining quality measures: “How will we know if a change is an improvement?”
    • 10 Improvement Team Reviewing Quality Indicators
    • Measuring Quality to Improve Care: Who Needs What Data for What Purpose? Facility Staff (managers, providers, staff -hospitals and clinics): -Need quality measures to assess and continuously improve services. Is care improving? Regional/District & Program Managers: -Need measures to assess and continuously strengthen essential system functions (e.g. competent workforce). Are essential system functions performing to standard? --------  Clients (users of care)  National Policy-makers (value, policy)  Global Stakeholders (leadership, advocacy, accountability)
    • Integrating Quality into RBF Projects: Prioritizing Health Conditions/Services for Improvement Focus on:  High-burden conditions in local context (leading causes of mortality and morbidity)  High-burden conditions for which there is strong evidence of effective health care interventions (preventive and curative)  Country government priorities: involve local decision makers and experts  Consider phasing improvement priorities: “impossible to improve everything at once”
    • Involve local and international experts to:  Review country standards against global evidence: evidence is constantly changing  Distill standards into minimum “interventionbundles”: focus attention on essential high- impact interventions  Illustrative quality of care process measures based on minimum standards:  % cases adherent with standards – “all or nothing adherence” (e.g. % PPH cases managed per minimum standard; % cases pediatric pneumonia treated per standard)  Average % adherencewith minimum standards (e.g. average % adherence with newborn sepsis case-management standards; N=30 cases) Integrating Quality into RBF Projects: Selecting Standards and Defining Quality of Care Measures
    • Ilustrative quality measure: Quality of Partogram Completion (not so simple!) Quality Measure Operational Definition  % partograms in last quarter completed per standard  NUMERATOR: Number partograms documenting cervical dilation, maternal BP, pulse, temperature at admission and at least every 4 hours until delivery  DENOMINATOR: Total number of partograms reviewed
    • Defining Quality of Care Measures (continued)  Review quality measures being used at country and global level; adapt or develop new measures  Consider including measures of care coordination and performance of essential system functions (e.g. referral/counter-referral; supportive supervision)  Streamline and harmonize indicators: the fewer the better!
    • Improving adherence with post-partum care best practices: Average compliance with PNC standards Herat Province Afghanistan- Nine Health Centers; 2009-2011 16
    • Measuring Quality: The challenges….. JAMA Nov 13. 2013 “Quality measurement is in rapid flux….despite the challenges of a rapidly expanding number of quality measures, much of health care remains poorly measured or unmeasured.”
    • Measurement Methods  Observation  Patient interviews & questionnaires (e.g. exit interview; household survey)  Death (and near-miss) audits  Simulations (provider competence)  Provider knowledge/problem-solving (e.g. vignettes/case studies)  Routine information systems (e.g. facility chart & register audits; routine health information system) -Regular measurement is a core principle of all improvement, but measurement alone will not improve care! 18
    • Common Measurement Challenges for QI and RBF efforts  Routine HMIS in low-resource settings include few (if any) quality/content indicators  Primary data sources (medical records/registers) often don’t include necessary data for constructing quality measures  Quality of clinical procedures (e.g. newborn resuscitation) cannot be assessed from a medical record; creative measurement approachesneeded  Few routine indicators of performance of essential system functions (e.g. % maternities in district with functional neonatal bag & mask at bedside)  Weak staff data management capacity (providers and managers)
    • Routine Versus Complications Care  Easier to measure routine best practices relevant for every patient: often simple intervention for which a “box” can be checked in a standard record (e.g. ENC, AMTSL; immunization)  More difficult to measure quality of complications care  Timely accurate diagnosis  Stabilization and successful timely referral (primary facility)  Prompt and ongoing treatment/monitoring (hospital)  Discharge planning and follow up 20
    • 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% J08 F08 M08 A08 M08 J08 J08 A08 S08 O08 N08 D08 Percentage of pre-eclampsia and eclampsia case management standards achieved Jan-Dec 2008, average of 120 cases analyzed per month Based on detection in 120,000 patient contacts at 31 MOH facilities Measuring Adherence with Pre-eclampsia/eclampsia Best Practices to Improve PE/E Care in Niger 31 facilities 21 Changes implemented: •Systematic screening at every contact •Organization daily tasks to prevent stock outs •Standardized emergency case management eclampsia •Standardized referral protocols
    • Pre-eclampsia/Eclampsia chart audit tool for primary facility 22 Charts Evaluation 1 2 3 4 5 1. Blood pressure (BP) recorded 1. Gestational age (GA) recorded (per one of criteria indicatedin GUIDE) 1. Urine protein quantified (dipstick +, ++, +++) 1. Danger signs assessed (see chart review guide) Diagnosis pre-eclampsia or eclampsia recorded if criteria met 1. DBP > 90 and at least 2+ proteinuria pre-eclampsia(+ seizure if eclampsia) First Treatment and referral if primary facility 1. 4 gm loading dose of MgSO4 IV ; monitor for toxicity (reflexes, urine output, respirations) 1. If GA < 34 weeks administer antenatal corticosteroids 1. Transfer with IV access (and provider if possible)
    • 23 Despite challenges it is possible to track quality measures from (modified) local records (e.g. add columns to registers; stamps to records/partograms, etc.)
    • WHO MNCH Quality of Care Consultation Geneva December 2013: Objectives Share global and regional experiences in the assessment and improvement of facility MNCH QoC Review/agree on a core and supplementary set of global indicators for accountability - monitoring and reporting on facility MNCH QoC Review assessment tools, methodologies and processes used in measuring facility MNCH QoC
    • WHO Consultation Draft Set Global Newborn Indicators: Mix of Structural, Quality of Care Process, Outcome indicators Proportionof health facilities with maternityservices that have functional bag& masks (2 neonatal mask sizes) in the deliveryareas Proportionof newborns whoreceived all four elements of essential newborn care:  immediate and thorough drying  immediate skin-to-skincontact  delayed cord clamping  initiation of breastfeeding in the first hour Proportion health facilities where Kangaroo Mother Care is operational, by level of facility Facility neonatal mortality rate disaggregated by birth weight: >4000 g, 2500-3999 g, 2000- 2499 g, 1500-1999 g, < 1500 g Proportionof health facilities offeringmaternityservices that have BFHI certification and recertification not older than two years
    • Aligning RBF with QI and HSS Efforts: Challenges and Opportunities  RBF projects can be integrated with and/or incentivize QI and HSS efforts  National QI strategy implementation  Pre and in-service training & performance-based supervision  Supply chain interventions  HMIS improvement interventions, etc.  RBF projects can incentivize QI activities and quality performance measures  Regular QI team meetings  District/facility routine collection & analysis of quality measures with action plans  District/facility continuous quality improvement (CQI): setting aims; developing and tracking quality measures; testing changes, sharing learning….  Accreditation, etc.
    • Country Case Example: Zimbabwe  Dr. Patron Mafouna, Provincial Medical Director, Maniacaland Province  Mr. Ronald Mutasa, World Bank
    • • Focus on register completion (“# columns checked”) rather than processes of clinical care (specific clinical interventions) • Heavy emphasis on “structural quality” (e.g. appearance of the facility, medications, supplies, staffing) • Poorly defined clinical quality indicators (e.g. numerator not clearly defined), inter-rater reliability issues • Limited analysis and monitoring of individual checklist items • Limited link between incentives and clinical quality of care Zimbabwe Experience With Original Checklist Quality Items (18 districts; Hospitals and Clinics)
    • Improving Quality Measurement Over Time: Illustrative example “Partogram completion” indicator RBF Project Phase Indicator Checklist Guidelines for calculating indicator Initial Phase % partograms correctlycompleted in last quarter No criteria specified weak quality measure Current Phase % partograms correctlycompleted in last quarter Randomly review 10 partograms from last quarter Calculate % partograms documenting specific criteria: FHR, cervical dilation, BP, pulse, temperature documented at admission and at least every 4 hoursuntil delivery
    • Beyond Structural Care: Phasing quality incentives for high-burden MNCH Conditions by project phase RBF Phase One (9-12 months) RBF Phase Two  Routine MNH/FP  PPH  Obstructed labor  Maternal & Newborn Sepsis  Pediatric pneumonia  Pediatric diarrhea  Pediatric malaria All Phase One priority conditions plus:*  Pre-eclampsia/Eclampsia  Newborn Asphyxia  Miscarriage/abortion c management  Management acute malnutrition
    • Zimbabwe: Beyond structural incentives  Innovations to introduce continuous quality improvement and rigorously evaluate  P4Q alone versus  P4Q plus continuous quality improvement  Supporting MOHCC to draft and implement national/regional QI strategy…..align RBF with strategy  Supporting MOHCC to introduce quality indicators into HMIS…..strengthening HMIS
    • Zimbabwe: Lessons being learned  Technical capacity & substance matters in P4Q, so does political economy & building ownership by local stakeholders  Complex trade-off between structural and clinical care indicators  Health providers respond to P4Q indicators and incentives  Source documents at provider level matter  Focus on QI principles and emerging global best practices
    • Country Case Study: Liberia  Rianna Mohammed, World Bank
    • • High infection rates and post-surgery complications • No systematic use of clinical guidelines • Accreditation scores on quality are worse than primary facilities Low Quality of care at Hospitals: 3 4 Hospitals characterized by: • Poor infrastructure; inadequate supply of drugs and equipment. • Insufficient numbers productive, responsive, competent staff • Long waiting times • Limited internal and external financing (e.g. 85% of health expenditure in 2009/10 directed almost entirely at the PHC-level). LIBERIA: Neglect of Hospitals...poor quality of care
    • ALIGNING PBF AND SYSTEM STRENGTHENING: LIBERIA PROJECT SEEKS TO IMPROVE QUALITY VIA PBF AND COMPETENCY BUILDING 35 Hospital PBF Competency Building Incentives for: a) improved quality of care (i.e. adherence to clinical protocols) b) quantity of services delivered (including in- service training to health workers) • Coaching and verification support Better competency to improve processes of care • Support to development of innovative Graduate Medical Residency Program. • In-service training of nurses, midwives and PAs Synergistic linkage of components Improved staff motivation
    • QUALITY ASSESSMENT/ MONITORING TOOLS 36 1 Complicated and assisted delivery (including C-section) Any labor that is made more difficult or complex by a deviation from the normal procedure. Complicated delivery is defined as: assisted vaginal deliveries (vacuum extraction or forceps), C-section, episiotomy and other procedures. 17 2 Normal deliveries of at risk referrals High-risk pregnant women referred by health center to the hospital but delivered normally. A high-risk pregnancy is defined as: evidence of edema, mal presentation, increased BP, multi-parity, etc. 17 3 Counter referral slips returned to health facilities Hospital returns counter referrals letter with feedback on the referred patient to the referring health center. The counter referral letter is completed in triplicate, with one also given to the patient, and one retained by the hospital. 2.5 4 Newborn referred for emergency neonatal care treatment and treated Newborns referred for emergency neonatal care due to: perinatal complications, low birth weight, congenital malformation, asphyxia, etc. 5 6 Referred infants and under-fives with fever Any surgical procedure that does not involve anesthesia or respiratory assistance. 2.5 7 Minor surgical intervention Any surgery in which the patient must be put under general spinal/anesthesia and given respiratory assistance. Major surgery in the case of this package of services is defined as any of the following: Herniarraphy, Appendectomy, Myomectomy, Sleenectomy, Salpingectomy, Hysterectomy, Thyrodectomy, Mastectomy. 5 8 Major surgery (excluding CS, including major trauma) Patients transferred from a lower-level facility (health center or health clinic) to the hospital for emergency treatment. 18 9 Patients transported by ambulance 2.5 10 Number of training sessions held by faculty for nurses, midwifes and PA according to in-service curriculum and defined protocols. These indicators will incentivize the in-service training activities. 50 11 Number of nurses, midwifes and PAs that received specialized in-service training, relevant to benchmarks 10 Verified Total EarningsDefinition Six Hospitals Total Fee (USD)Indicators Claimed (c) Quantity Checklist Actual % Earned Points 1. Obstructed Labor 0.80 3.87 100% 33% 1.29 2. Hemorrhage 1.00 4.84 100% 71% 3.45 3. Maternal Sepsis 1.00 4.84 100% 50% 2.42 4. Eclampsia 0.70 3.39 100% 47% 1.59 5. Neonatal Asphyxia 1.00 4.84 100% 67% 3.23 6. Neonatal Sepsis 1.00 4.84 100% 54% 2.61 7. Prematurity 0.50 2.42 100% 47% 1.14 8. Maternal Newborn Best Practices 1.00 4.84 100% 54% 2.61 9. ETAT 1.00 4.84 100% 33% 1.61 10. Malaria 1.00 4.84 100% 71% 3.45 11. Pneumonia 1.00 4.84 100% 50% 2.42 12. Acute Diarrhea 0.80 3.87 100% 47% 1.82 13. Severe Acute Malnutrition 0.60 2.90 100% 67% 1.94 14. Surgical Safety 1.00 4.84 100% 54% 2.61 100% 60.00 100% 53% 32.20Total/Average Childbirth: Maternal-Newborn Pediatric (in-patient care) Surgical Care Quarter I III. Process of Care Detailed Score Checklists Weight (by importance) Point Allocation Max % (b) Process of Care Quality Checklists Score 1.GENERAL MANAGEMENT (30pt) 2. HUMAN RESOURCES FOR HEALTH (16pt) 3. HYGIENE AND MEDICAL WASTE DISPOSAL (27pt) 4. DRUGS MANAGEMENT (30 pt) 5. EQUIPMENT AND SUPPLIES (84pt) TOTAL % Date of Verfication TOTAL (187pt) REPUBLIC OF LIBERIA Ministry of Health and Social Welfare (MOHSW) Hospital Quarterly Quality Assessment Name of the Hospital Name of Team Leader of Quality Verification Verification Period Quarterly Quality Verification Score I. Management II. Structural (a) Management and Structural Checklist Indicators Max Points Actual Points Quarter I 1. General Management 30 2.6 2. Human Resources for Health 16 9 3. Hygiene and Medical Waste Disposal 27 0 4. Drugs Management 30 8 5. Equipment and Supplies 84 48 6. Aggregated Process of Care Score 60 32 Total 247 100 Total Percentage 100% 40% Total Quality Bonuses (USD) 159,678 64,517 PBF Bonus Calculation Tool Business/Operation Plan Health Worker Bonus Allocation LHSSP Indices Tool for Bonus Allocation to Individual Health Workers for Hospitals 1 200 50 30 300,000 0 6,944 2 200 70 30 420,000 0 9,722 3 150 80 30 360,000 0 8,333 4 - - - 5 - - - 6 - - - 7 - - - 8 - - - 9 - - - 10 - - - 11 - - - 12 - - - Quarter: Total PBF Incentives Earned % for Individual Bonus Attendance points [C] Hospital Name Total Individual Bonus Redemption Hospital July-Sept 2013 No Name of staff Staff category Monthly salary [A] Perfor- mance points [B] $50,000 50% $25,000 Total points = [A] x [B] x [C] Indices of the period PBF individual bonus Signature of receipt Min 50% Max 50% ~60 % ~20% ~20% (1)Continuousmonitoring (d) Impact Evaluation
    • Quality of Care Checklists for High-burden MNCH Conditions (complications): Based on National and WHO Standards Process of Care Checklists Childbirth: Maternal-Newborn 1. Obstructed Labor 2. Hemorrhage 3. Maternal Sepsis 4. Eclampsia 5. Neonatal Asphyxia 6. Neonatal Sepsis 7. Prematurity Pediatric (in-patient care) 8. Maternal Newborn Best Practices 9. ETAT 10. Malaria 11. Pneumonia 12. Acute Diarrhea 13. Severe Acute Malnutrition Surgical Care 14. Surgical Safety 37
    • 38 Chart review elements (see chart review guide for specific criteria) ; each element if recorded = 1 point Charts 1. Admission 1 2 3 4 5 1. Cervical dilation recorded at admission (# of cm) 2. Contraction frequency and duration charted at admission 3. Fetal presentation charted at admission 4. Partograph started when cervical dilation 4 cm or greater Admission Score (x/4) 2. Labor Monitoring (partograph) 1. Cervical dilation recorded at least every 4 hours 2. Frequency and duration contractions recorded at least every 30 minutes 3. Fetal HR recorded at least every 30 minutes Labor Monitoring Score (x/3) Standards for Management Obstructed Labor: Illustrative Checklist Distilling Essential care Items (admission, labor) Each item has chart review guide that defines criteria Five patient charts reviewed: average score (% adherence best practices) links with bonus
    • VERIFYING QUALITY MEASURES: PATIENT CHARTS REVIEWED BY AN INDEPENDENT COUNCIL 39 Step 1: Find relevant patients from register (e.g. “Malaria” patient for Malaria checklist) Step 2: Record names and patient numbers Step 3: Request health workers to bring charts • Team of minimum 2 verifiers from Liberia Medical and Dental Council (LMDC) compare charts and guides/checklists • 5 charts each of 14 checklists reviewed quarterly • Total scores calculated as % adherence for each checklist • Aggregated points are tied to the performance bonus
    • INCENTIVIZING QI ACTIONS AS PART OF HOSPITAL PERFORMANCE: MANAGEMENT CHECKLIST 40 1.1 Performance managementstructure operational • Hospital Board meets every month and problem-solve issues • Senior Management Team (SMT) meets every month and problem-solve issues • Quality Improvement (QI) or other relevant Team meets at least every month and problem- solve issues 1.2 Business (operational) plan updated and implemented 1.3 Performance review 1.4 Reporting and filing of the key data 1.5 Financial Management 1.6 Community Involvement 1.7 Grievance mechanism 8pt 3pt 6pt 2pt 4pt 4pt 3pt Assessment Components Max points
    • INTEGRATING QI ACTIONS AND RBF: QI TEAM AND COACH MOTIVATES HEALTH WORKERS TO IMPROVE QUALITY THROUGH TRACKING QUALITY MEASURES AND COACHING 41 • Weekly/daily chart review • Monthly management and structural checklist review 1 Self-Assessment 2 15 20 30 40 0 50 W1 W2 W3 W4 % e.g., Sepsis • Post scores on a wall • Provide detailedfeedback to staff Tracking and Feedback 3 • Support improvement activities (e.g., standardize chart, waste disposal, cleaning) • Training on the treatment protocol and chart writing Support to Improvement QI Activities
    • Multiple levels of performancemanagement at hospital and MOHSW levels: Incentivizing QI Actions 42 Hospital QI team • Responsible for improving specific quality indicators • Carry out weekly/monthly self-quality assessments and track results at each ward • Provide feedback, coach and train health workers Senior management Team • Developbusinessplan with supervisors • Hold supervisors accountable for quality indicators • (At least ) monthly performance review with QI team & support of QI team action plans Hospital Board • Provide oversight to hospital management • Make SMT accountablefor results and implementation of business plan • Community involvement and grievance Technical Committee • Overall oversight of performance trendsand project implementation
    • Key lessons learned so far • Measuring quality can be really complicated – finding a right focus and starting from small is critical (e.g., pre-pilot, five hospitals only, inpatient only). • Management strengthening is essential but not easy to do – further experiments and hands-on coaching will be important. • Training and re-training on quality measurement is essential in ensuring that all stakeholders understand how quality will be measured, and trust the tools and the verifiers. • Understanding the political economy, and being open and flexible to change is important in building ownership among stakeholders. 43