Your SlideShare is downloading. ×
LQAS in Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

LQAS in Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment

926
views

Published on

Piloting a Performance Monitoring System in a Post-Conflict Environment, Stephanie Watson-Grant, Anna McCrerey

Piloting a Performance Monitoring System in a Post-Conflict Environment, Stephanie Watson-Grant, Anna McCrerey


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
926
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • 14 year period of civil conflict GDP fell by 90% and debt swelled to $4.6 billion
  • 14 year period of civil conflict Looking forward, Liberia must consolidate socioeconomic and political development foundations to minimize risk of regress. At the end of the conflict in 2003, Liberia was left with a Gross Domestic Product, which was only 50% of the 1988 level and is only now rising to pre-war levels of approximately $1 billion. The Government of Liberia (GOL) strives to become a middle-income country by 2030, which will require tremendous growth and accelerated progress. Recognizing that good health is essential to stability and economic growth, the Ministry of Health and Social Welfare (MOHSW) is working to improve access, availability, and quality of basic health services for all Liberians.
  • Infant Mortality Rate (IMR) decreased from 144/1,000 to 71/1,000 Under 5 Mortality Rate (U5M) decreased from 220/1,000 to 111/1,000 Important Details on Infant Mortality: Approximately 18% of <5 deaths are neonatal Most occur in the first 6 days if life. No neonatal-specific indicators in HMIS. Most associated with causes of maternal mortality. Important Details about still births Approximately 2,000 stillbirths per year in Liberia. Most occur just before birth and are associated with complications of labor and delivery or infections, e.g., STIs and malaria. Most could be averted. *other sources have been quoted as >40% stunting rate; we deferred to LDHS 2007 for this statistic Malaria prevalence has decreased by half in between 2007-09 Declines in IMR and U5M attributed to restoration of basic service provision and improvements in the health sector, especially in the area of malaria prevention and treatment. Even with the decrease, however, these mortality rates are still high. And despite the overall decline in mortality, high rates of key morbidities such as diarrhea, acute respiratory illnesses (ARI), malaria, and undernutrition are still concerning
  • In the 7 years since war ended, services have been restored at over 80% of public facilities. The government is currently contracting out services at 75% of these 303 health clinics, health centers and hospitals. 14 year period of civil conflict In 2007 - 300 facilities were NGO-managed with uncertain funding The health system was highly fragmented (information systems, salaries, drug procurement, etc.) Human resources for health crisis – only 51 practicing Liberian doctors and shortage of > 7,000 health workers.
  • Advantages - LQAS is a relatively rapid and inexpensive approach to data collection that provides a viable alternative to traditional surveys. The method allows for smaller sample sizes than standard probability surveys, and the lower associated costs allow for more frequent sampling . LQAS data can also be used in conjunction with other sources of health information , such as service statistics, to obtain information on coverage and quality of care at the population level . Also, the value of the exercise is its usefulness in local level decision making.
  • LQAS is currently the ONLY source of county level estimates in Liberia (both DHS and MIS only drill down to regional level, grouping several counties together) This health outcome performance monitoring system complements service statistics from the HMIS and is usable at the local level by County Health Teams (CHTs). Limitations: Does not produce point estimates for SAs Not intended to measure incremental change over time for the county-based catchment areas: Wide confidence intervals around point estimates Improvements will not register as statistically significant unless they are large Not useful for very low or high-prevalence indicators (<20% or >90%) at the SA-level
  • In LQAS application in Liberia, first step was to identify the catchment areas. A catchment area was defined as a county, and four counties were selected as LQAS pilot sites. Division of the entire program area, or catchment area, into meaningful sub-divisions, or “lots”, usually defined as program supervisory areas. At a stakeholder meeting in January, it was decided that the LQAS-based outcome monitoring survey would cover five program areas: water, sanitation and hygiene -+- immunization and child health -+- nutrition -+- malaria -+- maternal health -+- family planning Criteria were used to select indicators, such as programmatic relevance and viability of inclusion, given the structure of LQAS As a result, about 26 indicators were selected, and the following sub-groups were included in the study: women of reproductive age (15-49 years) and children aged 0-59 months -+- children aged 0-5 months -+- children aged 0-23 months -+- children aged 12-23 months -+- children aged 6-23 months -+- children aged 0-59 months that had in the previous 2 weeks: a diarrheal episode -+- a febrile episode -+- cough and difficulty breathing For each LQAS, all indicators included in the study have the same sample size. Each indicator has a denominator of 19 within a single supervisory area, which are enough observations to provide the acceptable/unacceptable determination for that indicator A simple random sample of 19 sample points was chosen within each supervisory area. A detailed protocol was developed to select the interview starting point. The LQAS protocol relies on interview teams to then randomize in the field, thus training and field supervision are key to the successful implementation of LQAS. At the household identified as the starting point, interviewers determined the number of sub-groups able to be completed at the household. The interviewer then proceeded to the household directly to the left (or to the nearest household) to continue completing sub-sections of the questionnaire. This approach is called “parallel sampling”, and is unique to LQAS. This method ensures that each indicator has the same number in the denominator regardless of household composition found at the identified starting point.
  • In Liberia, lots were developed according to the source of funding and implementing partner; for example, all facility catchment areas supported by USAID/IRC in Nimba County were considered to be within the same lot. Explain collaborative process of determining lots – consulting with USAID, RBHS, Africare, MOHSW Explain that facilities were used as base for determining catchments, but that it was a community-based data collection methodology Highlight that due to size, Africare was split to two lots…..reminding audience of the need for 5 lot minimum -- Africare assisted in determined how to split their catchment facilities/communities into three lots
  • Highlight how MOHSW has facilities here, as well as Irish Aid. Also, point out that USAID supports two different implementing partners in this county and LQAS provides opportunity to compare performance. Would not be possible with county level HMIS data that gets rolled into one figure.
  • Discuss how USAID/Liberia had preference to support MOHSW and its staff to do ALL parts of LQAS entirely, but how Asst. Min Wesseh pushed back explaining they didn’t have sufficient staff; MOHSW decision/recommendation to use private sub-partner…..because MOHSW had buy-in, sub-partner was well received and good relationship with MOHSW Discuss where there is still significant room for improvement here. There’s another opportunity to discuss this further along in the presentation (i.e. next steps with capacity building), but can allude the fact that this year was about assessment and laying foundation Mention that even local PRIVATE sub-partner, still required capacity building to underscore challenges facing Liberia
  • 32 participants at training in LQAS field survey work - 7 MoHSW staff from national and county offices, including: M&E Officers from Bomi, Bong, Lofa, and Nimba Counties, all of whom went through training and assisted in field survey work - 3 survey research staff from Subah Belleh and Associates (SBA) - 22 survey candidates recruited by SBA, who have varying levels of survey work experience Basic statistical concepts of LQAS -+- Techniques for random selection of community -+- Techniques for random selection of interviewees -+- Conducting parallel sampling of interviewees -+- Survey instruments, protocols, data recording -+- Oversight of survey teams by designated team leader After the training, 16 Survey Interviewers were selected for the 4 County Survey Teams (4 per Team); 4 Survey Team Leaders were also selected (1 per Team)
  • Interviewers reviewed their own work Team Supervisors reviewed all questionnaires Roving MEASURE Evaluation 5 th car visits ALL teams – REAL TIME Survey Teams Before leaving a sample point, Survey Interviewers were expected to review the questionnaire forms they had filled out, checking for accuracy and completeness Quality Assurance Team Weekly visits to teams in the field for review of questionnaires allowed for prompt identification of items that needed to be corrected or redone Coordination of visits with each of 4 survey teams facilitated by ongoing cell phone contact, although telecommunications coverage was variable
  • Interviewers reviewed their own work Team Supervisors reviewed all questionnaires Roving MEASURE Evaluation 5 th car visits ALL teams – REAL TIME Survey Teams Before leaving a sample point, Survey Interviewers were expected to review the questionnaire forms they had filled out, checking for accuracy and completeness Team Leader sometimes assisted in conducting interviews, but primary duty was to serve as first ‘external’ reviewer Quality Assurance Team Weekly visits to teams in the field for review of questionnaires allowed for prompt identification of items that needed to be corrected or redone Coordination of visits with each of 4 survey teams facilitated by ongoing cell phone contact, although telecommunications coverage was variable
  • Interviewers reviewed their own work Team Supervisors reviewed all questionnaires Roving MEASURE Evaluation 5 th car visits ALL teams – REAL TIME Survey Teams Before leaving a sample point, Survey Interviewers were expected to review the questionnaire forms they had filled out, checking for accuracy and completeness Team Leader sometimes assisted in conducting interviews, but primary duty was to serve as first ‘external’ reviewer Survey Interviewers’ performance improved through Quality Assurance Team’s second ‘external’ review of questionnaires and its observation of field survey practices, conducted weekly Quality Assurance Team Weekly visits to teams in the field for review of questionnaires allowed for prompt identification of items that needed to be corrected or redone Coordination of visits with each of 4 survey teams facilitated by ongoing cell phone contact, although telecommunications coverage was variable
  • Survey Interviewers’ performance improved through Quality Assurance Team’s second ‘external’ review of questionnaires and its observation of field survey practices, conducted weekly
  • Return of completed questionnaires to Monrovia at end of each week allowed for more efficient data processing 08 days development of indicators and lots for LQAS 07 days training in LQAS for Survey Team members 03 days training in data entry and analysis 05 days photocopying and pre-labeling of questionnaires 18 days 4 simultaneous LQAS undertaken concurrently with quality assurance of survey work and data entry 02 days additional time for completing data entry 07 days double entry of questionnaire responses for additional quality assurance
  • Example with performance-based indicator where lots had varying results
  • Example where all lots met performance-based indicator
  • Example where there was no performance-based indicator, so each lot received a different target based on county averages
  • After carefully reviewing health statuses, available resources and other donor activities, the USAID/Liberia health team decided to focus the bulk of its programming in three of these six counties: Bong, Lofa, and Nimba. USAID/Liberia believes that the progress and achievements within these three counties will have the greatest effect on overall national health outcomes. With a 1/3 of Liberia’s population residing in Bong, Nimba, and Lofa, there is the greatest potential for influencing national statistics. Additionally, these are counties with potential for strong County Health Team (CHT) leadership, and they are home to existing and planned training institutions. Because of their location in relation to the development corridor, there are also important opportunities for private-public partnerships. In short, these are the three counties, which hold the potential and promise to become model counties of prevention and health care practices, laying the groundwork for replication in other counties in the future.
  • After carefully reviewing health statuses, available resources and other donor activities, the USAID/Liberia health team decided to focus the bulk of its programming in three of these six counties: Bong, Lofa, and Nimba. USAID/Liberia believes that the progress and achievements within these three counties will have the greatest effect on overall national health outcomes. With a 1/3 of Liberia’s population residing in Bong, Nimba, and Lofa, there is the greatest potential for influencing national statistics. Additionally, these are counties with potential for strong County Health Team (CHT) leadership, and they are home to existing and planned training institutions. Because of their location in relation to the development corridor, there are also important opportunities for private-public partnerships. In short, these are the three counties, which hold the potential and promise to become model counties of prevention and health care practices, laying the groundwork for replication in other counties in the future.
  • Supply Chain Systems, add ’t details: 1. Lack of government leadership to mobilize and coordination donor funds to implement Supply Chain Master Plan…. is primarily due to lack of Government Ownership 2. Chronic stock outs due to lack of coordination/integration, particularly in areas of forecasting, procurement, distribution, and information systems Health Financing, add ’t details: Financing the health system in Liberia continues to be a challenge, and donor support finances the bulk of all primary and secondary service delivery. Approximately 6.7% of the overall GOL budget is allocated to MOHSW; however, the overall health sector expenses far exceed this $25 million – which covers roughly 25% of the overall health costs. The donor community covers approximately 40%, and private out-of-pocket expenditures account for almost 35% of total health expenditures.
  • At a stakeholder meeting in January, it was decided that the LQAS-based outcome monitoring survey would cover five program areas: water, sanitation and hygiene -+- immunization and child health -+- nutrition -+- malaria -+- maternal health -+- family planning Criteria were used to select indicators, such as programmatic relevance and viability of inclusion, given the structure of LQAS As a result, about 30 indicators were selected, and the following sub-groups were included in the study: women of reproductive age (15-49 years) and children aged 0-59 months -+- children aged 0-5 months -+- children aged 0-23 months -+- children aged 12-23 months -+- children aged 6-23 months -+- children aged 0-59 months that had in the previous 2 weeks: a diarrheal episode -+- a febrile episode -+- cough and difficulty breathing
  • Transcript

    • 1. LQAS In Liberia: Piloting a Performance Monitoring System in a Post-Conflict Environment Anna McCrerey USAID/Liberia Health Officer Stephanie Watson-Grant MEASURE Evaluation Country Portfolio Manager 13 July 2011
    • 2.
      • Liberia is recovering from economic collapse and armed conflict and is confronting challenges of high unemployment, human resource depletion, regional instability, and weak institutions.
      • Last 7 years, focus on transition from relief  development
        • Next 5 years, focus on consolidating these gains and building solid foundations to prevent backslide
      • GDP Estimates (billions); foreign aid is 771% of current GOL spending – among highest in Africa
      • 65-80% of population lives on less than $1.25 per day
      • Ranked 162 nd out of 169 by UNDP ’s Human Development Index, 2010
      • Very weak infrastructure
        • Only 650km of roads paved; majority of unpaved are impassable during rainy season
    • 3.
      • Health Status, 2007
      • Impact Indicator: Maternal Mortality Ratio (MMR), 994/100,000
        • Contributing Outcome Indicator : Skilled Birth Attendant at Birth (SBA), 32% in rural areas
      • Impact Indicator : Total Fertility Rate (TFR), 5.9
        • Contributing Outcome Indicator : Modern Contraception Prevalence Rate (MCPR), 7% in rural areas
      • Impact Indicator: Under 5 Mortality Rate (U5M), 111/1,000
        • Contributing Outcome Indicator: Children Fully Immunized, 52% *2009
        • Contributing Outcome Indicator: Children Treated with Antibiotics for ARI, 7% in rural areas
      • Impact Indicator: Under 5 Stunting Rate, 42% *2010
        • Contributing Outcome Indicator: Infants <6 months Exclusively Breastfed, 29%
    • 4.
      • The 2007 National Health Plan was a participatory plan to reconstruct the health system with four core components:
        • Health infrastructure
        • Human resources
        • Support systems (information, drugs, etc.)
        • Basic package of health services, BPHS (set of high-impact interventions targeting leading causes of morbidity & mortality)
    • 5. Current GOL/MOHSW Monitoring and Evaluation System Starting Point MOHSW Progress Key Challenges Remaining Fragmented vertical reporting structures Initiated roll-out of an integrated system with standardized indicators and data collection tools System is sporadically utilized and unreliable Not comprehensive; irregular reporting Initiated use of PBCs and annual reviews with incentives for proper reporting and data use Data culture ’ -- data is not systematically used for decision-making Lack of M&E human resource capacity at all levels Recruitment/retention for several upper management positions with requisite experience Shortage of mid-career professionals
    • 6. Importance of Monitoring & Evaluation for U.S. Government
      • Performance Monitoring Tool for Performance-Based Contracting (PBCs)
      • Procurement Reform and Use of Host Country Systems
      • Feed the Future
      • GHI and BEST
      • CDCS
    • 7. Key Challenges to Outcome Monitoring in a Post-Conflict, Resource-Constrained Environment
      • Neither donors nor implementing partners provides full county or district coverage.
      • Severe capacity challenges exist within the MOHSW.
      • HMIS is in nascent stages and data quality is poor overall
      • Decentralization of MOHSW has not reached full implementation.
    • 8. Piloting LQAS to determine feasibility as an appropriate Health Outcome Monitoring System
    • 9. Overview of Lot Quality Assurance Sampling (LQAS)
      • Outcome measures whether lot indicators meets target or not (i.e. health outcome is ‘met’ or ‘not met’)
      • No effort made to distinguish varying levels of ‘met’ or ‘not met’
      • Lots can be aggregated and coverage is estimated for each indicator for entire catchment area (i.e. counties)
      • Adapted from the manufacturing sector to determine whether a lot meets desired quality specifications
    • 10.
    • 11.
      • Challenge Lack of county- and district-wide coverage
        • Solution Data at county and lot level used for decision-making
      Ex: Bong County Donor Implementing Partner Communities EU MDM Doe, Dolo, Duta, Gbarnga, Jimmy, Jowah, Kowai, Mano Weansue I, Namue, Pelelel, Sayeta, Sheansue, Wainsue Pool Fund Save the Children, UK Allen Ta, Barrolle Farm, Bestman Farm, Cotton Tree, Gahnmue, Gbaota, Gbolo Ta, Grain Ta, Gwainyea, James Ta, Kakalon Ta, Monkpalania, Mulbah Quelleh Village, Sanoyea, Wesley Village, Wolonto Ta, Yogbo, Zeansue IIII USAID Africare Boway I, Farvey, Garlimon, Gbalatuah, Gbono Paye, Green Hill Quarry, Maryea, Matthew Village, Naama, Nlynwoe, Nyenwolo, Pallequelleh, Palaytandai, Quoipa, War-Ta, Water Side, Yila, Yowee, Zebay USAID Africare Bokai Ta, Bondo Garbo, Bong Mines, Dagquoi, Degei, Degulah, Fofana, Gbankalenta, Handii, Holder Farm, Molley Gollah, Mother Celeostine Boose Orphanage, Nakpuah, Salala, Shuffeling-Ta, Tumutu USAID Africare Beletanda I, Binda Lawoe Ta, Borkpala, Botota, Bue, Galai, Gargar, Gbarnga, Janyea, Kpoye, Molongbarnga, Nyan, Raymond Ta II, Varney Village, Wumai
    • 12. Ex: Lofa County Donor Implementing Partner Communities USAID IRC Bahn, Beadatuo, Beeplay, Boyee, Camp One Township, Central, Gelakpoah Viilage, Glarlay New Town, Graie Township, Guerkpahnah, Lorplay, Paye, Tappita City, Tengbein, Yekepa, Yreah, Zeanpea, Zorn Tiah USAID EQUIP Banlah, Beatuo, Boakai Village, Buutuo, Devongbin, Garyeesonnoh, Gbayblin, Gblarlay, Glann Town, Kparblee, Kpowin, Lepula, Rlekporlay, Saclapea, Sarlay, Tuudin, Yarsonnoh, Yenkpalah, Zuotuo USAID EQUIP Beo-Yoolar, Bonla, Duo Boe, Duo Gorton, Ganta, Gbonnie, Kormapea Village, Kpayelepula, Lugbye, Maoh, Nyoanplay 1, Slangonplay, Tahnplay, Yekepa, Yekepa-Barpa, Youbah, Younlay, Zorlepula Irish Aid IRC Behplay, Ganta, Geanplay, Guowee, Karnplay/Larpea, Kissayplay, Loe Lay, Sanniquelle, Sehyi-Geh, Sergeant Old Town, Sopeay, Zolowee ------- MOHSW Biapa, Cocopa Camp 2, Cocopa Camp 6, Gampa, Ganta, Nyansin Old, Nyoanplay, Saclepea
    • 13.
      • Challenge Severe capacity challenges within MOHSW
        • First steps towards a solution include:
        • Work plans to incorporate MOHSW participation at each stage to build capacity:
          • Lot determination
          • Indicator selection
          • Training and data collection
          • Data entry and analysis
          • Results dissemination
        • Use of a local sub-partner to supplement staffing as an intermediate step
        • Intensive, hands-on mentoring from MEASURE Evaluation
    • 14.
      • Challenge Data Quality Concerns
        • 1 st Solution Rigorous training for data collectors, including field testing instrument and necessary adjustments
    • 15.
      • 2 nd Solution 3-pronged approach to Quality Assurance in the field
      Survey Interviewers double-checked their forms for accuracy and completeness before leaving a sample point
    • 16.
      • 2 nd Solution cont’d 3-pronged approach to Quality Assurance in the field
      Team Leaders served as ‘external’ reviewers flagging questionnaires that needed to be re-done
    • 17.
      • 2 nd Solution cont’d 3-pronged approach to Quality Assurance in the field
      Real-time roving MEASURE Evaluation QA team visited every county team once a week and provided hands-on mentoring.
    • 18.
    • 19.
      • 3 rd Solution Immediate Date Entry and Double Data Entry
    • 20. Challenge Weak implementation of decentralization policy Solution Design dissemination activities that re-enforce and support decentralization including 4 county dissemination meetings with 72 participants mostly from County Health Teams
    • 21. Sample of Results for County Level Estimates
    • 22. Example 1
    • 23.
    • 24.
    • 25.
    • 26.
    • 27. Sample of Results at Lot Level
    • 28.
    • 29.
    • 30.
    • 31.
    • 32. Looking Forward
    • 33. MOHSW finalized its new 10-year National Health Plan and Policy with the following commitments:
      • Invest heavily in engaging and building human resource capacity at the county-level, with the objective of enabling CHTs to assume their expanded role as part of the County Administration.
      • County level shall be responsible for service delivery and partner oversight, while the central level will focus on establishing policies and standards, as well as resource mobilization and allocation.
      • Support community-based services as a vital component necessary to achieving primary health care goal of maximum participation in decision-making
    • 34.
      • Nationwide investments in capacity building and TA for policy formulation, strategy development, and systems strengthening
      • Intensive investments in three target counties for facility and community-based services and CHT capacity building
      • Strategic investments in six development corridor counties based on USAID ’s comparative advantage and leveraging other donor support
      Grand Gedeh Nimba Bong River Cess Grand Bassa Margibi Bomi Gbarpolu Lofa Sinoe Maryland Grand Kru River Gee USAID/Liberia Health Focus Counties . In support of MOHSW’s new 10-year National Health Plan, USAID/Liberia will: GOL Development Corridor
    • 35.
      • LQAS Role Moving Forward:
      • Maximizing use of country systems
      • (encouraging country ownership and investing in country-led plans)
      • Sharpening M&E capacity building activities
      • (building sustainability through health systems strengthening)
      • Supporting data use for policy and program decision making
      • (improving metrics, monitoring and evaluation)
    • 36. THANK YOU!
    • 37. There exist critical barriers within the health sector of Liberia, specifically surrounding health systems issues. Gov ’t Progress to Date Key Challenges Human Resources for Health
      • 8 training institutions re-opened
      • Only 1/3 of health workforce is ‘skilled’
      • Overall challenges of recruitment/retention – especially pronounced in rural areas
      Supply Chain Systems
      • Developed Supply Chan Master Plan
      • Chronic stock-outs due to lack of coordination/integration,
      • Critical warehouse and storage gaps
      Health Management Information System
      • Initiated roll-out of an integrated system
      • System is sporadically utilized and unreliable
      • Lack of ‘data culture’ -- data is not systematically used for decision-making
      Leadership and Management
      • 1 st Ministry to qualify for USAID host country contracting
      • Lack of capacity performance monitoring
      Health Financing
      • Currently working on a health financing policy
      • Overall expenses exceed GOL budget of $25 million, which only covers 1/4 th of overall health costs
    • 38.
      • Indicator selection
        • Program areas identified by stakeholders in January 2011
          • water, sanitation and hygiene (WASH); immunization and child health; nutrition; malaria; maternal health; family planning
        • Indicators used to identify the target groups:
          • mothers of reproductive age (15-49 years)
          • children aged 0-59 months
          • children aged 0-59 months with a cough, fever or diarrhea in the previous 2 weeks
      • Questionnaire development
        • Survey questions adapted from other studies (e.g., MIS, DHS, and LQAS), harmonized with MOHSW standard indicators
        • Draft of questions reviewed by stakeholders (USAID, RBHS, MOHSW)
        • Main questionnaire and eight sub-questionnaires created
    • 39.  
    • 40.  
    • 41.  
    • 42.  
    • 43.  
    • 44.  
    • 45.  
    • 46.  
    • 47.  
    • 48.  
    • 49.  
    • 50.  
    • 51.  
    • 52.  
    • 53.  
    • 54.  
    • 55.  
    • 56.  

    ×