Successfully reported this slideshow.
Your SlideShare is downloading. ×

Routine data use in evaluation: practical guidance

More Related Content

Slideshows for you

More from MEASURE Evaluation

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Routine data use in evaluation: practical guidance

  1. 1. Routine data use in evaluation: practical guidance Eva Silvestre, Ph.D., Tulane University, Data for Impact Webinar, September 24, 2020
  2. 2. Outline of webinar • Introduction • Overview of activity • Key points from guidance document • Examples: Ukraine – Zola Allen Kenya- Lyubov Teplitskaya • Questions
  3. 3. Routine data sources • Routine data sources are collected at regular intervals at public, private, and community-level health facilities and institutions. • There has been a lack of confidence in routine health information systems (RHIS). • USAID and other donors have made significant investments to improve these systems through a wide range of interventions.
  4. 4. • Historically, routine data have been passed over by evaluators in favor of other options, such as stand- alone surveys tailored to meet evaluation objectives. • But primary data collection can be expensive and time-intensive. • Availability of routine data, the perception of its cost- efficiency, and the complex nature of health interventions being implemented has led to its increased use in evaluations. Routine data in evaluation
  5. 5. Approach • Reviewed evaluations conducted by MEASURE Evaluation that used routine data. • Literature review to identify additional examples of studies that used routine data. • 18 evaluations selected for further investigation, based on the type of evaluation design, the source of data, and the health area being evaluated. • Focused on 13 briefs whose original authors could provide further information.
  6. 6. Understand the RHIS of the country or countries of interest • Become familiar with RHIS in general and for the country where the evaluation will take place. • It is helpful to know how data is collected, transmitted, aggregated, and used there. • Know the process for getting permission to access the data. • Look for foundational documents for countries, such as: • HIS strategic plan • Annual health statistics reports • Core health indicators reference sheets • PRISM assessment results and DQA and RDQA results
  7. 7. RHIS context • HIS across the world have evolved in unique ways. • The history and context is important, especially if the evaluation covers a long time period. • Contextual factors can affect data collection: • Disease outbreaks like Ebola or COVID-19 • Natural disasters • Labor disputes where data is withheld to extract concessions • Violent conflict
  8. 8. • During the evaluation period, have there been changes to the way the indicators of interest have been defined? • In the time period, have there been changes in the way data have been collected? • Have there been any changes to administrative alignment during this time? • Are there any expected changes to administrative alignment during the evaluation? • Have there been any contextual disruptions that may have affected service delivery, availability of required commodities (e.g., vaccines or therapeutics), or data entry? RHIS context questions
  9. 9. Understand specific data source(s) • Data sources present their own opportunities and challenges. • RHIS are set up to meet a country’s information needs and not set up for research or evaluation. • Some challenges: • Case-based versus visit-based sources • Ability to extract data for analysis • Presence (or not) of unique identifiers • Interoperability of multiple data sources or of older and newer systems
  10. 10. Data quality in RHIS Tools available Considerations • Investment in improving data quality and use • Several tools available that help assess • Dimensions of data quality
  11. 11. Data completeness • Understand reporting deadlines. • Review internal data quality reports. • Conduct data quality audits. • Options for dealing with missing values. • Approach will depend on to what extent data are missing and the likely cause. One could: • Ignore • Exclude districts or facilities • Impute
  12. 12. • Majority of the evaluations reviewed used additional data sources—including primary data collection—to answer all the evaluation questions. • Primary data: Qualitative methods (interviews and focus group discussions) • Secondary data: Demographic and Health Surveys, climatic data, service provision assessment data Other sources of data
  13. 13. • Routine data can be a valuable source of information to assess health program performance and impact on outcomes. • There is no perfect solution—each approach must be tailored to the goals of the evaluation and the nature of the data. • Evaluators may need to first assess the data and present a summary of it to donors who want to use it—so that all parties are aware of possible limitations. Conclusion
  14. 14. Strengthening Tuberculosis (TB) Control in Ukraine: Evaluation of the Impact of the TB-HIV Integration Strategy on Treatment Outcomes Zola Allen, Ph.D., Palladium, Data for Impact September 24, 2020
  15. 15. TB in Ukraine • One of 10 countries with a high burden of multi-drug resistant TB (MDR-TB) for 2016– 2020 • One of 20 countries with the highest estimated numbers of MDR-TB cases World Health Organization. Global Tuberculosis Report. 2019 eng.pdf?ua=1.
  16. 16. • To improve the delivery of TB and HIV services, with the goal of enhancing the timeliness of care and the life expectancy of patients with TB-HIV coinfection Strengthening TB Control in Ukraine (STbCU) project
  17. 17. • Examined the relationships among the strategies for integrating TB and HIV services, the use of TB-HIV services, and mortality outcomes • Employed a mixed-methods approach • A quasi-experimental quantitative evaluation design, complemented by qualitative interviews to inform the findings Impact evaluation of the STbCU project
  18. 18. 1. Completion of TB-HIV service cascade 2. Impact of service integration on the time lag between each step of service 3. Impact of service integration on all-cause mortality Questions addressed using routine data
  19. 19. • Sufficient to address the evaluation questions • Medical chart data aligned well with the evaluation period • Used abstracted data from routine data sources in all six study oblasts Usability and quality of the data
  20. 20. • Data captured in electronic registers • Electronic registers contained information by case, not by patient. • Necessary to de-duplicate data and include only the latest case in the sampling frame • Patient lists at AIDS centers included all HIV patients registered. • Necessary to exclude patients who do not receive treatment in the AIDS centers (e.g., patients who were diagnosed with HIV within the penitentiary system) Challenges in the usability of routine data for the evaluation
  21. 21. • Data availability • Missing data on several HIV disease characteristics (numbers of visits, clinical stage, and CD4 count) • At the TB facilities • At the AIDS centers (at baseline, >50% of the coinfected patients had data missing) • Missing data for several TB disease characteristics (classification, clinical form, and treatment category) • At the AIDS centers • Data on status of injecting drug users not collected by the facilities Challenges in the usability of routine data for the evaluation, cont.
  22. 22. • Data accuracy • Analysis of the time lag between each step of service • Dates of service incorrectly recorded according to the expected time sequence • Need for data cleaning • Analysis on the number of planned and received doses of treatment • These numbers did not always correspond with the duration of treatment • Low variation on the proportion of doses completed Challenges in the usability of routine data for the evaluation, cont.
  23. 23. • Inconsistency in the use of data collection tools across health facilities • HIV control card modifications in 2012 • Difficult to find out which version was used by each facility • Fields in the two forms meant different things [e.g., the code (T6) for “treatment completed" in the first version of the form was changed to “requires preventive treatment”] Challenges in the usability of routine data for the evaluation, cont.
  24. 24. • Missing data • Missing data in the medical charts and electronic registers • Developed and documented imputation rules and other decisions on how to handle missing data Challenges in the usability of routine data for the evaluation, cont.
  25. 25. • Constrained the analysis to the variables that were available from the records • Data from the records were better suited for the analysis of service cascades than for analyzing the effect of services on survival. Limitations in using routine data for evaluation
  26. 26. • Challenging and time-consuming fieldwork • It was difficult sometimes to find the necessary medical records in the archives. • TB forms were not always kept with forms containing HIV-related information. • Intensive and follow-up TB treatment were usually provided at different facilities. Limitations in using routine data for evaluation, cont.
  27. 27. • Inability to de- duplicate patients served by both types of facilities • Samples were collected and analyzed separately based on each patient’s point of service • Extra efforts to abstract handwritten information from medical charts Limitations in using routine data for evaluation, cont.
  28. 28. • Most data needed for the evaluation were available and accessible at the health facilities. • The data collection and data management challenges were expected and therefore sufficient time and financial resources were planned for the work. • Imputation rules and other decisions were developed and documented on how to handle missing and inconsistent data. • Rules and decisions applied during the baseline and end-line evaluation phases. What worked well
  29. 29. • Routine data were successfully used to address the evaluation questions. • Good planning, detailed documentation, and flexibility were important. • The development and documentation of imputation rules and other decisions on how to handle missing and inconsistent data were essential components. • The use of routine data worked better for some questions (e.g., service cascades), than for others (e.g., the effect of services on survival). Conclusion
  30. 30. Improving Maternal and Child Health Outcomes in Kenya: Impact of the Free Maternity Service Policy on Healthcare Use and Lives Saved Lyubov Teplitskaya, Palladium, Data for Impact Photo: Peter Kapuscinski / World Bank
  31. 31. 1989: Kenya introduces user fees 1990: Waiving policy for poor and children under five introduced 1991-2003: User fees re-introduced through phased approach 2004: Kenya introduces “10/20 policy” 2007: Removal of user fees for maternity care at public facilities June 2013: Kenya introduces Free Maternity Service Policy, abolishing user fees for maternal health at all public facilities The maternal mortality ratio in Kenya remains high, well above the Sustainable Development Goal (SDG) target of 70 Background: User Fees in Kenya 513 342 0 100 200 300 400 500 600 Free Maternity Service Policy Source: WHO, UNICEF, UNFPA, World Bank Group, and the United Nations Population Division. Trends in Maternal Mortality: 2000 to 2017. Geneva, World Health Organization, 2019
  32. 32. • What is the impact of the Free Maternity Service Policy on maternal healthcare utilization? • How many maternal and neonatal deaths were averted due to the Free Maternity Service Policy? Evaluation Questions
  33. 33. • Introduction of the Kenya Free Maternity Service Policy served as a natural experiment, with potential for use of quasi-experimental methods to evaluate the effect of the policy. • Interrupted time series (ITS) analysis is increasingly used to evaluate the impact of public health policies. • Monthly routine data is optimal for use in ITS and similar analyses because data is collected frequently. Rationale for Use of Routine Data
  34. 34. • Quasi-experimental statistical analysis • Used to quantify impact of an intervention • Requires sequential measures of the outcome variable before/after intervention • Frequently used in real world settings when RCTs are not possible • May be difficult to control for all time- varying confounders Interrupted Time Series (ITS) • Mathematical modeling tool • Estimates impact of coverage change on maternal and child mortality in low- and middle- income countries • Used for advocacy, evaluation, strategic planning • “If I increase coverage of intervention X, I could save Y number of lives”
  35. 35. DHIS2 outcomes: • Outpatient visits by children under age 5 • Outpatient visits by females over age 5 • Clients with more than 4 ANC visits • Normal deliveries at facilities • Postnatal care visits Description of Data Consulted Ownership: • Ministry of Health (MOH) • Faith-based organizations (FBO) • Private-for-profit (PFP) Aggregated all monthly national-level data from all 47 counties in Kenya
  36. 36. • Vetted specific coverage indicators for the analysis by assessing reporting completeness rates • Outliers • Adjustments for seasonality Assessment of Usability and Quality of Data Facility Type Reporting Completeness (%) MOH 69% PFP 71% FBO 65% Average Reporting Completeness (Jan 2011–Jul 2015)
  37. 37. • Time period • Before intervention: January 2011–May 2013 • After intervention: June 2013–August 2015 • Interrupted Time Series 𝒀 = 𝜷 𝟎 + 𝜷 𝟏 𝒕𝒊𝒎𝒆(𝒕) + 𝜷 𝟐 𝒍𝒆𝒗𝒆𝒍(𝒊) + 𝜷 𝟑 𝒕𝒓𝒆𝒏𝒅(𝒊𝒕) + 𝜺 (“itsa” command in Stata 14) • ITS utilization results used as inputs in LiST to estimate number of maternal and newborn deaths averted due to intervention • Counterfactual projection accounted for pre-policy coverage trend Data Analysis
  38. 38. • Incomplete facility reporting • Misalignment between DHIS2 intervention coverage and LiST service coverage Limitations DHIS2 Coverage Indicators LiST Inputs • Normal delivery at facility • ANC ≥ 4 visits • Outpatient visits by children under age 5 • Outpatient visits by females over age 5 • Postnatal care visits • Skilled birth attendance • Clean postnatal practices • ANC ≥4 visits • Iron supplementation • Antibiotics for treatment of dysentery • Zinc for treatment of diarrhea • Oral antibiotics for pneumonia • Artemesinin compounds (ACT) for treatment of malaria • Other extrapolation based on data available in previous years
  39. 39. • Assumed normal delivery at facility coverage = coverage of clean postnatal practices • For SBA: 𝐻𝐹𝐷𝑡 = 𝑁𝐷𝐸𝐿𝑡 𝑏𝑖𝑟𝑡ℎ 𝑡 ∗ 100% 𝑆𝐵𝐴 𝑡 = 𝐻𝐹𝐷𝑡 + 𝑓𝑡 Example of LiST input calculation using ITS coverage indicator
  40. 40. • Aggregated DHIS2 data from facilities can be used: • To assess differences in healthcare coverage following an intervention, such as the initiation of a new policy • As inputs into tools, such as LiST, to estimate impacts on health outcomes, such as mortality • Routine data such as those in DHIS2 have the advantage of time series • Recent approaches available to correct for incomplete facility reporting (Maina et al., 2017) Conclusions