Botswana's Integration of Data Quality Assurance into Standard Operating Procedures: Adaption of the routine data quality assessment tool


Published on

Presented at the November 2013 APHA Annual Meeting and Exposition.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Botswana's Integration of Data Quality Assurance into Standard Operating Procedures: Adaption of the routine data quality assessment tool

  1. 1. JOHN SNOW, INC. BACKGROUND To support improved data quality throughout the health system, the Botswana Ministry of Health (MoH) collaborated with experts from MEASURE Evaluation to develop a national procedure for routine monitoring of data quality and providing specific guidance on developing action plans to address challenges using a bottom-up approach. Because information is a key building block of a health system, efforts to improve data quality directly support improvements in a country’s information system across program areas. 1 2 3 METHODOLOGY The core objectives of the collaboration between the MoH and MEASURE Evaluation were to: 1. describe the process for ensuring data quality at the service delivery, district, and national levels, 2. and provide guidelines for data quality monitoring procedures. The development of the protocols and curriculum occurred over the course of a year (2012), followed by training workshops (2012/2013). January Established scope of work & objectives February Developed Botswana-RDQA Excel tool March Drafted standard operating procedures (SOPs) April Pre-tested B-RDQA tool in the field Reviewed SOPs May Finalized B-RDQA tool Revised SOPs June Finalized SOPs July-Oct. Developed data quality curriculum November Conducted 1st data quality training workshop April Conducteddataqualitytrainingoftrainers(ToT)workshop Supported 2nd data quality workshop KEY DELIVERABLES IN THE PROCESS INCLUDED: 1. Data Quality SOP—General, high level protocol for ensuring data quality at all levels of the health system 2. Routine Data Quality Assessment SOP—Protocol for the implementation of RDQAs as a monitoring tool 3. Customized RDQA Tool for Botswana (B-RDQA Tool) 4. B-RDQA Tool User Manual—Detailed guidance on implementing and using the B-RDQA Tool 5. Data Quality Curriculum—Curriculum for use in MoH trainings on data quality ADAPTATION OF THE B-RDQA TOOL The B-RDQA Tool is an Excel tool with multiple worksheets for a user to complete to verify data at various levels of the health system and conduct a system assessment to evaluate the key functional components of the M&E system. The tool was customized for the Botswana country context, with changes to the language used to describe the various levels of the health system to reflect the Botswana data flow. One of the significant additions to the tool was the addition of the “use of data for decision making” functional area in the system assessment component of the tool. BOTSWANA’S APPROACH The data quality protocols in Botswana were developed to ensure basic measures would be taken to ensure accuracy, timeliness and completeness of health data throughout the health system. The Botswana adaptation of the RDQA is unique as it provides a national protocol for routine data quality assessment across various levels of the health system, as well as across program areas. The ideal data flow for Botswana health data is illustrated below. Botswana health data currently flow through more than 39 different information systems, including both electronic and paper- based systems that feed into various data management systems. With the creation of a national M&E unit, the MoH is working to streamline processes and move towards this ideal flow. STANDARD OPERATING PROCEDURES The data quality SOP was written as a high level document on the various dimensions and considerations of data quality, intended for senior MoH officials, other policymakers, and M&E Officers. The RDQA SOP was written as a general protocol for conducting a RDQA, including responsibilities by level, intended for any MoH or district official responsible for initiating, managing or conducting routine assessments. The B-RDQA Tool User Manual was written specifically for thosestaffimplementingtheB-RDQAtoolandconductingassessments in the field at service delivery sites. Draft SOPs and a draft user manual were reviewed and discussed with stakeholders at consultative workshops. Both Ministry and external stakeholders participated in the consultations, and documents were finalized based on the recommendations from the workshops. Final documents were printed in country for distribution by the MoH. DATA QUALITY CURRICULUM & TRAINING A complete curriculum was developed by MEASURE Evaluation to train national and district M&E officers on how to implement and use the SOPs and the B-RDQA Tool. The curriculum supports a two and a half day training with a balance of presentations and hands-on exercises that give attendees first-hand experience using the tool, interpreting outputs, and developing action plans. The 1st training of 22 M&E Officers was conducted in November 2012, followed by a ToT workshop in April 2013 with select participants from the first data quality training. The trainers then conducted the 2nd training with support from MEASURE Evaluation. Overall feedback on the trainings was very positive, indicating that the SOPs and RDQA process would be useful both at the district and national levels as a routine tool for improving data quality. KEYS TO SUCCESS Country ownership: The development and implementation of protocols for improving data quality was initiated by the MoH, who requested technical assistance to adapt global tools to the Botswana context. The country-led foundation of this process has been essential in connecting with the correct stakeholders to give input and insight. Furthermore, responsibility for the in- country expenses proved beneficial with respect to the MoH taking ownership of the process. Going forward, the MoH will conduct an annual national M&E forum where data quality issues will be addressed by M&E Officers based on findings from their RDQAs. The MoH also has identified a consultant to develop the National Health M&E Plan for the ministry, which will include RDQAs as a routine activity for districts and at national level. Champions: Also key to the entire process was having a strong champion for data quality activities at the MoH. Without a strong technical voice supporting the investment in protocols to improve data quality, it would have been challenging to find the momentum to support the development and implementation of the protocols. Decentralization: Finally, the protocols decentralize the process of planning targeted activities to improve data quality, allowing service delivery sites and district-level officials to take ownership of data quality in a systematic and structured way. Service delivery sites and districts develop their own recommendations and action items, putting the power in local hands. Evidence of the success of the RDQA process was found at a district outside the capitol. A district M&E officer implemented a log book (right) to track data quality indicators after attending the April 2013 training workshop. Quality of health data impacts a government’s ability to make strategic health-related decisions, underscoring the importance of maintaining high quality data. At the national level, data ultimately inform budget and policy decisions. In the District Health Management Teams (DHMTs) and Service Delivery Sites, data enable providers and monitoring and evaluation (M&E) officers to understand the broader health activities and priorities in their respective areas. The development of this poster was supported by funds from the USAID MEASURE Evaluation project. SERVICE DELIVERY HEALTH WORKFORCE INFORMATION MEDICAL PRODUCTS, VACCINES & TECHNOLOGIES FINANCING LEADERSHIP/GOVERNANCE IMPROVED HEALTH (LEVEL AND EQUITY) RESPONSIVENESS SOCIAL AND FINANCIAL RISK PROTECTION IMPROVED EFFICIENCY SYSTEM BUILDING BLOCKS OVERALL GOALS / OUTCOMES ACCESS COVERAGE QUALITY SAFETY 4 5 M&E structures, functions & capabilities Use of data for decision making Indicator definitions and reporting guidelines Training Data management processes Data collection and reporting forms and tools Six functional areas of an M&E system Botswana’s integration of data quality assurance into standard operating procedures: 6 RESOURCE REQUIREMENTS Time & cost: The process of developing the SOPs, customized tool, training materials, and conducting the trainings took about 16 months. The total cost to MEASURE Evaluation, primarily in staff time and travel for in-country consultative workshops and training, was $300,000, funded by the United States Agency for International Development. The MoH was responsible for funding the in-country workshops, including venue, per diems, and transportation costs. Staff: At the MoH, the newly formed Department for Health Policy, Monitoring and Evaluation (DHPME) initiated the activities with MEASURE Evaluation. The Principal Health Officer was a key champion for the process, supported by the Chief Health Officer. The MEASURE Evaluation team that worked with the MoH included three Senior M&E Advisors and two M&E Associates. Travel: A total of five trips were made to work in-country with the MoH and other stakeholders including: 1. January 2012—Planning visit to develop the scope of work 2. April 2012—B-RDQA Tool customization workshop and pilot testing 3. June 2012—Consultative workshops to finalize SOPs and user manual 4. November 2012—Training of M&E officers 5. April 2013—Training of trainers and training of M&E officers 7 8 “This [process] will really reduce work burden… very exciting, can’t wait to implement. This was one of the best trainings which will really address our district data quality problems.” – District Health Officer “A very good training that came at the right time, providing skills that are sustainable and very easy to use…Bringing out very valuable results to improving health information systems, important to system improvement and decisions making.” - Trainee MEASURE Evaluation is funded by USAID through cooperative agreement GHA-A-00-08-00003-00 and implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government. ADAPTION OF THE ROUTINE DATA QUALITY ASSESSMENT TOOL FIGURE 1: INFORMATION IN THE WHO HEALTH SYSTEM BUILDING BLOCKS Health data are collected at service delivery sites Health data are aggregated at district & national levels Assessment of data impacts policy & budgets Policy & budget decisions impact health outcomes Data collection Aggregation & Analysis Impact on health FIGURE 2: DATA & HEALTH IMPACT FIGURE 3: IDEAL BOTSWANA HEALTH DATA FLOW FIGURE 4: BOTSWANA CONCEPTUAL FRAMEWORK FOR DATA QUALITY FIGURE 6: FUNCTIONAL AREAS OF THE M&E SYSTEM FIGURE 5: TIMELINE FOR IMPLEMENTATION OF RDQA Authors: Suzanne Cloutier, Sergio Lins, Amanda Makulec, David Boone, Ernest Fetogang, Rosinah T. Dialwa, and Segametsi Duge With growing interest and investment in health system strengthening measures, the Botswana adaptation of global data quality tools operationalizes a system for health information system improvements that could be adopted by other countries facing data quality challenges. Having conducted this adaptation in Botswana, the customization approach has been tested and streamlined, and select deliverables (e.g. data quality training curriculum) could be adapted to other country contexts. CONCLUSIONS: