Managing data-quality-in-an-integrated-surveillance-system
•Download as PPTX, PDF•
1 like•502 views
This document summarizes the challenges of managing data quality in an integrated public health surveillance system and proposes solutions. Historically, databases were siloed but integration provides benefits like reduced redundancy and standardized data collection. Electronic lab reporting increases standardization but also volume and velocity of data. Defining clear data quality roles, accountability, documentation, training, and standardized processes can help address current roadblocks. Metrics, flowcharts, and trainings on concepts like roles and responsibilities are proposed to improve data integrity and quality management going forward.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (13)
Similar to Managing data-quality-in-an-integrated-surveillance-system
Similar to Managing data-quality-in-an-integrated-surveillance-system (20)
More from State of Utah, Salt Lake City
More from State of Utah, Salt Lake City (20)
Recently uploaded
Recently uploaded (20)
Managing data-quality-in-an-integrated-surveillance-system
- 1. Managing Data Quality in an Integrated Surveillance System Rachelle Boulton, MSPH DCP Informatics Program April 27, 2016
- 2. NETSS STD MISTIMS eHARSArboNet Historically Siloed Databases
- 5. Integration Benefits Challenges Streamline data collection Acceptability Reduce redundancy User variety Standardization Standardization Disease overlap Siloed federal databases
- 6. Electronic Data Collection • More standardization • Volume and velocity
- 7. Electronic Data Collection • More standardization • Volume and velocity ELR
- 8. Where Do We Start?
- 9. DCP Informatics Program, 2014 Updated 4/26/2016 Jennifer Brown Division Director Kurt Liedtke Java Programmer Susan Mottice, PhD ELR Coordinator Jon Reid Health Informatics Manager Josh Ridderhoff PHP Programmer Rachelle Boulton, MSPH Epidemiology Liaison Data Management
- 10. DCP Informatics Program, 2016 Updated 4/26/2016 Jennifer Brown Division Director Kirk Benge, MPH ELR Coordinator Rachelle Boulton, MSPH Epidemiology Liaison Data Management Theron Jeppson, MEd, CHES Health Promotion Liaison ELR, Syndromic Surveillance Onboarding Vacant Health Informatics Manager Joel Hartsell, MPH eCR Coordinator Amanda Whipple, MPH Project Coordinator Rocio Ramos Research Analyst Glenda Garcia Office Specialist II Joe Jackson, MBA DTS IT Manager JoDee Baker, MPH NEDSS Product Manager Allyn Nakashima State Epidemiologist Kurt Liedtke Java Engineer Josh Ridderhoff PHP Developer Doug McGowan PHP Developer Mike Whisenant Java Engineer
- 12. Define Data Quality • Two separate concepts ▫ Data integrity management ▫ Process management • Two separate processes ▫ Quality control ▫ Quality assurance
- 13. Next Steps • Identify quantifiable parameters • Develop protocols • Test it!
- 14. Metrics • Completeness • Timeliness • Data source • Accuracy • Validity • Precision
- 15. Flowcharts Type Process Component Process Mapping Surveillance Quality Process Management Decision Support Investigation Quality Data Integrity Classification Data Quality Data Integrity
- 16. No Interest!
- 17. Trainings 1. Speak the same language 2. Roles and responsibilities 3. Identify barriers 4. Introduce metrics and flowcharts
- 18. Roadblocks to Data Quality • Undefined data quality roles • No accountability • High staff turnover • Poor documentation and dissemination • Poor training • Limited standardization • Difficult, ambiguous process for change
- 19. Solutions to Roadblocks Problem: Undefined data quality roles UDOH epidemiologists – surveillance managers NEDSS surveillance and data quality manager
- 20. Solutions to Roadblocks Problem: No accountability NEDSS manager position
- 21. Solutions to Roadblocks Problem: Poor documentation and dissemination Knowledge management system
- 22. Solutions to Roadblocks Problem: Poor training Prioritized higher Dedicated resources
- 23. Solutions to Roadblocks Problem: Process for change Streamlined protocol
- 24. What’s Next?
Editor's Notes
- My name is Rachelle Boulton and I work in the Division of Disease Control and Prevention’s Informatics Program. One of my primary responsibilities is to lead data quality and surveillance evaluation and improvement efforts for the work that our program does. Right now that is primarily focused on electronic reporting of laboratory data associated with reportable communicable diseases. I have also been working with the Bureau of Epidemiology to develop more active and standardized data quality processes for all aspects of communicable disease surveillance. Today I’m going to share with you where we are, how we got there, and what I think our next steps should be.
- Historically, communicable disease surveillance data was held in a number of different stand-alone databases, that were managed and funded independently. These databases were developed by CDC, and primarily used to transmitted surveillance data from the state health department to CDC. Local health departments did not have access to these databases, although they were responsible for collecting the data that would population them. I’ve listed some of our biggest databases here. NETSS collected data on approximately 70 different communicable diseases. TIMS, held tuberculosis case management and investigation data, STD MIS contained information on chlamydia, gonorrhea, and syphilis, ArboNet, captured data on diseases transmitted by mosquitoes, and eHARS, managed data on persons living with HIV.
- In 2009, Utah implemented a new, home-grown, integrated surveillance system, UT-NEDSS, that would hold surveillance data previously managed by these disparate systems. In addition to consolidating all communicable disease data into one database, UT-NEDSS also expanded access to the local health departments.
- Since that time, UT-NEDSS has expanded to collect and store blood lead testing results for the Environmental Epidemiology Program and data on Healthcare-Associated Infections.
- Integration of these databases improved public health communicable disease surveillance in a number of ways. It streamlined data collection. At the state data entry was, for the most part, consolidated into a single program. Additionally, integration reduced data collection and data entry redundancy that used to occur between the LHDs and the state. Integration forced us to standardize surveillance processes across different diseases, which was a significant benefit to LHDs, where often only one or two public health nurses were investigating all cases. Finally, sharing and consolidating information related to co-infections, like HIV and tuberculosis, became easier. However, the process was not, and is not, without it’s challenges. Acceptability was, and still is, a primary challenge. A brand new system meant new training, process and workflow changes, and often less individual control. Currently, we have over 200 active users from all 13 LHDs, 7 different UDOH programs, and one tribal health system. There tends to be frequent turnover in many of the positions, both state and local, that use UT-NEDSS. Some users are in the system all day, every day; others only access it a few times a year. We have users from Logan to St. George; and users with all levels of computer proficiency. As I previously mentioned, integration forced us to standardize surveillance processes across different diseases, which was not always as enthusiastically embraced at the state as it was at the LHDs, because it required compromise. We still have many investigation processes and protocols that need to be standardized. The differences are just remnants of the programs that they were originally developed in, and they could be easily standardized. Finally, one of the biggest remaining challenges is the siloed databases at the federal level that these data feed into. Although integration at the state and local level was heavily stressed by CDC, similar integration has not occurred at a federal level, and is not as enthusiastically embraced by all CDC programs. The development and use of an integrated surveillance system was the first driver for comprehensive data quality processes.
- In 2013, Utah began receiving laboratory data related to reportable communicable diseases electronically. This forced us again to standardize our data collection and data entry processes even more. Additionally, ELR increased the volume and velocity of data that we received.
- The way we interacted with our data significantly changed. Each piece of information was no longer individually collected, molded into the perfect shape, and hand placed where it belonged. It now came into our database on a conveyer belt. The informatics program at that time realized that we needed someone with the responsibility for watching this process, identifying defects or bottlenecks, ensuring cleanliness, and maintaining standards. At the time, management thought that it would be smart to have the position not just monitor ELR data quality, but to extend those efforts to all data captured by UT-NEDSS.
- Getting the process started was really challenging. There were a lot of unanswered questions. The two biggest unanswered questions were: Where does this position belong in our organizational structure? And What exactly do the job responsibilities entail?
- After about 6 months of discussion, the first question was answered. The position was placed in the informatics program, given to me, and my first responsibility was to answer the second question – What do my job responsibilities entail? This was our organizational structure when I joined the program 2 years ago in 2014.
- This is our organizational structure now. We’ve had considerable growth in the last two years, which has been really great, and we’ve got a wonderful team. But managing that growth, and the constant shifting of responsibilities has certainly been challenging. The Informatics Program is part of the Division of Disease Control and Prevention, which is composed of the Bureaus of Epidemiology and Health Promotion. The Informatics Program is a division resource for all informatics-related projects. Historically this has been focused in the Bureau of Epidemiology, but we have been expanding projects into the Bureau of Health Promotion recently, as well. About half of our program is programmers who develop and maintain UT-NEDSS and our electronic laboratory reporting system. The rest of the program is informaticians/former epidemiologists that manage UT-NEDSS development, electronic laboratory reporting, and electronic case reporting. That is just a little background about where I am located organization-wise in relation to the programs that maintain and use UT-NEDSS.
- The first thing I wanted to do was to define data quality, as is it a bit of a nebulous term. Essentially, data quality means that the data collected for surveillance are good enough for their intended purpose. Specifically related to public health surveillance, data quality means that the data accurately reflect the trends in the diseases or conditions under public health surveillance. Therefore, data quality is relative to what the data is being used for.
- All of my research really pointed to data quality involving two separate components. The first is data integrity management – which involves qualities like completeness and accuracy of the individual data values themselves. The other component is process management – which controls how data is entered, edited, and travels through the system. So data quality involves aspects related to the inherent nature of the data itself, as well as the external forces that we put on that data. Those components are managed through two distinct processes: quality control and quality assurance. Quality control (QC) is the reactive component that identifies and remedies errors that exist in the data. Quality assurance (QA) is the proactive component that prevents errors from being introduced into the database in the first place. The primary goal of QA is to put only the best quality data into the system, while the primary goal of QC is to maintain the best quality data in the database.
- So at this point, I was ready to start experimenting. I wanted to identify some quantifiable parameters - objective measurements of data quality. Develop specific protocols to guide data quality evaluation and improvement, using quantifiable parameters, and see if it worked. We got two very patient epidemiologists to volunteer to be experimented upon.
- In order to make things quantifiable, there needed to be metrics. Metrics help you identify data quality problems, measure your improvement, and evaluate ongoing efforts. Metrics should be used, and used often. I identified six metrics that seemed to give a comprehensive assessment of data quality, and identify problem areas. That included completeness, which is really just a percentage of missing or unknown values. Completeness is an indicator of the availability of the data that the system intends to collect TIMELINESS. Data is timely when it is available when you need it. Data that isn’t timely, may affect your completeness. DATA SOURCE. Oftentimes there are different sources whereby you can collect the same information, but the quality of the data may differ by source. One source may be easily accessible or more timely, but have lower quality. Making decisions that weigh the accessibility versus the quality can be challenging. ACCURACY. An accurate surveillance system has data that exactly measures or represents the true value. A primary way to measure accuracy is by determining how well the data in the surveillance system was transcribed from the original sources. Assessing accuracy requires external validation of the data and is the most time-consuming metric to assess. VALIDITY. For certain types of data, you expect the values to fall within a certain range. Examples include height, age, or body temperature. Data is valid when it conforms to pre-determined requirements. PRECISION. Precision has less to do with the actual data values, and more to do with the questions and value sets in the database. The quality of data can be significantly effected by questions that are ambiguous, or involve some sort of interpretation on the part of the individual collecting and/or entering the data. I think of precision as forcing a square peg of data into a round hole in the database. Precision is the most subjective metric, and it’s actually my favorite. Interestingly, I found that problems with precision were the most difficult for the epidemiologists to identify. And even when I identified a problem, I really had a tough fight getting them to see the issue. From our pilot projects, we found that accuracy and precision errors had the most profound implications on data quality, and these are not the more popular metrics that are often used for evaluating data quality.
- Before we began experimenting, I had hypothesized that data integrity issues (remember that has to do with the data values themselves) would lead to the majority of data quality problems. And while data integrity issues certainly existed, I was surprised to find that most data quality problems were an artifact of poor process management. Data quality problems often manifested as data integrity issues, but the solutions to improvement, were process related. To improve process management, we worked on developing surveillance flowcharts. Flowcharts allow you to visualize a process, identify flaws or bottlenecks, and clearly map decisions and actions. I identified three different types of flowcharts that could be used based on the type of data quality problems that were seen. Process mapping – identify steps in a chronological order. Include critical decision points, identify sources of data, person responsible, termination points. This seemed to be the most important. One of the process management flowcharts that we developed was surveillance guidelines for investigating Hepatitis C cases. The process walks the investigator through data collection and prioritize cases for investigation. It ensures that investigators appropriately investigate cases that should be investigated, and don’t spend their time on cases that don’t need further follow up. Decision support – primarily outcome, or in our case, intervention based. Focuses on what data needs to be collected and how that data should be used in making decisions about actions. We used a decision support flowchart during our measles outbreak to ensure that all LHDs and investigators were determining immunity status appropriately. We actually ended up automating this process on a website. Classification – data driven. Guides the appropriate interpretation of raw data. This is particularly useful for determining case status. The flowcharts that we have developed and used for standard or outbreak surveillance have been well received. https://elr.health.utah.gov/decision/
- Despite what we had learned, and despite my enthusiasm, I found very little interest from anyone in starting to develop any new documents, or conduct any assessments. People aren’t necessarily averse to having the documents available, and they are more than happy to have me do it. But I want to build data quality capacity. I want epidemiologists to identify and resolve data quality issues themselves. After some discussions with Cristie Chesler, the Director of the Bureau of Epidemiology, I ended up scheduling some data quality trainings, open to any state user of UT-NEDSS.
- The purpose of the trainings was four fold: 1. Get everyone speaking the same language. I introduced the concepts of data integrity and process management, as well as quality control and quality assurance. 2. Define roles and responsibilities. This was tough, and although we made some headway, there was still a lot left unresolved at the time. Since then, I’ve been able to conceptualize our relationships a bit more, and I think we’re starting to identify roles and responsibilities better. 3. Identify barriers to data quality improvement. This was a great discussion, and I’ll highlight the gaps we identified next. 4. Introduce metrics and flowcharts, and actually work through some examples hands-on. I wanted participants to walk away and be able to say that they created a flowchart, and assessed data quality through metrics. We ended up have four training sessions. They occurred once a month, and were about an hour long. We ended up with participants from 6 programs throughout Division, and had an average of 20 participants at each session. We actually just had our last training on Monday. I think they were successful in at least getting some conversations started and getting people thinking about the big picture. However, before we can really start to improve our data quality, we have to address the barriers.
- The first barrier to data quality is undefined roles. All of the epidemiologists at our training said they didn’t know what they were “allowed” to change. Additionally, users don’t fully understand how they interact with each other, and where one person’s role ends and another begins. Some users thought that all data quality should be my responsibility. The second barrier is a lack of accountability. There really are no defined responsibilities for UT-NEDSS users or agencies that utilize UT-NEDSS. This is a remnant of the integration process. As we integrated systems and put more constraints on people, users needed to feel like they were not losing control of their data and processes, and we needed to not lose their participation. So there really weren’t many responsibilities outlined, aside from “you will use UT-NEDSS to capture your data”. The third barrier is the high staff turnover. Users are constantly coming and going through the system, and UT-NEDSS is a complex database. While we can’t necessarily make any changes to workforce retention, we need tools to deal with the challenges of constantly revolving users. The third barrier is poor documentation and dissemination. UT-NEDSS is a living, evolving database. New functionality is constantly being pushed out, and we are constantly generating protocols and providing updates. But this information can be presented in a meeting with 15 attendees, added to a Wiki with 50 participants, or sent out in an email. And users have no accountability to stay up to date on current changes. The fourth barrier is poor training. Our materials are lacking, and there is no standard protocol for training new users, or training existing users when functionality changes. The fifth barrier is limited standardization. Again, during the integration process, we allowed users and agencies more freedom in order to incentivize participation, but this has lead to 5 different protocols for doing the same thing in UT-NEDSS solely based on the epidemiologist that manages the data collection. The good news is, I think users are ready for change. In the past couple of years, I’ve completed a few standardization projects, and they’ve all been well-received. I’m seeing more and more suggestions, especially from the LHDs, for additional standardization of processes. Users are realizing that defined responsibilities, protocols, and standardization aren’t constraints that make their jobs more difficult, but instead make their job easier. The final barrier is a difficult and ambiguous process for change. Many epidemiologists have had frustrating experiences with trying to change or improve a process, that they feel like improved data quality just isn’t worth the effort. Epidemiologists may have to form a workgroup, which develops recommendations, and those have to be presented to several other groups for approval. And if there is any concern, you have to cycle through the process all over again. It’s not uncommon for it to take a year or more to make any process-related changes.
- Undefined roles Each UDOH epidemiologist needs to embrace their role as a surveillance manager for the diseases under their purview. This means they are responsible for assessing and correcting their surveillance processes, ensuring that those processes conform to overall processes, and regularly assessing their quality through metrics A single NEDSS surveillance and data quality manager needs to be identified. This person would serve as the data quality expert and would be a resource to the epidemiologists or LHDs. They would ensure process standardization across diseases, manage a knowledge management system, identify and prioritize automation functionalities, standardize documentation, and incorporate more structured data capture into UT-NEDSS.
- Accountability We need to have a NEDSS Manager position in every UDOH program and every LHD. This position needs to have clear responsibilities for training users, ensuring their agency’s participation in the appropriate workgroups, dissemination of protocols, and resolution of data quality errors. This will help with the challenges related to high staff turnover, as well.
- Poor documentation and dissemination A knowledge management system is needed to organize and store documentation in a single repository.
- Poor training The development of better training materials just needs to be prioritized higher and resources need to be dedicated to developing those materials.
- Process for change A clear protocol for making surveillance changes needs to be developed, and the process needs to be streamlined so that sufficient review and feedback can be given, without complicating the process too much. Additionally, there need to be requirements for epidemiologists to hold formal trainings for users on the surveillance changes, and accountability that each LHD has representation there.
- What’s next? We need to formally organize a process for data quality management across all the UDOH programs and external agencies that utilize the system. This process needs to address all of the roadblocks that we’ve identified. Data quality needs to be prioritized higher in our surveillance processes. Data quality should be designed into our system and our processes. Oftentimes data quality is first addressed after systems and processes have been developed, and at that point, it really is too late to make any substantial corrections. We need to advocate for data quality prioritization at the national level. We need to have a discussion about what data quality entails in the US Public Health System, and specifically ensure that quality is not lost when data is transferred between systems. Use flowcharts to develop algorithms that can process data WITHIN UT-NEDSS and guide investigators through the surveillance and investigation process. Automation should focus on processes or data interpretation that is routine, time-consuming, or prone to error. We need integration and standardization at the national level.