Understanding EMR Error Control Practices Among Gynecologic Physicians


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Understanding EMR Error Control Practices Among Gynecologic Physicians

  1. 1. Understanding the EMR Error Control Practices among Gynecologic PhysiciansRitu Kharea, Yuan Anb, Sandra Wolfa, Paul Nyirjesy,a Longjian Liua, Edgar Choua aDrexel University College of Medicine bDrexel University College of Information Science and Technology Philadelphia, PA, USA iConference 2013, February 13 2013 1
  2. 2. Motivation: EMR Errors EMR Errors ◦ incomplete, inaccurate, or inconsistent information entered in Electronic Medical Records (Brown & Patterson, 2001; Phillips & Gong, 2009) Occur because ◦ unusable EMR interfaces situated within demanding clinical environment ◦ clinicians inadvertently make mistakes while documenting patient visits and diagnosis information Are expensive … ◦ poor data quality ◦ unsafe quality of care ◦ physicians liable for medical malpractice (Classen, Pestotnik, Evans, Lloyd, & Burke, 1997; Fichman, Kohli, & Krishnan, 2011). 2
  3. 3. Controlling EMR Errors Develop computational error control algorithms (Redwood, Rajakumar, Hodson, & Coleman, 2011). ◦ alert physicians in real time, and minimize further medical errors An “inside-out” approach Step1: Understand the existing error control mechanisms Step II: Design the algorithms according to the observed limitations, and opportunities. Existing EMRs offer limited error control functionality ◦ Clinicians resort to MANUAL techniques to review, detect, and resolve the errors (Phillips & Gong, 2009). We take the first step toward algorithm development, and investigate the manual practices. ◦ assess abilities of physicians to detect EMR errors ◦ elucidate their strategies ◦ derive implications for algorithm design 3
  4. 4. The Error Simulation User Study Outpatient Clinics  Study Workflow ◦ clinicians document the patient 1. Fabricate gynecologic visit visit information into the EMRs in scenarios and develop the an on-the-spot “narrative” corresponding EMR patient visit manner. notes. ◦ documentation occurs under extreme time constraints 2. Purposefully introduce several ◦ conducive to a variety of data EMR errors into the notes. errors (George & Bernstein, 2009). 3. Present experienced gynecologic physicians with the flawed notes, Gynecologic Field of Medicine and ask them to ◦ Physicians responsible for ◦ Identify any data errors documenting information on ◦ Reveal error detection and  yeast infections, bacterial vaginitis, resolution strategies menstrual cycle issues, pre-natal and post-natal complaints, regular gynecologic examination, etc. 4
  5. 5. Providers find it far more efficient to enter impromptuEMR Gynecologic Visit Notes narrative notes (as opposed to structured data) during patient visits (Doğan et al., 2010).ABC, ABC, 34 yr old F, DOB: DD-MM-YYType: Amb New Pt Visit A typical gynecoloigicOwner: XYZ, XYZ Status: Final visit note is organizedReason for VisitMs ABC presents for a new patient visit into 19 SectionsChief Complaints (1) Reason for VisitYeast Infections (2) Chief ComplaintHPIAfter having the PE and DVT had to go off birth control which she was on for endometriosis. Few months later, started to have yeast infections. (3) History of PresentInitially, noticed external itching, soreness, redness. Her OB/GYN told she had yeast and put her on fluconazole to take as two dose treatment. Illness (HPI)Sometimes it would work but not reliable. Treated herself monthly in this manner. Started to have more internal symptoms, as well as little (4) Allergiesdischarge. Started nystain-triamcinolone which helped the external itching. They then tried inserts x7 days , it helped again. Now finds that herpartner is getting sore at times. Finally went on fluconazole once a week, started it a few months ago. Has not been much better on this regimen. (5) Current Medicines,Allergies (6) Active ProblemsBenadryl: CAPS (7) Past Medical HistoryLovenox: SOLNCoumadin: TABS (PMH)Latex Exam Gloves MISC (8) Past Surgical HistoryPMH: (PSH)Bipolar disorderCurrent Meds (9) Family HistoryDiflucan 150 mg oral tablet: Qty0, R0, RPT (10) Personal or SocialTerconazole 0.8% Vaginal Cream; Qty0; R0; RPT HistoryProbiotic CAPS; ; Qty0; R0; RPTAssessment (11)Gynecologic History• Bacterial vaginosis (12)Obstetric HistoryTests (13)Review of SystemsLABS ORDERED; yeast cultureOrders (14)Vital SignsFluocinolone Acetonide 0.025% External Ointment; APPLY SPARINGLY TO AFFECTED AREA(S) TWICE DAILY; Qty1; R1; Rx. (15)Physical ExaminationPlan (PE)Reviewed at length w/pt. Discussed vulvar hygiene sheet. Discussed pathophysiology of lichen simplex. Start fluocinolone ointment. Stopfluconazole as she has no evidence of VVC today. RTC – 1 month. (16)AssessmentSignature (17)TestsElectronically Signed by XYZ, MM-DD-YY (18)Plan (19)Orders 5
  6. 6. The 5 Kinds of EMR Errors Inconsistent Information ◦ Contradictory information across sections Incorrect Information ◦ Based on note scenario ◦ Based on clinical guidelines Incomplete Information ◦ Essential information omitted Missing Section ◦ An entire section omitted Miscellaneous Errors ◦ Inappropriate placement ◦ Use of un-established acronyms 6
  7. 7. User Study Design Participants  Data ◦ 11 females, 9 males ◦ 7 fabricated visit notes ◦ Gynecologic physicians with  different hypothetical Drexel College of Medicine gynecologic patients  Using Allscripts EMRs since 2008 or later  Purposefully introduced errors  Gold standard errors (total 97) prepared by 2 clinical investigators with 20 years of patient visit Number of Participants documentation 1 year  Introduced more incomplete and experience missing section errors as they are 5 2 year more frequent in real world 8 experience 3 year 5 experience 2 Total Number of Introduced Errors 4 year Inconsistent experience 6 Incorrect 48 Incomplete 25 54 Missing Section Miscellaneous 7
  8. 8. User Study Design Objectives ◦ assess participants’ ability to detect/resolve errors ◦ explicate intuitive strategies ◦ infer guidelines for algorithm design. Conducted one-to-one session with each participant. ◦ Analysis Stage  present the paper prototypes of the patient notes  participant carefully studies the note, detect any data error(s), and document/annotate them on the same sheet of paper. ◦ De-briefing Stage  the participant answers follow-up questions regarding the detected errors  what makes you conclude that certain data are erroneous?  what in your medical training allowed you to detect this error?  what measures would you take to resolve a certain error?  why do you think these errors occur? 8
  9. 9. Results: Note Analysis Stage (avg. 32.6 min) Re-grouped Errors:  Correlation (Pearsons) between ◦ Mod-liability: missing section, missing ◦ Task performance(recall) and task information, misc. errors duration ◦ Hi-liability: incorrect and incomplete ◦ Task performance and years of information experience Error Precision: 100% ◦ Not significant at 0.05 level ◦ Each detected error could be mapped to a Time Experience gold standard item spent Hi-liability 0.29 0.43 Error Recall: Recall for hi-liability (0.49) Mod-liability 0.1 0.19 statistically higher(p<=0.05) than recall for mod-liability (0.36)  Best Recall Performance 70% ◦ Participant P5 had 4 years experience and spent 53 minutes reviewing the note  Lowest Recall 17% ◦ Participant P10 had 1 year experience and spent only 19 minutes reviewing the note 9
  10. 10. Results: Debriefing Stage (avg. 13.3 min)Participants were very confident of their performances during thenote analysis stage, and were very vocal about their experiences. How do you gain the  Why do the errors occur? ability to detect errors? ◦ Because of poor ◦ Field experience of writing EMR documentation practices notes in clinical settings  17 participants believe that  4 participants physicians ◦ Training in medical school  Should write for others to be  6 participants able to read  Should ask more questions of ◦ Both (Experience + Academic the patient training)  Should write in a list format  5 participants ◦ Because of system design  5 participants  Allscripts EMR propagates all problems through previous visits creating obsolete information  Clinicians tend to write free text note because structured interface is not friendly enough. 10
  11. 11. Results: Debriefing StageWhat are the triggers for detecting mod-liability errors? Detection of abnormal history events ◦ if history of abnormal pap smear in the gynecologic history section is observed, it must always accompany more information such as the diagnosis date ◦ If history of hypertension in the family history section is specified then it should also be specified which family member suffered hypertension. General Information Recall ◦ If the patient is having an annual visit, then HIV screening must be present in at least one of the sections. ◦ If the patient is over 60 years of age, then a health monitoring plan should be created and specified in the Plan section 11
  12. 12. Results: Debriefing StageWhat are the triggers for detecting hi-liability errors? Observation of discrepant information between two sections ◦ The reason for visit should be consistent with the active problem list. ◦ The information on the same drugs should match across different sections, e.g., in one of the study notes  strength of the drug “Fluconazole” is 200mg in the Plan section, and strength of the drug “Diflucan” (market brand name for same drug) is 20mg in the Orders section  Only 5 participants detected the above error in our study Detection of abnormal results ◦ Any abnormal body mass index should be alerted in the Plan and Assessment sections. Identification of broken information links ◦ Each abnormal result from Physical Examination, should be linked to a corresponding diagnosis in the Assessment section. Each diagnosis item should have a corresponding item in the Plan section. 12
  13. 13. General Implicationsfor Error Control Algorithm Design Despite the experience and expertise, error recall performance < 50% (3 participants had >55%, 5 participants had <30%)  It is imperative to replace existing manual strategies with effective computational algorithms The participants delivered statistically better performance for hi- liability errors than mod-liability errors  Underlines the significance of learning from their expert abilities to minimize potential physician liability The computed correlations between performance and experience/time were not significant  No clear conclusion regarding learning-based algorithms The results on the provenance of abilities suggest that future algorithms should  incorporate domain knowledge from a wide range of sources  learn and infer from the contextual information in the EMR data 13
  14. 14. Guidelines to programmatically fire the Error Detection TriggersTrigger Computational Technique Knowledge SourcesGeneral Information Recall Basic if-then rules Clinical guidelines Extraction of key information such as • Centers for Disease Control and Prevention age, visit type, etc. • American Congress of Obstetricians and GynecologistsDetection of Abnormal Basic if-then rules • Davis’s Laboratory and Diagnostic TestsResults Extraction of examination results. • Agency for Healthcare Policy and Research • Archimedes 360 Medical CalculatorDetection of abnormal Advanced if-then rules Conceptual model for drugs, disease conditions, and habits.history event Extraction of abnormal events and their • UMLS RxNorm attributes • DailyMed Extraction of medications and their • MedlinePlus attributes.Observation of discrepant Comparison of problems, and Controlled vocabulary for describing problems and drugs,information between two medications across sections linkages between drug ingredients, and brand names, drug-sections Drug and Disease recognition drug interactions. • DrugBank • UMLS • FDA National Drug Code Directory • Classification of Diseases, Functioning, and Disability • RxDrugs • Davis’s Drug GuideIdentification of broken Extraction of results, diagnosis, plan, Drug indication, prescriptions, physical examination resources.links across multiple order information • SIDER 2sections Linking items from different sections, • Health Assessment Through the Life Span and discovering the missing links. • Outlines in Clinical Medicine • DailyMed • NDF-RT 14
  15. 15. Study Summary and ContributionsAs a pre-step to design EMR error control algorithms, we learn algorithm design lessons fromtheir abilities and behaviors of physicians on gynecologic visit notes.  Participants could detect only 49% of the inaccuracy and inconsistency errors, and only 36% of the omission errors from the notes. ◦ Need for more effective and efficient error control solution.  An in-depth investigation of manual strategies helped develop guidelines for algorithm design. 1. 5 data triggers that naturally prompt participants to sense a potential error in gynecologic notes: detection of abnormal examination results, recall of generic clinical guidelines, detection of abnormal history events, observation of discrepant information, identification of broken information links. 2. Participants can identify the triggers using NLP abilities, and an immense amount of intuitive domain knowledge accumulated through experience and medical school training. ◦ In addition to sophisticated NLP techniques, the algorithms should incorporate a wide range of federally established free resources for clinical guidelines, controlled vocabularies, drugs, diseases, drug indications, gynecologic best practices, etc. 3. We briefly provide the linkages among triggers, NLP techniques, and the relevant trustworthy knowledge sources. 15
  16. 16. Study Limitations Demanding schedules of our participants ◦ Could devote limited time to the study ◦ Set of derived implications is by no means complete Study had inherent bias ◦ Participants knew in advance that the note contain errors ◦ This is contrary to real-world practices Frequency of errors introduced in each note (avg. 13) is not based on empirical evidence (due to lack of related work) ◦ Some participants might have assumed the notes to contain fewer errors and terminated their analysis earlier 16
  17. 17. Conclusions and Future WorkIn the USA, medical errors kill more people than highway accidents every year. (Kohn, Corrigan, & Donaldson, 1999). “30 years from now the EMRs should make sense – otherwise defeats the purpose of EMRs” – a study participant on documentation malpractices  the first step to algorithm design by exploring an untapped knowledge resource, i.e., the physicians.  Explored their abilities to detect EMR data errors  Derived algorithm design implications from their intuitive knowledge and personal strategies.  In comparison to the manual expert strategies, the existing automated algorithms only scratch the surface of error control  We plan to design customized algorithms for gynecologic notes by building on the identified triggers ◦ To simulate the narrative information extraction  existing NLP algorithms to extract drug, disease and specific clinical information from texts (Doğan et al., 2010; Li et al., 2012; Névéol & Lu, 2010). ◦ To simulate the physicians’ knowledge in the head  utilize, integrate, and organize several available trustworthy knowledge sources hosted by the US Government. 17
  18. 18. Acknowledgements - The 2 anonymous reviewers - The 20 User Study Participants - Drexel’s Jumpstart grant for health informatics - Drexel’ Institutional Review BoardTHANK YOU 18