0
October 26 th , 2011Brad Doebbeling, MD, MScVA HSR&D Center on Implementing Evidence-Based PracticeRegenstrief InstituteIU...
AcknowledgementsCo-authors:Mindy Flanagan, PhD (1 st author)Nicole Arbuckle, BADavid A. Haggstrom, MD, MASLaura G. Militel...
Background• Computerized clinical decision support (CDS) can  improve clinical decision making, support adherence to  evid...
Institution Selection“Benchmark institutions” for CDS (Chaudhry, Ann Intern Med,2006)• Regenstrief Institute, Partners Hea...
MethodsPart 1: Qualitative field study• Objective: Identify barriers to CDS workflow integration• Direct observation (with...
Part 1 Study Sites and Participants11 outpatient clinics from:• 2 VA Medical Centers (VAMCs)• Clinics affiliated with Rege...
VA ComputerizedPatient RecordSystemVAMC 1
AnalysisQualitative analysis: top-down vs. bottom-up codingCoding template based on the sociotechnical model• Social, tech...
Technical Subsystem Emergent ThemesNavigation• How logically information is organized• How easily information is locatedFu...
Technical Subsystem Emergent Themes(cont.)Paper Workarounds• Persistence of paper-based documentation or paper-  based not...
Part 2 Survey DevelopmentGenerated 19 items for initial scale for five dimensions3 content experts reviewed items and rate...
Workflow Integration SurveyNavigation• Patient information is easy to find in CPRS.• Patient information is easily accesse...
Workflow Integration Survey (cont.)Usability• CPRS is challenging to use.*• CPRS is easy to use.• CPRS is frustrating to u...
Simulation StudyNew design features for a clinical reminder weredevelopedParticipants tested new design features in simula...
AnalysisScores for the Workflow Integration Survey werecompared across the two versions of CDS usingWilcoxon signed-rank t...
ResultsPCP Participants -- 10 physicians, 2 NPs, from 5outpatient clinics)Excellent internal reliability• for CDS, α = 0.9...
ConclusionsWorkflow Integration Survey assesses multipledimensions of workflow (specific to healthcaresettings)Initial tes...
Publications from StudyHaggstrom, D.A., Saleem, J.J., Militello, L.M., Arbuckle, N., Flanagan, M., andDoebbeling, B.N. Exa...
Thank-you!Mindy Flanagan, meflanag@iupui.eduNicole Arbuckle, nicole.arbuckle@udri.udayton.eduJason Saleem, jason.saleem@va...
Form of Colorectal Cancer Screening CDS atStudy SitesVA Medical Center 1• Computerized clinical reminder for CRC screening...
Form of Colorectal Cancer Screening CDS atStudy SitesVA Medical Center 1• Computerized clinical reminder for CRC screening...
Regenstrief Medical Record SystemPrinted paper encounter form with clinical remindersat the bottom.Paper reminders automat...
Form of Colorectal Cancer Screening CDS atStudy SitesVA Medical Center 1• Computerized clinical reminder for CRC screening...
Partners – Longitudinal Medical Record
Upcoming SlideShare
Loading in...5
×

Amia Pres Oct 26 2011 Final

175

Published on

Paper at AMIA Meeting 2011/10/26 on Development & Validation of a Workflow Integration Survey for HIT tools

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
175
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Participants included ten physicians, two nurse practitioners (NPs).
  •  Workflow Integration Survey subscaleCDSEnhanced CDSp-value (two-tailed)Navigation2.5 (0.9)3.8 (0.6)0.011Functionality3.1 (0.7)4.0 (0.6)0.008Ease of use3.2 (1.0)3.6 (0.9)0.049Workload2.3 (0.8)2.9 (0.6)0.028Table 2. Means (standard deviations) and p-values from Wilcoxon signed rank tests for the Workflow Integration Survey for both versions of the CDS; the 12 survey items were grouped along four subscales (navigation, functionality, ease of use, and workload).
  • Transcript of "Amia Pres Oct 26 2011 Final"

    1. 1. October 26 th , 2011Brad Doebbeling, MD, MScVA HSR&D Center on Implementing Evidence-Based PracticeRegenstrief InstituteIU School of Medicine, IUPUI School of Informatics & Purdue University
    2. 2. AcknowledgementsCo-authors:Mindy Flanagan, PhD (1 st author)Nicole Arbuckle, BADavid A. Haggstrom, MD, MASLaura G. Militello, MAJason J. Saleem, PhDSupported by:• Agency for Healthcare Quality and Research (AHRQ) • HSA2902006000131 • VA Consortium for Healthcare Informatics Research (CHIR) travel
    3. 3. Background• Computerized clinical decision support (CDS) can improve clinical decision making, support adherence to evidence-based guidelines, and ultimately improve quality of care• Integration of CDS into workflow critical• Metrics for assessing workflow integration in clinical settings not well established
    4. 4. Institution Selection“Benchmark institutions” for CDS (Chaudhry, Ann Intern Med,2006)• Regenstrief Institute, Partners Healthcare, Veterans Health Administration, (Intermountain Healthcare) • Early-adopter institutions that implemented internally developed EHR systems, including CDS • Widely implemented CDS in these institutions • Ideal settings to study integration of CDS in workflow
    5. 5. MethodsPart 1: Qualitative field study• Objective: Identify barriers to CDS workflow integration• Direct observation (with opportunistic interviews)• Key Informant Interviews• Focus GroupsPart 2: Survey Development• Objective: Develop survey to assess CDS integration into workflow• Item generation and revision• Pilot test in simulated patient encounter
    6. 6. Part 1 Study Sites and Participants11 outpatient clinics from:• 2 VA Medical Centers (VAMCs)• Clinics affiliated with Regenstrief Institute• Partners Healthcare2-4 observers experienced in ethnographic observationParticipants:• Observation & opportunistic interviews: -120 providers (physicians, NPs, PAs, nurses) and staff (MAs, clerks)• 11 Key informant interviews• 2 Focus groups: 11 physicians
    7. 7. VA ComputerizedPatient RecordSystemVAMC 1
    8. 8. AnalysisQualitative analysis: top-down vs. bottom-up codingCoding template based on the sociotechnical model• Social, technical, and external subsystemsFindings integrated across sitesFocused on themes that emerged as part of thetechnical subsystem for survey development
    9. 9. Technical Subsystem Emergent ThemesNavigation• How logically information is organized• How easily information is locatedFunctionality• Extent to which computer system has tools or operations available to complete necessary tasks (e.g., order tests)Usability• How easy or hard it is to use information system
    10. 10. Technical Subsystem Emergent Themes(cont.)Paper Workarounds• Persistence of paper-based documentation or paper- based notes when EHR availableWorkload• Degree to which the computer system increases effort (e.g., data entry, number of clicks) to perform necessary actions
    11. 11. Part 2 Survey DevelopmentGenerated 19 items for initial scale for five dimensions3 content experts reviewed items and rated for clarityRevisions resulted in 15-item scaleNine primary care physicians completed 15-item scale andprovided feedbackScale revised and the Paper Workaround items removed• Physicians did not routinely use CDS during patient encounterWorkflow Integration Survey consists of 12 items
    12. 12. Workflow Integration SurveyNavigation• Patient information is easy to find in CPRS.• Patient information is easily accessed with CPRS.• With CPRS, it is difficult to search for patient information during face- to-face encounters.*Functionality• CPRS has all of the functions (e.g., order entry, medication list) needed to complete face-to-face patient encounters.• CPRS helps you perform the tasks (e.g., order entry, progress notes, record review) you need to during face-to-face patient encounters.• The same information is entered into CPRS multiple times during face-to-face patient encounters.*
    13. 13. Workflow Integration Survey (cont.)Usability• CPRS is challenging to use.*• CPRS is easy to use.• CPRS is frustrating to use.*Workload• Using CPRS during face-to-face patient encounters adds effort (e.g., typing, clicks).*• Using CPRS during face-to-face patient encounters increases workload.*• CPRS helps you complete face-to-face patient encounters efficiently.*Indicates items are reverse-scored; Response set: 1=strongly disagree, 5=strongly agree
    14. 14. Simulation StudyNew design features for a clinical reminder weredevelopedParticipants tested new design features in simulatedpatient examsWithin-subject design• Each participant completed 2 patient scenarios X 2 versions of CDS• Counterbalanced order of CDS versionsParticipants completed WIS twice (after both patientscenarios for given CDS version complete)
    15. 15. AnalysisScores for the Workflow Integration Survey werecompared across the two versions of CDS usingWilcoxon signed-rank testWithin-subject comparisons were made between thesurvey subscalesStatistical tests were two-tailed with a 0.05 level ofsignificance
    16. 16. ResultsPCP Participants -- 10 physicians, 2 NPs, from 5outpatient clinics)Excellent internal reliability• for CDS, α = 0.93; for enhanced CDS, α = 0.80Enhanced CDS version rated significantly higher(better) than current CDS for each subscale (p <0.05)
    17. 17. ConclusionsWorkflow Integration Survey assesses multipledimensions of workflow (specific to healthcaresettings)Initial tests encouragingEasily adaptable to assessing other specific CDS toolsFurther validation (Workaround subscale excluded)needed
    18. 18. Publications from StudyHaggstrom, D.A., Saleem, J.J., Militello, L.M., Arbuckle, N., Flanagan, M., andDoebbeling, B.N. Examining the Relationship Between Clinical Decision Supportand Performance Measurement. AMIA Symposium 2009; 2009:223-227.Saleem, J.J., Militello, L.M., Arbuckle, N., Flanagan, M., Haggstrom, D.A., andDoebbeling, B.N. Provider Perceptions of Clinical Decision Support at ThreeBenchmark Institutions. AMIA Symposium 2009; 2009:558-62.Doebbeling, B.N., Militello, L.G., Flanagan, M., Haggstrom, D., Arbuckle, N.B.,Kiess, C.L., Saleem, J.J. Integration of Computerized Decision Support intoClinical Workflow: Investigating Social, Technical and Contextual Factors, JAMIA(In revision), 2011.Saleem, J.J., Haggstrom, D.E., Militello, L.G., Flanagan, M.E., Arbuckle, C.L.,Doebbeling BN. Redesign of a Computerized Clinical Reminder for ColorectalCancer Screening: A Human-Computer Interaction Evaluation. BMC MedicalInformatics and Decision Making, (In revision), 2011.Militello, L.G., Arbuckle, N.B., Saleem, J.J., Patterson, E, Flanagan, M.,Haggstrom, D., Doebbeling, B.N. Sources of Variation in Clinical Workflow:Implications for the Design of Cognitive Support, Applied Clinical Inform. (Inreview), 2011.
    19. 19. Thank-you!Mindy Flanagan, meflanag@iupui.eduNicole Arbuckle, nicole.arbuckle@udri.udayton.eduJason Saleem, jason.saleem@va.govBrad Doebbeling, bdoebbel@gmail.com
    20. 20. Form of Colorectal Cancer Screening CDS atStudy SitesVA Medical Center 1• Computerized clinical reminder for CRC screeningVA Medical Center 2• Set of computerized clinical reminders for screening, follow-up, and surveillanceRegenstrief Institute• Paper encounter form reminder for CRC screeningPartners Healthcare• Electronic, template health maintenance list
    21. 21. Form of Colorectal Cancer Screening CDS atStudy SitesVA Medical Center 1• Computerized clinical reminder for CRC screeningVA Medical Center 2• Set of computerized clinical reminders for screening, follow-up, and surveillanceRegenstrief Institute• Paper encounter form reminder for CRC screeningPartners Healthcare• Electronic, template health maintenance list
    22. 22. Regenstrief Medical Record SystemPrinted paper encounter form with clinical remindersat the bottom.Paper reminders automatically generated by the CDSrules.* Annual FOBT and periodic sigmoidoscopy are recommended for all persons aged 50 or overto screen for colorectal cancer. If screening FOBT is positive, colonoscopy is recommended.* HEMOCCULT 1)Pt refused 2) Done Today (results: ___________________)
    23. 23. Form of Colorectal Cancer Screening CDS atStudy SitesVA Medical Center 1• Computerized clinical reminder for CRC screeningVA Medical Center 2• Set of computerized clinical reminders for screening, follow-up, and surveillanceRegenstrief Institute• Paper encounter form reminder for CRC screeningPartners Healthcare• Electronic, template health maintenance list
    24. 24. Partners – Longitudinal Medical Record
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×