This document details a study on integrating computerized clinical decision support (CDS) into healthcare workflows, emphasizing the importance of seamless integration for improving clinical decision-making and adherence to evidence-based guidelines. The study included qualitative observations and surveys assessing barriers and facilitators of CDS integration across various healthcare institutions, leading to the development of a validated workflow integration survey. Initial results indicate that enhanced CDS versions significantly improve usability, functionality, and overall workflow efficiency in clinical settings.

1. October 26 th , 2011
Brad Doebbeling, MD, MSc
VA HSR&D Center on Implementing Evidence-Based Practice
Regenstrief Institute
IU School of Medicine, IUPUI School of Informatics & Purdue University

2. Acknowledgements
Co-authors:
Mindy Flanagan, PhD (1 st author)
Nicole Arbuckle, BA
David A. Haggstrom, MD, MAS
Laura G. Militello, MA
Jason J. Saleem, PhD
Supported by:
• Agency for Healthcare Quality and Research (AHRQ)
• HSA2902006000131
• VA Consortium for Healthcare Informatics Research (CHIR) travel

3. Background
• Computerized clinical decision support (CDS) can
improve clinical decision making, support adherence to
evidence-based guidelines, and ultimately improve
quality of care
• Integration of CDS into workflow critical
• Metrics for assessing workflow integration in clinical
settings not well established

4. Institution Selection
“Benchmark institutions” for CDS (Chaudhry, Ann Intern Med,
2006)
• Regenstrief Institute, Partners Healthcare, Veterans Health
Administration, (Intermountain Healthcare)
• Early-adopter institutions that implemented internally
developed EHR systems, including CDS
• Widely implemented CDS in these institutions
• Ideal settings to study integration of CDS in workflow

5. Methods
Part 1: Qualitative field study
• Objective: Identify barriers to CDS workflow integration
• Direct observation (with opportunistic interviews)
• Key Informant Interviews
• Focus Groups
Part 2: Survey Development
• Objective: Develop survey to assess CDS integration into
workflow
• Item generation and revision
• Pilot test in simulated patient encounter

6. Part 1 Study Sites and Participants
11 outpatient clinics from:
• 2 VA Medical Centers (VAMCs)
• Clinics affiliated with Regenstrief Institute
• Partners Healthcare
2-4 observers experienced in ethnographic observation
Participants:
• Observation & opportunistic interviews:
-120 providers (physicians, NPs, PAs, nurses) and staff
(MAs, clerks)
• 11 Key informant interviews
• 2 Focus groups: 11 physicians

7. VA Computerized
Patient Record
System
VAMC 1

8. Analysis
Qualitative analysis: top-down vs. bottom-up coding
Coding template based on the sociotechnical model
• Social, technical, and external subsystems
Findings integrated across sites
Focused on themes that emerged as part of the
technical subsystem for survey development

9. Technical Subsystem Emergent Themes
Navigation
• How logically information is organized
• How easily information is located
Functionality
• Extent to which computer system has tools or
operations available to complete necessary tasks (e.g.,
order tests)
Usability
• How easy or hard it is to use information system

10. Technical Subsystem Emergent Themes
(cont.)
Paper Workarounds
• Persistence of paper-based documentation or paper-
based notes when EHR available
Workload
• Degree to which the computer system increases effort
(e.g., data entry, number of clicks) to perform
necessary actions

11. Part 2 Survey Development
Generated 19 items for initial scale for five dimensions
3 content experts reviewed items and rated for clarity
Revisions resulted in 15-item scale
Nine primary care physicians completed 15-item scale and
provided feedback
Scale revised and the Paper Workaround items removed
• Physicians did not routinely use CDS during patient encounter
Workflow Integration Survey consists of 12 items

12. Workflow Integration Survey
Navigation
• Patient information is easy to find in CPRS.
• Patient information is easily accessed with CPRS.
• With CPRS, it is difficult to search for patient information during face-
to-face encounters.*
Functionality
• CPRS has all of the functions (e.g., order entry, medication list)
needed to complete face-to-face patient encounters.
• CPRS helps you perform the tasks (e.g., order entry, progress notes,
record review) you need to during face-to-face patient encounters.
• The same information is entered into CPRS multiple times during
face-to-face patient encounters.*

13. Workflow Integration Survey (cont.)
Usability
• CPRS is challenging to use.*
• CPRS is easy to use.
• CPRS is frustrating to use.*
Workload
• Using CPRS during face-to-face patient encounters adds effort (e.g.,
typing, clicks).*
• Using CPRS during face-to-face patient encounters increases workload.*
• CPRS helps you complete face-to-face patient encounters efficiently.
*Indicates items are reverse-scored; Response set: 1=strongly disagree,
5=strongly agree

14. Simulation Study
New design features for a clinical reminder were
developed
Participants tested new design features in simulated
patient exams
Within-subject design
• Each participant completed 2 patient scenarios X 2 versions of CDS
• Counterbalanced order of CDS versions
Participants completed WIS twice (after both patient
scenarios for given CDS version complete)

15. Analysis
Scores for the Workflow Integration Survey were
compared across the two versions of CDS using
Wilcoxon signed-rank test
Within-subject comparisons were made between the
survey subscales
Statistical tests were two-tailed with a 0.05 level of
significance

16. Results
PCP Participants -- 10 physicians, 2 NPs, from 5
outpatient clinics)
Excellent internal reliability
• for CDS, α = 0.93; for enhanced CDS, α = 0.80
Enhanced CDS version rated significantly higher
(better) than current CDS for each subscale (p <
0.05)

17. Conclusions
Workflow Integration Survey assesses multiple
dimensions of workflow (specific to healthcare
settings)
Initial tests encouraging
Easily adaptable to assessing other specific CDS tools
Further validation (Workaround subscale excluded)
needed

18. Publications from Study
Haggstrom, D.A., Saleem, J.J., Militello, L.M., Arbuckle, N., Flanagan, M., and
Doebbeling, B.N. Examining the Relationship Between Clinical Decision Support
and Performance Measurement. AMIA Symposium 2009; 2009:223-227.
Saleem, J.J., Militello, L.M., Arbuckle, N., Flanagan, M., Haggstrom, D.A., and
Doebbeling, B.N. Provider Perceptions of Clinical Decision Support at Three
Benchmark Institutions. AMIA Symposium 2009; 2009:558-62.
Doebbeling, B.N., Militello, L.G., Flanagan, M., Haggstrom, D., Arbuckle, N.B.,
Kiess, C.L., Saleem, J.J. Integration of Computerized Decision Support into
Clinical Workflow: Investigating Social, Technical and Contextual Factors, JAMIA
(In revision), 2011.
Saleem, J.J., Haggstrom, D.E., Militello, L.G., Flanagan, M.E., Arbuckle, C.L.,
Doebbeling BN. Redesign of a Computerized Clinical Reminder for Colorectal
Cancer Screening: A Human-Computer Interaction Evaluation. BMC Medical
Informatics and Decision Making, (In revision), 2011.
Militello, L.G., Arbuckle, N.B., Saleem, J.J., Patterson, E, Flanagan, M.,
Haggstrom, D., Doebbeling, B.N. Sources of Variation in Clinical Workflow:
Implications for the Design of Cognitive Support, Applied Clinical Inform. (In
review), 2011.

19. Thank-you!
Mindy Flanagan, meflanag@iupui.edu
Nicole Arbuckle, nicole.arbuckle@udri.udayton.edu
Jason Saleem, jason.saleem@va.gov
Brad Doebbeling, bdoebbel@gmail.com

20. Form of Colorectal Cancer Screening CDS at
Study Sites
VA Medical Center 1
• Computerized clinical reminder for CRC screening
VA Medical Center 2
• Set of computerized clinical reminders for screening, follow-up, and surveillance
Regenstrief Institute
• Paper encounter form reminder for CRC screening
Partners Healthcare
• Electronic, template health maintenance list

21. Form of Colorectal Cancer Screening CDS at
Study Sites
VA Medical Center 1
• Computerized clinical reminder for CRC screening
VA Medical Center 2
• Set of computerized clinical reminders for screening, follow-up, and surveillance
Regenstrief Institute
• Paper encounter form reminder for CRC screening
Partners Healthcare
• Electronic, template health maintenance list

22. Regenstrief Medical Record System
Printed paper encounter form with clinical reminders
at the bottom.
Paper reminders automatically generated by the CDS
rules.
* Annual FOBT and periodic sigmoidoscopy are recommended for all persons aged 50 or over
to screen for colorectal cancer. If screening FOBT is positive, colonoscopy is recommended.
* HEMOCCULT 1)Pt refused 2) Done Today (results: ___________________)

23. Form of Colorectal Cancer Screening CDS at
Study Sites
VA Medical Center 1
• Computerized clinical reminder for CRC screening
VA Medical Center 2
• Set of computerized clinical reminders for screening, follow-up, and surveillance
Regenstrief Institute
• Paper encounter form reminder for CRC screening
Partners Healthcare
• Electronic, template health maintenance list

24. Partners – Longitudinal Medical Record