This document summarizes research on the effects of computerized physician order entry (CPOE) systems on patient outcomes. While CPOE systems aim to reduce errors, some studies have found mixed or even negative results. Two potential issues are automation bias, where doctors over-rely on the system, and reduced situational awareness. The proposed study will examine whether CPOE implementation led to changes in adverse drug events, medication errors, and malpractice suits using medical records. It will also test whether automation bias and situational awareness, as measured in a simulation, affect performance when using CPOE. The study hypothesizes CPOE increased some errors due to new cognitive biases and that experience level may interact with system

1. Running head: AUTOMATION BIAS AND SITUATIONAL AWARENESS IN PHYSICIAN
ORDER ENTRIES AFFECT PATIENT OUTCOME: IS THE PEN MIGHTIER THAN THE
KEYBOARD?
Automation Bias and Situational Awareness in Physician Order Entries Affect Patient Outcome: Is
the Pen Mightier than the Keyboard?
George Tsourdinis1
& Christopher Straus, MD2
1Biological Sciences Division, The University of Chicago, 5801 S Ellis Ave, Chicago, IL 60637
2Department of Radiology, The University of Chicago Medicine, 5841 S Maryland Ave., Chicago, IL 60637

2. 2
Introduction
History and Research on CPOE/CDSS Systems
Aside from the stethoscope and white coat, the pen and prescription (℞) pad are the next
most recognizable tools in the doctor’s repertoire of weapons against human pathology. A blank
slate, the prescription pad possessed the power to order any prescription at the physician’s will. With
the advent of laboratory chemistry, blood workups, medical imaging, and the systematization of
hospitals as we know them today in the United States, the medical requisition sheet accompanied the
prescription pad in the doctor’s arsenal. Better known as a physician order entry (POE), the
requisition form allowed the physician to order any laboratory test, request any radiogram/scan,
make any referral, and order any procedures he or she thought necessary for the patient’s
betterment. As computer technology continued to advance into the 21st
century, hospital systems
and software vendors seized the opportunity to wipe the slate clean and begin anew with the
computerized physician order entry (CPOE) system. CPOE is defined by the direct entry, and thus
authorization, of medical orders into a computer software by a physician9
. The motivation behind
CPOE was not just to make the famously inscrutable handwriting of a physician legible, but mainly
to introduce a system that would reduce the large amount of medical-related errors that occur daily
and to increase efficiency and workflow in the medical environment. CPOE’s functionality arises not
in itself, but from an added component in the software called a clinical decision support system
(CDSS). A CDSS provides live feedback in the form of prompt windows to the physician about
potential errors as the physician enters his or her order. Errors can occur from incompatible drug
interactions, unnoticed preexisting allergies to an ordered medication, and aberrant weight- and
kidney- based medication dosing1
. Pre-programmed “order sets” are installed into the CDSS and
CPOE systems and contain the algorithms that can make these predictions of error based on prior

3. 3
extensive research demonstrating contradictory multi-drug interactions, proper dosages, and any
unfavorable patient interactions to medications. Combined together, CPOE and CDSS have been
touted to set new standards of quality patient care, reduce costs, and increase workflow efficiency.
Naturally, to further achieve these goals of reduced error and increased quality of care, hospital
administrators and pharmaceutical companies have made great strides in lobbying for the
implementation of CPOE across all hospitals in the United States (US). Moreover, a consortium of
over 150 influential public and private healthcare benefit organizations called “the Leapfrog Group”
has incentivized the implementation and continued use of CPOE and CDSS systems by including
the presence of CPOE/CDSS as one of their criteria in granting hospitals A-F report card ratings
based on numerous safety factors2
.
Prima facie, computer-based health systems technologies are appealing, but initial reports
advocating for CPOE’s benefits have not accounted for other aspects of the CPOE system, and
positive initial results should be interpreted cautiously. One such aspect is that as of 2009, less than
5% of hospitals in the US have fully implemented CPOE. While implementation numbers have
certainly increased since 2009 to now, the Leapfrog Group predicts that only in twenty years’ time
will CPOE achieve maximum outreach and implementation within urban hospitals. Furthermore,
utilization is still low from a national perspective, with less than 50% of physicians entering at least
80% of their orders electronically1
. Therefore, initial reports may not have been capturing the
complete sample size of potential CPOE utilization, thus inadvertently excluding negative events as
a result of CPOE.
Studies investigating the effects of CPOE/CDSS have spawned out of an interest to evaluate
its efficacy, but many have reported mixed results. One way to evaluate CPOE efficacy is to measure
a broad category known as “patient outcome.” The spectrum of variables entailing patient outcome
lies anywhere from biologically adverse events to a psychological state pre-/post-treatment and even

4. 4
whether malpractice lawsuits have been filed as a result of poor medical treatment. There are a few
robust ways to measure patient outcome:
• Adverse drug events (ADEs) are defined as errors in drug use (or lack of
administering an intended medicine) that result in harm to the patient14
.
• A medication prescription error (MPE) is defined to be any error in the
prescribing of a medication regardless of outcome14
.
• Wrong-patient errors are detected by the “retract and reorder” method, which
determines orders place for a patient that were then rapidly discontinued by the same
physician. Next, the analysis searches for whether an identical order was placed for a
different patient by the same physician5
.
• Intercepted medication errors include errors that had the potential to cause great
harm to the patient but did not actually reach the patient due to an interception by
medical staff. Non-intercepted errors can result in preventable ADEs associated with
MPEs9
.
• A malpractice suit is a lawsuit issued forth by the patient and/or his/her family
members against the doctor for alleged mistreatment of the patient during his/her
time in the doctor’s care. The severity of malpractice suits are usually measured in
monetary losses of the defendant (i.e. the physician), imprisonment time (if
applicable), and number of total malpractice suits per physician career – regardless of
whether a settlement was pursued.
To add to the complexity, not all negative patient outcomes are even reported†
. One study found
that direct observation of physicians throughout the day yielded an objective measurement of a
17.9% error rate, while the logged error rate in the medical records system by the same physicians was
†
Further, each outcome above can be classified as preventable, non-preventable, or potential (“potential” meaning likely to have occurred).

5. 5
0.9%. Dramatic discrepancies in reported errors hide vital information that would otherwise be
helpful to improve future patient outcomes.
To highlight the benefits of CPOE, Shamliyan et al. conducted a meta-analysis of twelve
studies and discovered that all studies reportedly had a reduction in MPEs with CPOE
implementation (66% reduction in adult MPE’s). Furthermore, they found that CPOE
implementation mediated the avoidance of 775 MPEs per 1,000 orders in one study9
. Van Rosse and
colleagues came across similar findings, with uniform reporting of MPE reduction with CPOE
implementation across thirteen studies. However, their same meta-analysis found an elevated risk of
mortality with CPOE14
. Another analysis by the Shamliyan group analysis also found that even after
the implementation of CPOE, the rate of wrong-drug prescriptions did not decrease for some time9
.
Another study had uncovered a three-fold increase in the likelihood of wrong-patient errors among
emergency department physicians using CPOE, compared to those ordering through handwritten
means5
. Additionally, CPOE’s counterpart, CDSS, has been shown to introduce error, too, despite
its complex algorithms for suggesting useful advice to the ordering doctor. One cited study reported
a 26% increase in likelihood of incorrect decision being made by a physician when the CDSS system
suggested an erroneous course of treatment;3
still, an experiment conducted by Goddard et al. found
that physician decision accuracy was improved in 13.1% of cases when CDSS was employed3
. These
data are just a small sample of the heterogeneity surrounding research on CPOE/CDSS.
Mediators of Error in CPOE/CDSS Systems: Cognitive Biases
Apparently, there is still no consensus on CPOE/CDSS systems as they pertain to
benefitting the medical system. Indeed, it is expected and observed that CPOE systems can be more
efficient than handwritten requisition forms and improve communication between physicians,
nurses, and pharmacists14
, but the presence of benefits does not preclude the potential for CPOE to

6. 6
introduce new errors that were not anticipated before implementation. With increasing demands on
physicians in the face of a primary care physician shortage and increasing patient influxes due to the
Affordable Care Act, doctors may be more hard-pressed to see more patients within shorter
amounts of time. External forces like the ACA and the constantly evolving landscape of medicine
could be mediating the effects of a likewise-recently introduced CPOE/CDSS system. The
expediency with which some physicians are forced to carry out their tasks, and the co-occurrence of
newly implemented CPOE/CDSS systems, could be giving rise to unwanted cognitive biases. The
human mind is especially subject to cognitive biases under new environmental stressors, and while
biases affect nearly every aspect of social interaction, they especially play a large - and sometimes
detrimental role - in communication between physicians and supporting medical staff. Out of the
plethora of biases that barrage the mind daily, automation bias stands out as the most relevant bias to
interact with CPOE/CDSS systems.
Automation Bias and Situational Awareness
Automation bias (AB) is formally known as the tendency to over-rely on automated
systems4
, like the CDSS of CPOE. Given the opportunity via automated technology, AB not only
has the potential to alter the manner in which a physician conducts their work, but also change the
types of error a physician is likely to make. In other words, AB is a very applicable bias to investigate
in the realm of CPOE effects on healthcare, since it can alleviate some forms of human error (e.g.
efficiency) but introduce other errors – as mentioned prior. AB has been heavily studied in the realm
of aviation, a field reliant on “autopilot” systems – as research is showing, perhaps, too reliant. Over-
reliance on such automated interfaces in any context produce two distinct errors: commission error
(abiding by incorrect advice from the system) and omission error (not acting because one is not
prompted to act by the system)11
. The general notion surrounding AB is that, when presented with

7. 7
an automated system that is said to be trusted with completing/aiding with a task, a human operator
will tend to forego vigilance and information seeking, instead using the automation as a replacement
for the operator’s independent cognitive faculties. Preliminary studies have shown that CDSS
recommendations during use, for example, can reverse a physician’s correct order choice to an
incorrect one 8% of the time5
. While still a relatively new idea, AB can have grave consequences for
the patient and could underlie medical errors experienced through CPOE. While numerically small,
the effect of 8% preventable incorrect answers could be fatal for the patients on the receiving end.
Because it is such a relatively new concept, AB does not have a validated, widespread
method of measurement at this time. Nonetheless, another related psychological measurement has
been well-developed and strongly defined since the 1980’s, when Dr. Mica Endsley published her
model of situational awareness. Situational awareness (SA) is stratified into hierarchical ‘levels’
specifically defined to be a person’s perception of elements within a current situation (Level I SA),
that person’s comprehension of the meaning of a situation (Level II SA), and lastly their ability to
predict how elements in the situation will change in the future (Level III SA)10
. Each step in SA level
represents a leap from concrete observations to abstract thoughts and predictions. Endsley’s model
for SA is much more understandable in its visual form, below in Figure 1.
Figure 1 – Model of situational awareness, as proposed by Endsley, M.; adapted from Singh et al.10

8. 8
The Situation Awareness Global Assessment Technique (SAGAT) is the primary method by
which one measures SA. This tool is used to assess SA across all three levels of the Endsley model.
Experimental models for utilizing SAGAT always involve a simulation, whereby an experimental
subject is instructed to proceed routinely through a specific task in their field. Once the
experimenters believe the environmental stimuli have given the subject a chance to obtain enough
information to satisfy each SA level, the experimenters will “freeze” the simulation and issue
questions to the trainee to test their perception of the situation at that precise moment in time. The
simulation resumes and the the next “freeze” occurs at a subsequent time point. A SAGAT score is
generated by the end of the simulation, which simply compares the subject’s self-reported SA with
the objective “reality” of the situation, per the lead investigator’s expert opinion6,10
. The more
accurate the subject is, the higher their SA is, and thus the closer their subjective experience is to the
objective reality of the situation. Traditionally, medical teams in various departments utilize a simple
checklist that enumerates a sequential list of actions the performer must accomplish to properly
complete the task at hand. However, the SAGAT does what the traditional checklist cannot, and
that is to capture a snapshot of the subject’s decision-making processes in time, before errors are
resolved and before outcome is determined. While the checklist does measure the subject’s
knowledge of the task, the SAGAT measures actual SA. A significant positive correlation between
these two scoring systems is defined as a measure of “overall task expertise and management”6
.
Situational awareness has been shown to possess a relationship with automation, as
highlighted in Figure 1, and can be one of the many mediators influencing SA. Furthermore, Skitka
and colleagues who are at the forefront of AB research, found significant interactions between
automated systems and SA, showing that non-medical automated systems could “short circuit” SA
and cause its decline11
. Therefore, it is highly likely that SA is inversely correlated with AB.

9. 9
General Research Design
Given our understanding of AB, SA, and CPOE/CDSS systems, we will investigate the role of
AB, as measured via SA (due to prior demonstrated success in validated SA methods, with no well-
defined methods for AB at this time), in how a physician operates a CPOE automated system within
an emergency department context at the University of Chicago Medical Center. This study will also
be restricted to the specialty of radiology, meaning that all patient medical records and scenarios
should contain some form of medical imaging that was ordered for the patient. The study will be
divided into two parts:
• Experiment I will be a randomized, retrospective archival analysis of the University of
Chicago medical records pre- and post-CPOE implementation to determine whether patient
outcome was worse or better with CPOE versus handwritten requisition forms. The
independent variables here will be 1) the condition of strictly either handwritten or CPOE
orders – as shown by the medical record – and 2) the ordering physician’s medical expertise
in terms of ‘years after graduating medical school.’ The dependent variable for this portion
of the study will be patient outcome, as determined by reported ADEs, MPEs, and severity
of malpractice suits filed.
o We expect there to be a statistically significant increase in the number of ADE’s and
severity of malpractice suits filed after CPOE implementation, from the written
condition to the CPOE condition, validating prior studies’ estimates. In addition, we
predict a non-statistically significant decrease in MPEs, as the literature has shown to
occur, from written to CPOE conditions [Hypothesis 1.1]. This effect would likely
be due to the new set of cognitive biases, including AB and decreased SA, that arose
after CPOE implementation. While this serves as supportive evidence, it is not

10. 10
definitive. These effects await to be confirmed in Experiment II, in accordance with
our hypotheses.
o We also anticipate all independent variables to steadily decrease in frequency as they
vary with increasing medical expertise during the handwritten time period. However,
we predict that there will be a positive correlation between ADE’s and severity of
malpractice suits with increasing medical experience during the CPOE time period
[Hypothesis 1.2]. This effect is expected because there has been a demonstrated
negative correlation in the literature between susceptibility to change and physician
experience level3
. One study seems to verify this, as it found older CDSS users
making more overall errors4
. This implies that older/more experienced physicians
may be more reluctant to learning a completely new way of conducting order entries
and may spend less time learning to train on the system, while younger physicians
being trained in CPOE during medical school will have relatively fewer errors – but
still more errors overall compared to handwritten conditions – since they already are
naturally habituated to the system. This habituation may also be the reason we expect
to see a decrease of dependent variables during the handwritten period, since all
physicians, regardless of experience level, have been trained throughout their
education to conduct handwritten orders. Therefore, it is expected that younger
physicians will generally have more errors in the handwritten condition only, since
they simply have less experience in medicine.
• Experiment II will comprise an experimental simulation design with hired patient-actors,
employing the use of the SAGAT method for assessing the SA of physicians across varying
expertise levels in written requisition form versus CPOE conditions. There will be 3

11. 11
simulations in order to control for inpatient symptomatology and increase generalizability of
our results; in other words, we want to eschew from only holding one simulation type (e.g.
only a pulmonary embolism case), since we cannot disprove that the effect of cognitive
biases discovered might be specific to that illness case only – we hope to see the
hypothesized effects across a diverse array of pathologies. Lastly, we will implement a
specific type of order entry system in both paper and computerized forms called the
radiology order entry (ROE) that was created and validated by Rosenthal, et al.8
. The ROE
was chosen in order to measure the accuracy of physician’s choice of radiological imaging
orders, based on an “appropriateness criteria” scale adopted from the American College of
Radiology (ACR). The independent variables in this portion of the study will be 1) the
simulated condition of being given a requisite form to fill out a handwritten ROE versus
filling the order through ROE, and 2) the expertise of the recruited physician-subject
(stratified by third/fourth-year medical student, first-year resident “intern,” and attending
physician). The dependent variables here will be mean SAGAT score and a traditional
checklist score (both as a percentage of correct responses), along with the ACR
appropriateness score (as a measure of “utility”; please see Methods for utility score
breakdown).
o We anticipate to observe a significant decrease in the mean SAGAT score across all
experience groups from the handwritten ROE to computerized ROE conditions.
[Hypothesis 2.1]. This effect is expected due to the introduction of an automatic
system, and presumably a higher AB as indicated by the lower SA (lower mean
SAGAT).
o We also expect to see a significant increase in SAGAT as medical experience
increases within the handwritten ROE condition, as the constancy of trained

12. 12
handwriting across age groups does not impose any biases on physicians, and
SAGAT will be purely dependent on medical knowledge and expertise. However, we
expect to a decrease in mean SAGAT as medical experiences increases within the
computerized ROE condition, given that older physicians experience a reluctance to
a new system of training, as mentioned prior [Hypothesis 2.2].
o Finally, we expect to see a significant increase in the ACR appropriateness score with
increasing medical experience within the handwritten ROE condition, since
appropriateness (or utility) here would depend on experience alone. However, we
hypothesize that the ACR appropriateness score will decrease with medical
experience in the computerized ROE condition, due to the mediating effects of AB
[Hypothesis 2.3]. Here, the ACR appropriateness score is analogous to ADEs or any
other medical metric of patient outcome, since lower appropriateness scores imply a
radiogram was ordered that will be of less benefit/utility to the patient based on their
illness. Errors from lack of appropriateness can translate into a missed pathology (a
false negative), whereby the type of imaging ordered does not detect a particular
bodily cue that confirms/rejects a diagnosis.
Importance
United States hospitals collectively incur over $2 billion in annual costs of ADEs alone1
, and
potentially even higher costs arise from other forms of error in the medical system. Concurrently,
the ordering and utilization of diagnostic radiology continues to rise; for instance, the time period
between 1996 to 2010 saw a 7.8%, 10%, and 3.9% increase in computed tomography (CT), magnetic
resonance imaging (MRI), and ultrasound – respectively8
. It is perplexing that the costs of care and
medical errors continue to rise even with a steady increase in radiological diagnostic equipment

13. 13
utilization, since it would be expected that early diagnosis would reduce costs and errors. More
importantly, there are patients who are victims of rising costs of care and increasing medical errors.
We strongly suspect this increase in errors can be attributed to communication gaps in medicine,
which are further underpinned by sometimes unnoticeable cognitive bias. The introduction of
CPOE into our hospital institutions may have solved workflow inefficiencies, but it also appears to
be creating new problems that result from the automated nature of CPOE-CDSS systems. However,
the heterogeneity of data in the literature prevents one from drawing a definitive conclusion about
the effects of CPOE on patient outcome, and even less research has been done to investigate the
cognitive biases that mediate this interaction. Here, we combine a statistical retrospective analysis
with an experimental approach on situational awareness to create a new paradigm of CPOE
research. The novelty of our study lies in its dual-experimental approach, its focus on radiology, and
its application of the SAGAT technique to a live patient-actor simulation, which has never been
explored before. By studying these two methods in parallel and stressing the psychological mediators
of error, our research will shed more light on possible mechanisms by which medical errors occur,
leading to potential policy changes and solutions in the long run that can ameliorate these systematic
symptoms in our hospitals. The overall goal of this research is not to abolish computerized systems
and reinstate writing as our primary form of physician ordering, but instead to find ways to preserve
the benefits of CPOE while eliminating its newly introduced negative effects. Uncovering the
mechanisms at play will allow physicians to confront said biases to improve overall patient outcome.
Methods
Experiment I
This experiment will be a retrospective archival analysis of past physician requisition forms
pre-CPOE implementation and of past CPOE orders post-CPOE implementation. The University

14. 14
of Chicago Medical Center in Hyde Park, IL will serve as our electronic medical record sample. The
University of Chicago has received an “A” rating from the Leapfrog Group for the past consecutive
5 years13
. CPOE implementation is a major factor in determining higher hospital ratings. Assuming
the University of Chicago implemented its CPOE system in 2005*
, we will draw patient records
from the two-year time span from 2000 to 2002 for our pre-CPOE implementation condition (fully
handwritten orders). We will allow two years of acclimation to CPOE implementation transition
period before drawing our next sample time period; this prevents any confounding factors involving
a lack of familiarity with CPOE, contributing to error rates rather than AB itself. Thus, once all
physicians have been acclimated to CPOE, we will draw patient records from a two-year time span
from 2008 to 2010 for our post-CPOE implementation condition (fully computerized orders).
Patient medical records, and associated physician order entries (either handwritten scanned
copies or digital computerized copies) will be acquired with respect to patient privacy laws (HIPAA)
and IRB guidelines after receiving authorization and approval for both Experiments I and II. For
each time condition (pre-/post-CPOE implementation), we will screen collected patient files. Our
screening criteria for medical record selection include:
1. Patients that have no history of “hand-offs” and have complete continuity of care. Hand-
offs are defined as a state of transition for the patient from the care of one physician to
another. This criterion prevents the attribution of any detected errors to another physician
who had the patient handed to him or her. This further isolates the potential effects of AB
from other confounding factors.
2. Patients must have had at least one physician order entry (whether handwritten or
computerized) for any type of radiological examination. This further focuses our patient
*
The date of CPOE implementation is not known or reported on the University’s website or annual hospital records. For the
purposes of this proposal, we will assume implementation occurred at the beginning of 2005.

15. 15
population (which is still large due to high rates of radiological utilization) to patients that
may have experienced adverse events as a result of inappropriate radiological exam orders
and thus missed pathologies. Assuming a missed pathology occurred, certain drugs that
were subsequently prescribed could have had negative interactions with the missed
pathology (e.g. a missed tumor that could have been detected on MRI but instead an
ultrasound was ordered).
Once the patient records have been screened, we will randomly select a nationally representative
sample size of n=1,000 for each pre/post condition from this pool of screened records. A high
sample size plus randomization ensures a high statistical power that will be able to detect any
significant differences in these populations, while a nationally representative sample supports the
generalizability of our results in the future.
Individually, each patient medical record will be examined by trained research assistants, who
are fourth-year medical students naïve to the study hypothesis (i.e. “blinded”). It is necessary to have
research assistants who are sufficiently medically-literate and familiar with medical records to
conduct the following analysis. Furthermore, blinding our research assistants will prevent the biased
mis-categorization of patients or underestimation/overestimation of the severity of their reported
medical errors if they were un-blinded to the hypothesis of the study. Assistants will record the
following into a computer data collection software like Microsoft Excel or Prism Graphpad: whether the
order was handwritten or computerized; the physician’s number of years in medical practice (post-
graduation); and the dependent variables of ADEs, MPEs, and severity of malpractice suits filed.
The data will then undergo statistical analysis to compare the effects of the handwritten-vs.-
computerized and medical experience independent variables on the aforementioned dependent
variables.

16. 16
Experiment II
This experiment will be of an experimental simulation design with the intended purpose to
further support the results of Experiment I and to further define the psychological mechanisms of
cognitive biases in the physician ordering process, namely AB as measured by level of SA. We will
adopt a similar study design to that of Hogan et al., who utilized a trauma Human Patient Simulator
mannequin with programmable vital signs and realistic symptoms to emulate various scenarios of
pulmonary trauma in the emergency room (ER). They placed teams of physicians based on different
experience levels in this simulation room, conducted a SAGAT protocol (outlined in the Introduction),
and measured SA as it relates to medical experience. Similarly, we will utilize various scenarios to
generalize our findings across multiple situations in the ER. However, we will diverge from original
study in several ways. First, we plan to hire patient-actors per the hour with our research funds to
emulate a sick patient in need of emergency care. Each actor will be briefed by an ER physician on
what symptoms they should be displaying beforehand and would be allowed to practice their roles in
front of said physician to make sure their acting is convincing enough to the participants. Patient-
actors have been employed in simulations for medical school training for years and have been
shown to enhance the experience by making the experience more realistic than an unresponsive
mannequin12
. Actors will be in a room designed to feel like an ER, with all of the materials and
medical supplies the participants would expect to find in the environment.
Participants will be physicians recruited from the University of Chicago Medical Center.
They will be informed of their eligibility to partake in the study via invitations cards placed in their
individual mail boxes, only if they are ER attendings, ER residents, or are currently on clinical
rotations in the ER as medical students (either third or fourth years). Invitations will kindly invite
them to “participate in a study exploring decision making in medicine.” As mentioned, we will aim
to recruit the following levels of medical experience: third/fourth-year medical students, first-year

17. 17
resident “interns,” and attending physicians that have just completed residency. Because we will hold
three illness scenarios, across two conditions (handwritten and computerized ROEs), with four
participants desired for each scenario to obtain a mean score, and four experience levels of
physicians, we will ideally need to recruit 72 (3 x 2 x 4 x 4) participants with relatively equal spread of
medical experiences across the sample. We intend to conduct this experiment as a between-subject
design, whereby each participant is observed in and exposed to only one “treatment” (e.g. only one
scenario for each participant of each medical experience level). Such a between-subject design
permits us to compare the effects of the treatment on the dependent variables directly. A within-
subject design might require fewer participants, indeed, but it risks participant bias, in which the
physician participant begins to understand what is being tested across multiple ER scenarios and
begins to alter their behavior to suit the study objective.
A second difference is that we will design a SAGAT procedure with a similar template to the
Hogan et al. study, but with different inquiries that are more relevant to the ER and radiology. We
will utilize three illness scenarios common in the ER that require radiological tests to be properly
diagnosed and treated. Before each scenario, the physician participant will be briefed and perform a
20-minute training session, during which an experimenter gives them an orientation of the medical
record charting/ROE computer system and written form; this is a necessary step to ensure that the
participant is familiar with the tools they are using, thereby controlling for lack of familiarity that
could affect our dependent variables. In each scenario, the actor will be dressed in a hospital gown
lying upon an ER gurney connected to wires showing vital signs (heart rate, oxygen saturation, etc.)
on a monitor displaying false, programmed values per the settings encoded by the investigator.
These false values will be symptomatic of the illness being suffered by the patient in a specific
scenario. The participant will also be presented with edited blood workup/lab results that are
appropriate for diagnosis in the specific illness scenario. Each scenario will be scripted in such a way

18. 18
that the patient progressively reveals more information and medical history to the physician upon
interview; the actor will be instructed to be visibly experiencing discomfort and show pain when the
physician presses on a specific area only during the routine physical examination in all three
scenarios. After some arbitrary interval of time (at least three minute elapsed before the first freeze;
at least one minutes elapsed between freezes), the experimenters will “freeze” the scenario by
turning the audio of the monitors off and instructing the participant to turn towards a blank wall to
eliminate any audio-visual cues. During this “freeze,” the participant will be asked questions that are
meant to assess his or her SA at specific levels from memory, alone. Once the inquiry is complete,
the monitors will be reactivated and the participant resumes their examination. During the
examination, the physician may be charting the patient’s medical history and writing his or her ROE
order, or charting and placing an ROE on a computerized system. It is possible that a freeze will
occur while he or she is writing or typing. If we were to postpone the charting or ROE submission
until after all three levels of SA had been assessed, the very fact that the experimenters made the
physician more aware of the situation by asking these questions might artificially increase his or her
ACR appropriateness score for the ROE. By allowing the events of the examination to proceed
naturally and interjecting at certain time intervals unbeknownst to the the participants, we can more
accurately capture an organic “snapshot” of the physician’s decision making processes as measured
by SA. Below, are elaborated the three scenarios designed for this experiment and questions asked
during each freeze to assess levels I, II, and III of Endsley’s SA model. Levels of SA are defined in
the Introduction section, but will include a small reminder of their definition in parentheses, below:
1. Osteomyelitis Scenario: The patient presents with heel, back, and bone pain. They
experience chills, fatigue, fever, and occasional night sweats. Their blood sugar is heavily
elevated (this illness occurs mostly in diabetics) in the lab results and blood pressure is high.

19. 19
The skin may or may not have ulcers/redness painted on with makeup to simulate a visual
symptom.
2. Pulmonary Embolism Scenario: The patient presents with deep chest pain, shortness of
breath, and a dry cough. Their heart rate is abnormal or racing. The patient expresses
concerns of lightheadedness and difficulty with breathing.
3. Aortic Aneurism Scenario: The patient complains of sharp pain in their abdomen and back.
They may present with bleeding and/or strong headaches. The patient usually does not
present severe symptoms until the aneurism ruptures. It is up to the physician to determine
the cause and proceed with a “watch-and-wait” route or order the patient to surgery
depending on the case’s severity.
SAGAT Inquiries at each SA Level Freeze (applicable across all illness scenarios):
• Level 1 (perception of basic, factual information in the environment; symptoms):
“What is the patient’s workup?” “What is their heart rate?” “What your findings on
the neurological and respiratory exams?” “How long has the patient been
experiencing these symptoms?”
• Level 2 (comprehension of the situation; differential diagnosis/-es): “What is/are the
potential cause(s) of the patient’s physiological abnormalities?”
• Level 3 (projection of patient’s future status): “What would you expect to happen to
the patient’s blood pressure in the next five minutes?” “How will the patient’s
parameters change in the next fifteen minutes?” “What would you do to exclude
alternative diagnoses [if more than one exists]?”

20. 20
Once all freezes have been conducted, the physician will conclude the examination by debriefing the
patient on their condition, wishing them well, and exiting the room. Each session will be video and
audio recorded with cameras that will be made aware to the participant in their 20-minute
orientation briefing. In the analysis phase, medical student research assistants naïve to the hypothesis
(as in Experiment I) will watch the audio-visual recordings to assess the correctness of each
physician to determine a SAGAT score. The score will be calculated by the % correct responses (=
correct responses/total responses *100) Answers to each SA level freeze inquiry will have been pre-
determined by three, expert emergency medicine physicians who will have reached a consensus on
specific values for lab results and monitoring (assess Level I), probable differential diagnosis (or
diagnoses; assess Level II), and planned courses of action based on whether certain unexpected
conditions arise in the patient (assess Level III). Of note, physicians will be granted a ±10% error
range around the actual, pre-determined answers in Level I.
In the meantime, a separate group of medical student research assistants naïve to the
hypothesis will examine the participant’s written or computerized ROE and use objective criteria
based on the ACR appropriateness scale to assess the appropriateness of the radiogram order, given
the specific illness that was examined for. The ACR score is a nine-point scale that categorizes
appropriateness levels into ranges of utility: “low utility” (score: 1-3), “moderate utility” (score: 4-6),
and “high utility” (score: 7-9).
Lastly, another separate research assistant will analyze the audio-visual recordings and
establish a percent-correctness score for each participant based on a traditional checklist. The
purpose of the traditional checklist score is to make sure that the SAGAT score is measuring similar
qualities in the participants that are relevant to SA in a medical scenario. If the SAGAT and traditional
checklist scores correlate to a significant degree, we can conclude that our study displayed proper
construct validity, since the SAGAT was not testing anything we did not want to test for. The

21. 21
traditional checklist here functions as a guideline of sorts to ensure this relevance in the SAGAT
score6
.
Statistical analyses will be conducted in the same manner as Experiment I to compare the
effects of written-vs.-computerized ROE and medical experience on SAGAT/traditional checklist
scores and ACR appropriateness score for radiograms ordered.
Limitations
One central limitation to our research design lies in the sheer difficulty to separate medical expertise
and familiarity (or usability) with computerized systems. There could uncontrollable subjective past
negative experiences with computers that cause reluctance and lower usability for the ordering
physician, irrespective of age. We attempted to control for familiarity and set a baseline usability for
a system the doctors have most likely never encountered before in Experiment II by orienting the
participants to the user systems (written and computerized forms) for charting and ROE entry before
the procedure even began. However, subjective past experiences with technology should be
acknowledged as a particularly evasive confounding factor that can impact usability and participant
affect towards the automation technology. This negative affect could artificially induce errors
without the physician’s realization, since the bias could be unconscious. Therefore, studies in CPOE
should aim to increase the exposure of psychological biases that underpin automation-induced
medical error. Making physicians more aware could be one solution to the issue of medical error.
We hypothesized that increasing age and experience in medicine will be positively correlated
with an increase in ADEs and lawsuit severity. However, it could very well be argued that older
physicians have more diagnostic and medical experience overall, which allows them to circumvent
their biases and focus on what truly matters. Thus, this would conversely result in a decrease in
ADEs and lawsuit severity with increasing expertise. While the data in the literature show the

22. 22
opposite trend for now4
, a scenario in which expertise trumps computer interface familiarity can be
imagined.
Furthermore, we did not account for trust/reliance interactions. Here, trust can be defined as
the physician’s confidence in the system or him-/herself. Trust in the computer over oneself can
result in reliance, which leads to an increased susceptibility of committing an error due to AB.
However, greater trust in oneself can allow the individual to override cognitive shortcuts like AB in
times of stress and make more independent decisions that lead to better outcomes. Physicians have
actually been shown to accept CDSS recommendations that were incorrect when they reported less
confidence in their own diagnosis4
. This effect of a physician’s confidence in their diagnostic ability
could still be variable in physicians who have been in a medical career for many years. Increased age
does not necessarily imply increased confidence in one’s abilities, always. Therefore, future studies
repeating our paradigm should choose different subjective variables and administer surveys assessing
the participant’s confidence in their scenario. Accounting for subjective factors can reveal a hidden
side of physician decision making that can also mediate whether he or she commits an unintended
medical error.
One last consideration for future studies should consider the finding that additional options
have the potential to increase the difficulty of decision making. When confronted with either
sending a patient to receive hip replacement surgery or place them on ibuprofen, most physicians
significantly opted for the ibuprofen route, rather than the surgery. However, when another group
of physicians were confronted with the same scenario, but now with an additional drug option (two
total drug options and one surgery option), most physicians significantly opted their patient for the
surgery, instead!7
Psychologists take these results to mean that when presented with additional
options, making the choice between only two similar, yet slightly different options is much harder
than choosing between two different options. Adding alternative options coaxes the mind to follow

23. 23
the path of least resistance and choose the most different option available (in this case, surgery
instead of two similar pain pills). Others attribute this to a “technological imperative,”7
a so-called
inner feeling of unease when the option to simply ‘do nothing’ seems unappealing to us, given the
vast array of interventional technology at our disposal. Thus, the imperative to intervene may be
stronger when fewer alternatives are available to us than when many alternatives are present and the
decision-making process spends too much cognitive energy. Considerations of the technological
imperative in CPOE are important, too, as CDSS recommendations providing too many alternatives
could result in omission errors, thereby preventing the patient from receiving a necessary
intervention. Overcoming these biases is difficult work. But recognizing the principle of primum non
nocere, or “first, do no harm,” is a vital first step towards confronting one’s cognitive biases to reduce
adverse patient events.

24. 24
Works Cited
1. Dixon, B.E. & Zafar, A. (2009). Inpatient Computerized Provider Order Entry (CPOE):
Findings from the AHRQ Health IT Portfolio. Agency for Healthcare Research and Quality.
AHRQ Publication No. 09-0031-EF, 1-18. www.ahrq.gov
2. Eikel, C., Delbanco, S., M. John, M. (2003). Eisenberg patient safety awards: The Leapfrog
Group for Patient Safety: rewarding higher standards.
The Joint Commission Journal on
Quality and Patient Safety, 29(12), 634–639. www.jcrinc.com
3. Goddard, K., Roudsari, A.V., & Wyatt, J.C. (2014). Automation bias: Empirical results.
assessing influencing factors. I. J. Medical Informatics, 83(5), 368–375.
http://dx.doi.org/10.1016/j.ijmedinf.2014.01.001
4. Goddard, K., Roudsari, A.V., & Wyatt, J.C. (2012). Automation bias: a systematic review of
frequency, effect mediators, and mitigators. Journal of the American Medical Informatics
Association : JAMIA, 19(1), 121–127. http://doi.org/10.1136/amiajnl-2011-000089
5. Green, R.A., Hripcsak, G., Salmasian, H., Lazar, E.J., Bostwick, S.B., Bakken, S.R., &
Vawdrey, D.K. (2015). Intercepting wrong-patient orders in a computerized provider
order entry system. Annals of Emergency Medicine, 65(6), 679-686.
doi:10.1016/j.annemergmed.2014.11.017
6. Hogan, M.P, Pace, D.E., Hapgood, J., & Boone, D.C., (2006). Use of human patient
simulation and the situation awareness global assessment technique in practical trauma
skills assessment. Journal of Trauma Injury, Infection, and Critical Care, 61(5), 1047–1052.
doi: 10.1097/01.ta.0000238687.23622.89
7. Redelmeier, D.A., & Shafir, E. (1995). Medical Decision Making in Situations That Offer

25. 25
Multiple Alternatives. The Journal of the American Medical Association: JAMA, 273(4), 302-
305. doi: 10.1001/jama.1995.03520280048038.
8. Rosenthal, D.I., Weilburg, J.B., Schultz, T., Miller, J.C., Nixon, V., Dreyer, K.J., & Thrall,
J.H. (2006). Radiology order entry with decision support: initial clinical experience.
Journal of the American College of Radiology, 3(10), 799–806. doi: 10.1016/j.jacr.2006.05.006
9. Shamliyan, T. A., Duval, S., Du, J., & Kane, R. L. (2008). Just What the Doctor Ordered.
Review of the Evidence of the Impact of Computerized Physician Order Entry System
on Medication Errors. Health Services Research, 43, 32–53. doi: 10.1111/j.1475-
6773.2007.00751.x
10. Singh, H., Petersen, L.A., & Thomas, E.J. (2006). Understanding diagnostic errors in
medicine: a lesson from aviation. BMJ Quality & Safety in Health Care, 15(3), 159–164.
doi: 10.1136/qshc.2005.016444
11. Skitka, L. J., Mosier, K. L., & Burdick, M. (2000). Accountability and automation bias.
International Journal of Human-Computer Studies, 52, 701–717. doi.10.1006/ijhc.1999.0349
12. Spencer, J. & Dales, J. (2006). Meeting the needs of simulated patients and caring for the
person behind them? Medical Education, 40(1), 3-5.
13. The University of Chicago Medicine & Biological Sciences. (2014). Clinical Effectives Report
– Fourth Edition. 1-12.
14. van Rosse, F., Maat, B., Rademaker, C.M.A., van Vught, A.J., Egberts, A.C.G., & Bollen,
C.W. (2009). The Effect of Computerized Physician Order Entry on Medication
Prescription Errors and Clinical Outcome in Pediatric and Intensive Care: A Systematic
Review. Pediatrics, 123(4), 1184-1190. doi: 10.1542/peds.2008-1494