This document discusses the assessment of communication and interpersonal skills competencies for emergency medicine residents. It defines 10 specific communication competencies that were agreed upon at a consensus conference. The conference aimed to define the communication competency for emergency medicine, review assessment methods used in other specialties, identify methods suggested by the Accreditation Council for Graduate Medical Education, and analyze the applicability of these methods to emergency medicine. Standardized patients and direct observation were identified as the best assessment methods, but other methods like checklists, skills rating forms, and multi-source feedback were also discussed. The document concludes that while no single method can fully assess communication skills, these various approaches can provide formative or summative evaluation of residents' compet
General Principles of Intellectual Property: Concepts of Intellectual Proper...
Assessment Of Communication And Interpersonal Skills Competencies
1. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1257
Assessment of Communication and Interpersonal
Skills Competencies
Cherri D. Hobgood, MD, Ralph J. Riviello, MD, Nicholas Jouriles, MD,
Glen Hamilton, MSW, MD
Abstract
Excellent communication and interpersonal (C-IP) skills
are a universal requirement for a well-rounded emer-
gency physician. This requirement for C-IP skill excel-
lence is a direct outgrowth of the expectations of our
patients and a prerequisite to working in the increasingly
complex emergency department environment. Directed
education and assessment of C-IP skills are critical com-
ponents of all emergency medicine (EM) training pro-
grams and now are a requirement of the Accreditation
Council for Graduate Medical Education (ACGME) Out-
come Project. In keeping with its mission to improve the
quality of EM education and in response to the ACGME
Outcome Project, the Council of Emergency Medicine
Residency Directors (CORD-EM) hosted a consensus
conference focusing on the application of the six core
competencies to EM. The objective of this article is to
report the results of this consensus conference as it re-
lates to the C-IP competency. There were four primary
goals: 1) define the C-IP skills competency for EM, 2)
define the assessment methods currently used in other
specialties, 3) identify the methods suggested by the
ACGME for use in C-IP skills, and 4) analyze the appli-
cability of these assessment techniques to EM. Ten spe-
cific communication competencies are defined for EM.
Assessment techniques for evaluation of these C-IP com-
petencies and a timeline for implementation are also de-
fined. Standardized patients and direct observation were
identified as the criterion standard assessment methods
of C-IP skills; however, other methods for assessment are
also discussed. Key words: communication; interper-
sonal skills; core competency; assessment. ACADEMIC
EMERGENCY MEDICINE 2002; 9:1257–1269.
Emergency department (ED) patient encounters are
unique in medicine. They are often a single en-
counter for a new/unknown patient, and the du-
ration and quality of the encounter are influenced
by time pressures. As such, the emergency physi-
cian (EP) must quickly establish rapport and trust
with the patient, gather information, assess the
cause of the emergency, and design a treatment
plan. The EP must also be able to effectively com-
municate findings and plans with the patient, the
family, and other members of the health care team.
Communication and interpersonal (C-IP) skills are
a key component of a well-rounded EP. These skills
From the Department of Emergency Medicine, UNC School of
Medicine, Chapel Hill, NC (CDH); the Department of Emer-
gency Medicine, Thomas Jefferson University, Philadelphia, PA
(RJR); the Department of Emergency Medicine, Case Western
Reserve University (MetroHealth), Cleveland, OH (NJ); and the
Department of Emergency Medicine, Wright State University,
Kettering, OH (GH).
Presented at the Council of Emergency Medicine Residency Di-
rectors (CORD) Consensus Conference on the ACGME Core
Competencies: ‘‘Getting Ahead of the Curve,’’ March 2002,
Washington, DC.
Address for correspondence and reprints: Cherri D. Hobgood,
MD, Education Director, Department of Emergency Medicine,
University of North Carolina at Chapel Hill, CB #7495, Neuro-
sciences Hospital, Chapel Hill, NC 27599–7495. Fax: 919-966-
3049; e-mail: hobgood@med.unc.edu.
are a universal requirement, for even the most com-
petent and clinically sound physician can falter if
his or her communication skills are less than ade-
quate. Communication skills are not just verbal, but
encompass the spectrum of nonverbal communi-
cations including body language and written com-
munication (Table 1).
The graduate of a contemporary emergency med-
icine (EM) residency program should have a long
list of identifiable clinical competencies, but must
also be outstanding in the areas of C-IP skills. This
requirement for excellent communication is a direct
outgrowth of the expectations of our patients and
a prerequisite to working in the increasingly com-
plex ED environment. Therefore, our educational
methodology and assessment techniques for the C-
IP competency should be suitably robust. Further,
these methods must be fundamentally fair, and pro-
vide an accurate, reproducible, and valid means of
evaluation for all examinees. Assessment methods
for C-IP skills competencies are numerous and
many have been well studied in other specialties.
The objective of this article is to report the results
of a consensus conference that had four goals: 1)
define the C-IP skills competency for EM, 2) define
the assessment methods currently used in other
specialties, 3) identify the methods suggested by
2. 1258 Hobgood et al. • ASSESSMENT OF COMMUNICATION AND INTERPERSONAL SKILLS
TABLE 1. Emergency Medicine Communication and Interpersonal Skills Competencies
1. Demonstrate the ability to respectfully, effectively, and efficiently develop a therapeutic relationship with patients and their
families
2. Demonstrate respect for diversity and cultural, ethnic, spiritual, emotional, and age-specific differences in patients and other
members of the health care team
3. Demonstrate effective listening skills and be able to elicit and provide information using verbal, nonverbal, written, and tech-
nological skills
4. Demonstrate ability to develop flexible communication strategies and be able to adjust them based on the clinical situation
5. Demonstrate effective participation in and leadership of the health care team
6. Demonstrate ability to elicit patient’s motivation for seeking health care
7. Demonstrate ability to negotiate as well as resolve conflicts
8. Demonstrate effective written communication skills with other providers and to effectively summarize for the patient upon
discharge
9. Demonstrate ability to effectively use the feedback provided by others
10. Demonstrate ability to handle situations unique to emergency medicine:
a. Intoxicated patients
b. Altered mental status patients
c. Delivering bad news (death notification, critical illness)
d. Difficulties with consultants
e. Do-not-resuscitate/end-of-life decisions
f. Patients with communication barriers (non-English-speaking, hearing-impaired)
g. High-risk refusal-of-care patients
h. Communication with out-of-hospital personnel as well as nonmedical personnel (police, media, hospital administration)
i. Acutely psychotic patients
j. Disaster medicine
the Accreditation Council for Graduate Medical Ed-
ucation (ACGME) for use in C-IP, and 4) analyze
the applicability of these assessment techniques to
EM.
BACKGROUND
To identify what currently exists, a Medline search
was performed from 1996 to the present utilizing
the following Medical Subject Headings of the Na-
tional Library of Medicine: communication, physi-
cian–patient relations, education/medical, clinical com-
petence, human, English language. In addition, for
assessment methods, any journal article that was
recommended by the ACGME in the Toolbox of As-
sessment Methods was reviewed.1
Any English-lan-
guage article, with applicability to EM or insight
into the development or implementation of an as-
sessment methodology was included in the analy-
sis and maintained in the annotated bibliography.
In addition to the intellectual dimensions of med-
icine, physicians must be able to clearly express
themselves, use language the patient can under-
stand, and listen to their patients. For a good doc-
tor–patient relationship, the physician must also
explore the patient’s experience while also involv-
ing the patient in treatment decisions. The physi-
cian should always treat the patient with respect
and empathy.2
It has been documented in the literature that ex-
cellent interviewing skills can strengthen the bond
between the patient and physician. Effective C-IP
skills increase patient satisfaction and are associ-
ated with improvement in patient compliance,
health status, and symptom resolution with a pos-
itive influence on patient recall, understanding,
better patient adherence, and fewer malpractice
suits.3–5
The physician’s approach should be pa-
tient-centered, where the focus is on the disease as
well as the patient’s experience of the illness. The
patient-centered interview technique has been
shown to increase patient satisfaction. Some tips for
conducting a successful patient-centered interview
include following the patient’s lead, asking for the
patient’s opinion and suggestions, and responding
to the patient’s suggestions.
Patients’ satisfaction is directly related to the
amount of information provided to them. Patients
want information and are more satisfied when they
receive it. McGuire et al. found that 63–90% of phy-
sicians made no attempt to discover the patient’s
views and expectations, encourage questions, check
understanding, categorize information, or negotiate
a treatment plan.6
Another study found that nine
out of ten patients did not receive good explana-
tions of proposed treatments.5
Interestingly, when
physicians are asked, they tend to underestimate
the amount of information desired by the patient7
while overestimating the time spent explaining and
planning in the interview by up to 900%.5
There are several strategies for developing effec-
tive patient-centered interview techniques. The Cal-
gary-Cambridge Observation Guide divides the in-
terview session into five tasks: 1) initiating the
session (initial rapport and reason for visit); 2) gath-
ering information (understanding the patient’s per-
spective); 3) building the relationship; 4) explana-
3. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1259
tion and planning; and 5) closing the session.8
Laidlaw and colleagues used this model to look at
patients’ satisfaction with their family physicians.
In their study, they found that the study population
was generally very satisfied with the care provided
by their physicians. The patients were also satisfied
with their doctors’ communication skills related to
the ability to put them at ease, to listen to their
complaints, and to create good rapport. The phy-
sicians were found to be weak at using a patient-
centered perspective, providing an adequate
amount of information, and actively involving pa-
tients in the treatment plan.5
Another communication technique is the CLASS
protocol of Buckman.9
CLASS stands for physical
Context, Listening skills, Acknowledgement and ex-
ploration of emotions, Strategy for management,
and Summary closure. Clinical pearls provided in
this model are: ensure privacy, sit down, keep eyes
on the same level as the patient, look unhurried,
ask open questions, pause and tolerate short si-
lences, use repetition of the patient’s words to clar-
ify and assure understanding, develop an empathic
response, propose a management plan the patient
understands and in line with the patient’s goals
and expectations, inquire about uncovered issues,
and make a contract for the next contact.
Fortunately, the knowledge, skills, and attitudes
desirable for effective communication can be taught
and learned. Oh et al. initiated early intensive pa-
tient-centered interview training into their resi-
dency. They looked at residents prior to, immedi-
ately after, and late after the training course. They
found that housestaff were able to retain skills over
the course of the residency and that improvements
were still seen two years after training (compared
with baseline).3
Duke and colleagues found that medicine–pedi-
atric residents felt underprepared to care for am-
bulatory patients with regard to their ability to
counsel and evaluate psychosocial issues. Their res-
idency curriculum lacked the ability to provide spe-
cific feedback to individual residents, because res-
idents were seldom directly observed interviewing
patients. To meet this educational need they de-
signed and implemented a clinical performance ex-
perience with standardized patients (SPs) that em-
phasized complex psychosocial and ambulatory
skills.10
This model, as well as the development of
an objective structured clinical exam (OSCE), has
been successfully implemented at other pro-
grams.11–13
The University of California, Irvine, developed a
longitudinal communication skills curriculum for
medical students, residents, and faculty. This was
prompted by the realization the medical students’
scores on communication skills assessments were
deteriorating during training. The curriculum be-
gins in the first year of medical school and contin-
ues through all four years and includes an OSCE
in the third year and a hands-on introduction to
clinical teaching in the fourth year. Residents and
faculty participate in seminars using the same com-
munications paradigm presented to students. Res-
idents are evaluated using videotaped patient in-
teractions. After initiation of this curriculum,
medical students’ communication skills scores dra-
matically increased (from 69% to 88%) on a state-
administered OSCE. Also, a patient satisfaction sur-
vey from the same institution showed the
likelihood of patients’ recommending their institu-
tion increased from 15% to 82%.14
Written communication skills for the physician
are also important. A large percentage of interac-
tions by physicians are written. It has been shown
that written skills can be taught and assessed even
by an OSCE.15
Yet little documented teaching stan-
dards exist for this important communication
method.
So, how are residents taught the important as-
pects of C-IP skills? When asked about profession-
alism, residents listed three common attributes:
competence, respect, and empathy. When asked
about their preferred method of learning these at-
tributes, their top three responses were: contact
with a positive role model, contact with patients,
and contact with negative role models.16
This study
underscores the importance of learning through
contact with others and that, as academic physi-
cians, we teach by example. This is especially true
of teaching C-IP skills.
An interesting study by Côté and Leclère2
ex-
amined how clinical teachers perceived the doctor–
patient relationship and themselves as role models.
Clinical teachers were able to identify competencies
associated with the relationship and placed impor-
tance on the ability to conduct interviews politely
and effectively, understand and involve the patient,
and handle emotionally charged situations. Inter-
estingly, these educators tended to demand more of
their students with regard to the doctor–patient re-
lationship than they did of themselves. The teach-
ers also had difficulty describing situations or be-
haviors in which they modeled the doctor–patient
relationship. Several of the physicians felt that lack
of time and a negative attitude toward the doctor–
patient relationship were obstacles to teaching and
learning this competency.2
Therefore, we as clinical
teachers of C-IP skills may not currently be the best
role models.
4. 1260 Hobgood et al. • ASSESSMENT OF COMMUNICATION AND INTERPERSONAL SKILLS
SUMMARY OF ASSESSMENT METHODS
Equal to the complexity of the C-IP skills required
to be a competent EM physician are the variety of
assessment methods applicable to their evaluation.
No single evaluation methodology can assess each
of the C-IP skills in its entirety. Table 2 provides a
limited summary of published assessment methods
applicable to C-IP skills. Any of these methods can
be used in EM settings with minor modification. All
are suitable for either formative or summative as-
sessment, at the discretion of the residency director.
The ACGME has also developed a more general list
of the competencies and applicable assessment
methods in the evaluation process. These recom-
mendations have been included in Table 2. A brief
summary of the types of methods currently in use
is provided in the following section along with
those the Consensus Group believes will be most
applicable to EM.
The literature suggests that the most common
type of assessment instrument is a checklist evalu-
ation. Checklist evaluations are common to many
types of assessment measures. In fact, almost all
assessment measures will incorporate either a
checklist or a rating scale as the final common path-
way in the assessment. As the name implies, a
checklist offers only dichotomous responses. Either
an action occurred or it did not. There is no option
for scoring how well the action occurred. The
checklist may also serve as a framework for teach-
ing the competency and as a robust research tool.
The most widespread checklist in use currently
is the SEGUE Framework.17
SEGUE is an acronym
that prompts critical communication tasks (Set the
stage, Elicit information, Give information, Under-
stand the patient’s perspective, End the encounter)
and provides a flow for the encounter. This frame-
work has been well studied in multiple settings in-
cluding SP scoring, blind scoring of videotaped en-
counters, and audiotaped encounters. The overall
interrater reliability for clinical skill assessment
scoring with this instrument is acceptable with SP
Kn = 0.8, videotape Kn = 0.93, and audiotape Kn =
0.87.17
This framework and scoring system could be
easily applied to the EM assessment of C-IP com-
petencies.
Skills rating forms are another excellent way to
evaluate the effectiveness of C-IP skills. The rating
form provides a hierarchy of responses to indicate
how well the subject performed in the evaluation.
In certain situations and behaviors, this may be
more useful than simply identifying whether the
action occurred. For behaviors such as introducing
oneself, a rating may not be useful. As with check-
lists, rating scales are a component of many other
types of assessment modalities and can serve as
guides for teaching and research.
The rating procedure most commonly used is the
single global assessment rating scale. This system
of evaluation consists of one summative form com-
pleted at the end of the evaluation period by the
faculty or supervisor of the learner during the train-
ing interval. These types of evaluations have been
shown in internal medicine populations to have
limited reliability and suffer from significant rater
bias.18
Global rating scales also discriminate poorly
between learners and are poor at identifying spe-
cific educational deficiencies.19
With the exception
of internal medicine, the forms for global rating as-
sessment are not standardized between training
programs; therefore, we have limited data for sys-
temwide analysis of this assessment modality.
Other types of communication scoring forms have
received more specific study.
The Calgary-Cambridge Observation Guide,8
the
Brown University Interpersonal Skill Evaluation
(BUISE),20
and the Arizona Clinical Interview Rat-
ing Scale (ACIR)21
have all been used to assess med-
ical students in practice settings. Their interrater re-
liability ranges from 0.52–0.87 for the ACIR to 0.91
for the BUISE. The Amsterdam Attitudes and Com-
munication Scale is a summative assessment tool
for physician–patient communication and profes-
sional communication.22
It requires completion of a
brief assessment form and evaluation of a detailed
list of observable, criteria-specific behaviors. Its ap-
plicability to EM may be limited because six trained
raters are needed to achieve precision if the instru-
ment is scored for all ten criteria. Other types of
scales have been developed, applied, and evaluated
in either direct observation or SP encounters. The
Liverpool Communication Skill Assessment Scale is
a 12-item assessment method using a mixed
method of checklist and rating scale to evaluate C-
IP competency. This instrument was identified as
having adequate reliability predicted by a general-
izability coefficient of 0.67.23,24
All are readily adapt-
able to EM applications and are potentially useful
for assessing the verbal portion of the communi-
cation encounter. Other scales have been developed
for nonverbal communication skills.
The Relational Communication Scale for Obser-
vation measurement (RCS-O)25
is an adaptation of
the Relational Communication Scale of Burgoon
and colleagues26
that was developed for patient as-
sessment of physician behaviors. These scales rate
the affective tone of the communication and are
measured predominately through the nonverbal
portions of the communication. Both instruments
have adequate interrater reliability in the range of
5. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1261
TABLE 2. Communication and Interpersonal (C-IP) Skills Assessment Methods
Assessment
Type
Assessment Method/
Instrument
Annotated
Reference
No.* Description Instrument Provided
Checklist† Bedside manner com-
munication skills
1 Reviews the key points in the estab-
lishment of therapeutic relation-
ship
No
SEGUE framework 3 Provides an acronym to structure to
the medical communication en-
counter (Set the stage, Elicit infor-
mation, Give information, Under-
stand the patient’s perspective,
End the encounter)
Yes
Rating scale Amsterdam Attitudes &
Communication Scale
4 The AACS is a summative evaluation
instrument, which covers 9 dimen-
sions of a communication en-
counter
Yes
Relational Communica-
tion Scale
6 A 34-item doctor–patient relational
communication scale for doctor–
patient interactions
Yes
Written Communication
Skills Rating Scale
25 A 34-item scale for scoring content,
style, and global rating of resident
dictated consultation letters
Yes: 27 of 34 items are
presented
Self-assessment Doctor–Patient Attitude
Scale
22 An attitude scale devised to discrim-
inate between doctor-centered,
disease-oriented and patient-cen-
tered, problem-oriented physician
beliefs
Yes: sample instrument
questions
Relative Ranking Model
of Self Assessment
24 Self-assessment of communication
skills was compared with expert
evaluation of performance to
identify differences between the
two assessment modalities
No
Peer review Interview Performance
Scale
7 Comparison of self-assessment and
peer assessment for evaluation of
interviewing skills
No
Global Peer Review 17 Explored the utility of peer review,
as an additional method of evalu-
ation in a university internal medi-
cine residency program
Yes
Patient survey† Patient Rating Scale for
Physician Communi-
cation
14 An instrument that allows patients to
give ratings to their physicians;
scoring discrete, observable items
of behavior, and complex, multidi-
mensional attributes
Yes
360-degree as-
sessment†
360-degree 42, 43 Design and implementation sugges-
tions for development of a 360-
degree assessment system
No
Clinical work sampling 18 Assessment system where multiple
small work samples are obtained
at regular intervals from faculty,
patients, and a multidisciplinary
team
Yes: 3 of 4 forms are
presented
Nurse Rating Scale 20 A 13-item questionnaire designed
for scoring by nurses for assess-
ment of humanistic behaviors
Yes
Global rating
forms
Global Rating Scales 31 Review of the utility and application
of the Global Rating Scale in
medical education
No
continued
6. 1262 Hobgood et al. • ASSESSMENT OF COMMUNICATION AND INTERPERSONAL SKILLS
TABLE 2. Communication and Interpersonal (C-IP) Skills Assessment Methods (cont.)
Assessment
Type
Assessment Method/
Instrument
Annotated
Reference
No.* Description Instrument Provided
Direct observa-
tion
Davis Observation Code 21 Development and evaluation of a
20-item direct observation scale
for physician–patient interactions
Yes
Calgary-Cambridge
Observation Guides
28 Two observation guides for real-time
assessment of: 1) the patient inter-
view and 2) explanation and
planning with the patient
Yes
Arizona Clinical Inter-
view Rating Scale
32 Assessment of interview skill in the
areas of organization, timeline,
transitional statements, rapport,
and documentation
Yes
Brown University Inter-
personal Skill Evalua-
tion (BUISE)
43 BUISE assesses C-IP competency via
direct observation either in the
practice setting or via video re-
cordings
No
Portfolio Reflective portfolio 29 Provides background and educa-
tional rationale for portfolio based
learning and assessment; provides
a range of ways portfolios can be
implemented into medical educa-
tion
No
Video portfolio 29 Video artifacts of communication
encounters
No
Procedures and case
logs
Procedure and case log encounters
that document examples of clini-
cal or C-IP experiences
No
Multimodality
assessment
Triple-jump exercises 27 Three-step triple-jump examination
aims to assess clinical problem
solving by means of tiered subjec-
tive assessments administered by
faculty
No
PAME 42 Patient Assessment and Manage-
ment Examination (PAME)—multi-
modality assessment method for
surgical competency
No
Simulations
and models
Basic simulation 40 Range of simulations are available;
low-tech applications are more
applicable to C-IP skills
No
COSMOS 9 Computer-supported method for
operators’ self-assessment of ef-
fective communication during
high-tech, high-stress stimulations
No
Oral examina-
tion
40, 41 May be a chart-stimulated experi-
ence or simulation
No
Standardized
patient (SP)
examina-
tion†
Interpersonal Skills Rat-
ing Form
SP C-IP Rating Form
26
38
The Interpersonal Skills Rating Form
and an evaluation of its use in
standardized patients is presented
Comparison of the reliabilities of a 5-
item SP rating form and a single
global patient assessment of satis-
faction vs. a 17-item checklist
Yes
Yes
Objective
structured
clinical ex-
amination
(OSCE)†
Communication skills
OSCE
13 Development of specific OSCE sta-
tions for communication compe-
tency including difficult cases,
which address communication
skills beyond simple history taking
No
continued
7. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1263
TABLE 2. Communication and Interpersonal (C-IP) Skills Assessment Methods (cont.)
Assessment
Type
Assessment Method/
Instrument
Annotated
Reference
No.* Description Instrument Provided
Liverpool Communica-
tion Skills Assessment
Scale
8, 10 The evaluation technique combines
patient ratings (GSPRS) and health
professional ratings (LCSAS) in an
OSCE format; uses both a check-
list and a rating approach and
could be modified to direct obser-
vation or 360-degree assessment
modalities
Yes
Objective
structured
video exami-
nation (OSVE)
OSVE 11 A new, quick, and efficient method
of assessing specific cognitive
comprehension of clinical com-
munication skills
Yes
*Refers to the annotated bibliography available on the CORD website (http://www.cordem.org).
†Recommended by the Accreditation Council for Graduate Medical Education (ACGME).
0.58 to 0.88, and the majority of the items have been
shown to be internally valid.25,26
Little work has been done to assess the written
communication skills of residents. One instrument
has been developed to assess resident consultation
letters in general practice.27
Another study focused
on the use of clinical notes to evaluate the skills of
foreign medical graduates.28
No EM-specific assess-
ment tools for written communication were iden-
tified.
Standardized patients are well studied as assess-
ment methods for C-IP competency. Many formats
for use of SP encounters have been developed, in-
cluding stations in an OSCE, and unannounced SPs
in ‘‘real’’ clinical context, linked with post-encoun-
ter probes and long station SP encounters.29,30
Typ-
ically, SPs score the residents directly for their C-IP
skills and negate the need for indirect scoring or a
second faculty evaluation. However, faculty or
other trained observers may also score the encoun-
ter directly or via audio31
or video recordings.20,30
Instruments developed and validated for use by
SPs include the Interpersonal Skills Rating Form
(reliability coefficients 0.68–0.85)32
and the Inter-
personal and Communication Skills Checklist (In-
ter-case reliability 0.65).33
Other instruments have
also been used with SPs and they include the ACIR,
the BUISE, and the SEGUE.17,20,21
In summary, the
literature suggests that SPs are a remarkably flexi-
ble and well studied form of communication as-
sessment.
Communication skills may also be assessed by
peers,34
by patients,35
and through self-assessment
techniques.36
These modalities may serve as useful
adjuncts to other types of C-IP assessment, and
when combined may serve as the foundation for
the 360-degree evaluation. The 360-degree evalua-
tion is a multirater feedback system primarily de-
signed for acquisition of feedback from all compo-
nents of a person’s sphere of influence.37,38
In the
clinical setting the 360-degree evaluation can also
be performed using techniques such as clinical
work sampling,39
and evaluations from important
adjunct members of the health care team, such as
nurses.40
Other less well-studied modalities were
identified that could be very useful in EM and may
address the C-IP competency all or in part. These
include exercises such as the triple jump,41
which is
a three-step tiered assessment procedure; the objec-
tive structured video examination,42
a method de-
signed to assess the cognitive aspects of the C-IP
encounter; and portfolio-based learning,43
where
the learner gathers artifacts of the learning process
and provides personal reflection on the encounter
and his or her learning achievement. These multi-
dimensional assessment modalities are important
components of a diverse assessment strategy.
EMERGENCY MEDICINE PERSPECTIVE
So what about our specialty? Very little is reported
about C-IP skills as it relates to EM. Rosenzweig et
al. describe C-IP as the ‘‘art of emergency medi-
cine’’ and opine that as EM educators, it is our job
to teach this art to our residents.44
In 1979, the American College of Emergency Phy-
sicians (ACEP) and the Society of Teachers of Emer-
gency Medicine (STEM) developed the ‘‘Core Con-
tent of Emergency Medicine’’ that outlined the
central body of knowledge of EM and delineated
the educational scope of postgraduate teaching and
continuing medical education (CME).45
In 1986, the
Core Content was revised and included a section
on essential interpersonal skills: communication,
8. 1264 Hobgood et al. • ASSESSMENT OF COMMUNICATION AND INTERPERSONAL SKILLS
empathic listening, objectivity, pain management,
problem resolution, self-control, self-resolution, and
grief reactions.46
Interestingly, there were no pub-
lished guidelines for teaching these skills in EM res-
idencies and, furthermore, there were no standards
set for monitoring the success with which these
skills are taught.
In the June 1997 revision of the Core Content,
section 20 is titled ‘‘Skills’’ and Section 20.12.4 is
defined ‘‘Physician Interpersonal Skills.’’ Subhead-
ings of this section include effective patient–phy-
sician communication, diversity issues, hostile en-
counters/complaints, and grief reactions.47
One
textbook, ‘‘designed to present the body of knowl-
edge that forms the basis of emergency medicine’’
and that parallels the 1997 Core Content, does not
include any text discussion of C-IP skills.48
A survey of EM residency program directors on
the integration of interpersonal skills into the EM
curriculum found a range of teaching strategies.
Classroom instruction, workshops, journal clubs,
and problem patient conferences were used to teach
these skills to EM residents. Fifteen percent of pro-
grams used audiovisual aids and 26% used role-
playing techniques. When asked about bedside in-
struction/evaluation of residents’ interpersonal
skills, program directors reported mean supervi-
sion rates of 15.4% for history taking and 14.7% for
giving discharge instructions. Nearly all (98%) of
the program directors agreed that communication
skills should be taught.44
In 1989, ACEP sponsored a survey of EM grad-
uates to determine their perception of the adequacy
of their training with respect to the areas of the core
content. The lowest rated area of the core content
was physician interpersonal skills. Unfortunately,
10% or fewer of these physicians stated they would
be taking CME courses in this area.49
Rosenzweig et al. developed a process for assess-
ing residents communication skills that incorpo-
rated videotaping of actual patient encounters, a
faculty development program to increase knowl-
edge of the doctor–patient relationship, and crea-
tion of an observation checklist. They found high
interrater reliability in areas of personal introduc-
tions, conflict management, and nonverbal com-
munication. Poor or no agreement was seen in es-
tablishing rapport, gathering information, and
contracting or informing. They recommend the
need for more reliable means of assessing C-IP
skills to determine clinical competency in EM.50
These studies underscore the importance of de-
veloping C-IP skills in the EP and the actual lack
of training EM residencies provide for it. Given the
nature of our specialty, we need to be effective com-
municators. The EP essentially has one chance to
have a positive doctor–patient interaction. The ED
encounter is filled with impersonal elements. The
patient–physician encounter is essentially between
strangers in an emotionally charged environment
and is often dominated by tests and painful pro-
cedures. The patient is physically and emotionally
exposed to a frightening, confusing, and chaotic en-
vironment. It is the job of the EP to make this a
positive experience for both the patient and the
physician using the techniques already mentioned.
In addition, EP interpersonal and communication
skills are used with nurses, technicians, secretaries,
families, out-of-hospital providers, social workers,
and law enforcement agencies. These interactions
must also be effective and efficient and follow the
doctor–patient model. The EP must communicate
clinical information to primary care physicians and
admitting and/or consulting physicians. This often
occurs via verbal exchange; however, written doc-
umentation is of great importance for the EP. It pro-
vides information to other health care team mem-
bers, allows for continuous quality improvement
(CQI) review, and can affect subsequent medicole-
gal proceedings. Documentation is time-consum-
ing, with one study demonstrating that EM resi-
dents spent 21% of their time documenting,51
while
another found that in a pediatric ED, residents
spent 11.6 minutes charting per patient.52
Charting
must be legible, complete, and done in a timely
fashion. A 1999 American Board of Emergency
Medicine (ABEM) in-service exam study showed
that the handwritten medical record was the one
most often used for communication between med-
ical professionals (76%).53
In any educational pro-
gram, written communications skills must be
taught and evaluated.
In 2001, the Model of the Clinical Practice of
Emergency Medicine (Model) was developed to re-
place the 1997 version of the Core Content.54
This
new document was designed to describe more ac-
curately what the EP actually does, based on as-
sessment of patient acuity. The Model provides a
description of tasks that must be performed to pro-
vide appropriate emergency care. Several of these
physician tasks take into account C-IP skills such
as out-of-hospital information transfer and direc-
tion, performance of focused history and physical,
therapeutic interventions, consult and disposition,
documentation, and multitasking and team man-
agement.
EM DEFINITION OF THE C-IP CORE
COMPETENCY
The ACGME defines the C-IP competency as: ‘‘Res-
idents must be able to demonstrate interpersonal
9. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1265
and communication skills that result in effective in-
formation exchange and teaming with patients,
their patients families, and professional associates.
Residents are expected to create and sustain a ther-
apeutic and ethically sound relationship with pa-
tients, use effective listening skills and elicit and
provide information using effective nonverbal, ex-
planatory questioning, writing skills, and work ef-
fectively with others as a member of a health care
team or other professional group.’’1
In defining this competency for EM, the EM pro-
gram directors present at the March 2002 confer-
ence believed that there were several factors unique
to our specialty. First, it was recognized that the ED
encounter is different from many other medical
specialties. Emergency physicians usually have no
pre-existing relationship with the patient and have
only a brief period of time to establish rapport, take
a history, and communicate a plan to the patient.
For the most part, the EP has one shot to get it right.
Emergency physicians also see a diversity of pa-
tients (age, gender, race, ethnicity, spirituality, emo-
tionality, sexual orientation, etc.) and must be able
to meet all these patients’ needs. Next, EPs are truly
part of a medical team, and they must be able to
effectively lead that team and resolve conflict if it
develops. The EP must also be able to identify and
work with diverse factors of the staff as well. Third,
written communication skills are essential for the
EP for several reasons. The medical record serves
to document the patient visit, to convey informa-
tion to primary care physicians and consultants, to
allow CQI activities, and to inform in medicolegal
proceedings. Fourth, the nonverbal cues we pro-
vide to patients, are important, as well as our abil-
ity to accurately identify and address the patient’s
motivation for seeking health care. Unfortunately,
conflict often occurs in EM, and the ability to re-
solve it is an art form requiring directed education
and assessment. Fifth, to ensure long-term profes-
sional development, assessment of the resident’s
ability to handle feedback provided by others is an
important C-IP skill. Finally, several patient en-
counters that are unique to EM require special
training (delivering bad news, death notification,
dealing with emergency medical services, intoxi-
cated patients, high-risk patients who leave against
medical advice, psychotic patients, disaster prepar-
edness and management, etc.). Table 1 lists a work-
ing definition of C-IP competency for EM.
EM ASSESSMENT OF THE C-IP CORE
COMPETENCY
The following section summarizes the application
of known assessment methods to the EM-specific
definition of C-IP competency. The results are de-
picted in Table 3 for the ten specific EM C-IP com-
ponents. This table represents the consensus of the
EM educational experts present at the March 2002
conference. There are many opportunities for EM
program directors to demonstrate resident compe-
tence in the C-IP core competency. It is assumed
that program directors will individualize their ap-
proach. However, several themes emerged from
this discussion.
First, EM has an advantage over other specialties
in that faculty are present on a 24-hour basis with
residents as they perform their duties. The resi-
dents model and learn skills from faculty that will
make them capable of independent EM practice.
We in EM have considerably greater access to our
trainees than most other specialties. We can observe
and immediately assess skills and verify our as-
sessment with other ED staff in real time with the
patient. Second, of all the assessment tools avail-
able, SPs and direct observation (DO) appear to
hold the most promise for EM residency assess-
ment of C-IP.
Standardized patients offer many conveniences.
They can portray the majority of communication
scenarios seen in the ED. They are not costly (price
$10–20/hour for the SP, plus logistic costs), can be
scheduled at convenient times, can offer a gener-
alizable educational experience (i.e., all residents
cover the same topic), and can portray special sit-
uations (e.g., death notification). Since the U.S.
Medical Licensing Examination is moving toward
an SP-based clinical exam as part of the licensing
requirements for all medical school graduates, the
facilities needed for this activity will become more
available. Standardized patients are least helpful
for portrayals of life-threatening illness or injury
cases, or pediatric cases. The ability to present an
unannounced SP in the clinical context of the ED is
technically difficult and may have little application
in the ED environment. However, the use of post-
encounter probes, and the SP as one component of
a multidimensional assessment, is an important as-
sessment technique.
Direct observation (‘‘Mini-CEX’’) is similar to a
program used by internal medicine55
but is superior
for EM. In this program, a faculty member shadows
a resident and watches as he or she performs all
the duties of an EP. This is superior to the Mini-
CEX in that all aspects of performance are ob-
served, not just a brief portion. It also offers advan-
tages over SPs because the evaluator is an expert
in the field and can offer immediate feedback and
teaching in every facet of the case, not just the com-
munication encounter. The other advantage is that
the laboratory where DO occurs is universally
10. 1266 Hobgood et al. • ASSESSMENT OF COMMUNICATION AND INTERPERSONAL SKILLS
TABLE 3. Expert Consensus Regarding
Interpersonal and Communication Skills
1. Demonstrate the ability to respectfully, effectively, and
efficiently develop a therapeutic relationship with
patients and their families.
Type When* Level
Directed observation Now All
Global rating Now All
Patient survey Now All
Standardized patients Then All
2. Demonstrate respect for diversity and cultural, ethnic,
spiritual, emotional, and age-specific differences in pa-
tients and other members of the health care team.
Type When Level
Patient survey (family) Now All
Directed observation Now All
Portfolio Now All
Standardized patients Then All
Standard video (CORD
national product) Goal All
360-degree evaluation Goal All
3. Demonstrate effective listening skills and be able to elicit
and provide information using verbal, nonverbal, written,
and technological skills.
Type When Level
Patient survey Now All
Direct observation Now All
360-degree evaluation Then All
Chart review (documentation) Now All
Portfolio (document) Now All
Standardized patients Then All
Simulations Now All
Triple (standarized patients,
evidence-based medicine,
report) Then All
4. Demonstrate ability to develop flexible communication
strategies and be able to adjust them based on the clini-
cal situation.
What When Level
Global Now All
Direct observation Now All
360-degree evaluation Then All
Simulation Now All
(Teaching) (Later)
Simulators Then/Goal All
Standardized patients Then All
Peer review Now All
5. Demonstrate effective participation in and leadership of
the health care team.
What When Level
Peer review Now Can/should (have
emergency medi-
cine year-specific
objectives)
360-degree evaluation Then All
Simulation Now All
Simulator Then/Goal All
Portfolio (file, to each
grad storage issues) Now All
Video Then All
Global Now All
Direct observation Now All
6. Demonstrate ability to elicit patient’s motivation for seek-
ing health care.
What When Level
Standardized patients Then All
continued
TABLE 3. Expert Consensus Regarding
Interpersonal and Communication Skills (cont.)
What When Level
Observation Now All
Global Now All
7. Demonstrate ability to negotiate as well as resolve con-
flicts.
What When Level
Simulation Now (Level-specific
graduated
objectives)
360-degree evaluation Then All
Peer review Now All
Global Now All
Direct observation Now All
Standardized patients Then All
Video portfolio Then All
Self-assessment Now All
Triple jump Now All
8. Demonstrate effective written communication skills with
other providers and to effectively summarize for the pa-
tient upon discharge.
What When Level
Chart review Now All
Peer review Now All
Patient survey Now All
Standardized patients (follow-up
probe) Then All
Global Now All
Oral exam Now All
360-degree evaluation Then All
Procedure logs (procedure note) Now All
9. Demonstrate ability to effectively use the feedback pro-
vided by others.
What When Level
Portfolio Now All
Global Now All
360-degree evaluation Then All
Peer review Now All
Video Then All
10. Demonstrate ability to handle situations unique to emer-
gency medicine:
k. Intoxicated patients
l. Altered mental status patients
m. Delivering bad news (death notification, critical illness)
n. Difficulties with consultants
o. DNR/end-of-life decisions
p. Patients with communication barriers (non-English-
speaking, hearing-impaired)
q. High-risk refusal of care patients
r. Communication with out-of-hospital personnel as well
as nonmedical personnel (police, media, hospital ad-
ministration)
s. Acutely psychotic patients
t. Disaster medicine
What When Level
Direct observation Now All/graduated
Standardized patients Then
Portfolio Now
360-degree evaluation Then
Simulation Now
Simulators Then
*Now defined as available for use in 2002; Then defined as
work needed to make available in the near future; Goal de-
fined as a long-term goal; Later defined as availability some-
time between then and goal.
11. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1267
available and costs nothing—the ED. The disad-
vantage of DO is that it requires faculty time and
commitment, both of which come with premium
price tags.
Both SPs and DO are currently used in EM resi-
dencies. Burdick and Escovitz have done extensive
work with the development and implementation of
SPs in medical students,56
foreign medical gradu-
ates,57
and EM residents.58
At the University of
North Carolina (UNC) School of Medicine, SPs
have been used for both formative and summative
assessment of EM resident C-IP skills. The SP scor-
ing instrument used at UNC for C-IP skills is avail-
able upon request.
Cydulka and colleagues at Case Western Reserve
University, MetroHealth Medical Center, have ex-
tensive experience with DO of EM residents.59
This
program has been in place for more than eight
years and has provided a unique opportunity to
observe residents at work in the ED. This method
allows evaluation of all facets of EM practice, not
just C-IP skills. For this reason, this method has tre-
mendous potential as an EM assessment method,
as many performance parameters such as multi-
tasking will be difficult to assess in any other for-
mat. The MetroHealth DO instruments are avail-
able on request.
Although SPs and DO were noted to have the
most promise for comprehensive C-IP evaluation,
other types of modalities may also be useful for
certain types of encounters. Low-tech simulations
were thought to have promise for C-IP encounters
such as consultation and conflict resolution. These
simulations would allow the evaluator/simulator
to evaluate firsthand the competence of the learner
in resolving these potentially difficult C-IP encoun-
ters. These could be independent simulations, a
portion of a chart-simulated recall experience, or a
station in an OSCE.
Portfolio development was expanded to include
the acquisition of communication artifacts such as
video or audiotapes of communication encounters.
For example, many programs currently record
trauma resuscitations, which could be used to eval-
uate and assess the functionality and success of the
C-IP encounters that occurred during the resusci-
tation.
Standardized oral examinations could also be
used to assess C-IP skills. This could supplement
more traditional clinical competence assessments or
be performed as a defined C-IP encounter. Use of
these examinations for formative evaluation can be
done without a significant amount of test standard-
ization. However, use of the oral examination for
high-stakes assessment requires significant training
of the test administrators.
The Michigan State University EM Residency re-
cently reported a method for modifying existing
evaluation formats to assess residents based on the
ACGME general competencies. One important con-
cept is that in addition to revising evaluation and
assessment tools, curriculum revision must be
made to ensure that the general competencies are
taught to the residents.60
Other educators have sug-
gested similar formats and methods for ensuring
that the relative value of the learning objective to
the curriculum is matched by the rigor of the as-
sessment.61
VALIDATION
Little work has been done to validate SP assess-
ment measures in EM. Standardized patient ex-
aminations have been used with great success in
undergraduate medical education and clinical
EM.56,58
Factor analysis of SP examinations have de-
termined that the noncognitive components of clin-
ical performance, such as C-IP skills, have a high
degree of construct validity and tend to be stable
across examinations.29,30
These SP assessments have
been shown to discriminate performance at differ-
ent training levels and to provide a fair and mean-
ingful assessment.29
There are no data to suggest
that SPs introduce bias into the testing process
when noncognitive skills are evaluated.29
The vali-
dation of SPs for assessment of postgraduate EM-
specific C-IP core competency is a potential area for
future study; however, there is no reason to suspect
that this modality will not be valid.
Direct observation has been used in many EM
residencies but, to the best of our knowledge, has
never been validated. The Davis Observation Code
for DO in family medicine residencies has been val-
idated using consensus of an expert panel.62
How-
ever, much remains to be done to validate this po-
tentially rich assessment modality in EM.
The lack of validated instruments for assessment
of C-IP competency in EM represents an opportu-
nity for further research to focus the development
of evaluation tools, make them available to all EM
residencies, and maintain a national database of re-
sults to validate the instruments across the spe-
cialty. This would be most useful with the SP and
DO modalities, as these assessment techniques are
believed to represent the criterion standard assess-
ment methods for C-IP skills. This standardization
of instruments would be beneficial to the specialty,
our trainees, and, most importantly, our patients.
CONCLUSIONS/RECOMMENDATIONS
1. The C-IP core competency for EM can be de-
fined by ten major objectives (Table 1).
12. 1268 Hobgood et al. • ASSESSMENT OF COMMUNICATION AND INTERPERSONAL SKILLS
2. There are many potential assessment tools for
these objectives (Table 2).
3. The tools most likely to be helpful in assessing
the C-IP core competency are standardized pa-
tients (SPs) and direct observation (DO).
4. The use of SPs is more expensive to arrange,
but cheaper once the infrastructure is in place.
5. The use of SPs has been validated for the as-
sessment of interpersonal skills in other spe-
cialties.
6. DO is available now and has been successfully
used in EM as an assessment and teaching tool.
7. There are no logistical costs to DO, except the
cost of faculty time, which is expensive.
8. DO utilizes EM faculty experts to assess com-
petency.
9. Both SPs and DO offer the Council of Emer-
gency Medicine Residency Directors (CORD)
the opportunity to create national assessment
instruments. Use of these instruments by all
EM programs and collection of data will allow
validation of the instruments for EM.
An annotated bibliography regarding this topic is
available on the CORD website (http://www.
cordem.org).
References
1. ACGME Outcome Project. Accreditation Council for
Graduate Medical Education website: http://www.
acgme.org. Accessed Feb 25, 2002.
2. Côté L, Leclère H. How clinical teachers perceive the
doctor–patient relationship and themselves as role mod-
els. Acad Med. 2000; 75:1117–24.
3. Oh J, Segal R, Gordon J, Boal J, Jotkowitz A. Retention
and use of patient-centered interviewing skills after in-
tensive training. Acad Med. 2001; 76:647–50.
4. Smith RC, Lyles JS, Mettler JA, et al. A strategy for im-
proving patient satisfaction by the intensive training of
residents in psychosocial medicine: a controlled, random-
ized study. Acad Med. 1995; 70:729–32.
5. Laidlaw TS, Kaufman DM, Macleod H, Sargeant J, Lan-
gille DB. Patients’ satisfaction with their family physi-
cians’ communication skills: a Nova Scotia survey. Acad
Med. 2001; 76(10, Oct RIME suppl):S77–S79.
6. McGuire P, Fairbaim S, Fletcher C. Consultation skills of
young doctors. Most young doctors are bad at giving in-
formation. Br Med J. 1986; 292:1576–8.
7. Billings-Gagliardi S, Mazor KM. Belanger M. Explana-
tions of basic medical information by students: what lay
people find effective. Acad Med. 2001; 76(10, Oct RIME
suppl):S39–S41.
8. Kurtz SM, Silverman J, Draper J. The Calgary-Cambridge
Observation Guides: an aid to defining the curriculum
and organizing the teaching in communication training
programs. Med Educ. 1996; 30:83–9.
9. Buckman R. How to Break Bad News: A Guide for
Health Care Professionals. Baltimore, MD: Johns Hopkins
University Press, 1992.
10. Duke MB, Griffith CH, Haist SA, Wilson JF. A clinical
performance exercise for medicine-pediatrics residents
emphasizing complex psychosocial skills. Acad Med.
2001; 76:1153–7.
11. Warf BC, Donnelly MB, Schwartz RW, Sloan DA. The rel-
ative contributions of interpersonal and specific clinical
skills to the perception of global clinical competence. J
Surg Res. 1999; 86:17–23.
12. Donnelly MB, Sloan D, Plymale M, Schwartz R. Assess-
ment of residents’ interpersonal skills by faculty proctors
and standardized patients: a psychometric analysis. Acad
Med. 2000; 75(10, Oct RIME suppl):S93–S95.
13. Peskin E, O’Dell K. Communication skills in women’s
health care: helping students clarify values related to
challenging topics in ob–gyn. Acad Med. 2001; 76:509–
10.
14. Rucker L, Morrison E. A longitudinal communication
skills initiative for an academic health system. Med
Educ. 2001; 35:1087–8.
15. Keely E, Myers K, Dojeiji S. Can written communication
skills be tested in an objective structured clinical exami-
nation format? Acad Med. 2002; 77:82–6.
16. Brownell AKW, Côté L. Senior residents’ views on the
meaning of professionalism and how they learn about it.
Acad Med. 2001; 76:734–7.
17. Makoul G. The SEGUE framework for teaching and as-
sessing communication skills. Pat Educ Couns. 2001; Oct
45(1):23–34.
18. Gray JD. Global rating scales in residency education.
Acad Med. 1996; 71(1, Jan RIME suppl):S55–S63.
19. Noel G, Herbers JE, Caplow M, et al. How well do inter-
nal medicine faculty members evaluate the clinical skills
of residents? Ann Intern Med. 1992; 117:757–65.
20. Burchard KW, Rowland-Morin PA. A new method of as-
sessing the interpersonal skills of surgeons. Acad Med.
1990; 65:274–6.
21. Stillman PL. Construct validation of the Arizona Clinical
Interview Rating Scale. Educ Psychol Measure. 1977; 37:
1031–8.
22. de Haes JC, Oort F, Oosterveld P, Cate O. Assessment of
medical students’ communicative behavior and attitudes:
estimating the reliability of the use of the Amsterdam At-
titudes and Communication Scale through generalizabil-
ity coefficients. Pat Educ Couns. 2001; Oct 45(1):35–42.
23. Humphris GM, Kaney S. The Liverpool Brief Assessment
System for communication skills in the making of doc-
tors. Adv Health Sci Educ Theory Pract. 2001; 6(1):69–80.
24. Humphris GM, Kaney S. Assessing the development of
communication skills in undergraduate medical students.
Med Educ. 2001; 35:225–31.
25. Gallagher TJ, Hartung PJ, Gregory SW. Assessment of a
measure of relational communication for doctor patient
interactions. Pat Educ Couns. 2001; Dec 45(3):211–8.
26. Burgoon JK, Pfau M, Parrott R, et al. Relational commu-
nication, satisfaction, compliance-gaining strategies, and
compliance in communication between physicians and
patients. Commun Monogr. 1987; 54:307–24.
27. Myers KA, Keely EJ, Dojeiji S, Norman GR. Development
of a rating scale to evaluate written communication skills
of residents. Acad Med. 1999; 74 (10, OCT RIME Suppl):
S111–S113.
28. Stillman PL, Regan MB, Haley HLA, Norcini JJ, Fried-
man M, Sutnick AI. The use of a patient note to evaluate
clinical skills of first-year residents who are graduates of
foreign medical schools. Acad Med. 1992; 67(10, Oct
RIME suppl):S57–S79.
29. Vu NV, Barrows HS. Use of standardized patients in clin-
ical assessments: recent developments and measurement
findings. Educ Res. 1994; 23:23–30.
30. van der Vleuten C, Swanson DB. Assessment of clinical
skills with standardized patients: state of the art. Teach
Learn Med. 1990; 2:58–76.
31. Macdougall C, O’Hallorhan C. Keeping it simple—audio
13. ACAD EMERG MED • November 2002, Vol. 9, No. 11 • www.aemj.org 1269
taping in consultation performance assessment. Med
Educ. 2001; 35:1091.
32. Schnabl GK, Hassard TH, Kopelow ML. The assessment
of interpersonal skills using standardized patients. Acad
Med. 1991; 66(9, Sept RIME suppl):S34–S46.
33. Cohen DS, Colliver JA, Marcy MS, Fried ED, Swartz
MH. Psychometric properties of a standardized-patient
checklist and rating-scale form used to assess interper-
sonal and communication skills. Acad Med. 1996; 71(1,
Jan RIME suppl):S87–S89.
34. Thomas PA, Gebo KA, Hellmann DB. A pilot study of
peer review in residency training. J Gen Intern Med.
1999; 14:551–4.
35. Matthews DA, Feinstein AR. A new instrument for pa-
tients’ ratings of physician performance in the hospital
setting. J Gen Intern Med. 1989; 4:14–22.
36. de Monchy C, Richardson R, Brown RA, Harden RM.
Measuring attitudes of doctors: the doctor patient (DP)
rating. Med Educ. 1988; 22:231–9.
37. Antonioni D. Designing an effective 360-degree appraisal
feedback process. Organizational Dynamics. 1996; Apr:
24–38.
38. Carey R. Coming around to the 360-feedback. Perfor-
mance. 1995; Mar:56–60.
39. Turnbull J, MacFadyen J, Van Barneveld C, Norman G.
Clinical work sampling: a new approach to the problem
of in-training evaluation. J Gen Intern Med. 2000; 15:556–
61.
40. Butterfield PS, Mazzaferri EL, Sachs LA. Nurses as eval-
uators of the humanistic behavior of internal medicine
residents. Med Educ. 1987; 62:842–9.
41. Smith RM. The triple-jump examination as an assessment
tool in the problem-based medical curriculum at the Uni-
versity of Hawaii. Acad Med. 1993; 68:366–72.
42. Humphris GM, Kaney S. The objective structured video
exam for assessment of communication skills. Med Educ.
2000; Nov 34:939–45.
43. Challis M. AMEE Medical Education Guide No. 11: port-
folio-based learning and assessment in medical educa-
tion. Med Teach. 1999; 21:370–86.
44. Rosenzweig S, Brigham TP, Snyder RD, Xu G, McDonald
AJ. Assessing emergency medicine resident communica-
tion skills using videotaped patient encounters: gaps in
inter-rater reliability. J Emerg Med. 1999; 17:355–61.
45. Graduate/Undergraduate Education Committee of the
American College of Emergency Physicians. Emergency
medicine core content. J Am Coll Emerg Physicians. 1979;
8:58–65.
46. Special Committee on the Core Content Revision. Emer-
gency medicine content. Ann Emerg Med. 1986; 15:853–
63.
47. Task Force on the Core Content for Emergency Medicine
Revision. Core content for emergency medicine. Ann
Emerg Med. 1997; 29:791–811.
48. Aghababian RV, Allison EJ, Braen GR, Fleisher GR, Mc-
Cabe JB, Moorhead JC (eds). Emergency Medicine: The
Core Curriculum. Philadelphia: Lippincott-Raven, 1998.
49. Moorhead JC, Adams BE, Aghababian RV, et al. An as-
sessment of emergency medicine residency graduates’
perceptions of the adequacy of their residency training.
Ann Emerg Med. 1989; 18:701–4.
50. Rosenzweig S, Brigham TP, Snyder RD, Xu, G, Mc-
Donald A. Assessing emergency medicine resident com-
munication skills using videotaped patient encounter;
gaps in inter-rater reliability. J Emerg Med. 1999; 17:355–
61.
51. Hollingsworth JC, Chisolm CD, Giles BK. How do physi-
cians and nurses spend their time in the emergency de-
partment? Ann Emerg Med. 1998; 31:87–91.
52. Chin L, Fleisher G. Planning model of resource utiliza-
tion in an academic pediatric emergency department. Pe-
diatr Emerg Care. 1998; 14:4–9.
53. Howell J, Chisholm C, Clark A, Spillane L. Emergency
medicine resident documentation: results of the 1999
American Board of Emergency Medicine in-training ex-
amination survey. Acad Emerg Med. 2000; 7:1135–8.
54. Hockberger RS, Binder LS, Graber MA, et al. The model
of clinical practice. Ann Emerg Med. 2001; 37:745–70.
55. The Mini-CEX. Website: www.abim.org.
56. Burdick WP, Escovitz ES. Use of standardized patients in
a freshman emergency medicine course. J Emerg Med.
1992; 10:627–9.
57. Boulet JR, Ben-David MF, Ziv A, et al. Using standard-
ized patients to assess the interpersonal skills of physi-
cians. Acad Med. 1998; 73(10, Oct RIME suppl):S94–S96.
58. Burdick WP, Ben-David MF, Swisher L, et al. Reliability
of performance-based clinical skill assessment of emer-
gency medicine residents. Acad Emerg Med. 1996; 3:
1119–23.
59. Cydulka RK, Emerman CL, Jouriles NJ. Evaluation of
resident performance and intensive bedside teaching
during direct observation. Acad Emerg Med. 1996; 3:345–
51.
60. Reisdorff EJ, Hayes OW, Carlson DJ, Walker GL. Assess-
ing the new general competencies for resident education:
a model from an emergency medicine program. Acad
Med. 2001; 76:753–7.
61. Hall JC. Assessment dispersion matrices. Med Educ.
2001; 35:345–7.
62. Callahan EJ, Bertakis KD. Development and validation of
the Davis Observation Code. Fam Med. 1991; 23:19–24.