Can Unannounced Standardized PatientsAssess Professionalism and CommunicationSkills in the Emergency Department?Sondra Zab...
916                                                                  Zabar et al.   •   PROFESSIONALISM AND COMMUNICATIONT...
ACAD EMERG MED • September 2009, Vol. 16, No. 9    •                                                       ...
918                                                                   Zabar et al.   •   PROFESSIONALISM AND COMMUNICATION...
Upcoming SlideShare
Loading in...5

EMPACT: Emergency Medicine Professionalism and Communication Training


Published on

"Emergency Medicine Resident Training in Interprofessional Skills: Evaluating a Needs-Based Curriculum"

Sondra Zabar, M.D., Principal Investigator Associate Professor of Medicine
Linda Regan M.D., Co-Investigator New York University School of Medicine

EMPACT aims to expand on previous work by assessing and improving EM resident competency in communication and professionalism through the development, implementation, and evaluation of new curriculum and assessment measures.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

EMPACT: Emergency Medicine Professionalism and Communication Training

  1. 1. Can Unannounced Standardized PatientsAssess Professionalism and CommunicationSkills in the Emergency Department?Sondra Zabar, MD, Tavinder Ark, MSc, Colleen Gillespie, PhD, Amy Hsieh, MPA, Adina Kalet, MD,Elizabeth Kachur, PhD, Jeffrey Manko, MD, and Linda Regan, MDAbstract Objectives: The authors piloted unannounced standardized patients (USPs) in an emergency medicine (EM) residency to test feasibility, acceptability, and performance assessment of professionalism and com- munication skills. Methods: Fifteen postgraduate year (PGY)-2 EM residents were scheduled to be visited by two USPs while working in the emergency department (ED). Multidisciplinary support was utilized to ensure suc- cessful USP introduction. Scores (% well done) were calculated for communication and professionalism skills using a 26-item, behaviorally anchored checklist. Residents’ attitudes toward USPs and USP detec- tion were also surveyed. Results: Of 27 USP encounters attempted, 17 (62%) were successfully completed. The detection rate was 44%. Eighty-three percent of residents who encountered a USP felt that the encounter did not hinder daily practice and did not make them uncomfortable (86%) or suspicious of patients (71%). Overall, resi- dents received a mean score of 60% for communication items rated ‘‘well done’’ (SD ± 28%, range = 23%–100%) and 53% of professionalism items ‘‘well done’’ (SD ± 20%, range = 23%-85%). Resi- dents’ communication skills were weakest for patient education and counseling (mean = 43%, SD ± 31%), compared with information gathering (68%, SD ± 36% and relationship development (62%, SD ± 32%). Scores of residents who detected USPs did not differ from those who had not. Conclusions: Implementing USPs in the ED is feasible and acceptable to staff. The unpredictability of the ED, specifically resident schedules, accounted for most incomplete encounters. USPs may represent a new way to assess real-time resident physician performance without the need for faculty resources or the bias introduced by direct observation. ACADEMIC EMERGENCY MEDICINE 2009; 16:915–918 ª 2009 by the Society for Academic Emergency Medicine Keywords: standardized patients, graduate medical education, assessment, OSCE, professionalism, communication, assessmentW hat options exist for assessing communica- Accreditation Council for Graduate Medical Education tion and professionalism skills? As resi- (ACGME) Outcomes Project,1 robust modalities to eval- dency programs seek to comply with the uate clinical performance and effectiveness of educa- tion are in high demand. The ACGME’s ToolboxFrom the New York University School of Medicine (SZ, TA, contains numerous tools for assessing communicationCG, AH, AK, EK, JM, LR), New York, NY; and The Johns skills,2 but many of these rely on self-assessment orHopkins University School of Medicine (LR), Baltimore, MD. trained observers present during patient encounters.Received February 27, 2009; revision received May 21, 2009; Patient complaints and postvisit surveys are useful foraccepted May 22, 2009. obtaining information, but offer limited opportunityPresented at The Gold Foundation Symposium, ‘‘How Are We for physicians to translate feedback into practiceTeaching Humanism in Medicine and What is Working?’’ change. Unannounced standardized patients (USPs)3–5September 27–29, 2007, Chicago, IL; and the 9th Annual Inter- present a method of measuring physicians’ communi-national Meeting on Simulation in Healthcare (IMSH), January cation and professionalism skills in a real practice set-10–14, 2009, Lake Buena Vista, FL. ting without the artificiality inherent in observedSupported by Picker Institute Challenge Grant 2007. structured clinical exams (OSCEs).6–8Address for correspondence and reprints: Sondra Zabar, MD; We hypothesized that USPs can provide a real-time,e-mail: accurate alternative to direct observation and OSCEs.ª 2009 by the Society for Academic Emergency Medicine ISSN 1069-6563doi: 10.1111/j.1553-2712.2009.00510.x PII ISSN 1069-6563583 915
  2. 2. 916 Zabar et al. • PROFESSIONALISM AND COMMUNICATIONThe purpose of this project was to 1) describe the tion, 6) review and practice with rating checklist, andprocess of conducting a USP program in an emergency 7) preparatory observational visit to the ED. Actorsdepartment (ED), 2) determine if implementing USPs in were compensated at a rate of $25 ⁄ hour for both train-the ED is feasible, and 3) present preliminary results of ing and in-ED time.a USP performance assessment. USP Encounter. Unannounced standardized patientsMETHODS met project staff while residents attended a required conference. USPs were introduced to the triage nurse,Study Design the MR administrator, and the attending. The USPsThis was a prospective, nonrandomized, cohort study were triaged per standard assess professionalism and communication abilities During the encounter, USPs complied with any (non-of emergency medicine (EM) residents using USPs. invasive) exam and accepted all appointments andInformed consent was obtained from all participants. prescriptions, which were canceled postencounter. IfResearch activities in this study were approved by the the resident insisted on any course of action that madeNew York University School of Medicine Institutional the USP feel unsafe, the USP was to ask for the attending,Review Board through a resident registry wherein resi- send a short message service (SMS) text message todents are asked to consent to allow inclusion of their project staff, or simply leave the ED. Hospital billingeducational and performance data in a research data- canceled the visit at the end of the day. Total time inbase. Data, therefore, are reported only for those resi- the ED was 1.5 to 4 hours ⁄ visit. Immediately followingdents for whom such consent was obtained. the encounter, the USP debriefed and completed a behaviorally anchored checklist that assessed residentStudy Setting and Population skills and the USP’s satisfaction with the visit.The Bellevue ED is a busy Level 1 trauma center at anacademic medical center in New York City. The ED sees Post-USP Survey. At the end of the project, all EMapproximately 100,000 visits per year. residents (including those who did not see a USP; Fifteen EM residents in their second year of post- n = 30) were surveyed about their attitudes towardgraduate training (PGY-2) participated in the EM Pro- USPs using a four-point scale (1 = strongly disagree,fessionalism and Communication Training (EMPACT) 4 = strongly agree) and open-ended questions. ToProgram. At the conclusion of the EMPACT training, determine detection rates, residents were asked if theyresidents were informed that they might be visited by had encountered a USP and if so to identify the USP’sUSPs during their subsequent time working in the sex and chief complaint.ED. However, residents were blinded as to the exactdate of the visit or patient complaint. Data Analysis Unannounced standardized patients assessed residents’Study Protocol professionalism and communication skills and theirLogistics. We required involvement from most ED satisfaction with the patient-centeredness9,10 of thestaff areas including nurses, attending physicians, medi- visit using a three-point scale: ‘‘not done,’’ ‘‘partiallycal records (MRs), registration, informatics, and radio- done,’’ and ‘‘well done.’’ Scores were calculated as thelogy. To ensure fidelity for each USP visit, we created a percentage of well-done items (Table 1). Professional-preexisting MR with a unique number, patient name ism and communication skills were scored from 13and identifying information, prior visits, and test items and patient centeredness from eight items. Over-results. Each resident was scheduled to receive two all recommendation ratings were obtained using aUSPs in urgent care (where residents’ schedules were four-point scale. Reliability estimates (Cronbach’srelatively predictable) during the 4 to 6 weeks after the alpha ‘t’) and descriptive statistics (means, standardEMPACT curriculum. deviations [SDs], and ranges) are reported. Correla- tions (Pearson’s r) between scores earned in the twoUSP Scenarios. We used two USP cases previously separate cases are also reported to assess stability ofvalidated in OSCEs, representing common ED chal- performance.lenges and requiring only communication-based inter-ventions. In the first case (a misread x-ray), residents RESULTSneeded to educate an angry patient recalled for a mis-read x-ray (skills: delivering bad news, dealing with a Seventeen of 27 visits were successfully conducted andchallenging patient, accountability), and in the second evaluated. Resident scheduling problems explained(a repeat visitor), care for a dissatisfied patient with most incomplete encounters. Five residents were visitedchronic pain who repeatedly uses the ED (skills: han- by USPs from both cases, and seven residents from onedling emotion, patient education, accountability). case.USP Training. Eight actors were recruited. On aver- USP Detectionage, each received seven hours of training consisting of Seven of 12 residents who encountered a USP pro-1) discussion of character and situation, 2) calibration vided information on detection; four of their nineof emotional tone, 3) role play for standardization, encounters were detected (44% detection rate). Five of4) practice with attending and chief residents for realism, 18 residents who did not see a USP indicated that they5) review of ‘‘ground rules’’ for safety and nondetec- did (28% false-positive rate). One of the residents who
  3. 3. ACAD EMERG MED • September 2009, Vol. 16, No. 9 • 917Table 1Resident Performance with USPs Domains of Assessment Items Mean, % ±SD, % Range, % Reliability Communication* 60 28 23–100 0.91 Information gathering Used appropriate questions 68 36 0–100 0.82 Clarified information Allowed to talk without interrupting Relationship development Communicated concern 62 32 20–100 0.85 Nonverbal enhanced communication Acknowledged emotions Was accepting ⁄ nonjudgmental Used words you understood Education and counseling Asked questions to see what you understood 43 31 0–100 0.78 Provided clear explanations Collaborated with you in identifying next steps Professionalism* 53 20 23–85 0.62 Accountability Disclosed error 49 23 0–80 0.60 Personally apologized Took responsibility for situation Manage difficult situation Avoided assigning blame 91 16 60–100 0.85 Maintained professionalism Giving bad news Prepared you to receive news 42 34 0–83 0.63 Gave you opportunity to emotionally respond Provided appropriate next steps Treatment plan and management Assessed resources 50 39 0–100 0.66 Arranged for follow-up Discussed plan Patient Centeredness* Fully explored my experience 43 29 0–75 0.91 Explored my expectations Came to an agreement Took a personal interest in me Earned (regained) my trust Acknowledged impact of error Didn’t make me feel wasting time I was given enough information Recommendation  2.3 0.9 1.0–3.5 0.90 n = 12 residents, 17 visits; reliability assessed with Cronbach’s alpha. Four-point scale: 1 = not recommend; 2 = recommend with reservations; 3 = recommend; 4 = highly recommend. USPs = unannounced standardized patients. *Mean percentage of items rated as ‘‘well done’’ (not, partly, or well done).  ’’Would you recommend this physician to a family member of friend?’’reported a false detection reported ignoring that think more (29%), or led them to be more self-awarepatient. (43%).USP Performance DISCUSSIONThe reliability of scores (Table 1) suggests adequateinternal consistency (a > 0.60). Residents performed Our results show that developing and implementing abetter in the misread x-ray case than in the repeat USP program in the ED is feasible and acceptable tovisitor case in professionalism (70% vs. 35%, t = 2.81, residents. Considering the drawbacks of OSCE assess-p = 0.048) and patient-centeredness (66% vs. 40%, ment and direct observation, combined with increasingt = 1.96, p = 0.05). Communication (r = 0.73, p = 0.16) demands on faculty time and decreasing funding, USPsand recommendation scores (r = 0.81, p = 0.09) were may offer an objective, cost-effective method for evalu-highly, albeit not significantly, correlated between the ating accurate practice skills.two cases, but professionalism (r = 0.24, p = 0.70), and The biggest challenge faced while implementing thepatient-centeredness were not (r = 0.08, p = 0.90), sug- USP program was the unpredictability of the ED. Occa-gesting case content matters most in these domains. sionally, USPs were mistakenly examined by another resident. Both content (highly trained SP, realisticPostevaluation Survey cases) and logistic factors (dedicated program coordi-Eighty-three percent of residents who encountered a nator, electronic MRs, team collaboration) are neces-USP felt that it did not hinder their daily practice sary for successful integration. Total cost, in terms ofand did not make them uncomfortable (86%) or sus- both time and money, is likely greater up front, withpicious of patients (71%). A minority of those resi- decreased workload, time, and expense as USPs anddents who encountered a USP felt that the encounter staff become trained. Further study of the costs isimproved their practice behavior (14%), made them needed.
  4. 4. 918 Zabar et al. • PROFESSIONALISM AND COMMUNICATION Even with a high detection rate, residents reported at: in the USP program for learning and patient care. asp. Accessed Sep 20, 2008.It is possible that informing residents that USPs would 2. Accreditation Council for Graduate Medical Educa-be visiting them in the ED improved performance. tion, American Board of Medical Specialties.More importantly, the majority of residents did not feel Outcome Project Toolbox of Assessment Methods.that the possibility of encountering a USP had any neg- Available at: impact on their daily practice, suggesting that toolbox.asp. Accessed Jun 20, 2009.USPs in the ED will not risk real patient safety. The 3. Gorter S, Scherpbier A, Brauer J, et al. Doctor-case of the resident who reported ignoring a patient patient interaction: standardized patients’ reflec-thought to be ‘‘unannounced’’ represents an unantici- tions from inside the rheumatological office.pated and anomalous professionalism issue, we believe, J Rheumatol. 2002; 29(7):1496–500.not causally related to the use of USPs; it demonstrates 4. Kravitz RL, Epstein RM, Feldman MD, et al. Influ-how USPs can provide useful information to program ence of patients’ requests for direct-to-customerdirectors. advertised antidepressants: a randomized controlled trial. JAMA. 2005; 293(16):1995–2002.LIMITATIONS 5. Ozuah PO, Reznik M. Using unannounced standard- ised patients to assess residents’ professionalism.There was a small sample size, with a relatively large Med Educ. 2008; 42(5):532–3.proportion of failed USP visits. However, the failure 6. Fiscella K, Franks P, Srinivasan M, Kravitz RL,rate improved as the project progressed. Even with our Epstein R. Ratings of physician communication bysmall numbers, it appears that two cases and the items real and standardized patients. Ann Fam Med. 2007;on the behaviorally anchored checklist can discriminate 5(2):151–8.residents based on their communication skills. 7. Talente G, Haist SA, Wilson JF. The relationship between experience with standardized patientCONCLUSIONS examinations and subsequent standardized patient examination performance: A potential problem withWith the ACGME placing greater importance on evalua- standardized patient exam validity. Eval Health Prof.tion of patient outcomes, we believe that our project 2007; 30(1):64–74.represents a new way to assess real-time resident perfor- 8. Srinivasan M, Franks P, Meredith LS, Fiscella K,mance. Despite being time-consuming and subject to the Epstein RM, Kravitz RL. Connoisseurs of care?unpredictability of the ED, implementing unannounced unannounced standardized patients’ ratings of phy-standardized patients in the ED is feasible and acceptable sicians. Med Care. 2006; 44(12):1092– staff. Future comparison of unannounced standard- 9. Marshall GN, Hays RD. The Patient Satisfactionized patients with observed structured clinical exam Questionnaire Short Form (PSQ-18). RAND Corpo-scores will enable educators to determine how well these ration, Paper P-7865. Available at: http://www.methods assess performance in actual practice. Accessed Jun 20, 2009. 10. Elwyn G, Edwards A, Wensing M, Hood K, AtwellReferences C, Grol R. Shared decision making: developing the OPTION scale for measuring patient involvement. 1. Accreditation Council for Graduate Medical Educa- Qual Saf Health Care. 2003; 12:93–9. tion (ACGME). ACGME Outcome Project. Available