Reed G. Williams, PhD
the patients were debriefed after the exam, one described a physician who was hostile and
performed an uncomfortable exami...
competence. It also is important to give students broad experience, since performance may
    be related to experience.
• ...
As part of the Association of American Medical Colleges Medical School Objectives Project
(MSOP), a survey was conducted a...
•   There are many opportunities for further research, including psychometric studies and
    comparisons of global and ch...
Faculty development involving the Simulation Center takes many forms. There is informal
consultation, for example, with cl...
The CSA is a performance-based examination using standardized patients in an OSCE-type
process. Each examinee completes 10...
Pass rates to date are contained in the following table:

(1985) recommended that a test of clinical skills be incorporated into the examinations for
licensure as soon as it was sh...
Therefore, the class of 2005 (which will enter in the fall of 2001) will be the first to take the
standardized patient exa...
Upcoming SlideShare
Loading in …5



Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. STANDARDIZED/SIMULATED PATIENTS IN MEDICAL EDUCATION Reed G. Williams, PhD Professor and Vice Chair for Educational Affairs Department of Surgery, SIU School of Medicine Gregory Makoul, PhD Associate Professor and Director, Program in Communication and Medicine Northwestern University Medical School Richard Hawkins, MD Director, National Capital Area Medical Simulation Center Uniformed Services University of the Health Sciences James A. Hallock, MD President and Chief Executive Officer Educational Commission for Foreign Medical Graduates Peter Scoles, MD Vice President, Assessment Programs National Board of Medical Examiners Moderator Michael J. Reichgott, MD, Chair-elect Section on Medical Schools -------------------------------------------------------------------------------------------------------------------- - Dr. Reichgott A 1993 paper by Dr. Howard Barrows1 noted that he had been working in the field of standardized patients for 30 years. It is impressive how long this technique for teaching and evaluation has been in place. In medical education, we have a current challenge to develop validated outcome measures for performance. This is critical to ensure competence. Standardized patients can be a tool for such evaluation, as well as for education. Dr. Williams Significant Events in the use of Standardized Patients: The history of standardized patient use is a success story that documents what can be accomplished through systematic educational research, development, and program evaluation. Three significant events led to Howard Barrows’ creation of the first standardized patient. As a chief resident at the New York Neurological Institute, Barrows worked on the service of an attending physician who observed all medical students work up a patient from beginning to end. When asked why, the physician replied that no one else was watching students. Barrows noted that in the absence of observation and feedback, errors could persist. The second major event occurred as Barrows selected and managed patients for the neurology board examination. When
  2. 2. the patients were debriefed after the exam, one described a physician who was hostile and performed an uncomfortable examination. When told that the physician would be spoken to, the patient said that he had “fixed” the examinee by “changing his Babinsky from one side to the other” and changing his sensory findings. The third triggering event for Barrows came when he was developing a set of films on the neurological examination using an artist’s model. He noted that the films did not include the elements of observation and feedback, so important for learning. He began to think about teaching the model to display a neurological problem, like the patient who could change his findings at will. In his first case in 1963, he taught the model to portray the signs, symptoms, and history of a paraplegic patient with MS. He also developed a checklist for the standardized patient to fill out on what was done by the trainee. This is similar to the process that is used today. In a second landmark, Kretzschmar at the University of Iowa taught patients to be Gynecologic Teaching Associates (GTAs). The GTAs then taught the pelvic and breast examinations to students. This approach still is prevalent.2 Students liked this format because they were able to learn from the process. The Objective Structured Clinical Examination (OSCE) was introduced by Harden and colleagues in 1975. Previously, encounters with patients for education or evaluation had included a full examination. The OSCE format focused on the assessment of a specific competency, with a short encounter. In 1976, standardized patients first were sent into physician offices in unannounced visits. A study using this approach by Kopelow and colleagues in Manitoba asked physician faculty members to determine how many items on a checklist a competent physician should perform. Then, standardized patients were sent into these same faculty members’ practices. The physicians who set the standards, on average, performed 60% of the items that they said were essential for competent performance. This study highlighted the problems with developing reasonable standards of performance. The introduction of the ATLS course (in 1976-1978) was an innovative, systematic use of standardized encounters for training and assessment purposes. In 1984, Stillman and colleagues made the first report of a multi-institutional standardized patient examination, when a number of residency programs in the northeast gave the same examination to their residents. The first use of standardized patients in a licensure examination (by the Medical Council of Canada) occurred in 1993. This use of standardized patients in a “high stakes” examination is a significant landmark. The 1998 addition by the Educational Commission for Foreign Medical Graduates of an assessment of clinical skills was another such example. Importance of Standardized Patient Use • Case specificity – Since clinical performance is case-specific, there is a need to increase the number of cases and balance cases in order to make a general estimate about clinical Page 2
  3. 3. competence. It also is important to give students broad experience, since performance may be related to experience. • Convenience – Standardized patients provide the cases that you need at the time and in the place they are needed. • Direct comparisons of competence – Standardized patient evaluations allow a direct comparison of students’ clinical skills. Previously, direct comparisons among students could only be done in the cognitive domain (for example, through USMLE results). • Compression/expansion of time – Use of standardized patient simulations allows students to have a longitudinal experience with patients and to follow a case in a compressed time frame. • Safe practice – Simulations allow students to be put in clinical situations that they could not manage alone in a real clinical setting. • Efficient use of physician faculty time – A physician can train a number of standardized patients who can then teach/evaluate students. This leaves the physician free to concentrate on specific areas where his/her expertise is most useful. • Standardized patient evaluations are responsive to real differences in performance. What is likely to happen next? We are likely to see more of the same in the use of standardized patients. Cases have become stereotypical, and there is a need to build in more creativity. It is likely that there will be more use of standardized patients to make unannounced visits to physician practices to assess variation in practice for health care research. What should happen next? There should be more use of standardized patients for health care research, for training, and for clinical performance appraisal. Current methods of performance assessment (such as observation by faculty members and residents) do not pick up differences in performance among trainees; standardized patient methods can accomplish this better. Better use should be made of clinical practice data, health outcomes research data, and patient safety data to identify problem areas that will guide the development of standardized patient cases. Finally, increased sophistication would be useful in what data are captured from standardized patient encounters and how these data are interpreted. Summary • Clinical competence is highly case specific • Standardized patients can ensure broad exposure to a variety of cases. • Current assessment methods do not capture differences in clinical performance. • Standardized patients allow direct comparison of trainees’ clinical skills. • Standardized patients can be useful physician extenders, by sparing physician time. • Scores in standardized patient encounters reflect real differences in clinical performance. There is a need to broaden the scope of standardized patient use and to link the selection and design of cases to clinical data. Dr. Makoul In the past, students were not routinely observed working with patients. Standardized patients can help with this problem, but are not the only answer. Page 3
  4. 4. As part of the Association of American Medical Colleges Medical School Objectives Project (MSOP), a survey was conducted about methods used to teach and evaluate communication skills. Eighty-nine schools responded. Of the responding schools, 79% used interviews with simulated patients to teach these skills. Feedback by simulated patients was used in 76% of schools and assessment with simulated patients (e.g., in an OSCE) was used in 70% of schools. At Northwestern, communication skills are taught throughout the curriculum. There is a Communication Skills unit for first-year students, an assessment of clinical (including communication) skills at the end of the first and second years, and tailored communication skills modules during the clerkships. There are plans to reintroduce a clinical skills assessment at the end of the third year. First-year Communication Skills Unit This unit is part of the two-year “Patient, Physician, and Society” course. The unit begins early in the first year, to position communication as a fundamental clinical skill, prior to students learning to take a patient history. In the unit, students practice and obtain feedback on 25 communication tasks. During the unit, groups of about 44 students meet with faculty to discuss concepts. In groups of 11, students meet with clinicians. Practice groups of four students work with patient instructors. The patient instructors change their demeanor and personal details from student to student, allowing practice with a broad range of patient types. They provide feedback to the students after the sessions. Videotaping of sessions allows students also to do self- assessment. Clinical Skills Assessment As defined at Northwestern, standardized patients are used in assessment, and therefore must maintain a constant demeanor and identical story from student to student. It requires lengthy training to get the standardized patients to present a case in the same way and to record evaluations consistently. In the first-year clinical skills examination, students receive an inventory of their strengths and weaknesses. Students must pass the second-year skills assessment or attend two required workshops before starting their clerkships. Clerkships Specific communication skills modules are inserted into clerkships. These are developed in collaboration with the clerkship directors and address more advanced communication challenges (e.g., delivering bad news in the surgery clerkship). This format works if the clerkship takes ownership of the modules. It is not well received if it is perceived that the modules are taking students away from the clerkship. Lessons Learned • Standardized patients need to be carefully screened and trained. When recruiting standardized patients, it is important to ensure that they do not have a personal agenda or “lecture” students about their opinions. • The use of standardized patients in assessment requires reliability of performance and scoring. Page 4
  5. 5. • There are many opportunities for further research, including psychometric studies and comparisons of global and checklist ratings. Dr. Hawkins The National Capital Area Medical Simulation Center celebrated its first year of operation in April of 2001. The Center consists of four units: • Clinical Skills Area – This includes 12 examination rooms with audiotaping and videotaping capability. The rooms are built around a central core that allows ongoing monitoring and control. • Computer Laboratory – There is a 16-station computer simulation laboratory, which has CD, DVD and web-based software programs. • Surgical Simulation Laboratory – There is an operating theater with two computer-driven anesthesia mannikins. The facility also includes a virtual reality suite, where procedures such as peritoneal lavage and laparoscopy can be practiced. • VTC/Distance Learning Facility – This allows distance learning capabilities. In the Simulation Center, standardized patients are used in four general areas: teaching, learner evaluation, program evaluation. and outcomes assessment. Standardized patients also are used in faculty development. Users include the school of medicine, graduate school of nursing, graduate medical education programs, and other institutions in the region on a space-available basis. There are a number of educational activities in the school of medicine that use Simulation Center resources. In the Introduction to Clinical Medicine (ICM) course students are taught interviewing, including experience with difficult communication tasks. The final examination for the physical diagnosis section of the course also is held at the Center. Finally, about one-half of the students take the final segment of ICM, which combines history and physical examination, at the Center. The ability to videotape these students is an asset, in that faculty members can review the tapes according to their own schedules and students can use the tapes for self- assessment. In general, the use of standardized patients allows control over clinical content and ensures that patients are available on schedule. As part of student evaluation in the ICM course, there is a six-station OSCE given at the Simulation Center. The Simulation Center is involved in four of the required clerkships and will add a fifth. For example, there was concern in the family medicine clerkship about the consistency of students’ clinical experiences across many remote sites. In response, the faculty arranged to bring students back to campus for a two-week block. On the first day, a multiple-station examination is administered to identify deficiencies and to serve as a needs assessment for the two-week period. A similar final examination at the end of the two-week block provides an evaluation of how well the educational needs of each student were met. Support for graduate medical education (GME) also is provided by the Simulation Center. For example, program directors at the Naval Medical Center asked for an examination for interns that would simulate conditions in remote environments. This would help assess whether the internship was preparing individuals to deliver care in such settings. To make the OSCE simulations close to reality, the physical medicine and rehabilitation GME program went to the community to recruit amputees with ill-fitting prostheses. There also is an assessment focusing on clinical skills for “problem” residents. Page 5
  6. 6. Faculty development involving the Simulation Center takes many forms. There is informal consultation, for example, with clerkship directors. A course is held annually for GME program directors. A “direct observation” workshop is offered to train faculty to be consistent in rating students’ clinical skills. For this session, standardized patients and standardized residents are trained to portray a specific level of performance. Within each project carried out by the Simulation Center, specific research questions have been identified. The Graduate School of Nursing is examining different types of ratings (checklist, global) and raters (self, peer, faculty, standardized patient). The Department of Pediatrics is teaching patient advocacy skills, putting students in a simulated managed care environment where they interact with a case manager and discharge planner, and then with occupational and physical therapy. At the end of the educational intervention, students present the simulated patient’s home health care needs to a simulated ward attending. Among other activities, the Simulation Center developed a scenario for a hospital ship to react to a disaster (a hurricane). This 4-6 hour mass casualty drill included standardized patients who presented with accompanying x-rays and other background data. The cost for construction, information technology, and furnishings for the Simulation Center was $4.5 million. The Center has an annual operating budget of $1.3 million. Standardized patient costs are $300,000 and the cost of standardized patient trainers is $125,000. During the first year of the Simulation Center’s existence, the focus has been on clinical skills teaching and assessment, directed at learner needs; on teaching communication skills; on faculty development; and on research. The Simulation Center with its standardized patients offers a safe environment to learn, for example, communication skills related to difficult or sensitive areas. For more information, consult the Simulation Center’s web site at Dr. Hallock Dr. Hallock described the use of standardized patients in a “high stakes” examination, the Educational Commission for Foreign Medical Graduates (ECFMG) Clinical Skills Assessment. The purpose of ECFMG certification is to assure residency program directors and the public that international medical graduates have met the minimum standards to enter residency training in the United States. ECFMG certification requires the primary source verification of a diploma and passage of Steps 1 and 2 of the United States Medical Licensing Examination, a test of English, and the Clinical Skills Assessment (CSA). The CSA was implemented in July 1998. Over 16,000 assessments have been carried out to date (about 6,000 per year). The CSA is offered only at the ECFMG facilities in Philadelphia. The purpose of the CSA is to ensure that International Medical Graduates from non-Liaison Committee on Medical Education (LCME) accredited medical schools demonstrate the ability to gather and interpret clinical patient data and communicate effectively at a level comparable to what would be expected of a graduate of an LCME accredited school. Page 6
  7. 7. The CSA is a performance-based examination using standardized patients in an OSCE-type process. Each examinee completes 10 scored cases. There also may be an 11th case that is being pilot-tested. For the ECFMG, a critical issue is training the standardized patients. Many of the standardized patient trainers are former standardized patients. The standardized patients are taught to portray cases consistently and to score accurately. The standardized patients do not assess clinical skills; they document clinical skills using a checklist. They do evaluate interpersonal skills and spoken English proficiency (ability to communicate). The standardized patients are trained and constantly monitored. Examinees also are videotaped. The cases are designed to simulate a high quality encounter in a physician’s office. Examinees receive information as they enter. They are expected to take a history and perform a focused physical examination, discuss their findings with the patient, and answer the patient’s questions. The examination is balanced by gender, age, and acuity for each examinee. There are no pediatric patients. The cases cover important and/or common conditions and are appropriate for an entry-level resident physician. There are number of characteristics that are required for a high-stakes examination: • Validity (does the examination measure what you want to measure) – The skills that are evaluated are those considered important for an individual beginning residency training (history-taking, physical examination, interpersonal skills, ability to communicate). The national ambulatory medical care survey and other national data sources are used to select relevant cases. Every year, the test blueprint is examined to ensure that it matches the common cases seen in physician offices. There is a multispecialty test committee of physicians who develop and review the cases. For each form of the test, there is attention to case mix, acuity, gender, age. All four major content areas are covered: chest, abdomen, neuropsychiatric, and constitutional. There also is a requirement that a minimum number of physical findings be present in each examination. • Fairness – Cases are tested before being used for scoring. The different forms of the test are equated for difficulty across cases and standardized patients. Because all testing is at one site, there are uniform testing conditions. • Security – There is a large “bank” of cases and 90 standardized patients. The forms of the test change so that the standardized patients do not know beforehand which cases they will portray. • Reliability (is a score a reasonable reflection of the examinee’s true ability) – Reliability is ensured through case development and eliminating extraneous sources of error (by standardized patient training and quality assurance measures). In about 5-10% of cases, a second standardized patient judges the case. Cases also are videotaped. All cases assess all skills, so performance across cases can be considered, • Feasibility – Such a high stakes examination can be done. Use of a single testing center allows the control of many variables. The great majority of the standardized patients (78 of 90) have been with the program since the beginning. Scheduling of examinees is handled by computer. Page 7
  8. 8. Pass rates to date are contained in the following table: ----------------------------------------------------------------------------------------------------------------- Candidate Group Percent Passing Segment of the Exam Overall Doctor/Patient Integrated (Total Communications Clinical Exam) Encounter All IMGs 80.4% 89.9% 86.5% USIMGs 87.1% 98.6% 87.9% Non-USIMGs 78.7% 87.6% 86.2% ------------------------------------------------------------------------------------------------------------------ Dr. Scoles There is consensus that cognitive and affective skills are in different domains, and that mastering one domain does not ensure mastery of the other. Communication and information gathering skills are essential for physicians. The Association of American Medical Colleges Medical School Objectives Project (MSOP) includes the ability to take an accurate history and to communicate effectively with patients, families, and colleagues among the outcomes that medical students should achieve. Data from a study related to MSOP by Makoul and colleagues found that 80% of medical schools use standardized patients for teaching communication skills, but that fewer used standardized patients in student evaluation. Even rarer is the use of standardized patients in summative (high stakes) evaluations. The approaches that medical schools use vary widely. About one-third of schools have no standardized patient programs for teaching and evaluation. In those schools with programs, what students learn about communication skills in the first and second years, through use of standardized patients, may be undermined in years three and four and later. It would not be necessary to have separate examinations to test for information gathering and communication skills, if the current exams that test for cognitive achievement also identified individuals deficient in those areas. Research indicates, however, that this is not the case. In the Medical Council of Canada examination, students must pass the cognitive portion (Part 1) of the examination before taking the clinical skills (Part 2) component. There still are 4-12% failure rates on the clinical skills (Part 2) examination. Projections based on the model standardized patient examination being developed by the National Board of Medical Examiners (NBME) suggest that 5-7% of U.S./Canadian graduates will fail and that these individuals will be a different cohort than fail Step 2 of the USMLE. The correlation between Step 2 and the model NBME standardized patient examination is modest. Correlation is highest in cases that test information gathering and lowest in cases that evaluate communication skills. The NBME began consideration of a clinical skills examination in 1975. The developmental work has been funded by a wide range of organizations, including the Kellogg, Macy, Robert Wood Johnson, and Commonwealth Fund foundations. The Pathways for Licensure Task Force Page 8
  9. 9. (1985) recommended that a test of clinical skills be incorporated into the examinations for licensure as soon as it was shown that the test would be valid, reliable, and practical. The NBME now thinks that these conditions can be met. The NBME is charged with responsibility to certify that candidates have adequate information to enter into supervised practice in postgraduate training and unsupervised practice after graduate medical education. Steps 1, 2, and 3 of the USMLE allow this assurance in the cognitive domain. The goal of the standardized patient examination would be to certify that the candidate possesses adequate information gathering and communication skills to enter into supervised graduate medical education. In this, it would be the counterpart of the Step 2 examination. The format of the NBME standardized patient examination will be similar to the Clinical Skills Assessment given by the ECFMG. Each examinee will complete 10 scored cases from a large pool of cases. Content equivalent forms of the examination will be utilized. Some of the cases will primarily test information gathering and other cases will focus on communication skills. The scored elements will include checklists and questionnaires generated by the standardized patients and post-encounter notes generated by the candidate. Each encounter with a standardized patient will last about 15 minutes, and then the candidate will have 10 minutes to generate the post-encounter note. The final scoring mechanism has not yet been set. Options for scoring include the following: (1) use a numerical score, since this examination will be part of the licensing examination sequence and some states require a numerical grade; (2) report the exam as pass-fail; (3) give a successful candidate certification and require that an unsuccessful candidate repeat the exam. The NBME considered several delivery models for the standardized patient examination, based on the balance between convenience/accessibility and reliability/standardization/fairness. The spectrum ran from using about 60 medical school sites that each delivered the exam for a window of time to using only a few test sites that delivered a highly standardized examination. The ultimate delivery mechanism probably will be a group of nationally-distributed, fixed examination sites that operate year round. The cost of the exam to students has not yet been set, but a general idea can be gained by adding the costs that will be incurred in exam development and implementation. There will be costs for such things as: • standardized patient salaries, • test center staff salaries • test center overhead, • test scoring, • candidate scheduling/registration, • score archiving/reporting, • NBME central costs. Including all these factors, the total cost per student would be similar to the cost of the ECFMG CSA. If approved by the NBME Executive Board, large-scale field-testing will begin. This will build toward delivery of the standardized patient examination beginning in the spring of 2004. Page 9
  10. 10. Therefore, the class of 2005 (which will enter in the fall of 2001) will be the first to take the standardized patient examination as part of USMLE. The ultimate failure rate in the current USMLE is 1-2% (about 300-600 students do not pass). The NBME spends about $23 million on these exams. It is anticipated that the standardized patient examination will cost $22 million per year, and will have an ultimate failure rate of 1-3%. There is consensus at the NBME that it is essential to certify skills as well as cognitive knowledge. NOTES: 1. Barrows H. An overview of the uses of standardized patients for teaching and evaluating clinical skills. Academic Medicine 1993;68(6):443-453. 2. Data from the 2000-2001 Liaison Committee on Medical Education (LCME) Annual Medical School Questionnaire are provided below. The questionnaire, which was sent to the deans of all 125 LCME-accredited medical schools, had a 100% response rate. Number of medical schools that used standardized patients in instruction of medical students in the following skills: - history-taking (106 schools) - doctor-patient communication skills (104) - general physical examination skills (93) - specialized physical exam skills, such as the gynecologic or urologic exam (114) Page 10