Medical Teacher


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Medical Teacher

  1. 1. This article was downloaded by:[University of Maastricht] On: 26 January 2008 Access Details: [subscription number 780210644] Publisher: Informa Healthcare Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Medical Teacher Publication details, including instructions for authors and subscription information: Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training Alberto Alves de Lima a; Carlos Barrero a; Sergio Baratta a; Yanina Castillo Costa a ; Guillermo Bortman a; Justo Carabajales a; Diego Conde b; Amanda Galli a; Graciela Degrange a; Cees Van DER Vleuten c a Argentine Society of Cardiology Buenos Aires, Argentina b Instituto Cardiovascular de Buenos Aires, Education, Argentina c Educational Development and Research, Maastricht University, The Nederlands First Published on: 21 September 2007 To cite this Article: de Lima, Alberto Alves, Barrero, Carlos, Baratta, Sergio, Costa, Yanina Castillo, Bortman, Guillermo, Carabajales, Justo, Conde, Diego, Galli, Amanda, Degrange, Graciela and Van DER Vleuten, Cees (2007) 'Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training', Medical Teacher, 29:8, 785 - 790 To link to this article: DOI: 10.1080/01421590701352261 URL: PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.
  2. 2. 2007; 29: 785–790 Downloaded By: [University of Maastricht] At: 20:22 26 January 2008 Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training ALBERTO ALVES DE LIMA1, CARLOS BARRERO1, SERGIO BARATTA1, YANINA CASTILLO COSTA1, GUILLERMO BORTMAN1, JUSTO CARABAJALES1, DIEGO CONDE2, AMANDA GALLI1, GRACIELA DEGRANGE1 & CEES VAN DER VLEUTEN3 1 Argentine Society of Cardiology Buenos Aires, Argentina, 2Instituto Cardiovascular de Buenos Aires, Education, Argentina, 3 Educational Development and Research, Maastricht University, The Nederlands Abstract Aims: The purpose of the study was to determine the validity, reliability, feasibility and satisfaction of the Mini-CEX. Methods and Results: From May 2003 to December 2004, 108 residents from 17 cardiology residency programs in Buenos Aires were monitored by the educational board of the Argentine Society of Cardiology. Validity was evaluated by the instrument’s capability to discriminate between pre-existing levels of clinical seniority. For reliability, generalisability theory was used. Feasibility was defined by a minimum number of completed observations: 50% of the residents obtaining at least four Mini-CEX’s. Satisfaction was evaluated through a one to nine rating scale from the evaluators, and residents’ perspectives. The total number of encounters was 253. Regarding validity, Mini-CEX was able to discriminate significantly between residents of different seniority. Reliability analysis indicated that a minimum of ten evaluations are necessary to produce a minimally reliable inference, but more are preferable. Feasibility was poor: 15% of the residents were evaluated four or more times during the study period. High satisfaction ratings from evaluators’ and residents’ were achieved. Conclusion: Mini-CEX discriminates between pre-existing levels of seniority, requires considerable sampling to achieve sufficient reliability, and was not feasible within the current circumstances, but it was considered a valuable assessment tool as indicated by the evaluators’ and residents’ satisfaction ratings. Introduction Practice points The Mini-CEX has been designed to incorporate the skills that . The Mini-CEX has been designed to incorporate the residents require in both actual patient encounters and in the skills that residents require in both actual patient educational interactions that they routinely encounter with encounters and in the educational interactions that attending physicians during teaching rounds (Norcini et al. they routinely encounter with attending physicians 1995; Norcini et al. 1997; Holmboe et al. 1998; Norcini et al. during teaching rounds. 2003). A single faculty member observes and evaluates a . Regarding validity, Mini-CEX clearly was able to resident while that resident conducts a focused history and discriminate between pre-existing levels of global physical examination on an inpatient or outpatient, or in the competency between residents. emergency department setting. After asking the resident for a . Mini-CEX has insufficient reproducibility when only a diagnostic and treatment plan, the faculty member completes few evaluations are sampled. a short evaluation form and gives the resident feedback . Mini-CEX has satisfaction rates according to both (Holmboe 2004). It is a performance-based evaluation method evaluators and residents satisfaction rates. that is used to assess selected clinical competencies (e.g. . Regarding feasibility, it appeared not easy to achieve the patient charts and physical examination, also communication number of encounters required. and interpersonal skills) in the medical training context. It is a performance-based assessment tool that intended to evaluate candidates at the ‘does’ level, that is, in real life settings and not Assessment constitutes the most vital factor influencing in simulated situations (Miller 1990). As the interaction is student learning behavior (Newble et al. 1983; Newble et al. relatively brief and occurs as a natural part of the process in the 1990; Van der Vleuten 1996). When students see that the recall training environment, each individual can be evaluated on of factual information is a predominant requirement in the several occasions and by various faculty members. examination system, they tend to adopt a rote-learning or Correspondence: Dr A. Alves de Lima, Educational Department, Argentine Society of Cardiology, Azcuenaga 980 (C1115AAD), Buenos Aires, Argentina. Tel: 54 11 4961 6027; fax: 54 11 961 6029. Email: ISSN 0142–159X print/ISSN 1466–187X online/07/080785–6 ß 2007 Informa UK Ltd. 785 DOI: 10.1080/01421590701352261
  3. 3. A. Alves de Lima et al. surface approach; however if examiners wish to assess were estimated using the variance components: Dependability Downloaded By: [University of Maastricht] At: 20:22 26 January 2008 students at the ‘does’ level, they must evaluate the student’s Coefficients (D) and Standard Errors of Measurement (SEM), habitual performance in daily practice (van der Vleuten 2000). both as a function of the number of evaluations. The The major purpose of this study is to document if Mini-CEX D-coefficient can be interpreted as a reliability coefficient, applied in a broad range of clinical settings and in a big i.e. the expected correlation between other random evalua- number of cardiology residency programs leads to achieving tions of the number indicated using other evaluators and adequate levels of validity, reliability, feasibility and satisfac- patients at random. The SEM is an estimate of the standard tion rates from residents and teachers. error and can be used to estimate confidence intervals around the score of an individual resident on the original scoring scale. For a 95% confidence interval the SEM is multiplied by 1.96 Methods (z-score under which 95% of normal distribution lies). The For each Mini-CEX, a single faculty member observed and SEM should be below 0.26 (0.5/1.96) in order to produce a evaluated the resident while the latter conducts a history and reliable inference on the scoring scale of at least one unit. physical examination on an ‘in’ or outpatient or on a patient in Feasibility was defined, according to the American Board of the emergency department. After asking the resident for a Internal Medicine’s guidelines for Mini-CEX’s implementations, diagnosis and treatment plan, the faculty member completes a on average a minimum of four Mini-CEX per resident short evaluation form and gives direct feedback. All formal (American Board of Internal Medicine 2005). mini-CEX evaluation data was collected on a one-page form Satisfaction was evaluated through the examination of the that was the same in all of the different study sites where the Mini-CEX from the perspective of evaluators, emphasizing on study was carried out (Appendix 1). The form was previously the ratings of the residents and on their satisfaction of the translated and transculturally adapted into Spanish. format. Ratings were carried out on a nine-point rating scale. Research subjects were cardiology residents from 17 cardiology training programs from Buenos Aires, Argentina. All the programs are affiliated to the University of Buenos Aires Results and consist of a four-year training period. The total number of From May 2003 to December 2004, 253 Mini-CEX encounters residents of the entire program was 118. All the program were carried out. 108 residents and 53 evaluators from 17 directors were invited to participate. It was a completely new cardiology residency programs participated in the study. Each assessment strategy for all of them. Participation was resident had gone through 1 to 7 evaluations (mean: 2.34): voluntary, no incentives were provided. Written instructions 13.7% of the residents were in their first year, 34.8% in the about the application of the format were distributed and second, 41.2% in the third and 10.3% in their fourth year of required at least four encounters during the 19 months study residency. Each evaluator conducted between 1 and 21 (mean period. There were no sanctions for failing to participate, but 4.77) evaluations. The total numbers of encounters were all of them accepted to participate. The assessment was used 253 and constituted the basis of the analysis. Of the 253 as a maximum-performance but formative evaluation. Results encounters, 52% of the encounters occurred in the coronary were not used in evaluating residents for promotion. care unit setting, 30% were carried out in step-down care unit, 6% in the emergency room, 6% in the ambulatory care unit Statistical analysis and 6% in the cardiovascular intensive care unit. The overall competence ratings were similar in all settings. Forty-one Validity was evaluated by the ability of Mini-CEX to percent of the encounters represented the first visit of a patient discriminate between pre-existing levels of expertise. In this to a particular resident and 59% were return visits to that case it was expected that significant mean resident perfor- resident. The means total Mini-CEX time was 42.77 minutes mance differences are to be found between different years of (SD 19.97). If we divide the Mini-CEXs total time between training. The descriptive data were expressed as its means and assessment and feedback period, each individual assessment standard deviations. For testing significance across expertise period takes 25.80 (SD 11.95, range 5–65) minutes and groups the non-parametric Mann Whitney test was used. feedback period 17.31 (SD 11.28, range 5–65). The patient’s A value of p 5 0.05 was considered statistically significant. problems or diagnoses were specified by the evaluator and To evaluate reliability, generalizability theory was used covered a broad range of problems in cardiology such as AMI, (Brennan 2001). An ANOVA was carried out by identifying cardiac failure, unstable angina, atrial fibrillation, valvular Year-of-training with (Y), Residents-within-Year with (P:Y) and disease and post-CABG. The mean ratings given by the 53 Evaluations-within-Residents-within-Year with (E:P:Y), and evaluators are reported in Table 1. variance components were estimated using the URGenova program. Since there might be significant growth in mean ratings throughout the years, variance associated with year was Validity analysis estimated separately (Y) to arrive at a more unbiased estimate of the variance of trainees. Separate evaluations of a single Validity was evaluated by examining if the instrument was trainee could either be done by the same examiner or by a capable of discriminating between pre-existing levels of different one. This might have led to an underestimate of the clinical seniority. Mini-CEX discriminates between pre-existing variance across evaluations (intra-rater variability is probably levels of global competency between residents; first year smaller than inter-rater variability). Two indices of reliability residents 7.19 (SD 0.74), second 7.51 (SD 0.82), third 7.76 786
  4. 4. Validity, reliability, feasibility and satisfaction of the Mini-CEX for cardiology residency training Table 1. Mean Ratings given by all evaluators by year of training. Downloaded By: [University of Maastricht] At: 20:22 26 January 2008 Domain 1 year 2 year 3 year 4 year P Communication 7.16 ( Æ 0.64) 7.57 ( Æ 0.81) 7.57 ( Æ 0.92) 8.00 ( Æ 0.88) 0.002 Physical exam 7.12 ( Æ 0.84) 7.48 ( Æ 0.93) 7.59 ( Æ 0.96) 8.16 ( Æ 0.91) 0.0006 Professionalism 7.64 ( Æ 0.75) 7.82( Æ 0.84) 7.83 ( Æ 1.00) 8.20 ( Æ 0.93) 0.079 Clinical judgement 7.43 ( Æ 0.71) 7.56 ( Æ 0.86) 7.88 ( Æ 0.90) 8.20 ( Æ 0.93) 0.0004 Counselling 7.43 ( Æ 0.77) 7.44 ( Æ 0.84) 7.59 ( Æ 1.11) 8.12 ( Æ 0.90) 0.01 Organisation 7.32 ( Æ 0.94) 7.54 ( Æ 0.88) 7.68 ( Æ 1.01) 8.12 ( Æ 0.90) 0.008 Global Competency 7.19 (Æ 0.74) 7.51 ( Æ 0.82) 7.76 ( Æ 0.86) 8.16 ( Æ 0.91) 0.0008 Table 2. G-coefficients and SEM’s are reported as a function of Discussion the sample size of evaluations. The purpose of the study was to report logistic and Number of evaluations G SEM psychometric data for the Mini-CEX format. Regarding validity, 1 0.07 0.78 Mini-CEX clearly was able to discriminate between pre-existing 2 0.14 0.55 levels of global competency between residents, has insufficient reproducibility when only a few evaluations are sampled and 5 0.28 0.35 had high satisfaction rates according to both evaluators and 10 0.44 0.25 residents satisfaction rates. Regarding feasibility, it appeared 15 0.54 0.20 not possible to achieve the number of encounters required. 30 0.70 0.14 Some issues may explain this difficulty. It was a new assessment tool never applied earlier in this environment. 50 0.80 0.11 We only developed written instructions, and perhaps in vivo G: generalizability-coefficient, SEM: standard error of measurement. faculties training programs for Mini-CEX would be preferable. Regarding reliability, the variance components for Y, P:Y and E:P:Y one-fifth of the variance can be attributed to growth in competence throughout the years. Provided that the Mini-CEX (SD 0.86) and fourth year residents 8.16 (SD 0.91), this should offer information on growth towards a final level of difference reaches statistical significance. P ¼ 0.0008 (Table 1). competence and should be able to discriminate throughout years of training. Only approximately 6% of total variance is related to resident (or person) variance. Since the instrument is Reliability analysis designed to discriminate among residents, this constitutes The generalizability study theory yielded variance components desirable (true score or universe score) variance and the larger for Y, P:Y and E:P:Y which were respectively 0.1643 (19.88% it is the better. As usual in many competence and performance of total variance), 00482 (5.83%) and 0.614 (74.29%). Using the assessments, however, this component is relatively small SEM benchmark of 0.26, a minimum of 10 evaluations were (van der Vleuten 2005). necessary to produce a minimal reliable inference (Table 2). About three-quarters of variance is associated with differ- This corresponds to a D-coefficient of 0.44. ences between examiners/evaluation occasions and residual error. Using the SEM benchmark of 0.26, a minimum of ten evaluations are necessary to produce a minimal reliable Feasibility analysis inference. The D coefficient for ten observations was, however, rather low (0.44). Reliability coefficients of 0.80 or For feasibility analysis the data showed that only 14.81% of all higher are generally accepted as a threshold for high-stakes the cohort was evaluated four or more times during the study judgements, such as the registration of a doctor for licensure period. (Crossley et al. 2002). In our dataset this was achieved with 50 observations. The required number of encounters as derived from this Satisfaction analysis study is more demanding than reported by other studies in the The residents (108) were generally satisfied with the mini-CEX literature. Norcini et al. (2003) concluded that ten or more format; their ratings ranged from 5 to 9 (mean 8.08 Æ 0.83). encounters produced relatively tight confidential intervals and Satisfaction rate for first-year residents was of 8.1, second-year that an increase in the number of encounters beyond that residents 7.8, third-year residents 8.1 and fourth-year residents produced only small gains in consistency. Carline et al. (1992) 8.5. The evaluators were also satisfied with the mini-CEX concluded that seven independent ratings would be necessary format; their ratings ranged from 6 to 9 (mean 8.06 Æ 0.74). to judge overall clinical performance. Similar results have been 787
  5. 5. A. Alves de Lima et al. reported by Kreiter and colleagues (more than eight ratings dispersed in time and used for formative purposes (Alves de Downloaded By: [University of Maastricht] At: 20:22 26 January 2008 needed), Kwolek (7–8 ratings), Ramsey (11 ratings), Violato Lima et al. 2005). (10 ratings) and Kroboth (6–10 ratings) (Kroboth et al. 1992; Ramsey et al. 1993; Kwolek et al. 1997; Violato et al. 1997; Kreiter et al. 1998; Durning et al. 2002). All reports agree that Conclusion somehow between seven and 11 ratings are necessary to The direct observation of residents’ behaviour is essential to achieve a generalizable global estimate of competence when assess clinical skills (Holmboe et al. 2004a; Holmboe et al. ratings are based on a non-systematic sample of observations 2004b; Schuwirth & Van der Vleuten 2006). For decades, (Williams et al. 2003). Although there has been only limited clinical supervisors have taken at face value the veracity of the research on the component skills included under the broad history and physical examination presented on inpatient and category of clinical competence, it is reasonable to expect that outpatient rounds without ever directly observing how the these abilities will develop at different rates and may differ in trainee actually performed them (Holmboe 2004). their stability in different situations. Results of performed work Medical educators’ major challenge lying ahead is to find suggest that different numbers of observations will be required the way to ensure that they themselves have not only strong to establish stable estimates of clinical competence in various clinical skills, but also the necessary skills to effectively clinical competence areas (Williams et al. 2003). observe, evaluate, and provide constructive feedback to We have not analysed inter-rater reliability since we took a trainees regarding clinical skills. In this direction Mini-CEX time sampling perspective in which each observation is but ensures direct observation and feedback of residents from one observation in a longer time-framework. We analysed different faculties in a broad range of patients’ problems in reliability across these observations. Rater variance as well as various settings. Furthermore, as this demonstrated feasibility performance variability variance across observations will be is an issue. Application strategies should be reinforced. We part of our reliability estimation (although we can’t partition don’t think that Mini-CEX requires any modification in itself but out the two sources of variance). If only one observation is it should never be used as a unique assessment tool. Direct used for a Mini-CEX assessment it would be important to judge observation of trainees in clinical setting can be connected to the inter-rater reliability. Our data, unfortunately, do not other exercises that trainees may perform after their encoun- provide this information. ters with patients, such as oral case presentation, written Consistent with previous works, examiners were satisfied exercises that assess the clinical reasoning, and literature with the format (Williams et al. 2003). searches. In addition, review videos of encounters with patients offer a powerful means of evaluating and providing feedback on trainees’ skills in clinical interaction (Epstein Limitations of the study 2007). It can only be approved if this evaluation method The number of residents participating in our study was becomes part of the clinical routine of the clinician and relatively low and this group may perhaps not be fully resident. representative of broader populations. We are aware that improvement of performance across years of training is a weak form of construct validity. However, Notes on contributors it is a fundamental one: absence of performance differences ALBERTO ALVES DE LIMA, MD, MHPE is a cardiologist, Head of the across different expertise groups would be detrimental to Education Department at the Instituto Cardiovascular de Buenos Aires, construct validity of the instrument used. This not being the Buenos Aires, Argentina and a member of the educational department of the Argentine Society of cardiology. case, we conclude to a first indication of validity. Further CARLOS BARRERO, MD is a cardiologist and a member of the educational studies into construct validity should be the next step. Studies department of the Argentine Society of cardiology. looking at the incremental validity over existing more SERGIO BARATTA, MD is a cardiologist and a member of the educational standardized performance instruments would provide compel- department of the Argentine Society of cardiology. ling construct validity evidence. YANINA CASTILLO COSTA, MD is a cardiologist and a member of the The assessment used was a maximum-performance but educational department of the Argentine Society of Cardiology. formative evaluation. If we take into account that this GUILLERMO BORTMAN, MD is a cardiologist and a member of the assessment does not assign grades or certifications, this educational department of the Argentine Society of Cardiology. could seriously have affected the perception of the fellows JUSTO CARABAJALES, MD is a cardiologist and a member of the and influence their satisfaction rates. educational department of the Argentine Society of Cardiology. The reliability analysis carried out here used the usual DIEGO CONDE, MD is a cardiologist and a member of the Education assumption of local independence between the repeated Department at the Instituto Cardiovascular de Buenos Aires, Buenos Aires, Argentina. measurement moments (one measurement is not influenced AMANDA GALLI is educational Psychologist and a member of the by another measurement). This assumption is clearly violated Education Department of the Argentine Society of cardiology. in Mini-CEX studies including ours. Every evaluation is actually GRACIELA DEGRANGE, MD is a cardiologist and a member of the meant to provide feedback and to change the performance of educational department of the Argentine Society of cardiology. the person being assessed. The Mini-CEX evaluations are CEES VAN DER VLEUTEN, Professor, is Psychologist and Chairperson of therefore not independent of each other. This is a general the Department of Educational Development and Research, Maastricht problem in the literature of performance measure which are University, The Netherlands. 788
  6. 6. Validity, reliability, feasibility and satisfaction of the Mini-CEX for cardiology residency training consistency of a clinical evaluation exercise. J Gen Intern Med References 7:174–179. Downloaded By: [University of Maastricht] At: 20:22 26 January 2008 Alves de Lima A, Henquin R, Thierer J, Paulin J, Lamari S, Belcastro F, Kwolek CJ, Donnelly MB, Sloan DA, Birrell SN, Strodel WE, Schwartz RW. van der Vleuten CP. 2005. A qualitative study of impact on learning 1997. Ward evaluations: should they be abandoned? J Surg Res 69:1–6. of the mini clinical evaluation exercise in postgraduate training. Miller GE. 1990. The assessment of clinical skills/competence/performance. Med Teach 27:46–52. Acad Med 65:S63–67. American Board of Internal Medicine. The Mini-CEX: A quality tool Newble DI, Jaeger K. 1983. The effect of assessments and examinations on in evaluation. (24 Oct the learning of medical students. Med Educ 17:165–171. 2005). Newble DI, Hejka EJ, Whelan G. 1990. The approaches to learning of Brennan RL. 2001. Generalizability Theory (New York, Springer-Verlag). specialist physicians. Med Educ 24:101–119. Carline JD, Paauw DS, Thiede KW, Ramsey PG. 1992. Factors affecting the Norcini JJ, Blank LL, Arnold GK, Kimball HR. 1995. The mini-CEX (clinical reliability of ratings of students’ clinical skills in a medicine clerkship. evaluation exercise): a preliminary investigation. Ann Intern Med J Gen Intern Med 7:506–510. 123:795–799. Crossley J, Davies H, Humphris G, Jolly B. 2002. Generalisability: a key to Norcini JJ, Blank LL, Arnold GK, Kimball HR. 1997. Examiner differences in unlock professional assessment. Med Educ 36:972–978. the mini-CEX. Adv Health Sci Educ Theory Pract 2:27–33. Durning SJ, Cation LJ, Markert RJ, Pangaro LN. 2002. Assessing the Norcini JJ, Blank LL, Duffy FD, Fortna GS. 2003. The mini-CEX: a method reliability and validity of the mini-clinical evaluation exercise for for assessing clinical skills. Ann Intern Med 138:476–481. internal medicine residency training. Acad Med 77:900–904. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. 1993. Epstein R. 2007. Asessment in medical education. N Eng J Med Use of peer ratings to evaluate physician performance. JAMA 356:387–396. 269:1655–1660. Holmboe ES, Hawkins RE. 1998. Methods for evaluating the clinical Schuwirth LW, Van der Vleuten CP. 2006. A plea for new psychometric competence of residents in internal medicine: a review. Ann Intern Med models in educational assessment. Med Educ 40:296–300. 129:42–48. Van der Vleuten C. 1996. The assessment of professional competence: Holmboe ES, Hawkins RE, Huot SJ. 2004. Effects of training in direct developments, research and practical implications. Adv Health Sci Educ observation of medical residents’ clinical competence: a randomized Theory Pract 1:41–67. trial. Ann Intern Med 140:874–881. Van der Vleuten C. 2000. Validity of final examinations in undergraduate Holmboe ES. 2004. Faculty and the observation of trainees’ clinical skills: medical training. BMJ 321:1217–1219. problems and opportunities. Acad Med 79:16–22. Van der Vleuten C, Schuwirth L. 2005. Assessment of professional Holmboe ES, Yepes M, Williams F, Huot SJ. 2004. Feedback and the mini competence: from methods to programmes. Med Educ 39:309–317. clinical evaluation exercise. J Gen Intern Med 19:558–561. Violato C, Marini A, Toews J, Lockyer J, Fidler H. 1997. Feasibility Kreiter CD, Ferguson K, Lee WC, Brennan RL, Densen P. 1998. A and psychometric properties of using peers, consulting physicians, generalizability study of a new standardized rating form used to co-workers, and patients to assess physicians. Acad Med evaluate students’ clinical clerkship performances. Acad Med 72:S82–84. 73:1294–1298. Williams RG, Klamen DA, McGaghie WC. 2003. Cognitive, social and Kroboth FJ, Hanusa BH, Parker S, Coulehan JL, Kapoor WN, Brown FH, environmental sources of bias in clinical performance ratings. Teach Karpf M, Levey GS. 1992. The inter-rater reliability and internal Learn Med 15:270–292. 789
  7. 7. Downloaded By: [University of Maastricht] At: 20:22 26 January 2008 790 A. Alves de Lima et al. APPENDIX 1: The Mini-CEX form