2. Grade Transcription: Outline Introduction Student Quality of Life and Stress Subjectivity of Clerkship and its Relation to HPF Effect of Transcription on Post Grad Selection Honors and Patient Safety
3. Current events and options The Faculty of Medicine is engaged in a review of the way grades will be transcribed. Two options exist: CURRENT SYSTEM (Honours / Pass / Fail) REFORMED SYSTEM ( Credit / No Credit)
4. Why consider change? Focus groups and informal surveys have shown a review of the grading system is strongly favoured (76% clerks, 84% pre-clerks). The Faculty Council has invited students to voice their opinion on this matter.
5. What are the referendum questions? Q. What are the referendum questions? Are you adequately informed to make a choice? Yes/No Are you in favour of replacing the current form of grade transcription (Honours, Pass, Fail) with Credit/No credit for all four years of your medical education? Yes/No Requirements for a valid referendum result: Three most important numbers: Minimum class response rate: 80% for years I ⅈ 60% for year III &IV “Well informed” ballots: If > 20% of any class do not feel adequately informed the referendum will be postponed
6. Applicants to the faculty of medicine are encouraged to demonstrate well roundedness inside and outside the classroom. Medical education can be inherently stressful Massive amounts of material Faced with new situations only a minority of the population experiences CaRMS match anxiety A very high level of student performance is expected. Grade transcription and student life
7. Is energy on exams causing a decrease in pursuit of other valuable activities? Is balanced learning being sacrificed to achieve a numerical benchmark? In many courses it is considered an advantage to have strong background knowledge. Is the enthusiasm to pursue other academic endeavours which will allow one to excel during clerkship being stifled? Eg. Research etc. Grade transcription and student life
8.
9. Credit /No Credit = Less StressFallacy? Using student questionnaires, Robins et al. found that a P/F system does reduce anxiety and inter-student competition. There is no accepted consensus on the relationship between P/F and stress load. Studies show that under a P/F system the average performance on exams does decrease. Does this correspond to a loss in future competence?
11. H / P / F and Clerkship Past polling suggest that 76% of clerks support a credit / no credit system.
12. Preclerkship vs Clerkship Preclerkship 38% students are 3 marks above and below the honours cutoff owing to the 80% average and the 6% standard deviation of preclerkship classes. This raised the issue of whether the difference between the two extremes of 77% and 83% or even between 79% and 80% was as great as is reflected on the transcript. The same situation exists in Clerkship courses as last year’s overall third year and fourth year averages were 80.15% and 79.74% respectively. This distribution makes the subjectivity of grading that much more important as a change of only a few marks can make the difference between an honours and a pass for many students.
13. Clinical Assessments Evaluation is often subjective and not based on written exams Yet during clerkship the weighting of subjective evaluation varies from 60-10% of your final grade. EXAMPLES OF CLERKSHIP GRADING BREAKDOWN: Ophthalmology Rotation Clinical Assessment = 10% Written Exam = 65% Clinical Skills Exam = 25% Ambulatory Community Experience Rotation Clinical Assessment = 60% Academic Project = 40%
14. Clerkship grading and grade averages Clerkship clinical skills are evaluated on a 5 point scale. 3 / 5 = meet expectations Average final clerkship grade was 80%. Approx. 40% of this grade was awarded subjectively. The majority are just above or just below an honours grade. Possible exaggerated difference between students with H / P / F system
16. Graduation Questionnaire During each of the past six years, many Toronto medical graduates have identified a variety of weaknesses associated with the systems and processes employed to evaluate students, particularly during the clinical clerkship. “Wide variation in clinical evaluations between supervisors (clinical evaluation is useless and is more dependent on the supervisor than the trainee)” [2007] “Too much emphasis is placed on evaluation which is very subjective. More emphasis on learning less on evaluation would be good” [2005] “Grading system in clerkship is poor. Clinical evaluation really only depends on luck, i.e. what sort of preceptor you get and your clinical site” [2004]
17. Guiding Principles for Medical Education Some students question whether the feedback they receive is really helping them to reflect on their learning or performance.... Students in clerkship were concerned with inconsistencies in evaluator’s interpretation of the HPF system and blanket policies of giving students all 3/5s versus all 5/5s regardless of their performance. “If a person likes you, [for example in], anesthesia, I don't know anything about anesthesia, I haven't read a thing, I got all 5's because I got along with the person, whereas the next time I got all 2's... well, not 2 but 3 because he didn't like me or something. It's just not really an accurate reflection of what I'm doing.” (Fourth Year Medical Student) “A 3 is a meets expectations, which means that you are at the level of all of your peers, which seemingly should be a good thing, right? So there are doctors who say…"Yeah, you're meeting expectations, great. You're going to get all 3's." But if you ask someone on…the Residency Selection Committee, a 3 is like you're just passing…you're almost failing…just things like that where a 3 really doesn't mean a 3 it means 2… and some people only give 3's because they don't think anyone's above the rest of their peers.” (Fourth Year Medical Student)
19. Effect of Transcription on Post Grad Selection “ 38% of Canada’s program directors preferred a numerical grading system, 28% a letter grading system, 25% a H/P/F system, and only 9 % a P/F system. Provan JL, Cuttress L. Preferences of program directors for evaluation of candidates for postgraduate training. CMAJ 1995 Oct 1;153(7):919-23. ” Survey of directors of general surgery residency programs: found that 81% believed that the evaluation system …affects students’ abilities to obtain the residency position of their choice A P/F system is detrimental to applicants because it makes it harder to identify the students who performed best in the Pass category. Dietrick JA, Weaver MT, Merrick HW. Pass/fail grading: a disadvantage for students applying for residency. Am J Surg. 1991;162:63-66 20
20. Effect of Transcription on Post Grad Selection Graduates from a medical school with a two-interval, pass/fail system successfully matched with strong, highly-sought-after postgraduate training programs, performed in a satisfactory to superior manner, and compared favorably with their peer group. Vosti KL, Jacobs CD. Outcome measurement in postgraduate year one of graduates from a medical school with a pass/fail grading system. Acad Med. 1999;74:547-549. 21 “ ”
21. Effect of Transcription on Post Grad Selection The Faculty of Medicine Honour/Pass/Fail Grading Policy Taskforce of 2006 : Postgraduate directors at the University of Toronto prefer as much information as possible but are prepared to make selections without such information. Currently Honours standing is consistently seen as advantageous. The weighting of Honours standing/grades in general varies among programs. “ ” 22
22. Effect of Transcription on Post Grad Selection During the early 1990’s: During the time of the Faculty Taskforce 23
23. Effect of Transcription on Post Grad Selection Now: Is a Pass from University of Toronto seen as weaker than a Pass at another school? 24
24. Effect of Transcription on Post Grad Selection Given the fact that a majority of students will not achieve honours in all of their courses, one must therefore postulate that the existence of honours, by cheapening the value of a University of Toronto Pass, could allow most student’s transcripts to appear blemished in a way not experienced by students at other universities. 25 Pass = failure? “ ”
25. 26 Honors and Patient Safety Juvenile Rheumatoid Arthritis (JRA)
26. Honors and Patient Safety 27 “ ” It has been reported that students perform worse on examinations where a P/F system is used. Suddick DE, Kelly RE. Effects of transition from pass⁄no credit to traditional letter grade system. J Exp Educ 1981;50:88–90. “ ” After two schools switched to a P/F system, their students’ national licensing exam scores decreased. Moss TJ, Deland EC, Maloney JV Jr. Selection of medical students for graduate training: pass⁄fail versus grades. N Engl J Med 1978;299:25–7.
27. Honors and Patient Safety 28 “ Changing from a letter to a pass-fail grading system in the first-year anatomy course was favored by students and had no effect on the National Board of Medical Examiners subject examination or final cumulative grades. Jones KG, Pedersen RL, Carmichael SW, Pawlina W. Effects of pass/fail grading system on academic performance of first year medical students in gross anatomy course [abstract]. FASEB J. 2003;17:A385. Abstract 278.18. ” “ ” Longitudinal studies found no significant relationship between grade point average in medical school and subsequent performance in medical practice. Taylor CW, Albo D Jr. Measuring and predicting the performances of practicing physicians: an overview of two decades of research at the University of Utah. Acad Med. 1993;68(2, suppl):S65-S67.
29. Honors and Patient Safety 30 “ Society has the right to know that physicians who graduate from medical school and subsequent residency training programs are competent and can practice their profession in a compassionate and skillful manner. It is the responsibility of the medical school to demonstrate that such competence has been achieved and the responsibility of the accreditation agencies to certify that the educational programs in medical schools can do what they promise. Assessment is of fundamental importance because it is central to public accountability. ” Shumway JM and Harden RM. The Assessment of Learning Outcomes for the Competent and Reflective Physician. Medical Teacher25(6): 569-584. 2003.
30. Honors and Patient Safety A non-competency based education process might list an objective under interpersonal/communication skills as: “Student will understand how to effectively involve patients in decision making where appropriate.” A competency-based education process would write that objective as: “Student effectively involves patients in decision making where appropriate.” Focus on knowledge application rather than just the gaining of knowledge 31
31. Honors and Patient Safety 32 A two interval system such a C/NC is seen by many as the most appropriate way to acknowledge if a competency has been achieved. Any honors distinction would be seen as arbitrary . Litzelman DK., Cottingham AH. “The new formal competency-based curriculum and informal curriculum at Indiana University School of Medicine: overview and five-year analysis.”Acad Med. 82.4 (2007)
33. Your Questions Q. “So if the referendum "passes"...how many years will it be before any changes are made?ie....will it affect us, the 1T0's in any way at all?” A. Earliest Fall 2009, likely require second referendum Q. “I got a 70% on the first test. Someone told me that this shift would involve bumping the pass up to 70? Would residency directors see if you had to remediate?” A. Pass would likely increase from the current 60%. But keep in mind currently a pass it unofficially 70% at course director and Board of Examiners discretion. No residency directors would not have access to information on remediation. 34
34. Your Questions Q. “Will Honours allow me to crush the competition in residency?” A. Debatable. But with everyone in Ontario moving to C/NC its hard to perceive an advantage . Q. “Will they go back and change any marks retroactively?” A. No. At the time of the change, all four years will be transcripted C/NC. 35
35. Your Questions Q. “How will this change be disseminated to the schools that are expecting UofT students with Honours marks?” A. A formal letter by the Faculty will be sent to each residency program to inform them of the change. 36
