INMED 2019 Sharing Reliable & Valid OSCE Stations
•Download as PPTX, PDF•
1 like•79 views
This document summarizes the results of an OSCE assessment of 195 medical students. It found that using an "entrustment question" of whether examiners would allow students to examine family members identified more students as failing (10%) than using global rating scores alone (1%). The authors conclude that assessments should use an entrustment question rather than just a global rating score to reduce "false positive pass scores." They also say the common cutoff of 50% for passing does not make sense and that assessments need near 100% certainty that a pass truly indicates competency.
Recommended
More Related Content
Similar to INMED 2019 Sharing Reliable & Valid OSCE Stations
Similar to INMED 2019 Sharing Reliable & Valid OSCE Stations (20)
More from Dr Thomas Kropmans
More from Dr Thomas Kropmans (9)
Recently uploaded
Recently uploaded (20)
INMED 2019 Sharing Reliable & Valid OSCE Stations
- 1. Sharing reliable and valid OSCE stations so as to improve cross-institutional assessment strategies: Are we equipped for it? Thomas JB Kropmans Senior lecturer MIME, CMN&HS
- 2. 11 EU Universities in 2017
- 3. Entrustment and beloved family in modular OSCE assessment ‘True’ decisions on Global Rating and Pass/Fail scores using ‘red flags’ Rosemary Geoghegan1; Andrew Murphy1; Jose Jaferson Monteiro Silva2 . Thomas Kropmans1,2; 1College of Medicine, Nursing & Health Sciences; School of Medicine, National University of Ireland Galway; 2 Qpercom Ltd Galway Ireland
- 4. Methods • Penultimate year 4 OSCE in 2018, 15 stations n=195 • Primary Care/Mental Health & Women Child Health • Blind scoring of • Check list item score • Global Rating Score • Fail, Borderline, Pass, Good and Excellent • Entrustment question: “would you allow this student to examine one of your beloved family members?” • G-coeff > 0.70 with an SEM = 4% (95% CI = 8%) • Cut-off = 50% (UK and Ireland)
- 5. OSCE Results Primary Care & Mental Health Women & Child Health Item score ‘Fail’ Global Rating ‘Fail’ Item score ‘Fail’ Global Rating ‘Fail’ 60 27 75 19 3% failed overall 1% failed overall 84 8 93 14 1% failed overall 0% failed overall
- 6. ‘Entrustment’ Results Primary Care & Mental Health Women & Child Health Item score ‘Fail’ Global Rating ‘Fail’ Item score ‘Fail’ Global Rating ‘Fail’ 144 35 132 168 19 145 4% failed overall 10% 1% failed overall 9% G-coeff > 0.70 with an SEM = 4% (95% CI = 8%) Cut-off = 50% (UK and Ireland) Total number of ‘educational decisions’ = 1358 PCMH/1544 WCH
- 7. Conclusion • Add ‘entrustment question’ to station rather than a ‘Global Rating Score’ • ‘If you are not allowed to examine my family, you should not examine anybody’ • Add ‘95% CI to SEM’ rather than ‘SEM’ to rule out ‘doubtful decisions’ e.g. ’false positives pass scores’ • ‘Be close to 100% sure that a Pass means a ‘real Pass’ and NOT a ‘false positive pass’ • Cut-score of 50% doesn’t make sense at all!!!
Editor's Notes
- During last year’s EBMA we compared Quality Assurance data of 12 European Universities all using our in house developed OSCE Management Information system that is currently used within 25 Universities and Professional Bodies worldwide. Purpose are we equipped YES are we ready of sharing stations NO!
- Default in our system is that examiners mark ‘blind’, they don’t need to (equivalent for ‘can’t) add on scores of ‘item score sheets’; the system is doing that for them. What they do need to do is at the end of each score sheet complete a ‘Global rating Score’ Fail, Borderline, Pass, Good or Excellent performance observed in that particular student. The biggest surprise, outcome last year was that the discrepancy between ‘students scoring below the cut-score’ and the ‘Global Rating Score’. Our University is not an exception 114 student failed on their ‘overall score’ in this penultimate OSCE whereas only 16 ‘Failed’ on the Global rating Score’.
- We can look over the hedge of our neighbors in Europe, as we did last year. However, checking whether their grass is greener than ours we first need to clean up our own backyard. We therefore added an entrustment question to our penultimate OSCE score sheets. Apart from an ‘item score sheet’, and a ‘Global Rating Score’ we asked the examiner ‘Would you allow this student to examine one of your beloved family members?’ YES or NO, in case of NO a ‘red flag’ would appear in downloaded results sheets.
- So, 4 OSCE to assess the clinical skills ‘learning outcome’ of two modules, 15 stations, 195 students and in previous years the G-coefficient of these OSCEs is > 0.70 with an SEM of 4 % (68% of the students has an ‘OBSERVED SCORE’ plus or minus 4%). The cut-off score in Ireland like in the UK is 50%! Remember 5 patients die and 5 survive and your still a good doctor!
- Similar to what we presented last year we again see the discrepancy between the ‘blinded scores on the item score sheets’ and the ‘Global Rating Scores’. You would expect Irish examiners to be harsh, Celtic warriors looking for the best outcome!! What we find are ‘doves’; ‘peace angles’ being very mild to their students. Now what if the family gets involved?
- This is where the real Irish examiner shows his through self! In ‘Primary care and Mental health 35 students ‘FAILED’ on the GRS whereas of 135 students they said ‘Don’t get near my family!’ and similar happened within the WCH module.
- Thank you very much, the good news is that our presentation of last year is now drafted in a publication where 8 Universities opened up and co-authored this paper. I am not sure what will happen in our own backyard because the 50% is an Irish thing and I have no plans joining Irish politics. May I turned into that Celtic warrior? A Dutchman in an Irish skin, I reckon!