Introduction of Standard Setting in undergraduate medical education: re-emphasizing the importance of item-writing.<br />J...
Standard setting – preparation for pilot<br />Module selection<br />Nutrition & Energy<br />Junior Cycle Semester 1<br />n...
Standard setting – training of judges<br />59.2<br />60.7<br />Workshop<br />Dr Katherine Boursicot<br />November 2007<br ...
JC1 NE 2008: Modified Angoff<br />“Classic”<br />Angoff<br />MCQ cut score<br />50<br />58<br />MCQ Class mean<br />63.6<b...
JC1 NE 2008: Hofstee<br />57 %<br />Jane Holland, RCSI<br />
RCSI Curriculum Forum, 29th March 2008<br />Jane Holland, RCSI<br />
Item writing – training of writers<br />Workshop<br />Dave Swanson / Kathy Holtzman<br />April 2008<br />Guidelines<br />N...
Item writing – pilot within NE module<br />13 unusable<br />64 suitable!<br />7 -ve<br />4 T/F<br />2 misc<br />Call for q...
JC1 NE 2009: Modified Angoff<br />12 judges <br />30 MCQs<br />Cut-score = 48<br />Item difficulties<br />DIF<br />IDEAL C...
JC1 NE 2008:  DIF vs Prob<br />Prob<br />KR-20 = 0.735 <br />DIF<br />rs = 0.277 (p = 0.13)<br />Jane Holland, RCSI<br />
JC1 NE 2009:  DIF vs Prob<br />Prob<br />KR-20 = 0.769 <br />DIF<br />rs = 0.599 (p = 0.001)<br />Jane Holland, RCSI<br />
Summary of results <br />& take home message<br />2008<br />2009<br />MCQ cut score<br />58<br />48<br />MCQ Class mean<br...
Upcoming SlideShare
Loading in …5
×

Amee 2009 7 E5 JCH

611 views

Published on

Presented at AMEE 2009 - Item-Writing & Standard Setting

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
611
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Amee 2009 7 E5 JCH

  1. 1. Introduction of Standard Setting in undergraduate medical education: re-emphasizing the importance of item-writing.<br />Jane Holland, Alice McGarvey, <br />Richard Arnett, David Croke<br />Royal College of Surgeons in Ireland<br />
  2. 2. Standard setting – preparation for pilot<br />Module selection<br />Nutrition & Energy<br />Junior Cycle Semester 1<br />n = 281<br />19 - anatomy, biochemistry<br /> pharmacology, physiology<br />Lecturers / <br /> Judges<br />Examination<br />Multiple choice, n = 30<br />Method<br />Modified Angoff (1971)<br />Jane Holland, RCSI<br />
  3. 3. Standard setting – training of judges<br />59.2<br />60.7<br />Workshop<br />Dr Katherine Boursicot<br />November 2007<br />Principles & specific methods<br />Exercise<br />10 MCQs (JC1 NE 2007)<br />Jane Holland, RCSI<br />
  4. 4. JC1 NE 2008: Modified Angoff<br />“Classic”<br />Angoff<br />MCQ cut score<br />50<br />58<br />MCQ Class mean<br />63.6<br />54.6<br />Overall class mean<br />61.1<br />59<br />Fails<br />12<br />15.4<br />14 judges <br />30 MCQs<br />3 ½ hours!!<br />*All tabular values are percentages<br />Jane Holland, RCSI<br />
  5. 5. JC1 NE 2008: Hofstee<br />57 %<br />Jane Holland, RCSI<br />
  6. 6. RCSI Curriculum Forum, 29th March 2008<br />Jane Holland, RCSI<br />
  7. 7. Item writing – training of writers<br />Workshop<br />Dave Swanson / Kathy Holtzman<br />April 2008<br />Guidelines<br />NBME / MRCS<br />Jane Holland, RCSI<br />
  8. 8. Item writing – pilot within NE module<br />13 unusable<br />64 suitable!<br />7 -ve<br />4 T/F<br />2 misc<br />Call for questions<br />60 MCQs required<br />71 MCQs requested<br />Guidelines / SBA stems<br />77 MCQs received<br />83%<br />Jane Holland, RCSI<br />
  9. 9. JC1 NE 2009: Modified Angoff<br />12 judges <br />30 MCQs<br />Cut-score = 48<br />Item difficulties<br />DIF<br />IDEAL Consortium Item Analysis package<br />Item probabilities<br />Prob<br />Modified Angoff (1971)<br />Jane Holland, RCSI<br />
  10. 10. JC1 NE 2008: DIF vs Prob<br />Prob<br />KR-20 = 0.735 <br />DIF<br />rs = 0.277 (p = 0.13)<br />Jane Holland, RCSI<br />
  11. 11. JC1 NE 2009: DIF vs Prob<br />Prob<br />KR-20 = 0.769 <br />DIF<br />rs = 0.599 (p = 0.001)<br />Jane Holland, RCSI<br />
  12. 12. Summary of results <br />& take home message<br />2008<br />2009<br />MCQ cut score<br />58<br />48<br />MCQ Class mean<br />63.6<br />63.0<br />Item difficulties vs. <br /> item probabilities (rs)<br />0.277 <br />(p=0.13)<br />0.599 <br />(p=0.001)<br />A well-structured item, while designed to more accurately assess the candidate’s knowledge of the subject material, also enables more accurate rating of items during the standard setting process and is an important consideration when introducing these procedures de novo.<br />Jane Holland, RCSI<br />

×