Upcoming SlideShare
×

# MCQ Workshop - Dr Jane Holland

3,516 views

Published on

MCQ Workshop - Dr Jane Holland RCSI

Presented at Moodlemoot Edinburgh 2014
www.moodlemoot.ie/

10 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
3,516
On SlideShare
0
From Embeds
0
Number of Embeds
498
Actions
Shares
0
101
0
Likes
10
Embeds 0
No embeds

No notes for slide

### MCQ Workshop - Dr Jane Holland

1. 1. Multiple Choice Questions - Item writing & item analysis 15th April 2014 Jane Holland MD PhD MRCSI RCSI Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn including slides from Richard Arnett, PhD
2. 2. MCQ Item writing & item analysis - Outline  Introduction to item-writing  Response formats  Stimulus formats  Technical item flaws  Introduction to item analysis Jane Holland http://www.nbme.org/publications/item-writing-manual.html
3. 3. MCQ response formats • True / False  The candidate is required to make a determination as to whether each option provided is true or false • One Best Answer  The candidate is required to indicate a single response i.e. the most likely diagnosis NBME Manual p 13 – 18 Schuwirth: Med Educ. 2004 Sep;38(9):974-9. Jane Holland
4. 4. True / false vs. Single Best Answer False True  (Simple true/false items)  (Complex true false items) Q. True statements about dogs include: A. They were first domesticated in New Zealand. B. 35% of households own at least one dog. C. They have a field of vision up to 200°. D. They communicate by barking. Jane Holland
5. 5. True / false vs. Single Best Answer Q. Which Irish rugby union player holds the record for most international caps? A. D’Arcy B. Heaslip C. Kearney D. O’Connell E. O’Driscoll Jane Holland 
6. 6. Q. Which of the following statements about Ireland is true? A. Tayto exports exceed € 1 Bn B. The climate is sub-tropical C. The main export is zinc D. The population is 5 million E. Rainy weather frequently occursX True / false vs. Single Best Answer Trade / exports Weather Demographics D C B A E Jane Holland
7. 7.  True / false vs. Single Best Answer Q. What colour is the sky? A. A hazy shade of winter B. Blue C. Grey D. Orange E. Red Jane Holland
8. 8. Q. Which of the following calls is a rugby union referee most likely to make in the case of an intercepted knock-on? A. Advantage B. Line out C. Penalty kick D. Penalty try E. Scrum Q. Which of the following was an original recording by Meatloaf? A. Bat out of Hell B. One out of three ain’t bad C. Paradise by the Kitchen Light D. Stand by me E. Total Eclipse of the Heart Jane Holland The “cover test”
9. 9. MCQs – how many options? Jane Holland Swanson et al; Academic Medicine. 2005;80(10):S93-S6. Tarrant & Ware; Nurse Education Today. 2010;30(6):539-43.
10. 10. Stem / vignette Question (Lead-in) Options Single Best Answer Jane Holland
11. 11. MCQ stimulus formats  Context -free  Tests factual knowledge  Context-rich  Tests reasoning or application Schuwirth: Med Educ. 2004 Sep;38(9):974-9. Kennedy, 2007 Jane Holland Creating Evaluating Analysing Applying Understanding Remembering
12. 12. NBME Manual p 35 - 49 Q. Which dermatome is at the umbilicus? A. T6 B. T8 C. T10 D. T12 E. L2 Q. A 65-year-old librarian presents with skin lesions suggestive of a Herpes zoster infection affecting the dermatome which includes the skin of the umbilicus.  Which segmental spinal nerve is associated with that dermatome? Jane Holland
13. 13. Q. A 6 year old boy is admitted with a history of colicky abdominal pain which has now localised to his right iliac fossa.  The initial, visceral pain felt with appendicitis is referred along sympathetic fibres from which level of the spinal cord? A. T6 B. T8 C. T10 D. T12 E. L2 NBME Manual p 35 - 49 Jane Holland
14. 14. Stem Lead-in Options  Clinical (or contextual) vignette • Age / gender • Presenting Problem / history • Examination findings / test results  Which of the following is the most: • Likely diagnosis • Appropriate treatment  Homogenous • One correct & 4 distractors • Alphabetical (or logical) order • All plausible & same length (short!) NBME Manual p 35 - 49 Jane Holland
15. 15. Q. Which of the following Moodle activities allows participants to select a single option from a list? A. Checklist B. Choice C. Forum D. Survey E. Workshop Jane Holland
16. 16. Q. Gavin is running a conference, at which there will be a number of parallel workshops with limited places. Which of the following Moodle activities could he use to allow participants to preselect their preferred workshop? A. Checklist B. Choice C. Forum D. Survey E. Workshop Jane Holland
17. 17. Q. Which of the following Moodle activities allows participants to select one choice from a list of options? Word repeats Jane Holland A. Checklist B. Choice C. Forum D. Survey E. Workshop
18. 18. Grammatical clues Long correct answer Absolutes Word repeats Logical cues Convergence Technical Item flaws NBME Manual p 19 - 27 Jane Holland
19. 19. Q. Which of the following Moodle activities allows participants to select one choice from a list of options? Word repeats Jane Holland A. Checklist B. Choice C. Forum D. Survey E. Workshop
20. 20. Q. It is extremely important for teachers to maintain a healthy diet. Therefore, students should always give them an: A. Apple B. Banana C. Bread roll D. Chocolate bar E. Mohito Grammatical clues Jane Holland a / an plural / singular
21. 21. Q. A 76 year old diabetic is brought into the accident and emergency department with suspected diabetic ketoacidosis. After confirming that she is hyperglycaemic, the first step in management should be intravenous administration of: A. Chest X-ray B. Electrocardiogram C. Fasting blood glucose level D. Insulin E. Placement of an nasogastric tube Grammatical clues Jane Holland
22. 22. Word repeats Q. A 58 year old man with a history of heavy alcohol use and previous psychiatric hospitalisation is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called: A. Depersonalisation B. Derailment C. Derealisation D. Focal memory deficit E. Signal anxiety Jane Holland
23. 23. Logical clues Q. Crime is: A. Equally distributed among the social classes B. Overrepresented among the poor C. Overrepresented among the middle class & rich D. Primarily an indication of psychosocial maladjustment E. Reaching a plateau of tolerability for the nation Jane Holland Also be wary of overlapping numericals...
24. 24. Convergence Strategy Q. Local anaesthetics are most effective in the: A. Anionic form, acting from inside the nerve membrane B. Cationic form, acting from inside the nerve membrane C. Cationic form, acting from outside the nerve membrane D. Uncharged form, acting from inside the nerve membrane E. Uncharged form, acting from outside the nerve membrane Jane Holland
25. 25. Long correct answer Q. Secondary gain is: A. Synonymous with malingering B. A frequent problem in OCD C. A complication of a variety of illnesses and tends to prolong many of them D. Never seen in organic brain damage Jane Holland
26. 26. Absolutes... Q. Which of the following shoes are never worn by men? A. Boots B. Runners C. Sandals D. Stilettos E. Wellies Jane Holland
27. 27. Absolutes Holsgrove G, Medical Education 1998; 32: 343 - 350 Jane Holland 0 10 20 30 40 50 60 0% 1% 5% “Never” 0 10 20 30 40 50 60 80% 95% 99% 100% “Always”
28. 28. Jane Holland
29. 29.  Shape  Long vignette & stem  short options  Cover test • Ideally can know the correct answer before seeing the possible options  Use homogenous distracters • i.e. all arteries, all medications, all diagnoses • Grammatically consistent • Logically compatible • List in logical or alphabetical order  Don’t use vague frequency terms  Never use absolutes!  Do not write a negative question!! NBME Manual p 35 - 49 Some useful principles Jane Holland
30. 30. Jane Holland
31. 31. Difficulty Discrimination Corrected Point Biserial Distractor Choice Introduction to Item Analysis  Classical Test Theory  Item Response Theory (IRT)  Generalizability Theory (G-Theory) Richard Arnett De Champlain: Med Educ. 2010; 44:109-117.
32. 32.  How did the items / questions perform?? 1. What proportion of candidates got the question correct? 2. Which options were chosen most frequently? 3. Which options were chosen by low, medium and high performers on the test as a whole? 4. How did success on an item correlate with overall test score? Jane Holland Difficulty Discrimination Corrected Point Biserial Distractor Choice De Champlain: Med Educ. 2010; 44:109-117.
33. 33. Item difficulty (DIF) – or facility  What proportion of candidates got the question correct? 90% of candidates answered correctly  DIF = 0.90  < 0.40 – hard  > 0.80 – easy  0.40 – 0.80 – good question  0.45 – 0.55 are extremely discriminating Jane Holland
34. 34. Which options were chosen most frequently?  Distractor choice  Examine the proportion of candidates choosing each option  ALL options should be chosen by candidates  Replace any options not chosen Jane Holland
35. 35. Discriminating Index (or power)  Which options were chosen by low, medium and high performers on the test as a whole? Lower 27% Upper 27%Middle  L N U U – L N = DI  Ideally, DP should be > 0.40 Jane Holland
36. 36. Jane Holland
37. 37. Richard Arnett
38. 38. Additional measures of Discrimination  Point biserial correlation (RPB)  Equivalent to Pearson correlation co-efficient  Is a measure of association between a binary (nominal) variable and a continuous variable  Corrected point biserial correlation (CRPB)  The item is removed from the test total prior to calculation  Range = -1 to +1  >0.2 good, <0.2 weak  Negative – inverse relationship!!!! Jane Holland
39. 39. Jane Holland
40. 40. Jane Holland
41. 41. Jane Holland
42. 42. Moodle Quiz statistics  Item Difficulty... Jane Holland  In a quiz with random questions...  Position of items within the test may change  Therefore, analyses must distinguish between positions in the test, and test items  Students may answer different sets of questions  i.e. Answer 30 out of 50 possible questions from the bank Np N
43. 43.  Click on the quiz, then access:  Administration  Quiz administration  Results  Statistics Jane Holland
44. 44. Jane Holland
45. 45. Jane Holland
46. 46. Jane Holland
47. 47. Good Stats Good Questions Good stats don’t necessarily mean good questions and good questions don’t always get good stats... ...but it is a good place to start Richard Arnett