Your SlideShare is downloading. ×
0
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Workshop — The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams

1,297

Published on

The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams

The Art of Writing Good Multiple-Choice Questions for High-Stakes Exams

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,297
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. The Art of Writing GoodMultiple-Choice Questions for High-Stakes Exams Claire Touchie, MD, FRCPC University of Ottawa Medical Council of Canada
  2. Workshop Objectives• Describe what can be tested with multiple- choice items• Define the anatomy of a multiple-choice item• Define and identify technical flaws• Create multiple-choice items for own stated purpose• Define and criticize poor performing items 2
  3. How to write multiple-choice items1. The What2. The How 3
  4. The What• What can I test with MCQs? • Knowledge • Clinical decision making• Clearly define the purpose of your exam• Define what it is you want to test • For the overall test • For your specific question 4
  5. The WhatPrior to writing your question, ask thefollowing questions• What concept do I want to test?• Where does the learner go wrong? • Focus on areas of “challenge” for the learner 5
  6. Example of the WhatPurpose: To assess the clinical clerk’sknowledge and decision-making capabilityat the end of an Internal Medicine six-weekrotation• Concept/Objective: Management of CHF• Challenge to the learner: ??? 6
  7. The How – Anatomy of a MC item• Stem • Clinical vignette that describes the setting, the patient’s age and complaint along with pertinent historical facts, physical exam details, and/or laboratory findings• Lead-in question • The task• Answer and alternatives • The most correct answer and the plausible distractors 7
  8. The How – Anatomy of a MC itemStem• A 58-year old man presents to the ED with sudden onset of left-sided chest pain associated with shortness of breath, palpitations and dizziness. His past history is relevant for a recent diagnosis of lung carcinoma. His examination is only remarkable for a heart rate of 112/minute. 8
  9. The How-Anatomy of a MC itemLead-in question• Which one of the following diagnostic test would be most useful to confirm the diagnosis? 9
  10. The How – Anatomy of a MC itemCorrect/Best answer and distractors1. Chest radiograph2. CT of the chest *3. Sputum culture4. Electrocardiogram5. Echocardiogram 10
  11. What did you notice about this question?Could you answer it without seeing thealternatives?Could you answer without being given thediagnosis? 11
  12. The Stem• Short description of a clinical scenario • Common or clinically important • Clear and contains relevant information to the clinical problem – avoid window-dressing• Word the stem positively • Avoid EXCEPT questions • Use negative words with caution • Eg: contraindication, what to avoid 12
  13. The Stem• Provide sufficient information to answer the item • DO NOT create tricky items by omitting essential information • DO NOT add extraneous information • Stem should be a clinical vignette 13
  14. Lead-in Question• Ensure the directions are very clear with a clear task • Can the stem be administered in a short answer (constructed-response) format? • “Cover answer test”
  15. Lead-in Question• Different clinical tasks can be tested• Can be done with the same stem (cloning of question) • History • Diagnosis • Investigations • Management/Treatment/Drug therapy • Counseling
  16. Lead-in QuestionTry asking questions that lead to clinical decisionmaking?Which one of the following – … is the most likely diagnosis? – … investigations would you now order? – …is the next step in the work-up and management of this patient? – … is the most important step in the initial management of this patient?
  17. Distractors• Only one right choice with four distractors• Use plausible distractor choices• Keep distractors independent, they should not be overlapping • E.g.: 1. 11-20; 2. 15-30• Keep distractors homogeneous in content and grammatical structure• Keep the length about equal• Avoid specific determiners such as All, Never, Always, Completely and Absolutely• Do not use All of the Above or None of the Above
  18. Item testing clinical decision makingA 62 year-old woman with a history of confusion and constipation comes tothe office for a follow-up visit. Laboratory investigations reveal a serumcalcium of 2.9mmol/L, a creatinine of 146 µmol/L, and a hemoglobin of 108g/L.Which one of the following is the most likely diagnosis?1. Hyperparathyroidism2. Chronic renal failure3. Multiple myeloma*4. Vitamin D intoxication5. Renal cell carcinoma
  19. Item testing clinical decision makingA 62 year-old woman with a history of confusion and constipationcomes to the office for a follow-up visit. Laboratory investigationsreveal a serum calcium of 2.9mmol/L, a creatinine of 146 µmol/L, anda hemoglobin of 108 g/L.Which one of the following would help confirm the diagnosis?1. Parathyroid hormone2. Serum protein electrophoresis*3. 25-OH vitamin D4. Serum creatinine5. Abdominal ultrasound
  20. What is wrong with this item?A previously healthy person suddenly presents withpleuritic pain in the left chest and shortness of breath.Which one of the following is the most likelydiagnosis?1. Mycoplasma pneumonia2. Spontaneous pneumothorax3. Pulmonary embolism4. Acute pericarditis5. Epidemic pleurodynia
  21. What is wrong with this item?Which one of the following is true about pseudogout?1. It occurs frequently in women.2. Seldom associated with acute pain in a joint3. May be associated with a finding of chondrocalcinosis.4. It is hereditary in all cases5. It responds well to treatment with allopurinol 21
  22. What is wrong with this item?Aortic insufficiency may be caused by all of thefollowing, EXCEPT:1. syphilis2. Marfan’s syndrome3. aortic dissection4. bacterial endocarditis5. myocardial infarction*
  23. Exercise #1 – 30 minutesWriting MCQ items • Pair up with a partner • Using the MCQ item development worksheet, develop MCQ items that will be useful to you 23
  24. Exercise #2 – Pre-test A Test of General Rock and Roll Knowledge 24
  25. Technical FlawsViolations of test item writing principles• Flawed items are usually more difficult• Fail more students Downing, 2002 25
  26. Technical Flaws• Unfocused items• Negative stem or lead-in question• Heterogeneous options• Logical or grammatical cues• Long correct answer• Word repeats• Convergence strategy 26
  27. Unfocused itemWhich one of the following is true about pseudogout?1. It occurs frequently in women.2. Seldom associated with acute pain in a joint3. May be associated with a finding of chondrocalcinosis.4. It is hereditary in all cases5. It responds well to treatment with allopurinol 27
  28. Negative stem or lead-in questionWhich of the following does not cause aorticinsufficiency?1. syphilis2. Marfan’s syndrome3. aortic dissection4. bacterial endocarditis5. myocardial infarction* 28
  29. Heterogeneous optionA 24-year-old female presents to a walk-in clinic with fever,flank pain, frequency and dysuria. The urinalysis (urinemicroscopy) shows 1+proteinuria, 25 white blood cells per highpower field and a few granular casts.Which one of the following investigations is the next best step?1. Intravenous pyelography.2. Intravenous antibiotics.3. Creatinine clearance.4. Midstream urine culture.*5. Oral analgesia. 29
  30. Logical cuesA 47-year old man present with an acute episode ofpsychosis. Which one of the following treatment wouldyou consider prescribing?1. Alprazolam2. Lorazepam3. Haloperidol*4. Diazepam5. Quetiapine 30
  31. Grammatical cuesA 78-year old man undergoes a thoracentesis for a largepleural effusion. Three hours later, he develops suddenonset of shortness of breath. What is your most likelydiagnosis?1. Reaccumulation of fluid2. Pneumothorax*3. Lung infection4. Bleeding5. Blood clot 31
  32. Word repeatsAlso known as “clang association”:A 45-year-old woman presents with sudden loss of consciousness. Onexam, her vitals are normal, she is not pale and she is not diaphoretic.Which one of the following is more typical of “fainting” as a conversionsymptom than of a syncopal attack due to orthostatic hypotension?1. Bradycardia.2. Muscle twitching.3. Absence of pallor and sweating.*4. Urinary incontinence.5. Rapid recovery. 32
  33. Convergence strategyAn 86-year-old woman fell at the local nursing home andsustained an intertrochanteric fracture of her left hip. On clinicalexamination, you would expect to find her left leg:1. Shortened, abducted and internally rotated.2. Lengthened, abducted and internally rotated.3. Shortened, adducted and externally rotated.4. Shortened, abducted and externally rotated.*5. Lengthened, abducted and externally rotated. 33
  34. Post-test A Test of General Rock and Roll Knowledge 34
  35. Now review the items you have written using the checklist! 35
  36. Exercise #3 – Poor Performing ItemsTests used to assess those who meet thepurpose and those who don’t!The administered test must therefore liveup to this expectation. 36
  37. Poor Performing ItemsWhen an item is either:• Too hard (very few are getting it right)• Too easy (very few are getting it wrong)• Does not distinguish between an overall good performing candidate or a poor performing candidate (discrimination)• Wrongly keyed• Poor distribution of responses 37
  38. Item DifficultyPercentage or proportion of candidates thatselect its correct answer • Should be between 20-90% (0.20-0.90) • Best if it is around the cut score but need a range of difficulty
  39. Item DiscriminationPoint-biserial index of discrimination • Correlation between candidates performance on an item and performance on the entire test (the latter assumed to be a measure of a candidate’s overall ability) • Range from –1 to +1 • Stronger candidates should perform better than weaker candidates • Effectively discriminating item is positive and >0.20
  40. Item DiscriminationCorrect answer should have positivediscrimination • If not: is there another correct answer? • Is there a flaw in the question?Distractors should have negativediscrimination
  41. Distribution of ResponsesA report of the percentage of candidatesthat select each of the options presentedA question’s distractors may need to beassessed if no candidates are choosingthem.
  42. Poor performing itemsQuestion 1 Answer key: option 1Item difficulty: 0.12Item discrimination: 0.02 Distractors Response Rate Discrimination Option 1* 0.12 0.02 Option 2 0.03 -0.12 Option 3 0.05 -0.23 Option 4 0.09 -0.17 Option 5 0.71 0.24 42
  43. Poor Performing ItemQuestion 2 Answer key: option 1Item difficulty: 0.97Item discrimination: 0.28 Distractors Response Rate Discrimination Option 1* 0.97 0.28 Option 2 0.01 -0.20 Option 3 0.01 -0.34 Option 4 0.00 0 Option 5 0.00 -0.04 43
  44. Poor Performing ItemQuestion 3 Answer key: option 2Item difficulty: 0.59Item discrimination: 0.16 Distractors Response Rate Discrimination Option 1 0.00 -0.04 Option 2* 0.59 0.16 Option 3 0.00 -0.04 Option 4 0.39 -0.40 Option 5 0.02 -0.12 44
  45. Poor Performing ItemsQuestion 4 Answer key: option 5Item difficulty: 0.53Item discrimination: 0.26 Distrators Response Rate Discrimination Option 1 0.23 -0.26 Option 2 0.04 -0.24 Option 3 0.05 -0.15 Option 4 0.16 -0.33 Option 5* 0.53 0.26 45
  46. In SummaryPrior to writing MCQ items: • Determine the purpose of the test • Determine WHAT you want to testWrite questions avoiding technical flawsEnsure questions of are of good qualityreflecting the original purpose. 46
  47. References• Guidelines for the Development of Multiple-Choice Questions at http://www.mcc.ca/pdf/MCQ_Guidelines_e.pdf• Haladyna TM, Downing SM, Rodriguez MC. A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education 2002;15:309-334 47
  48. Thank you!Questions? Comments? ctouchie@mcc.ca 48

×