Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Effects of Question Formats on Student and Item Performance


Published on

Presented by David Caldwell, Director of Professional Affairs, and Adam Pate, Assistant Professor, both from University of Louisiana Monroe School of Pharmacy

Although multiple-choice examinations are one of the most commonly used assessment methods in higher education, often faculty members receive little to no formal training in the discipline. This results in wide variation of item quality, which has been shown to affect student performance. This presentation will describe concepts in multiple-choice item writing including common errors, the statistical effects associated with them, and simple solutions to improve flawed items. In this interactive session presenters will describe personal struggles and successes in item writing and review published literature describing the effects on student performance.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Effects of Question Formats on Student and Item Performance

  2. 2. The fundamental problem in shock is: a. Decreased diastolic blood pressure b. Decreased systolic blood pressure c. Decreased tissue perfusion d. Increased serum lactic acid Example
  3. 3. If you find a patient in shock, your first action should be: a. Begin CPR b. Call 911 c. Circulation, airway, breathing assessment d. Find an AED Example
  4. 4. Haladyna, Downing, & Rodriguez • Developed from an analysis of 27 textbooks and 27 research studies • Purpose was to validate each guideline based on agreementin studied sources Haladyna TM, Downing SM, Rodriguez SM. Applied Measurement in Education. 2002; 15(3):309-334.
  5. 5. Which of the following is the capital of Texas? a. Austin* b. Dallas c. El Paso d. Houston Anatomy of a MC question
  6. 6. Guidelines: a closer look Guideline For (%) Uncited (%) Against (%) Use positives, no negatives 63 19 18 Write as many plausible distractors as you can 70 26 4 Use None of the Above carefully 44 7 48 Avoid All of the Above 70 7 22
  7. 7. All of the following drugs are metabolized by the P450 enzyme system except: a. Atazanavir b. Darunavir c. Raltegravir d. Tipranavir Use positives, no negatives
  8. 8. Stem negation = most appropriatewhen used to measure relevant objectives
  9. 9. Haladyna and Downing (1993) • Two-thirds of all items had only 1-2 effectively performing distractors • Only 1-8% of items had three effective distractors • No measured effect on difficulty, but more effective distractors = more discrimination Write as many plausible distractors as possible Haladyna TM, Downing SM. Education and Psychological Measurement. 1993; 53:999-1010.
  10. 10. Effect of numerous distractorsunclear
  11. 11. Which of the following is the capital of Texas? A. Dallas B. El Paso C. Houston D. Lubbock E. None of the above Use “none of the above” carefully
  12. 12. 3 of 4 studies failed to demonstrate differences in discrimination with “None of the above”
  13. 13. Which of the following is true regarding transdermal NRT? A. These products may be more appropriate in patients who anticipate adherence problems B. The 16-hr patch reduces the risk of insomnia and vivid dreaming compared to the 24-hr C. Currently there is insufficient data to directly compare transdermal NRT to oral NRT D. Transdermal NRT patches should never be cut E. All of the above Avoid “all of the above”
  14. 14. 70% of referenced sources recommend avoiding “all of the above”
  15. 15. 50 60 70 80 90 100 Difficulty (p) Standard Flawed A retrospective study 0 0.2 0.4 0.6 0.8 1 Discrimination (rpb) p=0.01 p=0.64 ! 48 % 52 % Guideline # Times broken % Total Minimize reading 24 16.9 All of the above 24 16.9 Complex item format 19 13.38 Central idea in stem 17 11.97 Edit and proof 10 7.04 Choices homogenous 8 5.63 Plausible distractors 8 5.63 Trivial Content 6 4.23 Length of choices equal 6 4.23 None of the above 6 4.23 Other 13 9.86 Ratio Standard:Flawed Pate AN, Caldwell DJ. CPTL. 2014; 6(1):130-134.
  16. 16. A prospective study 5 authors Student randomization standard, n=55 flawed, n=54 Test administration Analysis: one-sided independent t-tests between the two scales Mean discrimination, rpb Mean item difficulty, p 40 exam items 25 35 45 55 65 75 Difficulty (p) Standard Flawed p=0.04 0 0.1 0.2 0.3 0.4 0.5 Discrimination (rpb) p=0.44 Caldwell DJ, Pate AN. J Pharm Ed. 2013; 77(4):Article 71. Investigator review Test form 1 20 standard items Test form 2 20 flawed items
  17. 17. Any questions?