Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MC Item Writing Workshop

635 views

Published on

How do your write MC Items

  • Be the first to comment

  • Be the first to like this

MC Item Writing Workshop

  1. 1. TESSERACT Educational Nursing Workshop 2/12/2013
  2. 2. Multiple ChoiceItem Writing Workshop Doug Mauger EdD School Of Instructor Education dmauger@vcc.ca Nursing Workshop 2/12/2013
  3. 3. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Validity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  4. 4. Introductions• Name:• What topics do you teach?• How long have you been teaching?• One thing you know about Item writing• What do you want to get out of the workshop? Nursing Workshop 2/12/2013
  5. 5. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Valdity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  6. 6. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  7. 7. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  8. 8. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  9. 9. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  10. 10. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  11. 11. Nursing Workshop 2/12/2013
  12. 12. Nursing Workshop 2/12/2013
  13. 13. What are yourattitudes tocreating exams? Nursing Workshop 2/12/2013
  14. 14. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Validity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  15. 15. Stakeholders Gathering Nursing Workshop 2/12/2013
  16. 16. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Validity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  17. 17. Instrument……Confidence Nursing Workshop 2/12/2013
  18. 18. Points to Ponder There were flaws in 46% of the total questions, with the number of flaws per test ranging from 36% to 65%. In a sample of examinations distributed to first and second year medical students (Downing, 2005) Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the con- sequences of using flawed test items on achievement ex- aminations in medical education. Advances in Health Science Education 10 (2), 133–143. Nursing Workshop 2/12/2013
  19. 19. Process Validity Activity Nursing Workshop 2/12/2013
  20. 20. Validity and Reliability Nursing Workshop 2/12/2013
  21. 21. Reliable but Not Valid Nursing Workshop 2/12/2013
  22. 22. NOT Valid and NOT Reliable Nursing Workshop 2/12/2013
  23. 23. Valid and Reliable Nursing Workshop 2/12/2013
  24. 24. Validity Content Predictive Construct Consequences Process Nursing Workshop 2/12/2013
  25. 25. Reliable Test – Retest Alternate Form Inter Rater Nursing Workshop 2/12/2013
  26. 26. Types of Validity Activity Nursing Workshop 2/12/2013
  27. 27. 1. Process, Content Validity (inadequate sample)2. Process Validity (stress)3. Content Validity (inadequate preparation)4. Predictive Validity (inconsistent results)5. Consequence Validity (induced anxiety)6. Reliability/Process Validity (possible bias) Nursing Workshop 2/12/2013
  28. 28. 7. Process Validity (students not able to provide bestperformance)8. Content Validity (different instructor’s test reflectsdifferent instruction)9. Construct Validity (improper form of assessment)10. Content Validity/Alignment (instruction andassessment do not match) Nursing Workshop 2/12/2013
  29. 29. Why is Validity andReliability importantto you as aprofessional? Nursing Workshop 2/12/2013
  30. 30. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Validity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  31. 31. Alignment Content and Assessment Alignment Are Critical For Valid Assessment Nursing Workshop 2/12/2013
  32. 32. IntendedCurriculumWHAT we planon teaching
  33. 33. ImplementedCurriculumHOW we teachthe curriculum
  34. 34. AchievedCurriculumAssessmentEvidence ofStudentLearning
  35. 35. Outcomes: Intended Implemented Achieved What How AssessmentComplexity of content Perfect Alignment
  36. 36. Complexity of content What How Assessment Scenario 1
  37. 37. Complexity of content What How Assessment Scenario 2
  38. 38. Complexity of content What How Assessment Scenario 3
  39. 39. Complexity of content What How Assessment Scenario 4
  40. 40. Complexity of content What How Assessment Scenario 5
  41. 41. Complexity of content What How Assessment Scenario 6
  42. 42. Complexity of content What How Assessment Scenario 7
  43. 43. Complexity of content What How Assessment Scenario 8
  44. 44. Complexity of content What How Assessment Perfect Alignment
  45. 45. How has Alignment Impacted You? Nursing Workshop 2/12/2013
  46. 46. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Valdity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  47. 47. Nursing Workshop 2/12/2013
  48. 48. Blooms Taxonomy revised Nursing Workshop 2/12/2013
  49. 49. Sorting skills/outcomes fromSimple to Complex Nursing Workshop 2/12/2013
  50. 50. Blooms Taxonomy revised Nursing Workshop 2/12/2013
  51. 51. Remembering/Understanding Level Describes the circulation through the heart Defines the term systole Nursing Workshop 2/12/2013
  52. 52. Application LevelApplication (AP)theories in providing care to clients (e.g., a Apply the concepts of aging in developing interventions for the elderly Nursing Workshop 2/12/2013
  53. 53. Critical Thinking Level revisedBlooms Taxonomy Develop a plan for delivering services to elderly persons who are homebound Evaluate nursing research based on predetermined criteria Analyzes the organizational structure of the community health agency and its impact on client services. Nursing Workshop 2/12/2013
  54. 54. Sortingskills/outcomes into3 classifications Nursing Workshop 2/12/2013
  55. 55. Four Knowledge Dimensions Factual Knowledge The basic elements students must know to be acquainted with a discipline or solve problems in it. Conceptual Knowledge The interrelationships among the basic elements within a larger structure that enable them to function together. Procedural Knowledge How to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods. Meta-Cognitive Knowledge Knowledge of cognition in general as well as awareness and knowledge of ones own cognition Nursing Workshop 2/12/2013
  56. 56. Verbs Nursing Workshop 2/12/2013
  57. 57. Nursing Workshop 2/12/2013
  58. 58. Identify skills/outcomes ByKnowledge Dimension & Cognitive Process Nursing Workshop 2/12/2013
  59. 59. Writing a MC Item What is the biggest challenge for you in sorting outcomes? Nursing Workshop 2/12/2013
  60. 60. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Valdity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  61. 61. Writing a MC Item Nursing Workshop 2/12/2013
  62. 62. Writing a MC Item Nursing Workshop 2/12/2013
  63. 63. MC Item Anatomy Nursing Workshop 2/12/2013
  64. 64. Steps to writing a MC Item item 1. Selecting the competency/outcome 2. Construct the stem 3. Write plausible distractors 4. Peer edit 5. Item analysis Nursing Workshop 2/12/2013
  65. 65. ATOA and NOTA Nursing Workshop 2/12/2013
  66. 66. Points to Ponder ItemConstructing a MC Results showed that when “all-of-the-above” is used as the correct answer, the item is more difficult for all students, despite the literature assumption that it provides a cueing effect to students. Research findings corroborate literature assumptions that high ability students score significantly higher than other ability students in this type of option. Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learning http://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University Nursing Workshop 2/12/2013
  67. 67. Constructing a MC Item Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learning http://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University Nursing Workshop 2/12/2013
  68. 68. Points to Ponder ItemConstructing a MC The present study examined the effect of NOTA on multiple choice questions in a large classroom setting. Specifically, we manipulated the place- ment of NOTA across versions of the test such that, for each of 20 experimental questions, NOTA was either not present, replaced the key, or re- placed the high, medium, or low frequency dis- tractors respectively. We found that NOTA sig- nificantly increased the difficulty of multiple choice questions only when it was used as the key. We found no effect of NOTA replacing any of the dis- tractors, regardless of their effectiveness. Addi- tionally, we found no significant effect of NOTA on discrimination scores, regardless of its place- ment. 2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint submitted to Advances in Health Science Education Nursing Workshop 2/12/2013
  69. 69. Constructing a MC Item Nursing Workshop 2/12/2013
  70. 70. Correct or Best Correct Answer Best Answer Nursing Workshop 2/12/2013
  71. 71. Nursing Workshop 2/12/2013
  72. 72. ContentFree Test Nursing Workshop 2/12/2013
  73. 73. Varieties of MC Items 1. Single Correct Answer 2. Multiple Response 3. Combined Response 4. Multiple Response Nursing Workshop 2/12/2013
  74. 74. Writing a MC Item 1. Design each item to measure an important learning outcome 2. Present a single clearly formulated problem in the stem of the item 3. State the stem of the item in simple, clear language 4. State the stem in positive form wherever possible 5. Word the alternatives clearly and concisely 6. Keep the alternatives mutually exclusive Nursing Workshop 2/12/2013
  75. 75. Writing a MC Item 7. Keep the alternatives homogeneous in content 8. Keep the alternatives free from clues as to which response is correct 9. Avoid the alternatives AOTA and NOTA 10. Use as many functional distractors as are feasible. 11. Include one and only one correct or clearly best answer for each item Nursing Workshop 2/12/2013
  76. 76. Writing a MC ItemWriting a MC Item 12. Present the answer in each of the alternative positions approximately an equal number of times, in a random order 13. Layout the items in a clear and consistent manner 14. Use proper grammar, punctuation, and spelling 15. Avoid using unnecessarily difficult vocabulary 16. Analyze the effectiveness of each item after each administration of the test. Nursing Workshop 2/12/2013
  77. 77. Rules Broken in theContent Free Test Nursing Workshop 2/12/2013
  78. 78. Writing a MC Item Item writing…… Nursing Workshop 2/12/2013
  79. 79. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Valdity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  80. 80. Blueprint or TOS Nursing Workshop 2/12/2013
  81. 81. Remembering/Understanding Level Nursing Workshop 2/12/2013
  82. 82. Application LevelApplication (AP)theories in providing care to clients (e.g., a Nursing Workshop 2/12/2013
  83. 83. Critical Thinking Level revisedBlooms Taxonomy Nursing Workshop 2/12/2013
  84. 84. Blueprint or TOS Nursing Workshop 2/12/2013
  85. 85. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Valdity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  86. 86. Writing a MC Item What So what Now what Nursing Workshop 2/12/2013
  87. 87. Writing a MC Item Nursing Workshop 2/12/2013
  88. 88. Workshop Agenda• Introductions• Overview• Perspectives and Attitudes• Stakeholders• Valdity and Reliability• Alignment• Classifying skills and outcomes• Best Practices in MC item writing• Table of Specification or Blueprint• Take Away• Next Workshop Nursing Workshop 2/12/2013
  89. 89. Writing a MC Item Thank You Nursing Workshop 2/12/2013
  90. 90. ReferencesKrathwohl, David R. (2001). A Taxonomy for Anderson, Lorin W. & Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy. New York. Longman Publishing. http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxono my%20Question%20Stems.pdf Nursing Workshop 2/12/2013

×