Upcoming SlideShare
×

1,804 views
1,638 views

Published on

4 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
1,804
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
0
0
Likes
4
Embeds 0
No embeds

No notes for slide

1. 1. Advanced Test Item Construction Jim Parry, CPT, M.Ed. Test Development Manager USCG Training Center Yorktown, VA 2013 Users Conference Baltimore | March 3rd – 6thCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
3. 3. Back to the BasicsAnatomy 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
4. 4. Test Item Construction Stem  The beginning part of the item that presents the problem to be solved as well as other relevant information  Question  Incomplete statement Alternatives (options)  Correct answer – Key  Incorrect answers - Distractors 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
5. 5. Levels of Item ComplexityMake them think! 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
6. 6. Considerations Consider complexity when developing test items  What do you want the test taker to be able to “show” you? Three levels  Low  Moderate  High 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
7. 7. Low Complexity Test items require factual knowledge. Require knowledge of single fact, not multiple facts. Examples:  Knowing the name of piece of equipment  Knowing what test procedure to use in a simple situation  Knowing what reference provides instructions  Knowing which tool to use to repair a particular piece of equipment 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
8. 8. Low Complexity Example 2013 Users Conference  BaltimoreCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
9. 9. Moderate Complexity Test items require test taker to know or derive multiple facts.  May be moderately complex in two ways  Require knowledge of more than one fact  Two or more pieces of information are combined in some fashion to produce a new fact  Require more complex knowledge about facts because information is combined or compounded  Correct answer depends on factual knowledge that is complete and definitive 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
10. 10. Moderate Complexity Example 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
11. 11. High Complexity Items require test taker to:  Analyze & evaluate multiple facts to solve problems  Diagnose solutions  Make decisions Correct answer may not be recognized until it is compared to incorrect responses to identify the best answer under conditions described in the stem 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
12. 12. High Complexity (cont.) The analysis or evaluation required to recognize the correct response requires:  Synthesis of multiple facts  Evaluation of each alternative to compare it to the stem  Occasionally, evaluation of each alternative for comparison and/or correlation to each other Typically scenario type test items  Derived from “real” on-the-job situations 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
13. 13. High Complexity Example 2013 Users Conference  BaltimoreCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
14. 14. Matching Complexity to CognitivePerformanceBloom and Gangé weren’t all wrong! 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
15. 15. Bloom’s Original Taxonomy (1956) System describes levels of cognitive functioning so testing cognitive performance could be improved ◦ Evaluation ◦ Synthesis ◦ Analysis ◦ Application ◦ Comprehension ◦ Knowledge Taxonomy assumes each level is subsumed by the next higher 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
16. 16. Bloom’s vs. Item Complexity • Evaluation High • Synthesis • Analysis Moderate • Application • Comprehension Low • Knowledge 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
17. 17. Simple Objectives1.1 Given an electrical circuit schematic with a power source and various loads indicated, CALCULATE the current draw in amps within ± 0.1 amps. 1.1.1 SELECT appropriate power formula 1.1.2 CONVERT watts to amps 1.1.3 CALCULATE total load in watts and amps 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
18. 18. Knowledge Learning objectives require learner to:  Recall from memory  Recall is identical to original presentation  Recite definitions, procedures, formulas, etc. Test items ask learner to:  State the steps in a procedure  Define a term  Identify an object 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
19. 19. Knowledge (Low Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
20. 20. Comprehension Learning objectives require learner to:  Restate in own words  Translate information  Apply designated rules  Recognize examples of concepts Test items ask learner to:  Identify unseen examples  Classify unseen examples 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
21. 21. Comprehension (Low Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
22. 22. Application Learning objectives require learner to:  Decide rule(s) to apply  Solve problems by selecting rule/method Test items ask learner to:  Solve problems presented without rules/formulas given  Decide course of action to take without prompting 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
23. 23. Application (Moderate Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
24. 24. Analysis Learning objectives require learner to:  Break down complex situations  Figure out how parts relate to each other  Figure out how one thing influences another Test items ask learner to:  Solve problems given an extensive scenario, description or data  Some information may not be pertinent 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
25. 25. Analysis (Moderate Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
26. 26. Synthesis Learning objectives require learner to:  Create totally original material, products, designs, equipment, etc. Test items ask learner to:  Create totally original material, products, designs, equipment, etc.  Normally cannot be multiple-choice or other closed-ended test items 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
27. 27. Synthesis (High Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
28. 28. Evaluation Learning objectives require learner to:  Judge appropriateness or worth of objects, plans, designs, etc. with some purpose in mind Test items ask learner to:  Use all five lower levels of Bloom’s Taxonomy to meet objective  Normally cannot be assessed with closed-ended test items 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
29. 29. Gagné’s Learned Capabilities (1985) Divides learning outcomes into five classes of behavior.  Intellectual Skills  Discriminations  Concrete concepts This is where we  Defined concepts typically test  Rules  Higher order rules (problem solving)  Cognitive Strategies  Verbal Information  Motor Skill  Attitudes 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
30. 30. Gagné vs. Item Complexity • Problem Solving High • Higher Order Rules • RulesModerate • Defined Concepts • Concrete Concepts Low • Discriminations 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
31. 31. Bloom’s TaxonomyGagné’s Learned Capabilities 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.