Your SlideShare is downloading. ×
0
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Advanced Test Item Construction
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Advanced Test Item Construction

1,321

Published on

Published in: Business, Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,321
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Advanced Test Item Construction Jim Parry, CPT, M.Ed. Test Development Manager USCG Training Center Yorktown, VA 2013 Users Conference Baltimore | March 3rd – 6thCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 2. References American Educational Research Association, (1999).The standards for educational and psychological testing: American Educational Research Association: Washington, DC Department of the Air Force. (2012). 2014E Promotion test development standards. Headquarters, United States Air Force, Air Education and Training Command A3, Airman Advancement Division, Randolph AFB, TX. (Restricted publication) Department of the Army. (2004). Systems approach to training: Testing. (TRADOC Pamphlet 350-70-5). Headquarters, United States Army, Training and Doctrine Command: Fort Monroe, Virginia. Department of the Navy. (2008). Advancement examination and administration process guide (AEAPG). United States Navy, Naval Education and Training Command:, Navy Advancement Center: Pensacola, FL (Restricted publication) Downing, S. M. & Haladyna, T. M., (Ed.)(2006). Handbook of test development. Lawrence Erlbaum Associates, Inc.: Mahwah, NJ. Rust, J. & Golomboc, S. 1999, Modern psychometrics: the science of psychological assessment. Routledge: New York, NY. Shrock, S. & Coscarelli, W. (2007), Criterion-referenced test development -Technical and legal guidelines for corporate training: John Wiley & Sons, Inc.: San Francisco, CA 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 3. Back to the BasicsAnatomy 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 4. Test Item Construction Stem  The beginning part of the item that presents the problem to be solved as well as other relevant information  Question  Incomplete statement Alternatives (options)  Correct answer – Key  Incorrect answers - Distractors 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 5. Levels of Item ComplexityMake them think! 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 6. Considerations Consider complexity when developing test items  What do you want the test taker to be able to “show” you? Three levels  Low  Moderate  High 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 7. Low Complexity Test items require factual knowledge. Require knowledge of single fact, not multiple facts. Examples:  Knowing the name of piece of equipment  Knowing what test procedure to use in a simple situation  Knowing what reference provides instructions  Knowing which tool to use to repair a particular piece of equipment 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 8. Low Complexity Example 2013 Users Conference  BaltimoreCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 9. Moderate Complexity Test items require test taker to know or derive multiple facts.  May be moderately complex in two ways  Require knowledge of more than one fact  Two or more pieces of information are combined in some fashion to produce a new fact  Require more complex knowledge about facts because information is combined or compounded  Correct answer depends on factual knowledge that is complete and definitive 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 10. Moderate Complexity Example 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 11. High Complexity Items require test taker to:  Analyze & evaluate multiple facts to solve problems  Diagnose solutions  Make decisions Correct answer may not be recognized until it is compared to incorrect responses to identify the best answer under conditions described in the stem 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 12. High Complexity (cont.) The analysis or evaluation required to recognize the correct response requires:  Synthesis of multiple facts  Evaluation of each alternative to compare it to the stem  Occasionally, evaluation of each alternative for comparison and/or correlation to each other Typically scenario type test items  Derived from “real” on-the-job situations 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 13. High Complexity Example 2013 Users Conference  BaltimoreCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 14. Matching Complexity to CognitivePerformanceBloom and Gangé weren’t all wrong! 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 15. Bloom’s Original Taxonomy (1956) System describes levels of cognitive functioning so testing cognitive performance could be improved ◦ Evaluation ◦ Synthesis ◦ Analysis ◦ Application ◦ Comprehension ◦ Knowledge Taxonomy assumes each level is subsumed by the next higher 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 16. Bloom’s vs. Item Complexity • Evaluation High • Synthesis • Analysis Moderate • Application • Comprehension Low • Knowledge 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 17. Simple Objectives1.1 Given an electrical circuit schematic with a power source and various loads indicated, CALCULATE the current draw in amps within ± 0.1 amps. 1.1.1 SELECT appropriate power formula 1.1.2 CONVERT watts to amps 1.1.3 CALCULATE total load in watts and amps 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 18. Knowledge Learning objectives require learner to:  Recall from memory  Recall is identical to original presentation  Recite definitions, procedures, formulas, etc. Test items ask learner to:  State the steps in a procedure  Define a term  Identify an object 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 19. Knowledge (Low Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 20. Comprehension Learning objectives require learner to:  Restate in own words  Translate information  Apply designated rules  Recognize examples of concepts Test items ask learner to:  Identify unseen examples  Classify unseen examples 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 21. Comprehension (Low Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 22. Application Learning objectives require learner to:  Decide rule(s) to apply  Solve problems by selecting rule/method Test items ask learner to:  Solve problems presented without rules/formulas given  Decide course of action to take without prompting 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 23. Application (Moderate Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 24. Analysis Learning objectives require learner to:  Break down complex situations  Figure out how parts relate to each other  Figure out how one thing influences another Test items ask learner to:  Solve problems given an extensive scenario, description or data  Some information may not be pertinent 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 25. Analysis (Moderate Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 26. Synthesis Learning objectives require learner to:  Create totally original material, products, designs, equipment, etc. Test items ask learner to:  Create totally original material, products, designs, equipment, etc.  Normally cannot be multiple-choice or other closed-ended test items 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 27. Synthesis (High Complexity) 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 28. Evaluation Learning objectives require learner to:  Judge appropriateness or worth of objects, plans, designs, etc. with some purpose in mind Test items ask learner to:  Use all five lower levels of Bloom’s Taxonomy to meet objective  Normally cannot be assessed with closed-ended test items 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 29. Gagné’s Learned Capabilities (1985) Divides learning outcomes into five classes of behavior.  Intellectual Skills  Discriminations  Concrete concepts This is where we  Defined concepts typically test  Rules  Higher order rules (problem solving)  Cognitive Strategies  Verbal Information  Motor Skill  Attitudes 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 30. Gagné vs. Item Complexity • Problem Solving High • Higher Order Rules • RulesModerate • Defined Concepts • Concrete Concepts Low • Discriminations 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 31. Bloom’s TaxonomyGagné’s Learned Capabilities 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 32. 2013 Users Conference  BaltimoreCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 33. Your Turn! Chose a topic you are familiar with Write an objective List steps to perform the objective Write test items to test the objective as follow:  Low complexity  Moderate complexity  High complexity As you complete each type ask facilitator(s) to evaluate your work 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 34. Suggested Homework Chose another topic you are familiar with Write an objective List steps to perform the objective Write test items to test the objective at each level of Bloom’s Taxonomy • Evaluation High • Synthesis • Analysis Moderate • Application • Comprehension Low • Knowledge 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 35. 2013 Users Conference  BaltimoreCopyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

×