Test construction (for content staff) eg feb08 erp


Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Test construction (for content staff) eg feb08 erp

  1. 1. Test Construction Procedure Zarko Vukmirovic Workshop: Test Construction: 2008 CAPS Forms Assembly
  2. 2. <ul><li>Scope and Objectives </li></ul><ul><li>Scope: </li></ul><ul><ul><li>To present and discuss the procedure for assembling CAPS operational and field-test forms. </li></ul></ul><ul><ul><li>The procedure is also designed to be a model for other assessment programs. </li></ul></ul><ul><li>Objectives: </li></ul><ul><ul><li>Participants will learn the principles of test construction and will be able to use the technical resources for assembling of test forms. </li></ul></ul><ul><ul><li>Participants will assemble 2008 CAPS operational forms. </li></ul></ul>
  3. 3. Definitions <ul><li>The purpose of the Test Construction </li></ul><ul><ul><li>Assemble the O perational and F ield-test forms </li></ul></ul><ul><ul><li>Use the Item Bank </li></ul></ul><ul><ul><li>Use Test Construction Specifications . </li></ul></ul><ul><li>The final products </li></ul><ul><ul><li>Pull Lists (list of items included in the test) </li></ul></ul><ul><ul><li>Test Map (precise specification of item locations in the test) </li></ul></ul>
  4. 4. Responsibilities <ul><li>Test Development Director </li></ul><ul><ul><li>Works with Psychometrician on Test Construction Specifications , </li></ul></ul><ul><ul><li>Coordinates the process and approves pull lists. </li></ul></ul><ul><li>Content Specialist </li></ul><ul><ul><li>Selects items using Test Construction guidelines, </li></ul></ul><ul><ul><li>Checks accuracy of the final pull list. </li></ul></ul>
  5. 5. Responsibilities <ul><li>Psychometrician </li></ul><ul><ul><li>Works with Test Development Director on Test Construction Specifications , </li></ul></ul><ul><ul><li>Provides training for Content Specialists and other staff, </li></ul></ul><ul><ul><li>Assists with item selection, </li></ul></ul><ul><ul><li>Interprets item and form statistics, </li></ul></ul><ul><ul><li>Approves pull lists </li></ul></ul><ul><li>Editorial Team </li></ul><ul><ul><li>Reviews the items in the final pull list. </li></ul></ul><ul><li>Quality Assurance Staff </li></ul><ul><ul><li>Verifies the final pull lists against source documents. </li></ul></ul><ul><li>IT Staff </li></ul><ul><ul><li>Provides support for using information technology </li></ul></ul>
  6. 6. Inputs & Assumptions <ul><li>Documents: </li></ul><ul><ul><li>Test Construction Specifications (including domain and cluster specifications). </li></ul></ul><ul><li>Item Bank: </li></ul><ul><ul><li>Item Content </li></ul></ul><ul><ul><li>Item Statistics </li></ul></ul><ul><li>Assumptions: </li></ul><ul><ul><li>Item content and statistics databases are populated with sufficient number of items. </li></ul></ul><ul><ul><li>All items are reviewed for content and psychometric qualities and accepted by review committees. </li></ul></ul><ul><ul><li>All staff is trained for test construction procedure </li></ul></ul>
  7. 7. Outputs <ul><li>Content Specialist will create </li></ul><ul><ul><li>Pull Lists </li></ul></ul><ul><ul><li>Test Maps. </li></ul></ul><ul><li>Psychometrician will create: </li></ul><ul><ul><li>Curves: TCCs, TIFs, SEMs, </li></ul></ul><ul><ul><li>Statistical Summary Sheets, </li></ul></ul><ul><ul><li>Test Construction Report </li></ul></ul>
  8. 8. Procedure (1) <ul><li>Content Specialist selects operational items using the Item Bank and following Test Construction Specifications . </li></ul><ul><ul><li>Match the blueprint and follow other content guidelines </li></ul></ul><ul><ul><li>Match statistical targets (average p-value and point-biserial correlations) </li></ul></ul><ul><li>Content Specialist provides the pull list to Psychometrician (Item Bank: an export file is generated). </li></ul>
  9. 9. Procedure (2) <ul><li>Psychometrician will use a statistical program to merge the pull list with the statistical data for the items (if using Item bank: provided in the export file). </li></ul><ul><li>Psychometrician will use a statistical program to create Statistical Summary Sheets and test curves (TCC, TIF, and SEM). </li></ul><ul><li>Psychometrician will verify that items and forms characteristics meet requirements specified in Test Construction Specifications . </li></ul>
  10. 10. Procedure (3) <ul><li>Item Statistics checks </li></ul><ul><ul><li>Number of Items (overall test and by domain) </li></ul></ul><ul><ul><li>Item difficulty </li></ul></ul><ul><ul><li>Item discrimination </li></ul></ul><ul><ul><li>Options analysis </li></ul></ul><ul><ul><li>DIF statistics </li></ul></ul><ul><li>Forms characteristics checks </li></ul><ul><ul><li>Maximum Points (overall test, by reporting category, by item type) </li></ul></ul><ul><ul><li>Does obtained TCC approximates target TCC sufficiently </li></ul></ul><ul><ul><li>Is TIF sufficiently high at al relevant scale points </li></ul></ul><ul><ul><li>Is SEM low at relevant points </li></ul></ul><ul><li>Linking items checks </li></ul><ul><ul><li>Content representation </li></ul></ul><ul><ul><li>Statistical representation </li></ul></ul><ul><ul><li>Position maintenance </li></ul></ul>
  11. 11. Procedure (4) <ul><li>If all the criteria are met Psychometrician will sign off the form. </li></ul><ul><li>If modifications are needed, Psychometrician specifies the guidelines for changes . </li></ul><ul><li>Optionally, Psychometrician will meet with Content Specialist to review and make the necessary changes. </li></ul><ul><li>When the changes are complete, steps 1 - 3 are repeated. </li></ul>
  12. 12. Procedure (5) <ul><li>Once all changes are complete and the criteria are met, Psychometrician, Content Specialist, and TDD will sign off the form. </li></ul><ul><li>If the customer wants to review the test map, it is at this point that the test map will be sent to the customer by the TDD. </li></ul><ul><li>The TDD will provide a cover letter along with an explanation of the test map. </li></ul><ul><li>After customer approval the test map is forwarded to composition staff to prepare the form for printing. </li></ul>
  13. 13. Test Construction System <ul><li>Item Bank </li></ul><ul><ul><li>Search and review items </li></ul></ul><ul><ul><li>Selecting items to be included in test forms </li></ul></ul><ul><ul><li>Printing forms preview </li></ul></ul><ul><ul><li>Exporting item IDs (and statistics) </li></ul></ul><ul><li>SPSS program </li></ul><ul><ul><li>Generating TCCs, TIFs, SEMs, </li></ul></ul><ul><ul><li>Generating statistical summary sheets </li></ul></ul>
  14. 14. Test Characteristic Curve
  15. 15. Test Information Curve
  16. 16. Standard Error of Measurement