Using the Angoff Method to Set Cut Scores

11,276 views

Published on

Published in: Business, Technology
4 Comments
6 Likes
Statistics
Notes
No Downloads
Views
Total views
11,276
On SlideShare
0
From Embeds
0
Number of Embeds
1,568
Actions
Shares
0
Downloads
211
Comments
4
Likes
6
Embeds 0
No embeds

No notes for slide

Using the Angoff Method to Set Cut Scores

  1. 1. Using The Angoff Method to Set Cut ScoresAlan Wheaton Jim Parry, CPTInstructional Systems Specialist Test Development ManagerUSCG Maritime Law Enforcement Academy USCG Training CenterCharleston, SC Yorktown, VA
  2. 2.  How do you know your test takers are really minimally competent? What is a passing score on a test? Can you defend your cut score in a court of law?
  3. 3.  What is a cut score? Do you need a cut score? Which method should I use? Should I change the cut score based on test takers actual results?
  4. 4. Criterion-referenced Norm-referenced
  5. 5. • Distinguished research scientist at the Educational Testing Service (ETS) for over 40 years• Harvard, Purdue, U.S. Air Force• Scholastic Aptitude Test• Prominent contributor to educational measurement 1919-1993
  6. 6.  Defensible Easy Minimal Training Widely accepted
  7. 7. 1. Select Raters2. Take the Assessment3. Rate the Items4. Review the Ratings5. Determine the Cut Score
  8. 8. Important! Prior to applying the Angoff method be sure the test is valid and reliable.
  9. 9.  5 – 12 Subject Matter Experts (SMEs) ◦ Diverse group (geographic location, age, gender, race, etc.) Most critical step to success of the process Handout pages 3 & 4
  10. 10.  SMEs take the test The average score can be used to ensure the final passing score is not set higher than the average score obtained by the SMEs This step increases the defensibility of the final score Handout page 4
  11. 11.  Train the SMEs Describe the characteristics of minimally competent performers ◦ Minimally qualified employee on the job ◦ Not a student Describe how to rate items: ◦ Increments of 5  Low 25  High 95 Handout pages 4 & 5
  12. 12.  Collect the SME test item rating forms Enter results into a spreadsheet Determine standard deviation Re-evaluate test items that exceed standard deviation of 10 Enter revised results onto spreadsheet Handout page 6
  13. 13. Average Percentage Expert 1 Expert 2 Expert 3 Expert 4 Expert 5 StandardTest Item Correct Name Name Name Name Name Deviation 1 76 90 80 50 70 90 16.73320053 2 68 65 70 60 75 70 5.700877125 3 74 85 85 60 60 80 12.94217911 4 85 85 90 80 85 85 3.535533906 5 79 95 85 60 80 75 12.94217911Average 76.4 84 82 62 74 80
  14. 14.  Add the numbers in the “test item” column and divide by the number of SMEs. ◦ Enter each test item average score in the “average percentage correct” cell Calculate the cut score by adding the numbers in the “average percentage correct” column and dividing by the number of test items Policy makers make final determination? Handout page 7
  15. 15. 1. Select the raters2. Take the assessment (calculate average score)3. Rate the items4. Review the ratings (Enter into spreadsheet)5. Determine cut score
  16. 16. ETS – A primer on Setting Cut Scores on Tests ofEducational Achievement
  17. 17.  Create and follow a process that is appropriate and defensible Since cut scores are based on opinions – it is critical to choose judges wisely ◦ Train them ◦ Provide relevant data Be willing to start over if expected benefits are outweighed by negative consequences
  18. 18. U. S. Coast GuardMaritime Law EnforcementAcademyCharleston, SC
  19. 19.  2008 - Cut score was arbitrarily set at 80% 2009 - Applied for accreditation ◦ Federal Law Enforcement Training Accreditation (FLETA) 2009 - Angoff: raised cut score to 85%
  20. 20.  Prior to Angoff – 4% failure rate After Angoff – 12% failure rate ◦ First academic disenrollment After Angoff – of the 12% that failed, 76% of the failures scored between 81% and 85%
  21. 21.  Had we not raised the cut score from 80% to 85%, 110 of our students (over an 18 month period) would have passed the exam at a less than minimally qualified standard. Angoff method was considered a “best practice” by the Federal Law Enforcement Training Accreditation (FLETA) Board.
  22. 22. Enlisted Advancement Testing
  23. 23.  Arbitrary “Passing” score traditionally set by SME ◦ Not defensible! All rating advancement tests now set by Angoff method Entire test item database evaluated ◦ Angoff score used for random selection from database
  24. 24.  Results before Angoff and item review ◦ 29% first time pass rate Results after Angoff and item review ◦ 34% first time pass rate Item scores are within predicted Angoff range!

×