Two phases of establishing cutoff score


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Two phases of establishing cutoff score

  1. 1. Two Phases of establishing cutoff score• (1) Obtaining estimated cutoff score using expert judgments.• (2) establishing the operational cutoff scoreThree Components of the estimation phase• (1) selection of an estimation method• (2) data collection• (3) analysis of data 1
  2. 2. Standard-setting Method• Normative Method• Absolute Method – Contrasting Group – Borderline group• Absolute methods based on evaluation of the test: – Jaeger method – Angoff method – Nedelsky method – Ebel Method 2
  3. 3. Normative method• The passing score is based on an individual’s ranking within a group.• Standard is established that passes a given percentage of a reference group of examines.• Ex. 10% of the seniors taking the test will be eligible for admission in Stanford University.• Advantage: The number of people who will pass the test is known.• Disadvantage: It does not guarantee that all individuals who are competent pass the test. 3
  4. 4. Absolute methods based on the Evaluation of Examinees• Contrasting groups – Experts classify each examinee into one or two groups, for example, master or nonmaster. – Other criterion can be used to classify them as maters/nonmasters, experts/novice (grades). – Assess the number of classification errors: qualified individuals who failed, unqualified individuals who passed. – Cutoff the scores in the distribution if the difference in the two groups is significant. – Disadvantage: the selection may not match the scores; A small number of high scores in the nonmaster group can raise the cutoff score, low scores in the master group can lower it. 4
  5. 5. Absolute methods based on the Evaluation of Examinees• Borderline group – Scores of examinees who are classified as borderline are used to set the cutoff score. – Borderline (neither masters nor nonmasters) – The cutoff score is placed at the mean or median of the borderline group. 5
  6. 6. Absolute methods based on Evaluation of the Test• Jaeger method – Each item is judged about its importance for the decision to be made. – “Should every high school graduate be able to answer the item correctly?” – Cutoff scores are established by counting the number of items for which a correct response is judged to be critical. – Advantage: No conceptualization of minimal competent student. – Disadvantage: Passing item could be denied due to the answer of a single item. 6
  7. 7. Absolute methods based on Evaluation of the Test• Angoff method – Judges must estimate the difficulty of the item for a hypothetical group of minimally competent examinees. – The estimated cutoff score for a judged is calculated by summing the item difficulty estimates. – Disadvantage: Difficulty rating is just looking at the item stem. 7
  8. 8. Absolute methods based on Evaluation of the Test• Nedelsky Method – Judges evaluate the attractiveness of each distractor on the basis of whether or not the minimally competent examinee would be able to identify the distractor as incorrect. – Item difficulty is then estimated by assuming borderline examinees will guess randomly among the remaining options. – Advantage: The responses are evaluated aside from the stem. – Disadvantage: method can only be used for multiple choice. 8
  9. 9. Absolute methods based on Evaluation of the Test• Ebel Method – Gives two judgement for each item: its difficulty (easy, moderate, difficult) and its relevance (essential, important, acceptable, questionable). – Items are classified in a 3 X 4 matrix: Easy Moderate Difficult Essential Important Acceptable Questionable 9
  10. 10. Absolute methods based on Evaluation of the Test• Ebel method (cont.) – A jusdgement of the percentage of items in each cell of the matrix that will be answered correctly by the minimally competent student is then used to weight the number of items assigned to each cell. – The cutoff scores is calculated using cell weights and the number of items assigned to each cell. – Advantage: items is evaluated not only on difficulty but on relevance – Disadvantage: Accuracy of judges 10
  11. 11. Compromise methods• Hofstee method – Identify limits of the absolute cutoff score and the limits of the relative failing rates. – Judges make judgments about the item: – (1) what is the lowest acceptable cutoff score, even if every examinee were to fail the examination if this cutoff score was accepted? – (2) What is the highest acceptable cutoff score? – (3) What is the minimum acceptable failing rate? – (4) What is the maximum acceptable failing rate? – 2 points are plotted: The minimum acceptable cutoff score, maximum acceptable failing rate and the maximum acceptable failing rate , maximum acceptable failing rate. – The intersection in the ogive curve is the cutoff point. 11
  12. 12. Compromise methods• Beuk method – A point defined by the average absolute cutoff score and the average passing rate is plotted. – The intersection of this point to the ogive curve is the compromise cutoff score. 12
  13. 13. Compromise methods• De Gruijter method – Incorporation of uncertainty measures. – Each judge provide estimates of his or her uncertainty with respect to the true value of the failing rate and the true value of the cutoff score. – Each judge plots their points in the ogive curve. – A family of ellipse is generated around the plots. – The ellipse that touches the ogive curve is the compromise cutoff score. 13