Slides from Mark Reckase's presentation on January 30, 2020

1. Translation of Policy to Cut-Score:
What Needs to be Considered?
Mark D. Reckase
Michigan State University

2. Standards Are
Common:
How are they
developed?
 Body mass index
 Impact on crash test dummies
 Nutritional information cereal
boxes
 Speed limits on highways.

3. Some of my History
 My Ph.D. is in psychology with emphasis on statistics and
measurement.
 For years, I avoided learning anything about standard setting.
 After 10 years as a faculty member, I went to work for ACT, Inc, a
large testing company known for college admissions testing.
 In 1992, ACT was contracted by the National Assessment Governing
Board (NAGB) to do standard setting on the National Assessment of
Educational Progress (NAEP).
 Despite my protests that I knew nothing about standard setting, I was
assigned to work on the contract.
 My knowledge of standard setting stems from learning on the job for
10 years of work on NAEP while at ACT.

4. Some Observations
 Different standard setting methods yield different results.
 This is troubling – it means an informed psychometrician can partially determine
the results.
 Policy boards sometimes decide that the standard set
through an accepted process is not right and adjustments
are made.
 Policy boards do not know what the correct standard is, but they know when it is
not consistent with their intentions.
 All standard setting processes are not equally good, but I
have not seen professional publications that indicate that
one approach is better than another one.
 Practitioners argue that there is no correct standard – they are a matter of
judgment.

5. Some Tentative
Conclusions
 Standard setting processes
are not equally good.
 All standard setting processes
are modified
 It is seldom that a “method” is
implemented in the same way twice.
 It is not true that all
standards are equally
credible.
 If that were true, policy boards would
not want to adjust the standards.
 There is an unobserved “intended
standard” and the process yields an
observed standard that is an estimate
of the intended standard.

6. These observations
and the need
to defend our work
led me to investigate
the credibility
of standard setting
methods.
But it is difficult
to do the
investigations
without a theory
of how standard
setting works.

7. There is no such thing as a
theory of standard setting!

8. Basic Concepts
for a Theory
of Standard Setting

9. Standard Setting
Is Not Standard Setting!
It is translation of statement of policy
to the language of the reporting score
scale.

10. A Theory of
Standard Setting
 Standard is set by the statement of
policy – the Policy Definition.
 The Policy Definition implies a continuum of
performance.
 The Policy Definition implies a level of
performance.
 The purpose of the standard
setting process is the translation of
the level in the policy to a point
on the reporting score scale that
operationalizes the implied
continuum.

11. The Policy Definition of the National
Council of State Boards of Nursing
 “Licensure is the process by which boards of nursing grant
permission to an individual to engage in nursing practice after
determining that the applicant has attained the competency
necessary to perform a unique scope of practice. Licensure is
necessary when the regulated activities are complex
and require specialized knowledge and skill and independent
decision making. The licensure process determines if the
applicant has the necessary skills to safely perform a specified
scope of practice by predetermining the criteria needed and
evaluating licensure applicants to determine if they meet the
criteria.”

12. The Implied
Continuum
 A test is developed or selected to
operationally define the continuum.
 Evidence for the validity of the test as
representing the policy is important.
 Test must have precision in the range
where the standard or standards are
likely to be.
 In some cases, tests have been revised to
support the accurate representation of a
standard.

13. Panelists Are Translators from Policy
to the Reporting Score Scale
The panelists are translating
policy not making policy.
To perform the translation,
panelists must know about
Policy
Content
Properties of the reporting score scale.

14. A Problem:
Policy Definitions Tend to Be Vague
 To make the definition more precise and
concrete, they are translated to content specific
definitions.
 Now this is often done early and then approved by the policy
agency.
 The content definition is often used to define the
minimum qualifications for a person above the
standard.
 Now this is often done through a job analysis.

15. A Problem:
Few Persons
Are Expert in
the Content
and the
Reporting
Score Scale
Can find persons who are expert
in the language of content.
Can find psychometricians that
are expert about the language of
the reporting score scale.
Few persons are expert in both
languages.

16. Panelist Selection and Training
•Content of the test
•The examinee population
Panelists need to be expert in
•The details of the policy
•The test and reporting score scale
•The methods to use for translation
Panelists need to be trained in

17. Schematic
for the
Theory
Agency
Policy
Definition
Content Specific
Definition
Hypothetical
Construct
Design and
Development of Test
for Construct
Description of
Minimally Qualified
Individual
Point on Reporting Score
Scale for Test Consistent
with Definitions
Standard
Setting Process
Agency Approves
Recommended Standard
Agency
Policy
Definition
Content Specific
Definition
Hypothetical
Construct
Design and
Development of Test
for Construct
Description of
Minimally Qualified
Individual
Point on Reporting Score
Scale for Test Consistent
with Definitions
Standard
Setting Process
Agency Approves
Recommended Standard

18. The Standard Setting Method
 The goal is to help panelists perform the
translation.
 The method needs to provide information that
can be used to estimate a point on the
reporting score scale.
 There are many methods, and variations of
each, and they are not equally good.

19. A Validity Argument for the Standard
Policy implies
continuum
Policy implies
continuum
Policy implies
level
Policy implies
level
Policy is
translated to
content specific
definition
Policy is
translated to
content specific
definition
Standard setting
method helps
panelists translate
policy to reporting
score scale
Standard setting
method helps
panelists translate
policy to reporting
score scale
Panelists are
capable of
performing the
necessary tasks
Panelists are
capable of
performing the
necessary tasks
Training and
feedback are
effective at
helping panelists
Training and
feedback are
effective at
helping panelists

20. Implications
of the Theory
 Policy definition should be carefully
crafted to support continuum and level
 Reporting score scale must be
consistent with policy
 Reporting score scale should be
accurate in the range containing the
standard
 Content definitions should be approved
by policy makers
 Panelists must be able to do the
translation and not be influenced by
external issues
 Estimation needs to be unbiased and
with small standard error

21. Reference
 Reckase, M. D. (2006). A conceptual framework
for a psychometric theory for standard setting
with examples of its use for evaluating the
functioning of two standard setting methods.
Educational Measurement: Issues and practices,
25(2), 4-18.

22. Questions?