vertical Moderate Standard setting

1. 1
Presented By: Tariq Mehmood
Presented To : Dr Muhammad Iqbal
University of Education
Lahore

2. After completion of this unit Students will be able to :
 Define vertically Moderated Standard setting.
 Describe the history of VMSS.
 Elaborate Approaches of VMSS.
 Describe Application of VMSS..
2

3. vertically-moderated standard setting(VMSS)
sometimes referred to as a process of vertical articulation of
standards—is a new and developing concept.
3

4.  Simply VMSS is a procedure or set of procedures, typically
carried out after individual standards have been set.
 Reasonable expectations are typically stated in terms of
percentages of students at or above a important performance
level, such as Proficient. To illustrate VMSS, consider the
following scenario for a typical state testing program. Six
groups of standard setters have gathered to set standards for
the state mathematics tests in Grades 3–8.
4

5. Table 14-1 Hypothetical Results of Independent
Standard Settings Across Six Grade Levels
5
Grade Percentage of Students Classified as At or
Above Proficient Performance Level
3 37
4 41
5 34
6 43
7 29
8 42

6.  Given the placement of the cut scores for the Proficient
category, percentages of students that would be classified
as Proficient or above are shown, by grade level, in Table
14-1.
 In this example, we see that the percentages of students
considered Proficient or better (e.g., Proficient or
Advanced), goes up from Grade 3 to Grade 4, drops back
down at Grade 5, goes up again at Grade 6, drops again at
Grade 7, and rises again at Grade 8.
6

7.  Each of the methods of setting performance standards we
described is routinely applied to contexts in which the need is for a
single cut score or set of performance levels for a single test.
 These contexts have included setting a single cut score on a
licensing examination, deriving three performance levels (e.g.,
Basic, Proficient, Advanced) on a single end-of-grade subject area
test, and obtaining multiple cut scores on a set of four or five
different subject tests used to determine eligibility for high school
graduation.
 In each instance, there was no apparent need to link cut scores on
one test to those on another.
7

8.  The challenges to creating meaningful systems of performance
standards across grades and subject areas.
 One approach has been proposed by Lissitz and Huynh
(2003), who introduced the concept of VMSS in a background
paper prepared for the Arkansas Department of Education.
 The paper, titled “Vertical Equating for the Arkansas
ACTAAP Assessments: Issues and Solutions in Determination
of Adequate Yearly Progress and School Accountability,”.
8

9.  As an alternative to vertical scaling (equating), Lissitz and
Huynh (2003b) recommended VMSS. Specifically, they
recommended that “new cut scores for each test be set for all
grades such that
 each achievement level has the same (generic) meaning across
all grades, and
 the proportion of students in each achievement level follow a
growth curve trend across these grades.” They then offered a
list of specific recommendations to carry out the vertical
moderation of a set of performance standards across Grades 3–
8, including an annual validation of the process.
9

10.  It remains to be seen if VMSS will provide satisfactory
solutions to the current contexts to which it is being applied, as
well as how adaptable VMSS will be to other contexts.
10

11.  Simply VMSS is a procedure or set of procedures,
typically carried out after individual standards have
been set.
 Reasonable expectations are typically stated in terms
of percentages of students at or above a important
performance level, such as Proficient. To illustrate
VMSS, consider the following scenario for a typical
state testing program. Six groups of standard setters
have gathered to set standards for the state
mathematics tests in Grades 3–8.
11

12. VMSS typically focuses on percentages of
students at various proficiency levels.
A simple solution to the problem of different
percentages of students reaching a given
performance level—say, the Proficient cut score—
at different grades would simply be to set all
standards at the same score point or such that
equal percentages of students would be classified
as Proficient at each grade level, by command.
12

13. An alternative would be to set standards only for the lowest
and highest grades and then align the percentages of
Proficient students in the intermediate grades accordingly. In
the preceding
example, we would simply take the 37% figure for Grade 3
and the 42% figure for Grade 8 and set cut scores for Grades
4–7 so that their resulting percentages of students at or
above Proficient would fall on a straight line between 37%
and 42% as shown in Table 14-2
13

14.  . If complete standard-setting procedures were used for
establishing cut scores for Grades 3 and 8, then there is
some reason to have confidence in at least the 37% and
42% values for the anchor grades.
14

15. Table 14-2 Results of Smoothing Standard-
Setting Results Across Six Grade Levels
15
Grade Percentage of Students Classified as At or
Above Proficient Performance Level
3 37
4 38
5 39
6 40
7 41
8 42

16. According to these authors, assumptions for standard setting are
based on the intersection of three growth models and four
expected growth amounts as depicted in Table 14-3.
The three growth models may be summarized as follows: (1)
Linear growth, which assumes that the proficiency of all
examinees increases by a fixed amount, and examinees retain
their positions relative to one another;
16

17.  (2) Remediation, which assumes that the proficiency
of examinees at the lower end of the score
distribution increases more than those of examinees
at the upper end; and (3) Acceleration, which
assumes that the proficiency of examinees in the
upper portion of the score distribution increases at a
greater rate than that of examinees at the lower end of
the score distribution.
17

18. 18

19. Because of the newness of VMSS, only recently have results of
VMSS approaches been presented in the psychometric literature.
As we noted
19
Growth
Trajectori
es
Growth Model Types
Linear Remediation Acceleration
Negative
growth
All groups
show
decline
over time.
Overall group
shows
negative
growth, but
remedial
group fares
better than
non remedial
group.
Overall group
shows negative
growth, but
non remedial
group fares
better than
remedial
group.

20. 20
Growth
Trajectori
es
Growth Model Types
Linear Remediation Acceleration
No growth All groups
show no
growth
over time.
Overall group
shows no
growth, but
remedial
group fares
better than
non remedial
group.
Overall group
shows no
growth, but
non remedial
group fares
better than
remedial
group.

21. 21
Growth
Trajectori
es
Growth Model Types
Linear Remediation Acceleration
Low
growth
All groups
show low
growth
over time.
Overall group
shows low
growth, with
most gain
coming from
remedial
group.
Overall group
shows low
growth, with
most gain
coming from
non remedial
group.

22. 22
Growth
Trajectorie
s
Growth Model Types
Linear Remediation Acceleration
Moderate
growth
All groups
show
moderate
growth
over time.
Overall group
shows
moderate
growth, with
most gain
coming from
remedial group.
Overall group
shows moderate
growth, with
most gain
coming from
nonremedial
group.