The Art and Science of Applied Test Development. This is the fifth in a series of PPT modules explicating the development of psychological tests in the domain of cognitive ability using contemporary methods (e.g., theory-driven test specification; IRT-Rasch scaling; etc.). The presentations are intended to be conceptual and not statistical in nature. Feedback is appreciated.
Applied Psych Test Design: Part E--Cacluate norms and derived scores
1. The Art and Science of Test DevelopmentβPart E
Calculate norms and derived scores
Kevin S. McGrew, PhD.
Educational Psychologist
Research Director
Woodcock-MuΓ±oz Foundation
The basic structure and content of this presentation is grounded extensively on the test
development procedures developed by Dr. Richard Woodcock
2. How do we construct age-based norms from
standardization norm data?
Answer: Curve fitting of sorted subsample data points is
the engine that drives the development of all derived scores
3. These Block 546
Rotation W-scores Block Rotation
are then used for Summary: Final
developing test Rasch for
βnormsβ and Publication test β
validity research graphic item map
n = 37 norming
items (0-74 RS
points)
ο n = 4,722 norm
ο
οο subjects
οο
οο
οο
οο
ο
οοο
οοο
οο ο ο
οοο
οο
οοο ο ο
ο οο οο
ο
Graphic display of
distribution of Block
Rotation person abilities
Pub. Test
W-score
scale
432
4. 1 6 11 WJ III βclassicβ norm calculation procedures
Age Age Age
W W W
2 7 12
Age Age Age
W W W
(each ball represents an individual norm subject)
3 8 13
Age Age Age
W W W n =8,000+ norm subjects
4 9 β¦
Age Age Age
W W W
5 10 8,000
Age Age Age
W W W
1. Sort 8,000 subjects from youngest (CA in months) to oldest
Oldest
Youngest
β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦β¦
Mdn CA x1 x2 β¦β¦..
2. Divide sorted pool of subjects in successively older blocks of n=50
Mdn W y1 y2 β¦β¦..
3. Calculate βweightedβ (US Census derived subject weights)
median (average) CAMOS (X) and REF W (Y) for each block
4. Plot mdn CAMOS (x1, x2,..) and REF W (y1, y2β¦) and smooth curve
5. 550 550
Example:
Letter-Word ID Ref W (20-120 months) raw data points
500 500
450 450
400 400
350 Each data point is a βsampleβ that 350
contains βsampling errorβ --- this
accounts for the βbounceβ between
data points. How do we deal with
300 this sampling error (bounce) to 300
construct norms and derived
scores?
250 250
20 40 60 80 100 120
6. Letter-Word ID Ref W (20-120 months) polynomial curve generated solution
(using special curve fitting software)
550 550
500 500
450 450
400 400
The smoothed curve
350 350
represents the best
approximation of the
300 population average norm 300
W-score for a test
(Reference W or REF W)
250 250
20 40 60 80 100 120
7. Obtaining Developmental Scores (age/grade equivalents)
A W-score of 450 (for
Letter-Word Identification
test) = 2.4 grade equivalent
A W-score of 400 = 1.3
grade equivalent
Smoothed age curves are
used in the same manner to
obtain age equivalents
8. Developing norms and derived scores: What does a tested personβs score
on a test mean when compared to the appropriate reference group (age
norms will be used as example)
The meaning of a Block
ο ο ο ο Rotation W-score of X
οο
ο οο
ο οο
ο οο
ο
οο
ο
ο οο
ο
ο οο
ο
ο οο
ο
ο (e.g., 477) will have
οο
οο
ο οο
οο
ο οο
οο
ο οο
οο
ο different interpretations
οο
οο
οο οο
οο
οο οο
οο
οο οο
οο
οο when compared to
οοο
οοο
οοο οοο
οοο
οοο οοο
οοο
οοο οοο
οοο
οοο
ο οο ο
ο οο ο
οο ο ο οο ο
ο οο ο
οο ο ο οο ο
ο οο ο
οοο ο οο ο
ο οο ο
οο ο different age group norm
subsamples
2 yr olds 3 yr olds 4 yr olds 5 yr olds
Measures of relative
standing (percentile
rank, standard score)
derive meaning based
on how far away the
431.6 477 545.7 personβs W-score is
from average (for age)
BBB
Block Rotation W-scale
9. Obtaining Measures of Relative Standing: A subjects W-score for a
specific measure is compared to the average W-scores for that subjects
specific age (age norms) or grade (grade norms). This is called the
Reference W (REF W)
Expected βaverage REF Wβ
for someone tested at grade
3.0 (grade norms) is 472.5
(obtained score of 472.5
would be SS=100; PR=50)
10. Obtaining Scores of Relative Standing: Subjects obtained W-score for a specific
measure is compared to the distribution (mean and SD) of W-scores for that
subjects specific age (age norms) or grade (grade norms)
βMeanβ is the
smoothed βRef Wβ
value for a specific
age/grade
βSDβ is the smoothed
SD (10/90) for a
specific age/grade
SS (M=100; SD=15) = (z x 15) + 100
β’ e.g. z = -1; SS = 85
11. X Y
Custom software generated
norm βsetupβ data file
example
(Block Rotation)
Input for graphing and
polynomial curve fitting
X Y
Note: These examples are from
original WJ III 2001 norms and
not the subsequent WJ III NU
(2007) norms
12. Original Block Rotation Reference W age-based curve fitting: A real-
world example of the βart + scienceβ of constructing norms
r^2=0.12670607 Eqn 8160 Line(a,b) Robust None
7667WLO Eqn 7667 Chebyshev=>Std Rational Order 8/9
Block Rotation Ref-W Age
510 510
505 505
500 500
495 495
490 490
Ref-W
Ref-W
485 485
480 480
475 475
470 470
465 465
460 460
12 120 1200
Age (in months)
Solution A: Up to 230 months (note:
age scale is a log scale)
Note: These examples are from original WJ III 2001 norms and not the subsequent WJ III NU (2007) norms
13. Original Block Rotation Reference W age-based curve fitting: A real-
world example of the βart + scienceβ of constructing norms
r^2=0.12670607 Eqn 8160 Line(a,b) Robust None
6870WHI Eqn 6870 Chebyshev=>Std Polynomial Order 20
Block Rotation Ref-W Age
510 510
505 505
500 500
495 495
490 490
Ref-W
Ref-W
485 485
480 480
475 475
470 470
465 465
460 460
12 276 540 804 1068
Age (in months)
Solution B: 231 to 1200 months
(note: age scale is regular interval
scale--not log scale)
Note: These examples are from original WJ III 2001 norms and not the subsequent WJ III NU (2007) norms
14. Original Block Rotation Reference W age-based curve fitting: A real-
world example of the βart + scienceβ of constructing norms
Curve solution A βfeathered/blendedβ with Curve Solution B at 230
months for single final solution. Sometimes more than 2 curve parts are
needed for age norms.
Note: These examples are from original WJ III 2001 norms and not the subsequent WJ III NU (2007) norms
15. Final smoothed curves serve
as the mechanism for the
published norms, either in the
form of equations in software
Or,
Note: These examples are from
original WJ III 2001 norms and
not the subsequent WJ III NU
(2007) norms
16. Age Reference
(in months) W
Age Reference
(in months) W
Tables of values for
published norms in
test manuals
Note: These examples are from
original WJ III 2001 norms and etcβ¦β¦
not the subsequent WJ III NU
(2007) norms
17. X Y
Custom software generated
norm βsetupβ data file
example
(Block Rotation)
Input for graphing and
polynomial curve fitting
X Y
Note: These examples are from
original WJ III 2001 norms and
not the subsequent WJ III NU
(2007) norms
18. Original Block Rotation SD90 age-based curve fitting: A real-world
example of the βart + scienceβ of constructing norms
Block Rotation SD90 Age
Rank 2502 Eqn 7938 y=(a+cx^(0.5)+ex+gx^(1.5)+ix^2)/(1+bx^(0.5)+dx+fx^(1.5)+hx^2+jx^(2.5)) [NL]
r^2=0.48235094 DF Adj r^2=0.41998358 FitStdErr=1.6978814 Fstat=8.6968999
a=15.791894 b=0.66619087 c=1.7270779 d=-0.2462822 e=-1.0287721
f=0.02543265 g=0.082451267 h=-0.00095281528 i=-0.0010522608 j=1.5044367e-05
15 15
13 13
11 11
9 9
SD90
SD90
7 7
5 5
3 3
1 1
12 120 1200
Age (in months)
Same is done for SD 10
Note: These examples are from original WJ III 2001 norms and not the subsequent WJ III NU (2007) norms
19. Original Block Rotation SD90 age-based curve fitting: A real-world
example of the βart + scienceβ of constructing norms
Block Rotation SD90 Age
Rank 2502 Eqn 7938 y=(a+cx^(0.5)+ex+gx^(1.5)+ix^2)/(1+bx^(0.5)+dx+fx^(1.5)+hx^2+jx^(2.5)) [NL]
r^2=0.48235094 DF Adj r^2=0.41998358 FitStdErr=1.6978814 Fstat=8.6968999
a=15.791894 b=0.66619087 c=1.7270779 d=-0.2462822 e=-1.0287721
f=0.02543265 g=0.082451267 h=-0.00095281528 i=-0.0010522608 j=1.5044367e-05
15 15
13 13
11 11
9 9
SD90
SD90
7 7
5 5
3 3
1 1
12 276 540 804 1068
Age (in months)
Same is done for SD 10
Note: These examples are from original WJ III 2001 norms and not the subsequent WJ III NU (2007) norms
20. Final smoothed
curves serve as
the mechanism
for the published
norms, either in
the form of
equations in
software
Or,
Note: These examples are from
original WJ III 2001 norms and
not the subsequent WJ III NU
(2007) norms
21. Age SD (in
(in months) W units)
Age SD (in
(in months) W units)
Tables of values for
published norms in
test manuals
Note: These examples are from
original WJ III 2001 norms and etcβ¦β¦
not the subsequent WJ III NU
(2007) norms
22. Obtaining Scores of Relative Standing: Subjects W-score for a specific
measure is then compared to the distribution of W-scores for that subjects
specific age (age norms) or grade (grade norms)
Smoothed SD90
Smoothed REF W (average)
Note: These are NOT
the curves for Block
Rotation. They are from
another measure. Used
here as example
Smoothed SD10
23. Use of bootstrap re-sampling methods in curve fitting
Special proprietary iterative curve fitting Q/A procedures for selecting
best possible curve from a pool of plausible curves
Different subject weighting procedures
Calculating cluster norms (combinations of tests)
Calculating differentially weighted cluster norms (e.g., WJ III GIA
cluster)
Calculating discrepancy norms
Special test-cluster consistency checks and procedures
Creating special Rasch (W-score) based interpretative scoring options
and features (e.g., RPI, instructional ranges) β explained in separate
PPT module
Special test-length correction procedures for calculation of reliabilities
and correlations
24. With publication of WJ III NU norms, we now use bootstrap
generated βsticksβ and not raw single data points
25.
26. WJ III NU boostraping: If you really want to know check out ASB9
27. WJ III NU boostraping: If you really want to know check out ASB9
28. WJ III NU boostraping: If you really want to know check out ASB9
29. End of Part E
Additional steps in test development process will be
presented in subsequent modules as they are developed