What is validation in psychological test construction? What are the techniques could be used in doing it? (Lecture slides mostly in English)

1. PDU 303 PSIKOMETRIK
VALIDITY
SEMESTER GASAL 2015/2016
FAKULTAS PSIKOLOGI UNIVERSITAS KATOLIK INDONESIA ATMA JAYA
KULIAH VIII - 20 OKTOBER 2015

2. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
DEFINITION
Validation is the process by which a test developer
or test users collect the evidence to support
the types of inferences that are to be drawn
from test scores (Cronbach, 1971 in Crocker & Algina, 2008: 217).
Validity, as applied to a test, is a judgment or estimate
of how well a test measures what it purports to measure
in a particular context.
More specifically, it is a judgment
based on evidence about the appropriateness
of inferences drawn from test scores (Cohen & Swerdlik, 2009: 172).
2

3. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
MAJOR TYPES OF VALIDATION STUDIES
3
CONSTRUCT IDENTIFICATION CONTENT DESCRIPTION CRITERION-RELATED
A judgment about the
appropriateness of inferences
drawn from test scores
regarding individual standings
on a variable called a construct.
A judgment of how adequately a
test samples behavior
representative of the universe of
behavior that the test was
designed to sample.
A judgment of how adequately a
test score can be used to infer an
individual’s most probable
standing on some measure of
interest—the measure of
interest being the criterion.
(Cohen & Swerdlik, 2009: 176, 180, 193)ICON CREDITS: Magnifying Glass by Creative Stall, Tomography by Jakob Vogel,
Crystal Ball by Gerardo Martín Martínez; from the Noun Project

4. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
MAJOR TYPES OF VALIDATION STUDIES
4
CONSTRUCT IDENTIFICATION CONTENT DESCRIPTION CRITERION-RELATED
The test user desires to draw an
inference from the examinee’s
test score performances that can
be grouped under the label of a
particular psychological
constuct.
The test user desires to draw an
inference from the examinee’s
test score to a larger domain or
items similar to those on the test
itself.
The test user desires to draw an
inference from the examinee’s
test score to performance on
some real behavioral variable of
practical performance.
(Crocker & Algina, 2008: 217-218)
TYPES OF INFERENCES DRAWN
ICON CREDITS: Magnifying Glass by Creative Stall, Tomography by Jakob Vogel,
Crystal Ball by Gerardo Martín Martínez; from the Noun Project

5. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
CONSTRUCT IDENTIFICATION VALIDITY
5
(Crocker & Algina, 2008: 230-231)ICON CREDIT: Magnifying Glass by Creative Stall;
from the Noun Project

6. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
AT THE MINIMUM,
CONSTRUCT
VALIDATION ENTAILS
THE FOLLOWING
STEPS:
Select (or develop) a measurement
instrument which consists of items
representing behaviors of the construct;
Gather empirical data which will permit
the hypothesized relationships to be
tested;
Formulate one or more hypotheses about
how those who differ on the construct are
expected to differ;
CONSTRUCT IDENTIFICATION VALIDITY
5
Determine if the data are consistent with
the hypotheses.
1
2
3
4
(Crocker & Algina, 2008: 230-231)ICON CREDIT: Magnifying Glass by Creative Stall;
from the Noun Project

7. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
CONSTRUCT IDENTIFICATION VALIDITY
6
EXTERNAL
CRITERIA
CORRELATION
WITH OTHER
TESTS
PROCE-
DURES
INTERNAL
CRITERIA
CONVERGENT &
DISCRIMINANT
VALIDATION
FACTOR
ANALYSIS
INTERNAL
CONSISTENCY
CORRELATION
BETWEEN A
CONSTRUCT &
DESIGNATED
THE
MULTITRAIT-
MULTIMETHOD
MATRIX
(Anastasi & Urbina, 1997: 127-132;
Crocker & Algina, 2008: 231-235)ICON CREDIT: Magnifying Glass by Creative Stall;
from the Noun Project

8. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
CONTENT DESCRIPTION VALIDITY
7
(Crocker & Algina, 2008: 218)ICON CREDIT: Tomography by Jakob Vogel;
from the Noun Project

9. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
AT THE MINIMUM,
CONTENT VALIDATION
ENTAILS THE
FOLLOWING STEPS:
Selecting a panel of qualified experts in
the content domain;
Providing a structured framework for the
process of matching items to the
performance domain;
Defining the performance domain of
interest;
CONTENT DESCRIPTION VALIDITY
7
Collecting and summarizing the data from
the matching process.
1
2
3
4
(Crocker & Algina, 2008: 218)ICON CREDIT: Tomography by Jakob Vogel;
from the Noun Project

10. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
Should objectives be weighted to
reflect their importance?
CONTENT DESCRIPTION VALIDITY
8
(Crocker & Algina, 2008: 219-222)
How should the item-matching task
be structured?
What aspects of the item should be
examined?
How should the results be
summarized?
PRACTICAL CONSIDERATIONS
ICON CREDITS: Tomography by Jakob Vogel,
Exclamation by Shawn Schmidt; from the Noun Project

11. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
KLEIN & KOSECOFF (1975)KATZ (1958) & EBEL (1956)
The expert reviewer read the item
and identify the correct response,
just as an examinee would.
Have each item written on
separate cards; compare each
item to the list of instructional
objectives; and record the
outcome of the matching decision
on a standard form:
Either the item matches or not.
The experts rate the degree of
match to a specified objective by
using a five-point scale, where 1
indicates a poor fit and 5 an
excellent fit.
CONTENT DESCRIPTION VALIDITY
9
(Crocker & Algina, 2008: 219)
THE QUANTIFICATION
HAMBLETON (1980)
ICON CREDIT: Tomography by Jakob Vogel;
from the Noun Project

12. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
CONTENT DESCRIPTION VALIDITY
One method of measuring content validity, developed by C. H. Lawshe (1975),
is essentially a method for gauging agreement among raters or judges
regarding how essential a particular item is.
10
(Cohen & Swerdlik, 177-178)
THE QUANTIFICATION
CVR =
ne - (N/2)
N/2
NUMBER OF PANELIST MINIMUM VALUE
5 0,99
6 0,99
7 0,99
8 0,75
9 0,78
10 0,62
ICON CREDIT: Tomography by Jakob Vogel;
from the Noun Project

13. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
CRITERION-RELATED VALIDITY
11
(Crocker & Algina, 2008: 224)ICON CREDIT: Crystal Ball by Gerardo Martín Martínez;
from the Noun Project

14. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
AT THE MINIMUM,
CRITERION
VALIDATION ENTAILS
THE FOLLOWING
STEPS:
Identify an appropriate sample of examinees
representative of those for whom the test will
ultimately be used;
Administer the test and keep a record of each
examinee’s score;
Identify a suitable criterion behavior and a
method for measuring it;
CRITERION-RELATED VALIDITY
11
Determine the strength of the relationship
between test scores and criterion
performance.
1
2
3
(Crocker & Algina, 2008: 224)
4
5
When the criterion data are available, obtain a
measure of performance on the criterion for
each examinee;
ICON CREDIT: Crystal Ball by Gerardo Martín Martínez;
from the Noun Project

15. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
RELIABILITY OF P & C
SAMPLE SIZETHE CRITERION PROBLEM
RESTRICTION OF RANGE
CRITERION CONTAMINATION
There must be a judicious tradeoff in
selecting a criterion which:
1. can be reliably measured within the
time and cost constraints of the
study;
2. will have a relationship to the
ultimate criterion of interest in most
test users.
When validity coefficients are estimated
for small samples, sampling errors are
relatively large, and the statistical power
of the inferential procedures used to
analyze data from the validation study
may be substantially reduced.
During a validation study persons who
will be able to influence examinees’
criterion scores should not have access
to examinees’ scores on the predictor.
Restriction of the range of scores on
either the redictor or criterion may result
in attenuation of the observed validity
coefficient.
In any attempt to asses the degree of
relationship between a predictor and
criterion, it would seem logical that
errors of measurement should be held
minimum.
CRITERION-RELATED VALIDITY
12
PRACTICAL PROBLEMS
(Crocker & Algina, 2008: 225-228)ICON CREDIT: Crystal Ball by Gerardo Martín Martínez;
from the Noun Project

16. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK
CRITERION-RELATED VALIDITY
13
THE QUANTIFICATION
CONCURRENT PREDICTIVE
The test are administered to a
group on whom criterion data
are already available.
Prediction from the test to any
criterion situation, or in the more
limited sense of prediction over a
time interval.
(Anastasi & Urbina, 1997: 118-119)ICON CREDIT: Crystal Ball by Gerardo Martín Martínez;
from the Noun Project
2.1.

17. FAKULTAS PSIKOLOGI
UNIVERSITAS KATOLIK INDONESIA ATMA JAYA © 2015 Ardhiati PDU 303 PSIKOMETRIK 14
WANT TO KNOW MORE?
ANASTASI, A. & URBINA, S. (1997). PSYCHOLOGICAL TESTING (7TH ED.). UPPER
SADDLE RIVER, NJ: PRENTICE-HALL, INC.
COHEN, R.J. & SWERDLIK, M.E. (2009). PSYCHOLOGICAL TESTING AND
ASSESSMENT: AN INTRODUCTION TO TESTS AND MEASUREMENT (7TH ED.).
BOSTON, MA: MCGRAW-HILL HIGHER EDUCATION.
CROCKER, L. & ALGINA, J. (2008). INTRODUCTION TO CLASSICAL AND
MODERN TEST THEORY. MASON, OH: CENGAGE LEARNING.