This document discusses validity, reliability, and sampling in research. It defines reliability as consistency of measurement, and validity as measuring what is intended. There are different types of reliability including test-retest, internal consistency, and inter-rater. Validity includes construct, criterion, and different types. Threats to internal and external validity are outlined. The document also discusses population versus sample in research and different sampling methods like random, stratified, and convenience sampling.

1. Validity, Reliability, &
Sampling

2. Errors in measurement
 Reliability
 If you measure the same thing twice do you get the same
values?
 Validity
 Does your measure really measure what it is supposed to
measure??
reliable
valid
reliable
invalid
unreliable
invalid

3. Reliability
 True score + measurement error
 A reliable measure will have a small amount of
error
 Multiple “kinds” of reliability
• Test-retest
• Internal consistency
• Inter-rater

4. Reliability
 Test-restest reliability
 Test the same participants more than once
• Measurement from the same person at two different times
• Should be consistent across different administrations
Reliable Unreliable

5. Reliability
 Internal consistency reliability
 Multiple items testing the same construct
 Extent to which scores on the items of a measure
correlate with each other
• Cronbach’s alpha (α)
• Split-half reliability
• Correlation of score on one half of the measure with
the other half (randomly determined)

6. Reliability
 At least 2 raters observe behavior
 Inter-rater reliability
 Extent to which raters agree in their observations
• Are the raters consistent?
 Requires some training in judgment

7. Validity
 Does your measure really measure what it is
supposed to measure?
 There are many “kinds” of validity

8. VALIDITY
CONSTRUCT
CRITERION-
ORIENTED
DISCRIMINANT
CONVERGENT
PREDICTIVE
CONCURRENT
FACE
INTERNAL EXTERNAL
Many kinds of Validity

9. VALIDITY
CONSTRUCT
CRITERION-
ORIENTED
DISCRIMINANT
CONVERGENT
PREDICTIVE
CONCURRENT
FACE
INTERNAL EXTERNAL
Many kinds of Validity

10. Construct Validity
 Usually requires multiple studies, a large body
of evidence that supports the claim that the
measure really tests the construct

11. Face Validity
 At the surface level, does it look as if the
measure is testing the construct?
“This guy seems smart to me,
and
he got a high score on my IQ measure.”

12. Internal Validity
 Did the change in the
DV result from the
changes in the IV or
does it come from
something else?
 The precision of the results

13. Threats to internal validity
 History – an event happens the experiment
 Maturation – participants get older (and other changes)
 Selection – nonrandom selection may lead to biases
 Mortality – participants drop out or can’t continue
 Testing – being in the study actually influences how the
participants respond
 The precision of the results

14. External Validity
 Are experiments “real life” behavioral situations,
or does the process of control put too much
limitation on the “way things really work?”

15. External Validity
 Variable representativeness
 Relevant variables for the behavior studied
along which the sample may vary
 Setting representativeness
 Are the properties of the research setting
similar to those outside the lab (Ecological validity)
 Subject representativeness
 Characteristics of sample and target
population along these relevant variables

16. Sampling
 Why do we do we use sampling methods?
 Typically don’t have the resources to test everybody,
so we test a subset

17. Sampling
Population
Everybody that the
research is
targeted to be
about
The subset of the
population that
actually
participates in the
research
Sample

18. Sampling
Sample
Inferential
statistics used
to generalize
back
Sampling to
make data
collection
manageable
Population

19. Sampling
 Why do we do we use sampling methods?
 Goals of “good” sampling:
– Maximize Representativeness:
– To what extent do the characteristics of
those in the sample reflect those in the
population
– Reduce Bias:
– A systematic difference between those in
the sample and those in the population

20. Sampling Methods
 Probability sampling
 Simple random sampling
 Systematic sampling
 Stratified sampling
 Non-probability sampling
 Convenience sampling
 Quota sampling
Have some element of
random selection
Susceptible to biased
selection

21. Simple random sampling
 Every individual has a equal and independent
chance of being selected from the population

22. Systematic sampling
 Selecting every nth person

23. Stratified sampling
 Step 1: Identify groups (strata)
 Step 2: randomly select from each group

24. Convenience sampling
 Use the participants who are easy to get

25. Quota sampling
 Step 1: identify the specific subgroups
 Step 2: take from each group until desired number
of individuals