Concept of reliability in scale development

1. Reliability of Scale

2. Conceptual Definition of
Reliability
• Tells you how similar the scores would be if
the same person completed the scale twice
– Replicability
• Represents the precision of your scale
• Does not tell you if you are measuring the
right thing, but does tell you how well you
are measuring it
• Has a mathematical definition

3. Reliability in Terms of True and
Observed Scores
• You have greater reliability when you have
less random error in your measurements
error
Random
Scores
True
in
y
Variabilit
Scores
True
in
y
Variabilit
Scores
Observed
in
y
Variabilit
Scores
True
in
y
Variabilit
y
Reliabilit




4. Reliability in Terms of Variances
and Covariances
• You have greater reliability when the
communal variability (covariances) is large
relative to the noncommunal variability
(variances)




s
Covariance
Variances
1
y
Reliabilit

5. Reliability as Internal
Consistency
• Internal consistency refers to the relations
among the items in your scale
• The reliability of a scale is usually
estimated by its internal consistency
• Chronbach’s alpha is the most common
method of determining internal consistency

6. Chronbach’s alpha
• Primarily determined by the mean inter-
item correlation
– Higher correlations = higher alpha
• Also influenced by the number of items
– More items = higher alpha
 r
k
r
k
1
1 




7. Reliability and Power
• Analyses will be more powerful when they
use variables with greater reliability
– Lower reliability = more random error
– Random error can’t relate to anything
• Largest correlation you can find with a scale
= square root of its reliability
    y
x
xy
xy r
r 

true
observed 

8. Estimating Reliability Using
Parallel Measurements
• You can also estimate reliability as the
correlation of two parallel measurements
– Random variability of the two measurements
shouldn’t relate
– Correlation between two measures of the same
person should just represent variability related
to the latent factor

9. Estimating Reliability Using
Parallel Measurements
• Even though both internal consistency and
parallel measurements can be used as
estimates of reliability, they are slightly
different
– “Random error” in internal consistency
reliability represents measurement error
– “Random error” in parallel measurements
reliability represents measurement error and
change in the construct over time

10. Split-Half Reliability
• One special case of parallel measurements
is when you divide your scale into two sets
of items and treat the scores on each half as
a parallel measurement
• People commonly use odd/even split
– First half/second half not appropriate because
scores in the second half might be affected by
fatigue where scores in the first half will not

11. Split-Half Reliability
• Must apply a correction because each of
your parallel measurements has half the
number of items found in the full scale
• To do this:
1. Estimate the mean inter-item correlation using
the formula
 split
split
split
split
1 r
k
k
r
r




12. Split-Half Reliability
2. Calculate reliability using the formula
• Chronbach’s alpha is actually the average
of all possible split-halves
 r
k
r
k
1
1
y
reliabilit
full
full


