Reliability and validity

1. UCE
Reliability and Validity
• Reliability – Are we measuring
accurately?
• Validity – Are we measuring the
right thing?

2. UCE
Reliability
• The extent to which an
experiment or measuring
procedure yields the same
result on repeated trials
– repeatedly used by the one researcher
– used once by different researchers

3. UCE
Reliability - example
• Measurement of body
temperature at different
altitudes
• The thermometer is calibrated
using boiling water

4. UCE
Reliability - example
• Measurement of body
temperature at different
altitudes
• The thermometer is calibrated
using boiling water
2000 m 93.4 C
4000 m 87.3 C
6000 m 81.3 C
8000 m 75.5 C

5. UCE
Importance of Reliability
• Replication of research by
independent observers is
essential so that conclusions
can be made about the
generalizability of findings.

6. UCE
Validity
• The degree to which a study
accurately assesses the
specific concept or parameter
that the researcher is trying to
measure.
• whether the research has
measured what it set out to
measure

7. UCE
Validity - Example
• Days off work with sickness
used as a measure of general
health.

8. UCE
Types of Reliability
• Equivalency reliability
• Stability reliability
• Internal consistency
• Interrater reliability

9. UCE
Equivalency reliability
• The extent to which two items
measure identical concepts at
an identical level of difficulty
• Can be tested using a
correlation coefficient

10. UCE
Equivalence reliability -
Example
• Estimation of fetal weight
• Measurement of exposure to
radiation

11. UCE
Stability reliability
Test re-test reliability
• The agreement of measuring
instruments over time
• To determine stability a test is
repeated on the same subject at
a future date

12. UCE
Stability reliability -
example
• Measuring instruments such as
scales should be checked
regularly and re-calibrated
• Tests and questionnaires should
be checked for consistency
over time

13. UCE
Internal consistency
• The extent to which tests
assess the same characteristic
or quality. It is a measure of the
precision between different
observers or instruments.

14. UCE
Internal consistency -
example
• A questionnaire includes a
number of questions on anxiety
• Analyzing the internal
consistency of these questions
will tells us which ones focus
on anxiety

15. UCE
Interrater reliability
• The extent to which two or
more individuals (scorers or
raters) agree
• Addresses the consistency of a
scoring system.

16. UCE
Interrater reliability -
example
• A scoring system 1-5 is used to
assess a students competence
in performance clinical
examination.
• Do different scorers award the
same mark?

17. UCE
Types of Validity
• Internal
• External

18. UCE
Types of Validity
• Internal
– Content validity
– Face validity
– Criterion validity
– Construct validity
• External

19. UCE
External Validity
• The extent to which the results
of a study are generalizable or
transferable
• How close do the controlled
conditions of the laboratory
match those of real life

20. UCE
External Validity
• To have strong external validity
you need a random sample of
subjects drawn from a clearly
defined population.
• Ideally, you will have a good
sample of groups and a sample
of measurements and situations

21. UCE
Internal Validity
• The rigor with which the study
was conducted (study design,
care with measurements,
decisions on what to measure)
• Consideration of alternative
explanations for any causal
relationships explored

22. UCE
Internal Validity
• A study is internally valid when the
the results or effects on the
dependent variable are attributable
to the independent variable and not
to other factors.
• How well these other factors are
controlled is related to the internal
validity of the study.

23. UCE
Internal Validity
• Content validity
• Face validity
• Criterion validity
• Construct validity

24. UCE
Content validity
• Involves assessing whether the
questions asked and the
measurements recorded
adequately represent the
domain(s) being studied
• Is the questionnaire
comprehensive?

25. UCE
Face validity
• How a measure or procedure
appears
• Involves using recognised
experts to judge the
appropriateness of the
approach and instrument to be
used for data collection

26. UCE
Criterion validity
• Demonstrating the accuracy of
a measure or procedure by
comparing it with another
procedure which has already
been demonstrated to be valid

27. UCE
Construct validity
• The most complex type of validity, and
involves relating an instrument for data
collection to a theoretical framework
• For example, a researcher inventing a
new IQ test might spend a great deal of
time attempting to "define" intelligence
in order to reach an acceptable level of
construct validity.

28. UCE
Threats to Reliability
• Unstable/ unrepeatable
measurements
• Bias
• Small sample size

29. UCE
Threats to validity
• Internal
• External

30. UCE
Threats to Internal
Validity
• Selection bias
• History
• Unreliable measurements and
procedures
• Small sample size
• Order effects

31. UCE
Threats to External
Validity
• Population validity
• Ecological validity

32. UCE
Summary
• Positivist research
• Reliability
• Validity
• Threats to reliability
• Threats to viability

33. UCE