Dr. Eman M. Mortada discusses threats to validity in experimental designs, including internal validity threats such as history, maturation, testing, instrumentation, and mortality. External validity threats include reactive arrangements like the Hawthorne effect and experimenter effects such as the halo effect. Control validity threats involve factors that could influence the dependent variable other than the independent variable, such as selection bias, statistical regression, and diffusion of treatment. Randomization, control groups, and blinding techniques can help address threats to validity. True experiments have higher internal but lower external validity compared to quasi-experimental designs.

1. Dr. Eman M. Mortada
Professor
In public health and preventive medicine
Validity Threats in experimental
designs

2. Experimental validity
internal validity threats
external validity threats
Control validity threats
Lecture Outline

3. Experimental Research
 Experimental designs are often touted as the "gold standard"
against which all other designs are judged.
Dr. Eman M.Mortada

4. Experimental Research
 Purpose
 To make causal inferences about the relationship between
the independent and dependent variable
Cause Effect
Dr. Eman M.Mortada

5. The major feature that distinguishes experimental
research from other types of research
 Characteristics
 Direct manipulation of the independent variable
 Control of extraneous variables
◼ Eliminate the variable from the study
◼ Statistically adjust for the effect of the variable
Dr. Eman M.Mortada

6. Three Criteria for Causation
❑ Concomitant variation (correlation):
Statistical relationship between two variables. The
cause and effect must be correlated with each other.
❑ Appropriate time ordre of occurrence
( temporality):
Change in an independent variable occured before
an observed change in the dependent variable.
❑ Elimination of other alternative explantations &
possible causal factor:
The correlation between cause and effect cannot be
explained by another variable.
Dr. Eman M.Mortada

7. Normal
lung
Cancer
Cause Effect
then
Time
Temporality means that:
Cause must precede effect (disease ) in time
Lack of temporality rules out causality
Does the cause precede the effect?
Dr / Eman Mortada
Temporality (time order)

8. Extraneous variable
 An extraneous variable is a variable other
than the independent variable that might be
affecting the dependent variable causing a
change in the dependent variable
Dr. Eman M.Mortada
Makes it difficult to determine if any change in the
DV was caused by only the IV.
Their influence must be controlled by:
❑ Randomization
❑ Statistical control
❑ Design control

9. Experimental Designs Should be
Developed to Ensure Internal and
External Validity of the Study

10. Levels of experimental Control
 Least control
 Pre-experimental designs
 More control
 Quasi-experimental designs
 Most control
 True experimental designs

11. Validity
 From validus, meaning “strong & effective”
Test
validity
Face
validity
Content
validity
Construct
validity
Criterion
validity
Experime
ntal
validity
Internal
validity
External
validity

12. Experimental Validity
 Refers to the degree with which
correct inferences can be made
from the results of a research study
 Two types:
(1) Internal Validity and
(2) External Validity.

13. Internal Validity
 The extent to which the independent variable, and not other
extraneous variables, produce the observed effect on the
dependent variable
 Threatened by
 Confounding variables
 Strengthened by
 Adding adequate controls to reduce or eliminate confounding

14. Designs & level of internal validity
Dr. Eman M.Mortada
pre
Quasi
true
Low validity
High validity  True Experiment is
probably the strongest
design with respect to
internal validity
 Quasi-experimental, and
pre-experimental studies
are especially susceptible
to threats to internal
validity

15. ❑ AKA Generalizability
❑ External validity: extent to which we can generalize the results
of a research study to people, settings, times, measures, and
characteristics other than those used in that study.
❑ Concerned with real-world applications
❑ Threatened by
Unrepresentative samples
❑ Strengthened by
Gathering a representative sample (if possible)
External validity

16. Random selection  Random assignment
External
validity
control
Internal
validity
control
Random assignment vs Random
selection of subjects

17. Trade-off between internal &external
validity.
Dr. Eman M.Mortada
 Internal and external validity are related reciprocally
 The high control and internal validity often mean a
reduction of external validity.
 Experimental research allows us to test hypotheses and infer
causality under controlled conditions designed to maximize
internal validity.

18. the more precise, constrained, and
artificial we become in the
experiment,
the less natural are the procedures
and findings.
the more they limit the populations
and settings to whom they can
generalize their findings.
difficulty generalizing
experimentation to the natural
environment

19. Threats to Internal Validity
Changes in the dependent variable may be due to a variety of
extraneous factors rather than to the manipulation of the independent
variable can cause the results of a research study to be incorrectly
interpreted.
Participant
• Selection
• Maturation
• History
• Mortality
Treatment
• Diffusion of
treatments
Procedure
• Testing
• Instrumentati
on
• Statistical
Regression
Researcher
• Experimenter
Effect
• Observer Bias

20. 1. History
❑ Changes to DV due to:
Unplanned External Event other than IV between pretest and
posttest during the course of study but not part of the manipulation.
❑ Most likely when conditions are measured at different times with
long delays

21. T
O1
O2
one group pretest-posttest design
Pitfall of:
e.g. exercise?

22. X
e.g. Program to reduce energy drinks
consumption
Energy
drink
consumpti
on level
Pretest :
measure
DV
Education
al session
IV
Energy
drink
consumpti
on level
Posttest:
measure
DV
Price Of Energy drinks
Increases 50% Between Pre
And Post
History threat
O1 O2
O
2
O
1

23. Possible Solution
Energy
drink
consumpti
on level
Pretest :
measure
DV
Education
al session
IV
Energy
drink
consumpti
on level
Posttest:
measure
DV
Energy
drink
consumpti
on level
Pretest :
measure
DV
Nothing
IV
Energy
drink
consumpti
on level
Posttest:
measure
DV
A control group which will be exposed to the same history
but not the new form of therapy in the same environment
will reduce this threat.

24. 2.Maturation of subjects =predictable
changes
❑ Changes to DV due to: natural Processes that normally
cause subjects to change across time happen as a function
of time not as a function of the experiment
 People change naturally over time.
 Grow older, wiser, stronger , healthier, more physical
growth, emotional maturity, fatigue, and getting tired OR
natural increases in skills
 This is a problem in research that measures a DV over a
period of time
 The longer the time between measurements, the greater the
possibility of maturation effects
.

25. X
height &
weight of
children
Pretest :
measure
DV
Nutritiona
l protocol
IV
height &
weight of
children
Posttest:
measure
DV
Physical Growth
.Maturation threat
e.g. nutrition counseling to improve
anthropometric measures of newborn
O1 O2
O
2
O
1

26. Possible Solution
height &
weight of
children
Pretest :
measure
DV
Nutritiona
l protocol
IV
height &
weight of
children
Posttest:
measure
DV
height &
weight of
children
Pretest :
measure
DV
Nothing
IV
height &
weight of
children
Posttest:
measure
DV

27. T
O1
O2
one group pretest-posttest design
Pitfall of:
 Changes in the DV over time
irrespective of the IV

28. Pre-test Post-test Randomised Group Comparison
 Using a comparison group will reduce this threat
O1 T O2
P O4
O3
R
Possible Solution

29. Time series
T
O1 O2 O3 O4 O5 O6
Possible Solution

30. 3. Testing effect
❑ Changes to DV due to: taking a pre-test which may affect scores on the
post-test
❑ AKA “pretest sensitization,”
❑ process of Pretesting and retesting subjects can change them, a
pretest may sensitize an individual and improve the score of post test.
❑ Individuals generally score higher during second test regardless of
treatment.
❑ Testing becomes a more viable threat to internal validity as the time
between pretest and posttest is shortened.

31. …but then respond
better to the T than
the P…
e.g.
O1 T O2
P
O3
R
O4
…so it is actually T+O1
that is better than P, not T
alone.
Assessing muscle mass here
could make them train
harder…
Pitfall of prestest/postest design

32. O1
T
O2
R
O2
P
O1
P O2
T
O2
Solomon Four-Group Design
Use a control group which is also measured, but without the therapy
or with an alternative form of therapy
Possible Solution

33. 4. Instrumentation Change
 Changes to DV due to: Changes in data collection, Changes in
instruments, calibration of instruments, observers may cause
changes in the measurements
 Human observers become better,
 mechanical instruments become worse
 Instrumentation is a threat to internal validity when the
instrumentation is either unreliable or is changed between pre-
and posttesting.
Therefore, solution = calibrate

34. Changes to DV due to: loosing subjects → can be death, or subject
drop-out during course of the study
❑ Subject does not complete study.
❑ Attrition rate different between groups
❑ The longer period of study the more chance for dropout.
❑ Reduced n = reduced statistical Power
❑ Not only challenges quality of data gathered (Internal Validity)
but also our ability to generalise (External Validity).
Therefore, solution = recruit sufficient participants
5. Mortality

36.  Subjects
◼ Representativeness of the sample
in comparison to the population
◼ Participant’s awareness of being
involved in a study
◼ Personal characteristics of the
subjects
people of a specific race such as
black have high prevalence of HTN
compared to the white.
Therefore, a generalization made for
black will not be applicable for
whites. Hence, this is threat to
external validity.

37. Factors affecting external validity
 Time
◼ explanations can change over time
If a research was carried out on a
community in 1990 & then again
in 2000, the results of these two
researches would be different.
Therefore, older results cannot be
generalized over periods of time as
societies & circumstances
constantly change.

38. Factors affecting external validity
 Situations - characteristics of the
setting
◼ Specific environment
◼ Special situation
people living in high altitudes have
high (Hb) levels because at higher
altitudes the requirement of oxygen is
more, due to which there is more
production of (RBCs).
However, the Hb level of the people
living on the plains is lower in
comparison, so a generalization for
people of hilly areas is not applicable
for people living on plains.

39. Major Threats to External Validity
❑ Subject effect= Reactive arrangements
✓ Hawthorne effect
✓ John Henry effect
✓ Placebo effect
✓ Novelty effect
❑ Experimenter effect
✓ Halo effect

40.  Changes to DV due to: study participants respond in a certain
manner because they are aware that they are being observed
 Attention causes differences
 A specific type of reactive effect in which merely being a research
participant in an investigation may affect behavior
 Called the Hawthorne Effect after a study conducted at the
Hawthorne Electric Plant in the 1920’s.
 It was observed that a group of workers that participated in the
study acted differently because they “felt special.”
A- Reactive arrangements
1- Hawthorne effect:

41. It appears that being observed by the researchers was
increasing productivity, not the intensity of the lights.
Studying the effect of lighting on worker
productivity

42. Common solution:
❑ double-blind designs help avoid these
problems.
❑ Also, using a control group which is
measured the same way without the treatment.

43. Reactive arrangements
2- John Henry Effect
 Changes to DV due to: Members of the control group
compete with the experimental group.
 Participants know that they are in a control group and that the
experimental group is supposed to be better, therefore, the
control group tries harder to outperform the experimental
group.
 Therefore, participants should…
◼ Not be aware of the group they are in
❑ The terms refers to the classic story of John Henry laying
railroad track.

44. Reactive arrangements
3- Resentful Demoralization of the Control Group
 The control group may become discouraged
because it is not receiving the special attention
that is given to the treatment group.
 They may perform lower than usual because of
this.

45. Reactive arrangements
4- Placebo effect:
 Changes to DV due to: a participants response is influenced by
the expectancy of the participant of how they are expected to
behave
 Placebo effect is Real responses from fake treatment with placebo
substances
 Expectations play an important role in placebo effect
 Expectation of an effect gives that effect.
 The tendency of human subjects (often 20% or more of experiment
subjects) to show a response even when administered a placebo

46. Nocebo effect
Health deterioration due to negative expectations(“I will harm”)
E.g.:
Side effects: Occur in 20-30% of healthy adults
▪ Edema, pain, diarrhea, CVD
▪ Allergic/immune response/Asthma
▪ ↑ Risk of death

47. Reactive arrangements
5- Novelty effect:
 When a treatment is new, subjects & researcher might behave in
different ways. They may be enthusiastic about new methods of
doing things. Once treatment is more familiar & as the novelty
wears off, results might different.
 A treatment may work because it is novel and the subjects respond
to the uniqueness, rather than the actual treatment.
 When participants are engaged in something different this may
increase attention, interest, behavior, learning, etc., just because
it is something new.

48. B- Experimenter Effect
 Changes to DV due to: Researcher characteristic or behavior
influence subjects behavior.
 Examples of influential characteristics
 Facial expression
 Clothing
 Age, Gender
 Body build
 The treatment might have worked because of the person
implementing it. Given a different person, the treatment might not
work at all.
 Common solution: Use independent judges or more objective
measurements of the dependent variable.

49. Halo effect
 Based on the expectations of the researcher about certain
subjects based on some subject characteristics.
 The halo effect is a type of cognitive bias in which our overall
impression of a person influences how we feel and think about
his or her character.
 Common solutions: blind judges, use more objective measures.
You’re attractive You’re successful

50. Defense against threats to validity
 for External Validity
❖ Random selection of subjects
 for Internal Validity
❖ Random assignment to conditions
❖ Controlled
❖ Blindness
❖ Appropriate research design
➢ True experiments have high internal but
low external validity
➢ Quasi-experiments have higher external
but lower internal validity