Here are some potential threats: Internal validity threats: - History effects (other events occurring simultaneously could influence outcomes) - Maturation effects (participants naturally changing over time independent of treatment) - Testing effects (pre-testing influencing post-test results) - Instrumentation effects (changes in how outcomes are measured over time) - Statistical regression (participants selected for extreme scores may regress to the mean) - Selection biases (non-random assignment of participants to conditions) - Mortality effects (differential dropout rates across conditions) External validity threats: - Interaction of selection and treatment (findings may not generalize to other populations) - Reactive effects of experimental arrangements (results due

1. Experiments,
Quasi-Experiments, &
Natural Experiments
causal inference and experimentation
designs and validity
natural experiments

2. randomized controlled experiment
Treatment or
R experimental
group
Population Sample
R
Control Group
R = randomization

3. Randomized Controlled Experiment
Pre-test Post-test
R Experimental Group Mexp1 x Mexp2
R Control Group Mcontrol1 Mcontrol2
X = experimental manipulation
M = measurements
Experimental effect = (Mexp2 - Mexp1)- (Mcontrol2 - Mcontrol1)
R = Random assignment of subjects to groups

4. designs and validity
• last time: validity of measures
and indicators
• this time: internal & external
validity as aspects of research
design

5. internal validity XZ
• extent to which a research design yields strong evidence of causality
• Key issue: is there a causal relationship between a “treatment” and the
“outcome(s)”?
Threats to internal validity:
1. History
2. Maturation
3. Testing
4. Instrumentation
5. Statistical Regression
6. Selection Threat
7. Experimental Mortality
8. Selection-Maturation Interaction

6. external validity XZ
• extent to which a research design yields findings that generalize to
• To subjects / objects / units other than those actually in your study
• To settings / circumstances other than that/those actually in your study
• To times other than those actually in your study
Threats to external validity:
1. Interaction of testing and experimental variable
2. Interaction of selection bias and experimental variable
3. Reactive Arrangements
4. Multiple-treatment interference

8. How might you design an experiment to test a new
drug?

9. How might you design an experiment to test a new
drug?

10. OXO
000 X 000
00 X 00 X 00 X 00

11. Dosage and Factorial Designs

12. regression effect:
matching and Head
Start

14. Natural Experiments

15. Posner
• Why are the Chekwas and
Tumbukas allies in Zambia and
adversaries in Malawi?
• unit of analysis (530)
• case selection criteria (531)
• survey design (532)
• lit review (535)
• additional tests (541)
• conclusions (543)
• Posner’s approach: find the
treatment

16. Dunning:
Important questions to ask of natural experiments

17. group activity
1. Characterize your project,
or a partner’s project, in
terms of its research design
2. What are some of the
potential threats to internal
and external validity?