This document discusses experimental validity, including internal and external validity. It defines internal validity as being about whether the independent variable caused changes in the dependent variable. Threats to internal validity include history, maturation, testing, instrumentation, regression, selection bias, mortality, and additive/interactive effects. External validity is about generalizing results beyond the experimental setting, and threats include interaction of selection/treatment, testing/treatment, setting/treatment, history/treatment, and the Hawthorne effect. Maintaining validity requires controlling for these threats in research design.

1. EXPERIMENTAL VALIDITY
-JIJO VARGHESE
RESEARCH SCHOLAR
DEPARTMENT OF EDUCATION
CENTRAL UNIVERSITY OF KERALA
jmecheril@gmail.com

2. Experimental Designs  Pre-Experimental Design
 Quasi-Experimental Design
 True Experimental Design

3. Experimental Design  Selecting the group(s), assigning the
treatments, controlling the intervening
variables, and assessing the
Dependent Variable

4. Characteristics of
Experimental Method
 Presence of IV and DV
 Manipulation of Variable
 Controlling of Intervening Variables
 Random assignments of subjects and
Treatments
 Verification

5. How can you understand
your experiment is valid?

6. Experimental Validity
An experiment is valid :
(1) if results obtained are due only to the manipulated
independent variable and
(2) if they are generalizable to individuals or contexts
beyond the experimental setting.
These two criteria are referred to, respectively, as the
internal validity and external validity of an
experiment.

7. Types of Experimental Validity
Internal validity is the degree to which observed differences on the
dependent variable are a direct result of manipulation of the independent
variable, not some other variable.
External validity, also called ecological validity, is the degree to which
study results are generalizable, or applicable, to groups and environments
outside the experimental setting.

8. Internal Validity
 Identifying cause-effect
relationship is the most frequent
purpose of psychological research
 Internal Validity is the degree to
which you can justifiably claim
from your empirical research
study that changes in the IV
caused changes in the DV
Term coined by Campbell and
Stanley (1963)

9. Required conditions for inferring causation
IV-DV relation
Need to obtain strong
evidence that the
presumed cause and effect
are related.
IV and DV must be related ,
changes in IV must
accordingly change DV
IV precedes DV
Temporal sequence of the
variables being
investigated
The cause precedes the
effect
No other influence
No plausible alternative
explanation for the
relationship exist.
No effect of intervening
variable must occur

10. Threats to Internal Validity
• To infer that one variable was the cause of an effect observed in
another variable, we must control for all other possible causes.
• The other possible causes are the threats to internal validity because
these threats represent rival and competing explanations for the
results obtained.
• When these rival explanations exist, it is impossible to reach a causal
explanation with any degree of certainty, leading to highly suspect
results that cannot and should not be taken seriously .
• Hence, it is necessary to control and eliminate the influence of these
threats.

11. HISTORY
 Refers to any event that can produce the
outcome, other than the treatment, that
occurs after the beginning of the study but
before the posttest measurement of the
dependent variable.
 Longer the time lapse of a research study,
the greater the possibility pf history
becoming a rival explanation
Occurs in a one-group pretest-
posttest research design
Pretest Posttest
Intervention
X
History

12. MATURATION
 Maturation effect may cause subjects to
respond differently on posttest because they
have grown older, stronger and healthier,
more experienced.
 The psychological and biological process in a
human body, such as age, learning, fatigue,
boredom which reside within individual may
affect the second measurement.
Change in the internal
conditions of the individuals
pretest posttest
Time interval between pre
and post test of DV is
longer

13. TESTING
 Changes in the scores participants make on
the second administration of a test that result
from having previously taken the test.
 Familiarity with the tool used in the pretest
 The experience of having taken a pretest
may alter the results obtained on the
posttest.
The potential effect of the
pretest

14. INSTRUMENTATION
 Any change that occurs in the measuring
instrument.
 Two ways in which this can happen:
 A). The measurement instrument that is used
during pretesting is different from that used
during posttesting.
 B). If the data were collected through
observation (human observers are subjected
to boredom, fatigue etc.)
Change in the way the
dependent variable is measured.

15. Regression Artifacts/
Statistical Regression
 Extreme scores will tend to regress or move
toward the mean of a distribution on a second
test.
 Regression artifacts occur because the first
and second measurements of performance
are not perfectly correlated.
 “…pseudoeffects that appear to be effects due
to some supposed causal variable but are
nothing more than regression toward the
mean” (Campbell & Kenny, 1999).
High pretest scores- Lower
posttest scores

16. DIFFERENTIAL SELECTION
 Placing research participants in the various
comparison groups.
 Ideally, participants are randomly assigned to
the various groups (control and
experimental).
 Random assignment is the best way for
equating the groups.
 Eg. Experiment of conformity behavious.,
review say women have more conformity, if
two groups are not equated, the researcher
may get false result.
Threat for a multigroup study

17. Mortality
 Loss of subjects from comparison groups
 Drop out occurs for specific reasons relate to
the experimental situation.

18. Additive and
Interactive Effects
 Selection-History Effects- if two comparison
groups experienced a different history event and
the history effects they experienced resulted in
their responding in different ways to the
dependent variable.
 When experimental groups come from different
settings so that local history of each group
influences the responses.
 Selection Maturation Interaction-
experimental groups experience maturational
change at different speeds.
 Selection- Instrumentation Effects- if the
groups respond differently to an instrumentation
effect because they they are composed of
different kinds of people.
 This is often called “floor and ceiling” effects
because the measurement scale is incapable of
accurately recording values above or below a
certain level.
Combination of history,
maturation, instrumentation

19. EXTERNAL VALIDITY
 Degree to which the study results can be generalized to and across other people, settings,
treatments, outcomes, and times.
 Usefulness of the information outside the experimental situation.
 Also called ‘generalizing Validity’
Population validity The ability to generalize the study results to individuals who were not
included in the study
Ecological Validity The ability to generalize the study results across settings
Temporal Validity The ability to generalize the study results across time
Treatment Variation Validity The ability to generalize the study results across variation of treatment
Outcome Validity The ability to generalize the study results across different but related
dependent variables
Term coined by Campbell and Stanley (1963)

20. Threats to External Validity
Threats to external validity involve the interaction of treatment with the
specific type of subjects tested, the specific setting in which the
experiment is carried out, or the time in history when the study is done.

21. Interaction of
Selection and
Treatment
 One major goal of research is to apply result
to a target population, that is, to individuals
who are not experimental subjects but who
are represented by the sample of the study.
 If subjects are sampled according to specific
characteristics, such as, subjects may be
restricted to a limited age range, one gender,
only urban area, or a defined level of function.
 When samples are confined to certain types
of subjects, it is not reasonable to generalize
results to those who do not have these
characteristics.

22. Interaction of
Testing and
Treatment
 External Validity may be affected due to
Interaction of Testing and Treatment.
 In some designs, the research
administers a pretest to get the present
status of the group on the dependent
variable under study.
 Now this group is given the treatment
as decided by the researcher and at
the end of the treatment, the same test
is administered again to the group and
it constitutes the posttest.
 The performance on the posttest is the
resultant of interaction between
treatment and pretest.
 So the finding of this group cannot be
generalized to the population where
pretesting is not done.

23. Interaction of Setting
and Treatment
 Suppose the researcher demonstrates a causal
relationship between Concept Attainment
Model and concept understanding of Class IX
students of Government Schools of Delhi.
 Can the finding of this study be generalized to
students of other states, students from urban
area, different family structure, personality
and schools, etc.?
 This can only be answered by replicating the
same research in different settings.
 Further all situational specifics e.g. treatment
conditions, time, location, lighting, noise,
treatment administration, investigator, timing,
scope and extent of measurement, etc. of a
study potentially limit generalizability.

24. Interaction of
History and
Treatment
 This threat to external validity concerns the
ability to generalize the results to different
periods of time in the past or future.
 For example, if one looks at the results of
nutritional studies of reducing cholesterol in
the diet, results may be quite different today
from the results obtained 20 years ago, when
knowledge about the effect of diet and
exercise on cardiovascular fitness was less
developed, and when the society and the
media were less involved in promoting
fitness and health.
 This type generalization is supported when
results are replicated in future studies
 previous research verifies the established
causal relationship.

25. Experimental Bias or
Hawthorne Effect
 Subjects often try their best to fulfill the researcher's expectation or to
present themselves in the best way possible, so that responses are no
longer representative of natural behaviour.
 This fact was documented in classical studies performed from 1924 to 1927
at the Hawthorne Works, a division of the Western Electric Company in
Chicago.
 Researchers were interested in studying how various levels of illumination
affected workers' output.
 What they found was that no matter what they did, lowering light or raising
light, the workers increased production.
 This phenomenon became known as the Hawthorne effect, which is the
tendency of persons who are singled out for special attention to perform
better merely because of the expectations created by the situation.
 Experimenters may also have certain expectancies that can influence how
subjects respond.
 They may react more positively to subjects in the experimental group or
give less attention to those in the control group, because of an emotional or
intellectual investment in their hypothesis.
 Rosenthal described several types of experimenter effects in terms of the
experimenter's active behaviour and interaction with the subject, such as,
verbal cues and smiling, and passive behaviour, such as those related to
appearance.
 This threat can be avoided by employing testers who are blinded to subject
assignment and the research hypothesis.

26. Placebo Effect
 A placebo is anything that seems to be a "real"
medical treatment but isn't. It could be a pill, a
shot, or some other type of "fake" treatment.
 What all placebos have in common is that they
do not contain an active substance meant to
affect health.
 Sometimes a person can have a response to a
placebo. The response can be positive or
negative. For instance, the person's symptoms
may improve.
 Or the person may have what appear to be side
effects from the treatment. These responses are
known as the "placebo effect".
 The placebo effect refers to the phenomenon in
which some people experience some benefit
after the administration of a placebo.

27. Internal vs. External Validity
 They are both factors that should be
considered when designing a study,
and both have implications in terms of
whether the results of a study have
meaning.
 Both are not "either/or" concepts, and
so you will always be deciding to what
degree your study performs in terms
of both types of validity.
 The essential difference
between internal and external
validity is that internal validity
refers to the structure of a
study and its variables while
external validity relates to how
universal the results are

28. References
 Sansanwal, D.S. (2020). Research Methodology and Applied Statistics, Delhi:
Shipra Publications
 Christensen et al. (2014). Research Methods Design, And Analysis, Noida:
Pearson.
 Johnson, B. & Christensen, L. (2008). Educational Research: Quantitative,
Qualitative, and Mixed Approaches, London: Sage.