3. c.2000DelSiegle
The main function of the
experimental research
design is to control
variance.
• Principle: maximize
systematic variance,
control extraneous
systematic variance,
and minimize error
variance.
In other words control variance.
4. c.2000DelSiegle
Therefore the researcher
attempts to:
• maximize the variance of the variable(s) of the research
hypothesis (i.e., maximize the difference in the
dependent variable [outcome] caused by maximizing
the differences in the independent variable [treatment]).
• control the variance of extraneous or "unwanted"
variables that may have an effect on the experimental
outcomes, but which he/she is not interested (limit
factors other than the treatment (IV) that could be
causing differences in the outcome (DV) .
• minimize the error or random variance (i.e., avoid
unreliable measurement instruments which have high
errors of measurement ).
5. c.2000DelSiegle
Maximization of Experimental
Variance
• experimental variance
– the variance due to the
manipulated (i.e., treatment)
or attribute (i.e., gender)
variables (IV)
research precept:
– design, plan and conduct
research so that experimental
conditions are as different as
possible on the independent
variable.
6. c.2000DelSiegle
Control of Extraneous
Variables (EV)
• eliminate the variable (i.e., if you are worried
about gender, only include one gender in the study).
• randomization (i.e., if you randomly assign subjects to
groups, the extraneous variable should be equally
distributed among the groups)
• build it into the design – make it a moderator variable (i.e.,
if you are worried about gender, build it into the analysis [2-
way ANOVA]—we’ll learn about this later)
• match subjects (i.e., match the characteristics of subjects
and put one of each matched pair in each group)
• statistically equate groups (i.e., use ANCOVA [Analysis of
Covariance] to analyze the data with the extraneous variable
used as a covariate—we’ll learn about this later)
7. c.2000DelSiegle
Minimizing Error Variance
has Two Principle Aspects:
• reduction of errors of
measurement through
controlled conditions (i.e.,
standardize testing
procedures)
• increase in the reliability of
measures (i.e., revise test
instruments or find more
reliable ones)
9. c.2000DelSiegle
Internal
Validity:
• Are the results of the
study (DV) caused by
the factors included in
the study (IV) or are
they caused by other
factors (EV) which
were not part of the
study?
There are 16 common threats to internal validity.
10. c.2000DelSiegle
(Selection Bias/Differential Selection) -- The
groups may have been different from the start. If
you were testing instructional strategies to
improve reading and one group enjoyed reading
more than the other group, they may improve
more in their reading because they enjoy it, rather
than the instructional strategy you used.
Subject
Characteristics
Threats to Internal Validity
11. c.2000DelSiegle
(Mortality) -- All of the high or low scoring
subject may have dropped out or were
missing from one of the groups. If we
collected posttest data on a day when the
honor society was on field trip at the
treatment school, the mean for the
treatment group would probably be much
lower than it really should have been.
Loss of Subjects
Threats to Internal Validity
12. c.2000DelSiegle
Perhaps one group was at a
disadvantage because of their
location. The city may have
been demolishing a building
next to one of the schools in
our study and there are
constant distractions which
interfere with our treatment.
Location
Threats to Internal Validity
13. c.2000DelSiegle
The testing instruments may not be
scores similarly. Perhaps the person
grading the posttest is fatigued and
pays less attention to the last set of
papers reviewed. It may be that those
papers are from one of our groups and
will received different scores than the
earlier group's papers
Threats to Internal Validity
Instrumentation
Instrument Decay
14. c.2000DelSiegle
The subjects of one group may react differently to the
data collector than the other group. A male
interviewing males and females about their attitudes
toward a type of math instruction may not receive the
same responses from females as a female
interviewing females would.
Threats to Internal Validity
Data Collector
Characteristics
15. c.2000DelSiegle
The person collecting data my favors one group,
or some characteristic some subject possess,
over another. A principal who favors strict
classroom management may rate students'
attention under different teaching conditions with
a bias toward one of the teaching conditions.
Threats to Internal Validity
Data
Collector
Bias
16. c.2000DelSiegle
The act of taking a pretest or posttest may influence the
results of the experiment. Suppose we were conducting
a unit to increase student sensitivity to prejudice. As a
pretest we have the control and treatment groups watch
Shindler's List and write a reaction essay. The pretest
may have actually increased both groups' sensitivity
and we find that our treatment groups didn't score any
higher on a posttest given later than the control group
did. If we hadn't given the pretest, we might have seen
differences in the groups at the end of the study.
Threats to Internal Validity
Testing
17. c.2000DelSiegle
Something may happen at one site during our study that
influences the results. Perhaps a classmate dies in a car
accident at the control site for a study teaching children bike
safety. The control group may actually demonstrate more
concern about bike safety than the treatment group.
Threats to Internal Validity
History
18. c.2000DelSiegle
There may be natural
changes in the subjects
that can account for the
changes found in a study.
A critical thinking unit may
appear more effective if it
taught during a time when
children are developing
abstract reasoning.
Threats to Internal Validity
Maturation
19. c.2000DelSiegle
The subjects may respond differently just because they are
being studied. The name comes from a classic study in
which researchers were studying the effect of lighting on
worker productivity. As the intensity of the factory lights
increased, so did the worker productivity. One researcher
suggested that they reverse the treatment and lower the
lights. The productivity of the workers continued to increase.
It appears that being observed by the researchers was
increasing productivity, not the intensity of the lights.
Threats to Internal Validity
Hawthorne Effect
20. c.2000DelSiegle
One group may view that it is in competition with the other
group and may work harder than they would under normal
circumstances. This generally is applied to the control group
"taking on" the treatment group. The terms refers to the
classic story of John Henry laying railroad track.
Threats to Internal Validity
John
Henry
Effect
21. c.2000DelSiegle
The control group may become discouraged
because it is not receiving the special attention
that is given to the treatment group. They may
perform lower than usual because of this.
Threats to Internal Validity
Resentful
Demoralization of
the Control Group
22. c.2000DelSiegle
(Statistical Regression) -- A class that scores
particularly low can be expected to score slightly
higher just by chance. Likewise, a class that
scores particularly high, will have a tendency to
score slightly lower by chance. The change in
these scores may have nothing to do with the
treatment.
Threats to Internal Validity
Regression
23. c.2000DelSiegle
The treatment may not be implemented as
intended. A study where teachers are
asked to use student modeling techniques
may not show positive results, not because
modeling techniques don't work, but
because the teacher didn't implement them
or didn't implement them as they were
designed.
Threats to Internal Validity
Implementation
24. c.2000DelSiegle
Threats to Internal Validity
Compensatory
Equalization of Treatment
Someone may feel sorry for the control
group because they are not receiving much
attention and give them special treatment.
For example, a researcher could be studying
the effect of laptop computers on students'
attitudes toward math. The teacher feels
sorry for the class that doesn't have
computers and sponsors a popcorn party
during math class. The control group begins
to develop a more positive attitude about
mathematics.
25. c.2000DelSiegle
Experimental Treatment
Diffusion
Threats to Internal Validity
Sometimes the control group
actually implements the
treatment. If two different
techniques are being tested in
two different third grades in the
same building, the teachers
may share what they are
doing. Unconsciously, the
control may use of the
techniques she or he learned
from the treatment teacher.
26. c.2000DelSiegle
Once the researchers are confident that
the outcome (dependent variable) of the
experiment they are designing is the
result of their treatment
(independent variable)
[internal validity],
they determine for which
people or situations
the results of
their study apply
[external validity].
27. c.2000DelSiegle
External
Validity:
• Are the results of the study generalizable to
other populations and settings?
External validity comes in two forms: population and ecological.
28. c.2000DelSiegle
1. ...the extent to which one can generalize from the study
sample to a defined population--
If the sample is drawn from an accessible population, rather
than the target population, generalizing the research results
from the accessible population to the target population is
risky.
2. ...the extent to which personological variables interact with
treatment effects--
If the study is an experiment, it may be possible that different
results might be found with students at different grades (a
personological variable).
Threats to External Validity
(Population)
Population Validity is the extent to which the results of a study
can be generalized from the specific sample that was studied to a
larger group of subjects. It involves...
29. c.2000DelSiegle
Ecological Validity is the
extent to which the results of an experiment can be
generalized from the set of environmental conditions
created by the researcher to other environmental
conditions (settings and conditions).
Threats to External Validity
(Ecological)
There are 10 common
threats to external
validity.
30. c.2000DelSiegle
(not sufficiently described for others to
replicate) If the researcher fails to
adequately describe how he or she
conducted a study, it is difficult to
determine whether the results are
applicable to other settings.
Threats to External Validity
(Ecological) Explicit
description of
the
experimental
treatment
31. c.2000DelSiegle
(catalyst effect)
If a researcher were to apply
several treatments, it is difficult to
determine how well each of the
treatments would work individually.
It might be that only the combination
of the treatments is effective.
Threats to External Validity
(Ecological)
Multiple-treatment
interference
32. c.2000DelSiegle
(attention causes differences)
Subjects perform differently because they
know they are being studied. "...External
validity of the experiment is jeopardized
because the findings might not generalize
to a situation in which researchers or
others who were involved in the research
are not present" (Gall, Borg, & Gall, 1996,
p. 475)
Threats to External Validity
(Ecological)
Hawthorne effect
33. c.2000DelSiegle
Threats to External Validity
(Ecological)
(anything different makes a difference)
A treatment may work because it is novel and the subjects
respond to the uniqueness, rather than the actual
treatment. The opposite may also occur, the treatment
may not work because it is unique, but given time for the
subjects to adjust to it, it might have worked.
Novelty and
disruption effect
34. c.2000DelSiegle
(it only works with this experimenter)
The treatment might have worked
because of the person implementing it.
Given a different person, the treatment
might not work at all.
Threats to External Validity
(Ecological)
Experimenter effect
35. c.2000DelSiegle
(pretest sets the stage)
A treatment might only work if a
pretest is given. Because they
have taken a pretest, the subjects
may be more sensitive to the
treatment. Had they not taken a
pretest, the treatment would not
have worked.
Threats to External Validity
(Ecological)
Pretest sensitization
36. c.2000DelSiegle
(posttest helps treatment "fall into place")
The posttest can become a learning
experience. "For example, the posttest might
cause certain ideas presented during the
treatment to 'fall into place' " (p. 477). If the
subjects had not taken a posttest, the
treatment would not have worked.
Threats to External Validity
(Ecological)
Posttest sensitization
37. c.2000DelSiegle
Interaction of
history and
treatment effect
Threats to External Validity
(Ecological)
(...to everything there is a time...)
Not only should researchers be cautious about
generalizing to other population, caution should be
taken to generalize to a different time period. As
time passes, the conditions under which treatments
work change.
38. c.2000DelSiegle
(maybe only works with M/C tests)
A treatment may only be evident with
certain types of measurements. A
teaching method may produce superior
results when its effectiveness is tested
with an essay test, but show no
differences when the effectiveness is
measured with a multiple choice test.
Threats to External Validity
(Ecological)
Measurement of
the dependent
variable
39. c.2000DelSiegle
Interaction of time
of measurement
and treatment
effect
Threats to External Validity
(Ecological)
(it takes a while for the treatment to kick in)
It may be that the treatment effect does not occur until
several weeks after the end of the treatment. In this situation,
a posttest at the end of the treatment would show no impact,
but a posttest a month later might show an impact.
40. c.2000DelSiegle
First, and foremost, an experiment
must have internal validity. If the
researchers cannot be certain that the
results of the experiment are
dependent on the treatment, it does
not matter to which people or
situations they wish to generalize
(apply) their findings. The importance
of external validity is reliant on
having internal validity in much the same way that the validity
of a measurement instrument is reliant on the instrument
being reliable.
However, the more tightly experimenters design their study,
the more they limit the populations and settings to whom
they can generalize their findings.
42. c.2000DelSiegle
Suppose a researcher wants to
study the effect of a reading
program on reading achievement.
She might implement the reading program with a
group of students at the beginning of the school year
X
and measure their achievement at the end of
the year.
O
This simple design is known as a one-shot
case study design.
43. c.2000DelSiegle
X O
Unfortunately, the students’ end of year reading scores
could be influenced by other instruction in school, the
students’ maturation, or the treatment.
We also do not know whether the students’ reading skills
actually changed from the start to end of the school year.
We could improve on this design by giving a pretest at the
start of the study.
O
This is known as a one-group pretest-posttest
design.
44. c.2000DelSiegle
X OO
Unfortunately, the students’ end of year reading scores still
could be influenced by other instruction in school, the
students’ maturation, or the treatment.
OO
Our researcher may wish to have a comparison group.
This is a static-group pretest-posttest design.
45. c.2000DelSiegle
If our researcher believes that the pretest has an impact
on the results of the study, she might not include it.
X O
O
O
O
This is a static-group comparison design.
46. c.2000DelSiegle
X O
O
O
O
R
R
Because our researcher did not pretest, she might wish
to randomly assign subjects to treatment and control
group.
Random assignment of subject to groups should spread
the variety of extraneous characteristics that subjects
possess equally across both groups.
This is a randomized posttest-only, control group
design.
47. c.2000DelSiegle
X O
O
O
O
This is a randomized pretest-posttest control group
design.
R
R
Of course, our researcher could also include a pretest
with her random assignment.
48. c.2000DelSiegle
Occasionally researchers combine the randomized
pretest-posttest control group designwith the
randomized posttest-only, control group design.
This is a randomized Solomon four-group design.
X O
O
R
R
X O
O
O
O
R
R
49. c.2000DelSiegle
With the randomized Solomon four-group design, all
groups are randomly assigned
X
X
O
O
O
O
O
O
R
R
R
R
and given the posttest.
Two of the groups are given pretests.
One of the pretest
groups is assigned to
treatment and one of
the non-pretest groups
is assigned to
treatment.
50. c.2000DelSiegle
Each of the designs described in this section has
advantages and disadvantages that influence the
studies internal and external validity.
This presentation was prepared by Del Siegle.
Some of the material is from an earlier presentation by Scott Brown.
