2. DEFINITION
What is it?
•The testing of an idea (or practice or procedure)
•Determine whether it affects any outcome or dependent variable
•It is the best of the quantitative designs to use to establish probable
cause and effect.
When do
you do it?
•When there is a need to establish possible cause and effect between
independent and dependent variables.
When did it
begin?
•In the late 19th to early 20th century
•1916 : the idea of randomly selecting individuals to groups started
•1925 : firmly established procedure of comparing groups
•1963: major experimental designs identified
•1979: the types of designs are elaborated
•2002: discussions about major experimental designs are refined
3. Before you even think of doing an experiment, you must take note of the
key characteristics first!
1. RANDOM ASSIGNMENT
KEY CHARACTERISTICS OF EXPERIMENT
• The process of assigning individuals at random to groups or to different groups in an
experiment.
• The random assignment of individuals to groups differentiates a rigorous, “true”
experiment from an adequate, but less-than-rigorous “quasi-experiment”.
• Biasness distributed equally among the groups.
• You can control for extraneous characteristics (e.g., student ability, attention span).
• The experimental term for this process is “equating” the groups “ = randomly assigns
individuals to groups and equally distributes any variability of individuals
• BUT, personal factors that participants bring to an experiment can never be totally
controlled—some bias / error will always affect the outcome of a study.
4. Cont..
1. RANDOM ASSIGNMENT
KEY CHARACTERISTICS OF EXPERIMENT
• Random assignment vs random selection
• Impossible to randomly select for
experimental because the participants
are usually those who are available or
volunteer to take part.
• Random assignment is more
sophisticated.
Random selection refers to
how sample are selected from the
population.
Random assignment refers to how
the participants are assigned to the
treatment or control group using
a random procedure.
5. 2. CONTROL OVER EXTRANEOUS VARIABLES
KEY CHARACTERISTICS OF EXPERIMENT
• Extraneous factors are any influences in the selection of
participants/procedures/design etc that are likely to affect the outcome.
• All experiments have some random error that you cannot control, but you can
try to control extraneous factors as much as possible.
• Random assignment is a decision made by the investigator before the
experiment begins.
• Other control procedures before and during the experiment:
• Pretests
• Covariates
• Matching of participants
• Homogeneous samples
• Blocking variables.
6. A measure on some attribute or characteristic that you assess for participants in an
experiment before they receive a treatment.
A measure on some attribute or characteristic that is assessed for participants in an
experiment after a treatment.
Variables that the researcher controls for using statistics and that relate to the
dependent variable but that do not relate to the independent variable.
Process of identifying personal characteristics that influence the outcome
assigning the characteristics equally to the experimental and control groups.
(gender, pretest scores, abilities etc)
People who vary little in their personal characteristics (only junior/senior students,
racial group etc). The more similar they are, the more controlled it is.
A variable the researcher controls before the experiment starts by dividing (or
“blocking”) the participants into subgroups (or categories). For example : gender –
M&F
Pretest
Posttest
(after)
Covariates
Matching of
Participants
Homogeneous
Samples
Blocking
Variables
7. After you select your participants, you can now:
3. MANIPULATE TREATMENT CONDITIONS.
In experimental treatment, the researcher physically
intervenes to alter the conditions experienced by the
experimental unit (e.g., a reward for good spelling
performance or a special type of classroom instruction, such
as small group discussion).
Independent is the treatment given to the experimental group
Needs to have 2 or more levels. Eg: math training theoretical (1),
math theoretical + practice (2), math theoretical + practice + extra
class (3).
Dependent is the measurable variable
Treatment variables = Independent & Dependent
KEY CHARACTERISTICS OF EXPERIMENT
8. 4. GROUP COMPARISONS
A group comparison is the process of a researcher obtaining
scores on the dependent variable and comparing the means
and variance both within the group and between the groups.
KEY CHARACTERISTICS OF EXPERIMENT
9. 5. THREATS TO VALIDITY
How do you know the inferences you
made from your outcome are valid? Are
they really true and correct?
Threats to validity refer to specific
reasons for why we can be wrong when
we make an inference in an experiment
because of covariance, causation
constructs, or whether the causal
relationship holds over variations in
persons, setting, treatments, and
outcomes ( Shadish, Cook, & Campbell,
2002 ).
KEY CHARACTERISTICS OF
EXPERIMENT
10. 5. THREATS TO VALIDITY (cont…)
• Refers to the appropriate use of statistics
• Eg: violating statistical assumptions, restricted range on a variable, etc to infer
whether the presumed independent and dependent variables covary in the
experiment.
Statistical Conclusion Validity
• The validity of inferences about the constructs (or variables) in the study.
Construct Validity
• The validity of inferences drawn about the cause and effect relationship between
the independent and dependent variables.
Internal Validity
• The validity of the cause-and-effect relationship being generalizable to other
persons, settings, treatment variables, and measures.
External Validity
11. 5. THREATS TO VALIDITY (cont…)
Threats to internal validity
Participant
History
Maturation
Regression
Selection
Mortality
Interactions with
selection
Treatment
Diffusion of
treatments
Compensatory
equalization
Compensatory rivalry
Resentful
demoralization
Procedure
Testing
Instrumentation
12. 5. THREATS TO VALIDITY (cont…)
Threats to external validity
•Inability to generalize beyond the groups in the experiment
•Eg: other racial, social, geographical, age, gender etc.
•So, increase generalizability by making participation as
convenient as possible for all individuals in a population.
Interaction of
selection and
treatment
•The inability to generalize from the setting where the
experiment occurred to another setting.
•Eg: private high schools different from public high schools
•So, analyze the effect of a treatment for each type of setting
Interaction of
setting and
treatment
•When the researcher tries to generalize findings to past and
future situations.
•So, replicate the study at a later time rather than trying to
generalize results to other times.
Interaction of
history and
treatment
13. ◆ Between Group Designs
• True experiments (pre- and posttest, posttest only)
• Quasi-experiments (pre- and posttest, posttest only)
• Factorial designs
◆ Within Group or Individual Designs
• Time series experiments (interrupted, equivalent)
• Repeated measures experiments
• Single subject experiments
TYPES OF EXPERIMENTAL DESIGN
14. ◆ Between Group Designs
It is a strong experimental design - random assignment.
Experimental group receives treatment, but not control group.
Then, compile average (or mean) scores on a posttest.
When collecting pretest scores, you may compare net scores
(the differences between the pre- and posttests).
You may relate the pretest scores for the control and
experimental groups to see if they are statistically similar
Then, compare the two posttest group scores.
As you randomly assign individuals to the groups, most of the
threats to internal validity do not arise.
When true experiments include only a posttest, it reduces the
threats of testing, instrumentation, and regression because
you don’t use a pretest.
• True experiments (pre- and posttest, posttest only)
15. ◆ Between Group Designs
NO random assignment of participants to groups.
WHY? This is because you cannot artificially create groups for the
experiment.
Eg: studying new math program requires using existing fourth-grade
classes (one as experimental and one as control group). Randomly
assigning students to the two groups would disrupt classroom
learning.
Has more threats to internal validity than the true experiment:
(threats to participants, treatments and procedure are possible)
While the quasi-experimental design has the advantage of utilizing
existing groups in educational settings, it introduces many threats
that you need to address in the design of the experiment
• Quasi experiments (pre- and posttest, posttest only)
16. ◆ Between Group Designs
A study of two or more categorical, independent variables; each
examined at two or more levels (Vogt, 2005).
For example, in the math training program, you wish to combine the
influence of all the types of instruction and their motivation to learn
math
Assume that you have a reason to believe that motivation is
important but its “interaction” or combination with type of
instruction is unknown.
So, you need factorial design.
Thus, “motivation” is a blocking variable and you make random
assignment of each “block” (high, medium, and low) to each
treatment instructional group.
So, if you have 2 independent type of math training and motivation
(3 levels) – you will have a “2 x 3” factorial design. This will then go
against the result of the posttest.
• Factorial Design
17. The study of one
group, over time,
with multiple pretest
and posttest.
Does not require
large numbers of
participants.
Requires only one
group for the study.
There two important
variations:
Interrupted
Equivalent
• Time series experiments (interrupted, equivalent)
Interrupted time series design
•Study one group, obtain multiple pretest over
time, administering an intervention
•Then, measure outcomes (posttests) several
times
•Data analysis: pretests vs posttests or
posttest-only scores and using the pretests as
covariates
Equivalent time series design
•Alternate a treatment/intervention with a
posttest measure.
•Data analysis: comparing posttest measures.
◆ Within Group or Individual Designs
18. All participants in a single group participate in all experimental
treatments, with each group becoming its own control.
It compares a group’s performance under one experimental
treatment with its performance under another experimental
treatment.
The experimenter decides on multiple treatments (as in factorial
designs) but administers each one by one to only one group.
• Repeated measures
◆ Within Group or Individual Designs
19. Assume that you want to learn about the behavior of single individuals
and not groups.
It is the study of single individuals:
Observe them over a period of time
Administer an intervention.
Another observation after the intervention to see if treatment affects outcome.
• Single Subject
◆ Within Group or Individual Designs
A/B SINGLE DESIGN
(A): Administering an
intervention
(B): Observing and
measuring the behavior
after the intervention
20. Multiple Baseline Design:
each participant receives
an experimental treatment
at a different time (hence,
multiple baselines exist)
WHY? So that treatment
diffusion will not occur
among participants.
Choose this design when
the treatment (e.g., skill or
strategy being taught)
cannot be reversed and
doing so would be unethical
or injurious to participants.
• Single Subject (CONT…)
◆ Within Group or Individual Designs
• An alternating treatment design
is a single-subject design in
which the researcher examines
the relative effects of two or
more interventions and
determines which intervention is
the more effective treatment on
the outcome.
21. 1. Decide if an
experiment can solve
your RQ.
• Do you need to know if
there is a cause-effect
relationship?
2. Form hypotheses to
test the C/E
relationship
3. Select an
experimental unit &
identify participants
•Unit – individual, groups
•How to choose participants?
•How to assign to groups?
4. Select experimental
treatment & introduce
•Should have little intrusion
•Based on small pilot test
•Good intervention can
predict change
5. Choose a type of
experimental design
6. Conduct the
experiment
• Do pretests
• Introduce treatment
• Observe & posttest
7. Organize &
analyze data
8. Develop an
experimental
research report
HOW TO CONDUCT AN EXPERIMENT?
22. For a good experiment, here are some criteria:
The experiment has a powerful intervention.
The treatment groups are few in number.
Participants will gain from the intervention.
The researcher derives the number of participants per group in some
systematic way.
An adequate number of participants were used in the study.
The researcher uses measures and observations that are valid,
reliable, and sensitive.
The researcher controls for extraneous factors that might influence
the outcome.
The researcher addresses threats to internal and external validity.
EVALUATE THE QUALITY
