EXPERIMENTS
© LOUIS COHEN, LAWRENCE
MANION & KEITH MORRISON
STRUCTURE OF THE CHAPTER
• Designs in educational experimentation
• True experimental designs
• A quasi-experimental desig...
CAUSALITY
• Experiments are held up to be able to identify
causality through control and manipulation of
variables.
• Exam...
INDEPENDENT AND
DEPENDENT VARIABLES
Development
planning
School
Effectiveness
Parents
and
community
Teaching and
learning
...
RANDOMIZATION
• Random sampling and random allocation to either a
control or experimental group.
• Randomization allows fo...
CONCERNS IN EXPERIMENTS
• It may not be possible or desirable to isolate
and control variables under laboratory
conditions...
BLIND AND DOUBLE-BLIND
EXPERIMENTS
• Blind experiment: participants do not know to
which group they are assigned.
• Double...
KINDS OF EXPERIMENT
• Laboratory experiments (controlled, artificial conditions):
– Pretest-post-test control and experime...
FEATURES OF A TRUE EXPERIMENT
• Random allocation of the sample to control or
experimental groups;
• Identification and is...
Randomly assign subjects
to two matched groups:
control and experimental group
Conduct pre-test
Isolate and control variab...
‘TRUE’ EXPERIMENTAL DESIGN
CONTROL CONTROL
EXPERIMENT EXPERIMENTIntervention
Matched on
Pre-test
Random group
assignation
...
MEASURING EFFECTS
Average causal effect (A):
(A) = (E1−E2) − (C1−C2)
where:
– E1 = post-test for experimental group;
– E2 ...
CAMPBELL’S AND STANLEY’S NOTATION
• X represents the exposure of a group to an
experimental variable or event, the effects...
CAMPBELL’S AND STANLEY’S
SYMBOLIC REPRESENTATION OF
‘TRUE’ EXPERIMENTS
RO1 X O2
RO3 O4
Campbell, D. T. and Stanley, J (196...
TWO CONTROL GROUPS AND ONE
EXPERIMENTAL GROUP PRETEST-
POST-TEST DESIGN
Experimental RO1 X RO2
Control1 RO3 RO4
Control2 X...
THE POST-TEST CONTROL AND
EXPERIMENTAL GROUP DESIGN
Experimental R1 X O1
Control R 2 O2
THE POST-TEST TWO
EXPERIMENTAL GROUPS DESIGN
Experimental1 R1 X1 O1
Experimental2 R2 X2 O2
THE PRETEST―POST-TEST TWO
TREATMENT DESIGN
Experimental1 RO1 X1 O1
Experimental2 RO3 X2 O4
THE TRUE EXPERIMENT ONE
CONTROL AND TWO EXPERIMENTAL
GROUPS
Experimental1 RO1 X1 O1
Experimental2 RO3 X2 O4
Control RO5 O6
THE PRE-TEST TWO TREATMENT
DESIGN
Experimental1 RO1 X1 O1
Experimental2 RO3 X2 O4
MATCHED PAIRS DESIGN
Step One: Measure the dependent variable.
Step Two: Assign participants to matched pairs,
based on th...
INDEPEND
ENT
VARIABLE
LEVEL
ONE
LEVEL
TWO
LEVEL
THREE
Availability
of
resources
limited
availability (1)
moderate
availabi...
0
20
40
60
80
100
15 16 17 18
Age
Motivationformathematics
Males
Females
Difference for motivation in mathematics is not c...
PARAMETRIC DESIGN
• Participants are randomly assigned to groups
whose parameters are fixed in terms of the
levels of the ...
REPEATED MEASURES
• Participants in the experimental groups are
tested under two or more experimental
conditions.
• The or...
REPEATED MEASURES
(two groups receiving both conditions)
Group 1
With no
intervention
Matched on pre-test
Random allocatio...
Independent
groups
Noise condition No noise
condition
  
Sara Rob Peter
  
Jane Jack Jim
  
Joan Susan John
  ...
Repeated
measures
Noise condition No noise
condition
  
Sara Rob Peter
  
Jane Jack Jim
  
Joan Susan John
  
...
QUASI-EXPERIMENTS: NON-
EQUIVALENT CONTROL GROUP
DESIGN
• Pre-experimental design: the one-group
pretest―post-test
Experim...
QUASI-EXPERIMENTS: NON-
EQUIVALENT CONTROL GROUP
DESIGN
• The pre-test―post-test non-equivalent
group design
Experimental ...
PROCEDURES IN CONDUCTING
EXPERIMENTS
1. Identify research problems
2. Formulate hypotheses
3. Select appropriate levels at...
A TEN-STEP MODEL FOR
CONDUCTING EXPERIMENTS
Step One: Identify the purpose of the experiment.
Step Two: Select the relevan...
PROCEDURES IN CONDUCTING
EXPERIMENTS: HYPOTHESES
• Null hypothesis (H1)
• Alternative hypothesis (H0)
• Direction of hypot...
OPERATIONALIZING HYPOTHESES
• Hypothesis: ‘people who study in quiet
surroundings achieve better than those who
study in n...
DIRECTIONAL AND NON-
DIRECTIONAL HYPOTHESES
Directional (one-tailed):
People who do homework without the TV
on produce bet...
DIRECTION
OF CAUSALITY
MATURATION
TESTING
THREATS TO
VALIDITY AND
RELIABILITY
TYPE 1 AND
TYPE 2
ERRORS
INSTRUMENT-
ATION
O...
TIMING OF PRE-TEST AND POST-TEST
• Pre-test: as close to the start of the experiment as
possible (to avoid contamination o...
INTERNET-BASED EXPERIMENTS
Four types:
1. Those that present static printed materials
(e.g. printed text or graphics)
2. T...
INTERNET-BASED EXPERIMENTS
• Check download speeds and time, anticipate
problems of different browsers and platforms.
• Ca...
Upcoming SlideShare
Loading in...5
×

Chapter16

235

Published on

Research Methods in Education 6th Edition

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
235
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Chapter16

  1. 1. EXPERIMENTS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON
  2. 2. STRUCTURE OF THE CHAPTER • Designs in educational experimentation • True experimental designs • A quasi-experimental design: the non-equivalent control group design • Single-case research: ABAB design • Procedures in conducting experimental research • Threats to internal and external validity in experiments • The timing of the pretest and the post-test • Examples from educational research • The design experiment • Internet-based experiments
  3. 3. CAUSALITY • Experiments are held up to be able to identify causality through control and manipulation of variables. • Examine the effect of an independent variable on a dependent variable. • Identifying the effects of causes by implementing interventions in a controlled environment. • Held up to be able to offer explanations for outcomes.
  4. 4. INDEPENDENT AND DEPENDENT VARIABLES Development planning School Effectiveness Parents and community Teaching and learning Professional development Management Leadership Culture and climate
  5. 5. RANDOMIZATION • Random sampling and random allocation to either a control or experimental group. • Randomization allows for the many additional uncontrolled and, hence, unmeasured, variables that may be part of the make-up of the groups in question. • Randomization operates the ceteris paribus condition (all other things being equal), assuming that the distribution of extraneous variables is more or less even and perhaps of little significance. • Randomization strives to address Holland’s (1986) ‘fundamental problem of causal inference’, which is that a person may not be in both a control group and an experimental group simultaneously.
  6. 6. CONCERNS IN EXPERIMENTS • It may not be possible or desirable to isolate and control variables under laboratory conditions. • The ‘real world’ is not the antiseptic, artificial world of the laboratory. • Cannot assume that a single cause produces a single effect. • The setting affects the outcomes.
  7. 7. BLIND AND DOUBLE-BLIND EXPERIMENTS • Blind experiment: participants do not know to which group they are assigned. • Double blind experiment: neither the researcher nor the participants know to which group the participants are assigned.
  8. 8. KINDS OF EXPERIMENT • Laboratory experiments (controlled, artificial conditions): – Pretest-post-test control and experimental group – Two control groups and one experimental group pretest-post-test – Post-test control and experimental group – Post-test two experimental groups – Pretest-post-test two treatment – Matched pairs; – Factorial design; – Parametric design; – Repeated measures design; • Field experiments (controlled conditions in the ‘real world’): – one-group pretest-post-test; – non-equivalent control group design; – time series • Natural experiments (no control over real world conditions)
  9. 9. FEATURES OF A TRUE EXPERIMENT • Random allocation of the sample to control or experimental groups; • Identification and isolation of key variables; • Control of the key variables; • Exclusion of any other variables; • Special treatment (the intervention) given to the experimental group (i.e. manipulating the independent variable) whilst holding every other variable constant for the two groups; • Ensuring that the two groups are entirely separate throughout the experiment (non-contamination); • Final measurement of outcomes to compare the control and experimental groups and to look at differences from the pre-test results (the post-test); • Comparison of one group with another.
  10. 10. Randomly assign subjects to two matched groups: control and experimental group Conduct pre-test Isolate and control variables, exclude other variables Administer intervention to experimental group Conduct post-test and compare control and experimental groups Stages in an experiment
  11. 11. ‘TRUE’ EXPERIMENTAL DESIGN CONTROL CONTROL EXPERIMENT EXPERIMENTIntervention Matched on Pre-test Random group assignation Isolate, control and manipulate variables Post-test PLUS
  12. 12. MEASURING EFFECTS Average causal effect (A): (A) = (E1−E2) − (C1−C2) where: – E1 = post-test for experimental group; – E2 = pretest for experimental group; – C1 = post-test for control group; – C2 = pretest for control group.
  13. 13. CAMPBELL’S AND STANLEY’S NOTATION • X represents the exposure of a group to an experimental variable or event, the effects of which are to be measured. • O refers to the process of observation or measurement. • Xs and Os in a given row are applied to the same persons. • Left to right order indicates temporal sequence. • Xs and Os vertical to one another are simultaneous. • R indicates random assignment to separate treatment groups. • Parallel rows unseparated by dashes represent comparison groups equated by randomization, while those separated by a dashed line represent groups not equated by random assignment.
  14. 14. CAMPBELL’S AND STANLEY’S SYMBOLIC REPRESENTATION OF ‘TRUE’ EXPERIMENTS RO1 X O2 RO3 O4 Campbell, D. T. and Stanley, J (1963) Experimental and Quasi-experimental Designs for Research on Teaching. Boston: Houghton Mifflin Co.
  15. 15. TWO CONTROL GROUPS AND ONE EXPERIMENTAL GROUP PRETEST- POST-TEST DESIGN Experimental RO1 X RO2 Control1 RO3 RO4 Control2 X RO5
  16. 16. THE POST-TEST CONTROL AND EXPERIMENTAL GROUP DESIGN Experimental R1 X O1 Control R 2 O2
  17. 17. THE POST-TEST TWO EXPERIMENTAL GROUPS DESIGN Experimental1 R1 X1 O1 Experimental2 R2 X2 O2
  18. 18. THE PRETEST―POST-TEST TWO TREATMENT DESIGN Experimental1 RO1 X1 O1 Experimental2 RO3 X2 O4
  19. 19. THE TRUE EXPERIMENT ONE CONTROL AND TWO EXPERIMENTAL GROUPS Experimental1 RO1 X1 O1 Experimental2 RO3 X2 O4 Control RO5 O6
  20. 20. THE PRE-TEST TWO TREATMENT DESIGN Experimental1 RO1 X1 O1 Experimental2 RO3 X2 O4
  21. 21. MATCHED PAIRS DESIGN Step One: Measure the dependent variable. Step Two: Assign participants to matched pairs, based on the scores and measures established from Step One. Step Three: Randomly assign one person from each pair to the control group and the other to the experimental group. Step Four: Administer the intervention to the experimental group and, if appropriate, a placebo to the control group. Ensure that the control group is not subject to the intervention. Step Five: Carry out a measure of the dependent variable with both groups and compare/measure them in order to determine the effect and its size on the dependent variable.
  22. 22. INDEPEND ENT VARIABLE LEVEL ONE LEVEL TWO LEVEL THREE Availability of resources limited availability (1) moderate availability (2) high availability (3) motivation for the subject studied little motivation (4) moderate motivation (5) high motivation (6) FACTORIAL DESIGN Performance in an examination may depend on availability of resources and motivation for the subject studied 9 combinations: 1+4; 1+5; 1+6; 2+4; 2+5; 2+6; 3+4; 3+5; 3+6
  23. 23. 0 20 40 60 80 100 15 16 17 18 Age Motivationformathematics Males Females Difference for motivation in mathematics is not constant between males and females, but varies according to age of participants: an interaction effect (age and sex) Factorial designs must address the interaction of the independent variables.
  24. 24. PARAMETRIC DESIGN • Participants are randomly assigned to groups whose parameters are fixed in terms of the levels of the independent variable that each receives. • Parametric designs are useful if an independent variable has different levels or a range of values which may have a bearing on the outcome (confirmatory research) or if the researcher wishes to discover whether different levels of an independent variable have an effect on the outcome (exploratory research).
  25. 25. REPEATED MEASURES • Participants in the experimental groups are tested under two or more experimental conditions. • The order in which the interventions are sequenced may have an effect on the outcome (e.g. the first intervention may have an influence – a carry-over effect – on the second, and the second intervention may have an influence on the third). • Early interventions may have a greater effect than later interventions. • Repeated measures designs are useful if it is considered that order effects are either unimportant or unlikely.
  26. 26. REPEATED MEASURES (two groups receiving both conditions) Group 1 With no intervention Matched on pre-test Random allocation to groups Group 2 With intervention Group 2 With no intervention Post-test Group 1 With intervention
  27. 27. Independent groups Noise condition No noise condition    Sara Rob Peter    Jane Jack Jim    Joan Susan John    Lyn Sally Alan
  28. 28. Repeated measures Noise condition No noise condition    Sara Rob Peter    Jane Jack Jim    Joan Susan John    Lyn Sally Alan    Jane Jack Jim    Sara Rob Peter    Lyn Sally Alan    Joan Susan John
  29. 29. QUASI-EXPERIMENTS: NON- EQUIVALENT CONTROL GROUP DESIGN • Pre-experimental design: the one-group pretest―post-test Experimental O1 X O2 • Pre-experimental design: the one-group post- test only design Experimental O1 • The Post-Tests only non-equivalent groups design Experimental O1 - - - - - - - - - - Control O
  30. 30. QUASI-EXPERIMENTS: NON- EQUIVALENT CONTROL GROUP DESIGN • The pre-test―post-test non-equivalent group design Experimental O1 X O2 - - - - - - - - - - Control O3 O4
  31. 31. PROCEDURES IN CONDUCTING EXPERIMENTS 1. Identify research problems 2. Formulate hypotheses 3. Select appropriate levels at which to test the independent variables 4. Decide which kind of experiment to adopt 5. Decide population and sampling 6. Select instruments for measurement 7. Decide how the data will be analyzed 8. Pilot experimental procedures 9. Carry out the refined procedures 10.Analyze results 11.Report the results
  32. 32. A TEN-STEP MODEL FOR CONDUCTING EXPERIMENTS Step One: Identify the purpose of the experiment. Step Two: Select the relevant variables. Step Three: Specify the level(s) of the intervention (e.g. low, medium high intervention). Step Four: Control the experimental conditions and environment. Step Five: Select appropriate experimental design. Step Six: Administer the pretest. Step Seven: Assign the participants to the group(s). Step Eight: Conduct the intervention. Step Nine: Conduct the post-test. Step Ten: Analyze the results.
  33. 33. PROCEDURES IN CONDUCTING EXPERIMENTS: HYPOTHESES • Null hypothesis (H1) • Alternative hypothesis (H0) • Direction of hypothesis: states the kind of difference or relationship between two conditions or two groups of participants • One-tailed (directional): e.g. ‘people who study in silent surroundings achieve better than those who study in noisy surroundings’ • Two-tailed (no direction): e.g. ‘there is a difference between people who study in silent surroundings and those who study in noisy surroundings’
  34. 34. OPERATIONALIZING HYPOTHESES • Hypothesis: ‘people who study in quiet surroundings achieve better than those who study in noisy surroundings’ • What do ‘work better’, ‘quiet’ and ‘noisy’ mean? Define the operations: – ‘work better’ = obtain a higher score on the Wechsler Adult Intelligence Scale – ‘quiet’ = silence – ‘noisy’ = CD music playing • Operationalized hypothesis: ‘people who study in silence achieve a higher score on the Wechsler Adult Intelligence Scale than those who study with CD music playing’
  35. 35. DIRECTIONAL AND NON- DIRECTIONAL HYPOTHESES Directional (one-tailed): People who do homework without the TV on produce better results than those who do homework with the TV on. Non-directional (two-tailed): There is a difference between work produced in noisy or silent conditions.
  36. 36. DIRECTION OF CAUSALITY MATURATION TESTING THREATS TO VALIDITY AND RELIABILITY TYPE 1 AND TYPE 2 ERRORS INSTRUMENT- ATION OPERATIONAL- IZATION REACTIVITY HISTORY EXPERIMENTAL MORTALITY CONTAMIN- ATION
  37. 37. TIMING OF PRE-TEST AND POST-TEST • Pre-test: as close to the start of the experiment as possible (to avoid contamination of other variables). • Post-test: as close to the end of the intervention as possible. • Too soon a post-test: misses long-term/delayed effect and only measures short-term gain (which may be lost later). • Too long a time lapse before a post-test: becomes impossible to determine whether it was a particular independent variable that caused a particular effect, or whether other factors have intervened since the intervention, to produce the effect.
  38. 38. INTERNET-BASED EXPERIMENTS Four types: 1. Those that present static printed materials (e.g. printed text or graphics) 2. Those that make use of non-printed materials (e.g. video or sound) 3. Reaction-time experiments 4. Experiments that involve some form of interpersonal interaction
  39. 39. INTERNET-BASED EXPERIMENTS • Check download speeds and time, anticipate problems of different browsers and platforms. • Can experience greater problems of dropout than conventional experiments.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×