Fixed Designs for Psychological ResearchPresentation Transcript
Fixed Designs Experimental &Quasi-Experimental Grant Heller, Ph.D. PYC 5040
Fixed Designs• Theory driven• Should always be piloted first• Manipulation check may be useful• Confirmatory (fixed design) vs. Exploratory approach• Reliability• Construct Validity – Face – Predictive criterion – Internal – External (generalizability)
Experimental Research• Issues o Random selection and assignment o Group equivalence• Control & Comparison groups o Control (no treatment, wait list, placebo) o Comparison (standard treatment)• Assessing the impact of the Intervention / Manipulation o Manipulation checks o Treatment fidelity
Validity• Internal Validity o extent to which the changes in the study DV can be attributed to changes in the IV• External Validity o extent to which the results can be generalized
Threats to Internal Validity (Campbell & Stanley, 1963)• History• Testing• Instrumentation• Statistical Regression (to the mean)• Differential Mortality• Maturation• Selection• Selection X Maturation (interaction)• Experimenter Bias• Ambiguity about causal direction (A B or B A?)• Diffusion of treatments• Compensatory equalization of treatments• Compensatory rivalry
Threats to Internal Validity• Remember the acronym: MRS SMITH – Maturation – Regression to the mean – Selection of subjects – Selection by maturation interaction – Mortality – Instrumentation – Testing – History
Controlling for Threats to Internal Validity• Random assignment *** – Random assignment vs. Random selection• Matching – To ensure equivalency between groups• Blocking – To determine effects of extraneous variables• Holding extraneous variables constant – Reduces generalizability• Controlling for effects of extraneous variables (covariates) statistically – ANCOVA, MMR, partial correlation, etc.
Blocking example Therapy Wait List IQ > 110 IQ > 110Therapy Wait List Therapy Wait List IQ < 110 IQ < 110
Maximizing Internal Validity (Fraenkel & Wallen, 1993)1. Standardization of conditions – Minimize history & instrumentation2. Obtain as much info on participants as possible – Minimize mortality & selection3. Tighten up procedures of study – Minimize history & instrumentation4. Choose appropriate research design – Helps control most threats to internal validity
Threats to External Validity (generalizability) (LeCompte & Goetz, 1982)• Selection – Address through random selection• Setting• History• Construct effects
Threats to External Validity cont.• Selection X Treatment Interaction• History X Treatment Interaction• Testing X Treatment Interaction• Demand Characteristics• Hawthorne Effect• Order Effects (aka carryover effects & multiple treatment interference)
Ways to Increase External Validity• Random sampling/selection *** – Stratified random sampling – Cluster sampling• Naturalistic Research – Internal validity at expense of external validity – Analogue research• Single- and Double-Blind Research• Counterbalancing
Defense against threats to validity• for External Validity o Random selection of subjects• for Internal Validity o Random assignment to conditions
Internal vs. External Validity• Tradeoff between Internal & External Validity• How do we prioritize one over the other? – Most would argue in favor of internal validity• Mook (1983) In Defense of External Invalidity – “to what populations, settings, and so on, do we want the effect to be generalized?” (p. 379) – “we are not making observations, but testing them.” (p. 380) – Lab experiments allow us to test theory, find out what is possible, and break down phenomenon.
Specific Research Designs & Strategies• True Experimental Research – Random assignment to groups, receive different levels of manipulated variable• Quasi-Experimental Research – Random assignment is not possible (pre-existing groups)• Correlational Research – To be covered at a later date – Variables measured rather than manipulated• Developmental Research• Time-Series Design• Single-Subjects Designs• Qualitative Research – will covered
Experimental Fixed Designs• Assignment of Ss to different conditions• Manipulation of at least 1 variable (IV)• Measurement of effects of manipulation on 1 or more variables (DV’s)• Control of all other variables• Experimental realism vs. Mundane realism• Demand characteristics – Deception: but at what cost?• Expectancy effects – Double blind procedures
3 Essential Properties of a Well Designed Experiment (Leary, 2004)1. Manipulation of 1 or more Independent Variables (IVs)2. Random assignment to groups 1. Assure initial group equivalence3. Adequate control of extraneous variables
3 Aspects of Experimental Design• 1.) the number of independent variables (IV’s)• 2.) the number of treatment conditions – Levels of IV’s• 3.) whether the same or different subjects are used in each treatment condition.
Types of Experiments• Between-Subjects Design• Within-Subjects Design (repeated measures)• Mixed-Design – Combines between & within subjects designs• Single-Subject Design
Three Pre-Experimental Designs• 1.) The one-shot case study X OVulnerable to: History, Maturation, Selection, Mortality, Selectio n X Treatment Avoid!
Three Pre-Experimental Designs• 2.) The one-group pretest-posttest design O1 X O2Vulnerable to: History, Maturation, Testing, Instrumentation, Regression (?), Selection X Maturation, Selection X Treatment Avoid!
Three Pre-Experimental Designs• 3.) The static group comparison __ __O1 X __ O2Vulnerable to: Selection, Mortality, Selection X Maturation, Maturation (?), Selection X Treatment Avoid!
Designs to Avoid• Post-test only design – Problem: impossible to determine change from pre- treatment (no baseline measure) – Suggestion: improve design or adopt case study methodology• Post-test only non-equivalent groups – Problem: no baseline measure, so any differences between groups cannot be attributed to treatment – Suggestions: incorporate a pre-test; employ random assignment when possible; consider case study• Pre-test post-test single group design – Problem: widely used, but vulnerable to history, maturation regression. – Suggestion: add 2nd pre-tested no-treatment control group
True Experimental Designs• Two group designs – Post-test-only randomized control trial (RCT) – Post-test-only two treatment comparison – Pre-test post-test RCT – Pre-test post-test two treatment comparison• Three (or more) group simple designs• Factorial designs• Parametric designs• Matched pairs designs• Repeated measures designs – Within-groups design
Three True Experimental Designs• 4.) The pretest-posttest control group design R O1 X O2 R O3 O4
Three True Experimental Designs• 5.) The Solomon four-group design R O1 X O2 R O3 O4 R X O5 R O6
Three True Experimental Designs• 6.) The posttest-only control group design R X O1 R O2
Within-Subjects Designs• Advantages • Disadvantages – Increased statistical – Order effects power • Address through • Fewer participants counterbalancing needed • Latin square design • Carryover effects may still exist 1st 2nd 3rd 4th Group 1 0 mg 100 mg 600 mg 300 mg Group 2 100 mg 300 mg 0 mg 600 mg Group 3 300 mg 600 mg 100 mg 0 mg Group 4 600 mg 0 mg 300 mg 100 mg
Posttest-Only One-Way Designs • Randomized groups design Random Initial IV DV assignment Sample manipulated measured to groups • Matched-subjects design Ss in each block Initial Matched randomly IV DV assigned to manipulated measuredSample into blocks groups • Repeated measures design Initial Receives 1 DV Receives DV another levelSample level of IV measured of the IV measured
Pretest-Posttest-Only One-Way Designs • Randomized groups design DV Random DV Initial IV manip- measured assignment measuredSample to groups ulated (pretest) (posttest) • Matched-subjects design Ss in blocks DV Match DV Initial randomly IV manip- measured into assigned to measuredSample ulated (pretest) blocks groups (posttest) • Repeated measures design DV Receive DV Receive DV Initial measured another measured measured one levelSample (pretest) of IV posttest level of IV posttest #1 #2
2 X 2 Factorial Design Independent Variable A A1 A2 Also notated: R X11 OIndependent B1 Variable B R X12 O R X21 O B2 R X22 O
3 X 2 Factorial Design Independent Variable A A1 A2 A3Independent B1 Variable B B2
2 X 2 X 2 Factorial Design Independent Variable A A1 A2 Independent Variable B Independent Variable B B1 B2 B1 B2Independent C1 C1 Variable C C2 C2
2 X 2 X 2 Factorial Design Same design, different notation A1 A2 B1 B2 B1 B2C1 C2 C1 C2 C1 C2 C1 C2
Factorial Design: the null outcome http://www.socialresearchmethods.net/
Factorial Designs: main effects 1 http://www.socialresearchmethods.net/
Factorial Designs: main effects 2 http://www.socialresearchmethods.net/
Factorial Designs: main effects 3 http://www.socialresearchmethods.net/
Experimental Designs: when to use• Matched designs – Matched variables correlate with DV; measurement of matched variable unlikely to influence treatment effect• Repeated measures designs – Order effects unlikely; IV’s lend to repeated measurement; would likely be exposed in real life; individual differences likely to mask treatment effects• Simple two group designs – Order effects likely; IV(s) don’t lend to repeated measurement; Ss may be sensitized by pretesting or matching; not likely to get all treatments in real life.• Before-after / pre-post design – Pre-testing unlikely to affect Tx effects; concerns whether random assignment has produced equivalent groups; individual differences may mask Tx effects• Factorial designs – Interested in > 1 IV & interaction effects a concern• Parametric designs – IV(s) have a range of values or levels of interest; wish to investigate form or nature of relationship between IV and DV
Quasi-experiments“A research design involving an experimentalapproach but where random assignment totreatment and comparison groups has notbeen used” (Campbell & Stanley, 1963).
Quasi-experiments• Experimental approach, but random assignment not used• Typically employ naturally occurring groups – Classrooms, clinics, organizations, geographic areas, etc.• Generally do not possess same degree of internal validity as true experiments
Common threats to internal validity of quasi-experimental designs• Pretest-posttest designs – History – Maturation – Regression (to the mean) – Pretest sensitization• Two or more nonequivalent groups – Selection bias – Local history