Comparing research designs fw 2013 handout version


Published on

This is an updated version of my Comparing Research Designs lecture, which now includes discussions on: (1) common considerations with research design such as bias, reliability, validity, and confounding; and (2) expanded discussion of RCT designs including factorial and cross-over designs.

Published in: Education, Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Comparing research designs fw 2013 handout version

  1. 1. Comparing Research Designs Patrick Barlow Statistical and Research Design Consultant,Graduate School of Medicine,UTK PhD Candidate in Evaluation,Statistics, and Measurement,UTK
  2. 2. On the Agenda  Important considerations in research design  Reliability & validity  Biases & confounding  Strength of evidence  Observational Research Designs  Cross-sectional study  Case-control study  Cohort study  Experimental Research Designs  The Basics of Factorial and CrossoverTrials
  3. 3. Important considerations in research design Confounding Bias Reliability Validity
  4. 4. Reliability &Validity Reliability Validity  Refers to the consistency of an instrument/measurement.  Thought of as an individual’s “true score” on the phenomenon you aim to measure minus “measurement error”  Two common types of reliability  Internal consistency: Cronbach’s alpha, KR20  Inter-Rater: Kappa statistic  Necessary but not sufficient in determining validity.  Refers to the accuracy of an instrument/measurement  In other words, “the degree to which you’re measuring what you claim to measure”  Two broad types of validity  Internal validity  External validity
  5. 5. Internal vs. ExternalValidity  One of the strengths of randomized designs are that they have substantially higher internal & external validity.  InternalValidity: refers to the integrity of the experiment itself. It is the ability to draw a causal link between your treatment and the dependent variable of interest.  ExternalValidity: refers to the ability to generalize your study findings to the population at large. In other words, are your findings from a sample of UTMCK patients with HTN going to apply to all patients with HTN?
  6. 6. Threats to InternalValidity Concerns the accuracy of measurement within the study  Shadish, Cook & Campbell (2002) summarized a number of possible threats to internal validity, which can severely jeopardize the findings of a study. In particular:  History, Mortality, & Maturation  Repeated Testing  Confounding  Diffusion & Compensatory Rivalry
  7. 7. Threats to InternalValidity  Diffusion & Compensatory Rivalry  Diffusion: Treatment effects can “spill over” or “spread” across treatment groups. EX: Patients from different groups live near each other and discuss / share their experiences or treatments.  Compensatory Rivalry: Patients perform in a certain way because they know they’re in the control / experimental groups.
  8. 8. Threats to InternalValidity  History, Mortality, & Maturation  History: events external to the experiment influence the participants’. EX: Superstorm Sandy hits during a crossover trial in New Jersey.  Mortality: Patients either die (mortality) or drop out of the study (attrition) at different rates.  Maturation: Patients change over the course of the treatment, which influences results. EX: Children grow up during the course of a pediatric clinical trial.  RepeatedTesting  Patients can become “test-wise” if given the same subjective test multiple times, or they become conditioned to being tested (EX: patient’s pulse increases before a needle stick).
  9. 9. ExternalValidity  The ability to generalize the findings of your study to the relevant population.  Threatened by  Bias  Confounding  Non-experimental design (i.e. case-control vs. RCT)  Lack of randomization  External validity is the strongest when a true experimental design is used.
  10. 10. Confounding  A confounder is a variable that is causally associated with the outcome (DV) and may or may not be causally associated with the exposure (IV)  Causes spurious conclusions & inferences to be made about a set of variables  Reduced through  Randomization  Matching  Statistically controlling (covariates)
  11. 11. Confounding Example Smoking Hx HPV Cervical Cancer ?
  12. 12. Bias in Research  The result of systematic error in the design or conduct of a study  Can artificially “trend” results  Toward the Null hypothesis  Toward the Alternative hypothesis  A major problem to consider when planning any study
  13. 13. Common Biases  Selection bias: one relevant group in the population (e.g. cases positive for predictor variable) has a higher probability of being included in the sample Misclassification can be either unsystematic (random) or systematic (bad)  Information: bias from erroneously classifying people in exposure/outcome categories Recall/Response: bias associated with inaccurate recall of exposure or representation of true exposure (self-report) Experimenter/Interviewer bias: Differential treatment of participants in treatment and control groups  Publication: the tendency to publish only “positive” or “significant” findings.
  14. 14. Strength of Evidence The Bradford Hill Criteria  Provides researchers with seven criteria for assessing strength of evidence.  Strength of association (i.e. effect size)  Consistency (i.e. reliability)  Specificity  Temporal relationship  Biological gradient  Plausibility  Coherence  Experiment (reversibility)  Analogy (consideration of alternate explanations)
  15. 15. Pyramid of Clinical Evidence RCT Cohort Studies Case Control Studies Case Series Case Reports Ideas, Editorials, Opinions Animal research In vitro (‘test tube’) research Systematic Reviews & Meta-analyses Evidence Summaries Level 2 Evidence Level 1 Evidence Level 3 Evidence Cross-Sectional Studies: Level 2.3
  16. 16. Observational Research Designs Cross-sectional Case-control Cohort
  17. 17. Cross-Sectional Studies  “Snapshot” of a population.  People are studied at a “point” in time, without follow-up.  Strength of evidence…  What are some research questions that can be answered with cross- sectional designs?
  18. 18. Advantages and Disadvantages of Cross- Sectional Studies Advantages Disadvantages  Fast and inexpensive  No loss to follow-up  Springboard to expand/inform research question  Can target a larger sample size  Can’t determine causal relationship  Impractical for rare diseases  “Garbage in, garbage out”  Risk for nonresponse
  19. 19. Case-Control Studies  Always retrospective  Prevalence vs. Incidence  A sample with the disease from a population is selected (cases).  A sample without the disease from a population is selected (controls).  Groups are compared using possible predictors of the disease state.
  20. 20. Advantages and Disadvantages of Case- Control Studies Advantages Disadvantages  High information yield with few participants  Useful for rare outcomes  Cannot estimate incidence of disease  Limited outcomes can be studied  Highly susceptible to biases
  21. 21. Strategies for Sampling Controls  Population versus hospital/clinic-based controls  Matching  Individual level  Group level  Using two or more control groups
  22. 22. Cohort Studies  A “cohort” is a group of individuals who are followed or traced over a period of time.  A cohort study analyzes an exposure/disease relationship within the entire cohort.  Groups selected based on exposure to a risk factor.  Level of evidence?
  23. 23. Cohort Design
  24. 24. Prospective vs. Retrospective Cohort Studies Exposure Outcome Prospective Assessed at the beginning of the study (present) Followed into the future for outcome Retrospective Assessed at some point in the past Outcome has already occurred
  25. 25. Advantages and Disadvantages of Cohort Studies Advantages Disadvantages  Establish population-based incidence  Temporal relationship inferred  Time-to-event analysis possible  Used when randomization not possible  Reduces biases (selection, information)  Lengthy and costly  Not suitable for rare/long-latency diseases  May require very large samples  Nonresponse, migration and loss-to- follow-up  Sampling, ascertainment and observer biases
  26. 26. Experimental Designs The Basics of Factorial and Cross-Over Designs
  27. 27. Experimental Designs What areThey?  Considered to be the “gold standard” of clinical evidence because:  Randomization is used to reduce the effect of biases and confounding variables  Patients (single) and researchers (double) can be blinded to the intervention  High internal and external validity allow for assessing cause and effect relationships.  The most basic experimental design is a “Parallel trial.”  Patients are randomized into one of two groups, and remain in the same group throughout the study. “Double-blind trials”
  28. 28. Factorial Designs What areThey?  Factorial designs allow for researchers to test multiple interventions or treatment combinations in a single study.  For example: drug A or Drug B and 3x per week or everyday dose cycle.  The simplest form of this design is a 2x2 factorial design.  Allows researchers to test individual treatment effects and/or interactions between different treatments.  Looks like a “grid”
  29. 29. Factorial Designs Why areThey Used?  Factorial design are commonly used to effectively test multiple treatments or “Main effects” in a single study.  More efficient and more statistically powerful than multiple single intervention studies  Especially useful for testing interactions among different interventions or treatments Main Effects Interactions
  30. 30. Factorial Designs Example Dose Cycle Statin Rosuvastatin (Crestor) Atorvastatin (Lipitor) 3x Per Week M LDL M LDL Everyday M LDL M LDL What is the effect of dose (3x pw or everyday) and statin (Rusuvastatin or Atorvastatin) regimen on mean LDL Cholesterol?
  31. 31. Cross-over Designs What areThey?  A cross-over trial design involves giving the two or more interventions/treatments to a single group of patients.  At its most basic, this trial tests the efficacy of two treatments where each patient spends a period of time under both treatment options.  Patients are randomized into which treatment they receive first, and then swap to the other treatment after a predetermined time.
  32. 32. Cross-over Designs What areThey? A B ‚Cross-over‛ A B
  33. 33. Cross-Over Designs Why areThey Used?  Cross-over trials are useful because they reduce confounding factors associated with between-subjects designs.  Patients serve as their own controls  Useful for time-dependent research questions  Higher statistical power than between subjects designs due to no between-subjects error (i.e. need less patients to find statistical significance).
  34. 34. Cross-Over Designs Example 3x Per Week Treatment Everyday Treatment Everyday Treatment 3x Per Week Treatment Week Six
  35. 35. Disadvantages of RCT Designs  Extremely time and resource demanding  Unethical in many situations  Poor external validity if the RCT is too highly controlled  Difficult to study rare events  Therapeutic misconception
  36. 36. In Pairs… Work together to brainstorm an example of how your topic could be addressed using 1) a Cross-Sectional design, 2) a case-control design, 3) a prospective or retrospective cohort design, and an RCT (Parallel, factorial, or cross-over).  Be prepared to share your responses