This is an updated version of my Comparing Research Designs lecture, which now includes discussions on: (1) common considerations with research design such as bias, reliability, validity, and confounding; and (2) expanded discussion of RCT designs including factorial and cross-over designs.

1. Comparing
Research Designs
Patrick Barlow
Statistical and Research Design Consultant,Graduate School of Medicine,UTK
PhD Candidate in Evaluation,Statistics, and Measurement,UTK

2. On the Agenda
 Important considerations in research design
 Reliability & validity
 Biases & confounding
 Strength of evidence
 Observational Research Designs
 Cross-sectional study
 Case-control study
 Cohort study
 Experimental Research Designs
 The Basics of Factorial and CrossoverTrials

3. Important considerations
in research design
Confounding
Bias
Reliability
Validity

4. Reliability &Validity
Reliability Validity
 Refers to the consistency of an
instrument/measurement.
 Thought of as an individual’s “true
score” on the phenomenon you aim to
measure minus “measurement error”
 Two common types of reliability
 Internal consistency: Cronbach’s
alpha, KR20
 Inter-Rater: Kappa statistic
 Necessary but not sufficient in
determining validity.
 Refers to the accuracy of an
instrument/measurement
 In other words, “the degree to which
you’re measuring what you claim to
measure”
 Two broad types of validity
 Internal validity
 External validity

5. Internal vs. ExternalValidity
 One of the strengths of randomized designs are that they have
substantially higher internal & external validity.
 InternalValidity: refers to the integrity of the experiment itself. It is
the ability to draw a causal link between your treatment and the
dependent variable of interest.
 ExternalValidity: refers to the ability to generalize your study findings
to the population at large. In other words, are your findings from a
sample of UTMCK patients with HTN going to apply to all patients with
HTN?

6. Threats to InternalValidity
Concerns the accuracy of measurement within the
study
 Shadish, Cook & Campbell (2002) summarized a
number of possible threats to internal validity, which
can severely jeopardize the findings of a study. In
particular:
 History, Mortality, & Maturation
 Repeated Testing
 Confounding
 Diffusion & Compensatory Rivalry

7. Threats to InternalValidity
 Diffusion & Compensatory Rivalry
 Diffusion: Treatment effects can “spill over” or “spread” across
treatment groups. EX: Patients from different groups live near each
other and discuss / share their experiences or treatments.
 Compensatory Rivalry: Patients perform in a certain way because
they know they’re in the control / experimental groups.

8. Threats to InternalValidity
 History, Mortality, & Maturation
 History: events external to the experiment influence the participants’. EX:
Superstorm Sandy hits during a crossover trial in New Jersey.
 Mortality: Patients either die (mortality) or drop out of the study (attrition) at
different rates.
 Maturation: Patients change over the course of the treatment, which
influences results. EX: Children grow up during the course of a pediatric clinical
trial.
 RepeatedTesting
 Patients can become “test-wise” if given the same subjective test multiple
times, or they become conditioned to being tested (EX: patient’s pulse
increases before a needle stick).

9. ExternalValidity
 The ability to generalize the findings of your study to the
relevant population.
 Threatened by
 Bias
 Confounding
 Non-experimental design (i.e. case-control vs. RCT)
 Lack of randomization
 External validity is the strongest when a true experimental
design is used.

10. Confounding
 A confounder is a variable that is causally associated with
the outcome (DV) and may or may not be causally
associated with the exposure (IV)
 Causes spurious conclusions & inferences to be made
about a set of variables
 Reduced through
 Randomization
 Matching
 Statistically controlling (covariates)

11. Confounding Example
Smoking Hx
HPV
Cervical
Cancer
?

12. Bias in Research
 The result of systematic
error in the design or
conduct of a study
 Can artificially “trend”
results
 Toward the Null hypothesis
 Toward the Alternative hypothesis
 A major problem to
consider when planning any
study

13. Common Biases
 Selection bias: one relevant group in the population (e.g. cases
positive for predictor variable) has a higher probability of being
included in the sample
Misclassification can be either unsystematic (random) or
systematic (bad)
 Information: bias from erroneously classifying people in
exposure/outcome categories
Recall/Response: bias associated with inaccurate recall of
exposure or representation of true exposure (self-report)
Experimenter/Interviewer bias: Differential treatment of
participants in treatment and control groups
 Publication: the tendency to publish only “positive” or
“significant” findings.

14. Strength of Evidence
The Bradford Hill Criteria
 Provides researchers with seven criteria for assessing
strength of evidence.
 Strength of association (i.e. effect size)
 Consistency (i.e. reliability)
 Specificity
 Temporal relationship
 Biological gradient
 Plausibility
 Coherence
 Experiment (reversibility)
 Analogy (consideration of alternate explanations)

15. Pyramid of Clinical Evidence
RCT
Cohort
Studies
Case Control
Studies
Case Series
Case Reports
Ideas, Editorials, Opinions
Animal research
In vitro (‘test tube’) research
Systematic Reviews
& Meta-analyses
Evidence
Summaries
Level 2 Evidence
Level 1 Evidence
Level 3 Evidence
Cross-Sectional
Studies: Level 2.3

16. Observational Research Designs
Cross-sectional
Case-control
Cohort

17. Cross-Sectional Studies
 “Snapshot” of a population.
 People are studied at a
“point” in time, without
follow-up.
 Strength of evidence…
 What are some research
questions that can be
answered with cross-
sectional designs?

18. Advantages and Disadvantages of Cross-
Sectional Studies
Advantages Disadvantages
 Fast and inexpensive
 No loss to follow-up
 Springboard to
expand/inform research
question
 Can target a larger sample
size
 Can’t determine causal
relationship
 Impractical for rare diseases
 “Garbage in, garbage out”
 Risk for nonresponse

19. Case-Control Studies
 Always retrospective
 Prevalence vs. Incidence
 A sample with the disease from a population is selected
(cases).
 A sample without the disease from a population is selected
(controls).
 Groups are compared using possible predictors of the
disease state.

20. Advantages and Disadvantages of Case-
Control Studies
Advantages Disadvantages
 High information yield
with few participants
 Useful for rare outcomes
 Cannot estimate incidence
of disease
 Limited outcomes can be
studied
 Highly susceptible to
biases

21. Strategies for Sampling Controls
 Population versus hospital/clinic-based controls
 Matching
 Individual level
 Group level
 Using two or more control groups

22. Cohort Studies
 A “cohort” is a group of individuals who are followed or
traced over a period of time.
 A cohort study analyzes an exposure/disease relationship
within the entire cohort.
 Groups selected based on exposure to a risk factor.
 Level of evidence?

23. Cohort Design

24. Prospective vs. Retrospective
Cohort Studies
Exposure Outcome
Prospective
Assessed at the
beginning of the
study (present)
Followed into
the future for
outcome
Retrospective
Assessed at some
point in the past
Outcome has
already occurred

25. Advantages and Disadvantages of
Cohort Studies
Advantages Disadvantages
 Establish population-based incidence
 Temporal relationship inferred
 Time-to-event analysis possible
 Used when randomization not possible
 Reduces biases (selection, information)
 Lengthy and costly
 Not suitable for rare/long-latency
diseases
 May require very large samples
 Nonresponse, migration and loss-to-
follow-up
 Sampling, ascertainment and observer
biases

26. Experimental Designs
The Basics of Factorial and Cross-Over Designs

27. Experimental Designs
What areThey?
 Considered to be the “gold standard” of clinical evidence
because:
 Randomization is used to reduce the effect of biases and
confounding variables
 Patients (single) and researchers (double) can be blinded to
the intervention
 High internal and external validity allow for assessing cause
and effect relationships.
 The most basic experimental design is a “Parallel
trial.”
 Patients are randomized into one of two groups, and
remain in the same group throughout the study.
“Double-blind trials”

28. Factorial Designs
What areThey?
 Factorial designs allow for researchers to
test multiple interventions or treatment
combinations in a single study.
 For example: drug A or Drug B and 3x per week or
everyday dose cycle.
 The simplest form of this design is a 2x2
factorial design.
 Allows researchers to test individual
treatment effects and/or interactions
between different treatments.
 Looks like a “grid”

29. Factorial Designs
Why areThey Used?
 Factorial design are commonly used to effectively test
multiple treatments or “Main effects” in a single study.
 More efficient and more statistically powerful than multiple single intervention studies
 Especially useful for testing interactions among different
interventions or treatments
Main Effects
Interactions

30. Factorial Designs
Example
Dose Cycle
Statin
Rosuvastatin
(Crestor)
Atorvastatin (Lipitor)
3x Per Week M LDL M LDL
Everyday M LDL M LDL
What is the effect of dose (3x pw or everyday) and statin
(Rusuvastatin or Atorvastatin) regimen on mean LDL Cholesterol?

31. Cross-over Designs
What areThey?
 A cross-over trial design involves giving the two or
more interventions/treatments to a single group of
patients.
 At its most basic, this trial tests the efficacy of two
treatments where each patient spends a period of time
under both treatment options.
 Patients are randomized into which treatment they
receive first, and then swap to the other treatment
after a predetermined time.

32. Cross-over Designs
What areThey?
A
B
‚Cross-over‛
A
B

33. Cross-Over Designs
Why areThey Used?
 Cross-over trials are useful because they reduce confounding
factors associated with between-subjects designs.
 Patients serve as their own controls
 Useful for time-dependent research questions
 Higher statistical power than between subjects designs due to no between-subjects
error (i.e. need less patients to find statistical significance).

34. Cross-Over Designs
Example
3x Per
Week
Treatment
Everyday
Treatment
Everyday
Treatment
3x Per
Week
Treatment
Week Six

35. Disadvantages of RCT Designs
 Extremely time and
resource demanding
 Unethical in many
situations
 Poor external validity if the
RCT is too highly controlled
 Difficult to study rare
events
 Therapeutic misconception

36. In Pairs…
Work together to brainstorm an example of how your topic
could be addressed using 1) a Cross-Sectional design, 2) a
case-control design, 3) a prospective or retrospective cohort
design, and an RCT (Parallel, factorial, or cross-over).
 Be prepared to share your responses