1. The unity of all science consists
alone in the method, not its material.
Pearson K. The grammar of science. London, Black, 1892.
2. Statistics is the study of uncertainty.
Savage LJ. The foundations of statistics. New York, Wiley,
1954.
3. The aim of statistics reviewing
Accurate and transparent description of the uncertainty in
presented findings.
“Statisticians are experts in handling uncertainty”.
Lindley DV. The philosophy of statistics. The Statistician 2000;49:293-337.
4. Medical research methodology
Ethics concerns
Randomization
Experi- Controlled conditions
mental Internal validity
Short follow up
Sample size restrictions
Study
design
Obser- Few ethics concerns
vational Bias adjustments
External validity
Long follow up
Few sample size restrictions
5. Statistical aspects – internal validity
Internal validity by design (blocking of
Experi- known risk factors and randomization of
mental other)
Potential for confounding: none
Study
design
Internal validity by statistical analysis
Obser- (confounding adjustment for known and
vational measured risk factors)
Potential for confounding: massive
6. Confounder (or case-mix) adjustment
How much of the variation in endpoints can be explained by known
factors, and how much has unknown causes?
Variation with unknown origin
95%-99% Arthroplasty revision
85%-95% EQ-5D, SF36
70%-80% Coronary heart disease
7. Risk factors, confounding, and the
illusion of statistical control
'...it is essential to remember that “statistical control” is nothing more
than a highly fallible process filled with judgment calls that often go
unnoticed in practice.'
Christenfeld NJS, Sloan RP, Carrol D, Greenland S.
Psychosomatic Medicine 2004;66:868–875
12. Statistics
We calculated odds ratios by logistic regression
analysis, to estimate the relationship between
failure of the osteotomy and possible preoperative
risk factors. We performed multivariate, stepwise
(backward) logistic regression and entered
variables with a p-value of ≤ 0.05 into the model.
13. Unified theory of bias
Bias can be reduced to or explained by 3 structures
1. Reverse causation
Outcome precedes exposure measurement or outcome can have
effect on exposure. Measurement error or Information bias.
2. Common cause
Confounding by association, confounding by indication.
3. Conditioning on common effects
Collider, selection bias, time varying confounding.
14. Covariate selection
Adequate Background Knowledge
Confounder identification must be based on understanding of the
causal structure linking the variables being studied (treatment and
disease).
Condition on the minimal set of variables necessary to remove
confounding.
Inadequate Background Knowledge
Remove known instrumental variables, colliders, intermediates
(variables with post treatment measurement.
15. Confounding
Under-adjustment
occurs when a confounder is not adjusted for.
Over-adjustment
can occur from adjusting instrumental variables, intermediate
variables, colliders, variables caused by outcome.
16. Confounder
Common cause, i.e., confounder
Confounder L distort the effect of
treatment A on disease Y
Always adjust for confounders, unless
small data set and confounder has
strong association with treatment and
week association with outcome
17. Confounder example
A = treatment
1: statin alone
0: niacin alone
L = Baseline Cholesterol
1: LDL ≥ 160 mg/dL
0: LDL < 160 mg/dL
Y = Myocardial infarction
1: Yes
0: No
18. Intermediate variable
Adjusting for intermediate variable I in a
fixed covariate model will remove the effect
of treatment A on disease/outcome Y
In a fixed covariate model we do not want to
include variables influenced by A or Y
19. Intermediate example
A = treatment
1: statin alone
0: niacin alone
I = Post-treatment Cholesterol
1: LDL ≥ 160 mg/dL
0: LDL < 160 mg/dL
Y = Myocardial infarction
1: Yes
0: No
20. Collider
Adjusting for a collider can produce bias
Conditioning on common effect F without
adjustment of U1 or U2 will induce an
association between U1 and U2, which will
confound the association between A and Y
22. Variables associated with
treatment or disease only
Inclusion of variables associated with treatment only can cause bias
and imprecision
Variables associated with disease but not treatment (risk factors)
can be included in models. They are expected to decrease variance
of treatment effect without increasing bias
Including variables associated with disease reduces the chance of
missing important confounders
27. Any claim coming from an observational study is
most likely to be wrong
12 randomised trials have tested 52 observational claims (about the
effects of vitamine B6, B12, C, D, E, beta carotene, hormone replace-
ment therapy, folic acid and selenium).
“They all confirmed no claims in the direction of the observational
claim. We repeat that figure: 0 out of 52. To put it in another way,
100% of the observational claims failed to replicate. In fact, five
claims (9.6%) are statistically significant in the opposite direction to
the observational claim.”
Young S, Karr A. Deming, data and observational studies.
Significance, September 2011.
28.
29. Medical research methodology
Hypothesis generation Pre-specified hypotheses
Exploration Confirmation
Academic analysis freedom Legislation, regulatory
guidelines
Uncertainty tolerance Uncertainty intolerance
Aetiology Study scope Treatment
30. Medical research methodology
Experi-
Laboratory Randomized clinical
mental
experiments trials
Study
design
Obser-
vational Epidemiological Patient register
studies studies
Aetiology Study scope Treatment
31. Statistical aspects - precision
Bonferroni correction Protected type-1 error rate
Experi- within endpoints for specified endpoints
mental
Few type-2 error Sample size based on the
considerations type-2 error rate
Study
design
Multiplicity issues Specified type-1 and -2 error
Obser- not addressed uncertainty
vational (confidence intervals)
Sample size not based on
type-2 error rate No multiplicity consideration
for safety endpoints
Aetiology Study scope Treatment
32. Drug development
Discovery
Phase 1
(Phase 0)
Phase 2
Experi- Phase 3
mental
Phase 4
Study
design
PMS
(Phase 5)
Obser-
vational
Aetiology Study scope Treatment
33. Device development
Biomechanics
in vitro, etc. Randomized
Experi- performance
mental trials
Study
design
Safety
follow-up
Obser- in registries
vational
Aetiology Study orientation Treatment
34. It is impossible to do clinical research so badly that
it cannot be published
“There seems to be no study too fragmented, no hypothesis too
trivial, no literature citation too biased or too egotistical, no design
too warped, no methodology too bungled, no presentation of
results too inaccurate, no argument too circular, no conclusions
too trifling or too unjustified, and no grammar and syntax too
offensive for a paper to end up in print.”
Drummond Rennie 1986 (editor of NEJM and JAMA)
35.
36.
37.
38.
39.
40. Arthroplasty registry analyses
Crucial issues
- Fulfillment of methodological assumptions (Gaussian distr,
homogeneity of variance, proportionality, linearity, etc.)
- Confounding adjustment (risk factors, causality, linearity, etc.)
- Clinical significance and estimation uncertainty (95%CI).
Should be avoided
- P-value culture
- Bonferroni correction
- Post-hoc power
- Predictions
Blocked backdoor path. F meets the criteria for traditional confounder, but it is not a counfounder and this is not a confounded study
Causal null: whether having low education increases risk for type II diabetes. We measured mother’s diabetes status, but do not have measures of family income when the individual was growing up or if the mother had any genes that would increase the risk for diabetes. Under the assumptions in the DAG, should we adjust for mother’s diabetes status? Assumptions that if poor during childhood, then poor as adult and poor associated with diabetes and low education. Mother’s diabetes status will be statistically associated with education. They share a common prior cause. Meets criteria for statistical association Conditioning on mother’s diabetes unblocks the blocked backdoor path and induces a spurious statistical association between low education and diabetes. Does not meet criteria for graphical confounder. Basketball player tall or fast.
