1. EBM /Critical Appraisal
Nahid Sherbini , MD
Internist, Pulmonary Fellow
Teacher Assistant , Principle & Practice of Clinical Research Course PPCR
,Harvard Medical School
KFH-Medina
2. Introductory Lecture: Objectives
1. What
What is evidence-based medicine?
What does it look like in practice?
2. How
Formulate Clinical Questions
Search for Evidence
Appraisal of research
Apply to clinical problem
•
•
•
1.
2.
3.
3. What is EBM?
“Evidence-based medicine is the integration of best
research evidence with clinical expertise and patient
values”
- Dave Sackett
Patient
Concerns
Clinical
Expertise
Best research
evidence
EBM
8. Components of Clinical Questions
Patient/
Population Outcome
Intervention/
Exposure
Comparison
In patients
with
acute MI
In post-
menopausal
women
does early treat-
ment with a statin
does hormone
replacement
therapy
compared to
placebo
compared to no
HRT
decrease cardio-
vascular
mortality?
increase the
risk of
breast cancer?
9. Types of Research Questions
Descriptive
Means, frequency,
prevalence
“74% of MRSA
isolates occur after
the 15th
day of
hospitalization”
Associative
Correlations, bivariates
“Smoking (yes vs.
no) is associated
with lung cancer”
“Duration of
smoking increases
risk for lung
cancer”
Comparison
T tests, ANOVA
“Drug A led to a 15
mg/dl reduction in
LDL cholesterol
versus placebo”
Increasing strength
11. Study Design Types
Case Report/Case-series
Case Control/Retrospective
Cohort/Prospective
RCT
12. Definition of meta-analysis
(1) “Meta-analysis refers to the analysis of analyses…the
statistical analysis of a large collection of analysis results
from individual studies for the purpose of integrating
findings. It connotes a rigorous alternative to the casual,
narrative discussions of research studies which typify our
attempts to make sense of the rapidly expanding
literature…”
GV Glass (1976). Primary, secondary, and meta-analysis of
research. Edu Researcher 5:3-8
•
•
13. Systematic review
Review that has been prepared using a systematic approach to minimizing
bias and random errors which is documented in the Methods section
Meta-
analyses
Systematic reviews
14. Egger, M. et al. BMJ 1997;315:1371-1374
Cumulative meta-analysis of RCTs of oral beta blockers after MI:
results for nal mortality
Forest plot
17. Users' Guides for RCT
I. Are the results of the study valid?
Primary Guides:
Was the assignment of patients to treatments randomized?
Were all patients who entered the trial properly accounted for and attributed at its
conclusion?
Was follow-up complete?
Were patients analyzed in the groups to which they were randomized?
Secondary Guides:
Were patients, health workers, and study personnel "blind" to treatment?
Were the groups similar at the start of the trial?
Aside from the experimental intervention, were the groups treated equally?
II. What were the results?
How large was the treatment effect?
How precise was the estimate of the treatment effect?
III. Will the results help me in caring for my patients?
Can the results be applied to my patient care?
Were all clinically important outcomes considered?
Are the likely treatment benefits worth the potential harms and costs?
•
•
•
•
•
•
•
•
•
•
•
•
•
•
18. Are the results of this study valid?
1. Were patients randomized to treatment?
2. Was follow-up long and complete?
3. Were patients & clinicians blinded?
4. Were groups treated equally?
5. Were the groups similar at the start?
1
19. Was the assignment of patients to treatment
randomized?
Was the randomization list concealed?
1.
Why random allocation?
Why concealed from the clinicians?
How the study was randomized?
•
•
•
22. Randomization
It can lead to long runs (stretches of
treatment assignment):
ABBAAAAAAABAAAAAABBBA
It can produce imbalances
•
•
Simple
randomization
Blocking
Stratification
•
•
•
ABBA or BBAA or
Random block size
Too many small strata can
defeat the balancing effects
23. Are the results of this study valid?
1. Were patients randomized to treatment?
2. Was follow-up long and complete?
3. Were patients & clinicians blinded?
4. Were groups treated equally?
5. Were the groups similar at the start?
24. 2. Was follow-up of patients sufficiently
long and complete?
If all patients who were entered into the trial were accounted for
at its conclusion.
Look at Tables:
Baseline characteristics
Outcome table
•
•
•
•
26. Adherence in Patient Practices
Primary concern and greatly affects treatment outcomes
Strongly tied with patient-provider rapport and communication
HIGHLY variable, lower for preventative measures and for chronic
conditions
- Evolving concern: Complex treatments, drug resistance, strategies
to improve adherence have become a priority
- Unintentional (e.g. forgetting) or Intentional (e.g. personal judgment)
Increased medication-related hospital admissions
- Adherence itself appears to offer some benefit! (i.e. placebo)
-
-
-
-
27. Are the results of this study valid?
1. Were patients randomized to treatment?
2. Was follow-up long and complete?
3. Were patients & clinicians blinded?
4. Were groups treated equally?
5. Were the groups similar at the start?
28. 4. Were Patients And Clinicians Kept
Blind To Treatment?
Investigator
Care taker
•
•
29. Are the results of this study valid?
1. Were patients randomized to treatment?
2. Was follow-up long and complete?
3. Were patients & clinicians blinded?
4. Were groups treated equally?
5. Were the groups similar at the start?
30. 4. Apart from The Experimental
Therapy, Were Groups Treated
Equally?
31. Are the results of this study valid?
1. Were patients randomized to treatment?
2. Was follow-up long and complete?
3. Were patients & clinicians blinded?
4. Were groups treated equally?
5. Were the groups similar at the start?
32. 5. Were the groups similar at the
start of the trial?
33. How To Use An Article On
Therapy
Are the results valid?
What are the results?
Will the results help me in patient care?
•
•
•
Appraisal:
2
34. Are The Valid Results Of This
Individual Study Important?
1. What is the magnitude of the treatment effect?
2. How precise is this estimate of the treatment effect?
Important
2
35. Variables
Dependent Variable: Your outcome
Death, stroke, blood pressure
Independent Variable: Your intervention, predictor
Drug A, blood pressure
Covariates: A secondary factor that may influence the
relationship between Dependent and Independent
variables
Age, gender, race
•
•
•
•
•
•
36. Surrogates
Sometimes the primary question doesn’t have a feasible, safe,
or cost-permissive direct answer
“Does long-term sodium restriction improve risk of diabetic
retinopathy?”
Evaluation of a closely associated surrogate can be more
practical
“Does 3 weeks of dietary sodium restriction improve glucose
tolerance as measured by an insulin clamp?”
•
•
•
•
37. Continuous data
Categorical /
binary
Normal
distribution
Non-normal
Compare two groups
(independent variable -
binary)
Unpaired and
paired t-test
Mann-Whitney
Wilcoxon
Chi-square
Fisher’s exact
Compare three or more
groups (independent
variable - categorical)
ANOVA (one or
n-way ANOVA)
Kruskal-Wallis
or Friedman test
Chi-square
Fisher’s exact
Association between
two variables
(independent variable -
continuous)
Pearson
correlation
Spearman
correlation
Association between
three or more variables
(independent variable -
continuous)
Multiple linear
regression
Multiple logistic
regression
39. What is the difference between covariate
adjustment and subgroup analysis?
Important difference between subgroup analysis and covariate
adjustment
It is all about assessing variability
•
•
40. Covariate adjustment
To achieve the most appropriate p-value for the treatment
difference
To improve the precision of the estimated treatment
difference, thus increasing the statistical power of the trial.
Here you are not interested in learning how groups respond to treatments,
only to increase efficiency
(popcock, 2002)
•
•
41. Subgroup analysis – the issue of different response by
treatment
Statistical tests for interaction (evidence for the treatment
difference varying between subgroups) – best approach.
IMPORTANT: Assessing the differences in each subgroup
separately is misleading as if the overall treatment is
significant then subgroups will likely be significant – and
others might not be due to the small sample size.
•
•
42. Definitions
“Control event rate”, CER
“Experimental event rate”, EER
Absolute risk reduction
Relative risk reduction
Number needed to treat (NNT)
Number needed to harm (NNH)
•
•
•
•
•
•
43. Disorder Intervention Event to
prevent
Control Exp NNT
Carotid
Stenosis
(severe)
Endarterectomy Stroke or
Death
20% 10%
Carotid
Stenosis
(mild)
Endarterectomy Stroke or
Death
2% 1%
Definitions
Event rates
n with event / total
Control event rate (CER)
Experimental event rate (EER)
•
•
•
•
44. Absolute risk reduction
difference in two event rates
CER - EER = ARR
Relative risk reduction
proportion of control rate
CER-EER / CER = RRR
Number needed to treat
1 / ARR = NNT
•
•
•
•
•
•
•
•
10/20 = 50%
20 – 10 = 10%
Disorder Intervention Event to
prevent
Control Exp NNT
Carotid
Stenosis
(severe)
Endarterectomy Stroke or
Death
20% 10%
1/ 10% = 100/10 = 10
45. Absolute risk reduction
difference in two event rates
CER - EER = ARR
Relative risk reduction
proportion of control rate
CER-EER / CER = RRR
Number needed to treat
1 / ARR = NNT
•
•
•
•
•
•
•
•
1/2 = 50%
2 – 1 = 1%
Disorder Intervention Event to
prevent
Control Exp NNT
Carotid
Stenosis
(mild)
Endarterectomy Stroke or
Death
2% 1%
1/ 1% = 100/1 = 100
46. Absolute risk reduction
difference in two event rates
CER - EER = ARR
Relative risk reduction
proportion of control rate
CER-EER / CER = RRR
Number needed to treat
1 / ARR = NNT
•
•
•
•
•
•
•
•
RRR = 50%
Disorder Intervention Event to
prevent
Control Exp NNT
Carotid
Stenosis
(severe)
Endarterectomy Stroke or
Death
20% 10% 10
Carotid
Stenosis
(mild)
Endarterectomy Stroke or
Death
2% 1% 100
47. Level I: N of 1 randomized trial (double-blinded, cross-over)
Level I (A): Systematic reviews of randomized trials
Level I (B): Single randomized trial
Level II (A): Systematic review of observational studies addressing
patient-important outcome
Level II (B): Single observational study addressing important outcome
Level III: Physiologic studies
Level IV: Unsystematic clinical observations (case-reports, anecdotal)
Levels of Evidence
Hierarchy of Strength of Evidence for Treatment Decisions
JAMA 2000; 284(10):1290-96
48. How To Use An Article On Therapy
Are the results valid?
What are the results?
Will the results help me in patient
care ?
•
•
•
Appraisal:
3
49. Is this study applicable to our patient?
1. Is our patient so different from those in the study?
2. Is the treatment feasible?
3. What are our patient’s potential benefits and harms from the
therapy?
4. What are our patient’s values and expectations?
