critical appraisal

1. Overview
Critical Appraisals
of Medical Literature

2. What is Evidence-Based
Medicine?
“Evidence-based medicine is the integration of
best research evidence with clinical expertise
and patient values” (Sackett & Straus)
Best
research
evidence
Clinical
expertise
Patient
values
EB
M

3. WHY EBM?
1. Information overload
2. Keeping current with literature
3. Our clinical performance deteriorates with
time (“the slippery slope”)
4. Traditional CME does not improve clinical
performance
5. EBM encourages self directed learning
process which should overcome the above
shortages

4. Medical evidence increasing at epidemic
rates:
we all need EBP skills to keep up-to-date
MEDLINE 2010
2,000 articles / day
approx 75 new
trials
published
every day
Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)

5. The research-to-awareness
gap
0
500000
1000000
1500000
2000000
2500000
Trials MEDLINE BioMedical
Medical
Articles
per
Year
5,000?
per day
1,500
per day
55 per
day

6. Years after graduation
Relative
% of
remaining
knowledge
2 4 6 8 10 12
$
100%
THE SLIPPERY SLOPE

7. The Prognosis of Ignorance is
Poor
Worse with “duration in practice”

8. 1. Formulate clinical problems in answerable
questions
2. Search the best evidence: use internet or other
on-
line database for current evidence
3. Critically appraise the evidence for
 Validity (was the study valid?)
 Importance (were the results clinically
important?)
 Applicability (could we apply to our
patient?)
Steps in EBM practice
VIA

9. Diagnosis
(Determination of disease or problem)
Etiology
(What causes the disease)
Prognosis
(Prediction of the outcome of the disease)
Treatment
(Intervention necessary to help the patient)
Main area

10. >25,000 journals worldwide
>2 million published articles per year
Many published articles have methodological
(including statistical) flaws – even in most
respected journals
Not all results can be applied due to many
reasons, a.o. dissimilarities of study subjects
with our patients
Limited time of physicians: focus on articles
relevant to your clinical practice
Rationale

11. All articles should be
critically appraised for: VIA
Validity : was the study valid?
Importance : were the results
clinically important?
Applicability : were the valid and
important results can
be applied to my
patients?

12. For educational purposes,
students are encouraged
to consider all parts of the report,
from title to list of references,
before specifically appraise the content.

13. Introduction - Why did I start?
Methods - What did I do?
Results - What did I find?
Discussion - What do they mean?
Anatomy & physiology of research reports
IMRAD

14. What is the main purpose
of critical appraisals?
To determine that the study was valid and the
results were important and can be applied to our
patients
How can we determine validity?
 Methods! (and Results)
How can we know that the results were
important?
 Your clinical judgment!
How can we know that the results can be
applied to our patients?
 Make sure that your patients are similar to
those in the study (You should assume that

15. General approach to asses
validity of the study: RAMMbo
R = Recruitment
A = Allocation
M = Maintenance
M = Measurements
blinded?
objective?

16. Relation of PICO to RAMMbo
Patient/Problem
Intervention
Comparison
Outcome
Recruitment
Allocation
Maintenance
Measurements
 blinded?
 objective?

17. RCT (Pragmatic trials): Validity
Were the study participants randomized?
Was the randomization technique described?
Was the randomization table concealed?
Were the characteristics of the subjects
similar at the start of the intervention?
Were all participants given equal treatment
apart from the intervention?
Were all relevant outcomes considered?
Were the results analyzed correctly?

18. RCT (Pragmatic trials):
Importance
Calculate: EER, CER, RRR, ARR, NNT
a b
c d
E
C
Success Failure
EER = a/(a+b)
CER = c/(c+d)
RRR = (CER-EER)/CER
ARR = CER-EER
NNT = 1/ARR

19. RCT: Applicability
Were the participations similar to your
patients?
May be intuitively concluded or use f
(factor indicating how much severe your
patient compared to the study participation
in terms of prognostic factor)

20. DIAGNOSTIC TEST: Validity
Was independent and blind comparison
to gold standard applied?
Was the diagnostic test include
spectrum of disease similar to your real
practice?
Was the gold standard applied
regardless of the diagnostic result?

21. Diagnostic Test: Importance
Calculate: Sensitivity, specificity,
predictive values, likelihood ratios
a b
c d
+
-
+ -
Se = a/(a+c)
Sp = d/(b+d)
PPV = a/(a+b)
NPV = d/(c+d)
LR+ = se/(1-sp)
LR - = (1-se) /sp
Posttest odds = Pretest odds x LR+
Test

22. Diagnostic Test: Applicability
Were the participations similar to your
patients?
Is the diagnostic test applicable,
acceptable, and affordable in your setting?
Will the result of the test help your
patient?

23. Prognostic Studies: Validity
Was the inception cohort assembled in usual
point of course of the disease?
Was the follow-up sufficient & complete?
Were outcome criteria applied in blind
fashion?
Was there any validation in other group of
patients?
Was subgroup analysis performed after
adjustment for prognostic factors?

24. Prognostic studies:
Importance
Calculate: Relative risk (RR) and 95% CI
a b
c d
+
-
+ -
Exposure
RR = a/(a+b) : c/(c+d)
RR > 1: risk factor
RR < 1 : protective factor
RR = 1 : not a risk factor

25. Prognostic studies:
Applicability
Were the patients similar to yours?
Will the evidence make a clinically
important impact for the care of your
patient?

26. Meta-analysis: validity
Is it the SR / Meta-analysis you are
interested?
Does it include Methods describing how to
find articles?
Does it include description of assessment
of individual validity?
Were the results consistent from study to
study?

27. Meta-analysis: Importance
Are the valid results clinically important?
Inspect individual studies: OR /
RR/mean difference - point estimates
and confidence intervals
Inspect combined OR / RR / mean
difference
Comments if any

28. Meta-analysis: Applicability
Is your patient similar to the patients in the
study?
How big is the benefit?
NNT:
calculate from the OR
select the OR of best study

29. Three common myths/mistakes
in reviewing the literature
It is in a “big name” journal it must be good
Many dangerous DOE‟s have been printed in NEJM,
JAMA, Annals of ____ etc.
The author is a big name, or well published, or
from an impressive institution
My article is newer than your article
Newer is often NOT better

30. Best available evidence
Best available evidence follows from critical
appraisal
Determine strength of evidence („level
or evidence‟) for patient’s case
Strength of evidence partially
determines the strength of the
recommendation („grade or
recommendation’) for patient in
general

31. Levels of Evidence
Classification study and level of evidence
Often focus on therapeutic studies
Often (only) based on study design (the best study
design varies per study subject D/P/T!)
No/little stress on individual methodological aspects
No/little stress on (uncertainty in) size and relevance
effect
Many different classification systems (what is A1, level
2, IIb etc?)
Variation in study design regarding best available
evidence?
Own system/assessment

32. Strength of evidence of study
Strength of evidence of study is
determined by
Quality (relevance and validity)
Effect size (clinically relevant difference?)
Uncertainty in effect
Consistency results in studies from best
available evidence

34. From strength of evidence to
recommendation
Recommendation for individual patient
(EBCR) and recommendation for group(s) of
patients (Guidelines)
Compact summary best available evidence
Saves time
Fewer differences in patient handling
between doctors
Improvement quality of care

35. From strength of evidence to
recommendation
A recommendation has to
be clear (no number/cipher)
be unambiguous
be a reflection of all the evidence of
sufficient/good quality
have a „strength‟ („Grade of
recommendation‟)

36. Example GRADE approach
Grading of Recommendations Assessment,
Development and Evaluation (short GRADE)
Systematic approach for developing guidelines
Approach
Quality of evidence (=strength of evidence)
Strength of recommendations (=Recommendation)
2 grades – weak or strong (for or
against)
Quality of evidence (=strength of
evidence) only one factor
Guyatt et al BMJ 2008

37. GRADE Quality of Evidence
“Degree of trust in the effect estimate to support the
decision”
Best available evidence usually only high and
moderate quality
Quality of
evidence
Suggested implications
High further research is unlikely to change the confidence in an
estimated effect; we are confident that we can expect very similar
effect in a population for which the recommendation is intended
Moderate further research is likely to have an important impact on the
confidence in an estimated effect and may change that estimate
Low further research is very likely to have an important impact on the
confidence in an estimated effect and is likely to change that
estimate
Very low any estimate of an effect is very uncertain

38. GRADE: Strength of
recommendation
“The „strength of a recommendation‟ gives
the degree to which we can be sure that the
positive effects will outweigh the negative
effects of the proposed treatment strategy for
the patient (group) for which the
recommendation is meant”

39. Determinants of the strength of
the recommendation
Strength of evidence or „Quality of evidence‟
Balance between positive and negative
effects
Standards and values
Costs

40. Questions?