How to critically appraise a meta-analysis article

1. Critical Appraisal of the Literature:
Overview or Meta-Analysis
Thomas E. Grissom, Lt Col, USAF, MC, FCCM
Wilford Hall Medical Center

4. The Question
In your preoperative clinic, a 63 y.o patient scheduled
for a left hemicolectomy due to cancer states that
his daughter, an anesthesiologist, told him to
request an epidural for postoperative pain
management and that it should be used during the
procedure as part of the anesthetic plan. Your
department “money guy” has been warning
everyone to put epidurals in preoperatively, but to
wait until the case is done to use it. This allows for
a better billing profile. Does this article help you
make a better decision or recommendation?

5. British Medical Journal 321:1493-7, 2001

6. Are the results of
the overview valid?

7. Did the overview question address a focused
clinical question?
• What was the focus of this particular
review?
– Outcome (postoperative morbidity and
mortality) for patients receiving intraoperative
neuraxial blockade (epidural or spinal)

8. Were the criteria used to select articles for
inclusion appropriate?
• “Trials were ineligible if they were not
randomised or were quasi-randomized or if
the data were not available before 1 January
1997.”
• Only criteria was whether patients were
randomized to receive intraoperative
neuraxial blockade

10. Is it unlikely that important, relevant
studies were missed?
• Publication bias
• Unpublished studies should be reviewed
with same criteria as published reports
– No unpublished studies were found
– No language limitation

11. Was the validity of the included studies
appraised?
• Same as those for articles on harm and therapy
– Were there clearly identified comparison groups that
were similar with respect to important determinants of
outcome, other than the one of interest?
– Were outcomes and exposures measured in the same
way in the groups being compared?
– Was the assignment of patients to treatments
randomized?
– Were all of the patients who entered the trial properly
accounted for and attributed at its conclusion?

12. Was the validity of the included studies
appraised?
• “We did not use quality scores, and the
definitions of events were those used in the
original trials.”

13. Were the assessments of studies
reproducible?
• They did use two reviewers, however, there
is no comment about agreement
• “A third reviewer compared the two sets of
data collection sheets and any differences
were resolved by discussion.”

14. Were the results similar from study to
study?
• Tests of homogeneity
– Measures the extent to which differences among
the results of individual studies are greater than
you would expect if all studies were measuring
the same underlying effect and the observed
differences were due only to chance.

18. What are the
results?

19. What are the overall results of the
review?
• Overall mortality-0.70 (95% CI 0.54-0.90)
• No specific reduction in cause of death was
identified as significant
• There was 1 per 100 fewer deaths at the 30-
day point in the neuraxial blockade group

20. Odds Ratio
Adverse event No adverse event
Exposed a b
Not exposed c d
Odds ratio=(a/b)/(c/d)

21. Odds Ratio
Death No death
Neuraxial blockade 103 4768
No blockade 144 4544
Odds ratio=0.022/0.032=0.69

24. How precise are the results?
• The odds ratio is only a point estimate
• Confidence intervals (CI) provide an
estimate of imprecision
– 95% confidence interval represents a 95%
likelihood of containing the true treatment
effect

26. Will the results help me
in caring for my
patients?

30. Can the results be applied to my patient
care?
• What about subgroup analysis? Your patient
is rarely identical to the “average” patient in
the overview.
• What do you think?

31. Were all clinically important outcomes
considered?
• What do you think?

32. Are the benefits worth the harms and
costs?
• To get a sense of the treatment effect, calculate the
number needed to treat (NNT)
– NNT=(1/ARR)
– Incorporates the baseline risk of bad outcomes without
therapy and the RR with therapy
• If the outcome is frequent, low NNT even with
low RRR
• If the outcome is rare, high NNT even with good
efficacy

33. Are the benefits worth the harms and
costs?
Relative risk reduction=32%
Baseline risk=3%

34. Sackett, Clinical Epidemiology, p. 207

35. Are the benefits worth the harms and
costs?
• What do you think?

39. New England Journal of Medicine 344:626-31, 2001