systematic review

1. Evidence Based Medicine
Meta-analysis and systematic reviews
Ross Lawrenson

2. A systematic review
• This is a systematic collection of the
relevant primary papers in human
populations that deal with a focused
question and includes a summary of
the evidence from the primary sources.

3. A systematic review
• This is a systematic collection of the relevant
primary papers in human populations that deal
with a focused question and includes a summary
of the evidence from the primary sources.
• This is in contrast to an overview which often
tackles a whole topic rather than a focused
question, may include opinion based articles and
whose summary may be influenced by clinical
experience or evidence from other sources such
as animal experiments.

4. Meta-analysis
• A particular type of systematic review that
uses quantitative methods to combine the
results from a number of studies

5. Meta-analysis
• Meta analysis is a statistical anlysis which
combines the results of several independent
studies considered by the analyst to be
combinable.
• Huque 1988

6. Meta-analysis
• A particular type of systematic review that uses
quantitative methods to combine the results from a
number of studies
• Since meta-analysis is a retrospective look at data, it
is important to make the process rigorous and well
defined to prevent opportunities for bias to distort
the results. Only in this way can it achieve the status
of a scientific discipline. This necessitates blinding
the selection of papers, extraction of data and
quality assessment in duplicate following an
established protocol at the start of the study.

7. Critically appraising an overview
paper

8. Critical Appraisal of an overview
- methodology
• When critically appraising a paper ask
yourself three questions:
– Are the results valid?
– What are the results?
– Will the results help me in caring for my
patients?
• Go to the overview worksheet for the
complete list of questions

9. I. Are the results valid?
• In other words was this thorough and
rigorously conducted overview?
• Go through the worksheet questions 1- 6 to
help you decide whether you are likely to
believe the results of the paper you are
considering.

10. Are the results valid?
• Did the overview address a clearly focused
question?

11. Are the results valid?
• Did the overview address a clearly focused
question?
• Is there a defined population, a clearly
identified treatment or exposure and one or
more clearly defined outcomes?

12. Are the results valid?
• Were the criteria used to select articles for
inclusion appropriate?

13. Are the results valid?
• Were the criteria used to select articles for
inclusion appropriate?
• The article should clearly set out the criteria for
inclusion of studies. This usually includes the type
of study (for therapy questions ideally an RCT),
the populations included in the studies, treatments
or exposures and relevant outcomes.
• The criteria should not be so restrictive that
important studies are likely to be missed. Ideally
the authors should list all trials reviewed with a
reject log and reasons for exclusion

14. Are the results valid?
• Is it likely that important, relevant studies
were missed?

15. Are the results valid?
• Is it likely that important, relevant studies
were missed?
• What was the search strategy used in
identifying papers. Is it clearly stated. What
databases were searched? Was the grey
literature searched? What about secondary
sources? Was it a wide (sensitive) search?

16. Is it likely that important,
relevant studies were missed?
• Sensitivity of search
– The search strategy
– Sources used
– Foreign language papers
– Grey and unpublished

17. The search strategy
• This should be very sensitive.
• Ideally the paper will include in the methods
section the words that were used in the
interrogation of the Medical Databases and
which terms were combined and which
intersected. (e.g. When looking for randomised
controlled trials the terms “random allocation,
randomised, double blind and controlled”
should be used.)
• The time period over which papers could be
included should also be included.

18. Sources used
• All the sources used should be listed. These
could include Medline, Cochrane, Embase,
BIDS, Cinahl, etc.
• This should also include secondary
references I.e. the references cited by the
papers that were retrieved from Medline
etc.

19. Foreign language papers
• Many well conducted studies are published
in languages other than English.
Researchers undertaking systematic reviews
are often tempted to exclude these papers
because of the cost of having them
translated (as well as the delay). However
this means that some perfectly valid studies
that deal with the question are not included.
• This can introduce bias

20. Grey Literature
• A comprehensive overview should also look for
grey and unpublished literature. Studies with
negative findings may be difficult to get
published. If only studies with positive results
are published then a meta-analysis of the
published papers will give a positive result.
Suggest contacting researchers known to be
active in the field of interest. Also drug
companies often have unpublished studies
which they are now releasing to Cochrane and
other interested parties.

21. Control and Measurement of
Bias.
• Data extraction bias
• Selection bias
• Source of support

22. Data extraction bias
• To ensure that the correct papers have been
included, and that bias in the selection of papers is
minimised, duplicate extraction is recommended.
Ideally this should occur blinded. I.e. The
reviewers should be blinded to the authors, the
location of study, the journal and the finding. This
can best be achieved by copying only the methods
section of the papers being selected.

23. Selection bias
• One way of looking for selection bias is to
construct a funnel plot where the largest
studies (with the most precision) are close
to the top and smaller studies are nearer the
X axis. If there is no bias then the studies
selected should be symmetrically
distributed around the results from the
larger studies which should be closer to the
true result and less influenced by random
error.

24. Include diagrams of four funnel plots - two
with bias and two that are symmetrical.

25. Sources of support
• The funding of the original studies should
be mentioned to allow the reader to make
conclusions as to any conflict of interest
although not to disqualify a study from
inclusion.