how to do systematic review and Meta analysis for health professionals

1. Reporting systematic reviews and
meta-analyses: PRISMA
Iveta Simera
The EQUATOR Network
Centre for Statistics in Medicine, Oxford, UK
April 2012

2. 2
Large number of studies – need to
synthesise & summarise
Source: Banzi et al. J of Med Internet Res;
2010,12 (3) adapted from Haynes RB. Evid
Based Med 2006;11(6):162-164.
Source: Evidence-based Nursing
http://ebp.lib.uic.edu/nursing/node/12

3. 3
Different types of reviews
• Narrative (overview)
• Systematic review
– Meta-analysis

4. 4
Narrative reviews (NR)
• Provide an overview of a particular topic
• Often cover a wide range of issues within a given topic
• Can be useful for understanding new concepts
• But there are problems associated with NR:
– they are rarely comprehensive
– they do not reveal many details about their methodology
– they are highly susceptible to reviewers’ bias
– they seldom take into account differences in the quality of
studies
– they can often come to the wrong conclusion – careful
interpretation needed

5. 5
Example of NR

6. 6
Systematic reviews (SR)
• SR is a scientific investigation that focuses on
a specific question and uses explicit,
prespecified scientific methods to identify,
select, assess, and summarise the findings of
similar but separate studies.
• It may include a quantitative synthesis (meta-
analysis), depending on the available data
[Eden et al. Finding what works in health care:
Standards for systematic reviews, Institute of Medicine,
2011]

7. 7
Systematic reviews (2)
• The importance of SR is increasingly
appreciated
– Clinical practice guideline development
– Clinical and policy decisions
BUT
• The quality of published SR is variable and
often inadequate
– In many cases we are unable to judge the quality of
SR because the methodology is poorly reported or the
SR is poorly conducted

8. 8
Key characteristics of SR
• Focused well defined research question
• Clearly stated title and objectives
• Comprehensive strategy for identification of all
relevant studies (published & unpublished)
• Explicit (and justified) predefined inclusion &
exclusion criteria
• Critical appraisal of studies
• Clear analysis of the results of eligible studies
– Quantitative (meta-analysis)
– Qualitative
• Structured report

9. 9
Cochrane SR
• Development of Cochrane SR is coordinated by
the Cochrane Collaboration
– Established in 1993
– International network of 28,000 from 100 countries
– About 4,600 Cochrane reviews published
– They are internationally recognised as a benchmark
for high quality information about the effectiveness of
healthcare
http://www.cochrane.org

10. 10
Cochrane Library (CLIB)
• All Cochrane reviews published in CLIB
• Published by Wiley-Blackwell (indexed by PubMed,
impact factor 6.1)
• Free access in the UK and many other countries
http://www.thecochranelibrary.com/

11. 11
Methodology of Cochrane reviews
• Methodology robustly
developed (continuous
improvements)
• Handbook – free online
access:
http://www.cochrane.org/
training/cochrane-
handbook
• Good to follow even if
doing “non-Cochrane”
SR
• UK Cochrane Centre -
training

12. 12
Process of conducting Cochrane SR
• PROTOCOL – important
– Minimise potential bias in the
review conduct:
Reviews are retrospective,
need to establish the methods
in advance
– Planning
– Review team
– Cochrane protocols are peer
reviewed and published

13. 13
Cochrane review conduct – key points
• Protocol
• Objectives
– Focused well defined research question
– Primary outcome (one)
– Minimum number of secondary outcomes
– Include adverse events (harms) if relevant
• Literature search
– Comprehensive (electronic databases, grey literature,
reference lists, personal communication, ..)
– Useful to involve an information specialist in developing
search strategies (consider ss peer review)
– Keep detailed record of search methods and search results!

14. 14
Cochrane review conduct – key points (2)
• Data collection and analysis
– Selection of studies using predefined selection criteria
– Independently done by more than one reviewer
– Important to determine how to solve disagreements
between reviewers
• Data extraction
– Data extraction form (pilot – items, format, ..)
– Independently done by more than one reviewer

15. 15
Cochrane review conduct – key points (3)
• Assessment of risk of bias
– Problems with the design and execution of individual studies of
healthcare interventions raise questions about the validity of their
findings
– In clinical trials, biases can be broadly categorized as selection
bias, performance bias, detection bias, attrition bias, reporting
bias and other biases that do not fit into these categories
– Cochrane Collaboration developed the ‘Risk of bias tool’
7 specific domains:
• sequence generation (selection bias)
• allocation concealment (selection bias)
• blinding of participants and personnel (performance bias)
• blinding of outcome assessment (detection bias)
• incomplete outcome data (attrition bias)
• selective outcome reporting (reporting bias)
• other potential sources of bias

16. 16
Cochrane review conduct – key points (4)
• Data synthesis
– Qualitative: descriptive summary
– Quantitative - meta-analysis: pooling data from a
number of studies when there are
• Minimal differences between studies
• Outcome measured in the same way
• Data are available
Study weight
Different statistical
methods for pooling
Subgroup analysis
Sensitivity analysis

17. 17
Interpretation of results
• Clear statement of findings
• Authors conclusions should reflect findings
• Clear presentation is important
• Summary of findings tables
– Key information in a quick and accessible format
– Relating the quality of evidence to the outcomes

18. 18
Publishing SR
• Differences between publishing SR in the Cochrane
Library and in a journal:
– Cochrane has some specific rules (e.g. titles structure: a
title cannot start with ‘A’ or ‘The’; should not not include ‘a
systematic review of’)
• Publishing in a journal: PRISMA Statement
– Preferred Reporting Items for Systematic Reviews and
Meta-Analyses (2009)
– 27-item checklist, flow diagram
– PRISMA authors are also heavily involved in the Cochrane
work, high compatibility of both guides
http://www.prisma-statement.org/

19. 19
Poor reporting of systematic reviews
• Good reporting of primary studies is crucial
for SR development
BUT
• Reviews are not immune to the problems of
poor reporting
– Moher et al. assessed epidemiological and reporting
characteristics and bias-related aspects of 300
systematic reviews (of which 125 were Cochrane
reviews). The overall quality of reporting of key
aspects of methodology was very inconsistent with
particularly discouraging findings for non-Cochrane
reviews.
[Moher; PLoS Medicine 2007]

20. Example of bad reporting
• Nowhere in the paper any mention of the
review methodology!
20

21. Example of good reporting
21

22. 22
PRISMA 2009 Checklist

23. 23
PRISMA 2009 Checklist (2)

24. 24
PRISMA 2009 Checklist (3)
Methods - continued

25. 25
PRISMA 2009 Checklist (4)
Present the main results of the review. If meta-analyses are done, include for each,
confidence intervals and measures of consistency
Mistake in the published
PRISMA papers: Item 21
should read

26. 26
PRISMA 2009 Checklist (5)

27. 27
PRISMA 2009 Flow diagram

28. 28
PRISMA explanation & elaboration paper
– Explanation and rationale for reporting of
suggested information (items)
– Examples of good reporting
– Relevant data about how this information is
reported presently
Long but recommend to read to avoid
basic mistakes in SR reports!
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche P, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D, the
PRISMA Group. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health
care interventions: explanation and elaboration.
• PLoS Med 2009 6(7): e1000100
• Annals of Internal Medicine 2009;151:w65-w94
• BMJ 2009; 339:b2700.
• Journal of Clinical Epidemiology 2009; PMID: 19631507

29. 29
www.prisma-statement.org
www.equator-network.org