Updated - What makes systematic review systematic - Anna Sidorchuk

What makes a systematic review
systematic? Epidemiologic perspectives
ANNA SIDORCHUK, MD, PHD, RESEARCH COORDINATOR
KAROLINSKA INSTITUTET
DEPARTMENT OF PUBLIC HEALTH SCIENCES
ANNA.SIDORCHUK@KI.SE
THE EPIDEMIOLOGY PHD PROGRAM SEMINAR
APRIL 29, 2013

Outline of the speech
 Non-systematic vs. systematic review
 Why do systematic reviews? What is the added value?
 PRISMA/MOOSE/AMSTAR guidelines and checklists
 Randomized vs. observational data
 Process (criteria, optimize search, determine data to be
abstracted, quality score)
Anna Sidorchuk 2

30 april 2013Anna Sidorchuk
Primary data versus secondary data
 Traditionally, researchers collect and analyze their own data
(referred to as primary data)
 Secondary data analysis is based on data collected by someone
else (or reanalysis of your own published data)
 Systematic review is one of the ways to analyze secondary data
Systematic review - systematic, qualitative review of published
research in a particular field
Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford

What The Lancet thinks about
systematic reviews
In 2005, the editors wrote:
“. . . we will require authors of clinical trials submitted to The
Lancet to include a clear summary of previous research findings,
and to explain how their trial’s findings affect this summary. The
relation between existing and new evidence should be illustrated
by direct reference to an existing systematic review and meta-
analysis…”

Stages of waste in the production and reporting
of research evidence relevant to clinicians and
patients by Chalmers and Glasziou (Lancet, vol. 374, 2009)
Anna Sidorchuk 5

Why do we do systematic reviews?
 To help busy clinicians to summarize current knowledge in
relation to the area of interest
 To provide high-quality research evidence to guide clinical
practice
 To support clinical decision-making
 To support research proposals

How systematic review can help busy
clinicians?
By systematically:
 identifying,
 appraising,
 synthesizing,
 and, if appropriate, statistically combining studies on a
specific topic

In classifications of levels of evidence,
systematic reviews are included in the highest level of
evidence

Hierarchy of evidence: a framework for ranking
evidence evaluating healthcare interventions
Journal of Clinical Nursing
Volume 12, Issue 1, pages 77-84, 20 DEC 2002 DOI: 10.1046/j.1365-2702.2003.00662.x
http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2702.2003.00662.x/full#f1

Meta-analysis
 A statistical method to combine findings across studies
 Should be considered within the framework of systematic reviews
review needs to use a systematic approach to minimize bias, address
the issues of the completeness of the evidence, quality of studies
and combinability of studies
Meta-analysis - the statistical pooling of the results of
studies that are part of a systematic review

When systematic review is required?
 Before undertaking a systematic review it is necessary to
check whether there are already existing or ongoing reviews,
and whether a new review is justified
 Search:
the Database of Abstracts of Reviews of Effects (DARE)
 the Cochrane Database of Systematic Reviews (CDSR)
Adapted from: Systematic Reviews. CRD’s guidance for undertaking reviews in health care (2009). Centre for Reviews and
Dissemination, University of York. ISBN 978-1-900640-47-3.

When is meta-analysis appropriate?
 There exists a critical mass of comparable studies designed to
address a common research question
 Data are presented in a form that allows the meta-analyst to
compute an effect size for each study
 Characteristics of each study are described in sufficient detail
to allow meta-analysts to compare characteristics of different
studies and to judge the quality of each study

Systematic reviews vs. non-systematic
 Compared to traditional literature reviews:
there is a definite methodology employed in the research analysis
(more like that used in primary research); and
the results of the included studies are quantified to a standard metric
thus allowing for statistical techniques for further analysis.
 Therefore process of reviewing research literature is more
objective, transparent, and replicable; less biased and
idiosyncratic to the whims of a particular researcher

Systematic reviews vs. non-systematic
Feature Systematic review Narrative review
Question Often a focused one question Often broad in scope
Sources and search Comprehensive sources and
explicit search strategy
Not usually specified,
potentially biased
Selection Criterion-based selection,
uniformly applied
Not usually specified,
potentially biased
Appraisal Rigorous critical appraisal Variable
Synthesis
Qualitative summary that
includes statistical synthesis
(meta-analysis)
Often a qualitative summary
Inferences Usually evidence-based Sometimes evidence-based
Anna Sidorchuk 14
Adapted from: Y Yuan and R H Hunt, Am J Gastroenterol 2009; 104:1086–1092; doi:10.1038/ajg.2009.118

18 September, 2009Anna Sidorchuk 15
Increase in the number of published
meta-analyses (PubMed references)
Publication year

• The essence of good science is replicable and generalisable results
Do we get the same answer to research questions when we run the study again?
• The primary aims of meta-analysis is to test the generalisability of results
across a set of studies designed to answer the same research question
Are the results consistent? If not, what are the differences in the studies that explain
the lack of consistency?
Why are systematic review and meta-
analysis important?

When there is systematic variation in outcomes from different studies,
meta-analysis tries to explain these differences in terms of study
characteristics:
measures used
study design
participant characteristics
controls for potential bias
etc.
If the results of individual studies are
inconclusive or controversial…

“… doing a meta-analysis is easy,
doing one well is hard.”
Ingram Olkin, Stanford University

Benefits of systematic reviews, incl. meta-
analysis
 Increased power: by combining information from many
individual studies, the meta-analyst is able to detect systematic
trends not obvious in the individual studies
 Conclusions based on the set of studies are likely to be more
accurate than any one study

Benefits of systematic reviews, incl. meta-
analysis
 Improved precision: based on information from many studies,
the meta-analyst can provide a more precise estimate of the
population effect size (and a confidence interval)
 Provides potential corrections for potential biases,
measurement error and other possible artefacts
 Identifies directions for further primary studies to address
unresolved issues

Benefits of systematic reviews, incl.
meta-analysis
 Typically there is study-to-study variation in results. When
this is the case, the meta-analyst can explore what
characteristics of the studies explain these differences (e.g.,
study design) in ways not easy to do in individual studies
 Easy to interpret summary statistics (useful if communicating
findings to a non-academic audience)

Limitations of systematic reviews, incl.
meta-analysis
 Comparing apples and oranges
 Quality of the studies included in the meta-analysis
 What to do when studies don’t report sufficient information
(e.g., “non-significant” findings)?
 Including multiple outcomes in the analysis (e.g., different
achievement scores)
 Publication bias
22

Limitations of systematic reviews, incl.
meta-analysis
 Meta-analysis conclusions may still differ if different studies
are sampled or excluded for different reasons
 Need to be explicit
 Quality standards
 Inappropriate handling of data can lead to wrong conclusions
 Sample size consideration
23

Publication bias
 Studies that are published are more likely to report
statistically significant findings. This is a source of potential
bias

Publication bias
 The debate about using only published studies:
peer-reviewed studies are presumably of a higher quality
VERSUS
significant findings are more likely to be published than non-
significant findings
 There is no agreed upon solution. However, one should retrieve all studies
that meet the eligibility criteria, and be explicit with how they dealt with
publication bias. Some methods for dealing with publication bias have
been developed (e.g., Fail-safe N, Trim and Fill method)

RCT vs. observational studies
 Both are fine as long as all the methodological issues are
addressed
 If your focus is effect of therapy or prevention – use RCTs
 If you are interested in prognosis, etiology, adverse
effects, association between risk factors and outcomes –
use observational studies
 Fine to use both types of studies in the same review
Anna Sidorchuk 26

Steps in a meta-analysis
Establish
research
question
Define
relevant
studies
Develop
code
materials
Locate and
collate
studies
Pilot coding;
coding
Data entry
and effect
size
calculation
Main
analyses
Supplementar
y analyses
30 april 2013Anna Sidorchuk Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford

Practical tips
 Define all key elements (PICO model) a priory
 Formulate the research question as clear as possible
 Make an agreement with all your co-authors on the key elements
 If needed ask the experts in the area of outcome or/and exposure for
advice
 Keep everything in the protocol
 When writing an article include the elements in the aim

Practical tips
 Define all inclusion/exclusion criteria and components of search a priory
 Make criteria and the components as clear as possible, but feel free to
make some changes during the actual literature search (usually due to
availability of data, e.g. if only retrospective studies are available on the
issue, or no info on ethnicity, etc)
 Make an agreement with all co-authors on abovementioned
 Again, keep everything in the review protocol

Practical tips
 Define coding and form for data extraction a priory
 Make an agreement with all co-authors on abovementioned
 If use Excel for data extraction and, therefore, for coding, keep the legend
for each coding element within the same file on the separate sheet
 As an alternative – use Access
 Keep everything in the review protocol!

Practical tips
 Define as detailed as possible by reading other meta-analysis and
systematic reviews on the similar outcome(s) and exposure(s):
 All key words
 MESH terms
 Define the electronic sources to be used for your search
 Ask librarians for advice on search strategy and technic
 Define if grey literature will be included
 Keep everything in the protocol!!!

Critical issue
Quality, quality, quality!

Quality Standards in Systematic
reviews and Meta-Analysis
 Several organizations work on this
Cochrane collaboration
http://www.cochrane.org
Campbell collaboration
http://www.campbellcollaboration.org
 Consort: Consolidated Standards of Reporting Trials, includes
Quorum: Quality of Reporting of Meta-analyses group
http://www.consort-statement.org/mod_product/uploads/QUOROM
Statement 1999.pdf
 PRISMA (Preferred Reporting Items for Systematic reviews and Meta-
Analyses), AMSTAR (A measurement tool to assess systematic reviews)

PRISMA: Checklist of items to include when reporting a systematic review or meta-
analysis.
Section/Topic # Checklist Item
Reported
on Page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both.
ABSTRACT
Structured
summary
2
Provide a structured summary including, as applicable:
background; objectives; data sources; study eligibility criteria,
participants, and interventions; study appraisal and synthesis
methods; results; limitations; conclusions and implications of key
findings; systematic review registration number.
INTRODUCTION
Rationale 3
Describe the rationale for the review in the context of what is
already known.
Objectives 4
Provide an explicit statement of questions being addressed with
reference to participants, interventions, comparisons, outcomes,
and study design (PICOS).

analysis.
Reported
on Page #
METHODS
Protocol and
registration
5
Indicate if a review protocol exists, if and where it can be
accessed (e.g., Web address), and, if available, provide
registration information including registration number.
Eligibility
criteria
6
Specify study characteristics (e.g., PICOS, length of follow-
up) and report characteristics (e.g., years considered,
language, publication status) used as criteria for eligibility,
giving rationale.
Information
sources
7
Describe all information sources (e.g., databases with dates of
coverage, contact with study authors to identify additional
studies) in the search and date last searched.
Search 8
Present full electronic search strategy for at least one
database, including any limits used, such that it could be
repeated.
Study selection 9
State the process for selecting studies (i.e., screening,
eligibility, included in systematic review, and, if applicable,
included in the meta-analysis).
Data collection
process
10
Describe method of data extraction from reports (e.g., piloted
forms, independently, in duplicate) and any processes for
obtaining and confirming data from investigators.
Data items 11
List and define all variables for which data were sought (e.g.,
PICOS, funding sources) and any assumptions and
simplifications made.

analysis.
Reported
on Page #
METHODS
Risk of bias in
individual studies
12
Describe methods used for assessing risk of bias of
individual studies (including specification of whether this
was done at the study or outcome level), and how this
information is to be used in any data synthesis.
Summary
measures
13
State the principal summary measures (e.g., risk ratio,
difference in means).
Synthesis of
results
14
Describe the methods of handling data and combining
results of studies, if done, including measures of consistency
(e.g., I2
) for each meta-analysis.
Risk of bias
across studies
15
Specify any assessment of risk of bias that may affect the
cumulative evidence (e.g., publication bias, selective
reporting within studies).
Additional
analyses
16
Describe methods of additional analyses (e.g., sensitivity or
subgroup analyses, meta-regression), if done, indicating
which were pre-specified.

analysis.
Reported
on Page #
RESULTS
Study selection 17
Give numbers of studies screened, assessed for eligibility,
and included in the review, with reasons for exclusions at
each stage, ideally with a flow diagram.
Study
characteristics
18
For each study, present characteristics for which data were
extracted (e.g., study size, PICOS, follow-up period) and
provide the citations.
Risk of bias
within studies
19
Present data on risk of bias of each study and, if available,
any outcome-level assessment (see Item 12).
Results of
individual studies
20
For all outcomes considered (benefits or harms), present, for
each study: (a) simple summary data for each intervention
group and (b) effect estimates and confidence intervals,
ideally with a forest plot.
Synthesis of
results
21
Present results of each meta-analysis done, including
confidence intervals and measures of consistency.
Risk of bias
across studies
22
Present results of any assessment of risk of bias across
studies (see Item 15).
Additional
analysis
23
Give results of additional analyses, if done (e.g., sensitivity
or subgroup analyses, meta-regression [see Item 16]).

analysis.
Reported
on Page #
DISCUSSION
Summary of
evidence
24
Summarize the main findings including the strength of
evidence for each main outcome; consider their relevance to
key groups (e.g., health care providers, users, and policy
makers).
Limitations 25
Discuss limitations at study and outcome level (e.g., risk of
bias), and at review level (e.g., incomplete retrieval of
identified research, reporting bias).
Conclusions 26
Provide a general interpretation of the results in the context of
other evidence, and implications for future research.
FUNDING
Funding 27
Describe sources of funding for the systematic review and
other support (e.g., supply of data); role of funders for the
systematic review.

Quality Assessment of Primary Studies
 A common practice is to assign a quality score to primary
studies
Preferably based on an explicit checklist usually
 The judgment to include or exclude a study must be stated
and justified explicitly
 This judgment is mostly based on quantitative criteria, but
involves some subjectivity

NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE
CASE CONTROL STUDIES
 Note: A study can be awarded a maximum of one star for each numbered item within the Selection and Exposure
categories. A maximum of two stars can be given for Comparability.
Selection
1) Is the case definition adequate?
a) yes, with independent validation ¯
b) yes, eg record linkage or based on self reports
c) no description
2) Representativeness of the cases
a) consecutive or obviously representative series of cases ¯
b) potential for selection biases or not stated
3) Selection of Controls
a) community controls ¯
b) hospital controls
c) no description
4) Definition of Controls
a) no history of disease (endpoint) 
b) no description of source

NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE
CASE CONTROL STUDIES
Comparability
1) Comparability of cases and controls on the basis of the design or analysis
a) study controls for _______________ (Select the most important factor.) ¯
b) study controls for any additional factor ¯ (This criteria could be modified to indicate specific control for a
second important factor.)
Exposure
1) Ascertainment of exposure
a) secure record (eg surgical records) ¯
b) structured interview where blind to case/control status ¯
c) interview not blinded to case/control status
d) written self report or medical record only
e) no description
2) Same method of ascertainment for cases and controls
a) yes ¯
b) no
3) Non-Response rate
a) same rate for both groups ¯
b) non respondents described
c) rate different and no designation

“Ugly face” of systematic reviews and
meta-analyses
 Can be misleading
 Can be misused

Updated - What makes systematic review systematic - Anna Sidorchuk

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Updated - What makes systematic review systematic - Anna Sidorchuk

Similar to Updated - What makes systematic review systematic - Anna Sidorchuk (20)

Recently uploaded

Recently uploaded (20)

Updated - What makes systematic review systematic - Anna Sidorchuk