Updated - What makes systematic review systematic - Anna Sidorchuk
What makes a systematic reviewsystematic? Epidemiologic perspectivesANNA SIDORCHUK, MD, PHD, RESEARCH COORDINATORKAROLINSKA INSTITUTETDEPARTMENT OF PUBLIC HEALTH SCIENCESANNA.SIDORCHUK@KI.SETHE EPIDEMIOLOGY PHD PROGRAM SEMINARAPRIL 29, 2013
Outline of the speech Non-systematic vs. systematic review Why do systematic reviews? What is the added value? PRISMA/MOOSE/AMSTAR guidelines and checklists Randomized vs. observational data Process (criteria, optimize search, determine data to beabstracted, quality score)Anna Sidorchuk 2
30 april 2013Anna SidorchukPrimary data versus secondary data Traditionally, researchers collect and analyze their own data(referred to as primary data) Secondary data analysis is based on data collected by someoneelse (or reanalysis of your own published data) Systematic review is one of the ways to analyze secondary dataSystematic review - systematic, qualitative review of publishedresearch in a particular fieldAdapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford
30 april 2013Anna SidorchukWhat The Lancet thinks aboutsystematic reviewsIn 2005, the editors wrote:“. . . we will require authors of clinical trials submitted to TheLancet to include a clear summary of previous research findings,and to explain how their trial’s findings affect this summary. Therelation between existing and new evidence should be illustratedby direct reference to an existing systematic review and meta-analysis…”
Stages of waste in the production and reportingof research evidence relevant to clinicians andpatients by Chalmers and Glasziou (Lancet, vol. 374, 2009)Anna Sidorchuk 5
30 april 2013Anna SidorchukWhy do we do systematic reviews? To help busy clinicians to summarize current knowledge inrelation to the area of interest To provide high-quality research evidence to guide clinicalpractice To support clinical decision-making To support research proposals
30 april 2013Anna SidorchukHow systematic review can help busyclinicians?By systematically: identifying, appraising, synthesizing, and, if appropriate, statistically combining studies on aspecific topic
30 april 2013Anna SidorchukIn classifications of levels of evidence,systematic reviews are included in the highest level ofevidence
Hierarchy of evidence: a framework for rankingevidence evaluating healthcare interventionsJournal of Clinical NursingVolume 12, Issue 1, pages 77-84, 20 DEC 2002 DOI: 10.1046/j.1365-2702.2003.00662.xhttp://onlinelibrary.wiley.com/doi/10.1046/j.1365-2702.2003.00662.x/full#f1
30 april 2013Anna SidorchukMeta-analysis A statistical method to combine findings across studies Should be considered within the framework of systematic reviewsreview needs to use a systematic approach to minimize bias, addressthe issues of the completeness of the evidence, quality of studiesand combinability of studiesMeta-analysis - the statistical pooling of the results ofstudies that are part of a systematic review
When systematic review is required? Before undertaking a systematic review it is necessary tocheck whether there are already existing or ongoing reviews,and whether a new review is justified Search:the Database of Abstracts of Reviews of Effects (DARE) the Cochrane Database of Systematic Reviews (CDSR)30 april 2013Anna SidorchukAdapted from: Systematic Reviews. CRD’s guidance for undertaking reviews in health care (2009). Centre for Reviews andDissemination, University of York. ISBN 978-1-900640-47-3.
When is meta-analysis appropriate? There exists a critical mass of comparable studies designed toaddress a common research question Data are presented in a form that allows the meta-analyst tocompute an effect size for each study Characteristics of each study are described in sufficient detailto allow meta-analysts to compare characteristics of differentstudies and to judge the quality of each study30 april 2013Anna Sidorchuk
Systematic reviews vs. non-systematic Compared to traditional literature reviews:there is a definite methodology employed in the research analysis(more like that used in primary research); andthe results of the included studies are quantified to a standard metricthus allowing for statistical techniques for further analysis. Therefore process of reviewing research literature is moreobjective, transparent, and replicable; less biased andidiosyncratic to the whims of a particular researcher30 april 2013Anna Sidorchuk
Systematic reviews vs. non-systematicFeature Systematic review Narrative reviewQuestion Often a focused one question Often broad in scopeSources and search Comprehensive sources andexplicit search strategyNot usually specified,potentially biasedSelection Criterion-based selection,uniformly appliedNot usually specified,potentially biasedAppraisal Rigorous critical appraisal VariableSynthesisQualitative summary thatincludes statistical synthesis(meta-analysis)Often a qualitative summaryInferences Usually evidence-based Sometimes evidence-basedAnna Sidorchuk 14Adapted from: Y Yuan and R H Hunt, Am J Gastroenterol 2009; 104:1086–1092; doi:10.1038/ajg.2009.118
18 September, 2009Anna Sidorchuk 15Increase in the number of publishedmeta-analyses (PubMed references)Publication year
• The essence of good science is replicable and generalisable resultsDo we get the same answer to research questions when we run the study again?• The primary aims of meta-analysis is to test the generalisability of resultsacross a set of studies designed to answer the same research questionAre the results consistent? If not, what are the differences in the studies that explainthe lack of consistency?Why are systematic review and meta-analysis important?30 april 2013Anna Sidorchuk
When there is systematic variation in outcomes from different studies,meta-analysis tries to explain these differences in terms of studycharacteristics:measures usedstudy designparticipant characteristicscontrols for potential biasetc.If the results of individual studies areinconclusive or controversial…30 april 2013Anna Sidorchuk
“… doing a meta-analysis is easy,doing one well is hard.”Ingram Olkin, Stanford University
Benefits of systematic reviews, incl. meta-analysis Increased power: by combining information from manyindividual studies, the meta-analyst is able to detect systematictrends not obvious in the individual studies Conclusions based on the set of studies are likely to be moreaccurate than any one study30 april 2013Anna Sidorchuk
Benefits of systematic reviews, incl. meta-analysis Improved precision: based on information from many studies,the meta-analyst can provide a more precise estimate of thepopulation effect size (and a confidence interval) Provides potential corrections for potential biases,measurement error and other possible artefacts Identifies directions for further primary studies to addressunresolved issues30 april 2013Anna Sidorchuk
Benefits of systematic reviews, incl.meta-analysis Typically there is study-to-study variation in results. Whenthis is the case, the meta-analyst can explore whatcharacteristics of the studies explain these differences (e.g.,study design) in ways not easy to do in individual studies Easy to interpret summary statistics (useful if communicatingfindings to a non-academic audience)30 april 2013Anna Sidorchuk
Limitations of systematic reviews, incl.meta-analysis Comparing apples and oranges Quality of the studies included in the meta-analysis What to do when studies don’t report sufficient information(e.g., “non-significant” findings)? Including multiple outcomes in the analysis (e.g., differentachievement scores) Publication bias22
Limitations of systematic reviews, incl.meta-analysis Meta-analysis conclusions may still differ if different studiesare sampled or excluded for different reasons Need to be explicit Quality standards Inappropriate handling of data can lead to wrong conclusions Sample size consideration23
Publication bias Studies that are published are more likely to reportstatistically significant findings. This is a source of potentialbias30 april 2013Anna Sidorchuk
Publication bias The debate about using only published studies:peer-reviewed studies are presumably of a higher qualityVERSUSsignificant findings are more likely to be published than non-significant findings There is no agreed upon solution. However, one should retrieve all studiesthat meet the eligibility criteria, and be explicit with how they dealt withpublication bias. Some methods for dealing with publication bias havebeen developed (e.g., Fail-safe N, Trim and Fill method)30 april 2013Anna Sidorchuk
RCT vs. observational studies Both are fine as long as all the methodological issues areaddressed If your focus is effect of therapy or prevention – use RCTs If you are interested in prognosis, etiology, adverseeffects, association between risk factors and outcomes –use observational studies Fine to use both types of studies in the same reviewAnna Sidorchuk 26
Steps in a meta-analysisEstablishresearchquestionDefinerelevantstudiesDevelopcodematerialsLocate andcollatestudiesPilot coding;codingData entryand effectsizecalculationMainanalysesSupplementary analyses30 april 2013Anna Sidorchuk Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford
Practical tips Define all key elements (PICO model) a priory Formulate the research question as clear as possible Make an agreement with all your co-authors on the key elements If needed ask the experts in the area of outcome or/and exposure foradvice Keep everything in the protocol When writing an article include the elements in the aim30 april 2013Anna Sidorchuk
Practical tips Define all inclusion/exclusion criteria and components of search a priory Make criteria and the components as clear as possible, but feel free tomake some changes during the actual literature search (usually due toavailability of data, e.g. if only retrospective studies are available on theissue, or no info on ethnicity, etc) Make an agreement with all co-authors on abovementioned Again, keep everything in the review protocol30 april 2013Anna Sidorchuk
Practical tips Define coding and form for data extraction a priory Make an agreement with all co-authors on abovementioned If use Excel for data extraction and, therefore, for coding, keep the legendfor each coding element within the same file on the separate sheet As an alternative – use Access Keep everything in the review protocol!30 april 2013Anna Sidorchuk
Practical tips Define as detailed as possible by reading other meta-analysis andsystematic reviews on the similar outcome(s) and exposure(s): All key words MESH terms Define the electronic sources to be used for your search Ask librarians for advice on search strategy and technic Define if grey literature will be included Keep everything in the protocol!!!30 april 2013Anna Sidorchuk
Critical issueQuality, quality, quality!30 april 2013Anna Sidorchuk
Quality Standards in Systematicreviews and Meta-Analysis Several organizations work on thisCochrane collaborationhttp://www.cochrane.orgCampbell collaborationhttp://www.campbellcollaboration.org Consort: Consolidated Standards of Reporting Trials, includesQuorum: Quality of Reporting of Meta-analyses grouphttp://www.consort-statement.org/mod_product/uploads/QUOROMStatement 1999.pdf PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses), AMSTAR (A measurement tool to assess systematic reviews)30 april 2013Anna Sidorchuk
30 april 2013Anna SidorchukPRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.Section/Topic # Checklist ItemReportedon Page #TITLETitle 1 Identify the report as a systematic review, meta-analysis, or both.ABSTRACTStructuredsummary2Provide a structured summary including, as applicable:background; objectives; data sources; study eligibility criteria,participants, and interventions; study appraisal and synthesismethods; results; limitations; conclusions and implications of keyfindings; systematic review registration number.INTRODUCTIONRationale 3Describe the rationale for the review in the context of what isalready known.Objectives 4Provide an explicit statement of questions being addressed withreference to participants, interventions, comparisons, outcomes,and study design (PICOS).
30 april 2013Anna SidorchukPRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.Section/Topic # Checklist ItemReportedon Page #METHODSProtocol andregistration5Indicate if a review protocol exists, if and where it can beaccessed (e.g., Web address), and, if available, provideregistration information including registration number.Eligibilitycriteria6Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered,language, publication status) used as criteria for eligibility,giving rationale.Informationsources7Describe all information sources (e.g., databases with dates ofcoverage, contact with study authors to identify additionalstudies) in the search and date last searched.Search 8Present full electronic search strategy for at least onedatabase, including any limits used, such that it could berepeated.Study selection 9State the process for selecting studies (i.e., screening,eligibility, included in systematic review, and, if applicable,included in the meta-analysis).Data collectionprocess10Describe method of data extraction from reports (e.g., pilotedforms, independently, in duplicate) and any processes forobtaining and confirming data from investigators.Data items 11List and define all variables for which data were sought (e.g.,PICOS, funding sources) and any assumptions andsimplifications made.
30 april 2013Anna SidorchukPRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.Section/Topic # Checklist ItemReportedon Page #METHODSRisk of bias inindividual studies12Describe methods used for assessing risk of bias ofindividual studies (including specification of whether thiswas done at the study or outcome level), and how thisinformation is to be used in any data synthesis.Summarymeasures13State the principal summary measures (e.g., risk ratio,difference in means).Synthesis ofresults14Describe the methods of handling data and combiningresults of studies, if done, including measures of consistency(e.g., I2) for each meta-analysis.Risk of biasacross studies15Specify any assessment of risk of bias that may affect thecumulative evidence (e.g., publication bias, selectivereporting within studies).Additionalanalyses16Describe methods of additional analyses (e.g., sensitivity orsubgroup analyses, meta-regression), if done, indicatingwhich were pre-specified.
30 april 2013Anna SidorchukPRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.Section/Topic # Checklist ItemReportedon Page #RESULTSStudy selection 17Give numbers of studies screened, assessed for eligibility,and included in the review, with reasons for exclusions ateach stage, ideally with a flow diagram.Studycharacteristics18For each study, present characteristics for which data wereextracted (e.g., study size, PICOS, follow-up period) andprovide the citations.Risk of biaswithin studies19Present data on risk of bias of each study and, if available,any outcome-level assessment (see Item 12).Results ofindividual studies20For all outcomes considered (benefits or harms), present, foreach study: (a) simple summary data for each interventiongroup and (b) effect estimates and confidence intervals,ideally with a forest plot.Synthesis ofresults21Present results of each meta-analysis done, includingconfidence intervals and measures of consistency.Risk of biasacross studies22Present results of any assessment of risk of bias acrossstudies (see Item 15).Additionalanalysis23Give results of additional analyses, if done (e.g., sensitivityor subgroup analyses, meta-regression [see Item 16]).
30 april 2013Anna SidorchukPRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.Section/Topic # Checklist ItemReportedon Page #DISCUSSIONSummary ofevidence24Summarize the main findings including the strength ofevidence for each main outcome; consider their relevance tokey groups (e.g., health care providers, users, and policymakers).Limitations 25Discuss limitations at study and outcome level (e.g., risk ofbias), and at review level (e.g., incomplete retrieval ofidentified research, reporting bias).Conclusions 26Provide a general interpretation of the results in the context ofother evidence, and implications for future research.FUNDINGFunding 27Describe sources of funding for the systematic review andother support (e.g., supply of data); role of funders for thesystematic review.
Quality Assessment of Primary Studies A common practice is to assign a quality score to primarystudiesPreferably based on an explicit checklist usually The judgment to include or exclude a study must be statedand justified explicitly This judgment is mostly based on quantitative criteria, butinvolves some subjectivity30 april 2013Anna Sidorchuk
NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALECASE CONTROL STUDIES Note: A study can be awarded a maximum of one star for each numbered item within the Selection and Exposurecategories. A maximum of two stars can be given for Comparability.Selection1) Is the case definition adequate?a) yes, with independent validation ¯b) yes, eg record linkage or based on self reportsc) no description2) Representativeness of the casesa) consecutive or obviously representative series of cases ¯b) potential for selection biases or not stated3) Selection of Controlsa) community controls ¯b) hospital controlsc) no description4) Definition of Controlsa) no history of disease (endpoint) b) no description of source30 april 2013Anna Sidorchuk
NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALECASE CONTROL STUDIESComparability1) Comparability of cases and controls on the basis of the design or analysisa) study controls for _______________ (Select the most important factor.) ¯b) study controls for any additional factor ¯ (This criteria could be modified to indicate specific control for asecond important factor.)Exposure1) Ascertainment of exposurea) secure record (eg surgical records) ¯b) structured interview where blind to case/control status ¯c) interview not blinded to case/control statusd) written self report or medical record onlye) no description2) Same method of ascertainment for cases and controlsa) yes ¯b) no3) Non-Response ratea) same rate for both groups ¯b) non respondents describedc) rate different and no designation30 april 2013Anna Sidorchuk
“Ugly face” of systematic reviews andmeta-analyses Can be misleading Can be misused30 april 2013Anna Sidorchuk