Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Systematic Reviews: the process, quantitative, qualitative and mixed methods reviews. Edoardo Aromataris


Published on

Health Libraries Australia Professional Development Day 2012 [Keynote address - part 2]

Published in: Health & Medicine, Technology
  • Water Hack Burns 2lb of Fat Overnight (video tutorial) ☞☞☞
    Are you sure you want to  Yes  No
    Your message goes here
  • Strange "water hack" burns 2lbs overnight. Do this hack to drop 2lb of fat in 8 hours (video tutorial) ✱✱✱
    Are you sure you want to  Yes  No
    Your message goes here
  • Water Hack Burns 2lb of Fat Overnight (video tutorial) ★★★
    Are you sure you want to  Yes  No
    Your message goes here
  • The Insider's Edge You've Been Looking For.... ■■■
    Are you sure you want to  Yes  No
    Your message goes here
    Are you sure you want to  Yes  No
    Your message goes here

Systematic Reviews: the process, quantitative, qualitative and mixed methods reviews. Edoardo Aromataris

  1. 1. Systematic Reviews: the process,quantitative, qualitative and mixed methods reviews Edoardo Aromataris
  2. 2. Systematic reviews• Different strokes... – Where I’m coming from• Introduction to reviews and systematic reviews – Terminology• Systematic steps• Journey beyond where you may be involved...• Where you’ll find differences in quantitative and qualitative
  3. 3. Some answers!?• What is a systematic review?
  4. 4. Some answers!?• An attempt to identify, appraise and synthesise all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question.• An attempt to sum up the best available research on a specific question. This is done by synthesising the results of several studies.
  5. 5. The Nature of Evidence• “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials” Archie Cochrane, 1972
  6. 6. Cochrane Collaboration• Established 1993• Leaders in EBM, some 28,000 members• Focus on effectiveness of interventions and therapies in medical practice – RCTs focus – More pluralistic – qualitative methods• Organised into Cochrane Review Groups (CRG)• Cochrane Library
  7. 7. Campbell Collaboration (C2)• Focus on systematic reviews of effectiveness in education, crime and justice, and social welfare• Coordinating groups (6)• Campbell Systematic Reviews• Synthetic Reviews
  8. 8. The Nature of Evidence• “It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials” Archie Cochrane, 1972.• Shift toward pluralistic, inclusive definitions of what evidence is, and subsequently of what evidence based practice is...
  9. 9. Joanna Briggs Institute• Established 1996• Feasibility, Appropriateness, Meaningfulness and Effectiveness• Qualitative methods• JBC – 70+ Centres conduct reviews• JBI Library of Systematic Reviews• Leaders in EBHC and evidence implementation
  10. 10. Introduction to Systematic Reviews• Variations in conduct• Different questions, different methods of synthesis – E.g. effectiveness, aetiology, harms, meta-ethnographic reviews, realist syntheses• Not all are validated – Term used loosely• Broad coverage, applicable to most reviews – Systematic!
  11. 11. Terminology• Literature review summarises, critiques, and synthesizes articles while not using systematic methodology• Systematic reviews adhere to explicit and rigorous methods to identify, critically appraise, and synthesise relevant primary/original studies. (Krainovich-Miller, 2006:87)
  12. 12. Systematic Review• Also called “Research Synthesis”• Is an attempt to integrate empirical data for the purpose of: – uncovering the international evidence and – producing statements about that evidence to guide decision making• Requires explicit and exhaustive reporting of the methods used in synthesis
  13. 13. Systematic Review• ‘…an attempt to minimize the element of arbitrariness…by making explicit the review process, so that, in principle, another reviewer with access to the same resources could undertake the review and reach broadly the same conclusions’ (Dixon-Wood et al. 1997:157 quoted by Seers, 2005:102)
  14. 14. Bias• ‘systematic deviation of results or inferences from truth’• ‘an error in the conception and design of a study- or in the collection, analysis, interpretation, reporting, publication, or review of data-leading to results or conclusions that are systematically (as opposed to randomly) different from truth’ (Porta, 2008:18)
  15. 15. Meta-analysis• Quantitative evidence – Questions of Effectiveness, Feasibility and/or Appropriateness• Use of statistical methods to combine the results of various independent, similar studies• More precise calculation of one estimate of treatment effect than could be achieved by any of the individual, contributing studies• Only forms a part of the systematic review in which it appears
  16. 16. Meta-synthesis• Qualitative evidence – Questions of Meaningfulness, Feasibility and/or Appropriateness• Qualitative analysis of a number of independent qualitative research studies and text• Use of qualitative methods of combining the findings of individual studies• Only forms a part of the systematic review in which it appears
  17. 17. Qualitative Research Findings asEvidence for Practice• Qualitative evidence is of increasing importance in health services policy, planning and delivery.• It can play a significant role in: – understanding how individuals / communities perceive health, manage their own health and make decisions related to health service usage; – increasing our understandings of the culture of communities and of health units; – Informing planners and providers; – evaluating components and activities of health services that cannot be measured in quantitative outcomes.
  18. 18. Systematic Review• The notion of and methods for establishing credibility in systematic reviews has been extensively developed and debated• In terms of quantitative evidence: – Emphasis on reducing bias and increasing validity – Degree of credibility established through critique and by applying levels of evidence• In terms of qualitative evidence: – Emphasis on rigour of research design and transferability – Degree of credibility established through critique and by applying levels of credibility
  19. 19. Comprehensive/Mixed methodReview• Series of questions• Combines both quantitative and qualitative findings and addresses multiple forms of evidence• 2 or more types of evidence• Different approaches - particularly to integration of qualitative findings
  20. 20. Characteristics of a SR• Protocol driven process• Clearly stated set of objectives with pre-defined eligibility criteria for studies• Explicit, reproducible methodology• Systematic search that attempts to identify all studies that would meet the eligibility criteria• Assessment of the validity of the findings of the included studies• Systematic presentation, and synthesis, of the characteristics and findings of the included studies (Green et al., 2008:6)
  21. 21. Steps in a Systematic Review• Formulate review question• Define inclusion and exclusion criteria• Locate studies• Select studies• Assess study quality• Extract data• Analysis/summary and synthesis of relevant studies• Present results• Interpret results/determining the applicability of results (Egger & Davey Smith, 2001:25; Glasziou et al., 2004:2)
  22. 22. Question DevelopmentDivide question following the PICO/PICo model• Reviews of effects & • Reviews of qualitative & economics: Textual data: – Population – Population – Intervention/Exposure – Phenomena of Interest – Comparator – Context – Outcome – Types of Study Design
  23. 23. Question?• Very important - guides entire review• Effectiveness? Meaningfulness?• Other components - How? Why? When?
  24. 24. Where to Start?• Conduct a search for previous Systematic Reviews on your topic: has this review already been conducted or is a review already in progress? – Yes = stop and refine your question – No = proceed with your protocol development
  25. 25. Protocol• Detailed review methods a priori• Guide process – Reasoned approach to the question asked• Decrease biased post hoc changes – Important to avoid “fishing”• Review proper may deviate – Needs clear explanation how and why
  26. 26. Protocol Development • The usefulness and success of the review stems from the robustness of the protocol • The protocol: – Guides the specific direction of the review – Describes inclusion criteria – Identifies the appropriate search sources and resources – Methods of appraisal, extraction and synthesis
  27. 27. Background• Describe the issue under review, including: – Target population, interventions, outcomes,• Should concisely overview the main elements of the review, and issues within the topic of choice• Provide adequate detail to justify the conduct of the review and choice of inclusion criteria• Provide necessary definitions of important terms and concepts
  28. 28. Inclusion Criteria: Quantitative data• Study designs• Population characteristics• Intervention or exposure• Comparator - active or passive• Outcomes of interest
  29. 29. Inclusion Criteria: Qualitative & Text• Study designs• Population characteristics• Phenomena of Interest• Context
  30. 30.
  31. 31. Search Strategy• Know and understand the question!• What is the relevant PICO / PICo?• Qualitative, quantitative or economic?• Feasibility, Appropriateness, Meaningfulness or Effectiveness?• Other questions? Extra concepts? Less concepts?
  32. 32. Search Strategy Steps• Initial Search – initial search of MEDLINE, CINAHL, followed by analysis of text words in the title and abstract• Second Search – all identified key words and index terms across all databases• Third Search – references of identified studies and unpublished studies
  33. 33. Search Strategy• Features of search strategy – Sensitivity – ability to identify all the relevant studies – Specificity – ability to exclude irrelevant studies• Inverse relationship between sensitivity and specificity – High sensitivity will tend to have low specificity – A large number of articles retrieved may not be relevant to the review question
  34. 34. 
  35. 35. Searching• Depend on methods (to minimise bias, or not?) – Exhaustive or Saturation based? – Limited or comprehensive?• Impact on transparency and auditability• Where appropriate? – How much is enough? – Are appropriate sources being covered?
  36. 36. Types of Sources• What to use? – Scientific databases – Scientific Journals – Organisations – Websites – Libraries – Experts
  37. 37. Types of Resources• Peer reviewed journal articles – Research – Opinion/commentary/letters• Grey Literature – Google Scholar, Mednar• Theses/Dissertations – DATAD• Data – Statistics• Circulars• Reports
  38. 38. Databases and sources…• Medline/PubMed • ScienceDirect• Scopus/Embase • TRIP• POPLINE • Wiley InterScience• CINAHL • SPORTDiscus• ERIC • ISI Web of Science• PsychINFO • + many more…! Consult your Librarian!
  39. 39. Research and Trials Registers• Cochrane library – CCTR• Clinical• Controlled Clinical Trials• NHS Research Register• REGARD (database of the ESRC)
  40. 40. Grey literature• Mednar• PsycEXTRA• OAIster• SIGLE• Google Scholar
  41. 41. Theses & Dissertations• Index to Thesis• Networked Digital Library of Theses and Dissertations (NDLTD)• ProQuest Dissertations and Theses Database• EThOS - Beta - Electronic Thesis Online System• TROVE (ADTP)• +…
  42. 42. Government Websites• Are there government agencies that may have evidence relevant to your review question? – NCCAM – CDC – NH&MRC – AHRQ – AIHW – +…
  43. 43. Search Logistics! Search Strategy • Apply Search Strategy to databases • Export to bibliographic Search Sources software – E.g. Endnote • Document process Export citations to bibliographic software
  44. 44. 
  45. 45. Study Selection• Study Selection is an initial assessment that occurs following the review Search• It addresses the question “should the paper be retrieved?” – 2nd assessment that occurs after retrieval and addresses the question “should the study be included in the review?” - this is CRITICAL APPRAISAL!• It is essential to use two assessors in both the selection and critical appraisal processes to limit the risk of error
  46. 46. Selection Process• Aims to select only those studies that address the review question and that match the inclusion criteria documented in your protocol• Scan titles and abstracts• Err on the side of caution - Inclusive!• If uncertain? - Retrieve - scan full text• The selection should be: – Transparent – Reproducible
  47. 47. Critical Appraisal• Aim is to establish validity – To establish the risk of bias – Internal validity• Every review must set out to use an explicit appraisal process. Essentially, – A good understanding of research design is required in appraisers; and – The use of an agreed checklist or scale is usual.
  48. 48. 1004 references Why Critically 172 duplicates Appraise? 832 references Scanned Ti/Ab • Combining results of 715 do not meet poor quality research Incl. criteria may lead to biased or 117 studies misleading results and retrieved 82 do not meet understandings Incl. criteria 35 studies forCritical Appraisal
  49. 49. Assessing the Risk of Bias• Numerous tools are available for assessing methodological quality of clinical trials and observational studies.• Generally requires the use of a specific tool for assessing risk of bias in each included study.• ‘High quality’ research methods can still leave a study at important risk of bias (e.g. when blinding is impossible)• Some markers of quality are unlikely to have direct implications for risk of bias (e.g ethical approval, sample size calculation)
  50. 50. Type of bias Quality Population assessment Allocation Selection Allocation Treatment Control concealmentPerformance Blinding Exposed to Not exposed intervention Detection Blinding Population Population Attrition ITT follow up Follow up Follow up
  51. 51. JBI-MAStARI Instrument
  52. 52. Critical Appraisal• Exclude studies on basis of appraisal?• Explore impact through sensitivity analyses/subgroups
  53. 53. Critical Appraisal & Qualitative Review• General acceptance of need for quality• Ongoing debate around the role of appraisal• Particular focus of debate on role of scales and sum scores in appraisal• In practice, appraisal instruments for qualitative research tend to focus on establishing the degree to which the evidence applies to practice (transferability) rather than internal validity (credibility)
  54. 54.
  55. 55. Considerations in Data Extraction• Source - citation and contact details• Eligibility - confirm eligibility for review• Methods - study design, concerns about bias• Participants - total number, setting, diagnostic criteria• Interventions - total number of intervention groups• Outcomes - outcomes and time points• Results - for each outcome of interest: sample size, etc• Miscellaneous - funding source, etc
  56. 56. Data Extraction• The data extracted for a systematic review are the results from individual studies specifically related to the review question.• Difficulties related to the extraction of data include: – different populations used – different outcome measures – different scales or measures used – interventions administered differently – reliability of data extraction (i.e: between reviewers)
  57. 57. Extracting Findings for Qualitativereviews• The units of extraction in this process are specific findings and illustrations from the text that demonstrate the origins of the findings• A finding is defined as: A conclusion reached by the researcher(s) and often presented as themes or metaphors
  58. 58. 
  59. 59. Analyses• Depend on data collected/question• Maybe simply narrative• Meta analysis?• Qualitative syntheses?• Mixed methods
  60. 60. Analyses and reporting – What interventions/activities have been evaluated – The effectiveness /appropriateness /feasibility /meaningfulness of the intervention/activity – Contradictory findings and conflicts – Limitations of study methods – Issues related to study quality – The use of inappropriate definitions – Specific populations excluded from studies – Future research needs
  61. 61. 1004 referencesMeta Analysis & 172 duplicates 832 referencesMeta Synthesis Scanned Ti/Ab 715 do not meet Incl. criteria 117 studies retrieved 82 do not meet Incl. criteria 35 studies for Critical Appraisal 9 excluded studies 26 studies incl. in review 6 studies incl. 20 studies incl. in meta analysis in narrative
  62. 62. Statistical methods for meta-analysis• Quantitative method of combining results of independent studies• Aim is to increase precision of overall estimate• Investigate reasons for differences in risk estimates between studies• Discover patterns of risk amongst studies
  63. 63. When is meta-analysis useful?• If studies report different treatment effects.• If studies are too small (insufficient power) to detect meaningful effects.• Single studies rarely, if ever, provide definitive conclusions regarding the effectiveness of an intervention.
  64. 64. When meta-analysis can be used• Meta analysis can be used if studies: – have the same population – use the same intervention administered in the same way. – measure the same outcomes• Homogeneity – studies are sufficiently similar to estimate an average effect.
  65. 65. Example meta-analysisTaken from Egger, M. et al. BMJ 1998;316:140-144
  66. 66. Meta-Synthesis• Analysis and synthesis of qualitative studies• Based on processed data• Aim of meta-synthesis • is to assemble findings; • categorise these findings into groups on the basis of similarity in meaning; • aggregate these to generate a set of statements that adequately represent that aggregation.
  67. 67. Data Synthesis Involves:• Step 1: Identifying findings• Step 2: Grouping findings into categories; and• Step 3: Grouping categories into synthesised findings
  69. 69. Comprehensive/Mixed method Review FOCUS technical brief #25, Angela Harden, 2010
  70. 70. Interpretation• Quality, strength and applicability of evidence for outcomes• Applicability for stakeholders/policy makers• Qualitative evidence can aid in interpretation• How, why, barriers, facilitators, experience• Conclusions/recommendations should be drawn based on available evidence• May be none available!
  71. 71. The JBI Software System for the Unified Management of the Assessment and Review of Information
  72. 72. Consists of the following componentsComprehensive Review and Management SystemKeeps all review information together and generates a report that may contain elements from all other SUMARI modules
  73. 73. • Protocol• Reviewers• Search strategy• Bibliography - retrieved studies• Bibliography - non selected studies
  74. 74. QualitativeAssessment andReview Instrument
  75. 75. Meta-Analysis ofStatisticsAssessment andReview Instrument(Used in place of Review Managerfor non-Cochrane reviews)
  76. 76. Narrative Opinionand TextAssessment andReview Instrument
  77. 77. Analysis of CostTechnology andUtilisationAssessment andReview Instrument