Challenges in Systematic Reviews—Can Using Existing Reviews Save Time? Be Valid? Evelyn P. Whitlock, MD, MPH Associate Director, Oregon Evidence-based Practice Center Senior Investigator, Center for Health Research, KPNW Jennifer Lin, MD, MCR Center for Health Research, KPNW Roger Chou, MD Oregon Health & Science University July 11, 2008

Acknowledgments Karen Robinson, MSc Paul Shekelle, MD, PhD Ed Reid, MS The USPSTF review team at Oregon EPC

Karen Robinson, MSc

Paul Shekelle, MD, PhD

Ed Reid, MS

The USPSTF review team at Oregon EPC

There has been an exponential growth in the production of systematic reviews Reviewers of commissioned reviews must address whether and how to use existing systematic reviews in their new reviews Current systematic review methods do not clearly address this common practice Background

There has been an exponential growth in the production of systematic reviews

Reviewers of commissioned reviews must address whether and how to use existing systematic reviews in their new reviews

Current systematic review methods do not clearly address this common practice

Estimated growth in annual MEDLINE-indexed English language Systematic Reviews *Moher D, et al. PLoS Med 2007

Growth in SRs within USPSTF topics 205 2007 Screening for cervical cancer 428 2006 Screening for adult depression 24 2005 Motor vehicle occupant injury (MVOI) prevention interventions 15 2004 Pediatric obesity screening and interventions Abstracts Year Topic

Reviewers’ Questions about Using Existing Systematic Reviews How can we adhere to comprehensive, transparent, and unbiased systematic review methods? Will this save time and resources ? Are there instances where this shouldn’t be done ? Will this produce valid, reliable results that are both acceptable and useful to decision makers?

How can we adhere to comprehensive, transparent, and unbiased systematic review methods?

Will this save time and resources ?

Are there instances where this shouldn’t be done ?

Will this produce valid, reliable results that are both acceptable and useful to decision makers?

Possible benefits in utilizing existing systematic reviews Save effort and resources for other reviews or evidence-based activities Achieve more timely and more current products through shortcutting parts of the review process Reduce confusion through unnecessary creation of multiple reviews (e.g., more reviews than primary trials in treatment of low back pain)

Save effort and resources for other reviews or evidence-based activities

Achieve more timely and more current products through shortcutting parts of the review process

Reduce confusion through unnecessary creation of multiple reviews (e.g., more reviews than primary trials in treatment of low back pain)

Types of Reviews and Available Methodological Guidance Current Guidance NHS; Cochrane Guidance underway Cochrane Manual None found Review of reviews, overview of reviews, umbrella review (e.g., NHS HDA reviews) Secondary research (i.e., systematic reviews, meta-analyses) (Complex) systematic review, evidence report (e.g., EPC report) Both primary and secondary research To inform policy-level medical decision making (e.g., clinical guidelines, health care coverage decisions) Complex review, multiple interventions or linked questions Systematic review, (e.g., Cochrane review) Primary research (i.e., individual studies) To answer a focused question (gain statistical power and applicability over individual studies) Single question, focused review Nomenclature (example) Admissible evidence Purpose Type

Methodological steps when using existing systematic reviews (SRs)

Does it save time/resources?

Growth in SRs within USPSTF topics *Not including use for reference checking 2 29 205 2007 Cervical Cancer screening 10 112 428 2006 Adult Depression screening 0 24 24 2005 MVOI prevention interventions 1 15 15 2004 Pediatric obesity screening and interventions Included* Articles Abstracts Year Topic

Does it save time/resources? May not save time and can’t tell up front Most likely incorporation is in not reproducing searches—and this can be very time consuming

May not save time and can’t tell up front

Most likely incorporation is in not reproducing searches—and this can be very time consuming

When may a new review be needed? Discordant reviews High-profile topics Substantial differences in methods employed Substantial differences in research questions

Discordant reviews

High-profile topics

Substantial differences in methods employed

Substantial differences in research questions

Other possible hazards in using existing SR Prior review may not always be valid Propagation of error Incomplete methods/relative inexperience may introduce reviewer bias or random error HOWEVER, rejecting prior reviews out of hand is illogical

Prior review may not always be valid

Propagation of error

Incomplete methods/relative inexperience

may introduce reviewer bias or random error

HOWEVER, rejecting prior reviews out of hand is illogical

Implications Ad hoc production of systematic reviews and their prestige has resulted in duplication and variable quality Multiple reviews do not confer the same confidence as multiple trials, but can be useful Redundancy in systematic reviews clearly wastes efforts and confuses users—however user needs aren’t all the same so what is redundant? Updates might be more appropriate than new reviews in some cases Resolving these issues will be useful to those prioritizing new systematic reviews

Ad hoc production of systematic reviews and their prestige has resulted in duplication and variable quality

Multiple reviews do not confer the same confidence as multiple trials, but can be useful

Redundancy in systematic reviews clearly wastes efforts and confuses users—however user needs aren’t all the same so what is redundant?

Updates might be more appropriate than new reviews in some cases

Resolving these issues will be useful to those prioritizing new systematic reviews

Steps forward… More standardized reporting Annals editorial is a start—international group needed. Protocol and review registries More formal updating or sunsetting processes for systematic reviews Refinement/standardization of methods through dialogue, case studies, and empirical investigations

More standardized reporting

Annals editorial is a start—international group needed.

Protocol and review registries

More formal updating or sunsetting processes for systematic reviews

Refinement/standardization of methods through dialogue, case studies, and empirical investigations

OHSU EPC – http://www.ohsu.edu/epc KPCHR – http://www.kpchr.org/public/default.aspx AHRQ’s Effective Health Care Program – http://effectivehealthcare.ahrq.gov/ USPSTF – http://www.ahrq.gov/clinic/uspstfix.htm Websites

Comments & Questions