• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
A mid flight correction: setting priorities
 

A mid flight correction: setting priorities

on

  • 1,995 views

 

Statistics

Views

Total Views
1,995
Views on SlideShare
1,991
Embed Views
4

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 4

http://apps1.jhsph.edu 4

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

A mid flight correction: setting priorities A mid flight correction: setting priorities Presentation Transcript

  • A mid flight correction: setting priorities David Moher Canadian Cochrane Centre and Network Cochrane Collaboration’s Bias Methods Group University of Ottawa’s Evidence based Practice Center CIHR and AHRQ Acknowledge members of the Chalmers Research Group
  • Outline of talk
    • Quality of reports of systematic reviews
    • Systematic review non-registration
    • Systematic review non-publication
    • Systematic review selective outcome reporting
    • Updating systematic reviews
  • Epidemiology and reporting characteristics of systematic reviews
    • Searched MEDLINE to identify systematic reviews
      • If the authors’ stated objective was to summarize evidence from multiple studies and the article described explicit methods, regardless of the details provided
      • Included reports of Cochrane reviews
    • November 2004
    • Extracted information about epidemiology and reporting characteristics
    Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007; (3):e78.doi:10.1371/journal.pmed.0040078
    • Identified 300 reviews
      • extrapolate to 2500 new English language systematic reviews published annually
      • 20% are Cochrane; 80% or non-Cochrane reviews
    • Reviews are getting bigger; More informative
      • median = 16 (iqi: 7-30); non-cochrane = 23
      • number of participants: median = 1112 (iqi: 322-3750)
    Epidemiology and reporting characteristics of systematic reviews
    • Most (91%) reports of systematic reviews are published in small specialty journals
      • Don’t sample the ‘usual’ 5 journals!
    • Some improvements over time
      • reports of use of risk of bias assessments (66.8%)
    Epidemiology and reporting characteristics of systematic reviews
    • Reports of Cochrane reviews are appreciably better than non-Cochrane ones
    • Reports of non-Cochrane reviews are particularly troubling
      • most reviews (17.7%) are not reports of updates
      • some suggestion of outcome reporting bias
      • sub-optimal reporting of publication bias assessment (23.1%)
    • No report is registered in the conventional sense
      • none had a formal registration number
    Epidemiology and reporting characteristics of systematic reviews
  • More recent data
    • Similar study
    • Looking at systematic reviews of RCTs
    • Sample (2000-2005)
      • 35% published in 2005
      • 51% published in 2004 and 2005
    • Similar results
      • Cochrane reviews better reported than non-Cochrane reviews
      • Still room for improvement in reporting all reviews
    Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, Liu P, Ye L, Li Y, Tian W. The reporting quality of meta-analyes improves: a random sampling study. Journal of Clinical Epidemiology 2008;61:770-775
    • Do not have data on how well evidence reports and comparative effectiveness reviews perform compared to Cochrane and other groups conducting systematic reviews
  • Methodological quality of systematic reviews of animal studies: a survey of reviews of basic research
    • Electronic search (1996-2004)
      • medline, embase
    • Contact experts; searched reference lists
    • Searched for systematic reviews
      • terminology ambiguity
    • Screened searched results
    • Included 30 animal studies: administered to live animals, laboratory measures
    • Compared 30 animal studies to 45 randomly selected bench studies
    Mignini LE, Khan KS. Methodological quality of systematic reviews of animal studies: a survey of basic research. BMC Medical Research Methodology. 2006;
  • Methodological quality of systematic reviews of animal studies: a survey of reviews of basic research
  • Push (priority setting) at the expense of quality
    • Little uptake, and adherence of reporting guidelines for systematic reviews
      • PRISMA
    • Inconsistent agreement (terminology) on what a systematic review is!
    • Pushing reviews out the door at what expense?
  • Setting priorities
    • NLM index term “systematic reviews”
    • Fund quality feedback loop
    • Funders need to work with publishers and editors to improve uptake and adherence of guidelines for reporting systematic reviews
      • It improves quality
    • Need to develop template for conducting systematic reviews
      • It improves quality
  • “ Accurate and transparent reporting is like turning on the light before you clean up a room: it doesn’t clean it for you but does tell you where the problems are ” (Frank Davidoff, editor of Annals of Internal Medicine, 2000).
  • Reporting guidelines
    • Reporting Guideline specify minimum information necessary for clear accounting of research methodology and findings
    • A good return on funder investment
    • Number of reporting guidelines
      • n ~ 80
    • As yet most guidelines have had limited impact
      • Passive dissemination through publication only - not widely known
      • Complience not required by journals
  • EQUATOR: E nhancing the QUA lity and T ransparency O f health R esearch
    • EQUATOR Network (set up in 2006) grew out of the work of CONSORT and other guidelines groups
    • ‘ Umbrella’ organization promoting transparent and accurate reporting of health research through the implementation of reporting guidelines
    • Key stakeholders:
      • Developers of reporting guidelines
      • Editors & peer reviewers of general and specialty journals
      • Researchers (authors), medical writers,
      • Research funders
      • Everyone interested in improving the quality of research publications and of research itself
  •  
  • EQUATOR Core program
    • Main objectives for the next 5 years:
      • Provide resources and training enabling the improvement of health research reporting
      • Monitor progress in the improvement of health research reporting
    www.equator-network.org
  • Improving quality
    • Shared responsibility
    • Moral obligation
    • All stakeholders need to develop and fund feedback loop for quality and systematic reviews
  • Registering systematic reviews “Moher and colleagues call for the registration of systematic reviews in the same manner as is now required for clinical trials: it would then, at least, be possible to track those that are performed.” The PLoS Medicine Editors. Many reviews are systematic but some are more transparent and completely reported than others. PLoS Medicine 2007;
    • reduce redundant research and publications
    • reduce systematic review non-publication bias
    • minimize selective outcome reporting
    • Improve priority setting activities
    • “ Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study”
      • 10 systematic reviews published between 2003 and 2005
    • “ Acetylcysteine in the prevention of contrast-induced nephropathy: a case study of the pitfalls in the evolution of evidence”
      • 11 meta-analyses
    Bagshaw SM, McAlister FA, Manns BJ, Ghali WA. Acetylcysteine in the prevention of contrast-induced nephropathy: a case study of the pitfalls in the evolution of evidence. Arch Intern Med 2006;166:161-166 Biondi-Zoccai GGL, Lotrionte M, Abbate A, Teta L, Remigi E, Burzotta F, Valgimigli M, Romagnoli E, Crea F, Agostoni P. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ 2006; 332(7535):202-9
  • Registration and priority setting
    • Need to know what’s out there in order to set priorities
  • “ It is surely a great criticism of our profession that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant randomised controlled trials” It is surely a great criticism that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant systematic reviews
  • Selective outcome bias
    • selective reporting of outcomes
      • typically statistically positive
      • selected by investigators (post hoc)
  • Outcomes reporting bias
    • Methods
      • compared the contents of 102 trial protocols, approved by the scientific-ethics committees for copenhagen and frederiksberg, denmark, during 1994 and 1995, with 122 subsequent publications
    Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457-2465.
  • Some salient results
    • Nearly two-thirds had a change in at least one primary outcome between the protocol and publication
    • Statistically significant outcomes had a higher likelihood of being reported compared to non-significant ones
  • Does selective outcome reporting exist within systematic reviews
    • Back to the 300 reports of systematic reviews
      • 80% of published literature is non-Cochrane
      • 161 systematic reviews with meta-analysis
    • Statistical significance of primary outcome (if stated)
      • Cochrane: ~ 75% reported primary outcomes; 14.4% statistically significant
      • Non-Cochrane: ~ 25% reported primary outcome; 50% statistically significance
    Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007; 4(3):e78.doi:10.1371/journal.pmed.0040078
  • Does systematic review non-publication bias exist?
    • Surveyed 625 authors of systematic reviews
      • random sample of systematic reviews published in 2005
      • 1 st author or corresponding author
    • Developed 26-item survey to tap into:
      • publication process
      • identification of unpublished reviews
    • Pilot tested
    • Completed using Dillman method
    • Completed on Survey Monkey
    1 Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli M, Hopewell S, Lavis J, Berlin JA, Moher D. Systematic review publication bias: an international survey
  • Does systematic review non-publication bias exist?
    • 64% survey response rate
    • 93% survey completion rate
    • 6.3% reported that they were never the lead on the review!
    • Respondents reported a total of 1385 published systematic reviews
      • median = 2
    • 52 respondents reported a total of 200 unpublished systematic reviews
      • median = 2
    • Most unpublished reviews (73%) completed after 2005
  • What is updating
    • “A discrete event aiming to search for and identify new evidence to incorporate into a previously completed systematic review with new evidence taken to mean any evidence not included in the previously completed review irrespective of its chronological appearance in the literature”
    • Most international organizations agree with this definition
    Moher D, Tsertsvadze A. When is an update an update. Lancet 2006 Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey
  • Why is updating important
    • Keeps systematic reviews informative
    • What happens if reviews are out-of-date
      • Impact on clinical practice
      • Policy recommendations
      • Future primary research
        • Funding agencies
    • Most reports of systematic reviews do not appear to be updates
  • Updating systematic reviews
    • To estimate the average time to changes in evidence sufficiently important to warrant updating systematic reviews
    • To determine the performance characteristics of various surveillance protocols to identify important new evidence
    • To assess the utility of rates and patterns of growth for evidence within clinical areas as predictors of updating needs
    • To establish typical timeframes for the production and publication of systematic reviews in order to assess the extent to which they impact shelf-life
    • Identify the updating experiences of healthcare organizations that fund and/or conduct systematic reviews
    Shojania K, Sampson M, Ji J, Ansari M, Garritty C, O’Rourke K, Rader, Moher D. Updating Systematic Reviews. Technical Review No. 16. (Prepared by the University of Ottawa Evidence-based Practice Center under Contract No. 290-02-0017). AHRQ Publication No 07-0087 Rockville, MD: Agency for Healthcare Research and Quality. September 2007
  • Updating systematic reviews
    • To estimate the average time to changes in evidence sufficiently important to warrant updating systematic reviews
    • To determine the performance characteristics of various surveillance protocols to identify important new evidence
    • To assess the utility of rates and patterns of growth for evidence within clinical areas as predictors of updating needs
    • To establish typical timeframes for the production and publication of systematic reviews in order to assess the extent to which they impact shelf-life
    • Identify the updating experiences of healthcare organizations that fund and/or conduct systematic reviews
    Shojania K, Sampson M, Ji J, Ansari MT, Doucette S, Moher D. How quickly do systematic reviews go out of date: A survival analysis . Annals of Internal Medicine 2007;147:224-233 .
  • How quickly do systematic reviews become out of date: a survival analysis
    • Systematic reviews evaluating benefit or harm
      • drug, device, or procedure
      • published between 1995 and 2005
      • included at least 1 meta-analysis
    • Quantitative signal to update
      • change in statistical significance
      • for primary outcome or any mortality outcome a relative change in effect estimate of at least 50%
    • Qualitative signal to update
      • substantial difference in characterization of effectiveness
      • new information about harms
      • superior alternative treatments
      • important caveats about previous findings that would influence clinical decision making
    Signals for updating
  • Structured Search Protocols
    • Identification of new SRs on the same topic
    • Clinical Queries filters in PubMed
    • Applied Related Articles function in PubMed to 3 largest and 3 most recent trials in the original review
    • Using a ‘citing references’ search engine (Scopus) to identify new RCTs that cited original review
    • Relevant sections of Clinical Evidence, UpToDate
    • Screened 325 reports to obtain cohort of 100
    • 13 (median) included studies and 2663 participants
    • 85% evaluated drug therapies
    • Five most common clinical areas
      • cardiovascular medicine, 20
      • gastroenterology, 13
      • neurology, 11
      • infectious diseases, 9
      • respiratory system, 9
    Results
    • identified at least one new eligible trial for 85 of the reviews
      • 4 new trials (median increase in evidence base: 25%)
      • 1160 participants (median increase in evidence base: 47%)
    Results
    • A quantitative or qualitative signal occurred in 57 of the reviews
      • median event-free survival = 5.5 years (95%, confidence interval: 4.6, 7.6)
      • 23% of reviews needed updating within 2 years
      • 15% within 1 year
      • 7% already needed updating at time of publication
    • Quantitative signal
      • occurred in 20 of systematic reviews
    • Qualitative signal
      • occurred in 54 of systematic reviews
    Results
  • International survey
    • Describe the updating practices and policies of agencies that sponsor or conduct systematic reviews
    Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey
  • International survey
    • Stratified purposeful sampling approach
    • Key international groups commonly involved in undertaking/funding reviews
      • 195 organizations
    • 48 questions
    • Pilot tested
    • Administered through Survey Monkey
    • 65% response rate from 26 countries
    • 79% considered importance of updating ‘high’ to ‘very high’
    • 57% have formal policy
      • although only 29% have written policy
    • About half (51%) respondents reported that more than 50% of their systematic reviews are estimated to be out-of-date
  • Setting priorities
    • Before embarking on setting priorities for new reviews ensure that existing reviews are kept up-to-date
    • Encourage journals to have an explicit policy on publishing systematic review updates
    • Centralize aspects of updating
      • Searching the literature
  •