This document summarizes key points from a presentation on improving the quality of systematic reviews. It discusses issues like selective reporting of outcomes, non-publication of reviews, lack of registration, and the need for funders to improve reporting guidelines and develop templates for reviews. It also presents results from studies showing most reviews are not updated and estimating the time needed to update reviews is around 5 years. Surveys found many organizations see updating as important but few have formal policies for it.

1. A mid flight correction: setting priorities David Moher Canadian Cochrane Centre and Network Cochrane Collaboration’s Bias Methods Group University of Ottawa’s Evidence based Practice Center CIHR and AHRQ Acknowledge members of the Chalmers Research Group

2. Outline of talk Quality of reports of systematic reviews Systematic review non-registration Systematic review non-publication Systematic review selective outcome reporting Updating systematic reviews

3. Epidemiology and reporting characteristics of systematic reviews Searched MEDLINE to identify systematic reviews If the authors’ stated objective was to summarize evidence from multiple studies and the article described explicit methods, regardless of the details provided Included reports of Cochrane reviews November 2004 Extracted information about epidemiology and reporting characteristics Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007; (3):e78.doi:10.1371/journal.pmed.0040078

4. Identified 300 reviews extrapolate to 2500 new English language systematic reviews published annually 20% are Cochrane; 80% or non-Cochrane reviews Reviews are getting bigger; More informative median = 16 (iqi: 7-30); non-cochrane = 23 number of participants: median = 1112 (iqi: 322-3750) Epidemiology and reporting characteristics of systematic reviews

5. Most (91%) reports of systematic reviews are published in small specialty journals Don’t sample the ‘usual’ 5 journals! Some improvements over time reports of use of risk of bias assessments (66.8%) Epidemiology and reporting characteristics of systematic reviews

6. Reports of Cochrane reviews are appreciably better than non-Cochrane ones Reports of non-Cochrane reviews are particularly troubling most reviews (17.7%) are not reports of updates some suggestion of outcome reporting bias sub-optimal reporting of publication bias assessment (23.1%) No report is registered in the conventional sense none had a formal registration number Epidemiology and reporting characteristics of systematic reviews

7. More recent data Similar study Looking at systematic reviews of RCTs Sample (2000-2005) 35% published in 2005 51% published in 2004 and 2005 Similar results Cochrane reviews better reported than non-Cochrane reviews Still room for improvement in reporting all reviews Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, Liu P, Ye L, Li Y, Tian W. The reporting quality of meta-analyes improves: a random sampling study. Journal of Clinical Epidemiology 2008;61:770-775

8. Do not have data on how well evidence reports and comparative effectiveness reviews perform compared to Cochrane and other groups conducting systematic reviews

9. Methodological quality of systematic reviews of animal studies: a survey of reviews of basic research Electronic search (1996-2004) medline, embase Contact experts; searched reference lists Searched for systematic reviews terminology ambiguity Screened searched results Included 30 animal studies: administered to live animals, laboratory measures Compared 30 animal studies to 45 randomly selected bench studies Mignini LE, Khan KS. Methodological quality of systematic reviews of animal studies: a survey of basic research. BMC Medical Research Methodology. 2006;

10. Methodological quality of systematic reviews of animal studies: a survey of reviews of basic research

11. Push (priority setting) at the expense of quality Little uptake, and adherence of reporting guidelines for systematic reviews PRISMA Inconsistent agreement (terminology) on what a systematic review is! Pushing reviews out the door at what expense?

12. Setting priorities NLM index term “systematic reviews” Fund quality feedback loop Funders need to work with publishers and editors to improve uptake and adherence of guidelines for reporting systematic reviews It improves quality Need to develop template for conducting systematic reviews It improves quality

13. “ Accurate and transparent reporting is like turning on the light before you clean up a room: it doesn’t clean it for you but does tell you where the problems are ” (Frank Davidoff, editor of Annals of Internal Medicine, 2000).

14. Reporting guidelines Reporting Guideline specify minimum information necessary for clear accounting of research methodology and findings A good return on funder investment Number of reporting guidelines n ~ 80 As yet most guidelines have had limited impact Passive dissemination through publication only - not widely known Complience not required by journals

15. EQUATOR: E nhancing the QUA lity and T ransparency O f health R esearch EQUATOR Network (set up in 2006) grew out of the work of CONSORT and other guidelines groups ‘ Umbrella’ organization promoting transparent and accurate reporting of health research through the implementation of reporting guidelines Key stakeholders: Developers of reporting guidelines Editors & peer reviewers of general and specialty journals Researchers (authors), medical writers, Research funders Everyone interested in improving the quality of research publications and of research itself

17. EQUATOR Core program Main objectives for the next 5 years: Provide resources and training enabling the improvement of health research reporting Monitor progress in the improvement of health research reporting www.equator-network.org

18. Improving quality Shared responsibility Moral obligation All stakeholders need to develop and fund feedback loop for quality and systematic reviews

19. Registering systematic reviews “Moher and colleagues call for the registration of systematic reviews in the same manner as is now required for clinical trials: it would then, at least, be possible to track those that are performed.” The PLoS Medicine Editors. Many reviews are systematic but some are more transparent and completely reported than others. PLoS Medicine 2007; reduce redundant research and publications reduce systematic review non-publication bias minimize selective outcome reporting Improve priority setting activities

20. “ Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study” 10 systematic reviews published between 2003 and 2005 “ Acetylcysteine in the prevention of contrast-induced nephropathy: a case study of the pitfalls in the evolution of evidence” 11 meta-analyses Bagshaw SM, McAlister FA, Manns BJ, Ghali WA. Acetylcysteine in the prevention of contrast-induced nephropathy: a case study of the pitfalls in the evolution of evidence. Arch Intern Med 2006;166:161-166 Biondi-Zoccai GGL, Lotrionte M, Abbate A, Teta L, Remigi E, Burzotta F, Valgimigli M, Romagnoli E, Crea F, Agostoni P. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ 2006; 332(7535):202-9

21. Registration and priority setting Need to know what’s out there in order to set priorities

22. “ It is surely a great criticism of our profession that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant randomised controlled trials” It is surely a great criticism that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant systematic reviews

23. Selective outcome bias selective reporting of outcomes typically statistically positive selected by investigators (post hoc)

24. Outcomes reporting bias Methods compared the contents of 102 trial protocols, approved by the scientific-ethics committees for copenhagen and frederiksberg, denmark, during 1994 and 1995, with 122 subsequent publications Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291:2457-2465.

25. Some salient results Nearly two-thirds had a change in at least one primary outcome between the protocol and publication Statistically significant outcomes had a higher likelihood of being reported compared to non-significant ones

26. Does selective outcome reporting exist within systematic reviews Back to the 300 reports of systematic reviews 80% of published literature is non-Cochrane 161 systematic reviews with meta-analysis Statistical significance of primary outcome (if stated) Cochrane: ~ 75% reported primary outcomes; 14.4% statistically significant Non-Cochrane: ~ 25% reported primary outcome; 50% statistically significance Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007; 4(3):e78.doi:10.1371/journal.pmed.0040078

27. Does systematic review non-publication bias exist? Surveyed 625 authors of systematic reviews random sample of systematic reviews published in 2005 1 st author or corresponding author Developed 26-item survey to tap into: publication process identification of unpublished reviews Pilot tested Completed using Dillman method Completed on Survey Monkey 1 Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli M, Hopewell S, Lavis J, Berlin JA, Moher D. Systematic review publication bias: an international survey

28. Does systematic review non-publication bias exist? 64% survey response rate 93% survey completion rate 6.3% reported that they were never the lead on the review! Respondents reported a total of 1385 published systematic reviews median = 2 52 respondents reported a total of 200 unpublished systematic reviews median = 2 Most unpublished reviews (73%) completed after 2005

29. What is updating “A discrete event aiming to search for and identify new evidence to incorporate into a previously completed systematic review with new evidence taken to mean any evidence not included in the previously completed review irrespective of its chronological appearance in the literature” Most international organizations agree with this definition Moher D, Tsertsvadze A. When is an update an update. Lancet 2006 Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey

30. Why is updating important Keeps systematic reviews informative What happens if reviews are out-of-date Impact on clinical practice Policy recommendations Future primary research Funding agencies Most reports of systematic reviews do not appear to be updates

31. Updating systematic reviews To estimate the average time to changes in evidence sufficiently important to warrant updating systematic reviews To determine the performance characteristics of various surveillance protocols to identify important new evidence To assess the utility of rates and patterns of growth for evidence within clinical areas as predictors of updating needs To establish typical timeframes for the production and publication of systematic reviews in order to assess the extent to which they impact shelf-life Identify the updating experiences of healthcare organizations that fund and/or conduct systematic reviews Shojania K, Sampson M, Ji J, Ansari M, Garritty C, O’Rourke K, Rader, Moher D. Updating Systematic Reviews. Technical Review No. 16. (Prepared by the University of Ottawa Evidence-based Practice Center under Contract No. 290-02-0017). AHRQ Publication No 07-0087 Rockville, MD: Agency for Healthcare Research and Quality. September 2007

32. Updating systematic reviews To estimate the average time to changes in evidence sufficiently important to warrant updating systematic reviews To determine the performance characteristics of various surveillance protocols to identify important new evidence To assess the utility of rates and patterns of growth for evidence within clinical areas as predictors of updating needs To establish typical timeframes for the production and publication of systematic reviews in order to assess the extent to which they impact shelf-life Identify the updating experiences of healthcare organizations that fund and/or conduct systematic reviews Shojania K, Sampson M, Ji J, Ansari MT, Doucette S, Moher D. How quickly do systematic reviews go out of date: A survival analysis . Annals of Internal Medicine 2007;147:224-233 .

33. How quickly do systematic reviews become out of date: a survival analysis Systematic reviews evaluating benefit or harm drug, device, or procedure published between 1995 and 2005 included at least 1 meta-analysis

34. Quantitative signal to update change in statistical significance for primary outcome or any mortality outcome a relative change in effect estimate of at least 50% Qualitative signal to update substantial difference in characterization of effectiveness new information about harms superior alternative treatments important caveats about previous findings that would influence clinical decision making Signals for updating

35. Structured Search Protocols Identification of new SRs on the same topic Clinical Queries filters in PubMed Applied Related Articles function in PubMed to 3 largest and 3 most recent trials in the original review Using a ‘citing references’ search engine (Scopus) to identify new RCTs that cited original review Relevant sections of Clinical Evidence, UpToDate

36. Screened 325 reports to obtain cohort of 100 13 (median) included studies and 2663 participants 85% evaluated drug therapies Five most common clinical areas cardiovascular medicine, 20 gastroenterology, 13 neurology, 11 infectious diseases, 9 respiratory system, 9 Results

37. identified at least one new eligible trial for 85 of the reviews 4 new trials (median increase in evidence base: 25%) 1160 participants (median increase in evidence base: 47%) Results

38. A quantitative or qualitative signal occurred in 57 of the reviews median event-free survival = 5.5 years (95%, confidence interval: 4.6, 7.6) 23% of reviews needed updating within 2 years 15% within 1 year 7% already needed updating at time of publication Quantitative signal occurred in 20 of systematic reviews Qualitative signal occurred in 54 of systematic reviews Results

39. International survey Describe the updating practices and policies of agencies that sponsor or conduct systematic reviews Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey

40. International survey Stratified purposeful sampling approach Key international groups commonly involved in undertaking/funding reviews 195 organizations 48 questions Pilot tested Administered through Survey Monkey

41. 65% response rate from 26 countries 79% considered importance of updating ‘high’ to ‘very high’ 57% have formal policy although only 29% have written policy About half (51%) respondents reported that more than 50% of their systematic reviews are estimated to be out-of-date

42. Setting priorities Before embarking on setting priorities for new reviews ensure that existing reviews are kept up-to-date Encourage journals to have an explicit policy on publishing systematic review updates Centralize aspects of updating Searching the literature