Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

When do systematic reviews become unsystematic? Julie Williams

3,080 views

Published on

Health Libraries Australia Professional Development Day 2012

Published in: Health & Medicine, Technology
  • Dating for everyone is here: ♥♥♥ http://bit.ly/39pMlLF ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating direct: ♥♥♥ http://bit.ly/39pMlLF ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

When do systematic reviews become unsystematic? Julie Williams

  1. 1. When Systematic Reviewsbecome unsystematic.UNSW Library Julie Williams
  2. 2. What makes a review systematic (as opposed to unsystematic) is the use of an explicit and auditable protocol for review.If what makes a review systematic is adherence to a protocol, what makes a reviewunsystematic is simply that it does not adhere to a protocol.Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.
  3. 3. My Systematic Review Line-up.
  4. 4. The breast feeding practices of the mountain tribes of Laos:a systematic review.
  5. 5. Disciplines adopting the Systematic Reviewmethodology.
  6. 6. Systematic Reviews in AcademiaIs it appropriate;d) to assign an individual the task of completing a systematic review as their first major postgraduate project?b) to assign a systematic review project to an individual, when clearly the majority of systematic reviews are completed by teams.h) Is Academia arming these researchers/students with enough knowledge and understanding of what an SR is and how to undertake such a large and involved project.
  7. 7. Cochrane Handbook guidelines on Searching“ When designing a search strategy, in order to be comprehensive it is necessary to include a widerange of free-text terms for each of the concepts selected.” “Use a wide-variety of search terms, combined with OR within each concept.”Use “both free-text and subject headings”.“Searches for systematic reviews aim to be as extensive as possible in order to ensure the asmany as possible of the necessary and relevant studies are included in the review”Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester,England ; Hoboken, NJ: Wiley-Blackwell
  8. 8. If what makes a review systematic is adherence to a protocol, what makes a reviewunsystematic is simply that it does not adhere to a protocol.Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.
  9. 9. Examples of Search Strategy Reporting in aJournal Based Systematic Review (2)Data Sources and SearchesWe searched the Cochrane Central Register of Controlled Trials and Cochrane Database of Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1 December 2008) for relevant studies and meta-analyses (16). We also conducted secondary referencing by manually reviewing reference lists of key articles and searching citations by using Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search results.Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
  10. 10. Seven recommended elements of the searchstrategy description (Yoshi 2009)
  11. 11. Examples of Search Strategy Reporting in aJournal Based Systematic Review (1)Data sources and searchesWe conducted the review by using the following protocol. Two reviewers independently searched the MEDLINE, Web of Science, EMBASE, CINAHL, PsycINFO, and ERIC databases for studies published through September 2006 whose title, abstract, or keywords included reference to both false-positive results and screening mammography.The search terms were (false positive OR abnormal OR benign) AND (breast cancer OR mammog*).We also manually searched the reference sections of relevant papers and circulated requests for unpublished studies among colleagues and the authors of the articles we identified. We limited the searches to English-language studies.Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal Medicine, 146(7), 502-510
  12. 12. Search Strategy Reporting: Same topic, differentsearch strategies and reporting.Psychological consequences of False-Positive Mammograms– The Bond et al search strategy, is over 18 pages– The Brewer et al search strategy is 2 lines.– Bond et al search 15 databases– Brewer search 6 databases including ERIC– Bond et al identified 5058 articles– Brewer identified 11,726 articles.Mary Bond, Toby Pavey, Chris Cooper, Chris Hyde, Ruth Garside. A systematic review of the psychological consequences of false-positive screening mammograms. PROSPERO 2011:CRD42011001345Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal Medicine, 146(7), 502-510
  13. 13. “I don’t really know much about systematic reviews butthat’s what I’ve got to do.”
  14. 14. If you are doing almost anything related to health care today, being “evidence based” is de rigeur.Steinberg (2005)Steinberg, E. P., & Luce, B. R. (2005). Evidence based? Caveat emptor! Health Affairs, 24(1), 80-92.
  15. 15. The Rise and Rise of Systematic Reviews Boell, Sebastian and Cezec-Kecmanovic, Dubravka, “Are systematic reviews better, less biased and of higher quality?" (2011). ECIS 2011 Proceedings. Paper 223. http://aisel.aisnet.org/ecis2 011/223
  16. 16. Systematic Review critiques– There can be large variation in conclusions of systematic reviews on the same topic. (Sandelowski 2008)– “Systematic reviews ostensibly addressing the same research question will not include the same reports nor necessarily come to the same conclusions.” (Sandelowski 2008)(Ezzo 2001) (Linde & Willich 2003)– “The outcomes of systematic reviews are as situated, partial and perspectival as any other human activity.”( Lather 1999) (Sandelowski 2008)
  17. 17. Systematic Review critiquesDijkers writes “Reviews are becoming indispensable in keeping up with an exponentially growing rehabilitation literature.Adherents of the systematic reviews that support evidence-based practice have been quite dismissive of narrative (traditional, qualitative, and non-systematic) reviews.However, the types of problems that plague the latter may also be found in systematic reviews, which, in addition, have problems of their own.”Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and Rehabilitation, 88(5), 423-430.
  18. 18. Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L.(2009).Screening for breast cancer: An update for the U.S. Preventive Services Task Force.Annals of Internal Medicine, 151(10), 727-737
  19. 19. Letters regarding Nelson et al Screening for breast cancer: An update for theU.S. Preventive Services Task Force 2009– We believe that this statement mischaracterises the empirical literature ( DeFrank & Brewer, 2010)– “unfortunately the report drifts away from the published evidence” (DeFrank & Brewer, 2010)– The USPSTF evaluation and supporting articles are plagued with ambiguity over the terms “screen” and “screening” when used alone and in conjunction with mammography.” (Dean, 2010)– Unfortunately these erroneous data from the early overview were used for both the 2002 and 2009 USPSTF evaluations. (Dean, 2010)– It is indefensible that Nelson and colleagues base their estimate of diagnosis on flawed studies when data from 600,000 randomly assigned women are available. (Jørgensen & Gøtzsche, 2010)– It is curious that Nelson and colleagues do not quote our Cochrane review as they searched the Cochrane Library.(Jørgensen & Gøtzsche, 2010)– The USPSTF study lacks evidence to support a reduction in mammography screening in African- American and Latino women” (Seewaldt, 2010)Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
  20. 20. Screening for breast cancer: An update for theU.S. Preventive Services Task ForceData Sources and SearchesWe searched the Cochrane Central Register of Controlled Trials and Cochrane Database of Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1 December 2008) for relevant studies and meta-analyses (16). We also conducted secondary referencing by manually reviewing reference lists of key articles and searching citations by using Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search results.Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
  21. 21. Roles for Librarians in Systematic ReviewsAs search, database and information experts;– Scope Searching– Assist in the formulation of Search Strategies.– Nominate suitable databases to search– Endnote Training– Provide Systematic Reviewers with an overview of currently available protocols or methodologies– Assist in the reporting of and layout of the search strategy– Provide a critiquing role on the reporting of the review (What have they included, what have they left out)– Librarian involvement in training for potential reviewers.– Being a member of the Review team– Recommending against a Systematic Review where appropriate.
  22. 22. BibliographyBoell, S. K., & Cecez-Kecmanovic, D. (2010). Literature reviews and the hermeneutic circle. Australian Academic and Research Libraries, 41(2), 129-144.Boell, S. K., & Cezec-Kecmanovic, D. (2011). Are systematic reviews better, less biased and of higher quality? Paper presented at the European Conference on Information Systems (ECIS). http://aisel.aisnet.org/ecis2011/223Dean, P. B. (2010). Comments and response on the USPSTF recommendation on screening for breast cancer [4]. Annals of Internal Medicine, 152(8), 539.DeFrank, J. T., & Brewer, N. T. (2010). The background review for the USPSTF recommendation on screening for breast cancer [2]. Annals of Internal Medicine, 152(8), 537-538.Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and Rehabilitation, 88(5), 423-430.Ezzo, J., Bausell, B., Moerman, D. E., Berman, B., & Hadhazy, V. (2001). Reviewing the reviews: How strong is the evidence? How clear are the conclusions? International Journal of Technology Assessment in Health Care, 17(4), 457-466.Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester, England ; Hoboken, NJ: Wiley-Blackwell.Jørgensen, K. J., & Gøtzsche, P. C. (2010). The background review for the USPSTF recommendation on screening for breast cancer [3]. Annals of Internal Medicine, 152(8), 538.Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33, 2004.McMichael, C., Waters, E., & Volmink, J. (2005). Evidence-based public health: What does it offer developing countries? Journal of Public Health, 27(2), 215-221.Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007). Epidemiology and Reporting Characteristics of Systematic Reviews. PLoS Med, 4(3), e78. doi: 10.1371/journal.pmed.0040078Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.Sampson, M., McGowan, J., Tetzlaff, J., Cogo, E., & Moher, D. (2008). No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology, 61(8), 748-754. doi: 10.1016/j.jclinepi.2007.10.009Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.Seewaldt, V. L. (2010). Comments and Response on the USPSTF Recommendation on Screening for Breast Cancer. Annals of Internal Medicine, 152(8), 541-542. doi: 10.1059/0003-4819-152-8-201004200-00205Yoshii, A., Plaut, D. A., McGraw, K. A., Anderson, M. J., & Wellik, K. E. (2009). Analysis of the reporting of search strategies in Cochrane systematic reviews. [Article]. Journal of the Medical Library Association, 97(1), 21-29.

×