When do systematic reviews become unsystematic? Julie Williams

2,769 views

Published on

Health Libraries Australia Professional Development Day 2012

Published in: Health & Medicine, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,769
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
56
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • This may be my quote for the day. I think I should be wearing a t-shirt (I used to believe, but no I’m not sure Talk about experiences with the Systematic Review Service What the service does Experience with the Students/Researchers
  • SR Line-up 1 Many of the examples I am using today are outside Clinical Medicine, but not all. Pause. The first in my Line-up.
  • First Review in my Line-up The breast feeding practices of the mountain tribes of Laos: a systematic review. .
  • The second in my line-up. A student from Public Health requests help developing and writing the search strategy for his SR. The only problem, when I see him.................................
  • If you are not already aware Systematic Reviews are being adopted and adapted by disciplines outside Medicine and clinical medicine. This slide shows some of the disciplines that are now producing systematic reviews I say adapted, because the Systematic Reviews presented in some disciplines only have a vague resemblance to systematic reviews from the EBM movement . Boell suggests that Systematic Reviews in Software Engineering are really just “Systematic Literature Reviews”. This appears true in other disciplines as well. Many of these disciplines don’t include any form of Meta-analysis in their reviews. Nor do they rely on a hierarchy of evidence. At the same time as they are being promoted in these disciplines, critics of systematic reviews have emerged. Criticisms include questioning the appropriateness of SR’s in disciplines relying on Qualitative studies for evidence and where SR’s outside Medicine, often accept any form of research Critics question the point of meta-analysis, where there is meta-analysis on studies that are not similar enough to be synthesised. Take International Health and International studies whose focus is Less Developed Countries. Critics argue that much of the knowledge in less developed countries isn’t published, so a systematic review isn’t going to include this knowledge. Also, not many RCt’s are undertaken in “third world countries”, and how much use is an RCT from a first world country. Yet AusAid and DFID Insist on research being undertaken in the form of systematic reviews.
  • Alongside the rise of SR in various disciplines, has been the increasing interest in SR in the Academic World where Postgraduate students are expected to produce a Systematic Review as their major research project. While the clinical medicine discipline seem to approach this reasonably well, assigning each SR topic to a team, other disciplines, including Public Health have, I believe, unrealistic expectations of their students. The students are expected to undertake their SR’s as an individual project, which I believe contradicts the idea of Systematic Reviews and is how my second example was able to do what he did.
  • Community based Disaster Risk Management Library contacted by the review team Search has caused the database to cut off UNSW Also, the search strings are too long to be pasted into Search box Review is on Community Based Disaster Risk Management Conducted by Global Health Research Centre The review is being funded by AusAid, who insist that it is a SR The actual question they are answering is How, why and when do Community Based Disaster Risk Management initiatives impact on social and economic costs of disasters? Not really a narrow focused question Quantitative review, with no Meta-analysis and no Hierarchy of Evidence
  • The problem here is the Search Strategy which is way too broad. The actual question they are answering is How, why and when do Community Based Disaster Risk Management initiatives impact on social and economic costs of disasters? Not really a narrow focused question Terms such as Capacity , empowerment , alongside economic aspect and indigenous knowledge. The team are following the Search Guidelines from the Cochrane Handbook to Use a wide variety of terms, use both free-text terms and subject headings
  • This guideline I believe can be misleading, or taken too literally as it has in this Review. I sometimes feel it is taken too literally in clinical medicine reviews as well. I see complicated searches that are OTT, and just create unnecessary work for the review team. A lot of my examples today relate to breast cancer screening, particularly mammography. The search strategies for this topic often search for Breast cancer or Breast neoplasm in addition to Mammography. In my opinion this is not necessary. Illustratates the inconsistency of search strategies (on same topic and different detail of coverage).
  • And according to the literature, one of the common pitfalls in a systematic review is the completeness of the reporting of its search strategy. If the search strategy reporting is inadequate, the review is not transparent and can’t be replicated.
  • This is a typical example of a Search Strategy Report in a journal based systematic review. This Report of a Search Strategy is from a Systematic Review that changed the United States Policy on who is recommended for Mammography.
  • Here is Yoshi’s recommended elements of the search strategy description What can make a SR unsystematic? “ Lack of detail in the reporting of the search strategy” so that it is both inadequate for critical appraisal and unable to be replicated. According to Moher, many non-Cochrane reviews do not report key aspects of their Systematic Review Methodology thus impairing confidence in their results and conclusions . An important difference between Cochrane reviews and journal based reviews are the word restriction issues imposed by Journals. Cochrane reviews, are not encumbered by word length restrictions as are many journal article based reviews. Moher. Ways around the restrictions are to refer the reader to a separate web page for more detail. However.
  • Continuing on the theme of Reporting of Search Strategies, I want to show you 2 search strategy reports in Systematic Reviews published as Journal articles, and I have to say that I believe these examples to be typical. Taking you back to my earlier quote, What makes a systematic review unsystematic is not adhering to a protocol. The search strategy reporting in this and the next example is inadequate to judge the quality and validity of the review,........nor is there enough detail to replicate the search. Both are required outcomes in a Systematic Review. The Search Strategy report informs us that the reviewers searched the ERIC Database. Thankfully they didn’t retrieve any articles from ERIC. Also note the search terms used. This was one of my two comparative searches on Psychological Outcomes of False-Positive Results in Screening Mammography.
  • I’m going to talk about this in more detail later, but in looking at some SR’s on the topic of psycholpgical consequences of False Positive mammograms . I have observed Reviews on ostensibly the same topic but with amazing variation of their search strategy and level of search strategy reporting. Inconsistency of Search Strategies in Systematic Reviews on the same topic
  • Psychologist based at a major teaching hospital. Looking to implement psychological therapy program for Breast Cancer patients. Is told she must undertake a Systematic Review before she can trial and report on her proposed therapy program.
  • I don’t really know much about systematic reviews but that’s what I’ve got to do.”
  • I love this quote. It sums up for me the feeling of how Systematic Reviews and EBM are becoming ubiquitous. Why this pressure to produce systematic reviews? Why is it so? I believe in part it is about language and the fact that EBM and Systematic Reviews have hijacked the language in their favour.
  • These terms are used to describe narrative literature reviews by the promoters of systematic reviews. Would you want to write a “non systematic” review. Should the status of the review article be based upon the fact that it is a “Systematic Review” rather than the quality of the review itself in other words, if it is a systematic review it must be good, if a narrative review, must be poor research, subjective and the author is biased. According to Dijkers Systematic Reviews have a surprising degree of subjective interpretation in summarising evidence and making recommendations(Dijkers 2009) Sandelowski points to “ a large variation in conclusions of systematic reviews on the same topic. And Systematic reviews ostensibly addressing the same research question will not include the same reports nor necessarily come to the same conclusions. So, are systematic reviews as impartial and scientific as we are led to believe.
  • The status of the review article is based upon the fact that it is a “Systematic Review” rather than the quality of the review itself in other words, if it is a systematic review it must be good, if a narrative review, must be poor research, subjective and the author is biased. According to Dijkers Systematic Reviews have a surprising degree of subjective interpretation in summarising evidence and making recommendations(Dijkers 2009)
  • This study is a journal based systematic review that has been used by the United States Preventive Services Task Force to formulate the Recommendations in the US to determine who and how often women are recommended for Breast Cancer Screening. This Review was published in the Annals of Internal Medicine. The following issue of the Annals included 11 letters commenting on the review. 9 of the letters were critical of various aspects of the review.
  • Mischaracterises the empirical literature Uses erroneous data Based on flawed studies. Don’t quote a cochrane review Don’t represent Afro-American or Latino Women . Supporting articles plagued with ambiguity. Conclusion drifts away from the published evidence. And what about the Search Strategy report for such an important study?
  • So, I think what I’m trying to say is “Don’t fall into the trap of “its a systematic review therefore it is superior research” Any research or review should be judged on its merits, not just its label.
  • If we as Librarians have skills and understanding of both EBM and Systematic Reviews I feel that we have an important role to play. We are certainly the experts when it comes to databases and searching databases. Reviewers might think they have enough knowledge, but in many cases I have seen, they do not. Also, our Systematic Review Service is seeing researchers contact us for advice. Every time I assist a systematic review team or individual I learn more about the process. We also have an important critical role as well, which I guess is partly what I am doing today. My final quote for today, “Don’t fall into the trap of “its a systematic review therefore it is superior research” Any research or review should be judged on its merits, not just its label. Thank you.
  • When do systematic reviews become unsystematic? Julie Williams

    1. 1. When Systematic Reviewsbecome unsystematic.UNSW Library Julie Williams
    2. 2. What makes a review systematic (as opposed to unsystematic) is the use of an explicit and auditable protocol for review.If what makes a review systematic is adherence to a protocol, what makes a reviewunsystematic is simply that it does not adhere to a protocol.Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.
    3. 3. My Systematic Review Line-up.
    4. 4. The breast feeding practices of the mountain tribes of Laos:a systematic review.
    5. 5. Disciplines adopting the Systematic Reviewmethodology.
    6. 6. Systematic Reviews in AcademiaIs it appropriate;d) to assign an individual the task of completing a systematic review as their first major postgraduate project?b) to assign a systematic review project to an individual, when clearly the majority of systematic reviews are completed by teams.h) Is Academia arming these researchers/students with enough knowledge and understanding of what an SR is and how to undertake such a large and involved project.
    7. 7. Cochrane Handbook guidelines on Searching“ When designing a search strategy, in order to be comprehensive it is necessary to include a widerange of free-text terms for each of the concepts selected.” “Use a wide-variety of search terms, combined with OR within each concept.”Use “both free-text and subject headings”.“Searches for systematic reviews aim to be as extensive as possible in order to ensure the asmany as possible of the necessary and relevant studies are included in the review”Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester,England ; Hoboken, NJ: Wiley-Blackwell
    8. 8. If what makes a review systematic is adherence to a protocol, what makes a reviewunsystematic is simply that it does not adhere to a protocol.Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.
    9. 9. Examples of Search Strategy Reporting in aJournal Based Systematic Review (2)Data Sources and SearchesWe searched the Cochrane Central Register of Controlled Trials and Cochrane Database of Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1 December 2008) for relevant studies and meta-analyses (16). We also conducted secondary referencing by manually reviewing reference lists of key articles and searching citations by using Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search results.Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
    10. 10. Seven recommended elements of the searchstrategy description (Yoshi 2009)
    11. 11. Examples of Search Strategy Reporting in aJournal Based Systematic Review (1)Data sources and searchesWe conducted the review by using the following protocol. Two reviewers independently searched the MEDLINE, Web of Science, EMBASE, CINAHL, PsycINFO, and ERIC databases for studies published through September 2006 whose title, abstract, or keywords included reference to both false-positive results and screening mammography.The search terms were (false positive OR abnormal OR benign) AND (breast cancer OR mammog*).We also manually searched the reference sections of relevant papers and circulated requests for unpublished studies among colleagues and the authors of the articles we identified. We limited the searches to English-language studies.Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal Medicine, 146(7), 502-510
    12. 12. Search Strategy Reporting: Same topic, differentsearch strategies and reporting.Psychological consequences of False-Positive Mammograms– The Bond et al search strategy, is over 18 pages– The Brewer et al search strategy is 2 lines.– Bond et al search 15 databases– Brewer search 6 databases including ERIC– Bond et al identified 5058 articles– Brewer identified 11,726 articles.Mary Bond, Toby Pavey, Chris Cooper, Chris Hyde, Ruth Garside. A systematic review of the psychological consequences of false-positive screening mammograms. PROSPERO 2011:CRD42011001345Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal Medicine, 146(7), 502-510
    13. 13. “I don’t really know much about systematic reviews butthat’s what I’ve got to do.”
    14. 14. If you are doing almost anything related to health care today, being “evidence based” is de rigeur.Steinberg (2005)Steinberg, E. P., & Luce, B. R. (2005). Evidence based? Caveat emptor! Health Affairs, 24(1), 80-92.
    15. 15. The Rise and Rise of Systematic Reviews Boell, Sebastian and Cezec-Kecmanovic, Dubravka, “Are systematic reviews better, less biased and of higher quality?" (2011). ECIS 2011 Proceedings. Paper 223. http://aisel.aisnet.org/ecis2 011/223
    16. 16. Systematic Review critiques– There can be large variation in conclusions of systematic reviews on the same topic. (Sandelowski 2008)– “Systematic reviews ostensibly addressing the same research question will not include the same reports nor necessarily come to the same conclusions.” (Sandelowski 2008)(Ezzo 2001) (Linde & Willich 2003)– “The outcomes of systematic reviews are as situated, partial and perspectival as any other human activity.”( Lather 1999) (Sandelowski 2008)
    17. 17. Systematic Review critiquesDijkers writes “Reviews are becoming indispensable in keeping up with an exponentially growing rehabilitation literature.Adherents of the systematic reviews that support evidence-based practice have been quite dismissive of narrative (traditional, qualitative, and non-systematic) reviews.However, the types of problems that plague the latter may also be found in systematic reviews, which, in addition, have problems of their own.”Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and Rehabilitation, 88(5), 423-430.
    18. 18. Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L.(2009).Screening for breast cancer: An update for the U.S. Preventive Services Task Force.Annals of Internal Medicine, 151(10), 727-737
    19. 19. Letters regarding Nelson et al Screening for breast cancer: An update for theU.S. Preventive Services Task Force 2009– We believe that this statement mischaracterises the empirical literature ( DeFrank & Brewer, 2010)– “unfortunately the report drifts away from the published evidence” (DeFrank & Brewer, 2010)– The USPSTF evaluation and supporting articles are plagued with ambiguity over the terms “screen” and “screening” when used alone and in conjunction with mammography.” (Dean, 2010)– Unfortunately these erroneous data from the early overview were used for both the 2002 and 2009 USPSTF evaluations. (Dean, 2010)– It is indefensible that Nelson and colleagues base their estimate of diagnosis on flawed studies when data from 600,000 randomly assigned women are available. (Jørgensen & Gøtzsche, 2010)– It is curious that Nelson and colleagues do not quote our Cochrane review as they searched the Cochrane Library.(Jørgensen & Gøtzsche, 2010)– The USPSTF study lacks evidence to support a reduction in mammography screening in African- American and Latino women” (Seewaldt, 2010)Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
    20. 20. Screening for breast cancer: An update for theU.S. Preventive Services Task ForceData Sources and SearchesWe searched the Cochrane Central Register of Controlled Trials and Cochrane Database of Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1 December 2008) for relevant studies and meta-analyses (16). We also conducted secondary referencing by manually reviewing reference lists of key articles and searching citations by using Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search results.Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
    21. 21. Roles for Librarians in Systematic ReviewsAs search, database and information experts;– Scope Searching– Assist in the formulation of Search Strategies.– Nominate suitable databases to search– Endnote Training– Provide Systematic Reviewers with an overview of currently available protocols or methodologies– Assist in the reporting of and layout of the search strategy– Provide a critiquing role on the reporting of the review (What have they included, what have they left out)– Librarian involvement in training for potential reviewers.– Being a member of the Review team– Recommending against a Systematic Review where appropriate.
    22. 22. BibliographyBoell, S. K., & Cecez-Kecmanovic, D. (2010). Literature reviews and the hermeneutic circle. Australian Academic and Research Libraries, 41(2), 129-144.Boell, S. K., & Cezec-Kecmanovic, D. (2011). Are systematic reviews better, less biased and of higher quality? Paper presented at the European Conference on Information Systems (ECIS). http://aisel.aisnet.org/ecis2011/223Dean, P. B. (2010). Comments and response on the USPSTF recommendation on screening for breast cancer [4]. Annals of Internal Medicine, 152(8), 539.DeFrank, J. T., & Brewer, N. T. (2010). The background review for the USPSTF recommendation on screening for breast cancer [2]. Annals of Internal Medicine, 152(8), 537-538.Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and Rehabilitation, 88(5), 423-430.Ezzo, J., Bausell, B., Moerman, D. E., Berman, B., & Hadhazy, V. (2001). Reviewing the reviews: How strong is the evidence? How clear are the conclusions? International Journal of Technology Assessment in Health Care, 17(4), 457-466.Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester, England ; Hoboken, NJ: Wiley-Blackwell.Jørgensen, K. J., & Gøtzsche, P. C. (2010). The background review for the USPSTF recommendation on screening for breast cancer [3]. Annals of Internal Medicine, 152(8), 538.Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33, 2004.McMichael, C., Waters, E., & Volmink, J. (2005). Evidence-based public health: What does it offer developing countries? Journal of Public Health, 27(2), 215-221.Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007). Epidemiology and Reporting Characteristics of Systematic Reviews. PLoS Med, 4(3), e78. doi: 10.1371/journal.pmed.0040078Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.Sampson, M., McGowan, J., Tetzlaff, J., Cogo, E., & Moher, D. (2008). No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology, 61(8), 748-754. doi: 10.1016/j.jclinepi.2007.10.009Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.Seewaldt, V. L. (2010). Comments and Response on the USPSTF Recommendation on Screening for Breast Cancer. Annals of Internal Medicine, 152(8), 541-542. doi: 10.1059/0003-4819-152-8-201004200-00205Yoshii, A., Plaut, D. A., McGraw, K. A., Anderson, M. J., & Wellik, K. E. (2009). Analysis of the reporting of search strategies in Cochrane systematic reviews. [Article]. Journal of the Medical Library Association, 97(1), 21-29.

    ×