This document discusses systematic reviews and provides guidance on conducting comprehensive literature searches for systematic reviews. It notes that adherence to an explicit protocol is what makes a review systematic, while lack of protocol adherence makes a review unsystematic. The document also discusses challenges with systematic reviews, such as variation in conclusions between reviews on the same topic. It provides examples of literature search strategies and reporting from published systematic reviews. Overall, the document outlines best practices for conducting thorough literature searches and reporting search methods in systematic reviews.

1. When Systematic Reviews
become unsystematic.
UNSW Library
Julie Williams

2. What makes a review systematic (as opposed to unsystematic) is the use of an explicit
and auditable protocol for review.
If what makes a review systematic is adherence to a protocol, what makes a review
unsystematic is simply that it does not adhere to a protocol.
Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.

3. My Systematic Review Line-up.

5. The breast feeding practices of the mountain tribes of Laos:
a systematic review.

7. Disciplines adopting the Systematic Review
methodology.

8. Systematic Reviews in Academia
Is it appropriate;
d) to assign an individual the task of completing a systematic review as their first major
postgraduate project?
b) to assign a systematic review project to an individual, when clearly the majority of systematic
reviews are completed by teams.
h) Is Academia arming these researchers/students with enough knowledge and understanding of
what an SR is and how to undertake such a large and involved project.

11. Cochrane Handbook guidelines on Searching
“ When designing a search strategy, in order to be comprehensive it is necessary to include a wide
range of free-text terms for each of the concepts selected.”
“Use a wide-variety of search terms, combined with OR within each concept.”
Use “both free-text and subject headings”.
“Searches for systematic reviews aim to be as extensive as possible in order to ensure the as
many as possible of the necessary and relevant studies are included in the review”
Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester,
England ; Hoboken, NJ: Wiley-Blackwell

12. If what makes a review systematic is adherence to a protocol, what makes a review
unsystematic is simply that it does not adhere to a protocol.
Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.

13. Examples of Search Strategy Reporting in a
Journal Based Systematic Review (2)
Data Sources and Searches
We searched the Cochrane Central Register of Controlled Trials and Cochrane Database of
Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1
December 2008) for relevant studies and meta-analyses (16). We also conducted secondary
referencing by manually reviewing reference lists of key articles and searching citations by using
Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search
results.
Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An
update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.

14. Seven recommended elements of the search
strategy description (Yoshi 2009)

15. Examples of Search Strategy Reporting in a
Journal Based Systematic Review (1)
Data sources and searches
We conducted the review by using the following protocol. Two reviewers independently searched the
MEDLINE, Web of Science, EMBASE, CINAHL, PsycINFO, and ERIC databases for studies
published through September 2006 whose title, abstract, or keywords included reference to both
false-positive results and screening mammography.
The search terms were (false positive OR abnormal OR benign) AND (breast cancer OR
mammog*).
We also manually searched the reference sections of relevant papers and circulated requests for
unpublished studies among colleagues and the authors of the articles we identified. We limited
the searches to English-language studies.
Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal
Medicine, 146(7), 502-510

16. Search Strategy Reporting: Same topic, different
search strategies and reporting.
Psychological consequences of False-Positive Mammograms
– The Bond et al search strategy, is over 18 pages
– The Brewer et al search strategy is 2 lines.
– Bond et al search 15 databases
– Brewer search 6 databases including ERIC
– Bond et al identified 5058 articles
– Brewer identified 11,726 articles.
Mary Bond, Toby Pavey, Chris Cooper, Chris Hyde, Ruth Garside. A systematic review of the psychological consequences of false-positive
screening mammograms. PROSPERO 2011:CRD42011001345
Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal
Medicine, 146(7), 502-510

18. “I don’t really know much about systematic reviews but
that’s what I’ve got to do.”

19. If you are doing almost anything related to health care
today, being “evidence based” is de rigeur.
Steinberg (2005)
Steinberg, E. P., & Luce, B. R. (2005). Evidence based? Caveat emptor! Health Affairs, 24(1), 80-92.

20. The Rise and Rise of Systematic Reviews
Boell, Sebastian and
Cezec-Kecmanovic,
Dubravka, “Are systematic
reviews better, less biased
and of higher quality?"
(2011). ECIS 2011
Proceedings. Paper 223.
http://aisel.aisnet.org/ecis2
011/223

23. Systematic Review critiques
– There can be large variation in conclusions of systematic reviews on the same topic.
(Sandelowski 2008)
– “Systematic reviews ostensibly addressing the same research question will not include
the same reports nor necessarily come to the same conclusions.”
(Sandelowski 2008)(Ezzo 2001) (Linde & Willich 2003)
– “The outcomes of systematic reviews are as situated, partial and perspectival as any other
human activity.”( Lather 1999) (Sandelowski 2008)

24. Systematic Review critiques
Dijkers writes “Reviews are becoming indispensable in keeping up with an exponentially growing
rehabilitation literature.
Adherents of the systematic reviews that support evidence-based practice have been quite dismissive
of narrative (traditional, qualitative, and non-systematic) reviews.
However, the types of problems that plague the latter may also be found in systematic reviews,
which, in addition, have problems of their own.”
Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and
Rehabilitation, 88(5), 423-430.

26. Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L.
(2009).
Screening for breast cancer: An update for the U.S. Preventive Services Task Force.
Annals of Internal Medicine, 151(10), 727-737

27. Letters regarding Nelson et al Screening for breast cancer: An update for the
U.S. Preventive Services Task Force 2009
– We believe that this statement mischaracterises the empirical literature (
DeFrank & Brewer, 2010)
– “unfortunately the report drifts away from the published evidence” (DeFrank & Brewer, 2010)
– The USPSTF evaluation and supporting articles are plagued with ambiguity over the terms
“screen” and “screening” when used alone and in conjunction with mammography.” (Dean, 2010)
– Unfortunately these erroneous data from the early overview were used for both the 2002 and 2009
USPSTF evaluations. (Dean, 2010)
– It is indefensible that Nelson and colleagues base their estimate of diagnosis on flawed studies
when data from 600,000 randomly assigned women are available. (Jørgensen & Gøtzsche, 2010)
– It is curious that Nelson and colleagues do not quote our Cochrane review as they searched the
Cochrane Library.(Jørgensen & Gøtzsche, 2010)
– The USPSTF study lacks evidence to support a reduction in mammography screening in African-
American and Latino women” (Seewaldt, 2010)
Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S.
Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.

28. Screening for breast cancer: An update for the
U.S. Preventive Services Task Force
Data Sources and Searches
We searched the Cochrane Central Register of Controlled Trials and Cochrane Database of
Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1
December 2008) for relevant studies and meta-analyses (16). We also conducted secondary
referencing by manually reviewing reference lists of key articles and searching citations by using
Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search
results.
Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An
update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.

29. Roles for Librarians in Systematic Reviews
As search, database and information experts;
– Scope Searching
– Assist in the formulation of Search Strategies.
– Nominate suitable databases to search
– Endnote Training
– Provide Systematic Reviewers with an overview of currently available protocols or methodologies
– Assist in the reporting of and layout of the search strategy
– Provide a critiquing role on the reporting of the review (What have they included, what have they
left out)
– Librarian involvement in training for potential reviewers.
– Being a member of the Review team
– Recommending against a Systematic Review where appropriate.

30. Bibliography
Boell, S. K., & Cecez-Kecmanovic, D. (2010). Literature reviews and the hermeneutic circle. Australian Academic and Research Libraries, 41(2),
129-144.
Boell, S. K., & Cezec-Kecmanovic, D. (2011). Are systematic reviews better, less biased and of higher quality? Paper presented at the
European Conference on Information Systems (ECIS). http://aisel.aisnet.org/ecis2011/223
Dean, P. B. (2010). Comments and response on the USPSTF recommendation on screening for breast cancer [4]. Annals of Internal Medicine,
152(8), 539.
DeFrank, J. T., & Brewer, N. T. (2010). The background review for the USPSTF recommendation on screening for breast cancer [2]. Annals of
Internal Medicine, 152(8), 537-538.
Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and
Rehabilitation, 88(5), 423-430.
Ezzo, J., Bausell, B., Moerman, D. E., Berman, B., & Hadhazy, V. (2001). Reviewing the reviews: How strong is the evidence? How clear are the
conclusions? International Journal of Technology Assessment in Health Care, 17(4), 457-466.
Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester,
England ; Hoboken, NJ: Wiley-Blackwell.
Jørgensen, K. J., & Gøtzsche, P. C. (2010). The background review for the USPSTF recommendation on screening for breast cancer [3].
Annals of Internal Medicine, 152(8), 538.
Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33, 2004.
McMichael, C., Waters, E., & Volmink, J. (2005). Evidence-based public health: What does it offer developing countries? Journal of Public
Health, 27(2), 215-221.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007). Epidemiology and Reporting Characteristics of Systematic Reviews.
PLoS Med, 4(3), e78. doi: 10.1371/journal.pmed.0040078
Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S.
Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.
Sampson, M., McGowan, J., Tetzlaff, J., Cogo, E., & Moher, D. (2008). No consensus exists on search reporting methods for systematic
reviews. Journal of Clinical Epidemiology, 61(8), 748-754. doi: 10.1016/j.jclinepi.2007.10.009
Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.
Seewaldt, V. L. (2010). Comments and Response on the USPSTF Recommendation on Screening for Breast Cancer. Annals of Internal
Medicine, 152(8), 541-542. doi: 10.1059/0003-4819-152-8-201004200-00205
Yoshii, A., Plaut, D. A., McGraw, K. A., Anderson, M. J., & Wellik, K. E. (2009). Analysis of the reporting of search strategies in Cochrane
systematic reviews. [Article]. Journal of the Medical Library Association, 97(1), 21-29.