Searching many databases by using a generic term gives high recall but is very imprecise. For example:
A July 23, 2009, Ovid search for “cancer.mp or neoplasms” yielded 786,978 citations.
That is too many citations to review to yield the 323 citations on the role of tamoxifen and raloxifene in cancer, the reason for the search.
The goal is to carefully construct a search to keep precision high while maximizing recall.
Balancing Precision and Recall Precision (specificity): the proportion of retrieved articles that are relevant Recall (sensitivity): the proportion of potentially relevant articles retrieved by search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
One evaluation compared the results of popular studies against others with populations and methodologies that were similar or superior.
Evaluation of 45 studies published from 1993 to 2000 in journals with high impact factors that reported positive findings and were cited more than 1,000 times.
Results were compared against other studies with the same comparators by employing populations and methodologies that were similar or superior.
Opposite/null findings or much more dramatic effects were found 31 percent of the time.
Eighty-three percent of nonrandomized studies and 23 percent of randomized controlled trials were subsequently contradicted.
Presenting Popular Studies Is Not Enough Ioannidis JP. JAMA 2005;294:218-28.
When a general topic is proposed, it is tempting to begin by extensively searching for primary literature.
Before performing the extensive literature search, it is crucial to understand the topic, devise an analytic framework, ask clearly defined key questions, and understand the scope of the review to be conducted.
Where To Begin
Although the topic area dictates the databases to be included, the following are common ones:
Cochrane Central (General)
EMBASE (General, more international in scope)
TOXNET (Adverse Events/Toxicology)
Specialized Electronic Databases Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press. Generally a minimum requirement for a thorough search of literature
Study 1 (Sassi et al., 2002): sensitivity and specificity of using MEDLINE versus nine databases
This study focused on ability to retrieve economic analyses from January to March 1997.
MEDLINE-only searches had a sensitivity of 72 percent and a specificity of 75 percent versus nine databases.
Study 2 (Betrán et al., 2005): systematic review of the prevalence of maternal mortality and morbidity from 1997 to 2002
Multiple databases were searched (MEDLINE, EMBASE, BIOSIS, LILACS).
60 percent of citations were found in more than one database.
The MEDLINE search had 20 percent of nonreplicated citations, EMBASE had 7.4 percent, and LILACS had 5.6 percent.
MEDLINE Alone Is Not Enough Sassi, F, et al. Med Care 2002;40:387-94; Betrán AP, et al. BMC Med Res Methodol 2005;5:6.
Perform a pilot search and compare to see if output matches articles already identified by the research group, key informants, and manual searches of references of these articles.
Pilot searching is usually limited to MEDLINE and then adapted to other databases.
After pilot searching, refine the search.
Searching the literature is an iterative process.
Identifying Key Articles To Refine a Pilot Search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Multiple searches within the same database may be necessary.
Efficacy and harms searches may be separate.
If the available literature is small, a single broad search is appropriate.
If not so, separate searches are prudent.
Multiple Searches Help With an Extensive Literature Base (I) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
An efficacy search may include the disease, intervention, and terms for randomized controlled trials (RCTs).
Efficacy evaluations frequently are limited to RCTs.
Fewer inherent biases and higher quality evidence
A harms search may be broader and include just the intervention and harms and/or disease.
The paucity of harms data in published RCTs frequently requires that non-RCTs be included.
Authors would likely be interested in harms regardless of the reason for the intervention.
Multiple Searches Help With an Extensive Literature Base (II)
“ OR” — makes the search broader
“ AND” — makes the search more selective
“ /” — means all subheadings are searched
When a search term is entered, you can selectively choose some subheadings (such as pharmacology for pharmacology studies).
Subheadings are generally imprecise; accepting all subheadings increases the yield.
“ adj” — cuts down on miscellaneous citations
“ Heart adj failure” would find only instances in which “heart” is next to “failure.”
“ $” — truncates a word with different endings
“ Analy$” would find words like analysis, analyses, analyze, and analyse
“ .ti” or “.ab” — searches in the title or abstract for the word of interest
Can find citations where the word you search for is not a keyword
“ .mp” or multiple posting — a text word search; does not need to be a key word
Boolean Operators Help To Balance Precision and Recall
If the specific adverse effects of interest are specified in the key questions, they can be searched for directly (e.g., rhabdomyolysis, cancer).
General harms subheadings for MEDLINE would include:
General harms subheadings for EMBASE would include:
/adverse drug reaction
Consider using a database dedicated to harms such as TOXNET (http://toxnet.nlm.nih.gov/).
Harms Searching Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Filters have been developed and validated that can help balance precision and recall.
Cochrane Collaboration http://www.cochrane-handbook.org section 6.4.11 Search Filters, National Health Service
Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/SysRev/!SSL!/WebHelp/SysRev3.htm
InterTASC Information Specialists’ Sub-Group http://www.york.ac.uk/inst/crd/intertasc/about.htm
Filters may not work in all circumstances
Filters or Hedges Are Validated Ways To Improve Precision and Recall Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
MEDLINE (Ovid) Search for Randomized Controlled Trials Using the Cochrane Highly Sensitive and Specific Search Strategy (Sensitivity and Precision Maximizing Version 2008)
1. randomized controlled trial.pt.
2. controlled clinical trial.pt.
5. clinical trials as topic.sh.
8. 1 or 2 or 3 or 4 or 5 or 6 or 7
Example Filter (Hedge) for Finding Randomized Controlled Trials Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
MEDLINE (Ovid) Search for Observational Studies Using the Scottish Intercollegiate Guidelines Network Observational Study Medline Search Filter
1. epidemiologic studies/
2. exp case control studies/
3. exp Cohort Studies/
4. case control.tw.
5. (cohort adj (study or studies)).tw.
6. cohort analy$.tw.
7. (follow up adj (study or studies)).tw.
8. (observational adj (study or studies)).tw.
11. cross sectional.tw.
12. Cross-Sectional Studies/
13. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12
Example Filter for Finding Nonrandomized Studies Coleman CI, et al. Comparative Effectiveness Review No. 18 . Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Items to consider before hitting enter:
Use of natural language terms (search as key word)
Appropriate word truncation and spelling variation
Appropriate use of limits such as language and years
Appropriate use of searching for field, publication type, and author
Peer Review of Electronic Search Strategies (PRESS) Checklist (I) Sampson M, et al. PRESS: peer review of electronic search strategies . 2008.
Items to consider before hitting enter:
Spelling errors/spelling in American and British English (e.g., anaesthesia and anesthesia, analyse and analyze, oestrogen and estrogen)
Line errors that occur when searches are combined by using line numbers (e.g., “1 AND 3” instead of “1 AND 4”)
Appropriate use of Boolean operators
Adaptation of search strategies for specific databases
Are all appropriate headings used (refer to the thesaurus for a specific database)
Appropriate use of subheadings and floating subheadings
Peer Review of Electronic Search Strategies (PRESS) Checklist (II) Sampson M, et al. PRESS: peer review of electronic search strategies . 2008.
Publication bias is the tendency of certain types of trials (such as those with the largest effects) to be published.
Publication bias increases the risk that the observed effect might not reflect the true effect.
It may negatively impact consistency, precision, and magnitude of effect.
Expanding searches to include additional languages, citation tracking, hand searching, and grey literature can help identify and possibly minimize publication bias.
Considering Publication Bias When Developing a Search Strategy Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Study (Turner et al., 2008): 74 antidepressant studies registered with the U.S. Food and Drug Administration
97 percent of positive studies published
39 percent of neutral or negative studies published
11 of 14 studies published in a way that conveyed the positive results but deemphasized the negative
When only published literature was meta-analyzed, a 32 percent increase in relative effect size occurred versus the more complete dataset of conducted trials.
Example of Publication Bias Turner EH, et al. N Engl J Med 2008;358:252-60.
It is not a requirement to search non-English–language publications.
Evidence-based Practice Centers should consider how inclusion or exclusion of non-English–language studies might or might not bias the search.
It is usually advisable to include non-English– language studies when the evidence base may be highly restricted in English-language but broader in non-English–language studies.
Such as, complementary and alternative medicine topics
Restricting Search by Language and Impact on Publication Bias Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Study 1 (Pham et al., 2008): evaluation of 42 systematic reviews, including 662 randomized controlled trials
For Western medicine, language-restricted systematic reviews did not introduce biased results when compared with language-inclusive ones, in terms of estimates of intervention effectiveness (risk odds ratio [ROR] = 1.02; 95% confidence interval [CI] = 0.83–1.26).
For complementary and alternative medicine, however, language-restricted systematic reviews resulted in a 63 percent smaller protective effect estimate when compared with language-inclusive reviews (ROR = 1.63; 95% CI = 1.03–2.60).
English Only or All Languages (I) Pham B, et al. J Clin Epidemiol 2005;58:769-76.
Study 2 (Vickers et al., 1998): evaluation of acupuncture trials conducted in England versus China, Japan, Russia, and Taiwan
Results of randomized controlled trials or controlled clinical trials in England were favorable 75 percent of the time.
Results in China, Japan, Russia, and Taiwan were favorable 99, 89, 97, and 95 percent of the time, respectively.
Researchers should carefully consider how to manage data from these countries.
English Only or All Languages (II) Vickers A, et al. Control Clin Trials 1998;19:159-66.
Forward citation tracking: databases that provide information on articles cited by other authors
This method relies on the author’s choice to cite an article, rather than key words or indexing.
Can turn up unique items and can complement a traditional database search.
Databases currently in use are Web of Science, Scopus, Google Scholar, PubReMiner, and PubFocus.
Web of Science and Scopus have access fees but have the most developed search and export interfaces.
Google Scholar, PuReMiner, and PubFocus are free access but are not as developed.
Citation Tracking (I) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Backward citation tracing: reading references of key articles
New search terms, poorly indexed journals, or inappropriate limiters in the original search can be identified
“ Related article” links: useful when designing and refining a search but not for a formal search
Difficult to perform systematically, report strategy with transparency, and reproduce the search
Citation Tracking (II) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Not all journals or sources can be indexed in databases.
Some journals index only certain article types.
Abstracts, conference proceedings, and supplements may not be indexed.
Backward citation tracking and key informant discussion can identify these journals.
Ulrich has a useful Web site to identify journals in a specific topic area (available at: http://www.ulrichsweb.com/ulrichsweb/).
If Ulrich indicates an important journal is not indexed fully in databases used, search the journal by hand.
Hand Searching To Reduce Publication Bias Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Study 1 (Hopewell et al., 2007): using MEDLINE to search 22 specialized journals to identify randomized controlled trials (RCTs) was compared with hand searching of the same journals
117 of the 462 RCTs (25%) were not retrieved by MEDLINE because of a lack of the publication-type terms “randomized controlled trial” or “controlled clinical trial.”
Additional articles were missed in journals without full journal indexing.
Study 2 (Jadad et al., 1993): using MEDLINE to search nine pain and anesthesia journals was compared with hand searching
13 percent of eligible articles were not retrieved with MEDLINE alone.
Benefits of Hand Searching Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010; Jadad AR, McQuay HJ. Online J Curr Clin Trials 1993;(33).
Grey literature: literature produced at all levels of government, academics, business, and industry that is not controlled by commercial publishers
Usually not systematically identified, stored, or indexed
Can help overcome publication bias
Least efficient type of literature to search and may be (or be perceived to be) of lower quality
Useful in areas with little published evidence, in fields that are rapidly changing (e.g., devices, surgery), or in fields that are highly interdisciplinary (e.g., mental health, alternative medicine)
Using Grey Literature To Reduce Publication Bias Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Study (Hopewell et al., 2008): reviewed five systematic reviews conducted to evaluate the impact of grey literature in meta-analyses of randomized controlled trials
All systematic reviews found more positive results with published literature than with grey literature (risk odds ratio = 1.09; 95% confidence interval = 1.03–1.16)
Grey Literature Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010 .
The approval process for new drugs and devices involves submission of data that may not be published or may be incompletely published elsewhere
Grey Literature: Online Registries Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Much grey literature is unlikely to be published.
When published, final data often differ from those in the conference abstract.
Use key informants to identify the most important meetings in the field of interest and hand search the abstracts and conference proceedings in journal supplements.
Some databases are also available: ProceedingsFirst; Biological Abstracts; Conference Papers Index; Conference Proceedings Citation Index (part of the Web of Science); and BioWizard Scholar Conference Abstract Search.
Grey Literature: Abstracts and Conference Proceedings Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Theses and dissertations are becoming increasingly available as institutions post them online.
ProQuest Dissertation and Theses
The most comprehensive collection in the United States, with full text of dissertations through their Digital Archiving and Access program
Index to Theses in Great Britain and Ireland
Networked Digital Library of Theses and Dissertations (NDLTD)
Dissertation Abstracts Online
Grey Literature: Theses and Dissertations Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Aside from governmental regulatory sites, other governmental agencies support or collect grey literature.
Computer Retrieval of Information on Scientific Projects (CRISP): http://crisp.cit.nih.gov
Searchable database of federally funded biomedical research
Health Services Research Projects in Progress (HSRPROJ): http://wwwcf.nlm.nih.gov/hsr_project/home_proj.cfm
Database of ongoing grants and projects in health services research
Collated list of sources for grey literature appropriate for health sciences by the Canadian Agency for Drugs and Technology in Health (CADTH)
System for Information on Grey Literature in Europe (SIGLE): http://opensigle.inist.fr/
Provides bibliographic references to grey literature in Europe
New York Academy of Medicine (NYAM): http://www.nyam.org/library.grey.shtml
Provides NYAM grey literature with a list of other organizations that do the same
Grey Literature: General Sources Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Researchers can be contacted to determine whether or not projects in the grey literature are published in a peer-reviewed form.
Prolific researchers in the field can be contacted to see if they know of any citations that were missed by the literature search.
Search engine (e.g., Yahoo, Google) queries with the individual’s name or institution can be a good place to start.
Contact information in manuscripts is also a reasonable place to start.
Directly Contacting Researchers Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
SIPs are provided by the company manufacturing the drug or device.
SIPs contain information about products, including the product label and published and unpublished trials or studies. They:
can help to overcome publication bias by identifying trials that remain unpublished;
can identify soon-to-be-released trials that can ultimately be captured in comparative effectiveness reviews (CERs); and
allow drug and device manufacturers to be explicitly involved in helping provide a literature base to a CER.
Scientific Information Packets (SIPs) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Scientific Information Packets Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Accurate and transparent reporting of search strategies is important.
Searches need to be updated toward the conclusion of a systematic review to ensure that no new information has come out in the intervening period.
Stakeholders and end-users want assurances that the systematic review was free of bias and that adequate search principles were followed.
Searches will need to be rerun if the systematic review is updated several years after it is completed.
Documenting the search strategy so that a skilled literature searcher can reproduce it is very important.
Reporting Search Strategies: Transparency Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Nondatabase methods used
Full electronic search strategy
End date of search
List of excluded references
Qualifications of searcher
Number of references identified
QUOROM - or PRISMA-style flow diagram accounting for all references
Evidence of search effectiveness
Statement of filters employed
Description of sampling strategy
Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
Transparent Reporting: QUOROM Figure Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Transparent Reporting: PRISMA Figure Included Screening Eligibility Identification
The next few slides present a comparison of a controlled trial search and an observational study search.
See how the search pieces are created and then come together.
Note the use of Boolean operators.
Example of Two Searches
Example of Controlled Trial Search (I) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Example of Controlled Trial Search (II) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Example of Controlled Trial Search (III) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Example of Observational Study Search (I) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Example of Observational Study Search (II) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Example of Observational Study Search (III) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Searching is important: balance precision and recall.
MEDLINE is not enough.
Multiple strategies improve completeness of search.
Filters are a good place to start.
More than one search may be needed (e.g., separately for “efficacy” and “harms”).
Report strategy with transparency.
Betran AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol 2005 ;5:6.
Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease. Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
Hopewell S, Clarke M, Lusher A, et al. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med 2002;21:1625-34.
Hopewell S, McDonald S, Clarke MJ, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;(2):MR000010.
Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005;294:218-28.
Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials 1993;(33).
Pham B, Klassen TP, Lawson ML, et al. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 2005;58:769-76.
Sampson M, McGowan J, Lefebvre C, et al. PRESS: peer review of electronic search strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.
Sassi F, Archard L, McDaid D. Searching literature databases for health care economic evaluations: how systematic can we afford to be? Med Care 2002;40:387-94.
Turner EH, Matthews AM, Linardatos E. et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60.
Vickers A, Goyal N, Harland R, et al. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials 1998;19:159-66.
This module was prepared by C. Michael White, Pharm.D., FCP, FCCP, a member of the University of Connecticut/Hartford Hospital Evidence-based Practice Center.
This module is based on chapter 5 in version 1.0 of the Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews (available at: http://effectivehealthcare.ahrq.gov/repFiles/2007_10Draft MethodsGuide.pdf).