Searching for Relevant Studies

  • 727 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
727
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
21
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Searching for Relevant Studies
  • Systematic Review Process Overview
  • Learning Objectives At the conclusion of this module, the successful learner will be able to substantiate why prudent literature searching is important, describe important tenets of literature searching, and describe why transparency in the search process is important.
  • Balancing Precision and Recall Systematic reviews require comprehensive searches. Searching many databases by using a very generic term gives high recall but is very imprecise. For example, searching for “cancer.mp or neoplasms” yields 786,978 citations — far too many citations to review to yield the 323 citations that mention the role of tamoxifen and raloxifene in cancer, the reason for the search. The goal is to carefully construct a search in the attempt to keep precision high while maximizing recall. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Presenting Popular Studies Is Not Enough An important aspect of a systematic review is the conduct of a comprehensive search. Simply considering the results of popular studies is insufficient. One evaluation compared the results of popular studies against others with populations and methodologies that were similar or superior. Investigators assessed 45 studies published from 1993 to 2000 in journals with high impact factors that reported positive findings and were cited more than 1,000 times. The results of these studies were compared against other studies with the same comparators by employing populations that were the same size or larger or methodologies that were similar or superior. Approximately one-third of the time, opposite findings, null findings, or much more dramatic effects were seen. Overall, 83 percent of popular nonrandomized studies and 23 percent of randomized control trials were subsequently contradicted by other studies or trials. Reference: Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005;294:218-28. http://www.ncbi.nlm.nih.gov/pubmed/16014596
  • Where To Begin Where should a systematic reviewer begin? When a general topic is proposed, it is tempting to begin by extensively searching for primary literature. Here, primary literature refers to new discovery: observational studies, controlled trials, and meta-analyses. However, this will lead to suboptimal search results. Before performing the extensive literature search, it is crucial to understand the topic, devise an analytic framework, ask clearly defined key questions, and understand the scope of the review to be conducted. Even when consulting with a medical librarian, the systematic reviewer needs to understand these things. Refer to the Topic Refinement and Analytic Frameworks modules to learn more about these processes. Topic refinement and analytic frameworks are discussed in detail in separate modules.
  • Specialized Electronic Databases For most topics, a general database will be valuable. MEDLINE , Cochrane Central , and EMBASE can be searched. Searching MEDLINE and Cochrane Central is generally considered a minimum requirement for a thorough search of the literature. EMBASE is more international in scope and can be useful, especially for topics that may have been evaluated extensively outside of the United States. There are also many specialty databases, including PsychLit/PsychINFO for psychology, AIDSLine for HIV or AIDS, CINAHL for nursing, and TOXNET for adverse events and toxicology. These databases may be of value if the topic under consideration is focused in these areas. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • MEDLINE Alone Is Not Enough Here we present the findings of two studies that demonstrate why a comprehensive search must include databases other than MEDLINE. In the first study, the sensitivity and specificity of using MEDLINE versus nine databases was evaluated. The investigators were interested in the ability to retrieve economic analyses from January to March 1997. MEDLINE-only searches had a sensitivity of 72% and a specificity of 75% when compared to the use of nine databases. In the second study, evaluating the prevalence of maternal mortality and morbidity from 1997 to 2002, multiple databases were searched (MEDLINE, EMBASE, BIOSIS, LILACS). Sixty percent of the citations were found in more than one database. Searching MEDLINE provided 20 percent of nonreplicated citations, followed by EMBASE with 7.4 percent of nonreplicated citations and LILACS with 5.6 percent of nonreplicated citations. Thus, the use of MEDLINE alone does not optimally capture important literature. References: Sassi F, Archard L, McDaid D. Searching literature databases for health care economic evaluations: how systematic can we afford to be? Med Care 2002;40:387-94. http://www.ncbi.nlm.nih.gov/pubmed/11961473 Betrán AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol 2005;5:6. http://www.ncbi.nlm.nih.gov/pubmed/15679886
  • Identifying Key Articles To Refine a Pilot Search At this point, the systematic reviewer understands the topic area, has an analytic framework, and appreciates the scope of the project. Now a pilot search should be conducted. First, determine whether or not the articles already identified to date are found in the pilot search. If not, then look for reasons why: Are the citations not included in MEDLINE or are the ways the terms are defined in that trial different in some way? If so, refine the search and try it again. In general, the pilot search is limited to MEDLINE and then adapted to other databases. The search is an iterative process. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Multiple Searches Help With an Extensive Literature Base Once you decide which databases to use, you then need to decide whether multiple searches within each database are necessary. If the available literature is small, a single broad search is appropriate. However, if the literature is extensive, a search for efficacy and a separate search for harms may be advisable. Just performing a very broad general search for efficacy and harms would reduce precision without much improvement in recall. Reviewers should also consider if the efficacy studies provide effectiveness information – evidence that is applicable to end-users. Refer to the modules on assessing applicability and when to use observational studies for more information. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Multiple Searches Help With an Extensive Literature Base (II) In comparative effectiveness reviews, an efficacy search may be limited to randomized controlled trials (RCTs) if the body of evidence is sufficient to answer key questions. That is because RCTs have fewer inherent biases and greater internal validity. Searches would include the disease, intervention, and terms for RCTs. In contrast, harms searches usually include both RCTs and observational trials. RCTs tend not to fully evaluate or present harms data, and observational trials are needed. In a harms search, including just the intervention and harms may be permissible. This is because the authors may be interested in harms regardless of the reason for the intervention. However, it is also sufficient to include the intervention, harms, and the disease of interest. Reviewers should also consider if the efficacy studies provide effectiveness information – evidence that is applicable to end-users. Refer to the modules on assessing applicability and when to use observational studies for more information.
  • Boolean Operators Help To Balance Precision and Recall When conducting a search, Boolean operators are very important in the quest for balancing precision and recall. The term “OR” makes the search broader, whereas “AND” makes the search more selective. Searching for studies of “men OR women” would yield studies with men alone, women alone, or both men and women, whereas searching for “men AND women” would only yield citations with both men and women. The operator “/” searches for all subheadings. Because subheadings are generally imprecise, accepting all subheadings prevents a drop in recall. “ Adj” is a very important and underutilized operator. For example, searching for “heart adj failure” would find citations only where the term “heart” appears next to “failure” and would eliminate citations where “heart” appears in the first sentence and “failure” appears in the seventh sentence. The operator “$” is also underutilized. It is used to truncate a word, thus allowing words that end in different ways to be found. For example, using the term “analy$” would find words like “analysis,” “analyses,” “analyze,” and “analyse.” Do not truncate too early; “anal$” will pick up all of these terms plus terms related to anal, anal retentive, anal fistula, and all the other terms would now be in your search results, thus reducing precision. The operators “.ti” and “.ab” will search not only the key words but also the title and the abstract, respectively. The operator “.mp” stands for multiple posting and it is a text-word search that will show up even when the search term is not a key word, even when searching for authors names. To maximize recall, the search may need to be expanded beyond simple key words. Some of these terms are vendor specific. For example, “adj” and “$” are specific for the Ovid subscription service for MEDLINE.
  • Harms Searching To make informed health care decisions, the balance of benefits to harms needs to be determined. There are many terms for harms in MEDLINE that are characteristically different than the terms in EMBASE. The “/” operator is used to accept all subheadings. A medical librarian can help adapt a MEDLINE search to other databases, or you can use the database thesaurus. In addition to searching broadly in MEDLINE and other general databases for harms, consider using a database specifically dedicated to harms, such as TOXNET (http://toxnet.nlm.nih.gov/). Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Filters or Hedges Are Validated Ways To Improve Precision and Recall Filters, which are sometimes called hedges, are validated search strategies aimed at certain aspects of searches and can help balance precision and recall. To give an analogy, building a de novo search from scratch is represented by the use of all the ingredients that are needed to make a pizza, starting with flour, oil, salt, tomatoes, basil, oregano, onions, and finally pepperoni. If important ingredients are forgotten, such as the tomatoes for the sauce, the pizza will not be very tasty. What if jarred sauce and premade pizza dough could be used? That is what filters are like — using premade, prepackaged search strategies for different parts of a thorough search. This slide lists four places where these filters can be found. The most prominent filters are the ones created by the Cochrane Collaborative. Now because an off-the-rack strategy is being used instead of a custom-built strategy, filters may not work in all circumstances. In general, they can improve precision. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Example Filter (Hedge) for Finding Randomized Controlled Trials This slide provides an example of a search for randomized controlled trials (RCTs). We selected the MEDLINE (Ovid) filter for RCTs by using the Cochrane Highly Sensitive and Specific Search Strategy. This search strategy is meant to balance recall and precision while trying to identify RCTs. Not all RCTs are listed as such in MEDLINE, so we start out by searching for study type terms, denoted by the “.pt” operator. First, we searched for “randomized controlled trial.pt” and “controlled clinical trial.pt.” In steps 3 and 4, we searched the abstract for the terms “randomized” and “placebo” with the “.ab” designation to allow MEDLINE to search the abstract for these words. Then we searched the subheading of clinical trials by using the term “clinical trials as topic.sh.” In step 6, we searched for the word “randomly” in the abstract, and in step 7 we searched for the word “trial” in the title by denoting “.ti” at the end of the word. Because each of these terms are important ways to identify RCTs, we want to use them all. We did this in step 8 by using the operator “OR” between each step, 1 through 7. Remember, if we used the “AND” operator, a single citation would have to have all of these terms in order to be included, and the recall would suffer dramatically. Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Example Filter for Finding Nonrandomized Studies In this example, we searched MEDLINE (Ovid) for Observational Studies by using the Scottish Intercollegiate Guidelines Network Observational Study Medline Search Filter (http://www.sign.ac.uk/methodology/filters.html). Some of the Boolean operators were used in this filter, and others were not. First, “.tw” searches for the term in the title and abstract, “adj” is adjacent, “/” accepts all subheadings, “$” truncates the word so that any ending after the “$” will be captured, and “exp” is an exploded term. As we read through the search strategy, notice that the search is of general terms for observational trials by using “epidemiology studies/” and then using each individual study type considered to be an observational, case-control, cohort, and cross-sectional trial. Finally, terms used to describe observational trials such as “retrospective,” “longitudinal,” and “follow up adj study or studies” are used. In step 13, all of these terms are combined with the “OR” operator to allow any citation containing any one of these terms to be identified. Professional searchers list all the search terms to be combined sequentially and then combine them at the end. This strategy is superior to combining two terms at a time, where the chance for human error is increased. Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Peer Review of Electronic Search Strategies (PRESS) Checklist (I) Also use natural language terms. Index terms are great, but there is a lag time for indexing, inappropriate indexing, lack of adequate indexing terms, and changes in terms over time, which are not retrospectively updated. Make sure that the truncated word is truncated in a place where it will retrieve the citations of interest but will not dramatically reduce precision. Be sure that appropriate limiters, such as languages and years, are used. Finally, be sure that field searches and searches for publication type and author are appropriate. Reference: Sampson M, McGowan J, Lefebvre C, et al. PRESS: peer review of electronic search strategies . Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.
  • Peer Review of Electronic Search Strategies (PRESS) Checklist (II) Make sure that there are no spelling errors and that both American and English versions of words have been included. Otherwise, the search will be either Americentric or Eurocentric. Watch out for line errors, such as saying lines 1 and 3 are to be combined when in reality lines 1 and 4 are to be combined. Make sure the Boolean operators are used correctly and adapt the search for specific databases. Make sure all of the appropriate headings have been used; either confer with a medical librarian or use the thesaurus for the database to be sure. Finally, use appropriate subheadings and floating subheadings. Reference: Sampson M, McGowan J, Lefebvre C, et al. PRESS: peer review of electronic search strategies . Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.
  • Considering Publication Bias When Developing a Search Strategy Publication bias is the tendency of certain types of trials (such as those with the largest effects) to be published. Publication bias increases the risk that the observed effect might not reflect the true effect. This can negatively impact consistency, precision, and magnitude of effect in a systematic review. There are some things that can be done to minimize the risk of publication bias, such as including additional languages, citation tracking, hand searching, and grey literature. We will deal with each of these techniques separately. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Example of Publication Bias In one example of publication bias, of the 74 studies of antidepressants registered with the U.S. Food and Drug Administration, 97% of the positive studies were published, but only 39% of the other studies were published, and most of them were published in a way that emphasized positive results. If you relied only on the published results, there would be a 32% increase in the relative effect size versus a more complete dataset of conducted trials. Reference: Turner EH, Matthews AM, Linardatos E, et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60. http://www.ncbi.nlm.nih.gov/pubmed/18199864
  • Restricting Search by Language and Impact on Publication Bias There are benefits and risks associated with the use of non-English – language publications. Evidence-based Practice Centers should consider how inclusion or exclusion of non-English – language studies might or might not bias the search. It is usually advisable to include non-English – language studies when the evidence base may be highly restricted in the English-language studies but broader in non-English – language studies, such as complementary and alternative medicine topics. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • English Only or All Languages (I) This study evaluated 42 systematic reviews comprising 662 RCTs. For western medicine, language restriction did not introduce biased estimates of intervention effectiveness. However, when complementary and alternative medicine was evaluated, the protective effect estimate was 63% smaller than when language-inclusive reviews were evaluated. Reference: Pham B, Klassen TP, Lawson ML, et al. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 2005;58:769-76. http://www.ncbi.nlm.nih.gov/pubmed/16086467
  • English Only or All Languages (II) One reason the use of non-English – language citations is controversial is that the risk of publication bias in non-English – language journals might be accentuated. For example, randomized controlled trials of acupuncture conducted in England were compared with those conducted in China, Japan, Russia, and Taiwan. In England, acupuncture was found to be effective 75 percent of the time when compared with between 89 and 99 percent of the time in the other countries. There are two possible explanations: either non-English – language publications are more prone to publication bias, or people in England do not really know how to do acupuncture. Reference: Vickers A, Goyal N, Harland R, et al. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials 1998;19:159-66. http://www.ncbi.nlm.nih.gov/pubmed/9551280
  • Citation Tracking (I) Citation tracking is less controversial than expanding searches to all languages for reducing publication bias. In forward citation analysis, databases are used that rely on articles cited by others instead of using index terms. As such, it relies on the author’s choice to cite an article and relate it to a topic area rather than an index term or key word. In this way, it can complement a traditional database search. Web of Science, Scopus, Google Scholar, PubReMiner, and PubFocus all utilize forward citation tracking. Of these, Web of Science and Scopus are the most developed, but both have access fees. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Citation Tracking (II) The other type of citation tracking is backward citation tracking. This is simply the use of references of key articles to identify citations that were not captured in a search. By locating these missing citations, they can be examined to determine why they did not appear in a search. Perhaps this can lead to new search terms or help identify inappropriate limiters in the original search. This underscores the fact that searching is an iterative process. However, journals that are not indexed in the chosen databases or are incompletely indexed can be identified. In some journals, only some of the article types are indexed. If important content is missed by this exclusion, then hand searching of that journal may be needed. In some databases, there is a “related article” link. This link is useful when designing and piloting a search but not for conducting the formal search. That is because it is difficult to perform systematically, report with transparency, and reproduce the search in the future. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Hand Searching To Reduce Publication Bias Even when multiple databases are searched, not all journals of interest may be indexed. Some journals index only certain article types. Similarly, abstracts, conference proceedings, and supplements may not be indexed. These journals should be hand searched. Backward citation tracking and key informant discussions can help identify journals that miss this content. Ulrich also has a useful Web site to identify journals in a specific topic area. If Ulrich indicates an important journal is not indexed fully in a database, search that journal by hand. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Benefits of Hand Searching The benefits of hand searching were explored in two studies. In the first, MEDLINE was used to search 22 specialized journals to identify Randomized Controlled Trials (RCTs), and then the journals were hand searched. Twenty-five percent of the articles retrieved by hand searching were not identified by MEDLINE. This was mainly because of a lack of the publication-type terms “randomized controlled trial” or “controlled clinical trial” being included in MEDLINE and the loss of citation in journals without full journal indexing. It is possible that the use of a filter for identifying RCTs could have reduced the number of lost citations as well. In the second study, MEDLINE was used to search nine pain and anesthesia journals and then those journals were hand searched. In this study, 13 percent of eligible articles were not retrieved with MEDLINE alone. References: Hopewell S, McDonald S, Clarke MJ, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev . 2007;(2):MR000010. http://www.ncbi.nlm.nih.gov/pubmed/17443631 Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials 1993;(33). http://www.ncbi.nlm.nih.gov/pubmed/8306000
  • Using Grey Literature To Reduce Publication Bias Grey literature is produced at all levels of government, academics, business, and industry and is not controlled by commercial publishers. It is usually not systematically identified, stored, or indexed. The use of grey literature can help overcome publication bias, that is, the tendency of authors, journals, or reviewers to preferentially publish positive studies. However, it is the least efficient type of literature to search and may be (or be perceived to be) of lower quality. Grey literature is useful in areas with little published evidence, in fields that are rapidly changing (e.g., devices, surgery), or in fields that are highly interdisciplinary (e.g., mental health, alternative medicine). Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Grey Literature In one study, five systematic reviews were evaluated to determine the impact of grey literature in meta-analyses of randomized controlled trials. All systematic reviews found more positive results with published literature than with grey literature. Reference: Hopewell S, McDonald S, Clarke MJ, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev . 2007; (2):MR000010. http://www.ncbi.nlm.nih.gov/pubmed/17443631
  • Grey Literature: Regulatory Search Although there are risks and benefits associated with using grey literature, the use of regulatory sites is an important way to help address publication bias. That is because the approval process for new drugs and devices requires submission of data that may not be published or may be incompletely published. This can be important for determining efficacy and harms. This slide provides Web sites for United States, Canadian, and European regulatory bodies. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Grey Literature: Online Registries Online registries are becoming another very important source for grey literature. In order to publish results of clinical trials in premier journals, the methods have to be submitted to a registry beforehand. Thus, if trials are proposed that meet inclusion criteria but are never published, or if they are published but do not report all the end points that were intended to be collected in the methods, the authors can be contacted for the information. This slide lists the most prominent Web sites for such registries. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Grey Literature: Abstracts and Conference Proceedings Evidence-based Practice Centers should determine the risks and benefits of using abstracts and conference proceedings. Reviewing abstracts and conference proceedings can identify studies that may meet inclusion criteria for a systematic review and can reduce publication bias. However, much of this literature is unlikely to be published, and the final results may differ from the data included in the abstract. Key informants can help identify the most important meetings in the field of interest and then the abstract and conference proceedings in journal supplements can be hand searched. This slide lists some databases that are available to identify abstracts. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Grey Literature: Theses and Dissertations Theses and dissertations are also grey literature sources. There are several databases available that catalog them, but the most comprehensive collection is the ProQuest Dissertation and Theses database. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Grey Literature Aside from governmental regulatory sites, other governmental agencies either support or collect grey literature. This slide lists four databases that can be searched. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Grey Literature: General Sources This slide lists three additional general Web sites for grey literature. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Directly Contacting Researchers When a study is identified in a grey literature search, researchers can be contacted to determine whether or not the study is published in a peer-reviewed form. Prolific researchers in a field can be contacted to see if they know of any citations that were missed with a traditional search. Search engines and the contact information in manuscripts can be reasonable places to start. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Scientific Information Packets (SIPs) Scientific information packets are provided by the company that manufactures the drug or device. They include information about products, including the product label and published and unpublished trials or studies. There are three advantages to using scientific information packets: They can help to overcome publication bias by identifying trials that remain unpublished; they can identify soon-to-be-released trials that can be captured in comparative effectiveness reviews (CERs); and they allow drug and device manufacturers to be explicitly involved in helping provide a literature base to a CER. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Scientific Information Packets This flow chart shows the steps in developing scientific information packets. CER = comparative effectiveness review EPC = Evidence-based Practice Center FDA = U.S. Food and Drug Administration SRC = Scientific Resource Center Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Reporting Search Strategies: Transparency The Effective Health Care Program is predicated on transparency. Accurate and transparent reporting of search strategies is important. Searches need to be updated toward the conclusion of a systematic review to assure that no new information has come out in the intervening period. Stakeholders and end-users want assurances that the systematic review was free of bias and that adequate search principles were followed. Searches will need to be rerun if the systematic review is updated several years after it is completed. Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Transparent Reporting This slide presents the important information to be included in order to maximize transparency: databases used, dates covered, search terms, language restrictions, nondatabase methods used, inclusion/exclusion criteria, full electronic search strategy, publication-related restrictions, end date of search, list of excluded references, qualifications of searcher, number of references identified, a QUOROM- or PRISMA-style flow diagram accounting for all references, evidence of search effectiveness, statement of filters employed, and description of sampling strategy. QUOROM = Quality of Reporting of Meta-analyses PRISMA = Preferred Reporting Items for Systemic Reviews and Meta-analyses Reference: Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
  • Transparent Reporting: QUOROM Figure The QUOROM figure illustrates transparent reporting of citation dispensation. The numbers of citations identified, the number ultimately included, and the reasons for exclusion of citations are accounted for. Although the QUOROM figure represents a great step forward in reporting, it has been improved upon with the PRISMA figure, which we will discuss next. ACE = angiotensin-converting enzyme ARB = angiotensin-receptor blocker IHD = ischemic heart disease QUOROM = Quality of Reporting of Meta-analyses RCT = randomized controlled trial SMT = somatostatin Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Transparent Reporting: PRISMA Figure The PRISMA figure is even more transparent than the QUOROM figure. In QUOROM, additional studies identified through means other than databases were added toward the end. In PRISMA, they are accounted for at the beginning in the identification of literature stage, thus making the dispensation of all identified citations in the screening, eligibility, and included stages more clear. Another difference is that in PRISMA under the included stage, studies that are included in qualitative synthesis and those included in quantitative synthesis are accounted for separately. CF = cystic fibrosis QUOROM = Quality of Reporting of Meta-analyses RCT = randomized controlled trial rhGH = recombinant human growth hormone
  • Example of Two Searches See how the search pieces are created and then come together in a broad search. Note the use of Boolean operators in the search.
  • Example of Controlled Trial Search (I) This slide presents an example of a controlled trial search of MEDLINE (Ovid) by using the Cochrane Highly Sensitive and Specific Search Strategy (Sensitivity and Precision Maximizing Version 2008). We begin with nine terms for vascular disease with a slash to denote that all subheadings were selected. The nine terms are: Coronary Artery Disease/ or Coronary Disease/ Myocardial Ischemia/ Angina Pectoris/ or Angina, Unstable/ Angina Pectoris/ or Arterial Occlusive Diseases/ Peripheral Vascular Diseases/ Vascular Diseases/ Atherosclerosis/ Cardiovascular Diseases/ Carotid Artery Diseases/ Then we use a variety of terms for preserved or stable disease using “adj” to denote that a word has to be adjacent to the word next to it: 10. (((preserved adj left) or (stable adj cad) or (stable adj chd) or (stable adj coronary) or (preserved adj coronary) or (preserved adj systolic) or (preserved adj ventricular) or (preserved adj lvef) or (preserved adj ef) or (preserved adj ejection)) or (intact adj left) or (intact adj systolic) or (intact adj ventricular) or (intact adj lvef) or (intact adj ef) or (normal adj systolic) or (normal adj ventricular) or (normal adj lvef) or (normal adj ef)).mp 11. In line 11 we combine lines 1 through 10 with “or” operators. Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Example of Controlled Trial Search (II) In lines 12 through 18, we use terms for controlled trials: 12. randomized controlled trial.pt. 13. controlled clinical trial.pt. 14. randomized.ab. 15. placebo.ab. 16. clinical trials as topic.sh. 17. randomly.ab. 18. trial.ti. Remember that .pt searches for study types, .ti searches for titles, and .ab searches for abstracts. In step 19, we combine all the terms for controlled trials with the “or” operator. Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Example of Controlled Trial Search (III) In line 20, we limit the search to humans as a subheading by using “.sh” and combine line 20 with line 19 so that only controlled human trials are identified. In line 22, all of the angiotensin-converting enzyme (ACE) inhibitors of interest are identified and searched as free text using “.mp.” In line 23, all of the angiotensin-receptor blockers (ARBs) are identified and searched as free text. In lines 24 and 25, the drug classes of ACE inhibitors and angiotensin II type-1 receptor blockers are searched and in line 27 all of the terms for ACE inhibitors and ARBs are combined by using the “or” operator. Finally, we combine lines 11, 21, and 27 by using the “and” operator, which allows us to search for all vascular disease studies in humans who are stable and have preserved ventricular function evaluating ACE inhibitors or ARBs. 20. humans.sh. 21. 19 and 20 22. (alacepril or benazepril or captopril or ceronapril or cilazapril or delapril or enalapril or fosinopril or imidapril or libenzapril or lisinopril or moexipril or moveltipril or pentopril or perindopril or quinapril or ramipril or spirapril or temocapril or teprotide or trandolapril or zofenopril).mp. 23. (losartan or olmesartan or telmisartan or valsartan or eprosartan or candesartan or tasosartan or irbesartan).mp. 24. Angiotensin-Converting Enzyme Inhibitors/ 25. Angiotensin II Type 1 Receptor Blockers/ 26. (ACEI or ARB).mp. 27. 22 or 23 or 24 or 25 or 26 28. 11 and 21 and 27 Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Example of Observations Study Search (I) The following is a search of MEDLINE (Ovid) for Observational Studies by using the Scottish Intercollegiate Guidelines Network Observational Study Medline Search Filter (available at http://www.sign.ac.uk/methodology/filters.html). In the first 12 lines, we use terms for observational studies with different operators such as “.tw” to search in title and abstract, “adj” to search for adjacent terms, a slash (/) to search for all subheadings, a dollar sign ($) to truncate words, and “exp” to explode terms. The terms used are: 1. epidemiologic studies/ 2. exp case control studies/ 3. exp Cohort Studies/ 4. case control.tw. 5. (cohort adj (study or studies)).tw. 6. cohort analy$.tw. 7. (follow up adj (study or studies)).tw. 8. (observational adj (study or studies)).tw. 9. longitudinal.tw. 10. retrospective.tw. 11. cross sectional.tw. 12. Cross-Sectional Studies/ In line 13, we combine all 12 terms with the “or” operator. 13. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Example of Observational Study Search (II) In lines 14 and 15, we use individual names of the angiotensin-converting enzyme (ACE) inhibitors of interest; and in line 16, we use the individual names of the angiotensin-receptor blockers (ARBs) and search for them as free-text terms. 14. (alacepril or benazepril or captopril orceronapril or cilazapril or delapril or enalapril or fosinopril or imidapril or libenzapril orlisinopril or moexipril).mp. 15. (moveltipril or pentopril or perindopril or quinapril or ramipril or spirapril or temocapril or teprotide or trandolapril or zofenopril).mp. 16. (losartan or olmesartan or telmisartan or valsartan or eprosartan or candesartan or tasosartan or irbesartan).mp. In lines 17, 18, and 19 we search for the drug classes. 17. Angiotensin-Converting Enzyme Inhibitors/ 18. Angiotensin II Type 1 Receptor Blockers/ 19. (ACEI or ARB).mp. In line 20, we combine all of the lines defining ACE inhibitors and ARBs with the “or” operator. 20. 14 or 15 or 16 or 17 or 18 or 19 Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Example of Observational Study Search (III) In line 21, we use all of the terms for preserved heart disease through the use of the terms as follows: 21. (((preserved adj left) or (stable adj cad) or (stable adj chd) or (stable adj coronary) or (preserved adj coronary) or (preserved adj systolic) or (preserved adj ventricular) or (preserved adj lvef) or (preserved adj ef) or (preserved adj ejection)) or (intact adj left) or (intact adj systolic) or (intact adj ventricular) or (intact adj lvef) or (intact adj ef) or (normal adj systolic) or (normal adj ventricular) or (normal adj lvef) or (normal adj ef)).mp Then in line 22, we combine lines 13, 20, and 21 with the “and” operator to find studies of observational trials evaluating ACE inhibitors or ARBs in preserved or stable ischemic heart disease as follows: 22. 13 and 20 and 21 Reference: Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease . Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • Key Messages This module stresses the importance of searching for relevant studies. Balancing precision and recall is important and should help guide search strategies. More than just MEDLINE is needed, and using multiple approaches is optimal. Understanding Boolean operators and using filters can help balance precision and recall, as can performing more than one search of the same database (such as, searching for efficacy and harms separately). Finally, report the search strategy with transparency so that it can be evaluated and, if needed, replicated.
  • References (I)
  • References (II)
  • References (III)
  • Author

Transcript

  • 1. Searching for Relevant Studies Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov
  • 2. Systematic Review Process Overview
  • 3.
    • To substantiate why prudent literature searching is important
    • To describe important tenets of literature searching
    • To describe why transparency in the search process is important
    Learning Objectives
  • 4.
    • Systematic reviews require comprehensive searches.
    • Searching many databases by using a generic term gives high recall but is very imprecise. For example:
      • A July 23, 2009, Ovid search for “cancer.mp or neoplasms” yielded 786,978 citations.
      • That is too many citations to review to yield the 323 citations on the role of tamoxifen and raloxifene in cancer, the reason for the search.
    • The goal is to carefully construct a search to keep precision high while maximizing recall.
    Balancing Precision and Recall Precision (specificity): the proportion of retrieved articles that are relevant Recall (sensitivity): the proportion of potentially relevant articles retrieved by search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 5.
    • One evaluation compared the results of popular studies against others with populations and methodologies that were similar or superior.
      • Evaluation of 45 studies published from 1993 to 2000 in journals with high impact factors that reported positive findings and were cited more than 1,000 times.
      • Results were compared against other studies with the same comparators by employing populations and methodologies that were similar or superior.
      • Opposite/null findings or much more dramatic effects were found 31 percent of the time.
        • Eighty-three percent of nonrandomized studies and 23 percent of randomized controlled trials were subsequently contradicted.
    Presenting Popular Studies Is Not Enough Ioannidis JP. JAMA 2005;294:218-28.
  • 6.
    • When a general topic is proposed, it is tempting to begin by extensively searching for primary literature.
    • Before performing the extensive literature search, it is crucial to understand the topic, devise an analytic framework, ask clearly defined key questions, and understand the scope of the review to be conducted.
    Where To Begin
  • 7.
    • Although the topic area dictates the databases to be included, the following are common ones:
      • MEDLINE (General)
      • Cochrane Central (General)
      • EMBASE (General, more international in scope)
      • PsychLit/PsychINFO (Psychology)
      • AIDSLine (HIV/AIDS)
      • CINAHL (Nursing)
      • TOXNET (Adverse Events/Toxicology)
    Specialized Electronic Databases Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press. Generally a minimum requirement for a thorough search of literature
  • 8.
    • Study 1 (Sassi et al., 2002): sensitivity and specificity of using MEDLINE versus nine databases
      • This study focused on ability to retrieve economic analyses from January to March 1997.
      • MEDLINE-only searches had a sensitivity of 72 percent and a specificity of 75 percent versus nine databases.
    • Study 2 (Betrán et al., 2005): systematic review of the prevalence of maternal mortality and morbidity from 1997 to 2002
      • Multiple databases were searched (MEDLINE, EMBASE, BIOSIS, LILACS).
      • 60 percent of citations were found in more than one database.
      • The MEDLINE search had 20 percent of nonreplicated citations, EMBASE had 7.4 percent, and LILACS had 5.6 percent.
    MEDLINE Alone Is Not Enough Sassi, F, et al. Med Care 2002;40:387-94; Betrán AP, et al. BMC Med Res Methodol 2005;5:6.
  • 9.
    • Perform a pilot search and compare to see if output matches articles already identified by the research group, key informants, and manual searches of references of these articles.
      • Pilot searching is usually limited to MEDLINE and then adapted to other databases.
      • After pilot searching, refine the search.
      • Searching the literature is an iterative process.
    Identifying Key Articles To Refine a Pilot Search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 10.
    • Multiple searches within the same database may be necessary.
    • Efficacy and harms searches may be separate.
      • If the available literature is small, a single broad search is appropriate.
      • If not so, separate searches are prudent.
    Multiple Searches Help With an Extensive Literature Base (I) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 11.
    • An efficacy search may include the disease, intervention, and terms for randomized controlled trials (RCTs).
      • Efficacy evaluations frequently are limited to RCTs.
        • Fewer inherent biases and higher quality evidence
    • A harms search may be broader and include just the intervention and harms and/or disease.
      • The paucity of harms data in published RCTs frequently requires that non-RCTs be included.
      • Authors would likely be interested in harms regardless of the reason for the intervention.
    Multiple Searches Help With an Extensive Literature Base (II)
  • 12.
    • “ OR” — makes the search broader
    • “ AND” — makes the search more selective
    • “ /” — means all subheadings are searched
      • When a search term is entered, you can selectively choose some subheadings (such as pharmacology for pharmacology studies).
      • Subheadings are generally imprecise; accepting all subheadings increases the yield.
    • “ adj” — cuts down on miscellaneous citations
      • “ Heart adj failure” would find only instances in which “heart” is next to “failure.”
    • “ $” — truncates a word with different endings
      • “ Analy$” would find words like analysis, analyses, analyze, and analyse
    • “ .ti” or “.ab” — searches in the title or abstract for the word of interest
      • Can find citations where the word you search for is not a keyword
    • “ .mp” or multiple posting — a text word search; does not need to be a key word
    Boolean Operators Help To Balance Precision and Recall
  • 13.
    • If the specific adverse effects of interest are specified in the key questions, they can be searched for directly (e.g., rhabdomyolysis, cancer).
    • General harms subheadings for MEDLINE would include:
      • /adverse effects
      • /poisoning
      • /toxicity
      • /chemically induced
      • /contraindications
      • /complications
    • General harms subheadings for EMBASE would include:
      • /side effect
      • /adverse drug reaction
      • /drug toxicity
      • /complication
    • Consider using a database dedicated to harms such as TOXNET (http://toxnet.nlm.nih.gov/).
    Harms Searching Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 14.
    • Filters have been developed and validated that can help balance precision and recall.
      • Cochrane Collaboration http://www.cochrane-handbook.org section 6.4.11 Search Filters, National Health Service
      • Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/SysRev/!SSL!/WebHelp/SysRev3.htm
      • Scottish Intercollegiate Guidelines Network http://www.sign.ac.uk/methodology/filters.html
      • InterTASC Information Specialists’ Sub-Group http://www.york.ac.uk/inst/crd/intertasc/about.htm
    • Filters may not work in all circumstances
    Filters or Hedges Are Validated Ways To Improve Precision and Recall Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 15.
    • MEDLINE (Ovid) Search for Randomized Controlled Trials Using the Cochrane Highly Sensitive and Specific Search Strategy (Sensitivity and Precision Maximizing Version 2008)
    • 1. randomized controlled trial.pt.
    • 2. controlled clinical trial.pt.
    • 3. randomized.ab.
    • 4. placebo.ab.
    • 5. clinical trials as topic.sh.
    • 6. randomly.ab.
    • 7. trial.ti.
    • 8. 1 or 2 or 3 or 4 or 5 or 6 or 7
    Example Filter (Hedge) for Finding Randomized Controlled Trials Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 16.
    • MEDLINE (Ovid) Search for Observational Studies Using the Scottish Intercollegiate Guidelines Network Observational Study Medline Search Filter
      • 1. epidemiologic studies/
      • 2. exp case control studies/
      • 3. exp Cohort Studies/
      • 4. case control.tw.
      • 5. (cohort adj (study or studies)).tw.
      • 6. cohort analy$.tw.
      • 7. (follow up adj (study or studies)).tw.
      • 8. (observational adj (study or studies)).tw.
      • 9. longitudinal.tw.
      • 10. retrospective.tw.
      • 11. cross sectional.tw.
      • 12. Cross-Sectional Studies/
      • 13. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12
    Example Filter for Finding Nonrandomized Studies Coleman CI, et al. Comparative Effectiveness Review No. 18 . Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 17.
    • Items to consider before hitting enter:
      • Use of natural language terms (search as key word)
      • Appropriate word truncation and spelling variation
      • Appropriate use of limits such as language and years
      • Appropriate use of searching for field, publication type, and author
    Peer Review of Electronic Search Strategies (PRESS) Checklist (I) Sampson M, et al. PRESS: peer review of electronic search strategies . 2008.
  • 18.
    • Items to consider before hitting enter:
      • Spelling errors/spelling in American and British English (e.g., anaesthesia and anesthesia, analyse and analyze, oestrogen and estrogen)
      • Line errors that occur when searches are combined by using line numbers (e.g., “1 AND 3” instead of “1 AND 4”)
      • Appropriate use of Boolean operators
      • Adaptation of search strategies for specific databases
      • Are all appropriate headings used (refer to the thesaurus for a specific database)
      • Appropriate use of subheadings and floating subheadings
    Peer Review of Electronic Search Strategies (PRESS) Checklist (II) Sampson M, et al. PRESS: peer review of electronic search strategies . 2008.
  • 19.
    • Publication bias is the tendency of certain types of trials (such as those with the largest effects) to be published.
    • Publication bias increases the risk that the observed effect might not reflect the true effect.
      • It may negatively impact consistency, precision, and magnitude of effect.
    • Expanding searches to include additional languages, citation tracking, hand searching, and grey literature can help identify and possibly minimize publication bias.
    Considering Publication Bias When Developing a Search Strategy Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 20.
    • Study (Turner et al., 2008): 74 antidepressant studies registered with the U.S. Food and Drug Administration
      • 97 percent of positive studies published
      • 39 percent of neutral or negative studies published
        • 11 of 14 studies published in a way that conveyed the positive results but deemphasized the negative
      • When only published literature was meta-analyzed, a 32 percent increase in relative effect size occurred versus the more complete dataset of conducted trials.
    Example of Publication Bias Turner EH, et al. N Engl J Med 2008;358:252-60.
  • 21.
    • It is not a requirement to search non-English–language publications.
      • Evidence-based Practice Centers should consider how inclusion or exclusion of non-English–language studies might or might not bias the search.
      • It is usually advisable to include non-English– language studies when the evidence base may be highly restricted in English-language but broader in non-English–language studies.
        • Such as, complementary and alternative medicine topics
    Restricting Search by Language and Impact on Publication Bias Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 22.
    • Study 1 (Pham et al., 2008): evaluation of 42 systematic reviews, including 662 randomized controlled trials
      • For Western medicine, language-restricted systematic reviews did not introduce biased results when compared with language-inclusive ones, in terms of estimates of intervention effectiveness (risk odds ratio [ROR] = 1.02; 95% confidence interval [CI] = 0.83–1.26).
      • For complementary and alternative medicine, however, language-restricted systematic reviews resulted in a 63 percent smaller protective effect estimate when compared with language-inclusive reviews (ROR = 1.63; 95% CI = 1.03–2.60).
    English Only or All Languages (I) Pham B, et al. J Clin Epidemiol 2005;58:769-76.
  • 23.
    • Study 2 (Vickers et al., 1998): evaluation of acupuncture trials conducted in England versus China, Japan, Russia, and Taiwan
      • Results of randomized controlled trials or controlled clinical trials in England were favorable 75 percent of the time.
      • Results in China, Japan, Russia, and Taiwan were favorable 99, 89, 97, and 95 percent of the time, respectively.
        • Researchers should carefully consider how to manage data from these countries.
    English Only or All Languages (II) Vickers A, et al. Control Clin Trials 1998;19:159-66.
  • 24.
    • Forward citation tracking: databases that provide information on articles cited by other authors
      • This method relies on the author’s choice to cite an article, rather than key words or indexing.
        • Can turn up unique items and can complement a traditional database search.
      • Databases currently in use are Web of Science, Scopus, Google Scholar, PubReMiner, and PubFocus.
        • Web of Science and Scopus have access fees but have the most developed search and export interfaces.
        • Google Scholar, PuReMiner, and PubFocus are free access but are not as developed.
    Citation Tracking (I) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 25.
    • Backward citation tracing: reading references of key articles
      • New search terms, poorly indexed journals, or inappropriate limiters in the original search can be identified
    • “ Related article” links: useful when designing and refining a search but not for a formal search
      • Difficult to perform systematically, report strategy with transparency, and reproduce the search
    Citation Tracking (II) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 26.
    • Not all journals or sources can be indexed in databases.
      • Some journals index only certain article types.
      • Abstracts, conference proceedings, and supplements may not be indexed.
    • Backward citation tracking and key informant discussion can identify these journals.
    • Ulrich has a useful Web site to identify journals in a specific topic area (available at: http://www.ulrichsweb.com/ulrichsweb/).
      • If Ulrich indicates an important journal is not indexed fully in databases used, search the journal by hand.
    Hand Searching To Reduce Publication Bias Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 27.
    • Study 1 (Hopewell et al., 2007): using MEDLINE to search 22 specialized journals to identify randomized controlled trials (RCTs) was compared with hand searching of the same journals
      • 117 of the 462 RCTs (25%) were not retrieved by MEDLINE because of a lack of the publication-type terms “randomized controlled trial” or “controlled clinical trial.”
        • Additional articles were missed in journals without full journal indexing.
    • Study 2 (Jadad et al., 1993): using MEDLINE to search nine pain and anesthesia journals was compared with hand searching
      • 13 percent of eligible articles were not retrieved with MEDLINE alone.
    Benefits of Hand Searching Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010; Jadad AR, McQuay HJ. Online J Curr Clin Trials 1993;(33).
  • 28.
    • Grey literature: literature produced at all levels of government, academics, business, and industry that is not controlled by commercial publishers
      • Usually not systematically identified, stored, or indexed
      • Can help overcome publication bias
    • Least efficient type of literature to search and may be (or be perceived to be) of lower quality
      • Useful in areas with little published evidence, in fields that are rapidly changing (e.g., devices, surgery), or in fields that are highly interdisciplinary (e.g., mental health, alternative medicine)
    Using Grey Literature To Reduce Publication Bias Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 29.
    • Study (Hopewell et al., 2008): reviewed five systematic reviews conducted to evaluate the impact of grey literature in meta-analyses of randomized controlled trials
      • All systematic reviews found more positive results with published literature than with grey literature (risk odds ratio = 1.09; 95% confidence interval = 1.03–1.16)
    Grey Literature Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010 .
  • 30.
    • The approval process for new drugs and devices involves submission of data that may not be published or may be incompletely published elsewhere
      • Good check for publication bias
    • Regulatory Web sites:
      • Drugs@FDA; http://www.accessdata.fda.gov/Scripts/cder/DrugsatFDA/ index.cfm?fuseaction=Search.search_Drug_Name
      • FDA Center for Drug Evaluation and Research (CDER) Archives: http://www.fda.gov/cder/archives/default.htm
      • Dockets for FDA: http://www.fda.gov/ohrms/dockets/default.htm
      • Devices@FDA: http://www.accessdata.fda.gov/Scripts/cdrh/devicesatFDA/
      • Health Canada Drug Product Search: http://www.hc-sc.gc.ca/dhp-mps/prodpharma/databasdon/index-eng.php
      • European Medicines Agencies European Public Assessment Reports for Authorized Medicinal Products for Human Use: http://www.emea.europa.eu/htms/human/epar/a.htm
    Grey Literature: Regulatory Search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 31.
    • Online registries include trial name, type, and brief methodology
      • May contain results of completed but unpublished trials
    • Registry Web sites:
      • ClinicalTrials.gov: http://clinicaltrials.gov/ct2/search
      • ClinicalStudyResults.org: http://www.clinicalstudyresults.org/home/
      • Current Controlled Trials: http://www.controlled-trials.com/mrct/
      • Australian New Zealand Clinical Trials Registry: http://www.anzctr.org.au/trialSearch.aspx
      • Netherlands Trial Registry: http://www.trialregister.nl
    Grey Literature: Online Registries Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 32.
    • Much grey literature is unlikely to be published.
    • When published, final data often differ from those in the conference abstract.
    • Use key informants to identify the most important meetings in the field of interest and hand search the abstracts and conference proceedings in journal supplements.
      • Some databases are also available: ProceedingsFirst; Biological Abstracts; Conference Papers Index; Conference Proceedings Citation Index (part of the Web of Science); and BioWizard Scholar Conference Abstract Search.
    Grey Literature: Abstracts and Conference Proceedings Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 33.
    • Theses and dissertations are becoming increasingly available as institutions post them online.
    • Databases:
      • ProQuest Dissertation and Theses
        • The most comprehensive collection in the United States, with full text of dissertations through their Digital Archiving and Access program
      • Index to Theses in Great Britain and Ireland
      • Networked Digital Library of Theses and Dissertations (NDLTD)
      • Dissertation Abstracts Online
    Grey Literature: Theses and Dissertations Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 34.
    • Aside from governmental regulatory sites, other governmental agencies support or collect grey literature.
      • Computer Retrieval of Information on Scientific Projects (CRISP): http://crisp.cit.nih.gov
        • Searchable database of federally funded biomedical research
      • Health Services Research Projects in Progress (HSRPROJ): http://wwwcf.nlm.nih.gov/hsr_project/home_proj.cfm
        • Database of ongoing grants and projects in health services research
      • Science Accelerator: http://www.scienceaccelerator.gov/
        • From the Office of Scientific and Technical Information and the U.S. Department of Energy
      • Defense Technical Information Center (DTIC): http://www.dtic.mil/dtic/
        • From the U.S. Department of Defense, including the military health system
    Grey Literature: Government Documents Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 35.
    • Grey Matters: http://www.cadth.ca/index.php/en/ cadth/products
      • A practical tool for evidence-based searching
      • Collated list of sources for grey literature appropriate for health sciences by the Canadian Agency for Drugs and Technology in Health (CADTH)
    • System for Information on Grey Literature in Europe (SIGLE): http://opensigle.inist.fr/
      • Provides bibliographic references to grey literature in Europe
    • New York Academy of Medicine (NYAM): http://www.nyam.org/library.grey.shtml
      • Provides NYAM grey literature with a list of other organizations that do the same
    Grey Literature: General Sources Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 36.
    • Researchers can be contacted to determine whether or not projects in the grey literature are published in a peer-reviewed form.
    • Prolific researchers in the field can be contacted to see if they know of any citations that were missed by the literature search.
    • Search engine (e.g., Yahoo, Google) queries with the individual’s name or institution can be a good place to start.
      • Contact information in manuscripts is also a reasonable place to start.
    Directly Contacting Researchers Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 37.
    • SIPs are provided by the company manufacturing the drug or device.
    • SIPs contain information about products, including the product label and published and unpublished trials or studies. They:
      • can help to overcome publication bias by identifying trials that remain unpublished;
      • can identify soon-to-be-released trials that can ultimately be captured in comparative effectiveness reviews (CERs); and
      • allow drug and device manufacturers to be explicitly involved in helping provide a literature base to a CER.
    Scientific Information Packets (SIPs) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 38. Scientific Information Packets Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 39.
    • Accurate and transparent reporting of search strategies is important.
      • Searches need to be updated toward the conclusion of a systematic review to ensure that no new information has come out in the intervening period.
      • Stakeholders and end-users want assurances that the systematic review was free of bias and that adequate search principles were followed.
      • Searches will need to be rerun if the systematic review is updated several years after it is completed.
    • Documenting the search strategy so that a skilled literature searcher can reproduce it is very important.
    Reporting Search Strategies: Transparency Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 40. Transparent Reporting
    • Databases used
    • Dates covered
    • Search terms
    • Language restrictions
    • Nondatabase methods used
    • Inclusion/exclusion criteria
    • Full electronic search strategy
    • Publication-related restrictions
    • End date of search
    • List of excluded references
    • Qualifications of searcher
    • Number of references identified
    • QUOROM - or PRISMA-style flow diagram accounting for all references
    • Evidence of search effectiveness
    • Statement of filters employed
    • Description of sampling strategy
    Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.
  • 41. Transparent Reporting: QUOROM Figure Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 42. Transparent Reporting: PRISMA Figure Included Screening Eligibility Identification
  • 43.
    • The next few slides present a comparison of a controlled trial search and an observational study search.
    • See how the search pieces are created and then come together.
      • Note the use of Boolean operators.
    Example of Two Searches
  • 44. Example of Controlled Trial Search (I) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 45. Example of Controlled Trial Search (II) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 46. Example of Controlled Trial Search (III) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 47. Example of Observational Study Search (I) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 48. Example of Observational Study Search (II) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 49. Example of Observational Study Search (III) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
  • 50.
    • Searching is important: balance precision and recall.
      • MEDLINE is not enough.
    • Multiple strategies improve completeness of search.
    • Filters are a good place to start.
    • More than one search may be needed (e.g., separately for “efficacy” and “harms”).
    • Report strategy with transparency.
    Key Messages
  • 51.
    • Betran AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol 2005 ;5:6.
    • Campbell R, Balshem H. Finding evidence for comparing medical interventions. J Clin Epidemiol 2010;63: in press.
    • Coleman CI, Baker WL, Kluger J, et al. Comparative Effectiveness of Angiotensin Converting Enzyme Inhibitors or Angiotensin II Receptor Blockers Added to Standard Medical Therapy for Treating Stable Ischemic Heart Disease. Comparative Effectiveness Review No. 18. (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, October 2009. AHRQ Publication No. 10-EHC002-EF. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.
    • Hopewell S, Clarke M, Lusher A, et al. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med 2002;21:1625-34.
    References (I)
  • 52.
    • Hopewell S, McDonald S, Clarke MJ, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;(2):MR000010.
    • Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005;294:218-28.
    • Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials 1993;(33).
    • Pham B, Klassen TP, Lawson ML, et al. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 2005;58:769-76.
    References (II)
  • 53.
    • Sampson M, McGowan J, Lefebvre C, et al. PRESS: peer review of electronic search strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.
    • Sassi F, Archard L, McDaid D. Searching literature databases for health care economic evaluations: how systematic can we afford to be? Med Care 2002;40:387-94.
    • Turner EH, Matthews AM, Linardatos E. et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60.
    • Vickers A, Goyal N, Harland R, et al. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials 1998;19:159-66.
    References (III)
  • 54.
    • This module was prepared by C. Michael White, Pharm.D., FCP, FCCP, a member of the University of Connecticut/Hartford Hospital Evidence-based Practice Center.
    • This module is based on chapter 5 in version 1.0 of the Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews (available at: http://effectivehealthcare.ahrq.gov/repFiles/2007_10Draft MethodsGuide.pdf).
    Author