• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Study Eligibility Criteria
 

Study Eligibility Criteria

on

  • 1,884 views

 

Statistics

Views

Total Views
1,884
Views on SlideShare
1,884
Embed Views
0

Actions

Likes
1
Downloads
22
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Study Eligibility Criteria
  • Systematic Review Process Overview This slide illustrates the steps in the systematic review process. This module focuses on study eligibility criteria.
  • Learning Objectives
  • Study Selection Criteria Selection criteria, also known as eligibility or inclusion/exclusion criteria, are used to set appropriate limits by which studies are included in a review. References : Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91. http://www.ncbi.nlm.nih.gov/pubmed/7718048 McPheeters ML, Briss P, Teutsch SJ, et al. Systematic reviews in public health. In: Brownson RC and Petitti DB, eds. Applied epidemiology. New York: Oxford University Press; 2006. p. 99-124.
  • Some Sample Criteria Examples of criteria include setting limits around a population (e.g., limiting to women or men, or to community-dwelling women); the intervention (which should be very specific); the country or other geographic limit; the specific setting where one would expect to have the intervention implemented; the size of the study population; and potentially the study design.
  • Using Broad Criteria Criteria range from being very broad to very narrow and reflect the goals of the particular review, as well as the content area of the research.
  • Using Narrow Criteria
  • Refining Criteria The overarching goal of setting criteria is to minimize bias related to which studies are selected. In doing so, review teams must balance the need to obtain adequate information to answer a given review question with the potential to include too much extraneous information.
  • Bias in This Context Decisions about eligibility criteria may bias the review, possibly resulting in an overestimation or underestimation of the true treatment effect. When bias is discussed in this context, it is defined as “distortion of the estimate of effect that comes from how studies are selected” and may affect how applicable the review will be in practice.
  • Examples of Bias in This Context For example, the included studies may not have been conducted in the patient population that is the focus of the review. In another example, the use of studies of twin pregnancies in a review of preterm labor management for low-risk women may not provide applicable information for the target population. In still another example, limiting a review to randomized controlled trials (RCTs), especially when there are few of them, is likely to provide data on only a highly specific group of individuals that may not reflect common practice. This could result in an overestimate of the effect of an intervention if its effectiveness in practice is lower than that seen in RCTs.
  • Selecting Criteria Broader criteria are often set at the abstract review phase and refined to be more specific for the full-text review phase, so this process is likely to occur twice in the review. Sometimes it is helpful for the team to set criteria, then do a sample set of reviews, and reconvene to discuss whether the criteria are meeting the goals of the review. Criteria can be refined at this point if needed but then should be adhered to henceforth.
  • Tie Criteria to PICOTS Because we will be discussing the use of PICOTS (population, intervention, comparator, outcome, timing, and setting) as a framework for setting criteria, please review the meaning of PICOTS before proceeding. It is helpful to use the PICOTS framework in setting criteria. Specifically, consider each of the PICOTS components individually and determine with the team whether criteria associated with each are warranted.
  • Other Consideration for Setting Criteria Other potential considerations are the inclusion of specific study designs, studies published in foreign languages, and studies from other countries (those published in English); the use of grey or fugitive literature; and whether dates of publication should be limited (e.g., to include only studies published after a certain year if that marks a point at which practice changed, a new technique emerged, or a new medication was developed).
  • Types of Studies To Be Included Limiting a review to randomized controlled trials is appropriate for questions of efficacy but may preclude understanding broader effectiveness of an intervention. If observational studies are included, teams may decide to include only certain subtypes (e.g., prospective and retrospective cohorts but not case series). There may also be situations in which case series provide important information, especially in emerging fields or for identifying harms.
  • Observational Studies (I)
  • Observational Studies (II)
  • Non-English–Language Study Reports The question of whether to include studies that were published in languages other than English depends on both methodologic considerations and logistical ones. It may be impractical to obtain translation services within the context of a review. However, positive findings may be more likely to be published in high-profile English-language journals, and some researchers have suggested that to include only English-language studies may overestimate the positive effect of an intervention. Empirically, the bias associated with limiting a review to English-language publications, however, has been shown to be small. References: Gregoire G, Derderain F, Le Lorier J. Selecting the language of the publication included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63. http://www.ncbi.nlm.nih.gov/pubmed/7853041 Moher D, Pham B, Klassen TP, et al. What contribution do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000; 53:964-72. http://www.ncbi.nlm.nih.gov/pubmed/11004423
  • Grey or “Fugitive” Literature Reference: McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet  2000;356:1228-31. http://www.ncbi.nlm.nih.gov/pubmed/11072941
  • Year of Publication
  • Using the Criteria
  • Exercise 1
  • Exercise 1: Basic PICOTS Questions We can use these questions to operationalize the PICOTS (population, intervention, comparators, outcomes, timing, and setting) into practical questions for establishing criteria.
  • Exercise 1: PICOTS Here are examples of further specification of the PICOTS (population, intervention, comparator, outcome, timing, and setting). These are only examples, and the review team should think through as many issues as are relevant to the particular review.
  • Exercise 1: What Would You Do With … Sometimes we set criteria and then find that exceptions arise, or papers are found that do not fit nicely into them. For example, if we were conducting this review of adolescents with autism spectrum disorder (ASD), what would we do with a paper that said it included “individuals over 10 years of age.” Adolescents, and potentially adults, would have been included in the review, but so would children under 13 years of age. By the same token, a paper might be selected that included not only individuals with ASD, but also some with other developmental disabilities.
  • Exercise 1: No “Right” Answers There are no “right” answers to these questions. The team should clearly define how eligibility criteria are operationalized and record these decisions throughout the review.
  • Exercise 2: Selecting Criteria for a Narrow Research Question Here is an example of a research question that, by its nature, led to very narrow selection criteria. Notice that the question focuses on “efficacy” (which suggests the inclusion of randomized controlled trials only) and on women at low risk of preterm birth (which would exclude women with a previous preterm birth or with multiple birth).
  • Exercise 2: Implications of a Question That Requires Narrow Criteria In a real review, these narrow criteria led to the identification of only 11 studies.
  • Exercise 3: Selecting Criteria for a Broad Research Question In another example—a review of the management of overactive bladder among women—the team faced several challenges. First, overactive bladder is a difficult condition to define, making it challenging to put bounds around the “right” population. Treatments vary and include drugs, surgery, and behavioral interventions. It is critical for teams to justify and document their decisions about selection criteria as these decisions are made. Later, this becomes an important part of the report that increases transparency in the methods.
  • Exercise 3: Impact of Study-Size Criteria on Excluded Studies Based on expert opinion and a preliminary review of the literature, the team set a minimum sample size of 50 for the review on overactive bladder. A total of 79 studies out of more than 2,000 were excluded on the basis of study size.
  • Exercise 3: Impact of Gender in Studies on Selection Criteria Recall that this review was to focus on the management of overactive bladder in women. However, many studies of treatment included some men, and the team determined that to exclude all of these studies might create a loss of important information. On the other hand, the fact that men who had bladder-related issues likely had a different etiology for those issues and could have a different response to treatment could bias the overall estimate of effect. Based on expert opinion and the size and scope of the literature, the team included only studies whose sample had at least 75 percent women. This resulted in the exclusion of 40 studies. Had the team limited the review to studies that included only women, another 27 studies would have been excluded. The team looked carefully at the excluded studies and, with the help of technical experts, determined that exclusion of studies with less than 75 percent women was appropriate for reducing the possibility of obtaining a biased estimate.
  • Example in a Maternal-Fetal Surgery Technical Brief In a technical brief on the use of maternal-fetal surgery, the purpose was to describe the state of the literature, rather than to present estimates of outcomes. Only three randomized controlled trials were available in all of the literature, and most of the observational studies were case series. Case series were included because this area of research is very new and the purpose was to describe the state of the field, rather than the outcomes of the different surgeries.
  • Example 2: Systematic Review of Cesarean Delivery In response to the rising rate of cesarean deliveries, the National Institutes of Health Office of Medical Applications of Research planned a State-of-the-Science Conference on the maternal and neonatal outcomes of cesarean delivery on maternal request, which was defined as cesarean delivery in the absence of maternal or fetal indications.
  • Example 2: The Challenge in the Systematic Review on Cesarean Delivery From preliminary reviews of the literature, the National Institutes of Health was aware that no studies compared outcomes of cesarean delivery on maternal request (CDMR) to other modes of delivery. As a proxy, they suggested that the review include studies comparing cesarean delivery for breech to other modes of delivery, but this body of evidence was also small and limited in applicability — that is, breech delivery was a reasonable proxy for CDMR for maternal outcomes, but not for neonatal outcomes, where fetal indications have been a confounding factor for both breech delivery and poor neonatal outcomes. With an urgent need for actionable evidence, and in the absence of data on the intervention of interest, the Research Triangle Institute – University of North Carolina Evidence-based Practice Center decided to expand the list of proxies and weight the evidence appropriately.
  • Example 2: The Solution in the Systematic Review on Cesarean Delivery A crucial element of the comparison between cesarean delivery on maternal request (CDMR) and alternatives is the planned aspect of delivery. The Research Triangle Institute – University of North Carolina Evidence-based Practice Center (RTI-UNC EPC) included studies that compared planned vaginal delivery to planned cesarean delivery (for maternal and neonatal indications) as lower-rung evidence. As lowest-rung evidence, the RTI-UNC EPC also included comparisons of actual modes of delivery.
  • Key Messages
  • References
  • Author

Study Eligibility Criteria Study Eligibility Criteria Presentation Transcript

  • Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov
  • Systematic Review Process Overview
    • To understand the role of study eligibility criteria (inclusion/exclusion) in framing a systematic review
    • To know when and how to set study eligibility criteria
    • To understand the effect of study eligibility criteria on interpretation of a review
    Learning Objectives
    • Determine which studies will be included in the analysis
    • Function the same in systematic reviews as in primary research
    • Should reflect the analytic framework and key questions
    • Are powerful tools for widening or narrowing the scope of a review
    • Provide information to determine whether reviews can be compared or combined
    Study Eligibility Criteria
    • Population type (adult, community-dwelling females)
    • Intervention (screening tool to identify depression risk)
    • Country (United States only)
    • Setting (hospital-based studies only)
    • Study population size (N > 200)
    • Study design (randomized controlled trials)
    Some Sample Criteria
    • Can be as broad as “all related studies”
    • Can be helpful for exploring “what is known”
    • May result in too much literature to feasibly review
    • Could pull in disparate literature that cannot be compared
    Using Broad Criteria
    • Can be helpful in culling homogenous literature
    • Can reduce size of the literature to a manageable scope
    • Could be appropriate for a narrow research question
    • May reduce applicability of the review and sometimes risks introducing bias
    Using Narrow Criteria
    • Choosing how broad or how narrow to set criteria requires a balance of obtaining adequate information to answer a given question without obscuring the results with irrelevant literature.
    • Review teams should work together to find this balance.
    • The overarching goal is to minimize bias related to which studies are selected.
    Refining Criteria
    • Distortion of the estimate of effect that comes from how studies are selected for inclusion
    • Affects the applicability, or “external validity,” of the review itself
    Bias in This Context
    • Inappropriate eligibilty criteria may limit applicability of the review.
      • Example: the use of studies of twin pregnancies in a review of preterm labor management for low-risk women
    • Eligibility criteria may result in the inclusion of more of a certain study type that either overestimates or underestimates effectiveness.
    Examples of Bias in This Context
    • Review study goals
    • Assess analytic framework and key questions
    • Set criteria before beginning abstract review
    Selecting Criteria
    • Population — condition, disease severity and stage, comorbidities, patient demographics
    • Intervention — dosage, frequency, method of administration
    • Comparator — placebo, usual care, or active control
    • Outcome — health outcomes, morbidity, mortality, quality of life
    • Timing — Duration of followup
    • Setting — Primary, specialty, inpatient, cointerventions
    Tie Criteria to PICOTS
    • What study designs should be included?
    • Include studies in foreign languages?
    • Include studies conducted in other countries but published in English?
    • Include “grey” or “fugitive” literature?
    • Include year of publication?
    Other Considerations for Setting Criteria
    • Limit to randomized controlled trials?
    • Include observational studies?
      • If so, what specific types?
    • What is the value of a case series?
    • How is the definition of case series operationalized?
    Types of Studies To Be Included
    • There are four main types of observational studies:
      • Cohorts (with comparisons)
      • Case controls
      • Case series
      • Registries/databases
    Observational Studies (I)
    • Well-conducted observational studies can address issues of applicability and the need for longer-term outcomes if they:
      • include more representative patient populations,
      • have relevant comparators, and
      • report more meaningful clinical outcomes over longer time frames.
    • Observational studies may be a better source of information about harms.
    Observational Studies (II)
    • Positive findings may be more likely to be published in high-profile English-language journals.
    • Therefore, to include only English-language journals may overestimate the positive effect of an intervention.
    • Empirically, the bias associated with limiting a review to English-language reports has been shown to be small.
    Non-English–Language Study Reports Gregoire G, et al. J Clin Epidemiol 1995;48:159-63; Moher D, et al. J Clin Epidemiol 2000;53:964-72.
    • Material that may be difficult to identify and retrieve:
      • Government reports and regulatory sites (Center for Drug Evaluation and Research/U.S. Food and Drug Administration)
      • Pharmacoepidemiologic databases, including postmarketing surveillance
      • Book chapters
      • Conference proceedings
      • Published dissertations
      • Scientific information packets (Agency for Healthcare Research and Quality)
    • Rationale for a search:
      • Because journals may publish positive or statistically significant results, finding grey literature of unpublished nonsignificant or null results may indicate the presence of publication bias.
    Grey or “Fugitive “Literature
    • Literature searches typically have date parameters (i.e., earliest or latest date of publication).
    • Date parameters are particularly important when there has been a change in policy, practice, or formulation that makes older studies less applicable.
    Year of Publication
    • The rest of this module contains a number of questions and some sample criteria.
    • Using your newly acquired knowledge of selection criteria and the guidelines described in the previous slides, work through the thought exercises and examples that follow.
    Using the Criteria
    • What would you do if you were asked to review the literature on transition support for adolescents with autism spectrum disorder who are entering adulthood?
    • Before seeing the key questions, consider the categories of criteria that you will want to apply.
    Exercise 1
    • What is the relevant population?
    • What is the intervention of interest?
    • To what exposure is the intervention being compared?
    • What outcomes are relevant?
    • What short-term or long-term outcomes should be considered?
    • In what setting would the results be applicable?
    Exercise 1: Basic PICOTS Questions PICOTS = population, intervention, comparators, outcomes, timing, and setting
  • Exercise 1: PICOTS Population What constitutes an adolescent? What constitutes a diagnosis of autism spectrum disorder (ASD)? Intervention How is transition support defined? Comparator Do we compare to no transition support or directly compare types of support? Outcome What are the goals for adolescents with ASD as they transition to adulthood? Timing How quickly should the outcomes be apparent? Setting Is transition support provided in multiple settings, such as schools, clinics, and the community?
    • A study that included “individuals over 10 years of age”?
    • A paper about an intervention for individuals with a range of developmental disabilities, not only autism spectrum disorder (ASD)?
    • A study of children with Asperger’s syndrome but not other forms of ASD?
    Exercise 1: What Would You Do With . . .
    • There are no “right” answers to these questions.
    • The team should consider how selection criteria may bias the review or affect applicability.
    • The team should clearly define how eligibility criteria are operationalized and record these decisions throughout the review.
    Exercise 1: No “Right” Answers
    • What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?
    Exercise 2: Selecting Criteria for a Narrow Research Question
    • Efficacy: randomized controlled trials only
    • Low risk: no previous preterm birth
    • Yield: 11 studies
    Exercise 2: Implications of a Question That Requires Narrow Criteria
    • What is the best management strategy for overactive bladder (OAB) among women?
    • consideration:
      • OAB is a fairly difficult condition to define.
      • The study of OAB is often combined with other types of urological conditions, such as stress incontinence or prostate issues, when studied.
      • OAB is treated pharmacologically, behaviorally, and surgically and with complementary and alternative interventions — each area includes very different types of studies.
    Exercise 3: Selecting Criteria for a Broad Research Question
    • The research team required a minimum sample size N ≥ 50 in each study at study start.
    • Implications:
      • Total excluded for N < 50: 79
        • Excluded for N < 20: 36
        • Excluded for N 20 to 29: 23
        • Excluded for N 30 to 39: 8
        • Excluded for N 40 to 49: 12
    Exercise 3: Impact of Study-Size Criteria on Excluded Studies
    • Studies had to include at least 75 percent women.
      • This decision was based on expert opinion and the size and scope of the literature.
    • Based on this criterion, 40 studies were excluded.
    • Another 27 studies would have been excluded had the review been limited to studies of only women.
    Exercise 3: Impact of Gender on Study Selection Criteria
    • Case series with N ≥ 2 were included.
    • Only 3 of 169 studies were randomized controlled trials; 122 were case series.
    • Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups.
    Example 1: A Maternal-Fetal Surgery Technical Brief
    • A systematic review on outcomes of cesarean delivery on maternal request.
    • Conducted for the National Institutes of Health-Office of Medical Applications of Research (NIH-OMAR) State-of-the-Science Conference
    Example 2: Systematic Review of Cesarean Delivery
    • No evidence on outcomes of cesarean delivery on maternal request (CDMR) versus other modes of delivery
    • Urgent need for actionable evidence
    • Need to recognize and account for confounders 
    Example 2: The Challenge in the Systematic Review on Cesarean Delivery
    • Expand search to include proxies
    • Weight rungs of evidence to account for confounding
      • Highest rung: trials of breech delivery, but only for maternal outcomes
      • Lower rung: planned cesarean versus planned vaginal delivery
      • Lowest rung: comparisons of maternal and neonatal outcomes from actual modes of delivery
    Example 2: The Solution in the Systematic Review on Cesarean Delivery
    • Eligibility criteria:
      • Specify the scope of the literature to be included
      • Should be tied to the analytic framework, key questions, and PICOTS
      • Can reduce bias and support applicability, but run the risk of introducing bias if not chosen appropriately.
      • Must be applied consistently among individuals on the review team
      • Need to be recorded throughout the project and in the report for transparency and replicability
    Key Messages PICOTS = population, intervention, comparator, outcome, timing, and setting
    • Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91.
    • Gregoire G, Derderain F, Le L. Selecting the language of the publication included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63.
    • McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet  2000;356:1228-31.
    • McPheeters ML, Briss P, Teutsch SJ, et al. Systematic reviews in public health. In: Brownson RC and Petitti DB, eds. Applied epidemiology . New York: Oxford University Press; 2006. p. 99-124.
    • Moher D, Pham B, Klassen TP, et al. What contribution do languages other than English make on the results of meta-analyses ? J Clin Epidemiol 2000; 53:964-72.
    References
    • This presentation was prepared by Melissa L. McPheeters, Ph.D., M.P.H., a member of the Vanderbilt University Evidence-based Practice Center.
    • The information in this module is currently not included in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf).
    Author