0
Quantitative Synthesis II Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systema...
Systematic Review Process Overview
<ul><li>To understand how to explore between-study heterogeneity in a meta-analysis </li></ul><ul><li>To understand the pr...
Statistical Homogeneity Barrington KJ.  Cochrane Database Syst Rev  2000;(2):CD000505.
Statistical Heterogeneity Reprinted from Pakos E, et al.  J Bone Joint Surg Am  2005;87:1438-45, with permission from Rock...
<ul><li>In meta-analysis, heterogeneity refers to between-study diversity. </li></ul><ul><li>This term is often used to re...
<ul><li>Methodological diversity pertains to specifics of study design and analysis: </li></ul><ul><ul><li>Type of study <...
<ul><li>Statistical heterogeneity exists when the results of individual studies are not “consistent” among themselves. </l...
<ul><li>Clinical and methodological diversity is abundant: Our aim in a meta-analysis is to explore that diversity and use...
<ul><li>One of the most important roles of meta-analytic methodologies is to quantify statistical heterogeneity and to exp...
Dealing With Heterogeneity (I)
Dealing With Heterogeneity (II) OVERALL ESTIMATE Combining Summary Data SUBGROUP ANALYSES Differentiating Effects in Subgr...
<ul><li>Subgroup analyses can help: </li></ul><ul><ul><li>identify modifiers of the treatment effect, </li></ul></ul><ul><...
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I) Risk Ratio (95% Confidence ...
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II) Largest Effect in the Subg...
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III) Treatment Effect Diminish...
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV) Treatment Effect Diminishe...
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V) No Evidence of a Treatment ...
Subgroup Analysis:  A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al.  Ann Intern Med . 2005;142:37-4...
<ul><li>Subgroup analyses are a form of multiple testing. </li></ul><ul><li>When uncontrolled, multiple testing can yield ...
<ul><li>In the example that follows, we discuss subgroup analyses from the Second International Study of Infarct Survival ...
Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (I) Reprinted from ISIS-2 Collaborative Group. ...
Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2)  (II) Reprinted from ISIS-2 Collaborative Group...
<ul><li>When analyzing data, it is important to distinguish subgroup analyses that are specified a priori (without knowing...
<ul><li>Most meta-analyses use data that are published (and potentially known). </li></ul><ul><li>When researchers adequat...
<ul><li>Meta-analysts should do their best to define subgroups that make methodological and biological sense. </li></ul><u...
<ul><li>Meta-regression can help examine how the treatment effect changes across the levels of a variable.  </li></ul><ul>...
Subgroup Analysis:  A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al.  Ann Intern Med . 2005;142:37-4...
Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality  Miller ER 3rd, et al.  Ann Int...
Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo Ioannidis JPA. In:  Publication bias in meta-ana...
<ul><li>Study level </li></ul><ul><ul><li>Examples: presence/absence of blinding, intervention dose (in experimental studi...
<ul><li>Aggregate-data meta-regressions on participant-level covariates can mislead, because they are susceptible to ecolo...
<ul><li>Therefore, associations of treatment effect and participant-level covariates should be interpreted with caution. <...
<ul><li>In control-rate meta-regression, we examine whether the treatment effect changes across studies with different eve...
Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (I)
Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (II)
Control Rate Meta-Regression in the Preceding Streptokinase Example Schmid CH, et al.  Stat Med  1998;17:1923-42.
<ul><li>Subgroup analyses, meta-regressions, and control-rate meta-regressions are tools to explore between-study heteroge...
<ul><li>Barrington KJ. Umbilical artery catheters in the newborn: effects of position of the catheter tip. Cochrane Databa...
<ul><li>Pakos E, Ntzani EE, Trikalinos TA. Patellar resurfacing in total knee arthroplasty. A meta-analysis. J Bone Joint ...
<ul><li>This presentation was prepared by Joseph Lau, M.D., and Thomas Trikalinos, M.D., Ph.D., members of the Tufts Medic...
Upcoming SlideShare
Loading in...5
×

Quantitative Synthesis II

634

Published on

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
634
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Quantitative Synthesis II
  • Systematic Review Process Overview This slide illustrates the steps in the systematic review process. This module focuses on a further discussion of quantitative synthesis.
  • Learning Objectives
  • Statistical Homogeneity This is a forest plot of a meta-analysis with statistically homogeneous effects. All study estimates are very similar to each other. The confidence intervals from all studies are wide relative to the scatter of the point estimates. Formal statistical testing for heterogeneity in this case is consistent with our visual impression. When all studies have practically the same relative risk, there are not many things to explore: Any of the five studies, or any subset of the five studies, has practically the same risk ratio as the whole meta-analysis. Even if there is variation in the characteristics of the studies, there is no variability of effects to explore. However, in general, visual tests of statistical heterogeneity are precarious and should not be used in lieu of formal statistical testing. Reference: Barrington KJ, Umbilical artery catheters in the newborn: effects of position of the catheter tip. Cochrane Database Syst Rev 2000;(2):CD000505. http://www.ncbi.nlm.nih.gov/pubmed/10796375
  • Statistical Heterogeneity This forest plot is an extreme example of a meta-analysis with statistically heterogeneous effects: The point estimates are quite different when compared with the width of their respective confidence intervals. There is a 10-fold difference in the calculated relative risk between the first and third study. The confidence intervals of the 3rd and 5th studies do not overlap at all (i.e., each study excludes the other’s estimate). Formal statistical testing for heterogeneity suggests that this is a very heterogeneous meta-analysis. When there is substantial statistical heterogeneity, however, many interesting questions can be asked. Is there any explanation for the discrepancy between the 1st and 3rd studies? Might the definitions of anterior knee pain (the outcome) differ across studies? Could it be related to whether the assessment of pain was blinded versus nonblinded or recorded actively versus passively, or whether it was evaluated at shorter versus longer followup? Does it have to do with the mean ages of the participants in each study or with the type of knee prosthesis? To address these and other questions, we need to explore the observed between-study heterogeneity. We will discuss some of the tools that we can use for this exploration. In many cases, there will be no obvious explanation for the observed heterogeneity in the treatment effects. Other times, we may succeed in reaching a plausible hypothetical explanation. Reference: Pakos E, Ntzani EE, Trikalinos TA. Patellar resurfacing in total knee arthroplasty. A meta-analysis. J Bone Joint Surg Am 2005;87:1438-45. http://www.ncbi.nlm.nih.gov/pubmed/15995109
  • What Is Heterogeneity?
  • Methodological and Clinical Diversity Methodological diversity pertains to specifics of study design and analysis (e.g., type of study, length of followup, proportion and handling of dropouts). For example, particular types of studies may have found larger effects than others. Sometimes studies without blinded outcome assessments have systematically larger effects when compared to studies with blinded outcome assessments. Systematic differences are often suggestive of biases, and these variations in methodological characteristics may explain the observed statistical heterogeneity. Clinical diversity pertains to differences in the populations, intervention and cointerventions, and the definition of measured outcomes. For example, a higher drug dose may produce a larger effect size versus placebo when compared to a lower drug dose. Similarly, particular categories of patients may see larger benefits than other categories. Studies that exclusively enrolled patients with a condition of short duration may have larger effects than studies that exclusively enrolled patients with chronic conditions.
  • Statistical Heterogeneity In practice, clinical and methodological diversity are almost always present. Sometimes they affect study findings and, thus, “translate” to statistical heterogeneity. However, biases and chance can also result in statistical heterogeneity.
  • Clinical and Methodological Diversity Versus Statistical Heterogeneity Clinical and methodological diversity is abundant. Our aim in a meta-analysis is to explore that diversity and use our observations to formulate interesting hypotheses. Often clinical and methodological heterogeneity results in a statistically significant test. Chance, technical issues, or biases — such as publication or other reporting biases — can result in statistically significant results in heterogeneity tests.
  • Exploration of Heterogeneity Is Central to Evidence Synthesis
  • Dealing With Heterogeneity (I) What are the options, then, when dealing with between-study heterogeneity? From left to right, we have four options: Ignore any heterogeneity and always perform a meta-analysis with a fixed effects model (see the Quantitative Synthesis I module for an explanation of what a fixed effect model is). This is not a very informative approach. Perform a statistical test for heterogeneity (e.g., the chi-squared test for Cochran’s Q). If the test is nonsignificant, report a fixed effects meta-analysis. If the test is significant, do not perform a meta-analysis. There are many problems with this approach, both philosophical and practical. The major practical problem is that typical heterogeneity tests (such as Cochran’s Q) are underpowered (i.e., insensitive) when there are relatively few studies in a meta-analysis (e.g., 10 or less). This is fairly common for meta-analyses in medicine. The first two options are not recommended. Consider using a random effects model, which explicitly allows for between-study heterogeneity and incorporates it in the calculations. However, a random effects model provides a single overall estimate that is difficult to interpret and does not augment our understanding of the topic at hand. Always explore and explain between-study heterogeneity by performing subgroup analyses or meta-regression analyses. The latter are often preferable for statistical reasons. There are different types of meta-regression analyses, as will be discussed later.
  • Dealing With Heterogeneity (II) A meta-analysis provides a “grand mean,” that is, an overall effect. Put another way, a meta-analysis summarizes the evidence as a single point estimate (and a confidence interval). However, it offers no insights about why study results differ or what we can learn from between-study variability. Subgroup analyses describe differential effects in subgroups of studies. Subgroup analyses can be used to compare two or more point estimates (one per subgroup). Subgroup analyses can also be specified in a meta-regression framework, which is advantageous from a statistical point of view. Meta-regressions summarize the evidence with a line (and a confidence band) rather than a point. A meta-regression demonstrates how the effect changes at different levels of a variable of interest. Shown is a schematic of a meta-regression of treatment effect against a single variable of interest. Especially when modeling individual patient data, one can perform multivariable analyses that describe the treatment effect as a “response surface.” Meta-analysis of individual patient data essentially uses regression models, typically multilevel (hierarchical) ones. However, meta-analyses of individual patient data are costly and, therefore, are rarely performed.
  • Promises of Subgroup Analyses
  • Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I) Shown is a meta-analysis of thrombolytic therapy in patients with acute myocardial infarction, wherein studies are ordered by the average time-to-treatment from symptom onset (left panel) and the corresponding cumulative meta-analysis (right panel). In cumulative meta-analysis, we first order studies in a specific way (mean time-to-treatment initiation), and starting from the first study, we sequentially recalculate the summary effect each time we add the next study in our ordering. Cumulative meta-analysis shows the evolution of the evidence along the variable of interest (mean time-to-treatment initiation). This analysis is not a meta-regression . Cumulative meta-analysis is used here as is an exploratory tool. Statistically rigorous approaches to cumulative meta-analysis involve Bayesian modeling. Note that as the mean time-to-treatment initiation becomes longer, the cumulative effect becomes smaller (the odds ratio moves towards 1). This implies that the effect is larger (the odds ratio is smaller) in trials with shorter average time-to-treatment initiation when compared with trials that have longer average times. It makes sense to evaluate summary effects of subgroups of trials with time-to-treatment initiation at 0 to 3 hours, 3.1 to 5 hours, 5.1 to 10 hours, and longer than 10 hours. Optimally, decisions on how to select subgroups are made before performing analyses. We examine trials on the basis of aggregate information on a patient-level factor (the average of time-to-treatment initiation across all patients in each trial). However, the average time-to-treatment in any trial is not the time-to-treatment of the “average patient” (there is no such patient). Analyzing studies by grouping on aggregate statistics of patient-level factors , such as mean time-to-treatment, mean age, and proportion of women, may lead to ecological fallacy. This means that the relationship between the study-level treatment effect and the mean time-to-treatment initiation does not necessarily reflect the actual relationship between treatment results and time-to-treatment in individual patients. Therefore, we may observe spurious associations between the treatment effect and the levels of the aggregate factor. In this case, we know that the relationship is not spurious and that it has a pathophysiological explanation (ischemic damage and reperfusion injury).
  • Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II) Among the four subgroups, the summary effect is largest in this one (look at the last step of the cumulative meta-analysis in the right panel).
  • Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III) Here there are no notable differences in the summary effect from the previous plot.
  • Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV) Here the effect is smaller; the last odds ratio in the cumulative meta-analysis plot is closer to 1.
  • Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V) In this subgroup, the overall effect is even smaller and statistically nonsignificant.
  • Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Here is a meta-analysis of trials on vitamin E supplementation versus no supplementation and all-cause mortality. This is another case of a meta-analysis with subgroup analyses according to dose of vitamin E (above or below 400 IU/d). Note that contrary to the previous example, subgroups are defined by a study-level factor (protocol dose). Study-level factors apply equally to all patients in each trial and, therefore, cannot result in ecological fallacies. Other study-level factors could be characteristics of the design, such as using placebo versus no treatment in the comparator arm, having blinded versus nonblinded outcome assessment, and so on. So why was the cutoff put at 400 IU/d? Is the result of this subgroup-based exploration sensitive to the selection of the cutoff? The choice of the cutoff was more or less arbitrary, and this will be true for many subgroup analyses. We will revisit this example a bit later, when we discuss meta-regression. Reference: Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. http://www.ncbi.nlm.nih.gov/pubmed/15537682
  • Hazards of Subgroup Analyses: Multiple Testing (I)
  • Hazards of Subgroup Analyses: Multiple Testing (II)
  • Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (I) This graph summarizes subgroup analyses in the Second International Study of Infarct Survival (ISIS-2) for the comparison of intravenous streptokinase versus placebo and the outcome of vascular death. Note that there is no difference in treatment effect for deaths by prior myocardial infarction in the streptokinase versus placebo comparison. And it is likewise for the arbitrary grouping by astrological birth sign. Reference: ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60. http://www.ncbi.nlm.nih.gov/pubmed/2899772
  • Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (II) In the aspirin versus placebo comparison, however, there is a difference in treatment effect for deaths by prior myocardial infarction. There is also a difference for the Gemini and Libra astrological birth signs versus the rest. The authors used the astrological sign as a way to point out a common hazard of subgroup analyses, namely the risk of finding false-positive findings or inflating type-I error. Reference:: ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60. http://www.ncbi.nlm.nih.gov/pubmed/2899772
  • Can You Avoid the Hazards of Subgroup Analyses? (I)
  • Can You Avoid the Hazards of Subgroup Analyses? (II) Most meta-analyses use data that are published (and potentially known). In fact, when researchers prepare adequately before embarking on a meta-analysis (as they should ), they will inevitably become acquainted with the very data they will eventually analyze. Often, an experienced analyst can identify patterns in treatment effects across studies before running any analysis. Other times, meta-analyses exist on the same topic and may include the subgroup analyses in which a researcher is interested. Is it really meaningful, then, for researchers to claim that they have specified their subgroup analyses a priori ? Given these circumstances, it may be difficult for researchers to claim that they specified subgroups without knowing anything about their data.
  • Can You Avoid the Hazards of Subgroup Analyses? (III)
  • Beyond Subgroup Analyses: Meta-Regression
  • Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Let us revisit the subgroup analyses in the meta-analysis of vitamin E supplementation versus no supplementation and all-cause mortality. For that meta-analysis, we wondered whether the choice of cutoff for vitamin E dose would affect the conclusions. A more powerful statistical approach to exploring the effects of dose would be to use a meta-regression. A meta-regression is a regression of the summary effect versus different values of a predictor, either continuous or categorical. Reference : Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. http://www.ncbi.nlm.nih.gov/pubmed/15537682
  • Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality Meta-regressions are performed by using well-developed statistical methods that account for the precision of each study’s estimate of the treatment effect. More precise studies are weighted more heavily than less precise studies. In this example, a complex quadratic-linear spline model is used, which assumes a partly linear and partly nonlinear relationship between risk difference for all-cause mortality and vitamin E dose. The black spline line, rather than a point estimate, is now used to summarize the synthesis of the studies over different doses. Instead of a confidence interval around a point estimate, there is a confidence band around the “summary line.” Here the summary line suggests increased risk for mortality with high-dose supplementation. Although this is an example of a meta-regression analysis performed by using a continuous predictor, meta-regressions can be performed by using categorical predictors. Do not use unweighted regression methods (e.g., ordinary least squares regression for linear relationships) to perform meta-regression analyses. Specific models have been developed that are theoretically motivated and statistically sound. Use specialized software or specialized routines in general statistical packages. Reference : Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. http://www.ncbi.nlm.nih.gov/pubmed/15537682
  • Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo This is another example of a meta-regression that uncovers an interesting phenomenon. The treatment effect of zidovudine monotherapy on death in early HIV trials was time delimited. The effect of monotherapy is largest among treatment naïve patients in the first months, but then diminishes as resistance to the antiviral builds up. In the graph, trials with shorter followup show bigger effects, and trials with longer followup show smaller effects. In a simple meta-analysis, we would not see the differences in the followup across the 12 trials (we would be oblivious to the ordering of the studies along the x-axis). We would observe their scatter across the y-axis, which is indicative of their extensive statistical heterogeneity. Observe that the scatter of the studies along the meta-regression line is much smaller. From this perspective, meta-regression helps explain this heterogeneity, because it reduces the amount of residual heterogeneity. Reference: Ioannidis JPA. Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication bias in meta-analysis: prevention, assessment and adjustments . Chichester, England: Wiley; 2005. p. 287-302.
  • Two Types of Covariates in Meta-Regressions
  • Spurious Associations in Meta-Regressions or Subgroup Analyses (I)
  • Spurious Associations in Meta-Regressions or Subgroup Analyses (II)
  • Control-Rate Meta-Regression Often, there is no single covariate that can capture the baseline risk of the population in each study. Populations at high risk may benefit more from treatment when compared with populations at lower risk. The frequency of the events in the control group (e.g., comparator, usual care) of a randomized trial can be viewed as a surrogate of the baseline risk of the patients in the trial. It is interesting to explore whether or not there are systematic differences in the treatment effect across trials with different control rates. As we will discuss, this poses statistical challenges and necessitates use of proper analyses (i.e., proper meta-regression models). This is most meaningful to explore when an experimental treatment is compared with a placebo/usual-care/no-treatment group.
  • Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (I) Here is a visual exploration of whether the treatment effect changes with the control rate using cumulative meta-analysis. Trials are ordered by decreasing control rate. The treatment effect size evolves from larger (among trials with high control rates) to smaller (among more trials with low control rates).
  • Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (II) This is the reciprocal ordering of studies: from low to high control rate. Remember that cumulative meta-analysis is used as a graphical exploratory tool, in this example, and not to obtain inferences.
  • Control Rate Meta-Regression in the Preceding Streptokinase Example This control-rate meta-regression plot is based on data from the previous example. The treatment effect (log risk ratio) is plotted against the control rate of each trial. The squares are the observed estimates of the studies, and the circles are shrunken estimates from the control-rate meta-regression model. Note that the control-rate meta-regression is a special type of meta-regression fitted with a specific model. Intuitively, the difficulty is that the control rate (predictor) is included in the calculation of the response (effect, log risk ratio), and this induces a correlation between the errors of the response and the control rate. An ordinary meta-regression does not account for this complexity and can mislead. For example, the dashed line (sloping downward) shows a simple meta-regression (not correct), and the solid line (sloping upward) shows the meta-regression estimate from the proper model. The proper model shows a nonsignificant slope. The simple meta-regression model shows a downward slope (matching the direction of our observations from the cumulative meta-analysis graphs) but is not statistically correct. Reference: Schmid CH, Lau J, McIntosh MW, et al. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998;17:1923-42. http://www.ncbi.nlm.nih.gov/pubmed/9777687
  • Key Messages
  • References (I)
  • References (II)
  • Authors
  • Transcript of "Quantitative Synthesis II"

    1. 1. Quantitative Synthesis II Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov
    2. 2. Systematic Review Process Overview
    3. 3. <ul><li>To understand how to explore between-study heterogeneity in a meta-analysis </li></ul><ul><li>To understand the pros and cons of subgroup analyses </li></ul><ul><li>To understand what meta-regression is and why it is useful </li></ul><ul><li>To understand the usefulness of control-rate meta-regression </li></ul>Learning Objectives
    4. 4. Statistical Homogeneity Barrington KJ. Cochrane Database Syst Rev 2000;(2):CD000505.
    5. 5. Statistical Heterogeneity Reprinted from Pakos E, et al. J Bone Joint Surg Am 2005;87:1438-45, with permission from Rockwater, Inc.. Patellar Resurfacing in Total Knee Arthroplasty for Pain RE = random effects model
    6. 6. <ul><li>In meta-analysis, heterogeneity refers to between-study diversity. </li></ul><ul><li>This term is often used to refer to: </li></ul><ul><ul><li>differences in study characteristics, and </li></ul></ul><ul><ul><li>variability of study results. </li></ul></ul>What Is Heterogeneity?
    7. 7. <ul><li>Methodological diversity pertains to specifics of study design and analysis: </li></ul><ul><ul><li>Type of study </li></ul></ul><ul><ul><li>Length of followup </li></ul></ul><ul><ul><li>Proportion and handling of dropouts </li></ul></ul><ul><li>Clinical diversity pertains to differences in: </li></ul><ul><ul><li>Populations </li></ul></ul><ul><ul><li>Interventions and cointerventions </li></ul></ul><ul><ul><li>Outcomes </li></ul></ul>Methodological and Clinical Diversity
    8. 8. <ul><li>Statistical heterogeneity exists when the results of individual studies are not “consistent” among themselves. </li></ul>Statistical Heterogeneity Clinical diversity Methodological diversity Biases Chance Statistical heterogeneity
    9. 9. <ul><li>Clinical and methodological diversity is abundant: Our aim in a meta-analysis is to explore that diversity and use our observations to formulate interesting hypotheses. </li></ul><ul><li>Often clinical and methodological heterogeneity results in a statistically significant test. </li></ul><ul><li>Chance, technical issues, or biases may result in statistically significant results in heterogeneity tests. </li></ul>Clinical and Methodological Diversity Versus Statistical Heterogeneity
    10. 10. <ul><li>One of the most important roles of meta-analytic methodologies is to quantify statistical heterogeneity and to explore whether and to what extent it is explained by clinical and methodological diversity. </li></ul>Exploration of Heterogeneity Is Central to Evidence Synthesis
    11. 11. Dealing With Heterogeneity (I)
    12. 12. Dealing With Heterogeneity (II) OVERALL ESTIMATE Combining Summary Data SUBGROUP ANALYSES Differentiating Effects in Subgroups META-REGRESSION Modeling Summary Data RESPONSE SURFACE Modeling Individual Patient Data
    13. 13. <ul><li>Subgroup analyses can help: </li></ul><ul><ul><li>identify modifiers of the treatment effect, </li></ul></ul><ul><ul><li>recognize biologically interesting phenomena, or </li></ul></ul><ul><ul><li>formulate hypotheses. </li></ul></ul><ul><li>We discuss two illustrative examples: </li></ul><ul><ul><li>Time-to-thrombolysis affects the treatment effect of thrombolytic drugs in patients with acute myocardial infarction. </li></ul></ul><ul><ul><li>Effects of vitamin E supplementation on mortality may differ by vitamin E dose. </li></ul></ul>Promises of Subgroup Analyses
    14. 14. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I) Risk Ratio (95% Confidence Interval) Effects of Thrombolytic Therapy on Mortality in Patients With Acute Myocardial Infarction (Mean Time-to-Treatment)
    15. 15. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II) Largest Effect in the Subgroup of Trials With Mean Time-to-Treatment of 0 to 3 Hours
    16. 16. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III) Treatment Effect Diminishes When Mean Time-to-Treatment Is Between 3.1 and 5 Hours
    17. 17. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV) Treatment Effect Diminishes Further When Mean Time-to-Treatment Is Between 5.1 and 10 Hours
    18. 18. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V) No Evidence of a Treatment Effect When Mean Time to Treatment Longer than 10 Hours
    19. 19. Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med . 2005;142:37-46. Reprinted with permission from the American College of Physicians.
    20. 20. <ul><li>Subgroup analyses are a form of multiple testing. </li></ul><ul><li>When uncontrolled, multiple testing can yield spurious findings. </li></ul><ul><li>Most meta-analyses do not perform statistical adjustments for multiple testing. </li></ul>Hazards of Subgroup Analyses: Multiple Testing (I)
    21. 21. <ul><li>In the example that follows, we discuss subgroup analyses from the Second International Study of Infarct Survival (ISIS-2), a 2x2 factorial study of streptokinase versus placebo and aspirin versus placebo in more than 17,000 patients with myocardial infarction. </li></ul><ul><ul><li>For streptokinase versus placebo, the treatment effect does not differ across subgroups by history of prior myocardial infarction. </li></ul></ul><ul><ul><li>For aspirin versus placebo, the treatment effect does differ. </li></ul></ul>Hazards of Subgroup Analyses: Multiple Testing (II)
    22. 22. Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (I) Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.
    23. 23. Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (II) Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.
    24. 24. <ul><li>When analyzing data, it is important to distinguish subgroup analyses that are specified a priori (without knowing what the data are) versus those that are specified post hoc (after the researcher has been exposed to the data). </li></ul><ul><li>This distinction is very clear when analyzing a prospective study. </li></ul><ul><li>In most meta-analyses, the distinction is not as clear. </li></ul>Can You Avoid the Hazards of Subgroup Analyses? (I)
    25. 25. <ul><li>Most meta-analyses use data that are published (and potentially known). </li></ul><ul><li>When researchers adequately prepare before embarking on a meta-analysis, they inevitably become acquainted with the data they will analyze. </li></ul><ul><li>This makes it difficult for researchers to claim that they specified subgroups without knowing anything about their data. </li></ul>Can You Avoid the Hazards of Subgroup Analyses? (II)
    26. 26. <ul><li>Meta-analysts should do their best to define subgroups that make methodological and biological sense. </li></ul><ul><li>Treat the results of subgroup analyses with a healthy dose of skepticism, especially when adjustments for multiple testing are not performed. </li></ul>Can You Avoid the Hazards of Subgroup Analyses? (III)
    27. 27. <ul><li>Meta-regression can help examine how the treatment effect changes across the levels of a variable. </li></ul><ul><li>All subgroup analyses can be formulated in a meta-regression framework, but meta-regression goes well beyond subgroup analyses. </li></ul>Beyond Subgroup Analyses: Meta-Regression
    28. 28. Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med . 2005;142:37-46. Reprinted with permission from the American College of Physicians.
    29. 29. Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med . 2005;142:37-46. Reprinted with permission from the American College of Physicians.
    30. 30. Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo Ioannidis JPA. In: Publication bias in meta-analysis: prevention, assessment and adjustments . 2005.
    31. 31. <ul><li>Study level </li></ul><ul><ul><li>Examples: presence/absence of blinding, intervention dose (in experimental studies) </li></ul></ul><ul><li>Participant level </li></ul><ul><ul><li>Examples: mean age, proportion of diabetic patients, mean intake of vitamin E (in observational studies) </li></ul></ul>Two Types of Covariates in Meta-Regressions
    32. 32. <ul><li>Aggregate-data meta-regressions on participant-level covariates can mislead, because they are susceptible to ecological fallacy. </li></ul><ul><li>The observed relationship between the study-level treatment effect and the mean of a patient-level factor does not necessarily reflect the corresponding true relationship at the individual patient level. </li></ul>Spurious Associations in Meta-Regressions or Subgroup Analyses (I)
    33. 33. <ul><li>Therefore, associations of treatment effect and participant-level covariates should be interpreted with caution. </li></ul><ul><li>Such associations can be biologically plausible and informative, or can mislead. </li></ul><ul><li>Unfortunately, there is no universal way to distinguish true from fallacious findings. </li></ul>Spurious Associations in Meta-Regressions or Subgroup Analyses (II)
    34. 34. <ul><li>In control-rate meta-regression, we examine whether the treatment effect changes across studies with different event rates in the controls. </li></ul><ul><li>The control rate is a surrogate for the baseline risk for the event of interest (severity of disease). </li></ul><ul><li>This approach examines whether the underlying risk of an event explains differences in the treatment effect across studies. </li></ul>Control-Rate Meta-Regression
    35. 35. Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (I)
    36. 36. Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (II)
    37. 37. Control Rate Meta-Regression in the Preceding Streptokinase Example Schmid CH, et al. Stat Med 1998;17:1923-42.
    38. 38. <ul><li>Subgroup analyses, meta-regressions, and control-rate meta-regressions are tools to explore between-study heterogeneity. </li></ul><ul><ul><li>Use them to understand data. </li></ul></ul><ul><li>They are mostly hypothesis-forming tools. </li></ul><ul><ul><li>Especially for meta-regressions on patient-level covariates, ecological fallacy may mislead. </li></ul></ul><ul><li>Beware when interpreting their results. </li></ul>Key Messages
    39. 39. <ul><li>Barrington KJ. Umbilical artery catheters in the newborn: effects of position of the catheter tip. Cochrane Database Syst Rev 2000;(2):CD000505. </li></ul><ul><li>DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177-88. </li></ul><ul><li>ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60. </li></ul><ul><li>Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. </li></ul>References (I)
    40. 40. <ul><li>Pakos E, Ntzani EE, Trikalinos TA. Patellar resurfacing in total knee arthroplasty. A meta-analysis. J Bone Joint Surg Am 2005;87:1438-45. </li></ul><ul><li>Ioannidis JPA. Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester, England: Wiley; 2005. p. 287-302. </li></ul><ul><li>Schmid CH, Lau J, McIntosh MW, et al. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998;17:1923-42. </li></ul>References (II)
    41. 41. <ul><li>This presentation was prepared by Joseph Lau, M.D., and Thomas Trikalinos, M.D., Ph.D., members of the Tufts Medical Center Evidence-based Practice Center. </li></ul><ul><li>The information in this module is based on Chapter 9 in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf). </li></ul>Authors
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×