Mumbai Call Girls Service 9910780858 Real Russian Girls Looking Models
Quantitative Synthesis II
1. Quantitative Synthesis II Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov
5. Statistical Heterogeneity Reprinted from Pakos E, et al. J Bone Joint Surg Am 2005;87:1438-45, with permission from Rockwater, Inc.. Patellar Resurfacing in Total Knee Arthroplasty for Pain RE = random effects model
12. Dealing With Heterogeneity (II) OVERALL ESTIMATE Combining Summary Data SUBGROUP ANALYSES Differentiating Effects in Subgroups META-REGRESSION Modeling Summary Data RESPONSE SURFACE Modeling Individual Patient Data
13.
14. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I) Risk Ratio (95% Confidence Interval) Effects of Thrombolytic Therapy on Mortality in Patients With Acute Myocardial Infarction (Mean Time-to-Treatment)
15. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II) Largest Effect in the Subgroup of Trials With Mean Time-to-Treatment of 0 to 3 Hours
16. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III) Treatment Effect Diminishes When Mean Time-to-Treatment Is Between 3.1 and 5 Hours
17. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV) Treatment Effect Diminishes Further When Mean Time-to-Treatment Is Between 5.1 and 10 Hours
18. Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V) No Evidence of a Treatment Effect When Mean Time to Treatment Longer than 10 Hours
19. Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med . 2005;142:37-46. Reprinted with permission from the American College of Physicians.
20.
21.
22. Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (I) Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.
23. Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (II) Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.
24.
25.
26.
27.
28. Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med . 2005;142:37-46. Reprinted with permission from the American College of Physicians.
29. Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med . 2005;142:37-46. Reprinted with permission from the American College of Physicians.
30. Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo Ioannidis JPA. In: Publication bias in meta-analysis: prevention, assessment and adjustments . 2005.
Systematic Review Process Overview This slide illustrates the steps in the systematic review process. This module focuses on a further discussion of quantitative synthesis.
Learning Objectives
Statistical Homogeneity This is a forest plot of a meta-analysis with statistically homogeneous effects. All study estimates are very similar to each other. The confidence intervals from all studies are wide relative to the scatter of the point estimates. Formal statistical testing for heterogeneity in this case is consistent with our visual impression. When all studies have practically the same relative risk, there are not many things to explore: Any of the five studies, or any subset of the five studies, has practically the same risk ratio as the whole meta-analysis. Even if there is variation in the characteristics of the studies, there is no variability of effects to explore. However, in general, visual tests of statistical heterogeneity are precarious and should not be used in lieu of formal statistical testing. Reference: Barrington KJ, Umbilical artery catheters in the newborn: effects of position of the catheter tip. Cochrane Database Syst Rev 2000;(2):CD000505. http://www.ncbi.nlm.nih.gov/pubmed/10796375
Statistical Heterogeneity This forest plot is an extreme example of a meta-analysis with statistically heterogeneous effects: The point estimates are quite different when compared with the width of their respective confidence intervals. There is a 10-fold difference in the calculated relative risk between the first and third study. The confidence intervals of the 3rd and 5th studies do not overlap at all (i.e., each study excludes the other’s estimate). Formal statistical testing for heterogeneity suggests that this is a very heterogeneous meta-analysis. When there is substantial statistical heterogeneity, however, many interesting questions can be asked. Is there any explanation for the discrepancy between the 1st and 3rd studies? Might the definitions of anterior knee pain (the outcome) differ across studies? Could it be related to whether the assessment of pain was blinded versus nonblinded or recorded actively versus passively, or whether it was evaluated at shorter versus longer followup? Does it have to do with the mean ages of the participants in each study or with the type of knee prosthesis? To address these and other questions, we need to explore the observed between-study heterogeneity. We will discuss some of the tools that we can use for this exploration. In many cases, there will be no obvious explanation for the observed heterogeneity in the treatment effects. Other times, we may succeed in reaching a plausible hypothetical explanation. Reference: Pakos E, Ntzani EE, Trikalinos TA. Patellar resurfacing in total knee arthroplasty. A meta-analysis. J Bone Joint Surg Am 2005;87:1438-45. http://www.ncbi.nlm.nih.gov/pubmed/15995109
What Is Heterogeneity?
Methodological and Clinical Diversity Methodological diversity pertains to specifics of study design and analysis (e.g., type of study, length of followup, proportion and handling of dropouts). For example, particular types of studies may have found larger effects than others. Sometimes studies without blinded outcome assessments have systematically larger effects when compared to studies with blinded outcome assessments. Systematic differences are often suggestive of biases, and these variations in methodological characteristics may explain the observed statistical heterogeneity. Clinical diversity pertains to differences in the populations, intervention and cointerventions, and the definition of measured outcomes. For example, a higher drug dose may produce a larger effect size versus placebo when compared to a lower drug dose. Similarly, particular categories of patients may see larger benefits than other categories. Studies that exclusively enrolled patients with a condition of short duration may have larger effects than studies that exclusively enrolled patients with chronic conditions.
Statistical Heterogeneity In practice, clinical and methodological diversity are almost always present. Sometimes they affect study findings and, thus, “translate” to statistical heterogeneity. However, biases and chance can also result in statistical heterogeneity.
Clinical and Methodological Diversity Versus Statistical Heterogeneity Clinical and methodological diversity is abundant. Our aim in a meta-analysis is to explore that diversity and use our observations to formulate interesting hypotheses. Often clinical and methodological heterogeneity results in a statistically significant test. Chance, technical issues, or biases — such as publication or other reporting biases — can result in statistically significant results in heterogeneity tests.
Exploration of Heterogeneity Is Central to Evidence Synthesis
Dealing With Heterogeneity (I) What are the options, then, when dealing with between-study heterogeneity? From left to right, we have four options: Ignore any heterogeneity and always perform a meta-analysis with a fixed effects model (see the Quantitative Synthesis I module for an explanation of what a fixed effect model is). This is not a very informative approach. Perform a statistical test for heterogeneity (e.g., the chi-squared test for Cochran’s Q). If the test is nonsignificant, report a fixed effects meta-analysis. If the test is significant, do not perform a meta-analysis. There are many problems with this approach, both philosophical and practical. The major practical problem is that typical heterogeneity tests (such as Cochran’s Q) are underpowered (i.e., insensitive) when there are relatively few studies in a meta-analysis (e.g., 10 or less). This is fairly common for meta-analyses in medicine. The first two options are not recommended. Consider using a random effects model, which explicitly allows for between-study heterogeneity and incorporates it in the calculations. However, a random effects model provides a single overall estimate that is difficult to interpret and does not augment our understanding of the topic at hand. Always explore and explain between-study heterogeneity by performing subgroup analyses or meta-regression analyses. The latter are often preferable for statistical reasons. There are different types of meta-regression analyses, as will be discussed later.
Dealing With Heterogeneity (II) A meta-analysis provides a “grand mean,” that is, an overall effect. Put another way, a meta-analysis summarizes the evidence as a single point estimate (and a confidence interval). However, it offers no insights about why study results differ or what we can learn from between-study variability. Subgroup analyses describe differential effects in subgroups of studies. Subgroup analyses can be used to compare two or more point estimates (one per subgroup). Subgroup analyses can also be specified in a meta-regression framework, which is advantageous from a statistical point of view. Meta-regressions summarize the evidence with a line (and a confidence band) rather than a point. A meta-regression demonstrates how the effect changes at different levels of a variable of interest. Shown is a schematic of a meta-regression of treatment effect against a single variable of interest. Especially when modeling individual patient data, one can perform multivariable analyses that describe the treatment effect as a “response surface.” Meta-analysis of individual patient data essentially uses regression models, typically multilevel (hierarchical) ones. However, meta-analyses of individual patient data are costly and, therefore, are rarely performed.
Promises of Subgroup Analyses
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I) Shown is a meta-analysis of thrombolytic therapy in patients with acute myocardial infarction, wherein studies are ordered by the average time-to-treatment from symptom onset (left panel) and the corresponding cumulative meta-analysis (right panel). In cumulative meta-analysis, we first order studies in a specific way (mean time-to-treatment initiation), and starting from the first study, we sequentially recalculate the summary effect each time we add the next study in our ordering. Cumulative meta-analysis shows the evolution of the evidence along the variable of interest (mean time-to-treatment initiation). This analysis is not a meta-regression . Cumulative meta-analysis is used here as is an exploratory tool. Statistically rigorous approaches to cumulative meta-analysis involve Bayesian modeling. Note that as the mean time-to-treatment initiation becomes longer, the cumulative effect becomes smaller (the odds ratio moves towards 1). This implies that the effect is larger (the odds ratio is smaller) in trials with shorter average time-to-treatment initiation when compared with trials that have longer average times. It makes sense to evaluate summary effects of subgroups of trials with time-to-treatment initiation at 0 to 3 hours, 3.1 to 5 hours, 5.1 to 10 hours, and longer than 10 hours. Optimally, decisions on how to select subgroups are made before performing analyses. We examine trials on the basis of aggregate information on a patient-level factor (the average of time-to-treatment initiation across all patients in each trial). However, the average time-to-treatment in any trial is not the time-to-treatment of the “average patient” (there is no such patient). Analyzing studies by grouping on aggregate statistics of patient-level factors , such as mean time-to-treatment, mean age, and proportion of women, may lead to ecological fallacy. This means that the relationship between the study-level treatment effect and the mean time-to-treatment initiation does not necessarily reflect the actual relationship between treatment results and time-to-treatment in individual patients. Therefore, we may observe spurious associations between the treatment effect and the levels of the aggregate factor. In this case, we know that the relationship is not spurious and that it has a pathophysiological explanation (ischemic damage and reperfusion injury).
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II) Among the four subgroups, the summary effect is largest in this one (look at the last step of the cumulative meta-analysis in the right panel).
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III) Here there are no notable differences in the summary effect from the previous plot.
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV) Here the effect is smaller; the last odds ratio in the cumulative meta-analysis plot is closer to 1.
Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V) In this subgroup, the overall effect is even smaller and statistically nonsignificant.
Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Here is a meta-analysis of trials on vitamin E supplementation versus no supplementation and all-cause mortality. This is another case of a meta-analysis with subgroup analyses according to dose of vitamin E (above or below 400 IU/d). Note that contrary to the previous example, subgroups are defined by a study-level factor (protocol dose). Study-level factors apply equally to all patients in each trial and, therefore, cannot result in ecological fallacies. Other study-level factors could be characteristics of the design, such as using placebo versus no treatment in the comparator arm, having blinded versus nonblinded outcome assessment, and so on. So why was the cutoff put at 400 IU/d? Is the result of this subgroup-based exploration sensitive to the selection of the cutoff? The choice of the cutoff was more or less arbitrary, and this will be true for many subgroup analyses. We will revisit this example a bit later, when we discuss meta-regression. Reference: Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. http://www.ncbi.nlm.nih.gov/pubmed/15537682
Hazards of Subgroup Analyses: Multiple Testing (I)
Hazards of Subgroup Analyses: Multiple Testing (II)
Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (I) This graph summarizes subgroup analyses in the Second International Study of Infarct Survival (ISIS-2) for the comparison of intravenous streptokinase versus placebo and the outcome of vascular death. Note that there is no difference in treatment effect for deaths by prior myocardial infarction in the streptokinase versus placebo comparison. And it is likewise for the arbitrary grouping by astrological birth sign. Reference: ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60. http://www.ncbi.nlm.nih.gov/pubmed/2899772
Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (II) In the aspirin versus placebo comparison, however, there is a difference in treatment effect for deaths by prior myocardial infarction. There is also a difference for the Gemini and Libra astrological birth signs versus the rest. The authors used the astrological sign as a way to point out a common hazard of subgroup analyses, namely the risk of finding false-positive findings or inflating type-I error. Reference:: ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60. http://www.ncbi.nlm.nih.gov/pubmed/2899772
Can You Avoid the Hazards of Subgroup Analyses? (I)
Can You Avoid the Hazards of Subgroup Analyses? (II) Most meta-analyses use data that are published (and potentially known). In fact, when researchers prepare adequately before embarking on a meta-analysis (as they should ), they will inevitably become acquainted with the very data they will eventually analyze. Often, an experienced analyst can identify patterns in treatment effects across studies before running any analysis. Other times, meta-analyses exist on the same topic and may include the subgroup analyses in which a researcher is interested. Is it really meaningful, then, for researchers to claim that they have specified their subgroup analyses a priori ? Given these circumstances, it may be difficult for researchers to claim that they specified subgroups without knowing anything about their data.
Can You Avoid the Hazards of Subgroup Analyses? (III)
Beyond Subgroup Analyses: Meta-Regression
Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Let us revisit the subgroup analyses in the meta-analysis of vitamin E supplementation versus no supplementation and all-cause mortality. For that meta-analysis, we wondered whether the choice of cutoff for vitamin E dose would affect the conclusions. A more powerful statistical approach to exploring the effects of dose would be to use a meta-regression. A meta-regression is a regression of the summary effect versus different values of a predictor, either continuous or categorical. Reference : Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. http://www.ncbi.nlm.nih.gov/pubmed/15537682
Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality Meta-regressions are performed by using well-developed statistical methods that account for the precision of each study’s estimate of the treatment effect. More precise studies are weighted more heavily than less precise studies. In this example, a complex quadratic-linear spline model is used, which assumes a partly linear and partly nonlinear relationship between risk difference for all-cause mortality and vitamin E dose. The black spline line, rather than a point estimate, is now used to summarize the synthesis of the studies over different doses. Instead of a confidence interval around a point estimate, there is a confidence band around the “summary line.” Here the summary line suggests increased risk for mortality with high-dose supplementation. Although this is an example of a meta-regression analysis performed by using a continuous predictor, meta-regressions can be performed by using categorical predictors. Do not use unweighted regression methods (e.g., ordinary least squares regression for linear relationships) to perform meta-regression analyses. Specific models have been developed that are theoretically motivated and statistically sound. Use specialized software or specialized routines in general statistical packages. Reference : Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46. http://www.ncbi.nlm.nih.gov/pubmed/15537682
Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo This is another example of a meta-regression that uncovers an interesting phenomenon. The treatment effect of zidovudine monotherapy on death in early HIV trials was time delimited. The effect of monotherapy is largest among treatment naïve patients in the first months, but then diminishes as resistance to the antiviral builds up. In the graph, trials with shorter followup show bigger effects, and trials with longer followup show smaller effects. In a simple meta-analysis, we would not see the differences in the followup across the 12 trials (we would be oblivious to the ordering of the studies along the x-axis). We would observe their scatter across the y-axis, which is indicative of their extensive statistical heterogeneity. Observe that the scatter of the studies along the meta-regression line is much smaller. From this perspective, meta-regression helps explain this heterogeneity, because it reduces the amount of residual heterogeneity. Reference: Ioannidis JPA. Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication bias in meta-analysis: prevention, assessment and adjustments . Chichester, England: Wiley; 2005. p. 287-302.
Two Types of Covariates in Meta-Regressions
Spurious Associations in Meta-Regressions or Subgroup Analyses (I)
Spurious Associations in Meta-Regressions or Subgroup Analyses (II)
Control-Rate Meta-Regression Often, there is no single covariate that can capture the baseline risk of the population in each study. Populations at high risk may benefit more from treatment when compared with populations at lower risk. The frequency of the events in the control group (e.g., comparator, usual care) of a randomized trial can be viewed as a surrogate of the baseline risk of the patients in the trial. It is interesting to explore whether or not there are systematic differences in the treatment effect across trials with different control rates. As we will discuss, this poses statistical challenges and necessitates use of proper analyses (i.e., proper meta-regression models). This is most meaningful to explore when an experimental treatment is compared with a placebo/usual-care/no-treatment group.
Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (I) Here is a visual exploration of whether the treatment effect changes with the control rate using cumulative meta-analysis. Trials are ordered by decreasing control rate. The treatment effect size evolves from larger (among trials with high control rates) to smaller (among more trials with low control rates).
Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (II) This is the reciprocal ordering of studies: from low to high control rate. Remember that cumulative meta-analysis is used as a graphical exploratory tool, in this example, and not to obtain inferences.
Control Rate Meta-Regression in the Preceding Streptokinase Example This control-rate meta-regression plot is based on data from the previous example. The treatment effect (log risk ratio) is plotted against the control rate of each trial. The squares are the observed estimates of the studies, and the circles are shrunken estimates from the control-rate meta-regression model. Note that the control-rate meta-regression is a special type of meta-regression fitted with a specific model. Intuitively, the difficulty is that the control rate (predictor) is included in the calculation of the response (effect, log risk ratio), and this induces a correlation between the errors of the response and the control rate. An ordinary meta-regression does not account for this complexity and can mislead. For example, the dashed line (sloping downward) shows a simple meta-regression (not correct), and the solid line (sloping upward) shows the meta-regression estimate from the proper model. The proper model shows a nonsignificant slope. The simple meta-regression model shows a downward slope (matching the direction of our observations from the cumulative meta-analysis graphs) but is not statistically correct. Reference: Schmid CH, Lau J, McIntosh MW, et al. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998;17:1923-42. http://www.ncbi.nlm.nih.gov/pubmed/9777687
