call girls in aerocity DELHI 🔝 >༒9540349809 🔝 genuine Escort Service 🔝✔️✔️
Quantitative Synthesis I
1. Quantitative Synthesis I Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov
12. Effect of the Choice of Metric on Meta-analysis Treatment Control Study Events Total Rate Events Total Rate Relative Risk Risk Difference A 100 1000 10% 200 1000 20% 0.5 10% B 1 1000 0.1% 2 1000 0.2% 0.5 0.1%
13. Effect of Small Changes on the Estimate Baseline case Effect of decrease of 1 event Effect of increase of 1 event Relative change of estimate 2/10 20% 1/10 10% 3/10 30% ±50% 20/100 20% 19/100 19% 21/100 21% ±5% 200/1,000 20% 199/1,000 19.9% 201/1,000 20.1% ±0.5%
14.
15. A Sample 2x2 Table ISIS-2 Collaborative Group. Lancet 1988;2:349-60. Binary outcomes data to be extracted from studies Vascular deaths Survival Total Streptokinase 791 7,801 8,592 Placebo 1,029 7,566 8,595
22. Treatment Effects Estimates in Different Metrics: S econd International Study of Infarct Survival ( ISIS-2) ISIS-2 Collaborative Group. Lancet 1988;2:349-60. Streptokinase vs. Placebo Vascular Death Estimate 95% Confidence Interval Risk ratio 0.77 0.70 to 0.84 Odds ratio 0.75 0.68 to 0.82 Risk difference -0.028 -0.037 to -0.019 Number needed to treat 36 27 to 54
25. Simpson’s Paradox (II) Stones < 2 cm Stones ≥ 2 cm Pooling Tables 1 and 2 Open (93%) > PN (87%) Open (73%) > PN (69%) Open (78%) < PN (83%) Charig CR, et al. BMJ 1986;292:879-82. PN = percutaneous nephrolithotomy Success Failure Open 81 6 PN 234 36 Success Failure Open 192 71 PN 55 25 Success Failure Open 273 77 PN 289 61
26. Combining Effect Estimates What is the average (overall) treatment -c ontrol difference in blood pressure? Study N Mean difference (mm Hg) 95% Confidence Interval A 554 -6.2 -6.9 to -5.5 B 304 -7.7 -10.2 to -5.2 C 39 -0.1 -6.5 to 6.3
27. Simple Average What is the average (overall) treatment -c ontrol difference in blood pressure? (-6.2) + (-7.7) + (-0.1) 3 = -4.7 mm Hg Study N Mean difference mmHg 95% CI A 554 -6.2 -6.9 to -5.5 B 304 -7.7 -10.2 to -5.2 C 39 -0.1 -6.5 to 6.3
28. Weighted Average What is the average (overall) treatment -c ontrol difference in blood pressure? (554 x - 6.2) + (304 x - 7.7) + (39 x - 0.1) 554 + 304 + 39 = - 6.4 mm Hg Study N Mean difference mmHg 95% CI A 554 -6.2 -6.9 to -5.5 B 304 -7.7 -10.2 to -5.2 C 39 -0.1 -6.5 to 6.3
29.
30.
31.
32.
33.
34.
35.
36. Cochran’s Q-Statistics: Chi-square ( 2 ) Test for Homogeneity d i = effect measure; d + = weighted average Q-statistics measure between-study variation.
37.
38.
39. Random Sampling From a Container With a Fixed Number of White and Black Balls (Equal Sample Size)
40. Random Sampling From a Container With a Fixed Number of Black and White Balls (Different Sample Size)
45. Example Meta-analysis Where Results from Fixed and Random Effects Models Will Differ Gross PA, et al. Inn Intern Med 1995;123:518-27. Reprinted with permission from the American College of Physicians.
46. Weights of the Fixed Effect and Random Effects Models Random Effects Weight Fixed Effect Weight where: v i = within study variance v * = between study variance
47.
48. Dealing With Heterogeneity Lau J, et al. Ann Intern Med 1997;127:820-6. Reprinted with permission from the American College of Physicians.
49.
50.
51.
52.
53.
54.
Editor's Notes
Quantitative Synthesis I
Systematic Review Process Overview This slide illustrates the steps in the systematic review process. This module focuses on quantitative synthesis.
Learning Objectives
Synonyms for Meta-Analysis
Reasons To Conduct Meta-Analyses
Commonly Encountered Comparative Effect Measures This table shows the type of data and corresponding effect measures that are commonly encountered in meta-analyses. For continuous data, the mean difference is most often used. Standardized mean difference, or “effect size,” is a dimensionless standardized unit; it is seldom used in medical meta-analyses, because it lacks direct clinical interpretation. Corresponding metrics for dichotomous data include the odds ratio, risk ratio, and risk difference. Hazard ratio is used in time-of-event meta-analyses.
Principles of Combining Data for Basic Meta-Analyses
Things To Know About the Data Before Combining Them The biological and clinical plausibility of the research question should be addressed first when deciding whether or not to carry out a meta-analysis. Will the answer to the research question be clinically meaningful? As will be discussed later, the scale of the effect measure may influence the decision of whether or how to combine data. Estimates from studies with small numbers of patients or studies with few events are unstable. The results of a meta-analysis based only on small studies, even if statistically significant, should be interpreted cautiously. Additional points about data being considered for meta-analysis are illustrated with hypothetical examples in the next five slides.
True Associations May Disappear When Data Are Combined Inappropriately In this figure, the two hypothetical groups of data show a strong correlation between the variable of interest and the effect of interest. Any relationship between variables can be obscured by inappropriate aggregation of data. Imagine that the oblongs represent two studies; if one were to conduct a meta-analysis by using only the mean values, the correlation would be lost. The correct approach would be to take the correlation (or effect) in each of the studies and use the weighted average.
An Association May Be Seen When There Is None In contrast with the last slide, there is no correlation of the variable with the effect within each data group. A correlation appears, however, if we inappropriately aggregate the data by ignoring the fact that the data are from two separate studies that may have addressed different research questions or that there is no clinical rationale to combine the data.
Changes in the Same Scale May Have Difference Meanings This hypothetical example illustrates the importance of appreciating baseline values and the scale used to measure them when interpreting changes in data. It can be seen that the same absolute change of one unit at both locations (A to B, C to D) has different meaning in term of relative change. A to B represents a 100% relative change, whereas C to D represents only a 14% relative change. Meta-analysis of studies that have different baseline values may not be appropriate. Serum creatinine is such an example; the same absolute increase of measurement from 1.0 to 1.5 mg/dL has very different clinical implications than a change from 6.0 to 6.5 mg/dL.
Effect of the Choice of Metric on Meta-analysis In this hypothetical example, two studies with the same number of patients and different event rates have identical relative risk but very different absolute risks (risk difference). A meta-analysis of the relative risk scale would be appropriate, but a meta-analysis of the absolute risk scale would probably not be appropriate.
Effect of Small Changes on the Estimate In this hypothetical example of three studies with different numbers of patients, the decrease or increase of one event (perhaps due to error of misclassification) leads to a very different effect on the estimate. The greatest effect will be seen in the small studies.
Binary Outcomes Sometimes continuous variables are dichotomized into binary outcomes. A threshold value may be used to categorize pain scores as improved or not improved. This approach is suboptimal because of information loss and because the choice of the threshold for defining success is often arbitrary.
A Sample 2x2 Table Reference: ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60. http://www.ncbi.nlm.nih.gov/pubmed/2899772
Treatment Effect Metrics That Can Be Calculated From a 2x2 Table Three treatment effect metrics can be calculated from a 2x2 table: the risk difference, odds ratio, and risk ratio. By convention, the risk difference is the treatment-group event rate minus the control-group event rate. If the outcome is undesirable, a negative risk difference is interpreted as a reduction of the bad outcome. The odds ratio is the ratio of the odds of having an event in the treatment group to the odds of having an event in the control group. The risk ratio is equal to the treatment-group event rate divided by the control-group event rate.
Some Characteristics and Uses of the Risk Difference
Some Characteristics and Uses of the Odds Ratio
Some Characteristics and Uses of the Risk Ratio
When the Complementary Outcome of the Risk Ratio Is Asymmetric This slide demonstrates that the complementary outcome for the odds ratio is symmetrical but the complimentary outcome for the risk ratio is not. Thus, one needs to be careful in using risk ratio as the outcome measure.
Calculation of Treatment Effects in the Second International Study of Infarct Survival (ISIS-2) This is an example of how metrics were calculated for one study. The metrics for all studies included in a meta-analysis are similarly calculated.
Treatment Effects Estimates in Different Metrics: Second International Study of Infarct Survival (ISIS-2) The methods of calculating the standard error and confidence interval for individual studies are not shown in this presentation. Many books include these formulae, and meta-analysis computer software typically performs and reports these calculations.
Example: Meta-Analysis Data Set In a meta-analysis, we are interested in an overall estimate of the effect on studies that meet eligibility criteria for a research question of interest. This slide depicts the outcome data of 17 studies assembled for a meta-analysis. The next few slides address the issues of how we should combine the data.
Simpson’s Paradox This example illustrates why pooling counts from 2x2 tables across studies into a single table may yield misleading results. This phenomenon is known as the Simpson’s Paradox. Reference: Charig CR, Webb DR, Payne, SR, et al. Comparison of treatment of renal calculi by operative surgery, percutaneous nephrolithotomy, and extracorporeal shock wave lithotripsy. BMJ 1986;292:879-82. http://www.ncbi.nlm.nih.gov/pubmed/3083922
Simpson’s Paradox (II) Each stone-size category reveals that open surgery leads to higher rate of success, but pooling the raw data yields paradoxical results. In this example, the paradox is due to the different rates of open surgery and percutaneous nephrolithotomy in patients with different stone sizes. Applying this concept to meta-analysis, calculating the study effect first and then combining the effects estimate across studies eliminates this problem. Reference: Charig CR, Webb DR, Payne, SR, et al. Comparison of treatment of renal calculi by operative surgery, percutaneous nephrolithotomy, and extracorporeal shock wave lithotripsy. BMJ 1986;292:879-82. http://www.ncbi.nlm.nih.gov/pubmed/3083922
Combining Effect Estimates This slide and the next few discuss the basic concept of combining the effect estimates from studies included in a meta-analysis. We can use the diastolic blood pressure reduction data from three studies to illustrate how an average (overall) effect is estimated. Notice the three studies differ along three parameters: 1. Studies 1 and 2 have similar clinical effects: a respective reduction in blood pressure of -6.2 and -7.7 mm Hg reduction in blood pressure. 2. Study 3 has no clinical effect; it has a much smaller sample size than the first two. 3. Study 3 also has a wider confidence interval and is, therefore, less precise in its estimate of the average (overall) effect. There are two ways to average these data: a simple average and a weighted average. We explore the implications of these approaches in the following slides.
Simple Average The simple average for the three blood pressure studies is -4.7 mmHg. However, this simple average is not a useful average for the three studies. A simple average weights the contribution of each study equally. When there are important differences in the sample size or in the results across studies, the simple average does not provide a reasonable answer that reflects the contribution of the data. Intuitively, we believe that larger studies should be trusted more than smaller ones.
37 Weighted Average The weighted average for blood pressure reduction, when weighted by sample size, is -6.4 mm Hg. This average is more consistent with the results of the first two studies, which were more precise. Although weighting by sample size is simplistically appealing, it offers only a rough approximation of the precision of the estimators. In this case, where the outcome is continuous, the better weight to use is the inverse of the variance/N. Consider two studies with equal sample sizes. It is common for the samples in the two studies to have different amounts of dispersion. Thus, the estimates of the mean would have different degrees of precision. Later, we will examine methods for weighting that incorporate both sample size and precision. The approach to calculating the weighted average for a binary outcome is similar. The variance formula for binary outcomes depends upon the metric selected for the meta-analysis and uses the number of events and sample size in the calculation.
38 General Formula: Weighted Average Effect Size (d + ) This general formula is almost identical to the weighted average formula we used earlier for the three blood pressure studies. The one exception is that the letter “d” is used to represent the measure of effect in this formula, whereas the previous formula used the letter “x.” This formula can be used to calculate weighted averages for any type of data, even those from one-arm studies. In some cases, when using relative measures, such as relative risk and the odds ratio, the effect must be first transformed. In the case of relative risk and the odds ratio, the transformation would be its natural logarithm. The primary difference in using this formula for various meta-analytic data sets is how the weights, represented by the letter “w i ,” are calculated.
Calculation of Weights
Heterogeneity (Diversity) It may be easier to think of homogeneity rather than heterogeneity. Homogeneity is the degree to which studies are sufficiently similar, such that a single average effect would be meaningful. Homogeneity is the basis for the “apples and oranges” debate on which many critics of this science base their argument. A useful way to consider homogeneity is from two perspectives: clinical and methodological diversity, and statistical heterogeneity.
Clinical Diversity
Example: A Meta-analysis With a Large Degree of Clinical Diversity This is an example of a meta-analysis with a large degree of clinical diversity in the included studies. Many different combinations of endoscopic treatments, conditions, and outcomes are possible. Given the wide range of clinical diversity, one might ask, “Should a meta-analysis be carried out?” If one uses a narrow set of criteria (e.g., a comparison of argon laser treatment with placebo in patients with active bleeding, using mortality as the outcome) for meta-analysis, none of the 25 randomized controlled trial might be suitable for a meta-analysis. However, if one frames a broader question that is nonetheless clinically meaningful (combining multiple categories into one), meta-analysis may be appropriate. For example, if we accept that all three listed outcomes are undesirable and that reducing their rates is the key question, then it may make sense to combine the outcomes into one category called “undesirable outcomes.” This decision requires clinical judgment and input from content experts. Managing clinical diversity is a key issue in meta-analysis and will be discussed further in the next module. Reference: Sacks HS, Chalmers TC, Blum AL, et al. Endoscopic hemostasis: an effective therapy for bleeding peptic ulcers. JAMA 1990;264:494-9. http://www.ncbi.nlm.nih.gov/pubmed/2142225
Methodological Diversity Methodological diversity refers to variations in the design and conduct of clinical trials. It includes issues such as (but not limited to) blinded or unblinded studies, method of random allocation, and choice of measurement methods. The impact of methodological diversity can be explored by examining the differential effects of subgroups.
Statistical Heterogeneity The next several slides discuss the statistical models used to combine data in a meta-analysis. The key concepts discussed are the fixed effect model and the random effects model.
Cochran’s Q-Statistics: Chi-square ( 2 ) Test for Homogeneity This slide depicts the formula for Q-statistics. Under the null hypothesis of homogeneity, the Q-statistics follow a 2 distribution with k-1 degrees of freedom. When the value of the Q-statistic is too high, the null hypothesis of homogeneity is rejected. The 2 test, however, has low sensitivity to detect heterogeneity. A significance level of 0.1 instead of 0.05 is generally recommended. Any of the effect measures described earlier can be used with appropriate transformation and corresponding inverse variance weights. The purpose of showing the formula is not to teach reviewers how to calculate but to convey the idea that Q-statistics measure between-study variation.
The I 2 Index and Its Interpretation Reference: Higgins JP, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557–60.
Example: A Fixed Effect Model
Random Sampling From a Container With a Fixed Number of White and Black Balls (Equal Sample Size) In this illustration, to estimate the proportion of black and white balls, we performed 6 draws of equal sample size (10 balls each). If the sample size is small (as shown here), there will likely be wide variations of results across samples. Each sample will not be a good estimate of the fixed proportion. However, if we combine the results (i.e., do a meta-analysis) across the samples, we will get a more reliable estimate.
Random Sampling From a Container With a Fixed Number of Black and White Balls (Different Sample Size) In this illustration, we drew five samples, but the sample size of each draw is different. A weighted average of the samples should be used to estimate the proportion of black and white balls.
Different Containers With Different Proportions of Black and White Balls (Random Effects Model) This slide illustrates the meaning of a random effects model. Instead of a single container, now we have seven containers with different proportions of black and white balls. We are interested in estimating the overall proportion of the black and white balls in all the containers. This illustration is analogous to a clinical scenario of many different population groups, each with a different response rate to a treatment. Nonetheless, we are interested in knowing the overall effect of an intervention in the entire population.
Random Sampling From Containers To Get an Overall Estimate of the Proportion of Black and White Balls If we are interested in an overall estimate of the proportion of black and white balls in all the containers, we would randomly sample from these containers and then calculate a weighted average. Note that not all the containers are sampled because there may be too many or by chance or by design we will not draw balls from them. We assume that the proportion of black and white balls in all the containers follows a normal distribution. The weight to be used for each study in this setting must consider both within-sample variations and between-sample variations.
Statistical Models of Combining 2x2 Tables
Example Meta-analysis Where Fixed and the Random Effects Models Yield Identical Results This slide presents a forest plot of a meta-analysis that compared newer antibiotics with amoxicillin for acute sinusitis. The risk ratios of 13 studies are plotted with their respective 95% confidence intervals. The estimates of the risk of clinical failure of each study hovers around 1 for all studies, suggesting no difference in results (the studies are homogeneous) across these studies. Not surprisingly, the overall estimate of the meta-analysis reflects this observation. In this data set the DerSimonian and Laird random effects model’s estimate of between-study heterogeneity is 0. In this case, the results from the random effects model are identical to those of the fixed effects model. The chi-squared test discussed earlier could be used to test for between-study heterogeneity. In this particular example it is not significant.
Example Meta-analysis Where Results from Fixed and Random Effects Models Will Differ This slide presents a forest plot of 23 studies used in a meta-analysis of observational studies on the efficacy of influenza vaccination in elderly people. There is large degree of heterogeneity in the odds ratios (ranging from 0.04 to 1.1) of overall mortality. The 95% confidence intervals of many studies do not overlap each other. This suggests that a fixed effects model will fit these data poorly. In this data set the DerSimonian and Laird random effects model’s estimate of between-study heterogeneity is non-0, and therefore its results will differ from those of the fixed effects model. Further, studies were conducted in different years (thus using different strains of viruses) and in different settings. The efficacy of vaccination is likely to vary with different viruses and in different populations; Therefore, there is probably a lot of clinical diversity, and the assumptions made by a fixed effects model (single summary effect) are probably not true. So which model to pick? In most meta-analyses clinical and methodological diversity are abundant, and therefore the basic assumption of the fixed effects model that there is a single treatment effect is probably too strong. Therefore, in most meta-analyses the random effects model is more justifiable. There are exceptions, but we will not discuss them here. In the meta-analysis in this example, a random effects model should be used, and the I2 index should be useful to quantify the extent of heterogeneity. Reference: Gross PA, Hermogenes AW, Sacks HS, et al. The efficacy of Influenza vaccine in elderly persons: a meta-analysis and review of the literature. Ann Intern Med 1995;123:518-27. http://www.ncbi.nlm.nih.gov/pubmed/7661497
Weights of the Fixed Effect and Random Effects Models This slide depicts two weight formulas: one for the fixed effect model and one for the random effects model. The difference between the fixed effect weight and the random effects weight is that the latter incorporates the between-study variance. When a group of studies is statistically homogeneous, the between-study variance is small (or zero). As the studies become increasingly heterogeneous, the between-study variance increases. The implications of this relationship are as follows: 1. When studies are homogenous (i.e., when between-study variance is small), the fixed effect and random effects study-specific weights are essentially equal, which, overall, results in similar weighted average effects and confidence intervals. As the between-study variance increases (increasing heterogeneity), the two weights will become increasingly different. If the between-study variance is large enough, this value becomes dominant, and the random effects weights for every study move toward equality. In cases of extreme heterogeneity, the random effects model may give what appears to be counterintuitive results. However, if one bears in mind the effect of the between-study variance component in the formula, this is entirely consistent with the formulation of this approach. In situations that give rise to counterintuitive results, the wisdom of doing a meta-analysis should be questioned. Alternative approaches to quantitatively synthesize data, such as meta-regression, should be considered.
Commonly Used Statistical Methods for Combining 2x2 Tables
Dealing With Heterogeneity This block diagram depicts four general approaches and the specific methods that can be used to handle heterogeneity that may be encountered in a meta-analysis. The first approach is to ignore heterogeneity and combine data using a fixed effect model. The second approach is to test for heterogeneity using a chi-squared test and to not combine the data if the test is significant. In the third approach, if after assessing heterogeneity of the data it is deemed appropriate to combine results to provide an overall effect, then the random effects model is used. The fourth approach is to seek to explain heterogeneity by performing subgroup analyses or meta-regressions. The next module focuses on exploring and identifying reasons for heterogeneity. Reference: Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic review. Ann Intern Med 1997;127:820-6. http://www.ncbi.nlm.nih.gov/pubmed/9382404
Statistical Models of Combining 2x2 Tables: Summary
