DEFINITION Meta-analysis is a quantitative approach for systematically combining results of previous research to arrive at conclusions about the body of research. Quantitative: numbers Systematic: methodical Combining: putting together Previous research: whats already done Conclusions: new knowledge A study collects data from individual subjects (such as 100 subjects = 100 “data points”) A meta-analysis collects data from individual studies (such as 100 studies = 100 “data points”)
HISTORY Karl Pearson is probably the first medical researcher to use formal techniques to combine data from different studies (1904): He synthesized data from several studies on efficacy of typhoid vaccination His rationale for pooling data: “Many of the groups… are far too small to allow of any definite opinion being formed at all, having regard to the size of the probable error involved.”
THE TERM META-ANALYSIS 1952: Hans J. Eysenck concluded that there were no favorable effects of psychotherapy, starting a raging debate which 25 years of evaluation research and hundreds of studies failed to resolve 1978: To proved Eysenck wrong, Gene V. Glass statistically aggregated the findings of 375 psychotherapy outcome studies Glass called the method “meta-analysis”
META ANALYSIS IN CONTEXT Traditional Systematic Meta- review Review analysisAuthor An individual A team A teamSearch Individual Based on a Based on astrategy protocol protocolSummary Author‟s Can be SummaryConclusion judgement qualitative statistical techniques
FUNCTION OF META-ANALYSIS1. Identify heterogeneity in effects among multiple studies and, where appropriate, provide summary measure2. Increase statistical power and precision to detect an effect3. Develop, refine and test hypothesis4. Reduce the subjectivity of study comparisons by using systematic and explicit comparison procedure5. Identify data gap in the knowledge base and suggest direction for future research6. Calculate sample size for future studies
COULD WE JUST ADD THE DATA FROM ALLTHE TRIALS TOGETHER? One approach to combining trials would be to add all the treatment groups together, add all the control groups together, and compare the totals This is wrong for several reasons, and it can give the wrong answer
If we just add up the columns we get From a meta-analysis, we get34.3% vs 32.5% , a RR of 1.06, RR=0.96 , a lower death ratea higher death rate in the steroids group in the steroids group
PROBLEMS WITH SIMPLE ADDITION OFSTUDIES Breaks the power of randomization Imbalances within trials introduce bias
* #In effect we are comparingthis experimental group directlywith this control group – this isnot a randomized comparison
STEPS IN META-ANALYSIS1. Define the research question and specific hypotheses2. Define the criteria for including and excluding studies3. Locate research studies4. Determine which studies are eligible for inclusion5. Classify and code important study characteristics (e.g., sample size; length of follow-up; definition of outcome; drug brand and dose)6. Select or translate results from each study using a common metric7. Aggregate findings across studies, generating weighted pooled estimates of effect size.8. Evaluate the statistical homogeneity of pooled studies9. Perform sensitivity analyses to assess the impact of excluding or down-weighting unpublished studies, studies of lower quality, out-of-date studies, etc.
PUBLICATION BIAS Studies with significant results are more likely to be published to be published in English to be cited by others to produce multiple publications Including only published studies can introduce publication bias Most reviews do not look for publication bias
APPROACHES TO PUBLICATION BIAS Fail-safe N Publications are filled with the 5% of studies with Type I error but file drawers are filled with the 95% with non- significant effects Funnel plots Trim and fill method Small studies are removed from funnel plot until it is symmetric Then replace the small studies and balance them with studies on the opposite side of the funnel Statistical analogues of funnel plot Egger test
QUALITY SCORE Some meta-analysts score the quality of a study. 1. Examples (scored yes=1, no=0): 2. Published in a peer-reviewed journal? 3. Experienced researchers? 4. Research funded by impartial agency? 5. Study performed by impartial researchers? 6. Subjects selected randomly from a population? 7. Subjects assigned randomly to treatments? 8. High proportion of subjects entered and/or finished the study? 9. Subjects blind to treatment? 10. Data gatherers blind to treatment? 11. Analysis performed blind?
AVERAGING STUDIES Starting with the summary statistic for each study, how should we combine these? A simple average gives each study equal weight This seems intuitively wrong Some studies are more likely to give an answer closer to the „true‟ effect than others
WEIGHTING STUDIES More weight to the studies which give us more information More participants More events Lower variance Calculated using the effect estimate and its variance Inverse-variance method: 1 1 weight variance of estimate SE2 sum of (estimate weight) pooled estimate sum of weights
WHAT IS HETEROGENEITY Heterogeneity is variation between the studies‟ results
CAUSES OF HETEROGENEITY1. Differences between studies with respect to:2. Patients: diagnosis, in- and exclusion criteria, etc.3. Interventions: type, dose, duration, etc.4. Outcomes: type, scale, cut-off points, duration of follow-up, etc.5. Quality and methodology: randomised or not, allocation concealment, blinding, etc.
HOW TO LOOK FOR HETEROGENEITY? Visually Forest plot: do confidence intervals of studies overlap with each other and the summary effect? Statistically Chi-square test for heterogeneity (Mantel-Haenszel test or Cochran Q test) Tests whether the individual effects are farther away from the common effect, beyond what is expected by chance Has poor power P-value < 0.10 indicates significant heterogeneity
HOW TO DEAL WITH HETEROGENEITY1. Do not pool at all2. Ignore heterogeneity: use fixed effect model3. Allow for heterogeneity: use random effects model4. Explore heterogeneity
FIXED EFFECT MODEL Fixed effects model assumes that the true effect of treatment is the same value in each study (fixed); the differences between studies is solely due to random error Specific methods for combining odds ratio 1. Mantel- Haenszel method 2. Peto‟s method 3. Maximum-Likelihood techniques 4. Exact methods of interval estimation
RANDOM EFFECT MODEL In random effects models, the treatment effects for the individual studies are assumed to vary around some overall average treatment effect Allows for random error plus inter-study variability Results in wider confidence intervals (conservative) Studies tend to be weighted more equally (relatively more weight is given to smaller studies) There are five approach for this model: 1. Weighted least squares 2. Un-weighted least squares 3. Maximum likelihood 4. Restricted Maximum likelihood 5. Exact approach to random effects of binary data.
SENSITIVITY ANALYSIS Several features of the meta-analysis can be altered to gauge the robustness of the results: Modifying the inclusion criteria Including and excluding questionable studies Including and excluding unpublished studies Weighting the analysis by study quality Trying different ways to impute missing data Removing each study, one by one, to see the change
FOREST PLOT The graphical display of results from individual studies on a common scale is a “Forest plot”. In the forest plot each study is represented by a black square and a horizontal line (CI:95%).The area of the black square reflects the weight of the study in the meta-analysis. A logarithmic scale should be used for plotting the Relative Risk.
FOREST PLOTS Headache at 24 hours HEADINGS EXPLAIN THE COMPARISON
FOREST PLOTS Headache at 24 hours LIST OF INCLUDED STUDIES
FOREST PLOTS Headache at 24 hours RAW DATA FOR EACH STUDY
FOREST PLOTS Headache at 24 hours TOTAL DATA FOR ALL STUDIES
FOREST PLOTS Headache at 24 hours WEIGHT GIVEN TO EACH STUDY
FOREST PLOTS Headache at 24 hours EFFECT ESTIMATE FOR EACH STUDY, WITH CI
FOREST PLOTS Headache at 24 hours EFFECT ESTIMATE FOR EACH STUDY, WITH CI
FOREST PLOTS Headache at 24 hours SCALE AND DIRECTION OF BENEFIT
FOREST PLOTS Headache at 24 hours POOLED EFFECT ESTIMATE FOR ALL STUDIES, WITH CI
INTERPRETING CONFIDENCE INTERVALS always present estimate with a confidence interval precision • point estimate is the best guess of the effect • CI expresses uncertainty – range of values we can be reasonably sure includes the true effect significance • if the CI includes the null value rarely means evidence of no effect effect cannot be confirmed or refuted by the available evidence • consider what level of change is clinically important
LIMITATIONS OF META-ANALYSIS A meta-analysis reflects only whats published or searchable. Its focused on mean effects and differences between studies. But what really matters is effects on individuals. (Aggression bias) Relation between group rates or and means may not resemble the relation between individual values of exposure and outcome. This phenomenon is known as aggregation bias or ecologic bias.
WHAT IS AN IPD META-ANALYSIS? Involves the central collection, checking and analysis of updated individual patient data Include all properly randomised trials, published and unpublished Include all patients in an intention-to-treat analysis Individual patient data used Analysis stratified by trial IPD does not mean that all patients are combined into a single mega trial The major advantage of a IPD over an MA is the use of individual-based rather than group-based data.
GUIDELINES FOR REPORTING META-ANALYSIS The QUORUM Statement: Quality of Reporting of Meta-analyses – For clinical Randomized Controlled Trials (RCT‟s) MOOSE guidelines: Meta-analysis Of Observational Studies in Epidemiology
META-ANALYSIS SOFTWARE Free Commercial RevMan [Review Comprehensive Meta- Manager] analysis Meta-Analyst Meta-Win Epi Meta WEasy MA Easy MA Meta-Test General stats packages Meta-Stat Stata SAS S-Plus