Introduction to Meta Analysis
Dr Santam Chakraborty
Associate Professor, Radiation Oncology,
Malabar Cancer Center
Have not personally performed
a meta analysis .....
.... however read quite a few
2. Cochrane Handbook of Systematic Reviews of Interventions*
3. How to conduct a meta-analysis : Dr Arindam Basu
4. PRISMA statement
*Most material referred
Cochrane Handbook Section 9 (Analyzing data)
9.1.1 Do not start here!
It can be tempting to jump prematurely into a statistical analysis when
undertaking a systematic review. The production of a diamond at the bottom of
a plot is an exciting moment for many authors, but results of meta-analyses
can be very misleading if suitable attention has not been given to formulating
the review question; specifying eligibility criteria; identifying, selecting and
critically appraising studies; collecting appropriate data; and deciding what
would be meaningful to analyse.
A Historical Perspective
● Key paper
● Published 1904
● Correlation between typhoid
inoculation & mortality
● In soldiers serving British
● Noted that individual studies showed consistent correlation.
● Noted the high probability for selection bias in the studies
● However values of correlation between studies were variable -
○ An example of Heterogeneity.
● Overall found that correlation coefficients had low value
○ Failed to convince him of the value of routine inoculations.
“Assuming that the inoculation is not more than a temporary inconvenience, it would
seem to be possible to call for volunteers... [and] only to inoculate every second
volunteer... with a view to ascertaining whether any inoculation is likely to prove
useful... In other words, the ‘experiment’ might demonstrate that this first step to a
reasonably effective prevention was not a false one.”
Historical Perspective ..
● Efforts to synthesize increasing research volume - lead to various methods
of quantitative synthesis.
● Gene Glass (1976) coined the term meta analysis
○ The statistical analysis of a large collection of analysis results from
individual studies for the purpose of integrating the findings”
● 1st randomized controlled
trial showed a weak effect
of Aspirin in preventing
● Subsequent trial results
were assembled and
● The statistical synthesis of the data from separate but similar, i.e.
comparable studies, leading to a quantitative summary of the pooled
● This is different from a systematic review:
○ The application of strategies that limit bias in the assembly, critical
appraisal, and synthesis of all relevant studies on a specific topic.
○ Meta analysis may be a part of the process of a systematic review.
Conceptually following major advantages:
1. Improved power and precision due to larger sample size
2. Quantify inconsistency in results between studies.
3. Generalization of results
4. Hypothesis testing can be applied on summary statistics
5. Investigate presence of publication bias
Also forms an important part of “research synthesis” process.
Why Meta Analysis
How to Conduct
Decide which variables
Select the model
Should be a well framed question - define the boundaries.
● Population (P)
● Intervention (I)
● Comparisons (C)
● Outcomes (O)
Formulating the Problem
● Define the disease / condition of interest:
○ Have explicit criteria
○ But take care not to be too exclusive
● Identify the important demographic characteristics
● Identify the setting of the studies
● Handling of studies that involve a “subset” of the above participants.
Defining the Population
● What is the intervention and control
● Does the intervention have variations ?
○ Dose / Mode of delivery / frequency of delivery / who delivers
● Which variations should be included?
● How will you handle studies where only part of the intervention is
● How will trials that combine the intervention with another intervention of
interest be handled.
1. Reporting of outcome does not usually determine eligibility for inclusion.
2. Include outcomes that are meaningful:
3. Address outcomes that relate the beneficial effects as well as adverse
4. Define how the outcomes will be measured.
5. Limit primary outcomes to 2 - 3 which have impact on patient care.
Defining Types of Outcomes
● A priori decision regarding study type needed.
● Randomized trials favoured:
○ Best way to eliminate bias and confounding.
○ Better and more “conservative” estimate of the effect as compared to
○ May eliminate “publication bias” to some extent.
Define Study Types
● It is good practice to have a trial search coordinator
● Characteristics of a good search:
○ Search diverse sources
● Don't restrict your search to a single database e.g. Medline
● Searching across multiple database also limits selection bias for studies
that are found.
Searching for Studies
Sources to Search
● Bibliographic Databases: MEDLINE, EMBASE, CENTRAL
○ Fast and easy
○ Search electronically
○ Various search methodology
○ Indexing terms available
● Citation Indexes: SciSearch, SCOPUS
○ Checks for citing articles for a given article
● Dissertations and Theses can be searched in specific databases: ProQuest,
Theses (UK), DissOnline (German).
● Grey Literature: Conference abstracts
Sources to Search
○ Not all trial reports are included in bibliographic databases
○ May not contain relevant search terms.
● Full text search of journals in internet (not indexed)
● Conference abstracts:
○ 50% of all trials reported dont reach full publication !!
● Trial registers : National (CTRI) and International
● Develop a search strategy
○ Develop a set of keywords
● Document the search strategy : Important if search is to be reporducible
● Understand use of filters in databases
● Understand use of “Boolean operators”
Planning the Search
● Use a reference manager:
● Most important step in the process.
● Multiple heads better than one.
● Define selection criteria and have a form
○ May have a pilot test for the selection criteria on 10 -12 randomly
● Using a reference manager can help sort out “studies” from “reports”
● Define a list of excluded studies also.
● Study design data
● Participants data
● Interventions data
● Outcome measures data
Reference: Checklist of data to collect Cochrane Handbook
What data to collect
● Original Report
● Correspondence with investigators
● Individual Patient data - Perhaps the most robust form of data collection.
Sources of Data
● Bias is a systematic deviation / error from the truth in result / inferences.
● Can lead to underestimation / overestimation
● Can vary in magnitude
● Can vary in direction : Same bias can lead to different effect on the
● However actual bias is difficult to understand
○ Hence better to consider a “risk of bias”
● Important to distinguish from imprecision: bias is systematic which means
if the same study is repeated again and again the same error will occur.
Imprecision is a random error that can be overcome by larger samples.
● Selection bias (Non randomized)
● Performance bias (Non blinded - patient & care givers)
● Detection bias (Non blinded - assessor)
● Attrition bias
● Reporting bias - may be one of the most important - can exist within
Sources of Bias
● Publication Bias: Results determine if published.
● Time lag bias : Results determine time taken to publish
● Multiple publication bias
● Location bias
● Citation bias: Results determine citations
● Language bias
● Outcome reporting bias : Selective reporting of only some outcomes
Types of Reporting Biases
● The objective of the analysis is to get an idea of the “effect”
● Effect = Difference between two groups.
● 4 Questions that need to be answered:
○ Direction (+ve/-ve)
○ Strength of evidence for the effect
● Meta-analysis can answer 1 - 3.
Planning the Analysis
● Dichotomous data : Risk ratio, Risk Difference, NNT and Odds Ratio
● Continuous data: Mean Difference
● Ordinal data: Proportional odds ratio
● Counts & rates of events data: Rate ratio
● time to event (e.g. Survival) data - Censored: Hazard ratio
● Risk : Is the probability of getting the event. Ranges between 0 - 1.
○ Risk Ratio: Ratio of risk in the two groups
○ Risk Difference : Difference between risk in the two groups.
● Odds: Ratio of the probability of occurrence of an event to that it does not
occur. Can range between 0 - infinity.
○ Odds Ratio: Ratio of odds of an event in the two groups.
Thus Odds = Risk / 1 - Risk
Hence for trials with larger number of events Odds >> Risk
Risk versus Odds
● Measures the absolute difference between mean values of two groups.
● Can be used with summary statistics of ALL studies are made in SAME
● Standardized mean difference: Used when studies measure the same
outcome but do it in different ways.
● Hazard ratio conceptually similiar to risk ratio:
○ However calculated on censored data
○ Changes instantaneously.
● After the summary estimate or “effect” for each trial is calculated - meta-
analytic methods allow obtaining a summary of the effect.
● A weighted average is calculated.
● Weights account for the study information
● May optionally incorporate an assumption that studies are not measuring
the SAME intervention.
○ Basis of random effects meta-analysis
○ Fixed effect : When assumption is each study is estimating the same
● Standard error and confidence intervals of the weighted average effect are
also reported - precision of the estimate.
Summary of the Estimate
● Can be of different types:
○ Clinical diversity : Arising due to participants / interventions /
○ Methodological diversity: Arising due to variation in study methods
○ Statistical diversity: Variability in the intervention effects being
studied in the meta-analysis.
● Statistical diversity (Heterogeneity) - Arises due to clinical and
● Thus observed differences in the effect between studies becomes MORE
than what would be expected by chance alone.
● The following information should be presented:
○ Characteristics of included studies
■ Risk of bias table
○ Data and analysis
○ Figures : Study flow, forest plots, funnel plots, risk of bias plots
○ Summary of findings
● PRISMA statement has a complete checklist to be followed when
reporting such studies. (http://prisma-statement.org)
Reporting a Meta-Analysis
● Results are susceptible to diversity in studies - outcomes studied should be
● Bias at each study influences the results
● Publication bias is a serious source of problems.
● Agenda driven bias - perhaps most serious !!
Problems with Meta-analysis
❖ Meta-analysis is a powerful technique for synthesizing study results.
❖ Allows improved power as well as precision
❖ Can make the results more generalizable.
❖ However very prone to bias
❖ A proper methodology to be followed