Detailed steps of conducting a meta analysis

2. Dr B R Rajeev
3rd MDS
Dept of Public Health Dentistry
SDM College of Dental Sciences

3. Introduction
How meta analysis came into being?
What is the rationale for meta analysis?
Steps in meta analysis
Biases in meta analysis
Strengths and weakness of meta analysis
Conclusion
Bibliography

4. Insert messed table pic

5. Dentists are challenging to manage sophisticated patient needs
and demands.
Advances in Dentistry Latest Techniques Relevant Literature
Application of this
knowledge
Practitioner’s Priority

6. Evidence Based Dentistry
integrates the best available evidence with clinical experience
and patient preference in making clinical decisions

7. Weaker
Stronger
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what
it is and what it isn't. British Medical Journal. 1996;312:71–72.

8. Incorporating research into practice is time consuming.
Busy clinicians need easy access to evidence
Systematic reviews and meta analyses
Research synthesis of multiple studies
Increased and efficient access to evidence

9. Meta-analysis is a statistical technique, or set of statistical
techniques, for summarising the results of several studies into a
single estimate.
“Meta-analysis refers to the analysis of analyses…the statistical
analysis of a large collection of analysis results from individual
studies for the purpose of integrating findings. “
Glass, 1976

10. The evidence-based practitioner, David Sackett, makes a
distinction between a review, an overview and a meta-analysis,
defining each as follows:
Review:
the general term for all
attempts to synthesise the
results and conclusions of
two or more publications on a
given topic.
Systemic review:
when a review strives to
comprehensively identify and
track down all the literature
on a given topic
Meta-analysis:
a specific statistical strategy for assembling the
results of several studies into a single estimate.

11. Review level
↓
Effect
Measure
Study A Effect measure
Outcome data
Effect measure
Outcome data
Study B
Effect measure
Outcome data
Study C
Effect measure
Outcome data
Study D
Study level
↓
Source: Jo McKenzie & Miranda Cumpston

12. Optional part of a systematic review
Systematic reviews do not have to have a meta-analysis.
Source: Julian Higgins
Systematic
reviews Meta-
analyses

13. THEEMERGENCEOF
META-ANALYSIS
 Karl Pearson
(1904)
 averaged
correlations for
studies of the
effectiveness of
inoculation for
typhoid fever

14. R. A. Fisher (1944)
“When a number of quite
independent tests of significance have
been made, it sometimes happens
that although few or none can be
claimed individually as significant, yet
the aggregate gives an impression
that the probabilities are on the
whole lower than would often have
been obtained by chance”

15.  W. G. Cochran (1953)
 Discussed a method of averaging means across independent studies
 Laid-out much of the statistical foundation that modern meta-analysis is built
upon (e.g., Inverse variance weighting and homogeneity testing)

16. 1952: Hans J. Eysenck concluded
that there were no favorable
effects of psychotherapy, starting
a raging debate
20 years of evaluation research
and hundreds of studies failed to
resolve the debate
1978: To prove Eysenck wrong,
Gene V. Glass statistically
aggregate the findings of 375
psychotherapy outcome studies
Glass (and colleague Smith)
concluded that psychotherapy did
indeed work.

17. Firstly, decision-makers are inundated with unmanageable
amounts of information.
Secondly, to access all or most of the studies often difficult, time
consuming and cost-ineffective.
Meta-analysis can provide access to information from many studies with
less effort and hassle.
It provides the option to read a summary prepared by others, relying on
those who have already spent time, money, and energy to summarize
information from multiple studies on the topic.

18. Thirdly, single studies rarely provide definitive answers to
clinical questions.
Fourthly, it helps resolve controversies and conflicting reports.
Meta-analysis of multiple studies helps establish whether:
• Scientific findings are consistent
• Can be generalized across populations, settings, and
treatment variations
• Findings vary by particular subsets.
1. It enhances precision
2. Provides robust estimates
3. Answers questions that single trials are under-powered or were
not designed to address.

19. Fifthly, in meta-analysis
limit bias (due to explicit methods used)
improve the reliability (precision) and accuracy (validity) of
conclusions.
provides a gain in statistical power for average estimates.
promising leads or small effects can be missed and researchers
can embark on studies of questions that have been already
answered.

20. Finally, meta-analyses
Identifies crucial areas and questions that have not been
adequately addressed with past research.
Thus, it documents the need for a major clinical trial.
It confirms the sufficiency of available literature on a particular
topic.
Thus, it helps to avoid the time and expense of conducting
another clinical trial.

21. Aim &
protocol
Literature
Searches
Quality
assessment
Study
Outcomes
Statistical
Analysis
Conclusions

22. It requires a clear statement of the intervention of interest,
relevant patient groups and appropriate outcomes.
Repeatedly asking “why is this clinical question important to
answer?” is helpful.
The objectives of the review should follow logically from the
question and be clearly stated.

23. A medical research question could contain four elements that
can be broken down as:
•Participants eg children with caries, smokers with
periodontal disease
•Intervention eg antibiotics, physiotherapy, powered
toothbrushes
•Comparison What the intervention is to be compared to:
eg another intervention, placebo
•Outcomes eg longevity of restorations, pain reduction,
quality of life

24. The published and unpublished literature should be carefully searched
for all reports of appropriate and relevant studies.
Systematic reviews of treatment and preventive interventions –RCT
The search must include accessing a number of electronic databases
and non-English sources.
The search strategy must be comprehensive.

25. Controlled vocabulary – what it is and where to find it.
Key word or free text searching
Boolean operators: AND, OR , NOT
Truncation : Caries- Dental Caries + Pit and Fissure Caries
Proximity operators
Ex- Wom$n- women + woman

26. Madhukar Pai et al, National Medical Journal of India, 2004

27. Inclusion Exclusion Criteria
The components of the question (type of intervention,
population, and outcome)
The studies identified - assessed against these criteria
Critical appraisal of studies –
two people working independently

28. Grey literature. It is the name given
to material produced by
government, academies, business
and industries; both in print and
electronic formats, but which is not
controlled by commercial publishing
interests and where publishing is
not the primary activity of the
organisation.

29. Quality refers to internal validity of the studies
The quality criteria used will depend on the study design
This should be presented clearly and allow to determine the
validity of the studies

30. A good quality systematic review will comment on all the
important study appraisal criteria outlined in the checklist for
study appraisal.

31. CONSORT
JADAD Score-
CHALMERS SCALE
NEWCASTLE OTTAWA SCALE
STROBE – Observational Studies
Randomised Controlled Trials

32. If appropriate, the findings from the individual included studies
can be aggregated to produce a summary estimate of the overall
effect of the intervention.
Aggregation is qualitative (i.e., individual descriptions of the
included studies
quantitative assessment - Meta-Analysis.
Meta-analysis should only be performed when the studies are
similar with respect to population, outcome and intervention.

33. A systematic review should attempt to place the findings of a
minimally-biased selection of studies in context.
This discussion should address issues such as the quality and
heterogeneity of the included studies, the likely impact of bias
and chance, and the applicability of the findings.

34. Data types and outcome measures
Summary effect estimates
Heterogeneity

35. Outcome
Binary Survival
Continuous
Odds Ratio (OR)
Relative Ratio (RR)
Risk Difference(RD)
Mean difference (MD)
Standardised mean
Difference (SMD)
Hazard Ratio (HR)

36. Odds ratio is the odds of success in the treatment group relative
to the odds of success in the control group.
Relative risk (RR) represents the probability of an event (failure)
in the treatment group relative to the probability of the same
event in the control group.
Risk difference is the difference of two binomial probabilities,
while RR is the ratio.

37. Event
(‘Success’)
No Event
(‘Failure’) Total
Experimental
intervention
SE FE NE
Control intervention SC FC NC

38. Mean Difference
The mean difference is a standard statistic that measures the
absolute difference between the mean value in two groups in a
clinical trial.
It estimates the amount by which the experimental intervention
changes the outcome on average compared with the control.
It can be used as a summary statistic in meta-analysis when
outcome measurements in all studies are made on the same
scale.

39. The standardized mean difference
Used as a summary statistic in meta-analysis when the studies
all assess the same outcome but measure it in a variety of ways.
The standardized mean difference expresses the size of the
intervention effect in each study relative to the variability
observed in that study.

40. Hazard ratio is the standard outcome measure in survival
analysis.
It is the ratio of the risk of having an event at any given time in
one group divided by the risk of an event in the other.
A hazard ratio which is equal to one represents no difference
between the groups.

41. Hazard ratios differ from relative risk ratios in that the latter are
cumulative over an entire study, using a defined endpoint, while
the former represent instantaneous risk over the study time
period, or some subset thereof.
In its simplest form the hazard ratio can be interpreted as the
chance of an event occurring in the treatment arm divided by the
chance of the event occurring in the control arm, or vice versa, of
a study.

42. Any kind of variability among studies in a systematic review may
be termed heterogeneity.
It refers to the various responses to a given treatment among the
included studies. Indicates that effect sizes vary considerably
across studies.
If heterogeneity is present, a common, summary measure is hard
to interpret.
If no significant heterogeneity:
Can perform meta-analysis and generate a common, summary
effect measure
If significant heterogeneity is found:
 Find out what factors might explain the heterogeneity
 Can decide not to combine the data

43. WHY IS IT CRUCIAL TO ASSESS HETEROGENEITY?
The presence versus the absence of true heterogeneity
(between-studies variability) can affect the statistical model that
the meta-analyst decides to apply to the meta-analytic database.

44. Sources of Heterogeneity
Variability
due to
sampling
error
• The sampling error variability is
always present in a meta-analysis,
because every single study uses
different samples.
Between-
studies
variability
• It is due to the influence of an
indeterminate number of
characteristics that vary among the
studies
1. There can be two sources of variability that explain the
heterogeneity in a set of studies in a meta-analysis.

45. Can be due to differences in:
Types of heterogeneity:
Patient populations
studied
Co-
interventions
Random error Study design
features (example:
length of follow-
up)
Interventions used Outcomes
measured
Study quality
Clinical
Heterogeneity
Statistical
Heterogeneity
Methodological
Heterogeneity

46. 1. Common sense
Are patients, intervention and outcome in each of the included studies
sufficiently similar
2. Visually
do confidence intervals of studies overlap with each other and the
summary effect
3. Statistical tests
1. Standard Chi-Square test
2. Plot of normalised (Z) score
3. Forest plot
4. Radial Plot
5. L’abbe` plot
Graphical Method

47. Chi-square test for heterogeneity (Mantel-Haenszel test or
Cochran Q test)
Tests whether the individual effects are farther away from the
common effect, beyond what is expected by chance
Has poor power when there are less number of studies

48. The Z score or standardised residuals for each study can be
calculated
The histogram of these Z scores should have an approximately
normal distribution, with mean zero and a variance of one
Large absolute z score signal - departure of individual studies
from the average results

49. The graphical display of results from individual studies on a
common scale is a “Forest plot”
In the forest plot each study is represented by a black square and
a horizontal line (CI:95%)
The area of the black square reflects the weight of the study in
the meta-analysis.

51. Obtained by plotting the outcome from each study divided by
the square of its variance against the reciprocal of SE
Points which form a homogenous set of trials will scatter
homoscedastically
Points a long way from the line of best fit indicate outliers that
will contribute considerably to the between study heterogeneity

53. Applicable to only meta analysis of studies with binary outcomes
Plots the risk (or odds) in the exposed against those of the
control group and often contains a regression line and a central
diagonal line indicating identical risks in each group

55. 1. Check again that the data are correct
2. Do not do a meta-analysis
3. Explore heterogeneity
4. Ignore heterogeneity
5. Perform a random effects meta-analysis
6. Change the effect measure
7. Exclude studies
Cochrane Handbook for Systematic Reviews of Interventions 4.2.6

56. Q – statistic: a measure of weighted squared deviations
Tau square test: a between study variance
Tau: between study standard deviation
I2 Index: a ratio of true to total variance

57. Collect a summary statistic from each contributing study
How do we bring them together?
• treat as one big study – add intervention & control data?
breaks randomisation, will give the wrong answer
• simple average?
weights all studies equally – some studies closer to the truth
• weighted average

58. More weight to the studies which give more information
• more participants, more events, narrower confidence interval
• calculated using the effect estimate and its variance
Inverse-variance method:
2
SE
1
estimate
of
variance
1
weight 

weights
of
sum
)
weight
estimate
(
of
sum
estimate
pooled



59. Headache Caffeine Decaf Weight
Amore-Coffea 2000 2/31 10/34
Deliciozza 2004 10/40 9/40
Mama-Kaffa 1999 12/53 9/61
Morrocona 1998 3/15 1/17
Norscafe 1998 19/68 9/64
Oohlahlazza 1998 4/35 2/37
Piazza-Allerta 2003 8/35 6/37

60. Headache Caffeine Decaf Weight
Amore-Coffea 2000 2/31 10/34 6.6%
Deliciozza 2004 10/40 9/40 21.9%
Mama-Kaffa 1999 12/53 9/61 22.2%
Morrocona 1998 3/15 1/17 2.9%
Norscafe 1998 19/68 9/64 26.4%
Oohlahlazza 1998 4/35 2/37 5.1%
Piazza-Allerta 2003 8/35 6/37 14.9%

61. Dichotomous or Continuous data
• Inverse-variance
• Straightforward, general method
Dichotomous data only
• Mantel-Haenszel (default)
Odd’s Ratio only
• Peto

63. Most meta-analyses are based on one of two statistical models:
The fixed-effect model The random-effects model

64. Summary Effect Estimates
FIXED EFFECT MODELS
• It assumes that the true effect of treatment is the same value in
each study (fixed); the differences between studies is solely due
to random error.
• In this model, all of the observed difference between the studies
is due to chance.
• Observed study effect = Fixed effect + error
Xi= θ + ei ei is sampling error
Xi = Observed study effect, θ = Fixed effect common to all studies

65. Summary Effect Estimates
FIXED EFFECT MODELS
Inverse variance method
Mantel- Haenszel method
Peto’s method

66. Summary Effect Estimates
RANDOM EFFECTS MODELS
The “random effects” model, assumes a different underlying
effect for each study.
In the treatment effects for the individual studies are assumed to
vary around some overall average treatment effect
Allows for random error plus inter-study variability
Results in wider confidence intervals
Studies tend to be weighted more

67. Summary Effect Estimates
RANDOM EFFECTS MODELS
• This model leads to relatively more weight being given to smaller
studies and to wider confidence intervals than the fixed effects
models.
• The use of this model has been advocated if there is
heterogeneity between study results.
• DerSimonian and Laird Test

69. Headache at 24 hours
• Headings explain the comparison

70. Headache at 24 hours
• List of included studies

71. Headache at 24 hours
• Raw data for each study

72. Headache at 24 hours
• Total data for all studies

73. Headache at 24 hours
• Weight given to each study

74. Headache at 24 hours
• Effect estimate for each study, with CI

75. Headache at 24 hours
• Effect estimate for each study, with CI

76. Headache at 24 hours
• Scale and direction of benefit

77. Headache at 24 hours
• Pooled effect estimate for all studies, with CI

78. Always present estimate with a confidence interval
Precision
• Point estimate is the best guess of the effect
• CI expresses uncertainty – range of values we can be reasonably sure
includes the true effect
Significance
• If the CI includes the null value
rarely means evidence of no effect
effect cannot be confirmed or refuted by the available evidence
• consider what level of change is clinically important

79. 1. English language bias
2. Availability bias
3. Cost bias
4. Familiarity bias
5. Database bias
6. Publication bias
7. Source selection bias
8. Citation bias
9. Multiple publication bias

80. Studies :
Reporting significant treatment effects are more likely to be published
Published in English
Published more quickly than non significant studies.
Problem associated – it affects the validity of the medical
literature as a whole because the obtained results may be
misleading.

81. The results of a meta-analysis may be biased if the included
studies are a biased sample of studies in general.
The only true test for publication bias is to compare effects in
the published studies formally with effects in the unpublished
studies.

82. Fail-safe N
Computing how many missing studies would be required to
retrieve and incorporate in the analysis before the p-value
became non significant.
Funnel plots
Trim and fill method
Small studies are removed from funnel plot until it is symmetric
Then replace the small studies and balance them with studies on
the opposite side of the funnel

84. Egger et al test
It is a test for funnel plot asymmetry is based on the linear regression
(not confined to passing through the origin) of standardized treatment
effects on their inverse standard errors.
Statistical significance of the intercept provides a test for funnel plot
asymmetry, since under ideal conditions the regression line should
pass through the origin.

85. It is a graphic method of assessing the publication bias.
It is plotted with effect size on the X axis and the sample size or
variance on the Y axis.
Large studies appear toward the top of the graph and generally
cluster around the mean effect size.
Smaller studies appear toward the bottom of the graph, and
(since smaller studies have more sampling error variation in
effect sizes) tend to be spread across a broad range of values.

86. Funnel plot can be a v shaped or an inverted v shaped.
If the readings are in units of SE (standard e) along the Y-Axis, it
gives a straight V shape.
If the units are in 1/SE over Y axis, it gives a picture of inverted V-
shaped.
The X-axis is usually Hazards Ratio, Risk Ratio, Odds Ratio or most
commonly "log" of all these values.

87. In the absence of publication bias, the studies will be distributed
symmetrically about the mean effect size, since the sampling
error is random. In this case the plot will be symmetrical.
In the presence of publication bias the studies are expected to
follow the model, with symmetry at the top, a few studies
missing in the middle, and more studies missing near the
bottom. In this case , the funnel plots will often be skewed and
asymmetrical.

89. Selection Bias (Publication and reporting bias, Biased inclusion criteria)
True Heterogeneity: size of effect differs according to study size
Intensity of interventions
Difference on underlying risk
Data irregularities
Poor methodological design of small studies
Inadequate analyses
Fraud
Artefactual – heterogenity due to poor choice of effect measure
Chance

90. CLINICALSITUATION
Patient c/o black spotted
cavity in the lower right back
tooth region..

91. On examination
G V Black’s Class 1 deep
dentinal caries

92. What Treatment??
Incomplete caries removal?
Complete caries removal ?
Pulp exposure
Post operative pulp symptoms
Treatment failure

93. Research Question- Incomplete Caries removal
Literature search
Criteria
Studies
Randomized or quasi-randomized controlled trials (RCTs) published in 1967
or later
Participants
Humans with primary dentin caries in deciduous or permanent teeth
requiring a restoration
Outcomes
Pulp exposure
Post operative pulp symptoms
Treatment failure

94. Search Strategy
(CochraneCentral Register of Controlled
Trials, MEDLINE, PUBMED, EMBASE)
between May 16, 1967 and July 23, 2012.

101. An increased risk of oral cancer was found for bidi smokers compared to
never smokers (OR 3.1, 95% confidence interval [CI] 2.0 –5.0) whereas
no significant pattern of risk was found for cigarette smokers (OR 1.1,
95% CI 0.7–1.8).

103.  Imposes a discipline on the process of summing up research
findings
 Represents findings in a more differentiated and sophisticated
manner than conventional reviews
 Capable of finding relationships across studies that are obscured
in other approaches
 Protects against over-interpreting differences across studies
 Can handle a large numbers of studies (this would overwhelm
traditional approaches to review)

104.  Requires a good deal of effort
 Mechanical aspects don’t lend themselves to capturing more qualitative
distinctions between studies
 “Apples and oranges” criticism
 Most meta-analyses include “blemished” studies to one degree or another
(e.g., a randomized design with attrition)
 Selection bias posses a continual threat
 Negative and null finding studies that you were unable to find
 Outcomes for which there were negative or null findings that were not
reported
 Analysis of between study differences is fundamentally correlational

105. In the hands of a careful consumer, a meta-analysis can provide
considerable guidance for basic and applied research.
Each new increment in research can be a firm-footed step
forward rather than a blind leap of faith.
Systematic reviews can summarize all of the available evidence
for a clinical question. If appropriate, a meta-analysis can be
performed to provide an estimate of the overall treatment effect
for a given therapy.

106. This is a tremendous timesaver and allow busy clinicians to in-
corporate evidence from trials that they may not have found
otherwise.
The combination of multiple studies also provides more powerful
evidence for making clinical decisions than an individual study.

107. Systematic reviews are scientific investigations and have many
potential biases, so it is important to systematically approach
their evaluation.
Meta analyses are becoming more common and will continue to
play a major role in translating research evidence into patient
care decisions.