Attikon 2014 - Software and model selection challenges in meta-analysis

[Poster title]
[Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4
1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]
Meta-analysis overview
Two-stage meta-analysis
One-stage meta-analysis
Summary
Software and model selection challenges in
meta-analysis
MetaEasy, model assumptions, homogeneity and IPD
Evangelos Kontopantelis
Centre for Health Informatics
Institute of Population Health
University of Manchester
Attikon Hospital
Athens, 2 June 2014
Kontopantelis Software and model selection challenges in meta-analysis

[Poster title]
Summary
Outline
1 Meta-analysis overview
2 Two-stage meta-analysis
3 One-stage meta-analysis
4 Summary

[Poster title]
Summary
The heterogeneity issue
Outline
4 Summary

[Poster title]
Summary
Timeline
Efforts to pool results from individual studies back as far as
1904
The first attempt that assessed a therapeutic intervention
was published in 1955
In 1976 Glass first used the term to describe a "statistical
analysis of a large collection of analysis results from
individual studies for the purpose of integrating the
findings"
Relatively young and dynamic field of research, reflecting
importance of MA and potential for conclusive answers

[Poster title]
Summary
Meta-analysing reported study results
A two-stage process
the relevant summary effect statistics are extracted from
published papers on the included studies
these are then combined into an overall effect estimate
using a suitable meta-analysis model
However, problems often arise
papers do not report all the statistical information required
as input
papers report a statistic other than the effect size which
needs to be transformed with a loss of precision
a study might be too different to be included (population
clinically heterogeneous)

[Poster title]
Summary
Individual Patient Data
IPD
These problems can be avoided when IPD from each
study are available
outcomes can be easily standardised
clinical heterogeneity can be addressed with subgroup
analyses and patient-level covariate controlling
Can be analysed in a single- or two-stage process
mixed-effects regression models can be used to combine
information across studies in a single stage
this is currently the best approach, with the two-stage
method being at best equivalent in certain scenarios

[Poster title]
Summary
Meta-analyses on the rise
Meta-analysis studies
are used more and more
since they seem to be a
useful ‘summary’ tool.
However critics argue
that one cannot combine
studies when they are
too diverse
(heterogeneous).

[Poster title]
Summary
Outline
4 Summary

[Poster title]
Summary
When Robin ruins the party...

[Poster title]
Summary
Heterogeneity
tread lightly!
When the effect of the intervention varies signiﬁcantly from
one study to another
It can be attributed to clinical and/or methodological
diversity
Clinical: variability that arises from different populations,
interventions, outcomes and follow-up times
Methodological: relates to differences in trial design and
quality
Detecting quantifying and dealing with heterogeneity can
be very hard

[Poster title]
Summary
Absence of heterogeneity
Assumes that the
true effects of the
studies are all
equal and
deviations occur
because of
imprecision of
results
Analysed with the
ﬁxed-effects
method

[Poster title]
Summary
Presence of heterogeneity
Assumes that
there is variation in
the size of the true
effect among
studies (in addition
to the imprecision
of results)
Analysed with
random-effects
methods

[Poster title]
Summary
Random or fixed?
two ‘schools’ of thought
Fixed-effect (FE)
‘what is the average result of trials conducted to date’?
assumption-free
Random-effects (RE)
‘what is the true treatment effect’?
various assumptions
normally distributed trial effects
varying treatment effect across populations although findings
limited since based on observed studies only
more conservative; findings potentially more generalisable
Researchers reassured when ˆτ2 = 0
FE often used when low heterogeneity detected

[Poster title]
Summary
Not simple
Start
(sort of)
Outcome(s)
continuous
Inverse Variance weighting methods (IV)
Yes
Fixed-effect by
conviction
Fixed-effect IV
model
Yes
No
Detected
heterogeneity
No Random-effects IV model
DL VC ML
REMLPL
Yes
Outcome(s)
dichotomous
No
Maentel-Haenszel methods (MH)
Fixed-effect by
conviction
Fixed-effect MH true
model
Yes
Detected
heterogeneity
No
Combining dichotomous
and continuous outcomes
Transform
dichotomous
outsomes to SMD
Feeling
adventurous?
Yes
Yes!
No!
Rare events
Very rare
events?
Estimate
heterogeneity (τ2
)
No
No
Random-effects MH-IV
hybrid model
Yes
Peto methods (P)
Fixed-effect Peto
true model
Yes
No
Outcome(s)
time-to-event
No
Fixed-effect Peto O-E
true model
Yes
Bayesian?
No
τ2
est
BP
MVa
MVb
Yes
Random-effects
IV model
DL
τ2
estimation
DL DL2 DLb VC
VC2ML REML PL
Non-zero
prior
Yes
τ2
est
B0
No

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Methods and performance
Cochrane database
Outline
4 Summary

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Challenges with two-stage meta-analysis
Heterogeneity is common and the fixed-effect model is
under fire
Methods are asymptotic: accuracy improves as studies
increase. But what if we only have a handful, as is usually
the case?
Almost all random-effects models (except Profile
Likelihood) do not take into account the uncertainty in ˆτ2.
Is this, practically, a problem?
DerSimonian-Laird is the most common method of
analysis, since it is easy to implement and widely available,
but is it the best?

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Challenges with two-stage meta-analysis
...continued
Can be difﬁcult to organise since...
outcomes likely to have been disseminated using a variety
of statistical parameters
appropriate transformations to a common format required
tedious task, requiring at least some statistical adeptness
Parametric random-effects models assume that both the
effects and errors are normally distributed. Are methods
robust?
Sometimes heterogeneity is estimated to be zero,
especially when the number of studies is small. Good
news?

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Outline
the MetaEasy add-in
metaeff & metaan
Cochrane database
4 Summary

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Organising
Data initially collected using data extraction forms
A spreadsheet is the next logical step to summarise the
reported study outcomes and identify missing data
Since in most cases MS Excel will be used we developed
an add-in that can help with most processes involved in
meta-analysis
More useful when the need to combine differently reported
outcomes arises

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
What it can do
Help with the data collection using pre-formatted
worksheets.
Its unique feature, which can be supplementary to other
meta-analysis software, is implementation of methods for
calculating effect sizes (& SEs) from different input types
For each outcome of each study...
it identiﬁes which methods can be used
calculates an effect size and its standard error
selects the most precise method for each outcome

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
What it can do
...continued
Creates a forest plot that summarises all the outcomes,
organised by study
Uses a variety of standard and advanced meta-analysis
methods to calculate an overall effect
a variety of options is available for selecting which
outcome(s) are to be meta-analysed from each study
Plots the results in a second forest plot
Reports a variety of heterogeneity measures, including
Cochran’s Q, I2, HM
2
and ˆτ2 (and its estimated conﬁdence
interval under the Proﬁle Likelihood method)

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Advantages
Free (provided Microsoft Excel is available)
Easy to use and time saving
Extracted data from each study are easily accessible, can
be quickly edited or corrected and analysis repeated
Choice of many meta-analysis models, including some
advanced methods not currently available in other software
packages (e.g. Permutations, Proﬁle Likelihood, REML)
Unique forest plot that allows multiple outcomes per study
Effect sizes and standard errors can be exported for use in
other meta-analysis software packages

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Installing
Latest version available from www.statanalysis.co.uk
Compatible with Excel 2003, 2007 and 2010
Manual provided but also described in:
Kontopantelis E and Reeves D
MetaEasy: A Meta-Analysis Add-In for Microsoft Excel
Journal of Statistical Software, 30(7):1-25, 2009
Play video clip

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Stata implementation
MetaEasy methods implemented in Stata under:
metaeff, which uses the different study input to provide
effect sizes and SEs
metaan, which meta-analyses the study effects with a
ﬁxed-effect or one of ﬁve available random-effects models
To install, type in Stata:
ssc install <command name>
help <command name>
Described in:
metaan: Random-effects meta-analysis
The Stata Journal, 10(3):395-407, 2010

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Many random-effects methods
which to use?
DerSimonian-Laird (DL): Moment-based estimator of both
within and between-study variance
Maximum Likelihood (ML): Improves the variance estimate
using iteration
Restricted Maximum Likelihood (REML): an ML variation
that uses a likelihood function calculated from a
transformed set of data
Proﬁle Likelihood (PL): A more advanced version of ML
that uses nested iterations for converging
Permutations method (PE): Simulates the distribution of
the overall effect using the observed data

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Performance evaluation
our approach
Simulated various distributions for the true
effects:
Normal
Skew-Normal
Uniform
Bimodal
Created datasets of 10,000
meta-analyses for various numbers of
studies and different degrees of
heterogeneity, for each distribution

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Performance evaluation
our approach
Compared all methods in terms of:
Coverage, the rate of true negatives when the overall true
effect is zero
Power, the rate of true positives when the true overall effect
is non-zero

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Coverage by method
Large heterogeneity across various between-study variance distributions
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Zero variance Normal Skew−normal
Bimodal Uniform
H=2.78 − I=64%
FE − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Coverage
0.70
0.75
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Coverage
0.80
0.85
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Coverage
0.90
0.95
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Coverage

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Power by method
Large heterogeneity across various between-study variance distributions
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
FE − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
DL − Power (25th centile)
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
ML − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
REML − Power (25th centile)
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PL − Power (25th centile)
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
%
2 5 8 11 14 17 20 23 26 29 32 35
number of studies
Bimodal Uniform
H=2.78 − I=64%
PE − Power (25th centile)

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Which method then?
Within any method, consistent results across all
distribution types
Robust against even severe violations of normality
PL has a ‘reasonable’ coverage in most situations,
especially for moderate and large heterogeneiry
REML and DL perform similarly and better than PL only
when heterogeneity is low (I2 < 15%)
Described in:
Performance of statistical methods for meta-analysis when
true study effects are non-normally distributed
Stat Methods Med Res, published online Dec 9 2010

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Cochrane Database for Systematic Reviews
Richest resource of meta-analyses in the world
Fifty-four active groups responsible for organising, advising
on and publishing systematic reviews
Authors obliged to use RevMan and submit the data and
analyses ﬁle along with the review, contributing to the
creation of a vast data resource
RevMan offers quite a few ﬁxed-effect choices but only the
DerSimonian-Laird random-effects method has been
implemented to quantify and account for heterogeneity
hidden data

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
‘Real’ Data
Python code to crawl Wiley website for RevMan files
Downloaded 3,845 relevant RevMan files (of 3,984
available in Aug 2012) and imported in Stata
Each file a systematic review
Within each file, various research questions might have
been posed
investigated across various relevant outcomes?
variability in intervention or outcome?

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
‘Real’ Data
Cochrane
database
CD000006
Group: Pregnancy and
Childbirth
Review name:
Absorbable suture
materials for primary
repair of episiotomy and
second degree tears
Meta-analysis 1
Synthetic sutures
versus catgut
Meta-analysis 2
Fast-absorbing synthetic versus
standard absorbable synthetic material
Meta-analysis 3
Glycerol impregnated catgut (softgut)
versus chromic catgut
Meta-analysis 4
Monofilament versus standard
polyglycolic sutures
Outcome 1.1
Short-term pain: pain at day 3 or less
(women experiencing any pain)
Subgroup 1.1.1
Standard synthetic; k=9
Subgroup 1.1.2
Fast absorbing; k=1
Outcome 1.9
Dyspareunia - at 3 months
postpartum
Subgroup 1.9.1
Standard synthetic; k=5
Subgroup 1.9.2
Fast absorbing; k=1
Main 1.9.0
k=6
Main 1.1.0
k=10
Outcome 2.1
Short-term pain: at 3 days or less
Main 2.1.0
k=3
Outcome 2.11
Maternal satisfaction: satisfied with
repair at 12 months
Main 2.11.0
k=1
Outcome 3.1
Short-term pain: pain at 3 days or
less
Main 3.1.0
k=1
Outcome 3.8
Dyspareunia at 6 - 12 months
Main 3.8.0
k=1
Outcome 4.1
Short-term pain: mean pain scores at
3 days
Main 4.1.0
k=1
Outcome 4.4
Wound problems at 8 - 12 weeks:
women seeking professional help for
problem with perineal repair
Main 4.4.0
k=1

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
The questions
Investigate the potential bias when assuming homogeneity
Assess method performance in relation to heterogeneity
estimates in simulations
Examine the distribution of measured heterogeneity in all
Cochrane meta-analyses
Present details on the number of meta-analysed studies
and model selection
Assess the sensitivity of results and conclusions using
alternative models

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Which method?
simulation results
Bayesian methods (assuming a prior heterogeneity value)
did very well for very small meta-analyses
Our suggested variant of the non-parametric bootstrap for
the standard estimate, DLb, seemed best method overall
especially in detecting heterogeneity
which appears to be a big problem: DL failed to detect high
τ2
for over 50% of small meta-analyses

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Meta-analyses numbers
Of the 3,845 ﬁles 2,801 had identiﬁed relevant studies and
contained any data
98,615 analyses extracted 57,397 of which meta-analyses
32,005 were overall meta-analyses
25,392 were subgroup meta-analyses
Estimation of an overall effect
Peto method in 4,340 (7.6%)
Mantel-Haenszel in 33,184 (57.8%)
Inverse variance in 19,873 (34.6%)
random-effects more prevalent in inverse variance methods
and larger meta-analyses
34% of meta-analyses on 2 studies (53% k ≤ 3)!

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Meta-analyses by Cochrane group
22
Figures
Figure 1: All meta-analyses, including single-study and subgroup meta-analyses
0
2000
4000
6000
8000
10000
12000
14000
PregnancyandChildbirth
Schizophrenia
Neonatal
MenstrualDisordersandSubfertility
DepressionAnxietyandNeurosis
Airways
Hepato-Biliary
FertilityRegulation
Musculoskeletal
Stroke
AcuteRespiratoryInfections
Renal
DementiaandCognitiveImprovement
PainPalliativeandSupportiveCare
InfectiousDiseases
Heart
BoneJointandMuscleTrauma
MetabolicandEndocrineDisorders
GynaecologicalCancer
DevelopmentalPsychosocialandLearning…
ColorectalCancer
Hypertension
Anaesthesia
HaematologicalMalignancies
DrugsandAlcohol
Incontinence
InflammatoryBowelDiseaseandFunctional…
MovementDisorders
NeuromuscularDisease
OralHealth
PeripheralVascularDiseases
BreastCancer
TobaccoAddiction
CysticFibrosisandGeneticDisorders
Back
Skin
HIV/AIDS
Injuries
EyesandVision
Wounds
EarNoseandThroatDisorders
Epilepsy
UpperGastrointestinalandPancreaticDiseases
EffectivePracticeandOrganisationofCare
ProstaticDiseasesandUrologicCancers
MultipleSclerosisandRareDiseasesofthe…
MultipleSclerosis
ConsumersandCommunication
LungCancer
SexuallyTransmittedDiseases
ChildhoodCancer
OccupationalSafetyandHealth
SexuallyTransmittedInfections
PublicHealth
Single Study Fixed-effect model (by choice or necessity) Random-effects model

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Meta-analyses by method choice
Figure 2: Model selection by number of available studies (and % of random-effects meta-analyses)*
*note that in many cases fixed-effect models were used when heterogeneity was detected
Figure 3: Comparison of zero between-study variance estimates rates in the Cochrane library data and in simulations,
21%
27%
31%
37%
41%
51%
15%
19%
22%
22%
27%
30%
0
2000
4000
6000
8000
10000
12000
2 3 4 5 6-9 10+
Number of Studies in meta-analysis
Peto (FE) Inverse Variance (FE) Inverse Variance (RE) Mantel-Haenszel (FE) Mantel-Haenszel (RE)

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Comparing Cochrane data with simulated
To assess the validity of a homogeneity assumption we
compared the percentage of zero estimates, in real and
simulated data
Calculated DL heterogeneity estimate for all Cochrane
meta-analyses
Percentage of zero estimates equivalent to moderate-high
heterogeneity simulated data
Suggests that mean true heterogeneity is higher than
generally assumed but fails to be detected; especially for
small meta-analyses

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Comparing Cochrane data with simulated
*note that in many case fixed-effect models were used when heterogeneity was detected
Figure 3: Comparison of zero between-study variance estimates rates in the Cochrane library data and in simulations,
using the DerSimonian-Laird method*
*Normal distribution of the effects assumed in the simulations (more extreme distributions produced similar
results).
Peto (FE) Inverse Variance (FE) Inverse Variance (RE) Mantel-Haenszel (FE) Mantel-Haenszel (RE)
0
10
20
30
40
50
60
70
80
90
100
2 3 4 5 10 20
%ofzeroτ^2estimateswithDerSimonian-Laird
Number of studies in meta-analyis
Observed
true I^2=15%
true I^2=35%
true I^2=64%

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Reanalysing the Cochrane data
We applied all methods to all 57,397 meta-analyses to
assess heterogeneity distributions and the sensitivity of the
results and conclusions
For simplicity discuss differences between standard
methods and DLb; not a perfect method but one that
performed well overall
As in simulations, DLb identiﬁes more heterogeneous
meta-analyses; ˆτ2
DL = 0 for 50.5% & ˆτ2
DLb = 0 for 31.2%

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Distributions for ˆτ2
0500100015002000
#ofmeta-analyses
0 .1 .2 .3 .4 .5
t
2
estimate
Zero est(%): DL=44.9, DLb=29.6, VC=48.9 REML=45.4
ML=62.2, B0=49.2, VC2=44.3, DL2=45.3
Non-convergence(%): ML=0.7, REML=1.4.
Inverse Variance
010002000300040005000
#ofmeta-analyses
0 .1 .2 .3 .4 .5
t
2
estimate
ML=75.0, B0=59.6, VC2=53.9, DL2=55.5
Mantel-Haenszel
0200400600
#ofmeta-analyses
0 .1 .2 .3 .4 .5
t
2
estimate
ML=70.0, B0=54.8, VC2=49.6, DL2=51.0
Peto & O-E
02000400060008000
#ofmeta-analyses
0 .1 .2 .3 .4 .5
t
2
estimate
ML=70.2, B0=55.6, VC2=50.2, DL2=51.6
all methods
non-zero estimates only
DL DLb VC ML
REML B0 VC2 DL2

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Changes in results and conclusions
e.g. inverse variance analyses
RevMan DerSimonian-Laird
Random-effects method says
heterogeneity is present
Analysis with bootstrap DL rarely
changes conclusions (although
higher heterogeneity estimates
and found in around 20% more
meta-analysis
Conclusions change for:
0.9% of analyses
No
Estimated heterogeneity
‘ignored’ by authors and a
fixed-effect model is chosen
Yes
Analysis with bootstrap DL rarely
changes conclusions
2.4% of analyses
No
Analysis with bootstrap DL makes
a difference in 1 in 5 analyses (as
would analysis with standard DL
but to a smaller extent)
19.1% of analyses
Yes

[Poster title]
Summary
the MetaEasy add-in
metaeff & metaan
Cochrane database
Findings
Cochrane database analysis
Detecting and accurately estimating heterogeneity in small
MAs is very difﬁcult; yet for 53% of Cochrane MAs, k ≤ 3
Estimates of zero heterogeneity should be a concern
Bootstrapped DL leads to a small improvement but
problem largely remains, especially for very small MAs
Assume high levels of heterogeneity in sensitivity?
Caution against ignoring heterogeneity when detected
Discussed in:
Kontopantelis E, Springate DA and Reeves D
A Re-Analysis of the Cochrane Library Data: The Dangers
of Unobserved Heterogeneity in Meta-Analyses
PLOS ONE, 10.1371/journal.pone.0069930, 26 July 2013

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Outline
4 Summary

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Forest plot
One advantage of two-stage meta-analysis is the ability to
convey information graphically through a forest plot
study effects available after the ﬁrst stage of the process,
and can be used to demonstrate the relative strength of the
intervention in each study and across all
informative, easy to follow and particularly useful for
readers with little or no methodological experience
key feature of meta-analysis and always presented when
two-stage meta-analyses are performed
In one-stage meta-analysis, only the overall effect is
calculated and creating a forest-plot is not straightforward
Enter ipdforest

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Outline
A practical guide
ipdforest
Examples
Power
4 Summary

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
The hypothetical study
Individual patient data from randomised controlled trials
For each trial we have
a binary control/intervention membership variable
baseline and follow-up data for the continuous outcome
covariates
Assume measurements consistent across trials and
standardisation is not required
We will explore linear random-effects models with the
xtmixed command; application to the logistic case using
xtmelogit should be straightforward
In the models that follow, in general, we denote ﬁxed
effects with ‘γ’s and random effects with ‘β’s

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model 1
fixed common intercept; random treatment effect; fixed effect for baseline
Yij = γ0 + β1jgroupij + γ2Ybij + ij ij ∼ N(0, σ2
j )
β1j = γ1 + u1j u1j ∼ N(0, τ1
2)
i: the patient
j: the trial
Yij: the outcome
γ0: fixed common intercept
β1j: random treatment
effect for trial j
γ1: mean treatment effect
groupij: group membership
γ2: fixed baseline effect
Ybij: baseline score
u1j: random treatment
effect for trial j
τ1
2: between trial variance
ij: error term
σ2
j : within trial variance for
trial j

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model 1

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model basics
Common ﬁxed-effect
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect
common intercept

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model basics
Random-effect
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
intercept
Random-
effects
intercept level

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model basics
Trial speciﬁc ﬁxed-effect
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect trial-
specific intercept

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model 1
Possibly the simplest approach
In Stata it can be expressed as
xtmixed Y i.group Yb || studyid:group,
nocons
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect
common intercept
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
treatment
Random-
effects
treatment
mean
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect
common covariate

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model 2
fixed trial intercepts; random treatment effect; fixed trial effects for baseline
Common intercept & fixed baseline difficult to justify
A more accepted model:
xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4
|| studyid:group, nocons
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect trial-
specific intercept
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
treatment
Random-
effects
treatment
mean
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect trial-
specific covariate

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model 3
random trial intercept; random treatment effect; ﬁxed trial-speciﬁc effects for baseline
Another possibility, althought contentious, is to assume trial
intercepts are random (e.g. multi-centre trial):
xtmixed Y i.group Yb1 Yb2 Yb3 Yb4 ||
studyid:group, cov(uns)
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
intercept
Random-
effects
intercept level
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
treatment
Random-
effects
treatment
mean
0
5
10
15
20
25
30
35
1 2 3 4
Study
Fixed-effect
common covariate

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Model 4
random trial intercept; random treatment effect; random effects for baseline
The baseline could also have been modelled as a
random-effect:
xtmixed Y i.group Yb || studyid:group Yb,
cov(uns)
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
intercept
Random-
effects
intercept level
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
treatment
Random-
effects
treatment
mean
0
5
10
15
20
25
30
35
1 2 3 4
Study
Random-
effects
covariate
Random-
effects
covariate level

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Interactions and covariates
One of the major advantages of IPD meta-analyses
A covariate or an interaction term can be modelled as a
ﬁxed or random effect
In Stata expressed as:
xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4
age i.group#c.age || studyid:group, nocons
If modelled as a random effect, non-convergence issues
more likely to be encountered

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
General
ipdforest is issued following an IPD meta-analysis that
uses mixed effects two-level regression, with patients
nested within trials and a
linear model (xtmixed)
logistic model (xtmelogit)
Provides a meta-analysis summary table and a forest plot
Trial effects are calculated within ipdforest
Can calculate and report both main and interaction effects
Overall effect(s) & variance estimates extracted from
preceding regression
Calculates and reports heterogeneity

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Depression intervention
We apply the ipdforest command to a dataset of 4
depression intervention trials
Complete information in terms of age, gender,
control/intervention group membership, continuous
outcome baseline and endpoint values for 518 patients
Fake author names and generated random continuous &
binary outcome variables, while keeping the covariates at
their actual values
Introduced correlation between baseline and endpoint
scores and between-trial variability
Logistic IPD meta-analysis, followed by ipdforest

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Dataset
. use ipdforest_example.dta,
. describe
Contains data from ipdforest_example.dta
obs: 518
vars: 17 6 Feb 2012 11:14
size: 20,202
storage display value
variable name type format label variable label
studyid byte %22.0g stid Study identifier
patid int %8.0g Patient identifier
group byte %20.0g grplbl Intervention/control group
sex byte %10.0g sexlbl Gender
age float %10.0g Age in years
depB byte %9.0g Binary outcome, endpoint
depBbas byte %9.0g Binary outcome, baseline
depBbas1 byte %9.0g Bin outcome baseline, trial 1
depC float %9.0g Continuous outcome, endpoint
depCbas float %9.0g Continuous outcome, baseline
depCbas1 float %9.0g Cont outcome baseline, trial 1
Sorted by: studyid patid

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
ME logistic regression model - continuous interaction
ﬁxed trial intercepts; ﬁxed trial effects for baseline; random treatment and age effects
. xtmelogit depB group agec sex i.studyid depBbas1 depBbas2 depBbas5 depBbas9 i
> .group#c.agec || studyid:group agec, var nocons or
Mixed-effects logistic regression Number of obs = 518
Group variable: studyid Number of groups = 4
Obs per group: min = 42
avg = 129.5
max = 214
Integration points = 7 Wald chi2(11) = 42.06
Log likelihood = -326.55747 Prob > chi2 = 0.0000
depB Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
group 1.840804 .3666167 3.06 0.002 1.245894 2.71978
agec .9867902 .0119059 -1.10 0.270 .9637288 1.010403
sex .7117592 .1540753 -1.57 0.116 .4656639 1.087912
studyid
2 1.050007 .5725516 0.09 0.929 .3606166 3.057303
5 .8014551 .5894511 -0.30 0.763 .189601 3.387799
9 1.281413 .6886057 0.46 0.644 .4469619 3.673735
depBbas1 3.152908 1.495281 2.42 0.015 1.244587 7.987253
depBbas2 4.480302 1.863908 3.60 0.000 1.982385 10.12574
depBbas5 2.387336 1.722993 1.21 0.228 .5802064 9.823007
depBbas9 1.881203 .7086507 1.68 0.093 .8990569 3.936262
group#c.agec
1 1.011776 .0163748 0.72 0.469 .9801858 1.044385
_cons .5533714 .2398342 -1.37 0.172 .2366472 1.293993
Random-effects Parameters Estimate Std. Err. [95% Conf. Interval]
studyid: Independent
var(group) 8.86e-21 2.43e-11 0 .
var(agec) 5.99e-18 4.40e-11 0 .

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
ipdforest
modelling main effect and interaction
. ipdforest group, fe(sex) re(agec) ia(agec) or
One-stage meta-analysis results using xtmelogit (ML method) and ipdforest
Main effect (group)
Study Effect [95% Conf. Interval] % Weight
Hart 2005 2.118 0.942 4.765 19.88
Richards 2004 2.722 1.336 5.545 30.69
Silva 2008 2.690 0.748 9.676 8.11
Kompany 2009 1.895 0.969 3.707 41.31
Overall effect 1.841 1.246 2.720 100.00
One-stage meta-analysis results using xtmelogit (ML method) and ipdforest
Interaction effect (group x agec)
Study Effect [95% Conf. Interval] % Weight
Hart 2005 0.972 0.901 1.049 19.88
Richards 2004 0.995 0.937 1.055 30.69
Silva 2008 0.987 0.888 1.098 8.11
Kompany 2009 1.077 1.015 1.144 41.31
Overall effect 1.012 0.980 1.044 100.00
Heterogeneity Measures
value [95% Conf. Interval]
I^2 (%) .
H^2 .
tau^2 est 0.000 0.000 .

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Forest plots
main effect and interaction
Overall effect
Kompany 2009
Silva 2008
Richards 2004
Hart 2005
Studies
0 2 3 4 5 6 7 8 9 101
Effect sizes and CIs (ORs)
Main effect (group)
Overall effect
Kompany 2009
Silva 2008
Richards 2004
Hart 2005
Studies
0 .2 .4 .6 .8 1.2 1.41
Effect sizes and CIs (ORs)
Interaction effect (group x agec)

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
Power calculations
to detect a moderator effect
The best approach is through simulations
As always, numerous assumptions need to be made
effect sizes (main and interaction)
exposure and covariate distributions
correlation between variables
within-study (error) variance
between-study (error) variance - ICC
Generate 1000s of data sets using the assumed model(s)
Estimate what % of these give a signiﬁcant p-value for the
interaction
Can be trial and error till desired power level achieved

[Poster title]
Summary
A practical guide
ipdforest
Examples
Power
ipdpower
Currently under review
Stata program that allows for:
continuous, binary or count outcome
various random effects as discussed in ipdforest
various distributional assumptions (e.g. skew-normal
outcome)
main effects and interactions (moderators effects)
Reports power, coverage, effect estimates and
heterogeneity estimates

[Poster title]
Summary
MetaEasy can help organise meta-analysis and can be
very useful if need to combine continuous and binary
outcomes.
Methods implemented in Stata under metaeff and metaan
Zero heterogeneity estimate is a reason to worry;
heterogeneity might be there but we cannot measure or
account for in the model
If detected, even if very small, use a random-effects model
for generalisability?
All methods likely to miss or underestimate heterogeneity
use sensitivity analysis approach?

[Poster title]
Summary
A few different approaches exist for conducting one-stage
IPD meta-analysis
Stata can cope through the xtmixed and the xtmelogit
commands
The ipdforest command aims to help meta-analysts
calculate trial effects
display results in standard meta-analysis tables
produce familiar and ‘expected’ forest-plots
The easiest way to calculate power to detect complex
effects is through simulations
see ipdpower, under review

[Poster title]
Appendix
Thank you!
References
Comments, suggestions:
e.kontopantelis@manchester.ac.uk

[Poster title]
Appendix
Thank you!
References
JSS Journal of Statistical Software
April 2009, Volume 30, Issue 7. http://www.jstatsoft.org/
MetaEasy: A Meta-Analysis Add-In for Microsoft
Excel
National Primary Care Research
and Development Centre
David Reeves
National Primary Care Research
and Development Centre
Abstract
Meta-analysis is a statistical methodology that combines or integrates the results of
several independent clinical trials considered by the analyst to be ‘combinable’ (Huque
1988). However, completeness and user-friendliness are uncommon both in specialised
meta-analysis software packages and in mainstream statistical packages that have to rely
on user-written commands. We implemented the meta-analysis methodology in an Mi-
crosoft Excel add-in which is freely available and incorporates more meta-analysis models
(including the iterative maximum likelihood and profile likelihood) than are usually avail-
able, while paying particular attention to the user-friendliness of the package.
Keywords: meta-analysis, forest plot, Excel, VBA, maximum likelihood, profile likelihood.
1. Introduction
Meta-analysis can be defined as the statistical analysis of a large collection of analysis results
from individual studies - usually Randomised Controlled Trials (RCTs) - for the purpose of
integrating the findings (Glass 1976). Although the debate regarding the quality and ap-
plication caveats of the method is ongoing (Egger and Smith 1997; Bailar 1997), a Medline
(http://medline.cos.com/) search by the authors reveals that the number of meta-analysis
studies published in peer-reviewed journals seems to be growing exponentially (Figure 1).
Published meta-analysis studies (search criterion: Publication Type=meta-analysis) have
risen from 274 in 1990 to 2138 in 2005, while published work that is either a meta-analysis
or deals with meta-analysis issues (search criterion: Keyword=meta-analysis) has increased
from 329 to 3350, in the same period.
A major issue in meta-analysis is the almost inevitable clinical or methodological heterogene-
ity among the combined studies (Eysenck 1994). If the study results differ greatly (large
A Re-Analysis of the Cochrane Library Data: The Dangers
of Unobserved Heterogeneity in Meta-Analyses
Evangelos Kontopantelis1,2,3
*, David A. Springate1,2
, David Reeves1,2
1 Centre for Primary Care, NIHR School for Primary Care Research, Institute of Population Health, University of Manchester, Manchester, United Kingdom, 2 Centre for
Biostatistics, Institute of Population Health, University of Manchester, Manchester, United Kingdom, 3 Centre for Health Informatics, Institute of Population Health,
University of Manchester, Manchester, United Kingdom
Abstract
Background: Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels
of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of
unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta-
analysis methods; c) apply the findings to published meta-analyses.
Methods and Findings: We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using
simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a
zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the
degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the
sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high,
especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both
detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this
new method we found that in cases where heterogeneity had originally been detected but ignored, 17–20% of the
statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it
into account, between 1% and 3%.
Conclusions: When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler
fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is
very likely present but undetected. Our new method represents a small improvement but the problem largely remains,
especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed
moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored.
The Stata Journal (2010)
10, Number 3, pp. 395–407
metaan: Random-effects meta-analysis
National Primary Care
Research & Development Centre
Manchester, UK
e.kontopantelis@manchester.ac.uk
David Reeves
Health Sciences Primary Care
Research Group
Manchester, UK
david.reeves@manchester.ac.uk
Abstract. This article describes the new meta-analysis command metaan, which
can be used to perform fixed- or random-effects meta-analysis. Besides the stan-
dard DerSimonian and Laird approach, metaan offers a wide choice of available
models: maximum likelihood, profile likelihood, restricted maximum likelihood,
and a permutation model. The command reports a variety of heterogeneity mea-
sures, including Cochran’s Q, I2
, H2
M , and the between-studies variance estimate
bτ2
. A forest plot and a graph of the maximum likelihood function can also be
generated.
Keywords: st0201, metaan, meta-analysis, random effect, effect size, maximum
likelihood, profile likelihood, restricted maximum likelihood, REML, permutation
model, forest plot
1 Introduction
Meta-analysis is a statistical methodology that integrates the results of several inde-
pendent clinical trials in general that are considered by the analyst to be “combinable”
(Huque 1988). Usually, this is a two-stage process: in the first stage, the appropriate
summary statistic for each study is estimated; then in the second stage, these statis-
tics are combined into a weighted average. Individual patient data (IPD) methods
exist for combining and meta-analyzing data across studies at the individual patient
level. An IPD analysis provides advantages such as standardization (of marker values,
outcome definitions, etc.), follow-up information updating, detailed data-checking, sub-
group analyses, and the ability to include participant-level covariates (Stewart 1995;
Lambert et al. 2002). However, individual observations are rarely available; addition-
ally, if the main interest is in mean effects, then the two-stage and the IPD approaches
can provide equivalent results (Olkin and Sampson 1998).
This article concerns itself with the second stage of the two-stage approach to meta-
analysis. At this stage, researchers can select between two main approaches—the fixed-
effects (FE) or the random-effects model—in their efforts to combine the study-level
summary estimates and calculate an overall average effect. The FE model is simpler
and assumes the true effect to be the same (homogeneous) across studies. However, ho-
mogeneity has been found to be the exception rather than the rule, and some degree of
true effect variability between studies is to be expected (Thompson and Pocock 1991).
Two sorts of between-studies heterogeneity exist: clinical heterogeneity stems from dif-
c 2010 StataCorp LP st0201
JSS Journal of Statistical Software
MMMMMM YYYY, Volume VV, Issue II. http://www.jstatsoft.org/
Simulation-based power calculations for mixed
effects modelling: ipdpower in Stata
NIHR School for Primary Care Research
David A Springate
Rosa Parisi
Manchester Pharmacy School
David Reeves
Abstract
Simulations are a practical and reliable approach to power calculations, especially for
multi-level mixed effects models where the analytic solutions can be very complex. In
addition, power calculations are model-specific and multi-level mixed effects models are
defined by a plethora of parameters. In other words, model variations in this context are
numerous and so are the tailored algebraic calculations. This article describes ipdpower in
Stata, a new simulations-based command that calculates power for mixed effects two-level
data structures. Although the command was developed having individual patient data
meta-analyses and primary care databases analyses in mind, where patients are nested
within studies and general practices respectively, the methods apply to any two-level
structure.
Keywords: Stata, ipdpower, power, coverage, meta analysis, multi level, mixed effects, random
effects, individual patient data, IPD, primary care databases, PCD.
1. Introduction
The size of primary care databases (PCDs) allows for investigations that cannot normally
be undertaken in much smaller randomised controlled trials (RCTs), such as the moderating
effect of a patient characteristic on the effect of an intervention. However, researchers quite
often underestimate the numbers needed to detect such effects and assume that the size of
the database alone guarantees adequate power for any type of investigation. The essential
Article
Performance of statistical
methods for meta-analysis
when true study effects are
non-normally distributed: A
simulation study
Evangelos Kontopantelis1
and David Reeves2
Abstract
Meta-analysis (MA) is a statistical methodology that combines the results of several independent studies
considered by the analyst to be ‘combinable’. The simplest approach, the fixed-effects (FE) model, assumes
the true effect to be the same in all studies, while the random-effects (RE) family of models allows the true
effect to vary across studies. However, all methods are only correct asymptotically, while some RE models
assume that the true effects are normally distributed. In practice, MA methods are frequently applied
when study numbers are small and the normality of the effect distribution unknown or unlikely. In this
article, we discuss the performance of the FE approach and seven frequentist RE MA methods:
DerSimonian–Laird, Q-based, maximum likelihood, profile likelihood, Biggerstaff–Tweedie, Sidik–
Statistical Methods in Medical Research
21(4) 409–426
! The Author(s) 2010
Reprints and permissions:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/0962280210392008
smm.sagepub.com

Attikon 2014 - Software and model selection challenges in meta-analysis

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Attikon 2014 - Software and model selection challenges in meta-analysis

Similar to Attikon 2014 - Software and model selection challenges in meta-analysis (20)

More from Evangelos Kontopantelis

More from Evangelos Kontopantelis (20)

Recently uploaded

Recently uploaded (20)

Attikon 2014 - Software and model selection challenges in meta-analysis