Amsterdam 2012 - one stage meta-analysis

[Poster title]
[Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4
1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]
Meta-analysis overview
A practical guide
ipdforest
Power
Summary
One-stage meta-analysis in Stata
power issues and analyses with ipdforest
Evan Kontopantelis
Centre for Primary Care
Institute of Population Health
Faculty of Medicine
University of Manchester
Amsterdam, 5 Nov 2012
Kontopantelis ipdforest

[Poster title]
A practical guide
ipdforest
Power
Summary
Outline
1 Meta-analysis overview
2 A practical guide
3 ipdforest
methods
examples
4 Power
5 Summary

[Poster title]
A practical guide
ipdforest
Power
Summary
Timeline
‘Meta’ is a Greek preposition meaning ‘after’, so
meta-analysis =⇒ post-analysis
Efforts to pool results from individual studies back as far as
1904
The first attempt that assessed a therapeutic intervention
was published in 1955
In 1976 Glass first used the term to describe a "statistical
analysis of a large collection of analysis results from
individual studies for the purpose of integrating the
findings"

[Poster title]
A practical guide
ipdforest
Power
Summary
Meta-analysing reported study results
A two-stage process
the relevant summary effect statistics are extracted from
published papers on the included studies
these are then combined into an overall effect estimate
using a suitable meta-analysis model
However, problems often arise
papers do not report all the statistical information required
as input
papers report a statistic other than the effect size which
needs to be transformed with a loss of precision
a study might be too different to be included (population
clinically heterogeneous)

[Poster title]
A practical guide
ipdforest
Power
Summary
Individual Patient Data
IPD
These problems can be avoided when IPD from each
study are available
outcomes can be easily standardised
clinical heterogeneity can be addressed with subgroup
analyses and patient-level covariate controlling
Can be analysed in a single- or two-stage process
mixed-effects regression models can be used to combine
information across studies in a single stage
this is currently the best approach, with the two-stage
method being at best equivalent in certain scenarios

[Poster title]
A practical guide
ipdforest
Power
Summary
Forest plot
One advantage of two-stage meta-analysis is the ability to
convey information graphically through a forest plot
study effects available after the ﬁrst stage of the process,
and can be used to demonstrate the relative strength of the
intervention in each study and across all
informative, easy to follow and particularly useful for
readers with little or no methodological experience
key feature of meta-analysis and always presented when
two-stage meta-analyses are performed
In one-stage meta-analysis, only the overall effect is
calculated and creating a forest-plot is not straightforward
Enter ipdforest

[Poster title]
A practical guide
ipdforest
Power
Summary
The hypothetical study
Individual patient data from randomised controlled trials
For each trial we have
a binary control/intervention membership variable
baseline and follow-up data for the continuous outcome
covariates
Assume measurements consistent across trials and
standardisation is not required
We will explore linear random-effects models with the
xtmixed command; application to the logistic case using
xtmelogit should be straightforward
In the models that follow, in general, we denote ﬁxed
effects with ‘γ’s and random effects with ‘β’s

[Poster title]
A practical guide
ipdforest
Power
Summary
Model 1
fixed common intercept; random treatment effect; fixed effect for baseline
Yij = γ0 + β1jgroupij + γ2Ybij + ij ij ∼ N(0, σ2
j )
β1j = γ1 + u1j u1j ∼ N(0, τ1
2)
i: the patient
j: the trial
Yij: the outcome
γ0: fixed common intercept
β1j: random treatment
effect for trial j
γ1: mean treatment effect
groupij: group membership
γ2: fixed baseline effect
Ybij: baseline score
u1j: random treatment
effect for trial j
τ1
2: between trial variance
ij: error term
σ2
j : within trial variance for
trial j

[Poster title]
A practical guide
ipdforest
Power
Summary
Model 1
fixed common intercept; random treatment effect; fixed effect for baseline
Possibly the simplest approach
In Stata it can be expressed as
xtmixed Y i.group Yb || studyid:group,
nocons
where
studyid, the trial identifier
group, control/intervention membership
Y and Yb, endpoint and baseline scores
note that the nocons option suppresses estimation of the
intercept as a random effect

[Poster title]
A practical guide
ipdforest
Power
Summary
Model 2
fixed trial specific intercepts; random treatment effect; fixed trial-specific effects for
baseline
Common intercept & fixed baseline difficult to justify
A more accepted model allows for different fixed intercepts
and fixed baseline effects for each trial:
Yi j = γ0j + β1j groupi j + γ2j Ybi j + i j
β1j = γ1 + u1j
where
γ0j the fixed intercept for trial j
γ2j the fixed baseline effect for trial j
In Stata expressed as:
xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4
|| studyid:group, nocons
where Yb‘i’=Yb if studyid=‘i’ and zero otherwise

[Poster title]
A practical guide
ipdforest
Power
Summary
Model 3
random trial intercept; random treatment effect; ﬁxed trial-speciﬁc effects for baseline
Another possibility, althought contentious, is to assume trial
intercepts are random (e.g. multi-centre trial):
Yi j = β0j + β1j groupi j + γ2j Ybi j + i j
β0j = γ0 + u0j
β1j = γ1 + u1j
wiser to assume random effects correlation ρ = 0:
i j ∼ N(0, σ2
j ) u0j ∼ N(0, τ2
0 )
u1j ∼ N(0, τ2
1 ) cov(u0j , u1j ) = ρτ0τ1
xtmixed Y i.group Yb1 Yb2 Yb3 Yb4 ||
studyid:group, cov(uns)
cov(uns): allows for distinct estimation of all RE
variance-covariance components

[Poster title]
A practical guide
ipdforest
Power
Summary
Model 4
random trial intercept; random treatment effect; random effects for baseline
The baseline could also have been modelled as a
random-effect:
Yi j = β0j + β1j groupi j + β2j Ybi j + i j
β0j = γ0 + u0j
β1j = γ1 + u1j
β2j = γ2 + u2j
as before, non-zero random effects correlations:
u0j ∼ N(0, τ2
0 ) u1j ∼ N(0, τ2
1 )
u2j ∼ N(0, τ2
2 ) cov(u0j , u1j ) = ρ1τ0τ1
cov(u0j , u2j ) = ρ2τ0τ2 cov(u1j , u2j ) = ρ3τ1τ2
xtmixed Y i.group Yb || studyid:group Yb,
cov(uns)

[Poster title]
A practical guide
ipdforest
Power
Summary
Model 5
Interactions and covariates
A covariate or an interaction term can be modelled as a
ﬁxed or random effect
Assuming continuous and standardised variable age we
can expand Model 2 to include ﬁxed effects for both age
and its interaction with the treatment:
Yi j = γ0j +β1j groupi j +γ2j Ybi j +γ3agei j +γ4groupi j agei j + i j
β1j = γ1 + u1j
xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4
age i.group#c.age || studyid:group, nocons
If modelled as a random effect, non-convergence issues
more likely to be encountered

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
General
ipdforest is issued following an IPD meta-analysis that
uses mixed effects two-level regression, with patients
nested within trials and a
linear model (xtmixed)
or
logistic model (xtmelogit)
Provides a meta-analysis summary table and a forest plot
Trial effects are calculated within ipdforest
Can calculate and report both main and interaction effects
Overall effect(s) and variance estimates are extracted from
the preceding regression

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Process
ipdforest estimates individual trial effects and their
standard errors using one-level linear or logistic
regressions
Following xtmixed, regress is used and following
xtmelogit, logit is used, for each trial
ipdforest controls these regressions for fixed- or
random-effects covariates that were specified in the
preceding two-level regression
User has full control over included covariates in the
command (e.g. specification as fixed- or random-effects)
But we strongly recommend using the same specifications

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Estimation details
In the estimation of individual trial effects, ipdforest
controls for a random-effects covariate (i.e. allowing the
regression coefficient to vary by trial) by including the
covariate as an independent variable in each regression
Control for a fixed-effect covariate (regression coefficient
assumed constant across trials and given by the coefficient
estimated under two-level model) is a little more complex.
Not possible to specify a fixed value for a regression
coefficient under regress and the continuous outcome
variable is adjusted by subtracting the contribution of the
fixed covariates to its values in a first step prior to analysis
For a binary outcome the equivalent is achieved through
use of the offset option in logit

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Heterogeneity
part I
Between-trial variability τ2 in the treatment effect, known
as heterogeneity, arises from differences in trial design,
quality, outcomes or populations
For continuous outcomes, ipdforest reports, I2 and H2
M,
based on the xtmixed output
For binary outcomes, an estimate of the within-trial
variance is not reported under xtmelogit and hence
heterogeneity measures cannot be computed
Between-trial variability estimate ˆτ2 and its conﬁdence
interval is reported under both models.

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Heterogeneity
part II
We are not calculating an IPD version of Cochran’s Q, the
orthodox χ2
k−1 homogeneity test, considering its poor
performance when the number of trials k is small
Besides, taking into account even low levels of τ2 by
adopting a random-effects model is a more conservative
approach than the ﬁxed-effect one
When between-trial variance is estimated to be close to
zero, results with the two approaches converge

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Depression intervention
We apply the ipdforest command to a dataset of 4
depression intervention trials
Complete information in terms of age, gender,
control/intervention group membership, continuous
outcome baseline and endpoint values for 518 patients
Results not published yet; we use fake author names and
generated random continuous & binary outcome variables,
while keeping the covariates at their actual values
Introduced correlation between baseline and endpoint
scores and between-trial variability
Logistic IPD meta-analysis, followed by ipdforest

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Dataset
. use ipdforest_example.dta,
. describe
Contains data from ipdforest_example.dta
obs: 518
vars: 17 6 Feb 2012 11:14
size: 20,202
storage display value
variable name type format label variable label
studyid byte %22.0g stid Study identifier
patid int %8.0g Patient identifier
group byte %20.0g grplbl Intervention/control group
sex byte %10.0g sexlbl Gender
age float %10.0g Age in years
depB byte %9.0g Binary outcome, endpoint
depBbas byte %9.0g Binary outcome, baseline
depBbas1 byte %9.0g Bin outcome baseline, trial 1
depC float %9.0g Continuous outcome, endpoint
depCbas float %9.0g Continuous outcome, baseline
depCbas1 float %9.0g Cont outcome baseline, trial 1
Sorted by: studyid patid

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
ME logistic regression model - continuous interaction
ﬁxed trial intercepts; ﬁxed trial effects for baseline; random treatment and age effects
. xtmelogit depB group agec sex i.studyid depBbas1 depBbas2 depBbas5 depBbas9 i
> .group#c.agec || studyid:group agec, var nocons or
Mixed-effects logistic regression Number of obs = 518
Group variable: studyid Number of groups = 4
Obs per group: min = 42
avg = 129.5
max = 214
Integration points = 7 Wald chi2(11) = 42.06
Log likelihood = -326.55747 Prob > chi2 = 0.0000
depB Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
group 1.840804 .3666167 3.06 0.002 1.245894 2.71978
agec .9867902 .0119059 -1.10 0.270 .9637288 1.010403
sex .7117592 .1540753 -1.57 0.116 .4656639 1.087912
studyid
2 1.050007 .5725516 0.09 0.929 .3606166 3.057303
5 .8014551 .5894511 -0.30 0.763 .189601 3.387799
9 1.281413 .6886057 0.46 0.644 .4469619 3.673735
depBbas1 3.152908 1.495281 2.42 0.015 1.244587 7.987253
depBbas2 4.480302 1.863908 3.60 0.000 1.982385 10.12574
depBbas5 2.387336 1.722993 1.21 0.228 .5802064 9.823007
depBbas9 1.881203 .7086507 1.68 0.093 .8990569 3.936262
group#c.agec
1 1.011776 .0163748 0.72 0.469 .9801858 1.044385
_cons .5533714 .2398342 -1.37 0.172 .2366472 1.293993
Random-effects Parameters Estimate Std. Err. [95% Conf. Interval]
studyid: Independent
var(group) 8.86e-21 2.43e-11 0 .
var(agec) 5.99e-18 4.40e-11 0 .

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
ipdforest
modelling main effect and interaction
. ipdforest group, fe(sex) re(agec) ia(agec) or
One-stage meta-analysis results using xtmelogit (ML method) and ipdforest
Main effect (group)
Study Effect [95% Conf. Interval] % Weight
Hart 2005 2.118 0.942 4.765 19.88
Richards 2004 2.722 1.336 5.545 30.69
Silva 2008 2.690 0.748 9.676 8.11
Kompany 2009 1.895 0.969 3.707 41.31
Overall effect 1.841 1.246 2.720 100.00
Interaction effect (group x agec)
Hart 2005 0.972 0.901 1.049 19.88
Richards 2004 0.995 0.937 1.055 30.69
Silva 2008 0.987 0.888 1.098 8.11
Kompany 2009 1.077 1.015 1.144 41.31
Overall effect 1.012 0.980 1.044 100.00
Heterogeneity Measures
value [95% Conf. Interval]
I^2 (%) .
H^2 .
tau^2 est 0.000 0.000 .

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Forest plots
main effect and interaction
Overall effect
Kompany 2009
Silva 2008
Richards 2004
Hart 2005
Studies
0 2 3 4 5 6 7 8 9 101
Effect sizes and CIs (ORs)
Main effect (group)
Overall effect
Kompany 2009
Silva 2008
Richards 2004
Hart 2005
Studies
0 .2 .4 .6 .8 1.2 1.41
Interaction effect (group x agec)

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
ME logistic regression model - binary interaction
ﬁxed trial intercepts; ﬁxed trial effects for baseline & age cat; random treatment effect
. xtmelogit depB group i.agecat sex i.studyid depBbas1 depBbas2 depBbas5 depBba
> s9 i.group#i.agecat || studyid:group, var nocons or
Mixed-effects logistic regression Number of obs = 518
Group variable: studyid Number of groups = 4
Obs per group: min = 42
avg = 129.5
max = 214
Integration points = 7 Wald chi2(11) = 42.67
Log likelihood = -326.24961 Prob > chi2 = 0.0000
depB Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
group 1.931763 .5702338 2.23 0.026 1.083151 3.445233
1.agecat .7389353 .213031 -1.05 0.294 .4199616 1.300179
sex .7173044 .154959 -1.54 0.124 .469698 1.095439
studyid
2 1.029638 .5608887 0.05 0.957 .3539959 2.994823
5 .828577 .6082301 -0.26 0.798 .1965599 3.492777
9 1.266728 .6812765 0.44 0.660 .4414556 3.634794
depBbas1 3.196139 1.515334 2.45 0.014 1.261999 8.094542
depBbas2 4.625802 1.923028 3.68 0.000 2.047989 10.44832
depBbas5 2.354493 1.692149 1.19 0.233 .5756364 9.630454
depBbas9 1.902359 .7183054 1.70 0.089 .9075907 3.987446
group#agecat
1 0 .9277371 .3558053 -0.20 0.845 .4374944 1.967331
1 1 1 (omitted)
_cons .6175744 .2800575 -1.06 0.288 .253914 1.502076
Random-effects Parameters Estimate Std. Err. [95% Conf. Interval]
studyid: Identity
var(group) 2.24e-19 1.24e-10 0 .

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
ipdforest
modelling main effect and interaction
. ipdforest group, fe(sex i.agecat) ia(i.agecat) or
Main effect (group), agecat=0
Mead 2005 3.594 1.133 11.402 19.88
Willemse 2004 2.814 1.034 7.657 30.69
Lovell 2008 3.750 0.585 24.038 8.11
Meyer 2009 1.010 0.475 2.147 41.31
Overall effect 1.792 1.079 2.976 100.00
Mead 2005 1.166 0.376 3.617 19.88
Willemse 2004 2.566 1.007 6.543 30.69
Lovell 2008 1.935 0.340 11.003 8.11
Meyer 2009 3.551 1.134 11.126 41.31
Overall effect 1.932 1.083 3.445 100.00
Heterogeneity Measures
value [95% Conf. Interval]
I^2 (%) .
H^2 .
tau^2 est 0.000 0.000 .

[Poster title]
A practical guide
ipdforest
Power
Summary
methods
examples
Forest plots
main effect for each age group
Overall effect
Meyer 2009
Lovell 2008
Willemse 2004
Mead 2005
Studies
0 2 4 6 8 10 12 14 16 18 20 22 24 261
Overall effect
Meyer 2009
Lovell 2008
Willemse 2004
Mead 2005
Studies
0 2 4 6 8 10 121

[Poster title]
A practical guide
ipdforest
Power
Summary
Power calculations
to detect a moderator effect
The best approach is through simulations
As always, numerous assumptions need to be made
effect sizes (main and interaction)
exposure and covariate distributions
correlation between variables
within-study (error) variance
between-study (error) variance - ICC
Generate 100s of data sets using the assumed model(s)
Estimate what % of these give a signiﬁcant p-value for the
interaction
Can be trial and error till desired power level achieved

[Poster title]
A practical guide
ipdforest
Power
Summary
Research on power calculations with simulations
The Stata Journal (2002)
2, Number 2, pp. 107–124
Power by simulation
A. H. Feiveson
NASA Johnson Space Center
alan.h.feiveson1@jsc.nasa.gov
Abstract. This paper describes how to write Stata programs to estimate the power
of virtually any statistical test that Stata can perform. Examples given include
the t test, Poisson regression, Cox regression, and the nonparametric rank-sum
test.
Keywords: st0010, power, simulation, random number generation, postfile, copula,
sample size
1 Introduction
Statisticians know that in order to properly design a study producing experimental
data, one needs to have some idea of whether the scope of the study is sufficient to give
a reasonable expectation that hypothesized effects will be detectable over experimental
error. Most of us have at some time used published procedures or canned software
to obtain sample sizes for studies intended to be analyzed by t tests, or even analysis
of variance. However, with more complex methods for describing and analyzing both
continuous and discrete data (for example, generalized linear models, survival models,
selection models), possibly with provisions for random effects and robust variance es-
timation, the closed-form expressions for power or for sample sizes needed to achieve
a certain power do not exist. Nevertheless, Stata users with moderate programming
ability can write their own routines to estimate the power of virtually any statistical
test that Stata can perform.
2 General approach to estimating power
2.1 Statistical inference
Before describing methodology for estimating power, we first define the term “power”
and illustrate the quantities that can affect it. In this article, we restrict ourselves
to classical (as opposed to Bayesian) methodology for performing statistical inference
on the effect of experimental variable(s) X on a response Y . This is accomplished by
calculating a p-value that attempts to probabilistically describe the extent to which the
data are consistent with a null hypothesis, H0, that X has no effect on Y . We would
then reject H0 if the p-value is less than a predesignated threshold α. It should be
pointed out that considerable criticism of the use of p-values for statistical inference has
COMMENTARY Open Access
Simulation methods to estimate design power:
an overview for applied research
Benjamin F Arnold1*†
, Daniel R Hogan2†
, John M Colford Jr1
and Alan E Hubbard3
Abstract
Background: Estimating the required sample size and statistical power for a study is an integral part of study
design. For standard designs, power equations provide an efficient solution to the problem, but they are
unavailable for many complex study designs that arise in practice. For such complex study designs, computer
simulation is a useful alternative for estimating study power. Although this approach is well known among
statisticians, in our experience many epidemiologists and social scientists are unfamiliar with the technique. This
article aims to address this knowledge gap.
Methods: We review an approach to estimate study power for individual- or cluster-randomized designs using
computer simulation. This flexible approach arises naturally from the model used to derive conventional power
equations, but extends those methods to accommodate arbitrarily complex designs. The method is universally
applicable to a broad range of designs and outcomes, and we present the material in a way that is approachable
for quantitative, applied researchers. We illustrate the method using two examples (one simple, one complex)
based on sanitation and nutritional interventions to improve child growth.
Results: We first show how simulation reproduces conventional power estimates for simple randomized designs
over a broad range of sample scenarios to familiarize the reader with the approach. We then demonstrate how to
extend the simulation approach to more complex designs. Finally, we discuss extensions to the examples in the
article, and provide computer code to efficiently run the example simulations in both R and Stata.
Conclusions: Simulation methods offer a flexible option to estimate statistical power for standard and non-
traditional study designs and parameters of interest. The approach we have described is universally applicable for
evaluating study designs used in epidemiologic and social science research.
Keywords: Computer Simulation, Power, Research Design, Sample Size
Background
Estimating the sample size and statistical power for a
study is an integral part of study design and has profound
consequences for the cost and statistical precision of a
study. There exist analytic (closed-form) power equations
for simple designs such as parallel randomized trials with
treatment assigned at the individual level or cluster
(group) level [1]. Statisticians have also derived equations
to estimate power for more complex designs, such as
designs with two levels of correlation [2] or designs with
two levels of correlation, multiple treatments and
attrition [3]. The advantage of using an equation to esti-
mate power for study designs is that the approach is fast
and easy to implement using existing software. For this
reason, power equations are used to inform most study
designs. However, in our applied research we have routi-
nely encountered study designs that do not conform to
conventional power equations (e.g. multiple treatment
interventions, where one treatment is deployed at the
group level and a second at the individual level). In these
situations, simulation techniques offer a flexible alterna-
tive that is easy to implement in modern statistical
software.
Here, we provide an overview of a general method to* Correspondence: benarnold@berkeley.edu
Arnold et al. BMC Medical Research Methodology 2011, 11:94
http://www.biomedcentral.com/1471-2288/11/94

[Poster title]
A practical guide
ipdforest
Power
Summary
What to take home
A few different approaches exist for conducting one-stage
IPD meta-analysis
Stata can cope through the xtmixed and the xtmelogit
commands
The ipdforest command aims to help meta-analysts
calculate trial effects
display results in standard meta-analysis tables
produce familiar and ‘expected’ forest-plots
The best way to calculate power to detect complex effects
is through simulations

[Poster title]
Appendix Thank you!
Comments, suggestions:
e.kontopantelis@manchester.ac.uk

Amsterdam 2012 - one stage meta-analysis

Recommended

Recommended

More Related Content

Similar to Amsterdam 2012 - one stage meta-analysis

Similar to Amsterdam 2012 - one stage meta-analysis (18)

More from Evangelos Kontopantelis

More from Evangelos Kontopantelis (20)

Recently uploaded

Recently uploaded (20)

Amsterdam 2012 - one stage meta-analysis