Moderator analysis in meta-analysis

Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 08/10
Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course
Moderator analysis in meta-analysis
Dr. S. A. Rizwan M.D.,
Public Health Specialist & Lecturer,
Saudi Board of Preventive Medicine – Riyadh,
Ministry of Health, KSA
With thanks to Ryan Williams
26.11.2019 1

Moderator analyses in meta-analysis
• To test our hypotheses about whether variation among studies in effect
size is associated with differences in study methods or participants
• A priori ideas, incorporating these characteristics of studies into our
coding forms
• Two major forms:
– categorical models analogous to ANOVA
– meta-regression
26.11.2019 2

Assumptions
• Focus on random effects models
• Based on the random effects variance component
26.11.2019 3

Categorical moderators
• When the moderator variable is categorical, estimate ANOVA analogue
• Typically, comparing the mean effect sizes for 2 or more groups
• For example, we will look at a meta-analysis where we compare the mean
effect size for studies published in three different sources: journals,
dissertations, and unpublished studies
26.11.2019 4

Categorical moderators
• With a one-way random effects ANOVA model, recall that we will compute
– A mean effect size and standard error for each group, and then test whether these
means are significantly different from one another
• The mean effect size and standard error require an estimate of the
variance component
• Options: Will we assume that each group has the same variance
component? Or, will we assume that each group has its own variance
component?
26.11.2019 5

Assumptions for separate subgroups
• We believe that the variation among studies is different
between groups.
• For example,
– if we are testing out an intervention and we have studies that use either a
low-income and a high-income group of students, we might believe that there
will be more variation in effectiveness among studies that have mostly low-
income participants
– the effectiveness of an intervention for juvenile delinquents will vary more for
the group that had a prior arrest than for those that do not have a prior arrest
26.11.2019 6

Assumptions for pooled estimate
• We believe that the variation among effect sizes are the same no matter
the group.
• For an intervention review, we may assume that the variation among
studies does not differ within the groups of interest
• Caveat
– We might have to use a pooled estimate if we have small sample sizes within subgroups
– We need at least 5 cases (in general) to be able to estimate a separate variance
component for each subgroup
26.11.2019 7

Flowchart for choosing variance
26.11.2019 8

Steps for a random effects ANOVA
• Make a decision about the use of a pooled or a separate
estimate of the variance component
• Compute the group mean effect sizes, and their standard
errors
• Compare the group mean effect sizes to see if they are
statistically different from one another
26.11.2019 9

Eagly, Johannesen-Schmidt & van Engen (2003)
• This synthesis examines the standardized mean difference estimated in
primary studies for the difference between men and women in their use
of transformational leadership.
• Transformational leadership involves “establishing onself as a role model
by gaining the trust and confidence of followers” (Eagly et al. 2003, p.
570).
• The sample data is a subset of the studies in the full meta- analysis, a set
of 24 studies that compare men and women in their use of
transformational leadership
• Positive effect sizes indicate males score higher than females
26.11.2019 10

Sub-groups
• Example
26.11.2019 11

Sub-groups
26.11.2019 12

Sub-groups
26.11.2019 13

Summary of results – separate variance
estimates for each group
• Journals have a significant
variance component, and
the mean is not different
from zero by magnitude
• Dissertations and
unpublished studies both
have a non-significant
variance component, but
both have medium to large
effect size
Group k Mean 95% CI τ2 p
Journals 13 -0.05 [-0.24, 0.12] 0.09 <0.001
Dissertations 7 -0.47 [-0.69,-0.26] 0.02 0.22
Unpublished 4 -0.16 [-0.30,-0.03] 0.00 0.87
TOTAL 24 -0.16 [-0.29, -0.03] 0.08 <0.001
26.11.2019 14

Summary of results – separate variance
estimates for each group
• The test of the variance
component as different
from zero is exactly the
fixed effects test of
homogeneity.
• To get this test, we
compute the test of
homogeneity within each
group of studies.
Group k Mean 95% CI τ2 p
Journals 13 -0.05 [-0.24, 0.12] 0.09 <0.001
Dissertations 7 -0.47 [-0.69,-0.26] 0.02 0.22
Unpublished 4 -0.16 [-0.30,-0.03] 0.00 0.87
TOTAL 24 -0.16 [-0.29, -0.03] 0.08 <0.001
26.11.2019 15

Test of between group differences
• To test between group differences in a random effects model, we test
whether the variance component for the variation among the random
effects means is equal to zero
• There are several ways to obtain this value
• We will use a test of homogeneity of the three means – we will treat the
three group means as a meta-analysis
26.11.2019 16

Test of between group differences
• We will compute a test of homogeneity using our three
means as if this is a meta-analysis
• We will use the means and their estimated variances to
compute the sums we need to compute the homogeneity test
26.11.2019 17

Computation of Q between groups
Source Mean Var Wt Wt*Mean Wt*Mean2
Journals -0.05 0.008 122.53 -6.13 0.031
Dissertations -0.47 0.012 86.46 -40.64 19.10
Unpublished -0.16 0.005 211.32 -33.82 5.41
SUM 420.31 -80.59 24.54
Q = 24.54 - (-80.59 * -80.59)
420.31
= 9.09 Compare 9.09 to a chi-square
with df=3-1=2. p-value is 0.011
26.11.2019 18

Using the same variance for subgroups
26.11.2019 19

Notes about the CMA forest plot
• Like RevMan, the confidence intervals around each study are the fixed
effects confidence intervals (they use the within-study fixed effects
variance)
• The group means are the random effects means computed using random
effects weights. Their confidence intervals are also use random effects.
• In this example, we are using the same variance component for all groups
26.11.2019 20

Summary from CMA
26.11.2019 21

Notes about CMA results
• We assumed that the variance component was 0.08 for all
studies
• Compared to our separate variance estimates, this value is
smaller than the separate variance estimate for journals, but
larger than the separate estimate for dissertations and
unpublished articles
26.11.2019 22

Another subgroup
example
26.11.2019 23

Reporting random effects categorical
analysis
• The assumption made about the random effects variance: separate
estimate for each group, or the same estimate for all groups.
• Rationale for the choice of variance component
• The random effects mean and CI
• The value of the variance components (or variance component)
• The test of the between-group differences, and its significance
26.11.2019 24

What is meta-regression?
• Meta-regression is a statistical technique used in a meta-
analysis to examine how characteristics of studies are related
to variation in effect sizes across studies
• Meta-regression is analogous to regression analysis but using
effect sizes as our outcomes, and information extracted from
studies as moderators/predictors
26.11.2019 25

What is meta-regression?
• When we have a heterogeneous set of effect sizes, we can use statistical
techniques to examine the association among characteristics of the study
and variation among effect sizes
• We have a plan for these analyses a priori – based on our understanding
of the literature, and a logic model or framework
• Meta-regression used when we have more than one predictor or
moderator (either continuous or categorical)
26.11.2019 26

Form of the meta-regression model
• ! "↓# = $↓0 + $↓1  %↓#1 +…+ $↓& %↓#&
• where ! "↓#  is our generic effect size estimate from study i,
%↓#1 , %↓#2 , … %↓#&  are the moderator variable values for
study i, and $↓0 , $↓1 , … $↓&  are the unknown regression
coefficients.
26.11.2019 27

Recall that the variance of the effect size
• Depends on the sample size for all types of effects we have talked about
• Thus, the precision of each study’s effect size depends on sample size
• This is different from our typical application of regression where we
assume every person has the same “weight”
• Thus, we need to use weighted least squares regression to account for the
fact that the precision of each effect size depends on sample size
26.11.2019 28

Random effects meta-regression
• As in the categorical analysis discussion, we will need an estimate of the
random effects variance for our studies that will be used as our weights in
the regression
• There are many ways to compute the variance component in a random
effects meta-regression
• For now, let’s assume a single variance component for all studies.
26.11.2019 29

Test for model fit in meta-regression
• As in a standard regression model, we can use the regression
ANOVA table for diagnostics about the fit of a meta-
regression
• Recall that in a standard regression analysis, we would get
the following regression ANOVA table:
26.11.2019 30

ANOVA table for normal regression
26.11.2019 31

Test of model fit in meta-regression
• In meta-regression, we use the ANOVA table to get two different Q
statistics:
• QM – model sum of squares, compare to chi-square distribution with p – 1
df (p is number of predictors in the model)
• QR – residual sum of squares, compare to chi-square distribution with k -
p – 1 df (k is the number of studies)
26.11.2019 32

QM - the model sum of squares
• Qmodel is the test of whether at least one of the regression coefficients (not
including the intercept) is different from zero
• We compare QM to a chi-square distribution with p–1 degrees of freedom
with p = # of predictors in model
• If QM is significant, then at least one of the regression coefficients is
different from zero
26.11.2019 33

QR - the error or residual sum of squares
• QR is the test of whether there is more residual variation than we would
expect IF the model “fits” the data
• We compare QR to a chi-square distribution with k - p – 1 degrees of
freedom with k = # of studies/effect sizes, and p = # of predictors in model
• If QR is significant, then we have more error or residual variation to
explain, or that is not accounted for by the variables we have in the model
26.11.2019 34

Significance of regression coefficients
• In a standard regression analysist-tests on the printout tell
which regression coefficients are significantly different from
zero
• Those significant regression coefficients indicate that these
predictors are associated with the outcome
26.11.2019 35

Results of meta-regression
26.11.2019 36

Plot of points & regression line – bubble plot
26.11.2019 37

Example
26.11.2019 38

What to report in a random effects meta-
regression?
• The software and/or method used to compute the results
• The method used to compute the random effects variance
component
• The goodness of fit tests: QModel , and QResidual
• The regression coefficients and their test of significance
26.11.2019 39

Software
• RevMan does not have the capacity for meta-regression
• CMA only allows one predictor in the meta-regression
• STATA, R programs more flexible
26.11.2019 40

Take home messages
• Moderator analysis is a way of understanding heterogeneity in
our meta-analysis
• Categorical moderators are analyzed individually by subgroups
like ANOVA model
• Multiple and/or continuous variables are analyzed by meta-
regression
26.11.2019 41

THANK YOU
Kindly email your queries to sarizwan1986@outlook.com
26.11.2019 42

Moderator analysis in meta-analysis

More Related Content

What's hot

Similar to Moderator analysis in meta-analysis

More from Rizwan S A

Recently uploaded

Moderator analysis in meta-analysis