- 1. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Fixed-effect vs. Random-effects models in meta-analysis Dr. S. A. Rizwan M.D., Public Health Specialist & Lecturer, Saudi Board of Preventive Medicine – Riyadh, Ministry of Health, KSA 25.11.2019 1 With thanks to Michael Borenstein
- 2. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Outline • Provide a description of fixed and of random effects models • Outline the underlying assumptions of these two models in order to clarify the choices a reviewer has in a meta-analysis • Discuss how to estimate key parameters in the model 25.11.2019 2
- 3. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Our choice between the two depends on • Our assumption about how the effect sizes vary in our meta-analysis • The two models are based on different assumptions about the nature of the variation among effect sizes in our research synthesis 25.11.2019 3
- 4. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sampling assumptions • Fixed-effect • Sampling takes place at one level only • Any between-study variance will be ignored when assigning weights • Random-effects • Sampling takes place at two levels • Any between-study variance will be used when assigning weights 25.11.2019 4
- 5. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Fixed-effect • When there is reason to believe that all the studies are functionally identical • When our goal is to compute the common effect size, for the studies in the analysis • Example of drug company has run five studies to assess the effect of a drug • Note that the effect size from each study estimate a single common mean – the fixed-effect • We know that each study will give us a different effect size, but each effect size is an estimate of a common mean, designated in the prior picture as θ 25.11.2019 5
- 6. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 6
- 7. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Random-effects • We assume two components of variation: • – Sampling variation as in our fixed-effect model assumption • – Random variation because the effect sizes themselves are sampled from population of effect sizes • In a random effects model, we know that our effect sizes will differ because they are sampled from an unknown distribution • Our goal in the analysis will be to estimate the mean and variance of the underlying population of effect sizes • All the studies are functionally equivalent • Example of studies culled from publications 25.11.2019 7
- 8. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Random-effects • We see in the picture that each distribution has its own mean that is sampled from the underlying population distribution of effect sizes • That underlying population distribution also has its own variance, τ2, commonly called the variance component • Thus, each effect size has two components of variation, one due to sampling error, and one from the underlying distribution 25.11.2019 8
- 9. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 9
- 10. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Definition of combined effect • Fixed effect model • There is one true effect • Summary effect is estimate of this value • Random effects model • There is a distribution of effects • Summary effect is mean of distribution 25.11.2019 10
- 11. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course How weights are assigned? 25.11.2019 11
- 12. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course W = 1/ (V1) Fixed-effects model 25.11.2019 12
- 13. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Random-effects model 25.11.2019 13 1W = 1/ (V +T 2 )
- 14. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 190 How weights shift? • If within-study variance only, W=1/V • If between-study variance only, W=1/T2 • If both, W=1/(V+T2) 25.11.2019 14
- 15. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 15
- 16. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 16
- 17. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Random vs. Fixed 25.11.2019 17
- 18. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Random vs. Fixed 25.11.2019 18 RE weights are more balanced
- 19. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Random vs. Fixed 25.11.2019 19 RE confidence interval is wider
- 20. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Large study has less impact under RE 25.11.2019 20
- 21. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Small study has more impact under RE 25.11.2019 21
- 22. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course FE 25.11.2019 22
- 23. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Summary effect in Fixed model 25.11.2019 23 244.215 101.171 = 0.414FE
- 24. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Variance of summary effect in fixed model 25.11.2019 24 1 244.215 = 0.004FE
- 25. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course RE 25.11.2019 25
- 26. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Summary effect in random model 25.11.2019 26 90.284 32.342 = 0.358RE
- 27. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Variance of summary effect in random model 25.11.2019 27 1 90.284 = 0.011RE
- 28. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Why does it matter? • One matches the sampling • One does not match the sampling • Wrong model yields incorrect weights • Estimate of mean is wrong • Estimate of CI is wrong • MUST choose based on sampling model • The meaning of the ES is different • Relative weights are closer under RE (effect size will shift) • Absolute weights are smaller under RE (CI will become wider) • p-value will change (less significant in long run but can go either way) 25.11.2019 28
- 29. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Test of null 25.11.2019 29 M M Z = SE M* M* Z* = SE Fixed Random Two sources of error One source of error
- 30. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Implications of the choice • If you expect that the effect size from each study arises from a different population then use random effects • Random effects model is more likely to give non-significant estimates • Recall that all our analyses in a meta-analysis are weighted by the inverse of the variance of the effect size, i.e., by the precision of the effect size estimate • Because we have more variation assumed in a random effects model, our weights for each study will be more equal to one another • In other words, in a fixed effect model, we will more heavily weight larger studies. In a random effects model, the larger studies will not be weighted as heavily 25.11.2019 30
- 31. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Question! • Suppose we had four studies, each with • N =1,000,000, and a true (mean) effect size of 50. • Under the two models, • What would the forest plot look like? • What would the diamond look like? 25.11.2019 31
- 32. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 32
- 33. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 33
- 34. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Which model should we use? • Base decision on the model that matched the way the data were collected • Not on test of homogeneity 25.11.2019 34
- 35. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course What you may hear • Fixed-effect is simple model • Random-effects is more complicated 25.11.2019 35
- 36. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Actually • Fixed-effect is more restricted model • Random-effects makes less assumptions 25.11.2019 36
- 37. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course An alternate view • Random-effects model only makes sense if we have a clear picture of the sampling frame • Otherwise, we should report the mean and CI for the studies in our sample without attempt to generalize to a larger universe 25.11.2019 37
- 38. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Goals in a fixed effect model • In a fixed effect model, we will be most interested in estimating our common effect, θ, and its standard error • We will also want to know if there is heterogeneity present by computing Q • HOWEVER, WE WILL NOT decide on the basis of a significant Q that we should really do a random effects model • WE WILL make a decision about fixed effect versus random effects models because of our substantive knowledge of the area of the systematic review 25.11.2019 38
- 39. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Goals in a random effects model • We also want to estimate a mean effect size, but now this is the mean effect size from the underlying population, μ • We also want to estimate the variance of the underlying effect size, τ2 • We will test heterogeneity by testing whether τ2 is different from 0. • Biggest difficulty: how to estimate τ2 25.11.2019 39
- 40. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Estimating the variance component, τ2 • There are two main methods for computing τ2 • Variously called the method of moments, the DerSimonian/Laird estimate • Restricted maximum likelihood • The method of moments estimator is easy to compute and is based on the value of Q, the homogeneity statistic • Restricted maximum likelihood requires an iterative solution 25.11.2019 40
- 41. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Method of moments estimator 25.11.2019 41
- 42. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Restricted maximum likelihood estimator • Many statisticians do not like the method of moments estimator • Can estimate τ2 using HLM, SAS Proc Mixed, or R 25.11.2019 42
- 43. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Differences between the two models 25.11.2019 43
- 44. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course ES assumptions Fixed effect model • We assume that the treatment effect is the same in all trials. • We use only the sampling variation within the trials. Random effects model • We assume that the treatment effect is the not same in all trials. • The trials are a sample from a population of possible of trials where the treatment effect varies. • We use the sampling variation within the trials and the sampling variation between trials. 25.11.2019 44
- 45. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Sampling assumptions Fixed effect model • If the treatment effect is the same in all trials, it is more powerful and easier. • No assumption about representativeness. Random effects model • Less powerful because P values are larger and confidence intervals are wider. • The trials are a sample from a population of possible of trials where the treatment effect varies. • They must be a representative or random sample. • Very strong assumption. 25.11.2019 45
- 46. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Variance Fixed effect model • Variance of treatment effect in trial = standard error squared. • Weight = 1/variance = 1/SE2 Random effects model • Variance of treatment effect in trial = standard error squared plus inter- trial variance • Weight = 1/variance. • 1 = -------------------------------- SE2 + inter-trial variance • Inter-trial variance has degrees of freedom given by number of trials minus one. • Typically small. 25.11.2019 46
- 47. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course When heterogeneity exists Fixed effect model • When heterogeneity exists we get: • a pooled estimate which may give too much weight to large studies, • a confidence interval which is too narrow, • a P-value which is too small. Random effects model • When heterogeneity exists we get: • possibly a different pooled estimate with a different interpretation, • a wider confidence interval, • a larger P-value. 25.11.2019 47
- 48. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course When heterogeneity doesn’t exist Fixed effect model • When heterogeneity does not exists: • a pooled estimate which is correct, • a confidence interval which is correct, • a P-value which is correct. Random effects model • When heterogeneity does not exist: • a pooled estimate which is correct, • a confidence interval which is too wide, • a P-value which is too large. 25.11.2019 48
- 49. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 49
- 50. Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 06/10 Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course Thank you Kindly email your queries to sarizwan1986@outlook.com 25.11.2019 50