Attikon 2014 - Software and model selection challenges in meta-analysis

  • 82 views
Uploaded on

Software and model selection challenges in meta-analysis: MetaEasy, model assumptions, homogeneity and individual patient data

Software and model selection challenges in meta-analysis: MetaEasy, model assumptions, homogeneity and individual patient data

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
82
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary Software and model selection challenges in meta-analysis MetaEasy, model assumptions, homogeneity and IPD Evangelos Kontopantelis Centre for Health Informatics Institute of Population Health University of Manchester Attikon Hospital Athens, 2 June 2014 Kontopantelis Software and model selection challenges in meta-analysis
  • 2. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 3. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 4. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Timeline Efforts to pool results from individual studies back as far as 1904 The first attempt that assessed a therapeutic intervention was published in 1955 In 1976 Glass first used the term to describe a "statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings" Relatively young and dynamic field of research, reflecting importance of MA and potential for conclusive answers Kontopantelis Software and model selection challenges in meta-analysis
  • 5. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Meta-analysing reported study results A two-stage process the relevant summary effect statistics are extracted from published papers on the included studies these are then combined into an overall effect estimate using a suitable meta-analysis model However, problems often arise papers do not report all the statistical information required as input papers report a statistic other than the effect size which needs to be transformed with a loss of precision a study might be too different to be included (population clinically heterogeneous) Kontopantelis Software and model selection challenges in meta-analysis
  • 6. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Individual Patient Data IPD These problems can be avoided when IPD from each study are available outcomes can be easily standardised clinical heterogeneity can be addressed with subgroup analyses and patient-level covariate controlling Can be analysed in a single- or two-stage process mixed-effects regression models can be used to combine information across studies in a single stage this is currently the best approach, with the two-stage method being at best equivalent in certain scenarios Kontopantelis Software and model selection challenges in meta-analysis
  • 7. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Meta-analyses on the rise Meta-analysis studies are used more and more since they seem to be a useful ‘summary’ tool. However critics argue that one cannot combine studies when they are too diverse (heterogeneous). Kontopantelis Software and model selection challenges in meta-analysis
  • 8. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Outline 1 Meta-analysis overview The heterogeneity issue 2 Two-stage meta-analysis 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 9. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue When Robin ruins the party... Kontopantelis Software and model selection challenges in meta-analysis
  • 10. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Heterogeneity tread lightly! When the effect of the intervention varies significantly from one study to another It can be attributed to clinical and/or methodological diversity Clinical: variability that arises from different populations, interventions, outcomes and follow-up times Methodological: relates to differences in trial design and quality Detecting quantifying and dealing with heterogeneity can be very hard Kontopantelis Software and model selection challenges in meta-analysis
  • 11. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Absence of heterogeneity Assumes that the true effects of the studies are all equal and deviations occur because of imprecision of results Analysed with the fixed-effects method Kontopantelis Software and model selection challenges in meta-analysis
  • 12. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Presence of heterogeneity Assumes that there is variation in the size of the true effect among studies (in addition to the imprecision of results) Analysed with random-effects methods Kontopantelis Software and model selection challenges in meta-analysis
  • 13. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Random or fixed? two ‘schools’ of thought Fixed-effect (FE) ‘what is the average result of trials conducted to date’? assumption-free Random-effects (RE) ‘what is the true treatment effect’? various assumptions normally distributed trial effects varying treatment effect across populations although findings limited since based on observed studies only more conservative; findings potentially more generalisable Researchers reassured when ˆτ2 = 0 FE often used when low heterogeneity detected Kontopantelis Software and model selection challenges in meta-analysis
  • 14. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary The heterogeneity issue Not simple Start (sort of) Outcome(s) continuous Inverse Variance weighting methods (IV) Yes Fixed-effect by conviction Fixed-effect IV model Yes No Detected heterogeneity No Random-effects IV model DL VC ML REMLPL Yes Outcome(s) dichotomous No Maentel-Haenszel methods (MH) Fixed-effect by conviction Fixed-effect MH true model Yes Detected heterogeneity No Combining dichotomous and continuous outcomes Transform dichotomous outsomes to SMD Feeling adventurous? Yes Yes! No! Rare events Very rare events? Estimate heterogeneity (τ2 ) No No Random-effects MH-IV hybrid model Yes Peto methods (P) Fixed-effect Peto true model Yes No Outcome(s) time-to-event No Fixed-effect Peto O-E true model Yes Bayesian? No τ2 est BP MVa MVb Yes Random-effects IV model DL τ2 estimation DL DL2 DLb VC VC2ML REML PL Non-zero prior Yes τ2 est B0 No Kontopantelis Software and model selection challenges in meta-analysis
  • 15. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 16. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Challenges with two-stage meta-analysis Heterogeneity is common and the fixed-effect model is under fire Methods are asymptotic: accuracy improves as studies increase. But what if we only have a handful, as is usually the case? Almost all random-effects models (except Profile Likelihood) do not take into account the uncertainty in ˆτ2. Is this, practically, a problem? DerSimonian-Laird is the most common method of analysis, since it is easy to implement and widely available, but is it the best? Kontopantelis Software and model selection challenges in meta-analysis
  • 17. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Challenges with two-stage meta-analysis ...continued Can be difficult to organise since... outcomes likely to have been disseminated using a variety of statistical parameters appropriate transformations to a common format required tedious task, requiring at least some statistical adeptness Parametric random-effects models assume that both the effects and errors are normally distributed. Are methods robust? Sometimes heterogeneity is estimated to be zero, especially when the number of studies is small. Good news? Kontopantelis Software and model selection challenges in meta-analysis
  • 18. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Outline 1 Meta-analysis overview 2 Two-stage meta-analysis the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 19. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Organising Data initially collected using data extraction forms A spreadsheet is the next logical step to summarise the reported study outcomes and identify missing data Since in most cases MS Excel will be used we developed an add-in that can help with most processes involved in meta-analysis More useful when the need to combine differently reported outcomes arises Kontopantelis Software and model selection challenges in meta-analysis
  • 20. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database What it can do Help with the data collection using pre-formatted worksheets. Its unique feature, which can be supplementary to other meta-analysis software, is implementation of methods for calculating effect sizes (& SEs) from different input types For each outcome of each study... it identifies which methods can be used calculates an effect size and its standard error selects the most precise method for each outcome Kontopantelis Software and model selection challenges in meta-analysis
  • 21. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database What it can do ...continued Creates a forest plot that summarises all the outcomes, organised by study Uses a variety of standard and advanced meta-analysis methods to calculate an overall effect a variety of options is available for selecting which outcome(s) are to be meta-analysed from each study Plots the results in a second forest plot Reports a variety of heterogeneity measures, including Cochran’s Q, I2, HM 2 and ˆτ2 (and its estimated confidence interval under the Profile Likelihood method) Kontopantelis Software and model selection challenges in meta-analysis
  • 22. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Advantages Free (provided Microsoft Excel is available) Easy to use and time saving Extracted data from each study are easily accessible, can be quickly edited or corrected and analysis repeated Choice of many meta-analysis models, including some advanced methods not currently available in other software packages (e.g. Permutations, Profile Likelihood, REML) Unique forest plot that allows multiple outcomes per study Effect sizes and standard errors can be exported for use in other meta-analysis software packages Kontopantelis Software and model selection challenges in meta-analysis
  • 23. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Installing Latest version available from www.statanalysis.co.uk Compatible with Excel 2003, 2007 and 2010 Manual provided but also described in: Kontopantelis E and Reeves D MetaEasy: A Meta-Analysis Add-In for Microsoft Excel Journal of Statistical Software, 30(7):1-25, 2009 Play video clip Kontopantelis Software and model selection challenges in meta-analysis
  • 24. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Outline 1 Meta-analysis overview 2 Two-stage meta-analysis the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 25. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Stata implementation MetaEasy methods implemented in Stata under: metaeff, which uses the different study input to provide effect sizes and SEs metaan, which meta-analyses the study effects with a fixed-effect or one of five available random-effects models To install, type in Stata: ssc install <command name> help <command name> Described in: Kontopantelis E and Reeves D metaan: Random-effects meta-analysis The Stata Journal, 10(3):395-407, 2010 Kontopantelis Software and model selection challenges in meta-analysis
  • 26. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Outline 1 Meta-analysis overview 2 Two-stage meta-analysis the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 27. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Many random-effects methods which to use? DerSimonian-Laird (DL): Moment-based estimator of both within and between-study variance Maximum Likelihood (ML): Improves the variance estimate using iteration Restricted Maximum Likelihood (REML): an ML variation that uses a likelihood function calculated from a transformed set of data Profile Likelihood (PL): A more advanced version of ML that uses nested iterations for converging Permutations method (PE): Simulates the distribution of the overall effect using the observed data Kontopantelis Software and model selection challenges in meta-analysis
  • 28. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Performance evaluation our approach Simulated various distributions for the true effects: Normal Skew-Normal Uniform Bimodal Created datasets of 10,000 meta-analyses for various numbers of studies and different degrees of heterogeneity, for each distribution Kontopantelis Software and model selection challenges in meta-analysis
  • 29. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Performance evaluation our approach Compared all methods in terms of: Coverage, the rate of true negatives when the overall true effect is zero Power, the rate of true positives when the true overall effect is non-zero Kontopantelis Software and model selection challenges in meta-analysis
  • 30. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Coverage by method Large heterogeneity across various between-study variance distributions 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% FE − Coverage 0.80 0.85 0.90 0.95 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% DL − Coverage 0.70 0.75 0.80 0.85 0.90 0.95 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% ML − Coverage 0.80 0.85 0.90 0.95 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% REML − Coverage 0.80 0.85 0.90 0.95 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% PL − Coverage 0.90 0.95 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% PE − Coverage Kontopantelis Software and model selection challenges in meta-analysis
  • 31. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Power by method Large heterogeneity across various between-study variance distributions 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% FE − Power (25th centile) 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% DL − Power (25th centile) 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% ML − Power (25th centile) 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% REML − Power (25th centile) 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% PL − Power (25th centile) 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 % 2 5 8 11 14 17 20 23 26 29 32 35 number of studies Zero variance Normal Skew−normal Bimodal Uniform H=2.78 − I=64% PE − Power (25th centile) Kontopantelis Software and model selection challenges in meta-analysis
  • 32. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Which method then? Within any method, consistent results across all distribution types Robust against even severe violations of normality PL has a ‘reasonable’ coverage in most situations, especially for moderate and large heterogeneiry REML and DL perform similarly and better than PL only when heterogeneity is low (I2 < 15%) Described in: Kontopantelis E and Reeves D Performance of statistical methods for meta-analysis when true study effects are non-normally distributed Stat Methods Med Res, published online Dec 9 2010 Kontopantelis Software and model selection challenges in meta-analysis
  • 33. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Outline 1 Meta-analysis overview 2 Two-stage meta-analysis the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 34. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Cochrane Database for Systematic Reviews Richest resource of meta-analyses in the world Fifty-four active groups responsible for organising, advising on and publishing systematic reviews Authors obliged to use RevMan and submit the data and analyses file along with the review, contributing to the creation of a vast data resource RevMan offers quite a few fixed-effect choices but only the DerSimonian-Laird random-effects method has been implemented to quantify and account for heterogeneity hidden data Kontopantelis Software and model selection challenges in meta-analysis
  • 35. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database ‘Real’ Data Cochrane Database for Systematic Reviews Python code to crawl Wiley website for RevMan files Downloaded 3,845 relevant RevMan files (of 3,984 available in Aug 2012) and imported in Stata Each file a systematic review Within each file, various research questions might have been posed investigated across various relevant outcomes? variability in intervention or outcome? Kontopantelis Software and model selection challenges in meta-analysis
  • 36. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database ‘Real’ Data Cochrane Database for Systematic Reviews Cochrane database CD000006 Group: Pregnancy and Childbirth Review name: Absorbable suture materials for primary repair of episiotomy and second degree tears Meta-analysis 1 Synthetic sutures versus catgut Meta-analysis 2 Fast-absorbing synthetic versus standard absorbable synthetic material Meta-analysis 3 Glycerol impregnated catgut (softgut) versus chromic catgut Meta-analysis 4 Monofilament versus standard polyglycolic sutures Outcome 1.1 Short-term pain: pain at day 3 or less (women experiencing any pain) Subgroup 1.1.1 Standard synthetic; k=9 Subgroup 1.1.2 Fast absorbing; k=1 Outcome 1.9 Dyspareunia - at 3 months postpartum Subgroup 1.9.1 Standard synthetic; k=5 Subgroup 1.9.2 Fast absorbing; k=1 Main 1.9.0 k=6 Main 1.1.0 k=10 Outcome 2.1 Short-term pain: at 3 days or less Main 2.1.0 k=3 Outcome 2.11 Maternal satisfaction: satisfied with repair at 12 months Main 2.11.0 k=1 Outcome 3.1 Short-term pain: pain at 3 days or less Main 3.1.0 k=1 Outcome 3.8 Dyspareunia at 6 - 12 months Main 3.8.0 k=1 Outcome 4.1 Short-term pain: mean pain scores at 3 days Main 4.1.0 k=1 Outcome 4.4 Wound problems at 8 - 12 weeks: women seeking professional help for problem with perineal repair Main 4.4.0 k=1 Kontopantelis Software and model selection challenges in meta-analysis
  • 37. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database The questions Investigate the potential bias when assuming homogeneity Assess method performance in relation to heterogeneity estimates in simulations Examine the distribution of measured heterogeneity in all Cochrane meta-analyses Present details on the number of meta-analysed studies and model selection Assess the sensitivity of results and conclusions using alternative models Kontopantelis Software and model selection challenges in meta-analysis
  • 38. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Which method? simulation results Bayesian methods (assuming a prior heterogeneity value) did very well for very small meta-analyses Our suggested variant of the non-parametric bootstrap for the standard estimate, DLb, seemed best method overall especially in detecting heterogeneity which appears to be a big problem: DL failed to detect high τ2 for over 50% of small meta-analyses Kontopantelis Software and model selection challenges in meta-analysis
  • 39. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Meta-analyses numbers Of the 3,845 files 2,801 had identified relevant studies and contained any data 98,615 analyses extracted 57,397 of which meta-analyses 32,005 were overall meta-analyses 25,392 were subgroup meta-analyses Estimation of an overall effect Peto method in 4,340 (7.6%) Mantel-Haenszel in 33,184 (57.8%) Inverse variance in 19,873 (34.6%) random-effects more prevalent in inverse variance methods and larger meta-analyses 34% of meta-analyses on 2 studies (53% k ≤ 3)! Kontopantelis Software and model selection challenges in meta-analysis
  • 40. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Meta-analyses by Cochrane group 22 Figures Figure 1: All meta-analyses, including single-study and subgroup meta-analyses 0 2000 4000 6000 8000 10000 12000 14000 PregnancyandChildbirth Schizophrenia Neonatal MenstrualDisordersandSubfertility DepressionAnxietyandNeurosis Airways Hepato-Biliary FertilityRegulation Musculoskeletal Stroke AcuteRespiratoryInfections Renal DementiaandCognitiveImprovement PainPalliativeandSupportiveCare InfectiousDiseases Heart BoneJointandMuscleTrauma MetabolicandEndocrineDisorders GynaecologicalCancer DevelopmentalPsychosocialandLearning… ColorectalCancer Hypertension Anaesthesia HaematologicalMalignancies DrugsandAlcohol Incontinence InflammatoryBowelDiseaseandFunctional… MovementDisorders NeuromuscularDisease OralHealth PeripheralVascularDiseases BreastCancer TobaccoAddiction CysticFibrosisandGeneticDisorders Back Skin HIV/AIDS Injuries EyesandVision Wounds EarNoseandThroatDisorders Epilepsy UpperGastrointestinalandPancreaticDiseases EffectivePracticeandOrganisationofCare ProstaticDiseasesandUrologicCancers MultipleSclerosisandRareDiseasesofthe… MultipleSclerosis ConsumersandCommunication LungCancer SexuallyTransmittedDiseases ChildhoodCancer OccupationalSafetyandHealth SexuallyTransmittedInfections PublicHealth Single Study Fixed-effect model (by choice or necessity) Random-effects model Kontopantelis Software and model selection challenges in meta-analysis
  • 41. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Meta-analyses by method choice Figure 2: Model selection by number of available studies (and % of random-effects meta-analyses)* *note that in many cases fixed-effect models were used when heterogeneity was detected Figure 3: Comparison of zero between-study variance estimates rates in the Cochrane library data and in simulations, 21% 27% 31% 37% 41% 51% 15% 19% 22% 22% 27% 30% 0 2000 4000 6000 8000 10000 12000 2 3 4 5 6-9 10+ Number of Studies in meta-analysis Peto (FE) Inverse Variance (FE) Inverse Variance (RE) Mantel-Haenszel (FE) Mantel-Haenszel (RE) Kontopantelis Software and model selection challenges in meta-analysis
  • 42. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Comparing Cochrane data with simulated To assess the validity of a homogeneity assumption we compared the percentage of zero estimates, in real and simulated data Calculated DL heterogeneity estimate for all Cochrane meta-analyses Percentage of zero estimates equivalent to moderate-high heterogeneity simulated data Suggests that mean true heterogeneity is higher than generally assumed but fails to be detected; especially for small meta-analyses Kontopantelis Software and model selection challenges in meta-analysis
  • 43. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Comparing Cochrane data with simulated *note that in many case fixed-effect models were used when heterogeneity was detected Figure 3: Comparison of zero between-study variance estimates rates in the Cochrane library data and in simulations, using the DerSimonian-Laird method* *Normal distribution of the effects assumed in the simulations (more extreme distributions produced similar results). Peto (FE) Inverse Variance (FE) Inverse Variance (RE) Mantel-Haenszel (FE) Mantel-Haenszel (RE) 0 10 20 30 40 50 60 70 80 90 100 2 3 4 5 10 20 %ofzeroτ^2estimateswithDerSimonian-Laird Number of studies in meta-analyis Observed true I^2=15% true I^2=35% true I^2=64% Kontopantelis Software and model selection challenges in meta-analysis
  • 44. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Reanalysing the Cochrane data We applied all methods to all 57,397 meta-analyses to assess heterogeneity distributions and the sensitivity of the results and conclusions For simplicity discuss differences between standard methods and DLb; not a perfect method but one that performed well overall As in simulations, DLb identifies more heterogeneous meta-analyses; ˆτ2 DL = 0 for 50.5% & ˆτ2 DLb = 0 for 31.2% Kontopantelis Software and model selection challenges in meta-analysis
  • 45. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Distributions for ˆτ2 0500100015002000 #ofmeta-analyses 0 .1 .2 .3 .4 .5 t 2 estimate Zero est(%): DL=44.9, DLb=29.6, VC=48.9 REML=45.4 ML=62.2, B0=49.2, VC2=44.3, DL2=45.3 Non-convergence(%): ML=0.7, REML=1.4. Inverse Variance 010002000300040005000 #ofmeta-analyses 0 .1 .2 .3 .4 .5 t 2 estimate Zero est(%): DL=54.2, DLb=32.7, VC=58.8 REML=55.6 ML=75.0, B0=59.6, VC2=53.9, DL2=55.5 Non-convergence(%): ML=1.3, REML=1.9. Mantel-Haenszel 0200400600 #ofmeta-analyses 0 .1 .2 .3 .4 .5 t 2 estimate Zero est(%): DL=50.8, DLb=27.3, VC=54.2 REML=51.4 ML=70.0, B0=54.8, VC2=49.6, DL2=51.0 Non-convergence(%): ML=0.6, REML=1.0. Peto & O-E 02000400060008000 #ofmeta-analyses 0 .1 .2 .3 .4 .5 t 2 estimate Zero est(%): DL=50.7, DLb=31.2, VC=55.0 REML=51.7 ML=70.2, B0=55.6, VC2=50.2, DL2=51.6 Non-convergence(%): ML=1.0, REML=1.6. all methods non-zero estimates only DL DLb VC ML REML B0 VC2 DL2 Kontopantelis Software and model selection challenges in meta-analysis
  • 46. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Changes in results and conclusions e.g. inverse variance analyses RevMan DerSimonian-Laird Random-effects method says heterogeneity is present Analysis with bootstrap DL rarely changes conclusions (although higher heterogeneity estimates and found in around 20% more meta-analysis Conclusions change for: 0.9% of analyses No Estimated heterogeneity ‘ignored’ by authors and a fixed-effect model is chosen Yes Analysis with bootstrap DL rarely changes conclusions Conclusions change for: 2.4% of analyses No Analysis with bootstrap DL makes a difference in 1 in 5 analyses (as would analysis with standard DL but to a smaller extent) Conclusions change for: 19.1% of analyses Yes Kontopantelis Software and model selection challenges in meta-analysis
  • 47. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary the MetaEasy add-in metaeff & metaan Methods and performance Cochrane database Findings Cochrane database analysis Detecting and accurately estimating heterogeneity in small MAs is very difficult; yet for 53% of Cochrane MAs, k ≤ 3 Estimates of zero heterogeneity should be a concern Bootstrapped DL leads to a small improvement but problem largely remains, especially for very small MAs Assume high levels of heterogeneity in sensitivity? Caution against ignoring heterogeneity when detected Discussed in: Kontopantelis E, Springate DA and Reeves D A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses PLOS ONE, 10.1371/journal.pone.0069930, 26 July 2013 Kontopantelis Software and model selection challenges in meta-analysis
  • 48. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 49. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Forest plot One advantage of two-stage meta-analysis is the ability to convey information graphically through a forest plot study effects available after the first stage of the process, and can be used to demonstrate the relative strength of the intervention in each study and across all informative, easy to follow and particularly useful for readers with little or no methodological experience key feature of meta-analysis and always presented when two-stage meta-analyses are performed In one-stage meta-analysis, only the overall effect is calculated and creating a forest-plot is not straightforward Enter ipdforest Kontopantelis Software and model selection challenges in meta-analysis
  • 50. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis A practical guide ipdforest Examples Power 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 51. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power The hypothetical study Individual patient data from randomised controlled trials For each trial we have a binary control/intervention membership variable baseline and follow-up data for the continuous outcome covariates Assume measurements consistent across trials and standardisation is not required We will explore linear random-effects models with the xtmixed command; application to the logistic case using xtmelogit should be straightforward In the models that follow, in general, we denote fixed effects with ‘γ’s and random effects with ‘β’s Kontopantelis Software and model selection challenges in meta-analysis
  • 52. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model 1 fixed common intercept; random treatment effect; fixed effect for baseline Yij = γ0 + β1jgroupij + γ2Ybij + ij ij ∼ N(0, σ2 j ) β1j = γ1 + u1j u1j ∼ N(0, τ1 2) i: the patient j: the trial Yij: the outcome γ0: fixed common intercept β1j: random treatment effect for trial j γ1: mean treatment effect groupij: group membership γ2: fixed baseline effect Ybij: baseline score u1j: random treatment effect for trial j τ1 2: between trial variance ij: error term σ2 j : within trial variance for trial j Kontopantelis Software and model selection challenges in meta-analysis
  • 53. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model 1 fixed common intercept; random treatment effect; fixed effect for baseline Kontopantelis Software and model selection challenges in meta-analysis
  • 54. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model basics Common fixed-effect 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect common intercept Kontopantelis Software and model selection challenges in meta-analysis
  • 55. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model basics Random-effect 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects intercept Random- effects intercept level Kontopantelis Software and model selection challenges in meta-analysis
  • 56. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model basics Trial specific fixed-effect 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect trial- specific intercept Kontopantelis Software and model selection challenges in meta-analysis
  • 57. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model 1 fixed common intercept; random treatment effect; fixed effect for baseline Possibly the simplest approach In Stata it can be expressed as xtmixed Y i.group Yb || studyid:group, nocons 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect common intercept 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects treatment Random- effects treatment mean 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect common covariate Kontopantelis Software and model selection challenges in meta-analysis
  • 58. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model 2 fixed trial intercepts; random treatment effect; fixed trial effects for baseline Common intercept & fixed baseline difficult to justify A more accepted model: xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4 || studyid:group, nocons 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect trial- specific intercept 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects treatment Random- effects treatment mean 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect trial- specific covariate Kontopantelis Software and model selection challenges in meta-analysis
  • 59. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model 3 random trial intercept; random treatment effect; fixed trial-specific effects for baseline Another possibility, althought contentious, is to assume trial intercepts are random (e.g. multi-centre trial): xtmixed Y i.group Yb1 Yb2 Yb3 Yb4 || studyid:group, cov(uns) 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects intercept Random- effects intercept level 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects treatment Random- effects treatment mean 0 5 10 15 20 25 30 35 1 2 3 4 Study Fixed-effect common covariate Kontopantelis Software and model selection challenges in meta-analysis
  • 60. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Model 4 random trial intercept; random treatment effect; random effects for baseline The baseline could also have been modelled as a random-effect: xtmixed Y i.group Yb || studyid:group Yb, cov(uns) 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects intercept Random- effects intercept level 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects treatment Random- effects treatment mean 0 5 10 15 20 25 30 35 1 2 3 4 Study Random- effects covariate Random- effects covariate level Kontopantelis Software and model selection challenges in meta-analysis
  • 61. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Interactions and covariates One of the major advantages of IPD meta-analyses A covariate or an interaction term can be modelled as a fixed or random effect In Stata expressed as: xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4 age i.group#c.age || studyid:group, nocons If modelled as a random effect, non-convergence issues more likely to be encountered Kontopantelis Software and model selection challenges in meta-analysis
  • 62. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis A practical guide ipdforest Examples Power 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 63. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power General ipdforest is issued following an IPD meta-analysis that uses mixed effects two-level regression, with patients nested within trials and a linear model (xtmixed) logistic model (xtmelogit) Provides a meta-analysis summary table and a forest plot Trial effects are calculated within ipdforest Can calculate and report both main and interaction effects Overall effect(s) & variance estimates extracted from preceding regression Calculates and reports heterogeneity Kontopantelis Software and model selection challenges in meta-analysis
  • 64. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis A practical guide ipdforest Examples Power 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 65. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Depression intervention We apply the ipdforest command to a dataset of 4 depression intervention trials Complete information in terms of age, gender, control/intervention group membership, continuous outcome baseline and endpoint values for 518 patients Fake author names and generated random continuous & binary outcome variables, while keeping the covariates at their actual values Introduced correlation between baseline and endpoint scores and between-trial variability Logistic IPD meta-analysis, followed by ipdforest Kontopantelis Software and model selection challenges in meta-analysis
  • 66. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Dataset . use ipdforest_example.dta, . describe Contains data from ipdforest_example.dta obs: 518 vars: 17 6 Feb 2012 11:14 size: 20,202 storage display value variable name type format label variable label studyid byte %22.0g stid Study identifier patid int %8.0g Patient identifier group byte %20.0g grplbl Intervention/control group sex byte %10.0g sexlbl Gender age float %10.0g Age in years depB byte %9.0g Binary outcome, endpoint depBbas byte %9.0g Binary outcome, baseline depBbas1 byte %9.0g Bin outcome baseline, trial 1 depBbas2 byte %9.0g Bin outcome baseline, trial 2 depBbas5 byte %9.0g Bin outcome baseline, trial 5 depBbas9 byte %9.0g Bin outcome baseline, trial 9 depC float %9.0g Continuous outcome, endpoint depCbas float %9.0g Continuous outcome, baseline depCbas1 float %9.0g Cont outcome baseline, trial 1 depCbas2 float %9.0g Cont outcome baseline, trial 2 depCbas5 float %9.0g Cont outcome baseline, trial 5 depCbas9 float %9.0g Cont outcome baseline, trial 9 Sorted by: studyid patid Kontopantelis Software and model selection challenges in meta-analysis
  • 67. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power ME logistic regression model - continuous interaction fixed trial intercepts; fixed trial effects for baseline; random treatment and age effects . xtmelogit depB group agec sex i.studyid depBbas1 depBbas2 depBbas5 depBbas9 i > .group#c.agec || studyid:group agec, var nocons or Mixed-effects logistic regression Number of obs = 518 Group variable: studyid Number of groups = 4 Obs per group: min = 42 avg = 129.5 max = 214 Integration points = 7 Wald chi2(11) = 42.06 Log likelihood = -326.55747 Prob > chi2 = 0.0000 depB Odds Ratio Std. Err. z P>|z| [95% Conf. Interval] group 1.840804 .3666167 3.06 0.002 1.245894 2.71978 agec .9867902 .0119059 -1.10 0.270 .9637288 1.010403 sex .7117592 .1540753 -1.57 0.116 .4656639 1.087912 studyid 2 1.050007 .5725516 0.09 0.929 .3606166 3.057303 5 .8014551 .5894511 -0.30 0.763 .189601 3.387799 9 1.281413 .6886057 0.46 0.644 .4469619 3.673735 depBbas1 3.152908 1.495281 2.42 0.015 1.244587 7.987253 depBbas2 4.480302 1.863908 3.60 0.000 1.982385 10.12574 depBbas5 2.387336 1.722993 1.21 0.228 .5802064 9.823007 depBbas9 1.881203 .7086507 1.68 0.093 .8990569 3.936262 group#c.agec 1 1.011776 .0163748 0.72 0.469 .9801858 1.044385 _cons .5533714 .2398342 -1.37 0.172 .2366472 1.293993 Random-effects Parameters Estimate Std. Err. [95% Conf. Interval] studyid: Independent var(group) 8.86e-21 2.43e-11 0 . var(agec) 5.99e-18 4.40e-11 0 . Kontopantelis Software and model selection challenges in meta-analysis
  • 68. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power ipdforest modelling main effect and interaction . ipdforest group, fe(sex) re(agec) ia(agec) or One-stage meta-analysis results using xtmelogit (ML method) and ipdforest Main effect (group) Study Effect [95% Conf. Interval] % Weight Hart 2005 2.118 0.942 4.765 19.88 Richards 2004 2.722 1.336 5.545 30.69 Silva 2008 2.690 0.748 9.676 8.11 Kompany 2009 1.895 0.969 3.707 41.31 Overall effect 1.841 1.246 2.720 100.00 One-stage meta-analysis results using xtmelogit (ML method) and ipdforest Interaction effect (group x agec) Study Effect [95% Conf. Interval] % Weight Hart 2005 0.972 0.901 1.049 19.88 Richards 2004 0.995 0.937 1.055 30.69 Silva 2008 0.987 0.888 1.098 8.11 Kompany 2009 1.077 1.015 1.144 41.31 Overall effect 1.012 0.980 1.044 100.00 Heterogeneity Measures value [95% Conf. Interval] I^2 (%) . H^2 . tau^2 est 0.000 0.000 . Kontopantelis Software and model selection challenges in meta-analysis
  • 69. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Forest plots main effect and interaction Overall effect Kompany 2009 Silva 2008 Richards 2004 Hart 2005 Studies 0 2 3 4 5 6 7 8 9 101 Effect sizes and CIs (ORs) Main effect (group) Overall effect Kompany 2009 Silva 2008 Richards 2004 Hart 2005 Studies 0 .2 .4 .6 .8 1.2 1.41 Effect sizes and CIs (ORs) Interaction effect (group x agec) Kontopantelis Software and model selection challenges in meta-analysis
  • 70. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis A practical guide ipdforest Examples Power 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 71. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power Power calculations to detect a moderator effect The best approach is through simulations As always, numerous assumptions need to be made effect sizes (main and interaction) exposure and covariate distributions correlation between variables within-study (error) variance between-study (error) variance - ICC Generate 1000s of data sets using the assumed model(s) Estimate what % of these give a significant p-value for the interaction Can be trial and error till desired power level achieved Kontopantelis Software and model selection challenges in meta-analysis
  • 72. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary A practical guide ipdforest Examples Power ipdpower Currently under review Stata program that allows for: continuous, binary or count outcome various random effects as discussed in ipdforest various distributional assumptions (e.g. skew-normal outcome) main effects and interactions (moderators effects) Reports power, coverage, effect estimates and heterogeneity estimates Kontopantelis Software and model selection challenges in meta-analysis
  • 73. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary Outline 1 Meta-analysis overview 2 Two-stage meta-analysis 3 One-stage meta-analysis 4 Summary Kontopantelis Software and model selection challenges in meta-analysis
  • 74. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary Two-stage meta-analysis MetaEasy can help organise meta-analysis and can be very useful if need to combine continuous and binary outcomes. Methods implemented in Stata under metaeff and metaan Zero heterogeneity estimate is a reason to worry; heterogeneity might be there but we cannot measure or account for in the model If detected, even if very small, use a random-effects model for generalisability? All methods likely to miss or underestimate heterogeneity use sensitivity analysis approach? Kontopantelis Software and model selection challenges in meta-analysis
  • 75. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary One-stage meta-analysis A few different approaches exist for conducting one-stage IPD meta-analysis Stata can cope through the xtmixed and the xtmelogit commands The ipdforest command aims to help meta-analysts calculate trial effects display results in standard meta-analysis tables produce familiar and ‘expected’ forest-plots The easiest way to calculate power to detect complex effects is through simulations see ipdpower, under review Kontopantelis Software and model selection challenges in meta-analysis
  • 76. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Appendix Thank you! References Comments, suggestions: e.kontopantelis@manchester.ac.uk Kontopantelis Software and model selection challenges in meta-analysis
  • 77. [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor] Appendix Thank you! References JSS Journal of Statistical Software April 2009, Volume 30, Issue 7. http://www.jstatsoft.org/ MetaEasy: A Meta-Analysis Add-In for Microsoft Excel Evangelos Kontopantelis National Primary Care Research and Development Centre David Reeves National Primary Care Research and Development Centre Abstract Meta-analysis is a statistical methodology that combines or integrates the results of several independent clinical trials considered by the analyst to be ‘combinable’ (Huque 1988). However, completeness and user-friendliness are uncommon both in specialised meta-analysis software packages and in mainstream statistical packages that have to rely on user-written commands. We implemented the meta-analysis methodology in an Mi- crosoft Excel add-in which is freely available and incorporates more meta-analysis models (including the iterative maximum likelihood and profile likelihood) than are usually avail- able, while paying particular attention to the user-friendliness of the package. Keywords: meta-analysis, forest plot, Excel, VBA, maximum likelihood, profile likelihood. 1. Introduction Meta-analysis can be defined as the statistical analysis of a large collection of analysis results from individual studies - usually Randomised Controlled Trials (RCTs) - for the purpose of integrating the findings (Glass 1976). Although the debate regarding the quality and ap- plication caveats of the method is ongoing (Egger and Smith 1997; Bailar 1997), a Medline (http://medline.cos.com/) search by the authors reveals that the number of meta-analysis studies published in peer-reviewed journals seems to be growing exponentially (Figure 1). Published meta-analysis studies (search criterion: Publication Type=meta-analysis) have risen from 274 in 1990 to 2138 in 2005, while published work that is either a meta-analysis or deals with meta-analysis issues (search criterion: Keyword=meta-analysis) has increased from 329 to 3350, in the same period. A major issue in meta-analysis is the almost inevitable clinical or methodological heterogene- ity among the combined studies (Eysenck 1994). If the study results differ greatly (large A Re-Analysis of the Cochrane Library Data: The Dangers of Unobserved Heterogeneity in Meta-Analyses Evangelos Kontopantelis1,2,3 *, David A. Springate1,2 , David Reeves1,2 1 Centre for Primary Care, NIHR School for Primary Care Research, Institute of Population Health, University of Manchester, Manchester, United Kingdom, 2 Centre for Biostatistics, Institute of Population Health, University of Manchester, Manchester, United Kingdom, 3 Centre for Health Informatics, Institute of Population Health, University of Manchester, Manchester, United Kingdom Abstract Background: Heterogeneity has a key role in meta-analysis methods and can greatly affect conclusions. However, true levels of heterogeneity are unknown and often researchers assume homogeneity. We aim to: a) investigate the prevalence of unobserved heterogeneity and the validity of the assumption of homogeneity; b) assess the performance of various meta- analysis methods; c) apply the findings to published meta-analyses. Methods and Findings: We accessed 57,397 meta-analyses, available in the Cochrane Library in August 2012. Using simulated data we assessed the performance of various meta-analysis methods in different scenarios. The prevalence of a zero heterogeneity estimate in the simulated scenarios was compared with that in the Cochrane data, to estimate the degree of unobserved heterogeneity in the latter. We re-analysed all meta-analyses using all methods and assessed the sensitivity of the statistical conclusions. Levels of unobserved heterogeneity in the Cochrane data appeared to be high, especially for small meta-analyses. A bootstrapped version of the DerSimonian-Laird approach performed best in both detecting heterogeneity and in returning more accurate overall effect estimates. Re-analysing all meta-analyses with this new method we found that in cases where heterogeneity had originally been detected but ignored, 17–20% of the statistical conclusions changed. Rates were much lower where the original analysis did not detect heterogeneity or took it into account, between 1% and 3%. Conclusions: When evidence for heterogeneity is lacking, standard practice is to assume homogeneity and apply a simpler fixed-effect meta-analysis. We find that assuming homogeneity often results in a misleading analysis, since heterogeneity is very likely present but undetected. Our new method represents a small improvement but the problem largely remains, especially for very small meta-analyses. One solution is to test the sensitivity of the meta-analysis conclusions to assumed moderate and large degrees of heterogeneity. Equally, whenever heterogeneity is detected, it should not be ignored. The Stata Journal (2010) 10, Number 3, pp. 395–407 metaan: Random-effects meta-analysis Evangelos Kontopantelis National Primary Care Research & Development Centre University of Manchester Manchester, UK e.kontopantelis@manchester.ac.uk David Reeves Health Sciences Primary Care Research Group University of Manchester Manchester, UK david.reeves@manchester.ac.uk Abstract. This article describes the new meta-analysis command metaan, which can be used to perform fixed- or random-effects meta-analysis. Besides the stan- dard DerSimonian and Laird approach, metaan offers a wide choice of available models: maximum likelihood, profile likelihood, restricted maximum likelihood, and a permutation model. The command reports a variety of heterogeneity mea- sures, including Cochran’s Q, I2 , H2 M , and the between-studies variance estimate bτ2 . A forest plot and a graph of the maximum likelihood function can also be generated. Keywords: st0201, metaan, meta-analysis, random effect, effect size, maximum likelihood, profile likelihood, restricted maximum likelihood, REML, permutation model, forest plot 1 Introduction Meta-analysis is a statistical methodology that integrates the results of several inde- pendent clinical trials in general that are considered by the analyst to be “combinable” (Huque 1988). Usually, this is a two-stage process: in the first stage, the appropriate summary statistic for each study is estimated; then in the second stage, these statis- tics are combined into a weighted average. Individual patient data (IPD) methods exist for combining and meta-analyzing data across studies at the individual patient level. An IPD analysis provides advantages such as standardization (of marker values, outcome definitions, etc.), follow-up information updating, detailed data-checking, sub- group analyses, and the ability to include participant-level covariates (Stewart 1995; Lambert et al. 2002). However, individual observations are rarely available; addition- ally, if the main interest is in mean effects, then the two-stage and the IPD approaches can provide equivalent results (Olkin and Sampson 1998). This article concerns itself with the second stage of the two-stage approach to meta- analysis. At this stage, researchers can select between two main approaches—the fixed- effects (FE) or the random-effects model—in their efforts to combine the study-level summary estimates and calculate an overall average effect. The FE model is simpler and assumes the true effect to be the same (homogeneous) across studies. However, ho- mogeneity has been found to be the exception rather than the rule, and some degree of true effect variability between studies is to be expected (Thompson and Pocock 1991). Two sorts of between-studies heterogeneity exist: clinical heterogeneity stems from dif- c 2010 StataCorp LP st0201 JSS Journal of Statistical Software MMMMMM YYYY, Volume VV, Issue II. http://www.jstatsoft.org/ Simulation-based power calculations for mixed effects modelling: ipdpower in Stata Evangelos Kontopantelis NIHR School for Primary Care Research University of Manchester David A Springate Institute of Population Health University of Manchester Rosa Parisi Manchester Pharmacy School University of Manchester David Reeves Institute of Population Health University of Manchester Abstract Simulations are a practical and reliable approach to power calculations, especially for multi-level mixed effects models where the analytic solutions can be very complex. In addition, power calculations are model-specific and multi-level mixed effects models are defined by a plethora of parameters. In other words, model variations in this context are numerous and so are the tailored algebraic calculations. This article describes ipdpower in Stata, a new simulations-based command that calculates power for mixed effects two-level data structures. Although the command was developed having individual patient data meta-analyses and primary care databases analyses in mind, where patients are nested within studies and general practices respectively, the methods apply to any two-level structure. Keywords: Stata, ipdpower, power, coverage, meta analysis, multi level, mixed effects, random effects, individual patient data, IPD, primary care databases, PCD. 1. Introduction The size of primary care databases (PCDs) allows for investigations that cannot normally be undertaken in much smaller randomised controlled trials (RCTs), such as the moderating effect of a patient characteristic on the effect of an intervention. However, researchers quite often underestimate the numbers needed to detect such effects and assume that the size of the database alone guarantees adequate power for any type of investigation. The essential Article Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study Evangelos Kontopantelis1 and David Reeves2 Abstract Meta-analysis (MA) is a statistical methodology that combines the results of several independent studies considered by the analyst to be ‘combinable’. The simplest approach, the fixed-effects (FE) model, assumes the true effect to be the same in all studies, while the random-effects (RE) family of models allows the true effect to vary across studies. However, all methods are only correct asymptotically, while some RE models assume that the true effects are normally distributed. In practice, MA methods are frequently applied when study numbers are small and the normality of the effect distribution unknown or unlikely. In this article, we discuss the performance of the FE approach and seven frequentist RE MA methods: DerSimonian–Laird, Q-based, maximum likelihood, profile likelihood, Biggerstaff–Tweedie, Sidik– Statistical Methods in Medical Research 21(4) 409–426 ! The Author(s) 2010 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/0962280210392008 smm.sagepub.com Kontopantelis Software and model selection challenges in meta-analysis