Anatomy of a meta analysis i like


Published on

Evaluates a meta analysis of family therapy interventions for families facing physical illness.

The slide presentation and article is discussed in greater detail at

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Anatomy of a meta analysis i like

  1. 1. Anatomy of a Meta-Analysis I Like James C Coyne, PhD University Medical Center, Groningen, the Netherlands
  2. 2. This set of slides maps items from the QUOROM (Quality of Reporting of Meta-Analyses, Lancet) statement into text from an article reporting a meta analysis of family interventions in chronic illness.
  3. 3. Mechthild Hartmann, Eva Bäznerk, Beate Wild, Ivan Eisler, Wolfgang Herzog. Effects of Interventions Involving the Family in the Treatment of Adult Patients with Chronic Physical Diseases: A Meta-Analysis. Psychotherapy and Psychosomatics, 2010, 7979(3), 136-148.(3), 136-148.
  4. 4. Meta-analysis uses the same basic steps as all research  Formulating problem (are there enough studies)  Gathering studies (file drawer problem)  Coding studies (creating data)  Computing effect sizes (how to do for nonstandard data)  Analyzing data (lots of options)  Interpreting and presenting results (graphics)
  5. 5. Like an experimental study, a report of a meta-analysis should report sufficient details of what was done so that a reader can critically evaluate what was done and how results were interpreted.
  6. 6. Transparency and Reproducibility
  7. 7. However, even more than in the case of an experimental study, the meta- analyses should report details of how studies were coded and analyzed and with what results so that readers can re-analyze results and come to their own independent conclusions.
  8. 8. The QUORUM statement (Quality of Reporting of Meta-Analyses, Lancet, 1999) is now PRISMA (Preferred Reporting Items for Systematic Reviews and Meta- Analyses)
  9. 9. Quorom: Abstract Objectives Uses a structured format states the clinical question explicitly Data Sources The databases and other information sources Review Methods The selection criteria: methods for validity assessment, data abstraction, study characteristics, and quantitative data synthesis Results Characteristics of the RCTs included and excluded; qualitative and quantitative findings, with sub-group analyses if appropriate Conclusion The main results
  10. 10. Quorom: Introduction The explicit clinical problem, biological rationale for the intervention and rationale for the review.
  11. 11. Quorom: Method Searching The information sources in detail and any restrictions (years considered, publication status, language of publication) . Selection The inclusion and exclusion criteria (population, intervention, principal outcomes, study design) . Validity assessment The criteria and process used. Data abstraction The process or processes used (e.g. independently or in duplicate) . Study characteristics The type of study design, participants’ characteristics, details of intervention, outcome definitions . Quantitative data synthesis The principal measures of effect, method of combining results, handling of missing data; how statistical heterogeneity was assessed; a rationale for any a priori sensitivity and sub-group analyses; and any assessment of publication bias .
  12. 12. Quorom: Results Trial flow Provides a meta-analysis profile summarizing trial flow. Study characteristics Presents descriptive data for each trial. Quantitative data synthesis Reports agreement on the selection and validity assessment; presents simple summary results; presents data needed to calculate effect sizes and confidence intervals in intention-to-treat analyses.
  13. 13. Quorom: Discussion Summarizes key findings; discusses clinical inferences based on internal and external validity; interprets the results in the light of the totality of available evidence; describes potential biases in the review process and suggests a future research agenda.
  14. 14. Step 1: Problem Conceptualization and Operationalization The first step is to conceptualize the problem, operationalize the variables, and create the hypotheses. The problem statement should include a specification of the relevant research literature and the major independent and dependent variables (Lipsey & Wilson, 2001).
  15. 15. MH: A separate analysis of chronic physical diseases is reasonable, as evidence suggests that this patient group is more resistant to psychosocial interventions than those presenting mental diseases [21, 22] . In view of the limited recovery rates for traditional biomedical approaches, there is a broad consensus that additional non-pharmacological interventions will be crucial in improving the health outcomes in chronic physical diseases [2, 23]. Here, we perform a systematic review and meta- analysis of randomised controlled trials (RCTs) to assess the effects of family-oriented interventions designed to improve the health of adults with chronic physical disease.
  16. 16. Step 2: Data Collection and Processing. Given that numerous articles will likely be identified, procedures should be established for tracking article collection and organizing citation information (e.g., a researcher may choose to create a bibliographic database).
  17. 17. Study Selection and Search Strategy MH: A study protocol and inclusion/exclusion criteria were established before the search. All psychosocial interventions with family involvement provided by health professionals to improve physical health outcomes were included when classifiable as (psycho)education or addressing relationships. Trials based on information brochures or films without additional education or discussion provided by a health professional were excluded. Trials focused on comparing hospital versus home-based care were also excluded.
  18. 18. MH: We hypothesized that characteristics of the intervention and the samples may explain the heterogeneity in the effect size. We therefore conducted subgroup analyses of the disease groups, types of interventions (psychoeducation, addressing relationships) and types of family members (spouse vs. mixed family members) to identify possible moderators. The influence of the intervention intensity was also assessed via meta- regression analysis.
  19. 19. Validity  Have studies been sought thoroughly: • Medline and other relevant bibliographic database • Cochrane controlled clinical trials register • Foreign language literature • "Grey literature" (unpublished or un-indexed reports: theses, conference proceedings, internal reports, non- indexed journals, pharmaceutical industry files) • Reference chaining from any articles found • Personal approaches to experts in the field to find unpublished reports • Hand searches of the relevant specialized journals.
  20. 20. MH: Our highly sensitive search strategy combined free-text words and medical subject heading (MeSH) terms targeting ‘chronic diseases’, ‘family’, ‘information / education / intervention / psychotherapy’ and ‘randomised controlled trials’. The search was conducted with the following electronic databases: the Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews (issue II, 2007), MEDLINE (1950), PsycIndex (1977), PsycINFO (1887) and CINAHL (1982). Finally, leading scientists in the field and the study authors of registered trials were contacted in order to locate trials not registered in the databases, including unpublished trials. In addition, reference lists of previously published meta-analyses or systematic reviews of all included studies were searched.
  21. 21. MH: Data Extraction and Quality Assessment. To determine which trials to assess, 2 review authors independently scanned the titles and abstracts of every retrieved record. Every article identified as relevant by at least 1 of the review authors was retrieved in full for relevance screening, which was also independently conducted by 2 individuals. Disagreements about eligibility were resolved by discussion with the other authors.
  22. 22. MH: Study characteristics were extracted by 2 review authors using a standardised form developed before the search. If the details of the data were unclear, the study authors were contacted for clarification. We contacted the authors of 18 papers and received replies from 12, of whom 6 were able to provide the statistical data needed. We assessed the methodological quality of the included trials with regard to randomization procedure, allocation concealment, loss to follow-up and intention-to-treat analysis [26].
  23. 23. Step 3: Setting Inclusion and Exclusion Criteria Planning for the inclusion or exclusion criteria is perhaps the most important component of meta-analysis. Such criteria are directly related to internal/external validity and generalizability (Gliner et al., 2003). Factors such as sampling methods, research methods, time frames, publication types, and cultural/language differences of studies should all be considered (Cooper & Hedges, 1994). These decisions are heavily dependent on a researcher’s purposes, goals, and available resources. Lipsey and Wilson (2001) suggest that researchers prepare a detailed, written specification of the criteria a study must meet for inclusion in the meta-analysis.
  24. 24. Validity  Have inclusion and exclusion criteria for studies been stated explicitly, taking account of the patients in the studies, the interventions used, the outcomes recorded and the methodology?
  25. 25. Validity  Have the authors considered the homogeneity of the studies: the idea that the studies are sufficiently similar in their design, interventions and subjects to merit combination. • this is done either by eyeballing graphs like the forest plot or by applications of chi-square tests (Q test)
  26. 26. MH: Given the great variety of diseases, interventions and outcomes, we decided a priori to use random-effects models to calculate the across-study effect size. To assess heterogeneity, we used the I2 statistic [26]. Publication bias was assessed through visual inspection of the funnel plot.
  27. 27. Step 4: Analysis and Results The basic analytic goals of meta-analysis are to: (a) combine and analyze the distribution of effect sizes and (b) examine the relationship between effect sizes and other descriptive variables to understand the variability of effect sizes across studies.
  28. 28. Step 4: Analysis and Results The four basic steps are: 1. Create independent effect sizes for each study. 2. Compute the weighted mean of effect sizes using inverse variance weights. 3. Determine the confidence interval for the mean. 4. Analyze for homogeneity.
  29. 29. MH: The quality of the included RCTs differed considerably. Intention-to-treat analyses were conducted in 29 of the 52 studies. Ten failed to conduct intention-to-treat analyses, and 13 made no statements in this respect. Concealment of allocation could only be verified in 18 studies, primarily because the majority of study authors did not provide this information. Similarly, descriptions of the randomization procedure were reported in only 20 studies. Most studies (n = 45) disclosed a loss to followup, and a high dropout rate of over 30% was found in only 2 trials. There was no evidence for publication bias upon visual inspection of the funnel plots for all outcomes.
  30. 30. MH: Sensitivity analyses were undertaken to determine whether the pooled effect sizes are dependent on study quality or on our decision to use specific outcome hierarchies. All analyses were therefore redone for alternative hierarchies and each hierarchy level separately. We also recalculated the effect sizes using only trials fulfilling the highest methodological standards.
  31. 31. MH: The results of our meta-analysis suggest that family- oriented interventions in chronic physical diseases are more effective for improving physical health outcomes and reducing mental health problems in both patients and caregivers than commonly used treatments. The pooled overall estimate was 0.32 (Hedges’ g ) for physical health of the patient and 0.28 (Hedges’ g ) for patient mental health. The interventions can further be considered useful to promote better health among family members (Hedges’ g = 0.35). All results can be described as small [30] statistically significant effects favouring family-oriented interventions. In light of the large variety in format, content and the intensity of the intervention, the effect was extremely stable. It corresponds to an OR of 1.72–1.84, meaning that patients in the family involvement group had a 72–84% higher chance of improved health compared to the usual care group.
  32. 32. Step 5: Discussion Summarizes key findings; discusses clinical inferences based on internal and external validity; interprets the results in the light of the totality of available evidence; describes potential biases in the review process and suggests a future research agenda.
  33. 33. MH: The primary limitation of our review is common to all meta-analyses, that is the dependency on the quality of the primary studies. We found that the methods for conducting these trials were suboptimal. By restricting our analysis to RCTs, however, we used the best evidence in this field for our review. To account for methodological weaknesses, we conducted sensitivity analyses that demonstrated that study quality did not affect the effect size estimates in a statistically significant manner. A second limitation may be the high degree of heterogeneity, which was present in the primary studies and was reflected in the I2 statistic.
  34. 34. MH: Our current results may have greater implications for future research in chronic medical care than for individual clinical decisions.