Detecting flawed meta analyses


Published on

Presentation at Society of Behavioral Medicine 2009

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Detecting flawed meta analyses

  1. 1. Meta-analyses to establish the effectiveness of psychosocial interventions in health psychology: Is the literature ready?James C. Coyne 1,2 , Brett Thombs3 ,MariëtHagedoorn21 University of Pennsylvania, School of Medicine, USA2 University Medical Center Groningen, the Netherlands3 McGill University, Montreal, Canada
  2. 2. Narrative Reviews Are an Endangered Species, Soon to be Extinct.
  3. 3. Systematic ReviewInvolves the application ofscientific strategies, in waysthat limit bias, to theassembly, critical appraisal,and synthesis of all relevantstudies that address aspecific clinical question.
  4. 4. There is often a perception that thestatistical combination of data acrossstudies is the most important part of asystematic review. We take such a viewcautiously. We believe that a well-reported, systematic qualitative reviewis much better than an inappropriatelyconducted and reported quantitativereview or meta-analysis. Moher et al., 1998
  5. 5. Exaggerated or Premature Conclusion of a Meta AnalysisDiscourages CommittingResources to Research—TheIssue is Settled, So Why Study It!Ultimately Gets Found Out.Loss of Credibility to the Field.
  6. 6. Zimmermann et al. "Does One Size Fit All?" Moderators inpsychosocial interventions for breast cancer patients: A meta- analysis. Ann Behav Med. 2007; 34: 225-239. First research question: Whether breast cancer patients had better outcomes when they received interventions in a study that only included breast cancer patients compared to studies that included patients with mixed diagnoses. Need to compare the outcome of studies in which breast cancer patients were treated alone to the effects for breast cancer patients in studies in which they were mixed with patients with other cancers. Unfortunately, the separate treatment effects for breast cancer patients cannot be isolated in any of these studies.
  7. 7. Coyne, Thombs, & Hagedoorn. A Meta-Analysis ofPsychosocial Interventions for Cancer Patients Gone Awry. Ann Behav Med. (2009).“Psychosocial intervention” never defined.A diversity of outcomes-- satisfaction with getting a tour,psychological distress, returning to work--consideredcomparable and collapsed within and across studies.Failed to evaluate methodological quality in selectingstudies.Conclusions were influenced by numerous coding andcomputational errors, inexplicable omission of studies,and multiple counting of the same study.
  8. 8. Hoffman, B. M. et al. (2007). "Meta-analysis ofpsychological interventions for chronic low back pain." Health Psychology , 26: 1-9.“Positive effects of psychological interventions… werefound for pain intensity, pain-related interference, health-related quality of life, and depression.”“Multidisciplinary approaches that included apsychological component, when compared with activecontrol conditions, were also noted to have positive short-term effects on pain interference and positive long-termeffects on return to work.“The robust nature of these findings should encourageconfidence among clinicians and researchers alike.”
  9. 9. Critique of Hoffman, B. M. et al. (2007). "Meta-analysis of psychological interventions for chronic low back pain."In 60% of the studies included in the meta-analysis, intervention andcontrol groups were not comparable on key variables at baseline.Only 3 of 34 studies assessed patient adherence to activities requiredby the intervention and only 3 of 34 restricted outside interventions.Less than a third of studies had manualized treatment or protocolsdescribing session by session, and a smaller proportion monitoredtreatment fidelity.Less than half of the studies provided adequate informationconcerning number of patients enrolled, treatment drop-out andreasons for drop-outs.Only 15% of trials provided intent-to-treat analyses.
  10. 10. Critique of Hoffman, B. M. et al. (2007). "Meta-analysis of psychological interventions for chronic low back pain."A lack of evidence that psychological interventionsare superior to other active treatments and of anyenduring effects of psychological interventionsbeyond immediate post-treatment assessments.Lack of benefit for depressive symptomatology.Arguments for the distinctive contribution ofpsychological interventions to multiple modaltreatments assume comparisons that are notpossible from available studies.
  11. 11. Reviewed 4 meta-analyses that recentlyappeared in Health Psychology.Problems with the transparency andcompleteness with which the meta-analyses were reported.Dependence of the meta-analyses onsmall, underpowered trials of generallypoor quality.Conclusions were of questionableclinical validity and utility.
  12. 12. Evidence Based Medicine Reviews in HealthPsychologyDixon, Keefe, Scipio, Perri, & Abernethy,2007Hoffman, Papas, Chatkoff, & Kerns, 2007Irwin, Cole, & Nicassio, 2006Jacobsen, Donovan, Vadaparampil, & Small,2007
  13. 13. Like Pornography? “I shall not attempt to define the kind of material I understand to be embraced within a shorthand description…But “I know it when I see it”, and the motion picture in question in this case is not that.” Justice Potter Stewart
  14. 14. Was the conduct of the meta-analysis accurately andadequately described in thearticle or supplementarymaterials?
  15. 15. Very Useful ResourceCooper, H., S. Maxwell, et al.(2008). "Reporting Standards forResearch in Psychology Why DoWe Need Them? What Might TheyBe?" American Psychologist63(9): 839-851.
  16. 16. Was there an adequate attempt todeal with the methodologicalquality of the originalintervention trials?
  17. 17. Look for:A scoring systemMore than one assessorA strategy for taking quality intoaccount in analyses
  18. 18. Was an adequate effort made to deal with the methodological quality of studies? “Wide variations in the nature of interventions, outcome measures, length of follow-up periods, and presentations of trials’ results prohibited us from using meta-analysis” (p. 561).”
  19. 19. To what extent did theresults of the meta-analysis depend on small, underpowered studies?
  20. 20. Are there enough sufficiently number of studies with adequate sample size to draw a conclusion? Studies with small sample sizes are prone to publication bias. Null results can be dismissed as the result of low statistical power and left unpublished, whereas positive results seem particularly impressive because they are obtained despite low statistical power and get published. Effect sizes found in small underpowered studies tend to overestimate the true effect size. Probability of detecting at least a moderate sized effect when it is present in a study with 35 patients per condition is unacceptably low, about 50-50.
  21. 21. Jacobsen et al.70-80% studies rated as only fair.< 40% intervention and control groupswere comparable at baseline<50% indicated the number of patientsenrolled, treatment drop-out and reasonsfor drop-out
  22. 22. Irwin et al.Exclusion of small trials (n < 35)would have eliminated all studies ofolder adults; five of these studiesincluded 15 or fewer participants percondition. Of the studies includingyounger adults, only one of the 15studies would have remained.
  23. 23. Does the meta analysisadequately deal with clinical heterogeneity and is there abasis for giving a meaningful interpretation to a single summary effect size?
  24. 24. Clinical HeterogeneityCombining studies may beinappropriate for a variety of thefollowing reasons: differences in patienteligibility criteria in the included trials,different interventions and outcomes,and other methodological differencesor missing information. Moher et al., 1998
  25. 25. Clinical ConnoisseurshipConsumers of meta-analyses should be intimately familiarwith clinical phenomena and the nature of interventions,and they need to be able to go beyond the numerical data indetermining whether it is appropriate to integrate studiesthat differ in patient populations and likely restrictions onwho will be willing and able to participate in anintervention, the nature of the intervention, and theappropriateness of outcomes for determining the efficacy ofan intervention. They need to be prepared to make criticalassessments of whether the attempted integration of dataexceeds even the metaphor of mixing apples and oranges.
  26. 26. To what extent does the meta analysisdistinguish interventions that explicitlytargeted clinically significant levels ofproblems (distress, fatigue) versusinterventions that did not have such anaim or that did not require clinicallysignificant levels of the problem?
  27. 27. In Dixon (pain) and Jacobsen(fatigue) patients did not have tomeet a threshold criterion ofpain/fatigue, primary objective ofthe intervention did not have toinvolve pain/fatigue reduction, andpain/fatigue did not have to be aprimary outcome.
  28. 28. Meta Analyses Have Become Powerful and Authoritative ToolsThe hope is that meta analysis provides definitiveconclusions concerning whether interventions workeven when drawing on contradictory and sometimesflawed individual studies.Risk is that bad meta analyses discourage revisitingthe original studies that were integrated in it.Risk is that inaccurate meta analyses will be used fordecisions to pay for delivery of services to patients,override patient preference, and define futureresearch priorities.