This document provides a critique of 4 recent meta-analyses published in Health Psychology. It finds problems with transparency and completeness in how the meta-analyses were reported. It also notes a dependence on small, underpowered original trials of generally poor quality. The document questions the clinical validity and utility of conclusions drawn from these meta-analyses due to issues like clinical heterogeneity among studies and lack of consideration of methodological quality. Overall, it aims to encourage more rigorous standards for meta-analyses to avoid inaccurate or exaggerated conclusions.

1. Meta-analyses to establish the
effectiveness of psychosocial
interventions in health psychology: Is
the literature ready?
James C. Coyne 1,2 , Brett Thombs3 ,Mariët
Hagedoorn2
1
University of Pennsylvania, School of Medicine, USA
2
University Medical Center Groningen, the Netherlands
3
McGill University, Montreal, Canada

2. Narrative Reviews Are
an Endangered
Species, Soon to be
Extinct.

3. Systematic Review
Involves the application of
scientific strategies, in ways
that limit bias, to the
assembly, critical appraisal,
and synthesis of all relevant
studies that address a
specific clinical question.

4. There is often a perception that the
statistical combination of data across
studies is the most important part of a
systematic review. We take such a view
cautiously. We believe that a well-
reported, systematic qualitative review
is much better than an inappropriately
conducted and reported quantitative
review or meta-analysis.
Moher et al., 1998

5. Exaggerated or Premature
Conclusion of a Meta Analysis
Discourages Committing
Resources to Research—The
Issue is Settled, So Why Study It!
Ultimately Gets Found Out.
Loss of Credibility to the Field.

6. Zimmermann et al. "Does One Size Fit All?" Moderators in
psychosocial interventions for breast cancer patients: A meta-
analysis. Ann Behav Med. 2007; 34: 225-239.
First research question: Whether breast cancer patients
had better outcomes when they received interventions in a
study that only included breast cancer patients compared
to studies that included patients with mixed diagnoses.
Need to compare the outcome of studies in which breast
cancer patients were treated alone to the effects for breast
cancer patients in studies in which they were mixed with
patients with other cancers.
Unfortunately, the separate treatment effects for breast
cancer patients cannot be isolated in any of these studies.

7. Coyne, Thombs, & Hagedoorn. A Meta-Analysis of
Psychosocial Interventions for Cancer Patients Gone
Awry. Ann Behav Med. (2009).
“Psychosocial intervention” never defined.
A diversity of outcomes-- satisfaction with getting a tour,
psychological distress, returning to work--considered
comparable and collapsed within and across studies.
Failed to evaluate methodological quality in selecting
studies.
Conclusions were influenced by numerous coding and
computational errors, inexplicable omission of studies,
and multiple counting of the same study.

8. Hoffman, B. M. et al. (2007). "Meta-analysis of
psychological interventions for chronic low back
pain." Health Psychology , 26: 1-9.
“Positive effects of psychological interventions… were
found for pain intensity, pain-related interference, health-
related quality of life, and depression.”
“Multidisciplinary approaches that included a
psychological component, when compared with active
control conditions, were also noted to have positive short-
term effects on pain interference and positive long-term
effects on return to work.
“The robust nature of these findings should encourage
confidence among clinicians and researchers alike.”

9. Critique of Hoffman, B. M. et al. (2007). "Meta-
analysis of psychological interventions for chronic
low back pain."
In 60% of the studies included in the meta-analysis, intervention and
control groups were not comparable on key variables at baseline.
Only 3 of 34 studies assessed patient adherence to activities required
by the intervention and only 3 of 34 restricted outside interventions.
Less than a third of studies had manualized treatment or protocols
describing session by session, and a smaller proportion monitored
treatment fidelity.
Less than half of the studies provided adequate information
concerning number of patients enrolled, treatment drop-out and
reasons for drop-outs.
Only 15% of trials provided intent-to-treat analyses.

10. Critique of Hoffman, B. M. et al. (2007). "Meta-
analysis of psychological interventions for chronic
low back pain."
A lack of evidence that psychological interventions
are superior to other active treatments and of any
enduring effects of psychological interventions
beyond immediate post-treatment assessments.
Lack of benefit for depressive symptomatology.
Arguments for the distinctive contribution of
psychological interventions to multiple modal
treatments assume comparisons that are not
possible from available studies.

11. Reviewed 4 meta-analyses that recently
appeared in Health Psychology.
Problems with the transparency and
completeness with which the meta-
analyses were reported.
Dependence of the meta-analyses on
small, underpowered trials of generally
poor quality.
Conclusions were of questionable
clinical validity and utility.

12. Evidence Based Medicine Reviews in Health
Psychology
Dixon, Keefe, Scipio, Perri, & Abernethy,
2007
Hoffman, Papas, Chatkoff, & Kerns, 2007
Irwin, Cole, & Nicassio, 2006
Jacobsen, Donovan, Vadaparampil, & Small,
2007

13. Like Pornography?
“I shall not attempt to
define the kind of
material I understand
to be embraced within
a shorthand
description…But “I
know it when I see it”,
and the motion picture
in question in this case
is not that.”
Justice Potter Stewart

14. Was the conduct of the meta-
analysis accurately and
adequately described in the
article or supplementary
materials?

15. Very Useful Resource
Cooper, H., S. Maxwell, et al.
(2008). "Reporting Standards for
Research in Psychology Why Do
We Need Them? What Might They
Be?" American Psychologist
63(9): 839-851.

16. Was there an adequate attempt to
deal with the methodological
quality of the original
intervention trials?

17. Look for:
A scoring system
More than one assessor
A strategy for taking quality into
account in analyses

18. Was an adequate effort made to deal with
the methodological quality of studies?
“Wide variations in the nature
of interventions, outcome
measures, length of follow-up
periods, and presentations of
trials’ results prohibited us from
using meta-analysis” (p. 561).”

19. To what extent did the
results of the meta-analysis
depend on small,
underpowered studies?

20. Are there enough sufficiently number of
studies with adequate sample size to draw a
conclusion?
 Studies with small sample sizes are prone to publication
bias.
 Null results can be dismissed as the result of low
statistical power and left unpublished, whereas positive
results seem particularly impressive because they are
obtained despite low statistical power and get published.
 Effect sizes found in small underpowered studies tend to
overestimate the true effect size.
 Probability of detecting at least a moderate sized effect
when it is present in a study with 35 patients per condition
is unacceptably low, about 50-50.

21. Jacobsen et al.
70-80% studies rated as only fair.
< 40% intervention and control groups
were comparable at baseline
<50% indicated the number of patients
enrolled, treatment drop-out and reasons
for drop-out

22. Irwin et al.
Exclusion of small trials (n < 35)
would have eliminated all studies of
older adults; five of these studies
included 15 or fewer participants per
condition. Of the studies including
younger adults, only one of the 15
studies would have remained.

23. Does the meta analysis
adequately deal with clinical
heterogeneity and is there a
basis for giving a meaningful
interpretation to a single
summary effect size?

24. Clinical Heterogeneity
Combining studies may be
inappropriate for a variety of the
following reasons: differences in patient
eligibility criteria in the included trials,
different interventions and outcomes,
and other methodological differences
or missing information.
Moher et al., 1998

25. Clinical Connoisseurship
Consumers of meta-analyses should be intimately familiar
with clinical phenomena and the nature of interventions,
and they need to be able to go beyond the numerical data in
determining whether it is appropriate to integrate studies
that differ in patient populations and likely restrictions on
who will be willing and able to participate in an
intervention, the nature of the intervention, and the
appropriateness of outcomes for determining the efficacy of
an intervention. They need to be prepared to make critical
assessments of whether the attempted integration of data
exceeds even the metaphor of mixing apples and oranges.

26. To what extent does the meta analysis
distinguish interventions that explicitly
targeted clinically significant levels of
problems (distress, fatigue) versus
interventions that did not have such an
aim or that did not require clinically
significant levels of the problem?

27. In Dixon (pain) and Jacobsen
(fatigue) patients did not have to
meet a threshold criterion of
pain/fatigue, primary objective of
the intervention did not have to
involve pain/fatigue reduction, and
pain/fatigue did not have to be a
primary outcome.

28. Meta Analyses Have Become Powerful and
Authoritative Tools
The hope is that meta analysis provides definitive
conclusions concerning whether interventions work
even when drawing on contradictory and sometimes
flawed individual studies.
Risk is that bad meta analyses discourage revisiting
the original studies that were integrated in it.
Risk is that inaccurate meta analyses will be used for
decisions to pay for delivery of services to patients,
override patient preference, and define future
research priorities.