This document summarizes a presentation given by James Coyne on issues with reliability and bias in positive psychology findings. Some key points: - John Ioannidis and others have shown that many positive findings in biomedical research do not replicate and are exaggerated or false due to biases. - Similar issues exist in psychology due to confirmatory bias, flexible data analysis and chasing statistical significance. - Reforms are needed like pre-registering studies, transparent reporting standards, and making data available for independent analysis. - However, challenges remain as journals prefer positive results and organizations have conflicts of interest that uphold certain findings. Overall, skepticism is needed regarding many claimed research findings.

1. Are most positive findings in
psychology false or
exaggerated? An activist's
perspective
Colloquium at ANU
Research School of Psychology
Sept 10, 2014
James C. Coyne, Ph.D.
University of Groningen University
Medical Center Groningen (UMCG),
Groningen, the Netherlands
jcoynester@gmail.com

2. John Ioannidis documented many positive
findings in the biomedical literature do not
replicate and many apparent “discoveries” turn
out to be exaggerated or simply false.
Many apparent discoveries are created by a
combination of confirmatory bias, flexible rules
of design, data analysis and reporting and
significance chasing.

3. “It can be proven that most claimed
research findings are false”
Ioannidis, JPA. (2005). Why most published
research findings are false. PLOS Medicine 2:
696-701.
Ioannidis, JPA. (2005). Contradicted and
initially stronger effects in highly cited clinical
research. JAMA 294: 218-228.
Young, NS., Ioannidis, JPA. et al. (2008). Why
Current Publication Practices May Distort
Science. PLOS Medicine 5: 1418-1422.

4. Ben Goldacre
Drugs are tested by the people who
manufacture them, in poorly designed
trials, on hopelessly small numbers of
weird, unrepresentative patients, and
analysed using techniques which are
flawed by design, in such a way that
they exaggerate the benefits of
treatments. Unsurprisingly, these
trials tend to produce results that
favour the manufacturer. When trials
throw up results that companies don't
like, they are perfectly entitled to hide
them from doctors and patients.

5. Efforts at reform
 Preregistration of clinical trials make it more
difficult to hide negative trials or alter analytic
plans after results are known.
 Reporting standards ensure more transparent
and detailed article so results can be
independently validated.
 Reporting standards and pre-registration of
plans for meta-analyses and systematic reviews.
 Making data available for independent
reanalysis.

6. Our modest contribution to
reform
Roseman, M, Milette, K, Bero, LA, Coyne, JC,
Lexchin, J., Turner, EH, & Thombs, BD. (2011).
Reporting of conflicts of interest in meta-analyses
of trials of pharmacological
treatments. JAMA, 2011;305(10):1008-17.
Roseman, M, Turner, EH, Lexchin, J., Coyne,
JC, Bero, LA, & Thombs, BD. (2012). Reporting
of conflicts of interest from drug trials in
Cochrane reviews: cross sectional study. BMJ,
2012; 345.

8. We documented with association of source of
funding with claimed size of effects.
We proposed considering conflict of interest as
a moderator, source of heterogeneity, and
another risk of bias in Cochrane risk of bias
assessment.
The Cochrane Collaboration agreed.

9. Cochrane Risk of Bias
 Sequence generation.
 Allocation concealment.
 Blinding of participants, investigators, outcome
assessors.
 Incomplete outcome data.
 Selective outcome reporting.
 Other threats to validity.

10. Trickle down of reform
 Changes forced upon Pharma slowly and
inconsistently reach clinical psychology.
 Larger data sets allow exploration of issues
such as choice of control groups and
investigator allegiance.
 Behavioral Medicine adopts changes
occurring in clinical psychology later and
inconsistently.

11. Psychology's crisis of credibility

14. Instances of outright fraud are unusual,
but confirmatory bias is rampant and
supported by diffuse institutional forces.
On the other hand, when fraud is
consistent with prevailing biases, it can
be tough to root out.

15. GGeettttiinngg HHaannss EEyysseenncckk’’ss ffrraauudduulleenntt
ddaattaa oouutt ooff tthhee lliitteerraattuurree
● Co-authored papers with
Grossarth-Maticek that claimed
improbably high associations
between personality and cancer
and personality and smoking and
published in own journal.
● Later it was revealed that he was
paid substantially high amounts of
money by attorneys for US
tobacco companies to publish
these data.

16. In a literature dominated by hazard ratios (HRs)
clustered around 1.0, Eysenck/Grossarth-
Maticek’s were extreme outliers, absurdly so.
Their influence nonetheless lives on in inflated
overall estimates of the effects of negative
affect on cancer in tertiary sources that do not
acknowledge their tainted origins.

17. Grossarth-Maticek/Eysenck
Fake Data

18. Coyne JC, Ranchor AV, Palmer SC. Meta-analysis of
stress-related factors in cancer. Nat Rev Clin Oncol.
2010;7(5).
Ranchor AV, Sanderman R, Coyne JC. Invited
Commentary: Personality as a Causal Factor in Cancer Risk
and Mortality-Time to Retire a Hypothesis? American
Journal of Epidemiology. 2010;172(4):386-8.
Coyne JC, Johansen C. Confirmatory bias and the
persistent influence of discredited data in interpreting the
stress-cancer link: Commentary on Michael et al. (2009).
Health Psychol. 2011;30(3):374-5

19. Confirmatory bias entrenched,
preached to be a virtue.

20. AAddvviiccee ooff aa FFaammoouuss SSoocciiaall
PPssyycchhoollooggiisstt
“There are two possible articles you can write: (1) the article
you planned to write when you designed your study or (2)
the article that makes the most sense now that you have
seen the results. They are rarely the same, and the correct
answer is (2).”
-Bem, 2003, pp. 171-172

21. Breakthrough “discoveries” are created
and perpetuated by a combination of
confirmatory bias, flexible rules of
design, data analysis and reporting and
significance chasing.

24. APS Journals
 Have not adopted CONSORT.
 Violate COPE in not having procedure for
appeal of negative decisions.
 Strong confirmatory bias, newsworthiness
over science.
 Aversion to null findings and replications.
 Don’t adhere to Pottery Barn Rule.

25. From: jcoyne@mail.med.upenn.edu <jcoynester@gmail.com
>
Date: Wed, Jun 11, 2014 at 4:26 PM
Subject: Fwd: Appeal re Manuscript PSCI-13-2172
To: Alan Kraut <akraut@psychologicalscience.org
I respect the autonomy of APS publications from the CEO. But I think
that the new editor of Psychological Science is only going to compound
his losses if he continues to fight giving us an appeal consistent with
the standards of Committee on Publication Ethics (COPE). I think that
COPE and the larger scientific community will be solidly on our side,
and will generate considerable outrage. Hopefully, you would not
consider it inappropriate to give him some sage advice on this.
Sorry to be writing to you under these circumstances.

26. Skepticism about the
replication movement.
May only serve to ghettoize null findings and
failures to replicate.
 Protect bias in prestigious journals toward
false-positive findings and discrimination
against attempted replication and now
findings.
 Sleeping with the enemy.

27. The spin ccyyccllee iinn sscciieennttiiffiicc ppuubblliisshhiinngg::
iiss iitt nneecceessssaarryy ttoo hhaavvee ppoossiittiivvee rreessuullttss
ttoo ggeett ppuubblliisshheedd??

28. SSppiinn
 “Spin”-- Distortion to impress readers that something is
noteworthy.
 Objective: To identify nature and frequency of spin in 62
published reports of RCTs with statistically nonsignificant
results.
 Conclusion: Reporting and interpretation of findings
frequently inconsistent with the results.

29. SSppiinn
 Title was reported with spin in 13 articles (18.0%).
 Spin was identified in the Results (37.5%) and Conclusions
(58.3%) with conclusions of 17 (23.6%) focusing only on
treatment effectiveness.
 Spin was identified in the main-text Results (29.2%),
Discussion (43.1%),, and Conclusions (50.0%).
 More than 40% of reports had spin in at least 2 of these
sections in the main text.

30. SSiiggnniiffiiccaannccee cchhaassiinngg bbiiaass
The greatest threat [to the credibility of clinical research]
may come from the poor relevance and scientific rationale
and thus low pre-study odds of success of research efforts.
Given that we currently have too many research findings,
often with low credibility, replication and rigorous evaluation
becomes as important or even more important than
discovery. Credibility, replication, and translation are all
desirable properties of research findings, but only modestly
correlated.

31. “It is a common failing – and one I have myself suffered
from – to fall in love with a hypothesis and to be unwilling to
take no for an answer. A love affair with a pet hypothesis
can waste years of precious time. There is very often no
finally decisive yes, though quite often there can be a
decisive no.”
-P. B. Medawar

32. IInncceennttiivveess aanndd pprreessssuurreess ffoorr aa
ccoonnffiirrmmaattoorryy bbiiaass
Loscalzo, J. Circulation 2005;112:3026-3029
Copyright ©2005 American Heart Association

33. AAddddiittiioonnaall ssoouurrcceess ooff ccoonnffiirrmmaattoorryy
bbiiaass
 Professional organizations need positive findings to support
guidelines and recommendations requiring services of their
members.
 Journals need positive findings to raise impact factor.
 Gurus and advocates of particular treatments need positive
findings to promote their interventions and workshops ($$$)
and feed their vanity.
 Authors need positive findings to get published and advance
their careers.

34. Pressures from journals
Competition for subscriptions, advertising, and
prestige depend on the JIF.
JIF depends on number of citations for articles
within 2 years of publication.
Newsworthy, citable articles with clock starting
at point of “early release.”
Articles with media buzz are more likely to be
immediately cited.
Null findings, replications less newsworthy.

35. JIF and Retractions Correlated

36. HHoott aarreeaass ppllaagguueedd bbyy ccoonnffiirrmmaattoorryy
bbiiaass
 Mind-body relations--psychoneuroimmunology (PNI)
 Gene-environment interactions--particularly serotonin
transport genes—GWAS, and genomics
 Neuroscience
 Screening for depression and distress
 Claims that psychotherapies are evidence-based

37. In approaching a hot area of research, you have to keep
stopping and asking,
“Are a particular set of results interesting because they
advance the field?”
OR
“Are they interesting results only because of some vague
relevance claimed to interesting ideas that are not actually
being tested?”

38. Need for activism
Cannot depend on journals to clean up editorial
policies and review processes.
Journals inadequately support reform or self-correct.
Institutionalization of confirmatory bias and
routine acceptance of bad science.

39. Promoting reforms
 Strengthen post publication peer review.
 Set standards for conducting and reporting
clinical trials and meta analyses/systematic
reviews
 Greater transparency and sharing of data.
 Preregistration of observational studies and
hypotheses, as well as clinical trials.
 Outing of conflicts of interest.
 Use of social media to level the playing field.

40. Targeted takedowns of bad science
highlight the problem and model
solutions.
 Social media campaigns: Critique by
blogging, Twitter and Facebook.
 Outing and confrontation with bad editorial
practices.
 Letters to the editor are ineffective, but
PubMed Commons is promising.
 Calls for retractions and erratum.
.

41. Pet peeves
Hype and hokum calculated to attract interest
despite being premature or exaggerated.
Ideas and practices which are potentially
harmful to consumers.
Undisclosed conflicts of interest, both
financial and nonfinancial.
Repeat offenders.

42. Pet peeves
 Privileged access to supposedly peer-reviewed
outlets.
 Editors and reviewers' abuse of the review
process to suppress alternative points of view
and embarrassing data.
 Gurus who undermine consumers' sense of
self efficacy and increase their need for
experts.
 Gurus who sell products based on claims of
being scientists.

43. ““OOffffiicciiaall”” sscciieennccee
Paradigms or ideas insulated from full critical scrutiny and
standards of evidence by endorsement by funding
agencies, professional organizations, and flagship
journals.

44. Beware of Professional
Organization Involvement as a Risk
of Bias

45. AHA Advisory on Screening for
Depression Not Guideline-Congruent!
 Ziegelstein RC, Brett D, Thombs BD, Coyne JC, de
Jonge P. Routine Screening for Depression in Patients
with Coronary Heart Disease: Never Mind. Journal of the
American Academy of Cardiology. 2009;54(10):886-90.
 Thombs, B. D., Jewett, L. R., Knafo, R., Coyne, J. C., &
Ziegelstein, R. C. (2009). Learning from history: a
commentary on the American Heart Association Science
Advisory on depression screening. American heart
journal, 158(4), 503.

46. What to watch for in meta analyses commissioned
by professional organizations such as Society of
Behavioral Medicine (SBM)
 Precommitment to conclusion that interventions work
and are ready for dissemination and reimbursement.
 Ignoring of preponderance of methodologically flawed,
unpowered sources of bias.
 Broad inclusion of diverse interventions into one
category.
 Ignoring statistical heterogeneity.
 Weak assessment of risk of bias (Cochrane criteria are
seldom used).

47. MMoosstt ppoossiittiivvee ffiinnddiinnggss aarree ffaallssee oorr
eexxaaggggeerraatteedd:: TToooollss ffoorr rreeffoorrmm
 A general heightened awareness of the problem and a
healthy skepticism to what is published in peer-reviewed
journals.
 Reviews and editors educated about the problem and the
need for enforcement of higher standards, conformity to
established guidelines and transparency.
 Readers/consumers armed with critical tools to evaluate
for themselves claims of the literature.
 Education of science journalists.

48. MMoosstt ppoossiittiivvee ffiinnddiinnggss aarree ffaallssee oorr
eexxaaggggeerraatteedd:: TToooollss ffoorr rreeffoorrmm
 Strengthened post-publication review.
 Reduced resistance to publishing null findings and failure
to replicate.
 High-profile examples of particularly bad articles being
retracted.
 Open access journals, blogs, and Twitter.

49. Going after…
Horrible meta-analyses written by
American antiabortion activist, and
accepted for publication by devout
Catholic editor, Peter Tyrer.

50. American Anti-Abortionist Gets
Flawed Meta Analysis into British
Journal of Psychiatry

51. Going after bad science that
misleads cancer patients

52. SSppiieeggeell DD,, BBlloooomm JJRR,, KKrraaeemmeerr HHCC,,
GGootttthheeiill EE ((11998899).. LLaanncceett 22::888888--889911..
• Claimed women with metastatic breast cancer who
received supportive-expressive group psychotherapy
survived almost twice as long as women in the
control group.
• Cited 1164 times

55. Takedown of Pseudoscience of
Positive Psychology and Health
Blogged at PLOS Mind the Brain
Led to exchange of letters in PNAS
Which led to detailed re-analyses exposing
claims as nonsense

57. Unfair editorial practices
 Articles published in peer-reviewed journals
that are not peer-reviewed or only minimally.
 Use of invited commentaries to selectively
highlight or dampen evidence and arguments
relevant to politicized topics.
 Special issues with restricted calls for papers.
 Authors’ control of whether critical letters to
the editor of published.
 Restrictions on letters to the editor,

60. Conflicts of interest
 Psychology editors neither solicit nor
routinely publish statements of conflict of
interest.
 Strong risk of bias associated with
investigator allegiance, particularly when a
particular treatment is being marketed.
 Non-financial conflicts of interest are real and
powerful, but not always easy to detect.

61. Exposures of conflicts of interests (COIs)
associated with promoters of treatments
conducting original studies and meta-analyses
has led to numerous erratum rectifying
nondisclosure and exposure of association of
COI with confirmatory bias and bad science.

63. Erratum: one, two, many more

66. Moving From Criticism and Activism to
Proposals for Constructive Change

67. Democratizing Post-Publication Peer
Review

68. PubMed Commons is a system that enables
researchers to share their opinions about
scientific publications. Researchers can
comment on any publication indexed by
PubMed, and read the comments of others.

69.  PubMed Commons is a forum for open
and constructive criticism and discussion
of scientific issues.
 It will thrive with high quality interchange
from the scientific community.

70. Join the Pubmed Commons
Revolution
No longer will a few people
grant an irrevocable judgment
of “must stand because it is
peer reviewed.”
You can take post publication
peer review out of the hands
of editors.

71. No criticism without offering
solutions?
It takes a lot of work to explicate bad science
and the more general context that supports and
rewards it.
Critics should not be required to offer
constructive solutions or positive steps,
particularly when there is substantial opposition
from the journals.
But when a receptive audience has been
created, identification of positive steps become
more timely.

74. Thank you
Follow me on Twitter
@CoyneoftheRealm
Blogging at Mind the Brain