Successfully reported this slideshow.
Your SlideShare is downloading. ×

Are most positive findings in psychology false or exaggerated? An activist's perspective

Are most positive findings in 
psychology false or 
exaggerated? An activist's 
Colloquium at ANU 
Research S...

YouTube videos are no longer supported on SlideShare

View original on YouTube

John Ioannidis documented many positive 
findings in the biomedical literature do not 
replicate and many apparent “discov...

Check these out next

1 of 75 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (20)


Similar to Are most positive findings in psychology false or exaggerated? An activist's perspective (20)

More from James Coyne (20)


Recently uploaded (20)

Are most positive findings in psychology false or exaggerated? An activist's perspective

  1. 1. Are most positive findings in psychology false or exaggerated? An activist's perspective Colloquium at ANU Research School of Psychology Sept 10, 2014 James C. Coyne, Ph.D. University of Groningen University Medical Center Groningen (UMCG), Groningen, the Netherlands
  2. 2. John Ioannidis documented many positive findings in the biomedical literature do not replicate and many apparent “discoveries” turn out to be exaggerated or simply false. Many apparent discoveries are created by a combination of confirmatory bias, flexible rules of design, data analysis and reporting and significance chasing.
  3. 3. “It can be proven that most claimed research findings are false” Ioannidis, JPA. (2005). Why most published research findings are false. PLOS Medicine 2: 696-701. Ioannidis, JPA. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218-228. Young, NS., Ioannidis, JPA. et al. (2008). Why Current Publication Practices May Distort Science. PLOS Medicine 5: 1418-1422.
  4. 4. Ben Goldacre Drugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analysed using techniques which are flawed by design, in such a way that they exaggerate the benefits of treatments. Unsurprisingly, these trials tend to produce results that favour the manufacturer. When trials throw up results that companies don't like, they are perfectly entitled to hide them from doctors and patients.
  5. 5. Efforts at reform  Preregistration of clinical trials make it more difficult to hide negative trials or alter analytic plans after results are known.  Reporting standards ensure more transparent and detailed article so results can be independently validated.  Reporting standards and pre-registration of plans for meta-analyses and systematic reviews.  Making data available for independent reanalysis.
  6. 6. Our modest contribution to reform Roseman, M, Milette, K, Bero, LA, Coyne, JC, Lexchin, J., Turner, EH, & Thombs, BD. (2011). Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments. JAMA, 2011;305(10):1008-17. Roseman, M, Turner, EH, Lexchin, J., Coyne, JC, Bero, LA, & Thombs, BD. (2012). Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ, 2012; 345.
  7. 7. We documented with association of source of funding with claimed size of effects. We proposed considering conflict of interest as a moderator, source of heterogeneity, and another risk of bias in Cochrane risk of bias assessment. The Cochrane Collaboration agreed.
  8. 8. Cochrane Risk of Bias  Sequence generation.  Allocation concealment.  Blinding of participants, investigators, outcome assessors.  Incomplete outcome data.  Selective outcome reporting.  Other threats to validity.
  9. 9. Trickle down of reform  Changes forced upon Pharma slowly and inconsistently reach clinical psychology.  Larger data sets allow exploration of issues such as choice of control groups and investigator allegiance.  Behavioral Medicine adopts changes occurring in clinical psychology later and inconsistently.
  10. 10. Psychology's crisis of credibility
  11. 11. Instances of outright fraud are unusual, but confirmatory bias is rampant and supported by diffuse institutional forces. On the other hand, when fraud is consistent with prevailing biases, it can be tough to root out.
  12. 12. GGeettttiinngg HHaannss EEyysseenncckk’’ss ffrraauudduulleenntt ddaattaa oouutt ooff tthhee lliitteerraattuurree ● Co-authored papers with Grossarth-Maticek that claimed improbably high associations between personality and cancer and personality and smoking and published in own journal. ● Later it was revealed that he was paid substantially high amounts of money by attorneys for US tobacco companies to publish these data.
  13. 13. In a literature dominated by hazard ratios (HRs) clustered around 1.0, Eysenck/Grossarth- Maticek’s were extreme outliers, absurdly so. Their influence nonetheless lives on in inflated overall estimates of the effects of negative affect on cancer in tertiary sources that do not acknowledge their tainted origins.
  14. 14. Grossarth-Maticek/Eysenck Fake Data
  15. 15. Coyne JC, Ranchor AV, Palmer SC. Meta-analysis of stress-related factors in cancer. Nat Rev Clin Oncol. 2010;7(5). Ranchor AV, Sanderman R, Coyne JC. Invited Commentary: Personality as a Causal Factor in Cancer Risk and Mortality-Time to Retire a Hypothesis? American Journal of Epidemiology. 2010;172(4):386-8. Coyne JC, Johansen C. Confirmatory bias and the persistent influence of discredited data in interpreting the stress-cancer link: Commentary on Michael et al. (2009). Health Psychol. 2011;30(3):374-5
  16. 16. Confirmatory bias entrenched, preached to be a virtue.
  17. 17. AAddvviiccee ooff aa FFaammoouuss SSoocciiaall PPssyycchhoollooggiisstt “There are two possible articles you can write: (1) the article you planned to write when you designed your study or (2) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (2).” -Bem, 2003, pp. 171-172
  18. 18. Breakthrough “discoveries” are created and perpetuated by a combination of confirmatory bias, flexible rules of design, data analysis and reporting and significance chasing.
  19. 19. APS Journals  Have not adopted CONSORT.  Violate COPE in not having procedure for appeal of negative decisions.  Strong confirmatory bias, newsworthiness over science.  Aversion to null findings and replications.  Don’t adhere to Pottery Barn Rule.
  20. 20. From: < > Date: Wed, Jun 11, 2014 at 4:26 PM Subject: Fwd: Appeal re Manuscript PSCI-13-2172 To: Alan Kraut < I respect the autonomy of APS publications from the CEO. But I think that the new editor of Psychological Science is only going to compound his losses if he continues to fight giving us an appeal consistent with the standards of Committee on Publication Ethics (COPE). I think that COPE and the larger scientific community will be solidly on our side, and will generate considerable outrage. Hopefully, you would not consider it inappropriate to give him some sage advice on this. Sorry to be writing to you under these circumstances.
  21. 21. Skepticism about the replication movement. May only serve to ghettoize null findings and failures to replicate.  Protect bias in prestigious journals toward false-positive findings and discrimination against attempted replication and now findings.  Sleeping with the enemy.
  22. 22. The spin ccyyccllee iinn sscciieennttiiffiicc ppuubblliisshhiinngg:: iiss iitt nneecceessssaarryy ttoo hhaavvee ppoossiittiivvee rreessuullttss ttoo ggeett ppuubblliisshheedd??
  23. 23. SSppiinn  “Spin”-- Distortion to impress readers that something is noteworthy.  Objective: To identify nature and frequency of spin in 62 published reports of RCTs with statistically nonsignificant results.  Conclusion: Reporting and interpretation of findings frequently inconsistent with the results.
  24. 24. SSppiinn  Title was reported with spin in 13 articles (18.0%).  Spin was identified in the Results (37.5%) and Conclusions (58.3%) with conclusions of 17 (23.6%) focusing only on treatment effectiveness.  Spin was identified in the main-text Results (29.2%), Discussion (43.1%),, and Conclusions (50.0%).  More than 40% of reports had spin in at least 2 of these sections in the main text.
  25. 25. SSiiggnniiffiiccaannccee cchhaassiinngg bbiiaass The greatest threat [to the credibility of clinical research] may come from the poor relevance and scientific rationale and thus low pre-study odds of success of research efforts. Given that we currently have too many research findings, often with low credibility, replication and rigorous evaluation becomes as important or even more important than discovery. Credibility, replication, and translation are all desirable properties of research findings, but only modestly correlated.
  26. 26. “It is a common failing – and one I have myself suffered from – to fall in love with a hypothesis and to be unwilling to take no for an answer. A love affair with a pet hypothesis can waste years of precious time. There is very often no finally decisive yes, though quite often there can be a decisive no.” -P. B. Medawar
  27. 27. IInncceennttiivveess aanndd pprreessssuurreess ffoorr aa ccoonnffiirrmmaattoorryy bbiiaass Loscalzo, J. Circulation 2005;112:3026-3029 Copyright ©2005 American Heart Association
  28. 28. AAddddiittiioonnaall ssoouurrcceess ooff ccoonnffiirrmmaattoorryy bbiiaass  Professional organizations need positive findings to support guidelines and recommendations requiring services of their members.  Journals need positive findings to raise impact factor.  Gurus and advocates of particular treatments need positive findings to promote their interventions and workshops ($$$) and feed their vanity.  Authors need positive findings to get published and advance their careers.
  29. 29. Pressures from journals Competition for subscriptions, advertising, and prestige depend on the JIF. JIF depends on number of citations for articles within 2 years of publication. Newsworthy, citable articles with clock starting at point of “early release.” Articles with media buzz are more likely to be immediately cited. Null findings, replications less newsworthy.
  30. 30. JIF and Retractions Correlated
  31. 31. HHoott aarreeaass ppllaagguueedd bbyy ccoonnffiirrmmaattoorryy bbiiaass  Mind-body relations--psychoneuroimmunology (PNI)  Gene-environment interactions--particularly serotonin transport genes—GWAS, and genomics  Neuroscience  Screening for depression and distress  Claims that psychotherapies are evidence-based
  32. 32. In approaching a hot area of research, you have to keep stopping and asking, “Are a particular set of results interesting because they advance the field?” OR “Are they interesting results only because of some vague relevance claimed to interesting ideas that are not actually being tested?”
  33. 33. Need for activism Cannot depend on journals to clean up editorial policies and review processes. Journals inadequately support reform or self-correct. Institutionalization of confirmatory bias and routine acceptance of bad science.
  34. 34. Promoting reforms  Strengthen post publication peer review.  Set standards for conducting and reporting clinical trials and meta analyses/systematic reviews  Greater transparency and sharing of data.  Preregistration of observational studies and hypotheses, as well as clinical trials.  Outing of conflicts of interest.  Use of social media to level the playing field.
  35. 35. Targeted takedowns of bad science highlight the problem and model solutions.  Social media campaigns: Critique by blogging, Twitter and Facebook.  Outing and confrontation with bad editorial practices.  Letters to the editor are ineffective, but PubMed Commons is promising.  Calls for retractions and erratum. .
  36. 36. Pet peeves Hype and hokum calculated to attract interest despite being premature or exaggerated. Ideas and practices which are potentially harmful to consumers. Undisclosed conflicts of interest, both financial and nonfinancial. Repeat offenders.
  37. 37. Pet peeves  Privileged access to supposedly peer-reviewed outlets.  Editors and reviewers' abuse of the review process to suppress alternative points of view and embarrassing data.  Gurus who undermine consumers' sense of self efficacy and increase their need for experts.  Gurus who sell products based on claims of being scientists.
  38. 38. ““OOffffiicciiaall”” sscciieennccee Paradigms or ideas insulated from full critical scrutiny and standards of evidence by endorsement by funding agencies, professional organizations, and flagship journals.
  39. 39. Beware of Professional Organization Involvement as a Risk of Bias
  40. 40. AHA Advisory on Screening for Depression Not Guideline-Congruent!  Ziegelstein RC, Brett D, Thombs BD, Coyne JC, de Jonge P. Routine Screening for Depression in Patients with Coronary Heart Disease: Never Mind. Journal of the American Academy of Cardiology. 2009;54(10):886-90.  Thombs, B. D., Jewett, L. R., Knafo, R., Coyne, J. C., & Ziegelstein, R. C. (2009). Learning from history: a commentary on the American Heart Association Science Advisory on depression screening. American heart journal, 158(4), 503.
  41. 41. What to watch for in meta analyses commissioned by professional organizations such as Society of Behavioral Medicine (SBM)  Precommitment to conclusion that interventions work and are ready for dissemination and reimbursement.  Ignoring of preponderance of methodologically flawed, unpowered sources of bias.  Broad inclusion of diverse interventions into one category.  Ignoring statistical heterogeneity.  Weak assessment of risk of bias (Cochrane criteria are seldom used).
  42. 42. MMoosstt ppoossiittiivvee ffiinnddiinnggss aarree ffaallssee oorr eexxaaggggeerraatteedd:: TToooollss ffoorr rreeffoorrmm  A general heightened awareness of the problem and a healthy skepticism to what is published in peer-reviewed journals.  Reviews and editors educated about the problem and the need for enforcement of higher standards, conformity to established guidelines and transparency.  Readers/consumers armed with critical tools to evaluate for themselves claims of the literature.  Education of science journalists.
  43. 43. MMoosstt ppoossiittiivvee ffiinnddiinnggss aarree ffaallssee oorr eexxaaggggeerraatteedd:: TToooollss ffoorr rreeffoorrmm  Strengthened post-publication review.  Reduced resistance to publishing null findings and failure to replicate.  High-profile examples of particularly bad articles being retracted.  Open access journals, blogs, and Twitter.
  44. 44. Going after… Horrible meta-analyses written by American antiabortion activist, and accepted for publication by devout Catholic editor, Peter Tyrer.
  45. 45. American Anti-Abortionist Gets Flawed Meta Analysis into British Journal of Psychiatry
  46. 46. Going after bad science that misleads cancer patients
  47. 47. SSppiieeggeell DD,, BBlloooomm JJRR,, KKrraaeemmeerr HHCC,, GGootttthheeiill EE ((11998899).. LLaanncceett 22::888888--889911.. • Claimed women with metastatic breast cancer who received supportive-expressive group psychotherapy survived almost twice as long as women in the control group. • Cited 1164 times
  48. 48. Takedown of Pseudoscience of Positive Psychology and Health Blogged at PLOS Mind the Brain Led to exchange of letters in PNAS Which led to detailed re-analyses exposing claims as nonsense
  49. 49. Unfair editorial practices  Articles published in peer-reviewed journals that are not peer-reviewed or only minimally.  Use of invited commentaries to selectively highlight or dampen evidence and arguments relevant to politicized topics.  Special issues with restricted calls for papers.  Authors’ control of whether critical letters to the editor of published.  Restrictions on letters to the editor,
  50. 50. Conflicts of interest  Psychology editors neither solicit nor routinely publish statements of conflict of interest.  Strong risk of bias associated with investigator allegiance, particularly when a particular treatment is being marketed.  Non-financial conflicts of interest are real and powerful, but not always easy to detect.
  51. 51. Exposures of conflicts of interests (COIs) associated with promoters of treatments conducting original studies and meta-analyses has led to numerous erratum rectifying nondisclosure and exposure of association of COI with confirmatory bias and bad science.
  52. 52. Erratum: one, two, many more
  53. 53. Moving From Criticism and Activism to Proposals for Constructive Change
  54. 54. Democratizing Post-Publication Peer Review
  55. 55. PubMed Commons is a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others.
  56. 56.  PubMed Commons is a forum for open and constructive criticism and discussion of scientific issues.  It will thrive with high quality interchange from the scientific community.
  57. 57. Join the Pubmed Commons Revolution No longer will a few people grant an irrevocable judgment of “must stand because it is peer reviewed.” You can take post publication peer review out of the hands of editors.
  58. 58. No criticism without offering solutions? It takes a lot of work to explicate bad science and the more general context that supports and rewards it. Critics should not be required to offer constructive solutions or positive steps, particularly when there is substantial opposition from the journals. But when a receptive audience has been created, identification of positive steps become more timely.
  59. 59. Thank you Follow me on Twitter @CoyneoftheRealm Blogging at Mind the Brain