When cherished beliefs clash with evidence


Published on

Presentation at 10th Anniversary of the Department of Cancer Epidemiology, Danish Cancer Institute 2011

Published in: Health & Medicine, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

When cherished beliefs clash with evidence

  1. 1. When Cherished Beliefs ClashWhen Cherished Beliefs Clash with Evidence, Don’t Expect towith Evidence, Don’t Expect to Be Thanked for What ScienceBe Thanked for What Science Does BestDoes Best James C. Coyne, Ph.D. University of Pennsylvania School of Medicine and University Medical Center, Groningen jcoyne@mail.med.upenn.edu
  2. 2. Unpicking dodgy claims, unpacking the evidence behind dodgy claims, isn’t a kind of nasty carping activity; it socially useful, but it’s also extremely valuable explanatory tool. Because real science is all about critically appraising the evidence for someone else’s position. --Ben Goldacre
  3. 3. MuckrakersMuckrakers  Investigative journalists who  reported on corruption, harmful  social conditions, pollution,  food and product safety  standards, sexual harassment,  unfair labor practices, and  fraud.    Came into prominence and  named as such in Progressive  Era of American Politics (1890- 1920).
  4. 4. The top journals are filled with noble lies and fairy tales, promoting myths and claiming great success in addressing important patient needs and improving health outcomes, often where there is none. Articles appearing in top journals are not reliable guides to the best evidence.
  5. 5. James C. Coyne 1,2 , Brett Thombs3 ,Mariët Hagedoorn2 1 University of Pennsylvania, School of Medicine, USA 2 University Medical Center Groningen, the Netherlands 3 McGill University and Jewish General Hospital, Montreal, Canada   Ain’t Necessarily So: Review andAin’t Necessarily So: Review and Critique of Recent Meta-AnalysesCritique of Recent Meta-Analyses of Behavioral Medicineof Behavioral Medicine Interventions inInterventions in Health PsychologyHealth Psychology
  6. 6. Storyline in Cancer CareStoryline in Cancer Care  Much of what we know about psychosocial aspects  of cancer is mythical, even if the myths are  widespread, contrary to fact, taken for granted as  background assumptions, and resistant to evidence.
  7. 7. Myth or Storyline inMyth or Storyline in Cancer CareCancer Care • Stress, personality, and emotion play substantial  roles in incidence, progression, and outcome of  cancer  and psychosocial interventions can  improve survival by strengthening the immune  system. • Storyline is highly consistent with entrenched  cultural beliefs and fables about the triumph of  human will over adversity and mind over body and  its frailities.
  8. 8. Operation of Storylines in Scientific Discourse  Storlines often have the quality of a promissory note, giving  favorable data more credence than is yet justified.   Storylines tend to exclude, minimize, or incorporate evidence  with a confirmatory distortion.   Yet storylines are the basis by which evidence enters into  clinical and public policy decisions and media depictions of  issues.
  9. 9.  Myths carry sentimentality, important emotional  baggage that must be dealt with in any effort to  counter the myth.  Sentimentality of this cancer storyline countered by  articulation of concern that it hurts patients, not only  by squandering resources, but by blaming patients for  their morbidity and ultimate mortality.
  10. 10. We must not allow a shared commitment to improving the wellbeing of cancer patients to be exploited with exaggerated claims and poorly conceived, poorly conducted, and poorly reported clinical trials.
  11. 11. We Thought We WereWe Thought We Were Done with Claims ThatDone with Claims That Psychotherapy PromotesPsychotherapy Promotes the Survival of Cancerthe Survival of Cancer Patients….Patients….
  12. 12. No trial has ever found that psychotherapy improved the median survival time of women with metastatic breast cancer. No trial in which survival was chosen as the outcome of interest ahead of time has demonstrated a survival effect for patients with any type of cancer, when psychotherapy was not confounded with improved medical surveillance or treatment.   Coyne JC, Stefanek M, Palmer SC. Coyne JC, Stefanek M, Palmer SC.  Psychotherapy and survival in cancer: the Psychotherapy and survival in cancer: the  conflict between hope and evidence. conflict between hope and evidence.  Psychol Bull.Psychol Bull. 2007;133:367-394.2007;133:367-394.
  13. 13. Now classic 1989 Lancet Now classic 1989 Lancet  study of Spiegel and study of Spiegel and  colleagues: Was there colleagues: Was there  ever an effect there?ever an effect there?
  14. 14.  No one can replicate odd survival curve of control group in intervention or observational studies, suggesting something went wrong.  Curve of Spiegel’s intervention group approximated by both intervention and control groups in subsequent studies, suggesting it was inert.
  15. 15. “The results suggest that we can help breast cancer patients make positive steps that may help them live longer and make recurrence less likely. We already knew a psychological intervention program could help breast cancer patients to handle their stress, function more effectively, and improve their health. Now we know it does even more.” Andersen, B. L., H. C. Yang, et al. (2008). Andersen, B. L., H. C. Yang, et al. (2008).  "Psychologic Intervention Improves "Psychologic Intervention Improves  Survival for Breast Cancer Patients A Survival for Breast Cancer Patients A  Randomized Clinical Trial." Randomized Clinical Trial." CancerCancer   113(12): 3450-3458.113(12): 3450-3458.
  16. 16. •No survival effect found in simple analyses, claims depend on inappropriate multivariate analyses. •No differences between intervention and control  groups in recurrence or survival. •Psychosocial intervention consisting of a mixture of relaxation training, problem solving and health behavior promotion. •Null and weak results across 8 measures of mood (No  effects on mood), 15 measures of immune function, and 4 measures of adherence. A Closer look at Andersen, et al.A Closer look at Andersen, et al. (2008). (2008). CancerCancer 113(12): 3450-3458. 113(12): 3450-3458.
  17. 17. Psychosocial Psychosocial  Intervention, Immune Intervention, Immune  Function, and Function, and  Progression of Cancer: Progression of Cancer:  Unproven MedicineUnproven Medicine
  18. 18. Cameron LD, et al. Cognitive and Cameron LD, et al. Cognitive and  affective determinants of decisions to affective determinants of decisions to  attend a group psychosocial support attend a group psychosocial support  program for women with breast cancer. program for women with breast cancer.  Psychosomatic MedicinePsychosomatic Medicine 2005;67:584-589. 2005;67:584-589. Many breast cancer patients are attending group therapy with the belief that they are fighting their cancer by strengthening their immune system.
  19. 19. Weak or No Effects, But Positive Results Emphasized, Strong Confirmatory Bias in Reporting New Results and Recounting of Past Studies. Doubtful Clinical Significance Even If Results Were Obtained. Confused, Simplistic View of Role of Immune System in Cancer Progression. Claims That Psychosocial Claims That Psychosocial  Intervention Strengthens the Intervention Strengthens the  Immune SystemImmune System
  20. 20. Each of the measures used in this literature represents only a small facet of a complex, highly redundant system. It would therefore be inappropriate to conclude that intervention-related changes in any specific immune parameter signal a state of "immune enhancement" or altered susceptibility to immune- mediated disease. The normal functioning range for most immune measures is very broad, and psychological interventions typically do not induce changes of sufficient magnitude to move people outside of these boundaries (p. 48). Miller GE, Cohen S. Psychological Miller GE, Cohen S. Psychological  interventions and the immune system: A interventions and the immune system: A  meta-analytic review and critique. meta-analytic review and critique. HealthHealth Psychology.Psychology. Jan 2001;20(1):47-63.Jan 2001;20(1):47-63.
  21. 21. How Does Sticking  to a How Does Sticking  to a  Storyline Get Enforced?Storyline Get Enforced? No Conspiracy Theory No Conspiracy Theory  Needed.Needed.
  22. 22. “Confirmatory Bias Has Existed Even in  Historical Periods When There Were No  Incentives, in Terms of Fame or  Fortune.” -Robert K. Merton
  23. 23. Copyright ©2005 American Heart Association Loscalzo, J. Circulation 2005;112:3026-3029 Incentives and Pressures to Fit Incentives and Pressures to Fit  Research to a StorylineResearch to a Storyline
  24. 24. ‘‘By the time many young people figure out the system, they are so much a part of it, so obsessed with keeping their grants, that their imagination and instincts have been so muted (or corrupted) that their best work is already behind them. This is made much worse by the US system in which assistant professors in medical schools will soon have to raise their own salaries. Who would dare to pursue risky ideas under these circumstances? Who could dare change their research field, ever?” -Ted Cox, Director of the Program on Biophysics, Princeton University
  25. 25. Do Americans Do ItDo Americans Do It Bigger and Better?Bigger and Better?
  26. 26. Scandinavian Data SetsScandinavian Data Sets  Democratic socialist countries with integrated health system and uniform medical and death records  Despite concern with social inequalities and disparities, Scandinavia lacks the gross differences found in the US due to race/ethnic status, income, and insurance status.
  27. 27. Nakaya et al (2010)Nakaya et al (2010)  The [Finnish] cohort consists of all …same-sex twin pairs born before 1958, of which both members were alive in 1975.  The Swedish twin cohort was identified from the population- based Swedish Twin Registry, the largest of its kind in the world, which has information on more than 140,000 twins.  Information on cancer diagnoses was obtained by record linkage to the national cancer registries …using the unique identification numbers assigned to everyone residing in those countries.  Data on emigration and death were obtained in Finland by record linkage to the Population Register Centre and in Sweden by record linkage to the National Population Register.
  28. 28. The Danish StudiesThe Danish Studies  Ross et al (2009): Between September 1996 and May 1999, we sought to include all patients aged 18 or over who were being treated for a primary colorectal cancer by abdominal surgery at the surgical departments of eight hospitals in the eastern part of Denmark.  Boesen, et al (2007): In this population-based, randomized trial conducted in a society with free hospital treatment, psychoeducation for patients treated for cutaneous malignant melanoma did not influence time to recurrence or survival.
  29. 29. Why Don’t the Danes Get theWhy Don’t the Danes Get the Attention They Deserve forAttention They Deserve for What is the Best-of-Its-KindWhat is the Best-of-Its-Kind Research?Research?
  30. 30. Danish Psycho-OncologyDanish Psycho-Oncology Research Lacks Hype,Research Lacks Hype, Confirmatory Bias of AmericanConfirmatory Bias of American ResearchResearch
  31. 31. Telling It Like It Ain’t: How toTelling It Like It Ain’t: How to Succeed in Psycho-OncologySucceed in Psycho-Oncology  Have Lots of Endpoints and Ignore Negative Results in Main Analyses of Primary Endpoints.  Favor Secondary Analyses, Subgroup Analyses, and Endpoints Developed Post Hoc Over Negative Findings for Primary Analyses.  Ignore Methodological Shortcomings That Would Make Trial or Meta Analyses Invalid.
  32. 32. Telling It Like It Ain’t: How toTelling It Like It Ain’t: How to Succeed in Psycho-OncologySucceed in Psycho-Oncology • Present Negative Findings as if Positive in Subsequent Publications and Exaggerate Findings That Are Positive  Assess Multiple Endpoints and Treat Any Significant Finding as if it were a Replication of Past Findings  Create a False Consensus and Seeming Unanimity in the Literature by Cherrypicking Findings That Can be Construed as Positive and Ignoring the Rest
  33. 33. A credible scientific journal should publish all studies with ‘‘null’’ results provided they acknowledge their limitations. Conversely, such a journal should be cautious about publishing ‘‘positive’’ results, most of which are false. Independent replication is important and should be done by different teams, preferably by competitors. ‘‘Null’’ results should be published promptly in print in short versions, with more extensive details in web-based files. Ioannidis, J. P. A. (2006). JournalsIoannidis, J. P. A. (2006). Journals should publish all "null" results andshould publish all "null" results and should sparingly publish "positive"should sparingly publish "positive" results.results. Cancer EpidemiologyCancer Epidemiology Biomarkers & PreventionBiomarkers & Prevention 15(1): 185-15(1): 185- 186.186.
  34. 34. Ioannidis, J. P. A. (2006). JournalsIoannidis, J. P. A. (2006). Journals should publish all "null" results andshould publish all "null" results and should sparingly publish "positive"should sparingly publish "positive" results.results. “Positive” results should be published equally promptly, but only on the web, pending independent replication; once refuted, the original article and the refutation could be printed as a single nice null report; the rare validated findings should appear in print with full details.
  35. 35. The GoalThe Goal  To Create a Climate in Which Honest Reporting Of Well Conducted, But Negative Trials is More Valued than Hyping and Hiding of Results.  To Value the Weight of Evidence, Whatever That May Be, Over Noble Lies and Fairy Tales That Make Health Psychology Look Good.
  36. 36. Taking Heat…Taking Heat…
  37. 37. Resistance to criticism of what isResistance to criticism of what is published.published. “[The authors’] frustration with the work of others is not enough to appoint themselves as the Supreme Judges of the work of others - however flawed this work might be.” [Anonymous reviewer, Tell It Like It Ain’t..article].
  38. 38. It is with a heavy heart that I have concluded that Dr. Coyne suffers from an incurable illness: narcissistic myopia. He is a depression researcher, so successful treatment of mere anxiety symptoms appears meaningless to his limited vision. It … seems to permit him to dismiss the results of 25 years of my and many other colleagues' research demonstrating positive effects of group support for cancer patients. I can live with his distortion of the published data…but when he insults my patients by informing us that our supportive/expressive groups are a "waste of seriously ill patients' potentially short remaining lives," my tolerance for his obvious impairment vanishes. One of my patients who attended our supportive/expressive group for six years said: "This group is the least superficial thing I do in my life." She and her family thanked me for it when I visited her home shortly before she died. -David Spiegel, M.D., Stanford University (Listserv post)
  39. 39. Response to David Spiegel:Response to David Spiegel: Well, David, seeing that no one, not even you, has been able to replicate the findings for survival reported in your 1989 supportive-expressive therapy article in Lancet, might you now consider the possibility that the 1989 findings were spurious? What further data would it take for you to consider that possibility?
  40. 40. Barbara Andersen on CoyneBarbara Andersen on Coyne “Dr. Coyne’s writing strategy is, in the first portion of a sentence, to mischaracterize an aspect of a study and then, in the second part, assert why the aspect is methodologically flawed. “Dr. Coyne uses wording such as “the authors’ claim” or sarcasm that are [sic] not appropriate to scientific discourse.”
  41. 41. Response to Barbara Andersen:Response to Barbara Andersen: Well, Barbara, even if you wouldn’t respond to our peer-reviewed article suggesting that you did not get an effect on survival, might you consider publishing standard, unadjusted outcomes, such as a Kaplan- Meier estimate of the survival function so that readers can decide for themselves? After all, strong claims require strong data.
  42. 42. When anWhen an investigator, wellinvestigator, well armed with aarmed with a hypothesishypothesis consistent withconsistent with entrenched culturalentrenched cultural beliefs...beliefs...
  43. 43. encounters a gang of unruly data...encounters a gang of unruly data...
  44. 44. only the investigator walks away,only the investigator walks away, with his hypothesis unscathed.with his hypothesis unscathed.