Birds, Bears, Turtles, Trains,Coupons, Cocks…and Moses           Gary Schwitzer             Publisher  Association of Heal...
We review storiesthat include claims      about…          •   Medical treatments          •   Tests          •   Products ...
Our criteria: Does the story explain…•   What’s the total cost?•   How often do benefits occur?•   How often do harms occu...
After 1,800 stories                       over 7 years~70% of stories fail to:  ✔ Discuss costs  ✔Quantify potential benef...
News stories often paint akid-in-candy-store picture    of U.S. health care Terrific Risk-free Without a   price tag
Ignore      or   MinimizePotential Harms
The award forworst newscoverage:THE WINNER IS:SCREENINGTESTS(often involving thisgland)
Turtles, birds, bears• Turtles move too slowly to ever be  dangerous and don’t need treatment.• Birds are so fast you’ll n...
Lead-time bias    Beth gets on in Boston                                       Nancy boards in NY                        W...
While running for president Rudy Giuliani announced in a 2007 campaign ad:“I had prostate cancer, 5, 6 years ago. My chanc...
Susan G Komen for the Cure’s mammography advertisement during breast cancer awareness                                     ...
Common flaws: too much stenography – notindependent vetting of studies in journalsGlorifying big names/big journals - Publ...
Common flawsFailure to evaluate inherently weak scienceIdolatry of the surrogate – Not understanding or simply notreportin...
What CNN didn’t report…• Resignations, questions about conflicts of  interest and fraud probes of the program in  question...
Prostate cancer scan advance helps Mayo Clinic doctors with earlydetection                                                ...
Stories about tests should explain       sensitivity and specificity• Sensitivity = how many true positives• Specificity =...
Evidence – not just excitement – about     expensive new technologies
Using causal language to describe        observational studies• Observational studies cannot prove cause and  effect so it...
“Relative finger length could be used as a                               simple test for prostate cancer risk”            ...
Association ≠ Causation
Exaggerating effect size –        Absolute vs. Relative Risk• Two ways of saying the same thing• One way – relative risk r...
Nolvadex (tamoxifen)Reducing breast cancer risk by 50   .
For the first time, there is a clinically proven way formany women at high risk of developing breastcancer to significantl...
Women who took Nolvadex had 48% fewer breastcancers….                48% of what ???
Its like a coupon…   Extremely Fancy Store    What if selected items were..                            TVs, washing machin...
What is the coupon really worth?Chance of getting breast cancer over 6 yearsPlacebo:The REGULAR price!         PLACEBO    ...
What is the effect of Nolvadex?How good is the sale?          PLACEBO                    NOLVADEX            3.3%         ...
Describing the effect of NOLVADEX   So finally....this is how you get to            PLACEBO                        NOLVADE...
Two ways of saying the same thing:       the benefit of NOLVADEX   Extremely Fancy Store                Extremely Fancy St...
Relative vs.           "% off" risk                    absolute                     "savings"                             ...
The proof? In a landmark study…womenBenefit         who took Nolvadex had 48% fewer breast                cancers than wom...
Present benefits and harms the same way
The proof? In a landmark study…women                who took Nolvadex had 48% fewer breast                cancers than wom...
Over the next 6 years, what happened …                                PLACEBO     NOLVADEXBenefits: Nolvadex lowered chanc...
• Surrogate markers may not tell the whole story• Does The Language Fit The Evidence? – Association Versus  Causation• 7 W...
Progression free survival = The length of time during and after the treatment ofcancer, that a patient lives with the dise...
Read John Ioannidis to learn pitfalls of a steady diet of journal stories                           PLoS Med 2005; 2(8): e...
Journals complicit in miscommunication             • Editors of the HEART Group journals recently               stated tha...
Annals of Oncology: “Bias in reporting of endpoints of efficacy and toxicity in randomized,clinical trials for women with ...
What the authors point out:• “Investigators go overboard to make their  studies look positive.”• In 2/3 of studies, that m...
September 2012Positive “spin” was identified in about half of press releases andnews stories. The main factor associated w...
Public Misunderstanding of Cancer Risk• Studies have consistently shown people  overestimate their own risk of cancer• Oth...
“Unrealistic Optimism in Early-Phase OncologyTrials”People tended to overestimate the benefits ofthe trial they were enrol...
We hold a mirror up to journalists – Ask yourselves:• If you cover studies/clinical  news, how many of those stories  are ...
Drowning from a firehose of infoxication
@garyschwitzer@ivanoransky
Thank you for your attentionThanks to Drs. Steve Woloshin & LisaSchwartz of Dartmouth Medical School forslides on absolute...
Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses:  my talk to AHCJ 2013 Boston
Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses:  my talk to AHCJ 2013 Boston
Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses:  my talk to AHCJ 2013 Boston
Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses:  my talk to AHCJ 2013 Boston
Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses:  my talk to AHCJ 2013 Boston
Upcoming SlideShare
Loading in …5
×

Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses: my talk to AHCJ 2013 Boston

2,144 views

Published on

My workshop talk at Association of Health Care Journalists 2013 conference in Boston.

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,144
On SlideShare
0
From Embeds
0
Number of Embeds
786
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses: my talk to AHCJ 2013 Boston

  1. 1. Birds, Bears, Turtles, Trains,Coupons, Cocks…and Moses Gary Schwitzer Publisher Association of Health Care Journalists Boston – March 14, 2013
  2. 2. We review storiesthat include claims about… • Medical treatments • Tests • Products • Procedures
  3. 3. Our criteria: Does the story explain…• What’s the total cost?• How often do benefits occur?• How often do harms occur?• How strong is the evidence?• Is the condition exaggerated?• Is this really a new approach?• Is it available?• Are there alternative choices?• Who’s promoting this?• Do they have a financial conflict of interest?
  4. 4. After 1,800 stories over 7 years~70% of stories fail to: ✔ Discuss costs ✔Quantify potential benefits ✔ Quantify potential harms ✔ Evaluate the quality of the evidence
  5. 5. News stories often paint akid-in-candy-store picture of U.S. health care Terrific Risk-free Without a price tag
  6. 6. Ignore or MinimizePotential Harms
  7. 7. The award forworst newscoverage:THE WINNER IS:SCREENINGTESTS(often involving thisgland)
  8. 8. Turtles, birds, bears• Turtles move too slowly to ever be dangerous and don’t need treatment.• Birds are so fast you’ll never catch them. Too late to try to treat.• Bears are dangerous, but move slowly enough that you can catch them.• Screening tests can only make a difference for the bears.
  9. 9. Lead-time bias Beth gets on in Boston Nancy boards in NY Washington Train crashes in Washington. Both die. It may look like Beth was a longer-term survivor of the train ride, but only because she got on earlier than Nancy.
  10. 10. While running for president Rudy Giuliani announced in a 2007 campaign ad:“I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer—andthank God, I was cured of it—in the United States? Eighty-two percent. My chanceof surviving prostate cancer in England? Only 44 percent under socializedmedicine.”Yet despite this impressive difference in the five year survival rate, the mortalityrate was about the same in the US and the UK.How can that be? Remember the train? Lead-time bias and overdiagnosis. Far more USmen get PSA test than men in the UK, contributing to US’ higher survival rate.
  11. 11. Susan G Komen for the Cure’s mammography advertisement during breast cancer awareness month, 2011. Woloshin S , and Schwartz L M BMJ 2012;345:bmj.e5132©2012 by British Medical Journal Publishing Group
  12. 12. Common flaws: too much stenography – notindependent vetting of studies in journalsGlorifying big names/big journals - Publication in a medical journal does not guarantee the findings are true (or even important).Not ready for prime time – journals meant for conversation among scientistsNever intended to be sources of daily news. So if you’re going to use them that way, you simply must be aware of the landscape: • retractions, research fraud, fabrication, falsification of data • unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”) • ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)
  13. 13. Common flawsFailure to evaluate inherently weak scienceIdolatry of the surrogate – Not understanding or simply notreporting that surrogate outcomes (like tumor shrinkage) may nottranslate into clinically meaningful outcomes (longer life).Reckless extrapolation - Predicting what may happen in humans –and soon - based on very preliminary animal / lab science.Lack of awareness of conflicts of interest & other ethical issuesGoing soft on business stories or on local stories
  14. 14. What CNN didn’t report…• Resignations, questions about conflicts of interest and fraud probes of the program in question.• Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.
  15. 15. Prostate cancer scan advance helps Mayo Clinic doctors with earlydetection Minnesota Public Radio “The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”
  16. 16. Stories about tests should explain sensitivity and specificity• Sensitivity = how many true positives• Specificity = how many true negatives• What the Mayo story didn’t report was that studies have shown false positive tests of 15-47% - meaning biopsies and more risks.• And one Mayo report noted 11% false negatives• Shouldn’t that be reported alongside “The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”
  17. 17. Evidence – not just excitement – about expensive new technologies
  18. 18. Using causal language to describe observational studies• Observational studies cannot prove cause and effect so it is inaccurate to use terms like “benefits…protects…reduces risk”• These studies can only show a statistical association, so all you can say is that. We offer a primer, “Does The Language Fit The Evidence? – Association Versus Causation.”
  19. 19. “Relative finger length could be used as a simple test for prostate cancer risk” Based on what?The men were shown pictures of different finger length patterns and asked toidentify the one most similar to their own.Men whose index and ring fingers were the same length -- about 19 percentof those studied -- had a similar prostate cancer risk, but men whose indexfingers were longer than their ring finger were 33 percent less likely to haveprostate cancer.
  20. 20. Association ≠ Causation
  21. 21. Exaggerating effect size – Absolute vs. Relative Risk• Two ways of saying the same thing• One way – relative risk reduction – makes effect size seem larger• Other way – absolute risk reduction – makes effect size seem smaller.• We absolutely think you should use the absolute figures.
  22. 22. Nolvadex (tamoxifen)Reducing breast cancer risk by 50 .
  23. 23. For the first time, there is a clinically proven way formany women at high risk of developing breastcancer to significantly reduce that risk.The proof? In a landmark study…women who tookNolvadex had 48% fewer breast cancers thanwomen taking sugar pills.
  24. 24. Women who took Nolvadex had 48% fewer breastcancers…. 48% of what ???
  25. 25. Its like a coupon… Extremely Fancy Store What if selected items were.. TVs, washing machines?48 On selected items! % OFF save $100s Things like candy? save pennies “48% of what” matters! Know the REGULAR price!
  26. 26. What is the coupon really worth?Chance of getting breast cancer over 6 yearsPlacebo:The REGULAR price! PLACEBO Nolvadex: The SALES price! NOLVADEX 3.3% 1.7% How much do you save?Absolute risk Savings = Regular price – Sales price reduction 3.3% 1.7% = 1.6%Chance of getting breast cancer (over 6 years) with for 6 years, If 100 women took NOLVADEX instead of placebo NOLVADEX there would be about 2 lower than with breast cancer. was 1.6 % points fewer cases of placebo
  27. 27. What is the effect of Nolvadex?How good is the sale? PLACEBO NOLVADEX 3.3% 1.7% Chance of outcome (intervention) Relative Risk = Chance of outcome (control)
  28. 28. Describing the effect of NOLVADEX So finally....this is how you get to PLACEBO NOLVADEX the48% off sale! 3.3% 1.7% Its the relative risk reduction 1.7% ??? Relative Risk = = 0.52 ??? 3.3% "% Lower" format =1 - RR =1 - .52 =.48At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.
  29. 29. Two ways of saying the same thing: the benefit of NOLVADEX Extremely Fancy Store Extremely Fancy Store One feels big One feels small48 % OFF 1.6% How you say it matters!On selected items! SAVINGS On selected items! "Framing"
  30. 30. Relative vs. "% off" risk absolute "savings" reductionsChance of death at 1 year Risk reduction Relative AbsolutePlacebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)30% 10% 67% ?? 20% ?? 3% 1% 67% 2%0.003% 0.001% 67% 0.002%
  31. 31. The proof? In a landmark study…womenBenefit who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.Harm "Nolvadex isnt for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women". ”Strokes, cataracts more common with No numbers Nolvadex. Most women experience some level of hot flashes and vaginal discharge".
  32. 32. Present benefits and harms the same way
  33. 33. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.210% more uterine cancer and potentially 210% more life threatening blood clots in the lung and legs.
  34. 34. Over the next 6 years, what happened … PLACEBO NOLVADEXBenefits: Nolvadex lowered chance Getting breast cancer 3.3% 1.7%Harms: Nolvadex increased chance Having a serious blood clot 0.5% 1.0%Getting uterine cancer 0.5% 1.1% Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer
  35. 35. • Surrogate markers may not tell the whole story• Does The Language Fit The Evidence? – Association Versus Causation• 7 Words (and more) You Shouldn’t Use in Medical News• Problems with Reporting on News from Scientific Meetings• Absolute vs. Relative Risk• Number Needed to Treat (more on this in the afternoon)• Commercialism• Single Source Stories• Phases of Drug Trials• Medical Devices• Animal & Lab Studies
  36. 36. Progression free survival = The length of time during and after the treatment ofcancer, that a patient lives with the disease but it does not get worse.VersusOverall survival = do people live longer?
  37. 37. Read John Ioannidis to learn pitfalls of a steady diet of journal stories PLoS Med 2005; 2(8): e124
  38. 38. Journals complicit in miscommunication • Editors of the HEART Group journals recently stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.” – J AM COLL CARDIOL, Vol. 60, No. 23, 2012 • “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies” – Am J Obstet Gynecol, published online 11/29/12 • “Spin and Boasting in Research Articles.” - Commentary in Arch Pediatr Adolesc Med: [published online October 2012]
  39. 39. Annals of Oncology: “Bias in reporting of endpoints of efficacy and toxicity in randomized,clinical trials for women with breast cancer” (published online January 9, 2013)Reuters: “Doctors relying on studies published intop journals for guidance about how to treatwomen with breast cancer may not be gettingthe most accurate information.”
  40. 40. What the authors point out:• “Investigators go overboard to make their studies look positive.”• In 2/3 of studies, that meant not listing serious side effects in the abstract – which is all many may read• In 1/3 of studies, if the treatment didn’t work as hoped, researchers reported results the study was not designed to test – “secondary endpoints”
  41. 41. September 2012Positive “spin” was identified in about half of press releases andnews stories. The main factor associated with “spin” in pressreleases was the presence of “spin” in the journal article abstractconclusion.In other words, a direct link frompublished study news release news story.Where is the reader left behind in this food chain?
  42. 42. Public Misunderstanding of Cancer Risk• Studies have consistently shown people overestimate their own risk of cancer• Other studies show people overestimate risk factors that have not been proven and underestimate risk factors that are well- established
  43. 43. “Unrealistic Optimism in Early-Phase OncologyTrials”People tended to overestimate the benefits ofthe trial they were enrolled in andunderestimate its risks. -- IRB: Ethics & Human Research 2011
  44. 44. We hold a mirror up to journalists – Ask yourselves:• If you cover studies/clinical news, how many of those stories are just about new treatments, tests, products or procedures?• Do you think you might be reporting too much of this?• If so, have you spoken with editors about this? Do you want our help?• If you don’t think you report too much of this, what kind of grades would you get on our 10 criteria?• If you don’t accept or agree with our 10 criteria, what criteria do you use?
  45. 45. Drowning from a firehose of infoxication
  46. 46. @garyschwitzer@ivanoransky
  47. 47. Thank you for your attentionThanks to Drs. Steve Woloshin & LisaSchwartz of Dartmouth Medical School forslides on absolute vs. relative risk My email: Gary@HealthNewsReview.org

×