Successfully reported this slideshow.

Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses: my talk to AHCJ 2013 Boston

1

Share

Loading in …3
×
1 of 52
1 of 52

Birds, Bears, Turtles, Trains, Coupons, Cocks...and Moses: my talk to AHCJ 2013 Boston

1

Share

Download to read offline

Description

My workshop talk at Association of Health Care Journalists 2013 conference in Boston.

Transcript

  1. 1. Birds, Bears, Turtles, Trains, Coupons, Cocks…and Moses Gary Schwitzer Publisher Association of Health Care Journalists Boston – March 14, 2013
  2. 2. We review stories that include claims about… • Medical treatments • Tests • Products • Procedures
  3. 3. Our criteria: Does the story explain… • What’s the total cost? • How often do benefits occur? • How often do harms occur? • How strong is the evidence? • Is the condition exaggerated? • Is this really a new approach? • Is it available? • Are there alternative choices? • Who’s promoting this? • Do they have a financial conflict of interest?
  4. 4. After 1,800 stories over 7 years ~70% of stories fail to: ✔ Discuss costs ✔Quantify potential benefits ✔ Quantify potential harms ✔ Evaluate the quality of the evidence
  5. 5. News stories often paint a kid-in-candy-store picture of U.S. health care  Terrific  Risk-free  Without a price tag
  6. 6. Ignore or Minimize Potential Harms
  7. 7. The award for worst news coverage: THE WINNER IS: SCREENING TESTS (often involving this gland)
  8. 8. Turtles, birds, bears • Turtles move too slowly to ever be dangerous and don’t need treatment. • Birds are so fast you’ll never catch them. Too late to try to treat. • Bears are dangerous, but move slowly enough that you can catch them. • Screening tests can only make a difference for the bears.
  9. 9. Lead-time bias Beth gets on in Boston Nancy boards in NY Washington Train crashes in Washington. Both die. It may look like Beth was a longer-term survivor of the train ride, but only because she got on earlier than Nancy.
  10. 10. While running for president Rudy Giuliani announced in a 2007 campaign ad: “I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer—and thank God, I was cured of it—in the United States? Eighty-two percent. My chance of surviving prostate cancer in England? Only 44 percent under socialized medicine.” Yet despite this impressive difference in the five year survival rate, the mortality rate was about the same in the US and the UK. How can that be? Remember the train? Lead-time bias and overdiagnosis. Far more US men get PSA test than men in the UK, contributing to US’ higher survival rate.
  11. 11. Susan G Komen for the Cure’s mammography advertisement during breast cancer awareness month, 2011. Woloshin S , and Schwartz L M BMJ 2012;345:bmj.e5132 ©2012 by British Medical Journal Publishing Group
  12. 12. Common flaws: too much stenography – not independent vetting of studies in journals Glorifying big names/big journals - Publication in a medical journal does not guarantee the findings are true (or even important). Not ready for prime time – journals meant for conversation among scientists Never intended to be sources of daily news. So if you’re going to use them that way, you simply must be aware of the landscape: • retractions, research fraud, fabrication, falsification of data • unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”) • ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)
  13. 13. Common flaws Failure to evaluate inherently weak science Idolatry of the surrogate – Not understanding or simply not reporting that surrogate outcomes (like tumor shrinkage) may not translate into clinically meaningful outcomes (longer life). Reckless extrapolation - Predicting what may happen in humans – and soon - based on very preliminary animal / lab science. Lack of awareness of conflicts of interest & other ethical issues Going soft on business stories or on local stories
  14. 14. What CNN didn’t report… • Resignations, questions about conflicts of interest and fraud probes of the program in question. • Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.
  15. 15. Prostate cancer scan advance helps Mayo Clinic doctors with early detection Minnesota Public Radio “The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”
  16. 16. Stories about tests should explain sensitivity and specificity • Sensitivity = how many true positives • Specificity = how many true negatives • What the Mayo story didn’t report was that studies have shown false positive tests of 15-47% - meaning biopsies and more risks. • And one Mayo report noted 11% false negatives • Shouldn’t that be reported alongside “The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”
  17. 17. Evidence – not just excitement – about expensive new technologies
  18. 18. Using causal language to describe observational studies • Observational studies cannot prove cause and effect so it is inaccurate to use terms like “benefits…protects…reduces risk” • These studies can only show a statistical association, so all you can say is that. We offer a primer, “Does The Language Fit The Evidence? – Association Versus Causation.”
  19. 19. “Relative finger length could be used as a simple test for prostate cancer risk” Based on what? The men were shown pictures of different finger length patterns and asked to identify the one most similar to their own. Men whose index and ring fingers were the same length -- about 19 percent of those studied -- had a similar prostate cancer risk, but men whose index fingers were longer than their ring finger were 33 percent less likely to have prostate cancer.
  20. 20. Association ≠ Causation
  21. 21. Exaggerating effect size – Absolute vs. Relative Risk • Two ways of saying the same thing • One way – relative risk reduction – makes effect size seem larger • Other way – absolute risk reduction – makes effect size seem smaller. • We absolutely think you should use the absolute figures.
  22. 22. Nolvadex (tamoxifen) Reducing breast cancer risk by 50 .
  23. 23. For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
  24. 24. Women who took Nolvadex had 48% fewer breast cancers…. 48% of what ???
  25. 25. It's like a coupon… Extremely Fancy Store What if selected items were.. TVs, washing machines? 48 On selected items! % OFF save $100s Things like candy? save pennies “48% of what” matters! Know the REGULAR price!
  26. 26. What is the coupon really worth? Chance of getting breast cancer over 6 years Placebo:The REGULAR price! PLACEBO Nolvadex: The SALES price! NOLVADEX 3.3% 1.7% How much do you save? Absolute risk Savings = Regular price – Sales price reduction 3.3% 1.7% = 1.6% Chance of getting breast cancer (over 6 years) with for 6 years, If 100 women took NOLVADEX instead of placebo NOLVADEX there would be about 2 lower than with breast cancer. was 1.6 % points fewer cases of placebo
  27. 27. What is the effect of Nolvadex? How good is the sale? PLACEBO NOLVADEX 3.3% 1.7% Chance of outcome (intervention) Relative Risk = Chance of outcome (control)
  28. 28. Describing the effect of NOLVADEX So finally....this is how you get to PLACEBO NOLVADEX the48% off sale! 3.3% 1.7% It's the relative risk reduction 1.7% ??? Relative Risk = = 0.52 ??? 3.3% "% Lower" format =1 - RR =1 - .52 =.48 At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.
  29. 29. Two ways of saying the same thing: the benefit of NOLVADEX Extremely Fancy Store Extremely Fancy Store One feels big One feels small 48 % OFF 1.6% How you say it matters! On selected items! SAVINGS On selected items! "Framing"
  30. 30. Relative vs. "% off" risk absolute "savings" reductions Chance of death at 1 year Risk reduction Relative Absolute Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG) 30% 10% 67% ?? 20% ?? 3% 1% 67% 2% 0.003% 0.001% 67% 0.002%
  31. 31. The proof? In a landmark study…women Benefit who took Nolvadex had 48% fewer breast cancers than women taking sugar pills. Harm "Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women". ”Strokes, cataracts more common with No numbers Nolvadex. Most women experience some level of hot flashes and vaginal discharge".
  32. 32. Present benefits and harms the same way
  33. 33. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills. 210% more uterine cancer and potentially 210% more life threatening blood clots in the lung and legs.
  34. 34. Over the next 6 years, what happened … PLACEBO NOLVADEX Benefits: Nolvadex lowered chance Getting breast cancer 3.3% 1.7% Harms: Nolvadex increased chance Having a serious blood clot 0.5% 1.0% Getting uterine cancer 0.5% 1.1% Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer
  35. 35. • Surrogate markers may not tell the whole story • Does The Language Fit The Evidence? – Association Versus Causation • 7 Words (and more) You Shouldn’t Use in Medical News • Problems with Reporting on News from Scientific Meetings • Absolute vs. Relative Risk • Number Needed to Treat (more on this in the afternoon) • Commercialism • Single Source Stories • Phases of Drug Trials • Medical Devices • Animal & Lab Studies
  36. 36. Progression free survival = The length of time during and after the treatment of cancer, that a patient lives with the disease but it does not get worse. Versus Overall survival = do people live longer?
  37. 37. Read John Ioannidis to learn pitfalls of a steady diet of journal stories PLoS Med 2005; 2(8): e124
  38. 38. Journals complicit in miscommunication • Editors of the HEART Group journals recently stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.” – J AM COLL CARDIOL, Vol. 60, No. 23, 2012 • “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies” – Am J Obstet Gynecol, published online 11/29/12 • “Spin and Boasting in Research Articles.” - Commentary in Arch Pediatr Adolesc Med: [published online October 2012]
  39. 39. Annals of Oncology: “Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer” (published online January 9, 2013) Reuters: “Doctors relying on studies published in top journals for guidance about how to treat women with breast cancer may not be getting the most accurate information.”
  40. 40. What the authors point out: • “Investigators go overboard to make their studies look positive.” • In 2/3 of studies, that meant not listing serious side effects in the abstract – which is all many may read • In 1/3 of studies, if the treatment didn’t work as hoped, researchers reported results the study was not designed to test – “secondary endpoints”
  41. 41. September 2012 Positive “spin” was identified in about half of press releases and news stories. The main factor associated with “spin” in press releases was the presence of “spin” in the journal article abstract conclusion. In other words, a direct link from published study news release news story. Where is the reader left behind in this food chain?
  42. 42. Public Misunderstanding of Cancer Risk • Studies have consistently shown people overestimate their own risk of cancer • Other studies show people overestimate risk factors that have not been proven and underestimate risk factors that are well- established
  43. 43. “Unrealistic Optimism in Early-Phase Oncology Trials” People tended to overestimate the benefits of the trial they were enrolled in and underestimate its risks. -- IRB: Ethics & Human Research 2011
  44. 44. We hold a mirror up to journalists – Ask yourselves: • If you cover studies/clinical news, how many of those stories are just about new treatments, tests, products or procedures? • Do you think you might be reporting too much of this? • If so, have you spoken with editors about this? Do you want our help? • If you don’t think you report too much of this, what kind of grades would you get on our 10 criteria? • If you don’t accept or agree with our 10 criteria, what criteria do you use?
  45. 45. Drowning from a firehose of infoxication
  46. 46. @garyschwitzer @ivanoransky
  47. 47. Thank you for your attention Thanks to Drs. Steve Woloshin & Lisa Schwartz of Dartmouth Medical School for slides on absolute vs. relative risk My email: Gary@HealthNewsReview.org

Description

My workshop talk at Association of Health Care Journalists 2013 conference in Boston.

Transcript

  1. 1. Birds, Bears, Turtles, Trains, Coupons, Cocks…and Moses Gary Schwitzer Publisher Association of Health Care Journalists Boston – March 14, 2013
  2. 2. We review stories that include claims about… • Medical treatments • Tests • Products • Procedures
  3. 3. Our criteria: Does the story explain… • What’s the total cost? • How often do benefits occur? • How often do harms occur? • How strong is the evidence? • Is the condition exaggerated? • Is this really a new approach? • Is it available? • Are there alternative choices? • Who’s promoting this? • Do they have a financial conflict of interest?
  4. 4. After 1,800 stories over 7 years ~70% of stories fail to: ✔ Discuss costs ✔Quantify potential benefits ✔ Quantify potential harms ✔ Evaluate the quality of the evidence
  5. 5. News stories often paint a kid-in-candy-store picture of U.S. health care  Terrific  Risk-free  Without a price tag
  6. 6. Ignore or Minimize Potential Harms
  7. 7. The award for worst news coverage: THE WINNER IS: SCREENING TESTS (often involving this gland)
  8. 8. Turtles, birds, bears • Turtles move too slowly to ever be dangerous and don’t need treatment. • Birds are so fast you’ll never catch them. Too late to try to treat. • Bears are dangerous, but move slowly enough that you can catch them. • Screening tests can only make a difference for the bears.
  9. 9. Lead-time bias Beth gets on in Boston Nancy boards in NY Washington Train crashes in Washington. Both die. It may look like Beth was a longer-term survivor of the train ride, but only because she got on earlier than Nancy.
  10. 10. While running for president Rudy Giuliani announced in a 2007 campaign ad: “I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer—and thank God, I was cured of it—in the United States? Eighty-two percent. My chance of surviving prostate cancer in England? Only 44 percent under socialized medicine.” Yet despite this impressive difference in the five year survival rate, the mortality rate was about the same in the US and the UK. How can that be? Remember the train? Lead-time bias and overdiagnosis. Far more US men get PSA test than men in the UK, contributing to US’ higher survival rate.
  11. 11. Susan G Komen for the Cure’s mammography advertisement during breast cancer awareness month, 2011. Woloshin S , and Schwartz L M BMJ 2012;345:bmj.e5132 ©2012 by British Medical Journal Publishing Group
  12. 12. Common flaws: too much stenography – not independent vetting of studies in journals Glorifying big names/big journals - Publication in a medical journal does not guarantee the findings are true (or even important). Not ready for prime time – journals meant for conversation among scientists Never intended to be sources of daily news. So if you’re going to use them that way, you simply must be aware of the landscape: • retractions, research fraud, fabrication, falsification of data • unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”) • ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)
  13. 13. Common flaws Failure to evaluate inherently weak science Idolatry of the surrogate – Not understanding or simply not reporting that surrogate outcomes (like tumor shrinkage) may not translate into clinically meaningful outcomes (longer life). Reckless extrapolation - Predicting what may happen in humans – and soon - based on very preliminary animal / lab science. Lack of awareness of conflicts of interest & other ethical issues Going soft on business stories or on local stories
  14. 14. What CNN didn’t report… • Resignations, questions about conflicts of interest and fraud probes of the program in question. • Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.
  15. 15. Prostate cancer scan advance helps Mayo Clinic doctors with early detection Minnesota Public Radio “The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”
  16. 16. Stories about tests should explain sensitivity and specificity • Sensitivity = how many true positives • Specificity = how many true negatives • What the Mayo story didn’t report was that studies have shown false positive tests of 15-47% - meaning biopsies and more risks. • And one Mayo report noted 11% false negatives • Shouldn’t that be reported alongside “The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”
  17. 17. Evidence – not just excitement – about expensive new technologies
  18. 18. Using causal language to describe observational studies • Observational studies cannot prove cause and effect so it is inaccurate to use terms like “benefits…protects…reduces risk” • These studies can only show a statistical association, so all you can say is that. We offer a primer, “Does The Language Fit The Evidence? – Association Versus Causation.”
  19. 19. “Relative finger length could be used as a simple test for prostate cancer risk” Based on what? The men were shown pictures of different finger length patterns and asked to identify the one most similar to their own. Men whose index and ring fingers were the same length -- about 19 percent of those studied -- had a similar prostate cancer risk, but men whose index fingers were longer than their ring finger were 33 percent less likely to have prostate cancer.
  20. 20. Association ≠ Causation
  21. 21. Exaggerating effect size – Absolute vs. Relative Risk • Two ways of saying the same thing • One way – relative risk reduction – makes effect size seem larger • Other way – absolute risk reduction – makes effect size seem smaller. • We absolutely think you should use the absolute figures.
  22. 22. Nolvadex (tamoxifen) Reducing breast cancer risk by 50 .
  23. 23. For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
  24. 24. Women who took Nolvadex had 48% fewer breast cancers…. 48% of what ???
  25. 25. It's like a coupon… Extremely Fancy Store What if selected items were.. TVs, washing machines? 48 On selected items! % OFF save $100s Things like candy? save pennies “48% of what” matters! Know the REGULAR price!
  26. 26. What is the coupon really worth? Chance of getting breast cancer over 6 years Placebo:The REGULAR price! PLACEBO Nolvadex: The SALES price! NOLVADEX 3.3% 1.7% How much do you save? Absolute risk Savings = Regular price – Sales price reduction 3.3% 1.7% = 1.6% Chance of getting breast cancer (over 6 years) with for 6 years, If 100 women took NOLVADEX instead of placebo NOLVADEX there would be about 2 lower than with breast cancer. was 1.6 % points fewer cases of placebo
  27. 27. What is the effect of Nolvadex? How good is the sale? PLACEBO NOLVADEX 3.3% 1.7% Chance of outcome (intervention) Relative Risk = Chance of outcome (control)
  28. 28. Describing the effect of NOLVADEX So finally....this is how you get to PLACEBO NOLVADEX the48% off sale! 3.3% 1.7% It's the relative risk reduction 1.7% ??? Relative Risk = = 0.52 ??? 3.3% "% Lower" format =1 - RR =1 - .52 =.48 At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.
  29. 29. Two ways of saying the same thing: the benefit of NOLVADEX Extremely Fancy Store Extremely Fancy Store One feels big One feels small 48 % OFF 1.6% How you say it matters! On selected items! SAVINGS On selected items! "Framing"
  30. 30. Relative vs. "% off" risk absolute "savings" reductions Chance of death at 1 year Risk reduction Relative Absolute Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG) 30% 10% 67% ?? 20% ?? 3% 1% 67% 2% 0.003% 0.001% 67% 0.002%
  31. 31. The proof? In a landmark study…women Benefit who took Nolvadex had 48% fewer breast cancers than women taking sugar pills. Harm "Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women". ”Strokes, cataracts more common with No numbers Nolvadex. Most women experience some level of hot flashes and vaginal discharge".
  32. 32. Present benefits and harms the same way
  33. 33. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills. 210% more uterine cancer and potentially 210% more life threatening blood clots in the lung and legs.
  34. 34. Over the next 6 years, what happened … PLACEBO NOLVADEX Benefits: Nolvadex lowered chance Getting breast cancer 3.3% 1.7% Harms: Nolvadex increased chance Having a serious blood clot 0.5% 1.0% Getting uterine cancer 0.5% 1.1% Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer
  35. 35. • Surrogate markers may not tell the whole story • Does The Language Fit The Evidence? – Association Versus Causation • 7 Words (and more) You Shouldn’t Use in Medical News • Problems with Reporting on News from Scientific Meetings • Absolute vs. Relative Risk • Number Needed to Treat (more on this in the afternoon) • Commercialism • Single Source Stories • Phases of Drug Trials • Medical Devices • Animal & Lab Studies
  36. 36. Progression free survival = The length of time during and after the treatment of cancer, that a patient lives with the disease but it does not get worse. Versus Overall survival = do people live longer?
  37. 37. Read John Ioannidis to learn pitfalls of a steady diet of journal stories PLoS Med 2005; 2(8): e124
  38. 38. Journals complicit in miscommunication • Editors of the HEART Group journals recently stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.” – J AM COLL CARDIOL, Vol. 60, No. 23, 2012 • “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies” – Am J Obstet Gynecol, published online 11/29/12 • “Spin and Boasting in Research Articles.” - Commentary in Arch Pediatr Adolesc Med: [published online October 2012]
  39. 39. Annals of Oncology: “Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer” (published online January 9, 2013) Reuters: “Doctors relying on studies published in top journals for guidance about how to treat women with breast cancer may not be getting the most accurate information.”
  40. 40. What the authors point out: • “Investigators go overboard to make their studies look positive.” • In 2/3 of studies, that meant not listing serious side effects in the abstract – which is all many may read • In 1/3 of studies, if the treatment didn’t work as hoped, researchers reported results the study was not designed to test – “secondary endpoints”
  41. 41. September 2012 Positive “spin” was identified in about half of press releases and news stories. The main factor associated with “spin” in press releases was the presence of “spin” in the journal article abstract conclusion. In other words, a direct link from published study news release news story. Where is the reader left behind in this food chain?
  42. 42. Public Misunderstanding of Cancer Risk • Studies have consistently shown people overestimate their own risk of cancer • Other studies show people overestimate risk factors that have not been proven and underestimate risk factors that are well- established
  43. 43. “Unrealistic Optimism in Early-Phase Oncology Trials” People tended to overestimate the benefits of the trial they were enrolled in and underestimate its risks. -- IRB: Ethics & Human Research 2011
  44. 44. We hold a mirror up to journalists – Ask yourselves: • If you cover studies/clinical news, how many of those stories are just about new treatments, tests, products or procedures? • Do you think you might be reporting too much of this? • If so, have you spoken with editors about this? Do you want our help? • If you don’t think you report too much of this, what kind of grades would you get on our 10 criteria? • If you don’t accept or agree with our 10 criteria, what criteria do you use?
  45. 45. Drowning from a firehose of infoxication
  46. 46. @garyschwitzer @ivanoransky
  47. 47. Thank you for your attention Thanks to Drs. Steve Woloshin & Lisa Schwartz of Dartmouth Medical School for slides on absolute vs. relative risk My email: Gary@HealthNewsReview.org

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

×