Successfully reported this slideshow.

Covering Medical Studies: How Not to Get It Wrong

6

Share

Loading in …3
×
1 of 44
1 of 44

Covering Medical Studies: How Not to Get It Wrong

6

Share

Download to read offline

Description

AHCJ Boston 2013 presentation on reading medical studies for journalists

Transcript

  1. 1. Covering Medical Studies: How Not to Get It Wrong AHCJ Boston, March 2013 Ivan Oransky, MD Executive Editor, Reuters Health Co-Founder, Retraction Watch @ivanoransky
  2. 2. Put Down That Coffee!
  3. 3. Put Down That Coffee! STUDY LINKS COFFEE USE TO PANCREAS CANCER New York Times, March 12, 1981 “Although the statistical association does not prove that coffee causes cancer, Dr. Brian MacMahon of Harvard, leader of the research group, said he stopped drinking coffee a few months ago when the results of the study became clear. In a telephone interview, he said that he would not presume to advise others.”
  4. 4. Or Get A Refill
  5. 5. Or Get A Refill CRITICS SAY COFFEE STUDY WAS FLAWED New York Times, June 30, 1981 ''This otherwise excellent paper may be flawed in one critical way,'' said a letter from Dr. Steven Shedlofsky of the Veterans Administration Hospital in White River Junction, Vt. He questioned the comparison of pancreatic cancer patients with persons hospitalized for noncancerous diseases of the digestive system.
  6. 6. Or Get A Refill CRITICS SAY COFFEE STUDY WAS FLAWED New York Times, June 30, 1981 “Such patients, he noted, might be expected to give up coffee drinking because of their illness. This, he argued, would tilt the proportion of coffee drinkers away from the ''control'' group who were being compared with the cancer patients. Amplifying the letter in an interview, Dr. Shedlofsky said many patients with digestive diseases give up coffee because they believe it aggravates their discomfort, and others do so because their doctors have advised them to.
  7. 7. We Cured Cancer 15 Years Ago
  8. 8. We Cured Cancer 15 Years Ago The New York Times, May 3, 1998 “Within a year, if all goes well, the first cancer patient will be injected with two new drugs that can eradicate any type of cancer, with no obvious side effects and no drug resistance -- in mice.” … ''Judah is going to cure cancer in two years,'' said Dr. James D. Watson, a Nobel laureate who directs the Cold Spring Harbor Laboratory, a cancer research center on Long Island. Dr. Watson said Dr. Folkman would be remembered along with scientists like Charles Darwin as someone who permanently altered civilization.”
  9. 9. Or Maybe We Didn’t. Here’s Why. The New York Times, February 11, 2013
  10. 10. Or Maybe We Didn’t. Here’s Why. The New York Times, February 11, 2013 “The study’s findings do not mean that mice are useless models for all human diseases. But, its authors said, they do raise troubling questions about diseases like the ones in the study that involve the immune system, including cancer and heart disease.”
  11. 11. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials?
  12. 12. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials? 5 How many of those made it to FDA approval?
  13. 13. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials? 5 How many of those made it to FDA approval? 1 Source: http://www.phrma.org/issues/intellectual-property (PhRMA)
  14. 14. How to Get It Wrong • Write about compounds in pre-clinical trials as if they were about to be on pharmacy shelves • Write about every drug in phase I and phase II trials as if it would definitely be approved
  15. 15. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  16. 16. How Often Are Studies Wrong?
  17. 17. Retractions on the Rise -The Wall Street Journal
  18. 18. “But My Editor Wants More Copy” So you’ll be covering studies.
  19. 19. How Not to Get It Wrong Source: Jamie Smith http://inksnow.blogspot.com/2012/07/context.html
  20. 20. Always Read the Study
  21. 21. Always Read the Study Get the full study and read it – “I think it’s journalistic malpractice to not have the full study in front of you when you’re reporting,” Oransky says.
  22. 22. How to Get Studies • www.EurekAlert.org for embargoed material • AHCJ membership includes access to Cochrane Library, Health Affairs, JAMA, and many other journals www.healthjournalism.org • ScienceDirect (Elsevier) gives reporters free access to hundreds of journals www.sciencedirect.com • Open access journals (e.g., Public Library of Science www.plos.org) • Ask press officers, or the authors
  23. 23. Ask “Dumb” Questions If you lack experience dealing with scientific material, don’t be afraid to ask for definitions of jargon and scientific terms. This is no time to pretend you understand everything. Oransky says the science and medical industries are full of jargon that mask important details. “You’ll get off the phone and have a notebook full of gibberish and jargon,” he says. “You can’t be afraid of asking a dumb question.”
  24. 24. Ask Smart Questions • Was it: – Peer-reviewed? – Published? Where? Not all journals are created equal. “Dr. X said they published in Y rather than a clinical journal because the paper was too long for the word limits in the clinical journals. I'm not sure where a detail like that would go…but he was impressed with my question.”
  25. 25. Ask Smart Questions • Was it in humans? – It’s remarkable there are any mice left with cancer, depression, or restless leg syndrome
  26. 26. Ask Smart Questions • Size matters Look for the power calculation, and ask if you don’t see one
  27. 27. Ask Smart Questions • Was it well-designed? From Covering Medical Research, Schwitzer/AHCJ
  28. 28. Ask Smart Questions • “Were those your primary endpoints?” • “Looks as though that endpoint reached statistical significance. Is that difference clinically significant?”
  29. 29. Read the Discussion Good journals will insist that authors include limitations. Read accompanying editorials, too.
  30. 30. What’s Your Angle? • Are you trying to help readers, listeners, and viewers make better health care decisions? • Covering a study because it has a good business angle, or it’s about a local project, is perfectly OK, but it doesn’t mean readers deserve less evidence and skepticism
  31. 31. Who Could Benefit? • How many people have the disease? • Keep potential disease-mongering in mind
  32. 32. How Effective is the Treatment? • Clinically significant endpoints, or surrogates – does this matter? • Preventing complications? How many? • Always remember to quantify results, not just “patients improved”
  33. 33. What Are The Side Effects? • Every treatment has them • Where to look: – Go beyond press releases and abstracts – Look at tables, charts, and results sections
  34. 34. Who Dropped Out? • Why did they leave the trial? • Intention to treat analysis
  35. 35. How Much Does it Cost? • If it’s ready to be the subject of a story, someone has projected the likely cost and market. – At least ask.
  36. 36. Who Has an Interest? • Disclose conflicts • PharmedOut.org • Dollars For Docs series http://projects.propublica.org/docdollars/
  37. 37. Are There Alternatives? • Did the study compare the new treatment to existing alternatives, or to placebo? • What are the advantages and disadvantages (and costs) of those existing alternatives? • Consider alternative explanations. Remember coffee and pancreatic cancer?
  38. 38. Don’t Rely Only on Study Authors • Find outside sources. Here’s how:
  39. 39. Use Anecdotes Carefully • Is the story representative? • Does the source of the story have any conflicts?
  40. 40. Watch Your Language • Lifestyle/diet – are they randomized controlled trials, or just observational? • If observational, make the language fit the evidence: – YES: “tied,” “linked” – NO: “reduces,” “causes”
  41. 41. A Dirty Little Secret Keep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr
  42. 42. Keep Yourself Honest • Use HealthNewsReview.org
  43. 43. Acknowledgement/Contact • Nancy Lapid, Reuters Health ivan-oransky@erols.com Twitter: @ivanoransky

Description

AHCJ Boston 2013 presentation on reading medical studies for journalists

Transcript

  1. 1. Covering Medical Studies: How Not to Get It Wrong AHCJ Boston, March 2013 Ivan Oransky, MD Executive Editor, Reuters Health Co-Founder, Retraction Watch @ivanoransky
  2. 2. Put Down That Coffee!
  3. 3. Put Down That Coffee! STUDY LINKS COFFEE USE TO PANCREAS CANCER New York Times, March 12, 1981 “Although the statistical association does not prove that coffee causes cancer, Dr. Brian MacMahon of Harvard, leader of the research group, said he stopped drinking coffee a few months ago when the results of the study became clear. In a telephone interview, he said that he would not presume to advise others.”
  4. 4. Or Get A Refill
  5. 5. Or Get A Refill CRITICS SAY COFFEE STUDY WAS FLAWED New York Times, June 30, 1981 ''This otherwise excellent paper may be flawed in one critical way,'' said a letter from Dr. Steven Shedlofsky of the Veterans Administration Hospital in White River Junction, Vt. He questioned the comparison of pancreatic cancer patients with persons hospitalized for noncancerous diseases of the digestive system.
  6. 6. Or Get A Refill CRITICS SAY COFFEE STUDY WAS FLAWED New York Times, June 30, 1981 “Such patients, he noted, might be expected to give up coffee drinking because of their illness. This, he argued, would tilt the proportion of coffee drinkers away from the ''control'' group who were being compared with the cancer patients. Amplifying the letter in an interview, Dr. Shedlofsky said many patients with digestive diseases give up coffee because they believe it aggravates their discomfort, and others do so because their doctors have advised them to.
  7. 7. We Cured Cancer 15 Years Ago
  8. 8. We Cured Cancer 15 Years Ago The New York Times, May 3, 1998 “Within a year, if all goes well, the first cancer patient will be injected with two new drugs that can eradicate any type of cancer, with no obvious side effects and no drug resistance -- in mice.” … ''Judah is going to cure cancer in two years,'' said Dr. James D. Watson, a Nobel laureate who directs the Cold Spring Harbor Laboratory, a cancer research center on Long Island. Dr. Watson said Dr. Folkman would be remembered along with scientists like Charles Darwin as someone who permanently altered civilization.”
  9. 9. Or Maybe We Didn’t. Here’s Why. The New York Times, February 11, 2013
  10. 10. Or Maybe We Didn’t. Here’s Why. The New York Times, February 11, 2013 “The study’s findings do not mean that mice are useless models for all human diseases. But, its authors said, they do raise troubling questions about diseases like the ones in the study that involve the immune system, including cancer and heart disease.”
  11. 11. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials?
  12. 12. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials? 5 How many of those made it to FDA approval?
  13. 13. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials? 5 How many of those made it to FDA approval? 1 Source: http://www.phrma.org/issues/intellectual-property (PhRMA)
  14. 14. How to Get It Wrong • Write about compounds in pre-clinical trials as if they were about to be on pharmacy shelves • Write about every drug in phase I and phase II trials as if it would definitely be approved
  15. 15. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  16. 16. How Often Are Studies Wrong?
  17. 17. Retractions on the Rise -The Wall Street Journal
  18. 18. “But My Editor Wants More Copy” So you’ll be covering studies.
  19. 19. How Not to Get It Wrong Source: Jamie Smith http://inksnow.blogspot.com/2012/07/context.html
  20. 20. Always Read the Study
  21. 21. Always Read the Study Get the full study and read it – “I think it’s journalistic malpractice to not have the full study in front of you when you’re reporting,” Oransky says.
  22. 22. How to Get Studies • www.EurekAlert.org for embargoed material • AHCJ membership includes access to Cochrane Library, Health Affairs, JAMA, and many other journals www.healthjournalism.org • ScienceDirect (Elsevier) gives reporters free access to hundreds of journals www.sciencedirect.com • Open access journals (e.g., Public Library of Science www.plos.org) • Ask press officers, or the authors
  23. 23. Ask “Dumb” Questions If you lack experience dealing with scientific material, don’t be afraid to ask for definitions of jargon and scientific terms. This is no time to pretend you understand everything. Oransky says the science and medical industries are full of jargon that mask important details. “You’ll get off the phone and have a notebook full of gibberish and jargon,” he says. “You can’t be afraid of asking a dumb question.”
  24. 24. Ask Smart Questions • Was it: – Peer-reviewed? – Published? Where? Not all journals are created equal. “Dr. X said they published in Y rather than a clinical journal because the paper was too long for the word limits in the clinical journals. I'm not sure where a detail like that would go…but he was impressed with my question.”
  25. 25. Ask Smart Questions • Was it in humans? – It’s remarkable there are any mice left with cancer, depression, or restless leg syndrome
  26. 26. Ask Smart Questions • Size matters Look for the power calculation, and ask if you don’t see one
  27. 27. Ask Smart Questions • Was it well-designed? From Covering Medical Research, Schwitzer/AHCJ
  28. 28. Ask Smart Questions • “Were those your primary endpoints?” • “Looks as though that endpoint reached statistical significance. Is that difference clinically significant?”
  29. 29. Read the Discussion Good journals will insist that authors include limitations. Read accompanying editorials, too.
  30. 30. What’s Your Angle? • Are you trying to help readers, listeners, and viewers make better health care decisions? • Covering a study because it has a good business angle, or it’s about a local project, is perfectly OK, but it doesn’t mean readers deserve less evidence and skepticism
  31. 31. Who Could Benefit? • How many people have the disease? • Keep potential disease-mongering in mind
  32. 32. How Effective is the Treatment? • Clinically significant endpoints, or surrogates – does this matter? • Preventing complications? How many? • Always remember to quantify results, not just “patients improved”
  33. 33. What Are The Side Effects? • Every treatment has them • Where to look: – Go beyond press releases and abstracts – Look at tables, charts, and results sections
  34. 34. Who Dropped Out? • Why did they leave the trial? • Intention to treat analysis
  35. 35. How Much Does it Cost? • If it’s ready to be the subject of a story, someone has projected the likely cost and market. – At least ask.
  36. 36. Who Has an Interest? • Disclose conflicts • PharmedOut.org • Dollars For Docs series http://projects.propublica.org/docdollars/
  37. 37. Are There Alternatives? • Did the study compare the new treatment to existing alternatives, or to placebo? • What are the advantages and disadvantages (and costs) of those existing alternatives? • Consider alternative explanations. Remember coffee and pancreatic cancer?
  38. 38. Don’t Rely Only on Study Authors • Find outside sources. Here’s how:
  39. 39. Use Anecdotes Carefully • Is the story representative? • Does the source of the story have any conflicts?
  40. 40. Watch Your Language • Lifestyle/diet – are they randomized controlled trials, or just observational? • If observational, make the language fit the evidence: – YES: “tied,” “linked” – NO: “reduces,” “causes”
  41. 41. A Dirty Little Secret Keep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr
  42. 42. Keep Yourself Honest • Use HealthNewsReview.org
  43. 43. Acknowledgement/Contact • Nancy Lapid, Reuters Health ivan-oransky@erols.com Twitter: @ivanoransky

More Related Content

Similar to Covering Medical Studies: How Not to Get It Wrong

Related Audiobooks

Free with a 30 day trial from Scribd

See all

×