Evaluating Medical Evidence       for Journalists          Ivan Oransky, MD   Executive Editor, Reuters Health  Associatio...
Can You Trust Journal Studies?• How good is peer review?• Positive publication bias:    Publish a trial that will bring US...
Embargoes and the Ingelfinger RuleBy the late 20th century, journals needed to compete notjust with each other but with ne...
Even Without Embargoes, We’d Still Have              Ingelfinger
How Often Are Studies Wrong?               Ioannidis JPA. PLoS Med 2005; 2(8): e124
How Often Are Studies Wrong?
Retractions on the Rise                 -The Wall Street Journal
Retractions on the Rise                     -Neil Saunders
The Unofficial Record Holder
This is Transparency?
Conference Pitfalls
Conference Pitfalls• Conferences select presenters based on < 1000 words• Urologists at U of Florida & Indiana U studied 1...
Conference Pitfalls• RCTs are the “gold standard” of medical evidence• But the quality of that evidence wasn’t pretty• No ...
Just Say NoSometimes, it’s better not to cover something.But if you must…
Always Read the StudyWriting about a study after reading just a press             release or an abstract     – without rea...
How to Get Studies• www.EurekAlert.org for embargoed material• Association of Health Care Journalists membership  includes...
How Good Was The Study?• Was it:  – Peer-reviewed?  – Published? Where?• Was it in humans?  – It’s remarkable there are an...
From Covering Medical Research, Schwitzer/AHCJ
What’s Your Angle?• Are you trying to help readers, listeners, and  viewers make better health care decisions?• Covering a...
Who Could Benefit?• How many people have the disease?• Keep potential disease-mongering in mind
How Effective is the Treatment?• Clinically significant endpoints, or surrogates –  does this matter?• Preventing complica...
What Are The Side Effects?• Every treatment has them• Where to look:  – Go beyond press releases and abstracts  – Look at ...
Who Dropped Out?• Why did they leave the trial?• Intention to treat analysis
How Much Does it Cost?• If it’s ready to be the subject of a story,  someone has projected the likely cost and  market.   ...
Who Has an Interest?• Disclose conflicts• PharmedOut.org• Dollars For Docs series  http://projects.propublica.org/docdolla...
Are There Alternatives?• Did the study compare the new treatment to  existing alternatives, or to placebo?• What are the a...
Don’t Rely Only on Study Authors• Find outside sources. Here’s how:
Use Anecdotes Carefully• Is the story representative?• Does the source of the story have any conflicts?
Watch Your Language• Lifestyle/diet – are they randomized controlled  trials, or just observational?• If observational, ma...
Absolute vs. Relative Risk• Consider the risk for blindness in a patient with  diabetes over a five-year period• The risk ...
Absolute vs. Relative Risk• Expressed as an absolute difference, the new drug  reduces the five-year risk for blindness by...
Number Needed To Treat• Same concept as number needed to screen• Can be calculated from absolute risk:   – 100/absolute ri...
A Dirty Little SecretKeep a biostatistician in your back pocket                               Photo by Peyri Herrera, on F...
Keep Yourself Honest• Use HealthNewsReview.org
Acknowledgement/Contact• Nancy Lapid, Reuters Health          ivan-oransky@erols.com            Twitter: @ivanoransky
Upcoming SlideShare
Loading in...5
×

Evaluating medical evidence for journalists

6,122

Published on

How to evaluate medical evidence for journalists, AHCJ 2012

Evaluating medical evidence for journalists

  1. 1. Evaluating Medical Evidence for Journalists Ivan Oransky, MD Executive Editor, Reuters Health Association of Health Care Journalists Atlanta April 19, 2012
  2. 2. Can You Trust Journal Studies?• How good is peer review?• Positive publication bias: Publish a trial that will bring US$100,000 of profit or meet the end-of-year budget by firing an editor. -- Former BMJ editor Richard Smith• Over-reliance on embargoed studies• How often it turns out to be wrong
  3. 3. Embargoes and the Ingelfinger RuleBy the late 20th century, journals needed to compete notjust with each other but with newspapers and othermedia…In 1969, the Journal articulated this relationshipin its Ingelfinger Rule, a policy against publishing anythingthat had already appeared elsewhere. Other journalsfollowed suit. This rule, combined with embargopolicies, has led to a carefully choreographedproduction in which medical journals and the popularpress work cooperatively and competitively to influencethe news cycle. -- NEJM, April 19, 2012
  4. 4. Even Without Embargoes, We’d Still Have Ingelfinger
  5. 5. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  6. 6. How Often Are Studies Wrong?
  7. 7. Retractions on the Rise -The Wall Street Journal
  8. 8. Retractions on the Rise -Neil Saunders
  9. 9. The Unofficial Record Holder
  10. 10. This is Transparency?
  11. 11. Conference Pitfalls
  12. 12. Conference Pitfalls• Conferences select presenters based on < 1000 words• Urologists at U of Florida & Indiana U studied 126 randomized controlled trials presented in 2002-2003
  13. 13. Conference Pitfalls• RCTs are the “gold standard” of medical evidence• But the quality of that evidence wasn’t pretty• No abstract said how trial subjects were randomly assigned to different treatments or placebos• None told how the study ensured that neither the researchers nor their doctors knew which they got• Only about a quarter said how long researchers followed the subjects in the trial
  14. 14. Just Say NoSometimes, it’s better not to cover something.But if you must…
  15. 15. Always Read the StudyWriting about a study after reading just a press release or an abstract – without reading the entire paper – is journalistic malpractice
  16. 16. How to Get Studies• www.EurekAlert.org for embargoed material• Association of Health Care Journalists membership includes access to Cochrane Library, Health Affairs, JAMA, and many other journals www.healthjournalism.org• ScienceDirect (Elsevier) gives reporters free access to hundreds of journals www.sciencedirect.com• Open access journals (e.g., Public Library of Science www.plos.org)• Ask press officers, or the authors
  17. 17. How Good Was The Study?• Was it: – Peer-reviewed? – Published? Where?• Was it in humans? – It’s remarkable there are any mice left with cancer, depression, or restless leg syndrome• Size matters• Was it well-designed?
  18. 18. From Covering Medical Research, Schwitzer/AHCJ
  19. 19. What’s Your Angle?• Are you trying to help readers, listeners, and viewers make better health care decisions?• Covering a study because it has a good business angle, or it’s about a local project, is perfectly OK, but it doesn’t mean readers deserve less evidence and skepticism
  20. 20. Who Could Benefit?• How many people have the disease?• Keep potential disease-mongering in mind
  21. 21. How Effective is the Treatment?• Clinically significant endpoints, or surrogates – does this matter?• Preventing complications? How many?• Always remember to quantify results, not just “patients improved”
  22. 22. What Are The Side Effects?• Every treatment has them• Where to look: – Go beyond press releases and abstracts – Look at tables, charts, and results sections
  23. 23. Who Dropped Out?• Why did they leave the trial?• Intention to treat analysis
  24. 24. How Much Does it Cost?• If it’s ready to be the subject of a story, someone has projected the likely cost and market. – At least ask.
  25. 25. Who Has an Interest?• Disclose conflicts• PharmedOut.org• Dollars For Docs series http://projects.propublica.org/docdollars/
  26. 26. Are There Alternatives?• Did the study compare the new treatment to existing alternatives, or to placebo?• What are the advantages and disadvantages (and costs) of those existing alternatives?
  27. 27. Don’t Rely Only on Study Authors• Find outside sources. Here’s how:
  28. 28. Use Anecdotes Carefully• Is the story representative?• Does the source of the story have any conflicts?
  29. 29. Watch Your Language• Lifestyle/diet – are they randomized controlled trials, or just observational?• If observational, make the language fit the evidence: – YES: “tied,” “linked” – NO: “reduces,” “causes”
  30. 30. Absolute vs. Relative Risk• Consider the risk for blindness in a patient with diabetes over a five-year period• The risk for blindness is 2 in 100 (2%) in people who get the conventional treatment and 1 in 100 (1%) with a new drug• The absolute difference is derived by subtracting the lower risk from the higher risk: 2% - 1% = 1%. From Covering Medical Research, Schwitzer/AHCJ
  31. 31. Absolute vs. Relative Risk• Expressed as an absolute difference, the new drug reduces the five-year risk for blindness by 1%.• The relative difference is the ratio of the two risks.• Given the data above, the relative difference is: 1% ÷ 2% = 50%• Expressed as a relative difference, the new drug cuts the risk of blindness in half. From Covering Medical Research, Schwitzer/AHCJ
  32. 32. Number Needed To Treat• Same concept as number needed to screen• Can be calculated from absolute risk: – 100/absolute risk difference (as a percentage)
  33. 33. A Dirty Little SecretKeep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr
  34. 34. Keep Yourself Honest• Use HealthNewsReview.org
  35. 35. Acknowledgement/Contact• Nancy Lapid, Reuters Health ivan-oransky@erols.com Twitter: @ivanoransky
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×