America Is the Target; Israel Is the Front Line _ Andy Blumenthal _ The Blogs...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer & Oransky
1. Lessons from
HealthNewsReview.org
Gary Schwitzer
Publisher, HealthNewsReview.org
Center for Media Communication & Health
University of Minnesota School of Public Health
@garyschwitzer
@HealthNewsRevu
gary@healthnewsreview.org
Association of Health Care Journalists April 23, 2015
2. In health care news, we
have our own version of
the Rolling Stone rape
story every day
• “The failure encompassed reporting, editing, editorial supervision and fact-
checking.”
• “failing to state where important information had come from.”
• “There is a tension between crafting a readable story - a story that flows - and
providing clear attribution of quotations and facts.”
• “The editors invested Rolling Stone’s reputation in a single source.”
• “The problem of confirmation bias - the tendency of people to be trapped by
pre-existing assumptions and to select facts that support their own views
while overlooking contradictory ones - is a well-established finding of social
science.”
• Other critics: “The writer wanted it to be true, and the editor and fact-checker
failed to push.” – “There’s no substitute for skepticism.”
3. • HealthNewsReview.org reviews news stories
that include claims about interventions:
treatments, tests, products, procedures
• We regularly check:
Boston Globe
Los Angeles Times
New York Times
Philadelphia Inquirer
USA Today
Wall Street Journal
Washington Post
Associated Press
Bloomberg News
Websites of ABC, CBS, CNN,
Fox, NBC
NPR health & science page
HealthDay
Reuters Health
Vox.com
Slate.com
FiveThirtyEight.com
BuzzFeed.com
Websites of TIME, Newsweek,
U.S. News & World Report
4. As of last week we also now systematically review
health care news releases from many sources
5. • 13 journalists
• 8 AHCJ members
• 13 MDs
• 7 PhDs
• 5 communication scholars and/or
academic public information officers
• 4 public health grad students
• 3 women with breast cancer (one w/PhD,
two with Master’s degrees)
Every story &
release is analyzed
by 3 reviewers,
each applying the
same 10 criteria
6. Our criteria: Does the story
explain…
• What’s the total cost?
• How often do benefits occur?
• How often do harms occur?
• How strong is the evidence?
• Are there alternative choices?
• Is the condition exaggerated?
• Is this really a new approach?
• Is it available?
• Who’s promoting this?
• Do they have a financial conflict of interest?
69%
66%
65%
61%
57%
Percent
unsatisfactory
after 1,980
story reviews
7.
8. Most common flaw:
Conveying certainty
that doesn’t exist
• Exaggerating effect size – relative not absolute data
• Using causal language to describe observational studies
• Idolatry of the surrogate: Failing to explain limitations of
surrogate markers/endpoints
• Tyranny of the anecdote: telling success stories but rarely
profiling dropouts, dissatisfied, those who choose conservative
route or lifestyle change instead of treatment
• Single source stories with no independent perspective
• Failing to independently analyze quality of evidence
10. Dr. Richard Lehman has
published reviews of
journal articles for 8 years
on BMJ website
“I too was once a conclusion-of-the-abstract
reader, and was quite smug that I had even got
that far. It took me some years to become
aware of perhaps the most important principle
of critical reading: never believe the stated
bottom line without confirming it from the data.
And beware of the limitations of the data.”
11. Advice from a crusty, but trusted senior colleague
“You have been doing yeoman work fighting multiple battles. Is it time to
tackle the larger war? What constitutes evidence? A case report? A
journal publication? A presentation?
The tenor of the times proclaims that we are practicing evidence-based
medicine despite the fact that the level of evidence is scant. Many
studies have high risk of bias & the strength of evidence is weak.
You are proposing some variant of educational immunization to help
journalists resist the poor information they receive.
Another strategy is to address fundamental problems. Should not
journal editors be urged to improve the review process to weed out the
large numbers of low quality work? Should we do more to expose the
low level of evidence?”
13. But It Was Peer-Reviewed –
It Must Be Right!
AHCJ 2015
Santa Clara
Ivan Oransky
Vice President, Global Editorial Director,
MedPage Today
Co-Founder, Retraction Watch
17. But It Was Peer-Reviewed!
“Fifty-three papers were deemed 'landmark' studies. It was
acknowledged from the outset that some of the data might
not hold up, because papers were deliberately selected that
described something completely new, such as fresh
approaches to targeting cancers or alternative clinical uses
for existing therapeutics. Nevertheless, scientific findings
were confirmed in only 6 (11%) cases. Even knowing the
limitations of preclinical research, this was a shocking
result.”
28. Our learning examples are online at:
http://www.healthnewsreview.org/2015/04/sch
witzeroransky-workshop-how-to-accurately-
report-on-medical-research-findings/
Editor's Notes
These aren’t off-the-cuff observations.
For 8 years, I’ve published HealthNewsReview.org, a project that tries to improve the public dialogue about health care by analyzing media messages. The team I led conducted systematic reviews of nearly 2,000 news stories by leading organizations…applying these 10 criteria.
The report card is not good.
60-70% of those nearly 2,000 stories were judged unsatisfactory on what are arguably the five most important of our 10 criteria. For the information consumers need.
That means 60-70% of our huge sample was not ready for prime time. These stories made most interventions sound terrific…risk-free…and without a price tag….and to hell with the evidence and to hell with meaningful comparisons with existing alternatives.
This work was summarized this summer in a paper I had published in JAMA Internal Medicine.
The most common flaws, summarized in that paper, were news stories that:
Conveyed a certainty that doesn’t exist
Exaggeraed effect size
Used causal language to describe observational studies
Failed to explain limitations of surrogate markers/endpoints
Single source stories with no independent perspective
Failed to independently analyze quality of evidence