Delivered in Washington, DC, on November 16, 2014. These slides also became the basis for a talk I gave via Skype to Doug Starr's class in the graduate Program in Science and Medical Journalism at Boston University on November 19.

1. Evaluating media messages on
cancer studies
Project LEAD workshop
Gary Schwitzer
Publisher, HealthNewsReview.org
Adjunct Assoc. Prof., School of Public Health, University of Minnesota

2. Our criteria: Does the story
explain…
• What’s the total cost?
• How often do benefits occur?
• How often do harms occur?
• How strong is the evidence?
• Are there alternative choices?
• Is the condition exaggerated?
• Is this really a new approach?
• Is it available?
• Who’s promoting this?
• Do they have a financial conflict of interest?
69% 
66% 
65% 
61% 
57% 
Percent
unsatisfactory
after 1,889
story reviews –
7 years

4. Most common flaws
• Conveying a certainty that doesn’t exist
– Exaggerating effect size
– Using causal language to describe observational
studies
– Failing to explain limitations of surrogate
markers/endpoints
– Single source stories with no independent
perspective
– Failing to independently analyze quality of evidence

5. Exaggerating effect size –
Absolute vs. Relative Risk
• Two ways of saying the same thing
• One way – relative risk reduction – makes effect size
seem larger
• Other way – absolute risk reduction – makes effect
size seem smaller.
• We absolutely think you should use the absolute
figures.

6. Relative vs. absolute risk reductions
Placebo DRUG Relative Absolute
??
Chance of death at 1 year
30330%
030%%
%
10%
Risk reduction
67%
30% 10% 67% 20%
67%
3% 1% 67% 2%
67%
2?0?%
2%
0.002%
0.003% 0.001% 67% 0.002%

7. Nolvadex (tamoxifen)
Reducing breast cancer risk by 50 .

8. You must ask: 50% of what?
• The “of what?” is the absolute risk.
• When you’re only told the relative risk – 50% -
it could be risk reduction from 90 in 100 down
to 45 in 100…so 45 benefit.
• Or it could be from 2 in 100 to 1 in 100….so
only 1 benefits.
• You want to know the absolute size of the
effect, not just one relative to another.

9. What if we used only relative terms on
harms?
• In that same Nolvadex/tamoxifen study
example, the tamoxifen users had 210% more
uterine cancers and potentially life-threatening
blood clots in the lungs and legs.

10. Over 6 years, what happened …
PLACEBO NOLVADEX
Benefits: Nolvadex lowered chance
Getting breast cancer
Harms: Nolvadex increased chance
Having a serious blood clot
3.3% 1.7%
0.5% 1.0%
Getting uterine cancer 0.5% 1.1%
Net effect of Nolvadex for every 1000 women:
16 fewer women get breast cancer
5 more women get serious blood clots
6 more get uterine cancer

11. USING CAUSAL LANGUAGE TO
DESCRIBE OBSERVATIONAL STUDY
RESULTS…FAILING TO DESCRIBE
LIMITATIONS OF SUCH DATA

12. Association ≠ Causation

13. Limits of observational studies
• HealthDay News: “Going up a skirt size every
10 years raised chances of developing breast
cancer by 33%.”
Reported by many news organizations.
What is the impact of this on a woman who
goes from size 8 at age 40 to size 10 at age 50?

14. Sometimes even journals get it wrong
BMJ news release in
June, 2014
Many news stories used the “raises breast cancer risk” language.
But The Guardian in the UK used just 6 little words:
“association does not necessarily imply causation.”
I wrote about a half dozen such BMJ news releases in recent years, until finally, they
started using boilerplate language:
“This is an observational study so no definitive conclusions can be drawn about
cause and effect.”

15. FAILING TO EXPLAIN LIMITATIONS OF
SURROGATE MARKERS/ENDPOINTS

16. An example of journalism excellence
Many cancer drugs approved on the basis of progression-free survival -
which means how long patients survived before doctors detected a
tumor worsening. It does not necessarily mean an improvement in
overall survival. And many of these drugs have failed to show
improvements in overall survival after they were approved by the FDA.

17. RELYING ON A SINGLE SOURCE WITH
NO INDEPENDENT PERSPECTIVES
(SOMETIMES RELYING SOLELY OR LARGELY ON NEWS
RELEASES OR PUBLIC RELATIONS ANNOUNCEMENTS)

18. The Tyranny
Of The
Anecdote
• Was urged by co-workers to have 1st mammogram on the air.
• Didn’t disclose details of results, but announced she’d get bilateral
mastectomies.
• She wrote:
“I can only hope my story will… inspire every woman who hears
it to get a mammogram, to take a self-exam. No excuses. It is the
difference between life and death.”
• No excuses? Mammography should be informed choice, not mandate.
• Women can have rational reasons for declining mammograms.
• Mammogram will NOT be the difference between life and death for
most women.

19. Just a few out of 1000s of stories
• Boston Globe: Futuristic sports bra could detect early
signs of breast cancer
• ABC: Is a new heat sensing bra the breast medicine?
• CBS: Bra aims to detect breast cancer before
mammogram.
• WCBS, NY: Cancer-detecting bra may save women’s
lives
• CBC: High tech ‘smart bra’ could replace
mammograms
• CNN: A bra that could detect cancer

20. Take notes, then we’ll discuss

21. We blogged & raised ?s
• When news was released, NBCC staffer couldn’t
find clinical trial data on company website.
• Details about what was actually detected were
lacking
• Existing thermography unreliable for
differentiating cancer from normal tissue
• Ted Gansler, MD, editor of CA: A Cancer Journal
for Clinicians said:
“At this time, based on evidence currently available, I think the
vast majority of doctors will agree that a woman who chooses
any breast cancer screening test based on temperature
measurements, instead of mammography, would be making a
serious mistake that could have fatal consequences.”

23. Dr. Russell Harris, Univ. of North Carolina Cancer Center:
• The problems lie not with their numbers, but with what they left out.
• They are talking about a one-time screen. But for multiple screens over 10
years, say, the number of women with at least one positive mammogram is
more like 500 out of the 1,000.
• And the biopsy rate (at least one biopsy) is also higher.
• And they don’t discuss the anxiety caused by having a false positive or from
having to return for a follow-up surveillance mammogram.
• And they don’t say how many of the 5 women with breast cancer diagnosed
not by screening but by symptoms would have also had a “high cure
rate” (only 1 additional 40 year old women in 1,000 who are screened for 10
years have their lives extended by screening – 2 in 1,000 for 50 year olds, 3 in
1,000 for 60 year olds – and these are optimistic figures – it may be less than
this).
• And they don’t say how many of those 5 women diagnosed with breast
cancer are overdiagnosed – i.e., would never have been diagnosed at all if
they had not been screened.
• But they do emphasize that “Mammography Saves Lives: One of them may
be yours.”

24. • Breakthrough?
• Exclusive?
• Developing Story?
• Cancer Cure within
reach?
It was all based on a medical
center’s public relations
announcement…not any new
research or any new finding.
What about all the other research
at all the other cancer centers
aiming to do the same thing?

25. Remember 2 of our 10 story criteria…
• Did the story go beyond a PR effort or news release? NO
• Did the story disclose conflicts of interests in its sources?
NO. CNN didn’t report on….
• Resignations, questions about conflicts of interest and
fraud probes of the program in question.
• Not likely to be the kinds of angles and issues one
pursues when there are “exclusive” reporting
arrangements between a news organization and a
medical center.

26. How did this…..
….become this?

27. And all of this…. 327,000 results on a
Google search!

28. A university news release played a role

29. FAILURE TO INDEPENDENTLY
ANALYZE THE QUALITY OF THE
EVIDENCE…..OR, STENOGRAPHY

30. “Good journalism has a subtle feature of
reticence. We don’t publish everything we hear.
We filter. We curate. The goal of the traditional
journalist is to create a reputation for accuracy,
fairness, relevance and timeliness, and this
requires the willingness to not publish things
that are unlikely to be true. … There’s nothing at
stake here except the survival of credible
journalism.”
- Joel Achenbach
Washington Post

31. Look for the extra dimension
in health care news
June 25: Faithful to the latest
journal article, many news
organizations dutifully reported
what they were told by authors of
a study published in the Journal of
the American Medical Association,
“Breast Cancer Screening Using
Tomosynthesis in Combination
with Digital Mammography.“
Many stories used sensational
language – “breakthrough, game-changer,
best way of detection,
any woman should have this,
lifesaver.”

32. Some stories delivered more…
• USA Today: “3D mammograms find more cancers,
but do they save lives?”
– “Other breast cancer advocates caution that doctors
still don’t know whether finding these extra cancers
will actually save lives, however, or simply lead more
women to treatment.” Later, this:
“The absolute differences between (3-D and standard
mammograms), in terms of the number of cancers found, is
very small, says Fran Visco, president of the National Breast
Cancer Coalition. And other benefits are uncertain.
The study didn’t follow women over time. So doctors don’t
know whether undergoing tomosynthesis actually saves lives,
Visco says.”

33. Some stories delivered more…
• NY Times: “The verdict is still out on the long-term
worth of this new technology. The new
results are promising but not definitive…Even so,
more and more mammography centers are
buying the equipment.”
• Boston Globe: “Finding more cancers isn’t our
goal,” said Dr. Lisa Schwartz, professor of
medicine at the Dartmouth Institute for Health
Policy and Clinical Practice. “It’s about reducing
breast cancer deaths. We don’t know whether
this new technology finds more cancers destined
to kill people.”

34. Some stories delivered more…
• MedPage Today: “If we step back and ask
whether these papers show that better breast
imaging will actually translate into lives saved, it’s
unclear. The differences are small enough that
you want to analyze very carefully the trade-offs
in terms of extra biopsies, extra imaging effort,
and the extra cost involved, before thinking that
it should be a routine piece of our public health
screening for breast cancer,” said Harold Burstein,
MD, PhD, of Dana-Farber Cancer Institute in
Boston.

35. That’s quite a bit different than
“breakthrough...game-changer…best way of
detection…any woman should have
this…lifesaver” of ABC, NBC, Fox, Chicago Sun-
Times, Washington Post and others.
This gives you an idea of what a difference
independent perspectives, asking tough
questions, and independently vetting claims –
not just practicing journal stenography – can
make for ensuring accuracy, balance and
completeness in health care news.

36. Pharma says that for every 5,000 compounds in pre-clinical
testing, only 5 make it to human trials. And of those, only 1
will get FDA approval.
Another estimate: only 8% of new drugs in Phase I studies ever get
approved
But we keep seeing stories like:
– The Toronto Star: “New breast cancer drug heralded as breakthrough”
(Phase 2 study)
– CNN: “Stem cell medical breakthrough?” (Phase 1 study)
– Medscape: ‘Truly Remarkable’ Response with Combination for
Melanoma (Phase I study – 37 people followed 12 weeks)

37. “BREAKTHROUGH” on all 3 main TV
networks on a Phase I study
Note that all 3 practice
“question mark
journalism” – which
allows you to say
anything, followed by ?

38. Cover story “breakthrough”
16 words: “moving into
the testing phase”

39. Stenography, not journalism
Reported on Phase 2 breast cancer drug study – directly from a Pfizer news release –
which didn’t include any data.
But the story lifted a glowing quote from the company news release:
“The study suggests ‘the potential to transform the standard of care,’ said Mace
Rothenberg, Pfizer Oncology's chief medical officer. ‘This is encouraging information
for these women.’ “
No data. No independent perspective. Free publicity on results no one will see for two
more months when they are presented at a meeting.
But Reuters and AP also reported on the Pfizer news release. Those stories were picked
up by Huffington Post, ABC, Charlotte Observer, Boston Herald, Idaho Statesman,
Washington Times, Bradenton Herald, Fort Worth Star Telegram, Myrtle Beach Sun News,
Philly.com, others

40. 2013: The year we “cured” cancer – three times !
“Breaking Through Cancer’s Shield” - Oct 2013

41. “The trouble with people
is not that they don't know, but
that they know so much that ain't so.
…
I honestly believe it is better to know
nothing that to know what ain’t so.”
JOSH BILLINGS (PEN NAME OF HUMORIST
HENRY WHEELER SHAW, 1818 – 1885)

42. Journal of Incidentalomas
Journal of Experimental Efficacy
Journal of Preclinical Self-Importance
Journal of Surrogate Markers Only

43. September 2012
Positive “spin” was identified in about half of press releases and
news stories. The main factor associated with “spin” in press
releases was the presence of “spin” in the journal article abstract
conclusion.
In other words, a direct link from
published study news release news story.
Who thinks about the reader, the consumer,
the patient…at the end of this food chain?

44. Mayo’s Dr. Victor Montori to a journal club:
“Beware spin: composite endpoints, surrogate
markers, subgroup analyses, inadequate
comparators (too much or too little of an effective
alternative or placebo when an effective alternative
exists). Above all, avoid the intro/discussion
sections which is where most of the interpretational
spin is introduced.”
But many journalists – who feed off a steady diet
of journal studies in order to meet their story
quotas and click rates – have no idea what any of
these terms mean.

45. Dr. Richard Lehman has published reviews of
journal articles for 8 years on BMJ website
“I too was once a conclusion-of-the-abstract
reader, and was quite smug that I had even got
that far. It took me some years to become
aware of perhaps the most important principle
of critical reading: never believe the stated
bottom line without confirming it from the data.
And beware of the limitations of the data.”

46. Dr. Lehman reminds us:
350 years ago the Royal Society of London for Improving
Natural Knowledge used as its motto,
Nullius in verba
Rough translation:
“Don’t take anyone’s word for it.”
We still have much to learn from those notable 17th
century skeptics.

47. “Bad science is
no excuse for
bad journalism.”

48. Unprecedented
peaks of
excellence in
health care
journalism

49. But the valleys between those
peaks are becoming wider and
deeper – what we call…..
Daily Drumbeat of Dreck

50. Journalists could help people understand
and deal with the clash between:
• Science
• Evidence
• Data
• Recommendations for
entire population
• What we can prove
• Grasping uncertainty and
helping people apply
critical thinking to
decision-making issues
 Intuition
 Emotion
 Anecdote
 Decision-making by an
individual
 What we believe, wish, or hope
 Promoting false certainty and
non-evidence-based,
cheerleading advocacy

51. Thank you
Gary@HealthNewsReview.org