AHCJ 2012 Atlanta conf. talk

What are your criteria inreporting on health careresearch? Gary Schwitzer, Publisher, HealthNewsReview.org

Our plan for next 2 hours:  Background on HealthNewsReview.org, what we do, how we do it, why, and what we’ve found  Recurring pitfalls we’ve seen in 6 years of daily monitoring  An editor’s perspective on addressing these issues  We’ll review a story – together – using HealthNewsReview.org criteria.  Plenty of time for Q & A – tapping Ivan’s editor expertise  Handouts:  Summary of our 6-year, 1,700 story experience  Resources for reporting on studies  Story to review & scoresheet

We review stories thatinclude claims about…  Medical treatments  Tests  Products  Procedures

Site stats 28 reviewers: 19 clinicians or researchers, 7 journalists, 2 breast cancer survivors Site launched April 17, 2006 – 6th anniversary week! >1,700 stories reviewed Blog launched in 2004 - >2,500 posts Sole funding from the Informed Medical Decisions Foundation, founded by Jack Wennberg et al

Global Reach for such efforts= HealthNewsReview.org (U.S.), Media Doctor sites inAustralia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites inUK and Austria= talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, PuertoRico only in two years’ time

Our criteria: Does the story explain… What’s the total cost? How often do benefits occur? How often do harms occur? How strong is the evidence? Is the condition exaggerated? Is this really a new approach? Is it available? Are there alternative choices? Who’s promoting this? Do they have a financial conflict of interest?

After 6 years and 1,700 stories 70% fail to adequately discuss costs. 66% fail to quantify benefit – often exaggerating potential benefit 65% fail to quantify harm - often minimizing potential harm 62% fail to evaluate the quality of the evidence 57% fail to compare new idea with existing options

Kid-in-candy-store picture of U.S. health care Everything is terrific Nothing is risky No price tags

Recurring themes seen after 6 years of daily monitoring Failure to explain limitations of observational studies. Stories that conflate association and causation. Framing numbers to exaggerate benefit, minimize harms Stories about screening tests that emphasize only benefits, minimizing or ignoring potential harms

Be skeptical about observational studiesRemind readers that the findings may be moreabout the people being studied than the "exposure".They can point to a strong statisticalassociation, but they cannot prove cause and effect.

White rice increases risk of type 2 diabetesThursday, March 15, 2012The risk of type 2 diabetes is significantly increased if white rice is eatenregularly, claims a study published today on bmj.com.More than 400 words in the news release but NONE about the limitations ofsuch an observational study.

Does your language fit the evidence of observational studies?

Absolute versus relative risk

Following slides courtesyDrs. Steve Woloshin & Lisa Schwartz Dartmouth Medical School White River Junction, Vermont VA Part of their syllabus at: NIH Medicine in the Media MIT Medical Evidence Boot Camp

Nolvadex (tamoxifen)Reducing breast cancer risk by 50 .

For the first time, there is a clinically provenway for many women at high risk ofdeveloping breast cancer to significantly reducethat risk. The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

Women who took Nolvadex had 48% fewer breastcancers…. 48% of what ???

Its like a coupon… Extremely Fancy Store What if selected items were.. TV’s, washing machines?48 On selected items! % OFF save $100s Things like a pack of gum? save pennies ―48% of what‖ matters! Know the REGULAR price!

What is the "Breast cancer risk"48% off Chance of getting breastcoupon really cancerworth? Placebo NOLVADEX ??? ??? How much do you save?

The chance of getting breast cancer over 6 years in the PLACEBO group PLACEBO 3.3% The REGULAR price! The "base rate"Absolute risk in the placebo group Event rate in the placebo group

The chance of getting breast cancer over 6 years in the INTERVENTION group PLACEBO NOLVADEX 3.3% 1.7% The REGULAR price! The SALES price!

What is the effect of Nolvadex?How good is the sale? The REGULAR price! PLACEBO The SALES price! NOLVADEX 3.3% 1.7% How much do you save?Absolute risk reduction Regular price – Sales price = 1.6% Savings = 3.3% 1.7%Chance of women breast cancer (over 6 years)placebo for 6 If 100 getting took NOLVADEX instead of with NOLVADEX years, there would be about 2 fewer cases placebo cancer. was 1.6 % points lower than with of breast

What is the effect of Nolvadex?How good is the sale? PLACEBO NOLVADEX 3.3% 1.7% Chance of outcome (intervention) Relative Risk = Chance of outcome (control)

Describing the effect of NOLVADEX So finally....this is how you get to PLACEBO NOLVADEX the48% off sale! 3.3% 1.7% Its the relative risk reduction 1.7% ??? Relative Risk = = 0.52 ??? 3.3% "% Lower" format =1 - RR =1 - .52 =.48At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.

Two ways of saying the same thing: the benefit of NOLVADEX One feels big Extremely Fancy Store One feels small Extremely Fancy Store4On selected items! % OFF 1.6% matters! SAVINGS How you say it On selected items! "Framing"

Relative vs. absolute"savings" "% off" risk reductionsChance of death at 1 Risk reductionyear Relative AbsolutePlacebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)30% 10% 67% ?? 20% ?? 3% 1% 67% 2%0.003% 0.001% 67% 0.002%

The proof? In a landmarkBenefit study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.Harm "Nolvadex isnt for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women". ‖Strokes, cataracts more common with No numbers Nolvadex. Most women experience some level of hot flashes and vaginal discharge".

Present benefits and harms the same way

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.210% more uterine cancer and potentially 210% more life threatening blood clots in the lung and legs.

Harm6-yr chance of venous thromboembolic event: PLACEBO NOLVADEX 0.5% 1.0% Like inflation…doubled the price!

Over the next 6 years, what happened when women… PLACEBO NOLVADEXBenefits: Nolvadex lowered chance Getting breast cancer 3.3% 1.7%Harms: Nolvadex increased chance Having a serious blood clot 0.5% 1.0%Getting uterine cancer 0.5% 1.1% Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer

Take home messagesExtrapolate with caution! Dont tell people what to worry about – or do – based on very preliminary animal / lab science.Recognize pseudo-evidence Publication in a medical journal - even "the New England Journal of Medicine" - does not guarantee the findings are true (or even important).Be wary of inherently weak science Without a comparison group, it is impossible to be sure if the drug was responsible for the findings.Pay attention to the outcome Surrogate outcomes (like tumor shrinkage) do not reliably translate into clinically meaningful outcomes (longer life).Be skeptical about observational studies Remind readers that the findings may be more about the people being studied than the "exposure".Avoid exaggerated numbers Use absolute risks for both benefits and harms.

Pitfalls of a steady diet of journal stories PLoS Med 2005; 2(8): e124

The problem begins with the public’s rising expectations ofscience. Being human, scientists are tempted to show thatthey know more than they do. The number of investigators—and the number of experiments, observations and analysesthey produce—has also increased exponentially in manyfields, but adequate safeguards against bias are lacking.Research is fragmented, competition is fierce and emphasis isoften given to single studies instead of the big picture. Muchresearch is conducted for reasons other than the pursuit oftruth. Conflicts of interest abound, and they influenceoutcomes.

Spinning results of randomized clinical trialsBoultron et al, JAMA May 26, 2010, 303 (20): 2058In this representative sample of RCTs published in2006 with statistically non-significant primaryoutcomes, the reporting and interpretation of findingswas frequently inconsistent with the results.

Reporting bias in medical research - a narrative review McGauran et al. Trials 2010, 11:37 We identified reporting bias in 40 indications comprising around 50 different pharmacological, diagnostic, and preventive interventions. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.

BE AWARE OF…..• What Ivan’s project has shown about retractions, research fraud, fabrication, falsification of data• Unpublished data (BMJ recently published 8 articles in one edition on “the extent, causes and consequences of unpublished evidence”)• Conflicts of interest – guideline setting

• Commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies (Carl Elliott)• Ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)• The focus on surrogate markers in many studies may be hurting patient care. (primer in HealthNewsReview.org online toolkit)

News coverage & poor public discussion of screeningtests is one of the most concerning public policy issues.Worst, most biased coverage I’ve seen in 37 years

What the Task Force actually wrote:“The decision to start regular, biennial screeningmammography before the age of 50 yearsshould be an individual one and take patientcontext into account, including the patientsvalues regarding specific benefits and harms.”

As 37-year ChiTrib & NYT vet John Crewdson wrote in The Atlantic… “There are multiple reasons women are ill- informed about breast cancer. The fault lies primarily with their physicians, the cancer establishment, and the news media--especially the news media. Until coverage of breast cancer rises above the level of scary warnings mixed with heartwarming stories of cancer survivors, women are likely to go on being perplexed."

Other examples: Chicago Sun-Times Wall Street Journal Washington Post New York Daily News Minneapolis Star Tribune All TV networks

The inkisn’t even dry on studies beforemarketing begins

HealthDay wire service didn’t challengeresearcher promoting universal pancreaticcancer screening for everyone over 50.  After study of tissue from 7 people!

We could have given the finger to this story "Relative finger length could be used as a simple test for prostate cancer risk, particularly in men aged under 60," said one of the researchers.

Screening Madness Crusading one-sided advocacy

Why don’t we deliver this message? “All screening tests cause harm; some may do good.”But much health journalism consistently emphasizesbenefits & minimizes harms

A form of disease-mongering Selling sickness Selling the search for weapons of mass destruction inside everyone Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK ―Screening the apparently healthy potentially saves a few lives. But it definitely drags many others into the system needlessly — into needless appointments, needless tests, needless drugs and needless operations. This process doesn’t promote health; it promotes disease. People suffer from more anxiety about their health, from drug side effects, from complications of surgery. A few die. And remember: these people felt fine when they entered the health care system.‖ Communication of the evidence is a key health policy issue

―I honestly believe it is better to know nothingthan to know what ain’t so.‖ Josh Billings (pen name of humorist Henry Wheeler Shaw, 1818 – 1885)

Thank youGary@HealthNewsReview.org

http://philippehavinh.wordpress.com	197
http://www.healthnewsreview.org	70
https://si0.twimg.com	16
https://twimg0-a.akamaihd.net	4
http://us-w1.rockmelt.com	2

AHCJ 2012 Atlanta conf. talk

by Gary Schwitzer on Apr 21, 2012

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 289

Statistics

AHCJ 2012 Atlanta conf. talk Presentation Transcript