MIT Medical Evidence Bootcamp for Journalists 2012
Holding a mirror up to two industries We were awarded a Mirror Award – hosted by Syracuse U. Newhouse School of Communications for those “who hold a mirror to their own industry for the public’s benefit.”
Lessons to improve our reporting on evidence: An analysis of 1,800+ stories over 7 years Gary Schwitzer Publisher MIT Medical Evidence Boot Camp December 4, 2012
What others are saying about usThe editors of the journal PLoS Medicine wrote: “Schwitzer’s alarming report card of thetrouble with medical news stories is thus a wake-up call for all of us involved indisseminating health research-researchers, academic institutions, journal editors,reporters, and media organizations-to work collaboratively to improve the standards ofhealth reporting.”Susan Perry, on her MinnPost.com column referred to HealthNewsReview.org as:“indispensable to consumers & journalists”Dan Gillmor, director of the Knight Center for Digital Media Entrepreneurship at ArizonaState University’s Walter Cronkite School of Journalism and Mass Communication: “Calmand thorough analysis of health news journalism from HealthNewsReview.org”Dr. Steven Kussin, Director of The Shared Decision Center, wrote: “All doctors shouldsuggest patients register at HealthNewsReview.org.”Science journalist Paul Raeburn on the Knight Science Journalism Tracker:“I’m beginning to think that Schwitzer’s criteria for judging stories ought to be printed onwallet cards for reporters, like Miranda warnings, to remind them what questions to ask. Icould use one of those myself.”
Our criteria: Does the story explain…• What’s the total cost?• How often do benefits occur?• How often do harms occur?• How strong is the evidence?• Is the condition exaggerated?• Is this really a new approach?• Is it available?• Are there alternative choices?• Who’s promoting this?• Do they have a financial conflict of interest?
After 1,800 stories over 6+ years~70% of stories fail to: ✔ Discuss costs ✔Quantify potential benefits ✔ Quantify potential harms ✔ Evaluate the quality of the evidence
In an election year, when we should have spent more time helping citizens weigh why we spend 17% of GDP on health care, yet leave 17% of our neighbors uninsured….
…instead, news often painted a kid-in-candy-store pictureof U.S. health care interventions Terrific Risk-free Without a price tag
What CNN didn’t report…• Resignations, questions about conflicts of interest and fraud probes of the program in question.• Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.
Robert Bazell: “For the first time ever an experimental drug is showing great promise of slowing the progression of Alzheimers disease.” NBC News - October 8, 2012 FIRST TIME EVER?July 17, 2012: New Alzheimer’s Drug Slowing Progression of the Disease – CBSMarch 8, 2012: Alzheimer’s treatment in late stages of disease does slow progression(The Independent)October 20, 2011: Antiviral Drugs May Slow Alzheimers Progression (Science Daily)May 11, 2009: Can New Drug Slow Progress of Alzheimers? (ThirdAge.com)August 3, 2008: Drugs May Slow Progress Of Alzheimer’s (NPR on two drugs)5-minute Google search = 6 different approaches reported to slow progression of AD just in past 4 years
Questions you might ask:“a revised look at two trials that were declared failures” – What does that mean?How big was potential benefit? AP reported: “Taken separately, the studies on the drug - Eli Lilly & Co.s solanezumab - missed their main goals of significantly slowing the mind-robbing disease or improving activities of daily living. But pooled results found 34 percent less mental decline in mild Alzheimers patients compared to those on a fake treatment for 18 months.”How big were potential harms? Angina…brain swelling…brain bleeds
Common flaws: too much stenography – notindependent vetting of studies in journalsDeifying big names/big journals - Publication in a medical journal does not guarantee the findings are true (or even important).Not ready for prime time – journals meant for conversation among scientistsNever intended to be sources of daily news. So if you’re going to use them that way, you simply must be aware of the landscape: • retractions, research fraud, fabrication, falsification of data • unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”) • ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)
Common flaws (Examples provided later upon request)Failure to evaluate inherently weak scienceIdolatry of the surrogate – Not understanding or simply notreporting that surrogate outcomes (like tumor shrinkage) may nottranslate into clinically meaningful outcomes (longer life).Using causal language to describe observational studiesExaggerating effect size - Using relative, not absolute riskreduction numbers.Reckless extrapolation - Predicting what may happen in humans –and soon - based on very preliminary animal / lab science.Lack of awareness of conflicts of interest & other ethical issues• commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies
Misreporting of Observational Studies:Can Scientific Journals Help? (more on journals later)NBC: “The science that just might justify an American addiction…Bona fide science. A realstudy of a lot of people. …There are benefits… may decrease heart disease by as much as37 percent.”The piece ended by quoting Mark Twain:“The only way to keep your health is to eat what you don’t want, drink what you don’t like,and do what you’d rather not.”We cited another Mark Twain quote – just substitute “network TV news” for “newspaper”– imagine if Twain had lived to see today’s TV news:“If you don’t read the newspaper, you’re uninformed. If you read the newspaper, you’remis-informed.”
Two birds killed with one stone in HealthDay story1. Over-reliance on news releases2. Idolatry of the surrogate
Press release: ”In recent years, drugs that inhibit BRAF activity have rapidly halted andreversed tumor growth in about 90 percent of treated patients, but most patients’response is temporary, with tumor growth resuming in six or seven months.Investigations into how this resistance emerges have suggested that the MAPK pathwaygets turned back on through activation of MEK, another protein further down thepathway.”Story: “Drugs that inhibit BRAF activity can rapidly stop and reverse tumor growth inabout 90 percent of patients. But the response is temporary in most cases, and tumorgrowth resumes in six or seven months, the researchers explained. Previous researchsuggested that this drug resistance develops because the MAPK pathway gets turnedback on through activation of MEK, another protein that is part off the MAPK pathway.”---------------------------------------------Quality of the evidence?The story dutifully reports on the primary outcome of the study, progression-freesurvival, but doesn’t explain that this may not translate to an overall increase in survivaltime for patients taking the drug combination.
Our review:The story fails to point out the intermediate endpoints that look at risk factors for heartattack and stroke (such as cholesterol, and signs of thickening or calcium deposits inarteries) are not what we care about, which is whether heart attacks and strokes wereany more or less common. Since it takes longer to detect such events, the researcherschose to look at markers for future disease rather than cardiovascular events. That’s alegitimate way to explore a hypothesis, but must be emphasized as preliminary. Don’tforget that we got into the widespread but misguided practice of prescribing estrogenfor cardiovascular prevention back in the 80s based on just such assumptions, based ondata that showed HT had favorable effects on cholesterol and other markers–yet in theend it caused more heart attacks and strokes.
“The numbers don’t lie….or explainanything about what they mean.”
I am not anti-screening• I do oppose imbalanced, incomplete messages on screening tests• I do oppose messages that emphasize benefits and minimize or ignore harms• I do oppose messages that only tell anecdotes of those who claim their life was saved by screening – something that can never be proven.
A physician wrote to me after seeing this:"Could a political reporter say Vote for Obama?!"
Crusading advocacy on CBS News“Well, we’ve had a conclusion for many, many years at Susan G. Komen,almost a generation. Screening saves lives. The 5-year survival rates for breastcancer diagnosed early is 98 percent…and this is largely due to screening andearly diagnosis.” Left CBS, left “journalism,” just joined ABC
Can you imagine a politicalreporter wearing a button saying, “Vote for Romney” ?
Crossing line from journalism to advocacy "For you women out there and for the men who love you, screening matters. Do it. This disease can be completely curable if you find it at the right time."Katherine O’Brien has metastatic breastcancer – on her ihatebreastcancer blog:“Early detection is not a cure. ‘Completelycurable’ is a like a fat man wearing a hockeyjersey. It covers a lot of ground.”
This is what he wrote: The question was simple enough. "Do you want to be scanned?" asked the hospital PR woman. Um, sure?(More on Minneapolis Star Tribune later)
“All screening programmes do harm;some do good as well.” - Dr. Muir GraySome screening stories we should hear more about….
Track this yourself:• When you disagree with evidence, use politics to attack the process• There has been public ridicule of fact that a woman pediatrician heads USPSTF.• Since when does it require subspecialty in urology or radiology to evaluate evidence?
Leaders of ZERO: The Project to End Prostate CancerWhen the US Preventive Services Task Force recommended against routine screening ofmen with the PSA blood test, CEO Skip Lockwood said the UPSTF’s advice “condemnstens of thousands of men to die this year and every year going forward…”COO Jamie Bearse wrote about the chief medical officer of the American Cancer Society: “Otis Brawley has killed more men by giving them an excuse to not be tested.”Track this yourselves: Why is it that evidence-based, shareddecision-making advocates call for choice but screen-everybodyadvocates promote mandate not choice?
Brawley on offensive:“Many of these free screenings are designed more to getpatients for hospitals and clinics and doctors than they are tobenefit the patients. Thats a huge ethical issue that needs to beaddressed.Were not against prostate cancer screening. Were against a manbeing duped and deceived into getting prostate cancerscreening."
Duped & Deceived?Tim Glynn, lawyer, age 47when doctor “decided I’d havea PSA test without consultingme.”“Men should be aware of the truly Profiled in Shannon Brownlee’sterrible consequences. As a NYT Sunday mag piece,screening tool, you could do as “Can Cancer Ever Be Ignored?”well by throwing dice on a table.”
DECISIONS Study: PSA Decision• 70% of men reported a discussion before PSA test• MD generally initiated discussion (65%)• MD generally recommended screening (73%)• “Pros” discussed some/a lot in 71% of discussions, “cons” not discussed at all in 68%• 55% reported being asked their opinion• 48% couldn’t answer any of three knowledge questions Hoffman, et al. Arch Intern Med 2009; 169:1611
The marketing of screeningIn 2010 after the National Lung Screening Trialresults were released, the ACS posted on its blog:"Its only been a few days since researchersreleased preliminary results...(which) althoughenormously promising…not enough to call forroutine use of this screening test, even in heavysmokers. ...But our greatest fear was that forceswith an economic interest in the test wouldsidestep the scientific process and use therelease of the data to start promoting CT scans.Frankly, even we are surprised how quickly thathas happened."
TheNNT.com: number needed to treat, to screen, to harm
• Began with virtual colonoscopy –after routine physical.• But more than just colon is captured & CT showed something on kidney, liver, and lungs.• Kidney and liver issues benign after liver biopsy, PET scan and more CT scans.But lung questions led to major lung surgery, sawing him open through ribs“I awoke in the recovery room after 5 hours, with a chest tube, a Foley catheter, asubclavian central venous catheter, a nasal oxygen catheter, an epidural catheter, an arterialcatheter, subcutaneously administered heparin, a constant infusion of prophylacticantibiotics, and patient-controlled analgesia with intravenously administerednarcotics…..Excruciating pain.”• No malignant disease – all “incidentalomas”• Total cost > $50,000• All precipitated by a screening test
Sharon Begley in Newsweek:Dr. Stephen Smith, Professor emeritus offamily medicine at Brown UniversitySchool of Medicine, tells his physician notto order a PSA blood test for prostatecancer or an annual electrocardiogram toscreen for heart irregularities, sinceneither test has been shown to save lives.Rather, both tests frequently findinnocuous quirks that can lead to a Dr. Rita Redberg, professor of medicine atdangerous odyssey of tests and the University of California, San Francisco,procedures. and editor of the prestigious Archives of Internal Medicine, has no intention of having a screening mammogram even though her 50th birthday has come and gone. That’s the age at which women are advised to get one. But, says Redberg, they detect too many false positives (suspicious spots that turn out, upon biopsy, to be nothing) and tumors that might regress on their own, and there is little if any evidence that they save lives.
• Reported on woman with DCIS – ductal carcinoma in situ• Reported on her choice to pursue active surveillance rather than immediate aggressive intervention such as bilateral prophylactic mastectomy• Reported on the shared decision-making program at UCSF that helped her understand the tradeoffs.THAT IS GOOD JOURNALISM
Home-cooking on health biz stories • Minneapolis Star Tribune lags behind smaller paper, smaller market on MSP medical device co. troublesome issues • Apparent cheerleading on local Medtronic, St. Jude Medical, Boston Scientific stories. • Search “Star Tribune” on our site to see 4 story examples in a month • They’re not alone
Journalists could help people understand and deal with the clash between:• Science Intuition• Evidence Emotion• Data Anecdote• Recommendations for Decision-making by an entire population individual• What we can prove What we believe, wish, or hope• Grasping uncertainty and Promoting false certainty helping people apply where it does not exist critical thinking to decision-making issues
• Surrogate markers may not tell the whole story• Does The Language Fit The Evidence? – Association Versus Causation• Resources for Reporting on Costs of Medical Interventions• “Off-label” Drug Use and Marketing• 7 Words (and more) You Shouldn’t Use in Medical News• Problems with Reporting on News from Scientific Meetings• Absolute vs. Relative Risk• Number Needed to Treat• Commercialism• Single Source Stories• FDA Approval Not Guaranteed• Phases of Drug Trials• Medical Devices• Animal & Lab Studies
Read John Ioannidis to learn pitfalls of a steady diet of journal stories PLoS Med 2005; 2(8): e124
Journals complicit in miscommunication • Editors of the HEART Group journals last week stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.” – J AM COLL CARDIOL, Vol. 60, No. 23, 2012 • “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies” – Am J Obstet Gynecol, published online 11/29/12 • “Spin and Boasting in Research Articles.” - Commentary in Arch Pediatr Adolesc Med: [published online October 2012]
“I honestly believe it is better to knownothing than to know what ain’t so.” JOSH BILLINGS (PEN NAME OF HUMORIST HENRY WHEELER SHAW, 1818 – 1885)
Ask yourselves• If your beat includes covering studies/clinical news, what is mix of stories just about “new stuff”?• Do you think you might be reporting too much of this?• If so, have you spoken with editors about this? Do you want our help?• If you don’t think you report too much of this, how would you stack up on our 10 criteria?• If you don’t accept or agree with our 10 criteria, what criteria do you use?