How Do We Know What We Know?


Published on

My keynote at the University of Texas-Pan American PACE Ethics Conference, April 5, 2013

Published in: News & Politics
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How Do We Know What We Know?

  1. 1. How Do We Know What We Know? UTPABioethics: Creating and Challenging Knowledge in Health Edinburg, Texas, April 2013 Ivan Oransky, MD Executive Editor, Reuters Health Co-Founder, Retraction Watch @ivanoransky
  2. 2. Retractions on the Rise -The Wall Street Journal
  3. 3. How Often Are Studies Wrong?
  4. 4. “Winner Takes All” Incentives Scientific American, August 2012
  5. 5. “Winner Takes All” Incentives“The winner-take-all aspect of the priority rulehas its drawbacks, however. It can encouragesecrecy, sloppy practices, dishonesty and anexcessive emphasis on surrogate measures ofscientific quality, such as publication in high-impact journals.” -- Fang and Casadevall, Scientific American
  6. 6. Anonymous Whistleblowers Step Up
  7. 7. Blogs Get Aggressive
  8. 8. Blogs Get Aggressive
  9. 9. Blogs Get Aggressive
  10. 10. Blogs Get Aggressive
  11. 11. Journals are Listening
  12. 12. Retraction Watch
  13. 13. Post-Publication Peer Review Nature (22 Dec 2011) doi:10.1038/480449a
  14. 14. Post-Publication Peer Review
  15. 15. Post-Publication Peer Review
  16. 16. Post-Publication Peer Review
  17. 17. Post-Publication Peer Review
  18. 18. Alt Metrics
  19. 19. Alt Metrics
  20. 20. How Often Are Medical Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  21. 21. How Often Are Medical Studies Wrong?
  22. 22. Does The Literature Reflect Reality?
  23. 23. Does The Literature Reflect Reality? Publish a trial that will bring US$100,000 of profit or meet the end-of-year budget by firing an editor. -- Former BMJ editor Richard Smith
  24. 24. Positive Publication Bias
  25. 25. Positive Publication Bias“The overall frequency of positive supports has grown by over 22% between 1990 and 2007, with significant differences between disciplines and countries.”“…the strongest increase in positive results was observed in disciplines—like Clinical Medicine, Pharmacology & Toxicology, Molecular Biology” Fanelli, Scientometrics 2012.
  26. 26. Publish All Data?
  27. 27. Publish All Data?
  28. 28. FDA: “Black-or-White Approval”
  29. 29. FDA: “Black-or-White Approval”“…abandon the current black-or-white approvalprocess in favor of an incremental, conditional one.In such a process, drugs could be provisionallyapproved after promising early-stage data, with theFDA retaining the option to revoke that approvallater on, should unexpected data come to light.”“A ‘conditional approval’ approach would grantlimited marketing authorization to new drugs aftersuccessful Phase II trials.”
  30. 30. Confirmation Biases“Facts do not accumulate on the blank slates ofresearchers minds and data simply do not speak forthemselves. Good science inevitably embodies atension between the empiricism of concrete dataand the rationalism of deeply held convictions.”“…awareness of the systematic errors that can occurin evaluative processes may facilitate the selfregulating forces of science and help producereliable knowledge sooner rather than later.” -- Kaptchuk, BMJ, 2003;326:1453–5
  31. 31. Believers vs. Snails -- Kaptchuk, BMJ, 2003;326:1453–5
  32. 32. Contact/Acknowledgements @ivanoranskyThanks to Nancy Lapid, Reuters Health