This document summarizes a presentation by Ivan Oransky on how we know what we know in health sciences and challenges to creating knowledge. It notes that retractions are on the rise, studies are often wrong, and "winner-takes-all" incentives can encourage dishonest practices. It discusses how anonymous whistleblowers, aggressive blogs, journals listening to post-publication peer review, and altmetrics are helping address these issues. The presentation cautions about positive publication bias and confirmation biases affecting knowledge, and calls for publishing all data and more conditional drug approvals to improve reliability of knowledge over time.

1. How Do We Know
What We Know?
UTPA
Bioethics: Creating and Challenging Knowledge in Health
Edinburg, Texas, April 2013
Ivan Oransky, MD
Executive Editor, Reuters Health
Co-Founder, Retraction Watch
@ivanoransky

2. Retractions on the Rise
-The Wall Street Journal

3. How Often Are Studies Wrong?

4. “Winner Takes All” Incentives
Scientific American, August 2012

5. “Winner Takes All” Incentives
“The winner-take-all aspect of the priority rule
has its drawbacks, however. It can encourage
secrecy, sloppy practices, dishonesty and an
excessive emphasis on surrogate measures of
scientific quality, such as publication in high-
impact journals.”
-- Fang and Casadevall, Scientific American

6. Anonymous Whistleblowers Step Up
http://www.labtimes.org

7. Blogs Get Aggressive
http://abnormalscienceblog.wordpress.com/

8. Blogs Get Aggressive

9. Blogs Get Aggressive
http://md-anderson-cc.blogspot.com

10. Blogs Get Aggressive
http://www.science-fraud.org/

11. Journals are Listening

12. Retraction Watch

13. Post-Publication Peer Review
Nature (22 Dec 2011) doi:10.1038/480449a

14. Post-Publication Peer Review
http://crossref.org/

15. Post-Publication Peer Review

16. Post-Publication Peer Review
http://precedings.nature.com/

17. Post-Publication Peer Review

18. Alt Metrics
http://total-impact.org/

19. Alt Metrics
http://sciencecard.org/

20. How Often Are Medical Studies Wrong?
Ioannidis JPA. PLoS Med 2005; 2(8): e124

21. How Often Are Medical Studies Wrong?

22. Does The Literature Reflect Reality?

23. Does The Literature Reflect Reality?
Publish a trial that will bring US$100,000
of profit or meet the end-of-year budget
by firing an editor.
-- Former BMJ editor Richard Smith

24. Positive Publication Bias

25. Positive Publication Bias
“The overall frequency of positive supports
has grown by over 22% between 1990 and
2007, with significant differences between
disciplines and countries.”
“…the strongest increase in positive results
was observed in disciplines—like Clinical
Medicine, Pharmacology & Toxicology,
Molecular Biology”
Fanelli, Scientometrics 2012.

26. Publish All Data?

27. Publish All Data?
www.AllTrials.net

28. FDA: “Black-or-White Approval”

29. FDA: “Black-or-White Approval”
“…abandon the current black-or-white approval
process in favor of an incremental, conditional one.
In such a process, drugs could be provisionally
approved after promising early-stage data, with the
FDA retaining the option to revoke that approval
later on, should unexpected data come to light.”
“A ‘conditional approval’ approach would grant
limited marketing authorization to new drugs after
successful Phase II trials.”
http://www.manhattan-institute.org/

30. Confirmation Biases
“Facts do not accumulate on the blank slates of
researchers' minds and data simply do not speak for
themselves. Good science inevitably embodies a
tension between the empiricism of concrete data
and the rationalism of deeply held convictions.”
“…awareness of the systematic errors that can occur
in evaluative processes may facilitate the self
regulating forces of science and help produce
reliable knowledge sooner rather than later.”
-- Kaptchuk, BMJ, 2003;326:1453–5

31. Believers vs. Snails
-- Kaptchuk, BMJ, 2003;326:1453–5

32. Contact/Acknowledgements
ivan-oransky@erols.com
http://retractionwatch.com
@ivanoransky
Thanks to Nancy Lapid, Reuters Health