Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Björn Brembs
Universität Regensburg - Neurogenetics
http://brembs.net - @brembs
15th century: Adoration of the lamb
21st century: Adoration of the Glam
Publikationstätigkeit
(vollständige Publikationsliste, darunter Originalarbeiten als Erstautor/in,
Seniorautor/in, Impact-...
• Negotiable
• Irreproducible
• Mathematically
unsound
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
• Rockefeller University Press bought their data from Thomson Reuters
• Up to 19% deviation from published records
• Secon...
• Left-skewed distributions
• Weak correlation of individual
article citation rate with journal IF
Seglen PO (1997): Why t...
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
Is journal rank like astrology?
Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1...
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neur...
Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, ...
Brown, E. N., & Ramaswamy, S. (2007).
Quality of protein crystal structures. Acta
Crystallographica Section D Biological
C...
DOI: 10.1186/s13059-016-1044-7
-omics studies
Cog. Neurosci & PsychDOI: 10.1101/071530
“High-Impact” journals attract the
most unreliable research
Source: Daniel Lakens DOI: 10.1177/1745691614528520
Source: Daniel Lakens DOI: 10.1177/1745691614528520
Source: Daniel Lakens DOI: 10.1177/1745691614528520
Research questions:
True:
False:
Significant:
200
100
100
40
88
44
44
37
“Publish-or-Perish” disadvantages
meticulous scientists
Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1...
Counting ‘Quality’ & Productivity
=> Selecting the sloppy scientists
an obscenely expensive anachronism
Dysfunctional scholarly literature
• Limited access
• Link-rot
• No scientific impact analysis
• Lousy peer-review
• No global search
• No functional hyperli...
Scientific data in peril
Report on Integration of Data and Publications, ODE Report 2011
http://www.alliancepermanentaccess.org/wp-content/plugins/...
Non-existent software archives
Antiquated and missing functionality
Costs[thousandUS$/article]
Legacy Modern
(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishin...
Wasting billions on a parasitic industry
The disaster that is our scholarly
infrastructure
“Pretty please be open!”
The Department of Psychology embraces the values of open science
and strives for replicable and reproducible research. For...
“The decision, based on market and competitor analysis, will bring Emerald’s
APC pricing in line with the wider market, ta...
Save time and money (and make science
open by default as an added benefit)
(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. doi:10.1038/495426a, Packer, A. L. (20...
LEGAL
The square traversal process has been the
foundation of scholarly communication for nearly
400 years!
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
A replication crisis in the making: how we reward unreliable science
Upcoming SlideShare
Loading in …5
×

A replication crisis in the making: how we reward unreliable science

591 views

Published on

Presentation at the 2016 annual meeting of the Mind and Brain College of the university of Lisbon on the infrastructural causes for the apparent replication crisis in the experimental/biomedical sciences.

Published in: Science
  • Be the first to comment

A replication crisis in the making: how we reward unreliable science

  1. 1. Björn Brembs Universität Regensburg - Neurogenetics http://brembs.net - @brembs
  2. 2. 15th century: Adoration of the lamb
  3. 3. 21st century: Adoration of the Glam
  4. 4. Publikationstätigkeit (vollständige Publikationsliste, darunter Originalarbeiten als Erstautor/in, Seniorautor/in, Impact-Punkte insgesamt und in den letzten 5 Jahren, darunter jeweils gesondert ausgewiesen als Erst- und Seniorautor/in, persönlicher Scientific Citations Index (SCI, h-Index nach Web of Science) über alle Arbeiten) Publications: Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.
  5. 5. • Negotiable • Irreproducible • Mathematically unsound
  6. 6. https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
  7. 7. • Rockefeller University Press bought their data from Thomson Reuters • Up to 19% deviation from published records • Second dataset still not correct Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091
  8. 8. • Left-skewed distributions • Weak correlation of individual article citation rate with journal IF Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497http://www.bmj.com/cgi/content/full/314/7079/497
  9. 9. https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
  10. 10. Is journal rank like astrology?
  11. 11. Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1002273
  12. 12. Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
  13. 13. Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77
  14. 14. Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847
  15. 15. DOI: 10.1186/s13059-016-1044-7 -omics studies
  16. 16. Cog. Neurosci & PsychDOI: 10.1101/071530
  17. 17. “High-Impact” journals attract the most unreliable research
  18. 18. Source: Daniel Lakens DOI: 10.1177/1745691614528520
  19. 19. Source: Daniel Lakens DOI: 10.1177/1745691614528520
  20. 20. Source: Daniel Lakens DOI: 10.1177/1745691614528520
  21. 21. Research questions: True: False: Significant: 200 100 100 40 88 44 44 37
  22. 22. “Publish-or-Perish” disadvantages meticulous scientists
  23. 23. Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002165 61% (n=100) Open Science Collaboration
  24. 24. Counting ‘Quality’ & Productivity => Selecting the sloppy scientists
  25. 25. an obscenely expensive anachronism
  26. 26. Dysfunctional scholarly literature
  27. 27. • Limited access • Link-rot • No scientific impact analysis • Lousy peer-review • No global search • No functional hyperlinks • Useless data visualization • No submission standards • (Almost) no statistics • No content-mining • No effective way to sort, filter and discover • No semantic enrichment • No networking feature • etc. …it’s like the web in 1995!
  28. 28. Scientific data in peril
  29. 29. Report on Integration of Data and Publications, ODE Report 2011 http://www.alliancepermanentaccess.org/wp-content/plugins/download-monitor/download.php?id=ODE+Report+on+Integration+of+Data+and+Publications
  30. 30. Non-existent software archives
  31. 31. Antiquated and missing functionality
  32. 32. Costs[thousandUS$/article] Legacy Modern (Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126) (SciELO Ubiquity Scholastica ScienceOpen PeerJ F1000Research Frontiers etc.)
  33. 33. Wasting billions on a parasitic industry
  34. 34. The disaster that is our scholarly infrastructure
  35. 35. “Pretty please be open!”
  36. 36. The Department of Psychology embraces the values of open science and strives for replicable and reproducible research. For this goal we support transparent research with open data, open material, and pre-registrations. Candidates are asked to describe in what way they already pursued and plan to pursue these goals. Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications. versus
  37. 37. “The decision, based on market and competitor analysis, will bring Emerald’s APC pricing in line with the wider market, taking a mid-point position amongst its competitors.” Emerald spokesperson
  38. 38. Save time and money (and make science open by default as an added benefit)
  39. 39. (Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. doi:10.1038/495426a, Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126) Potentialforinnovation:9.8bp.a. Costs[thousandUS$/article] Legacy SciELO
  40. 40. LEGAL
  41. 41. The square traversal process has been the foundation of scholarly communication for nearly 400 years!

×