Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Björn Brembs
Universität Regensburg
http://brembs.net - @brembs
Scientists produce publications, data
and code
Scientists produce publications, data
and code
Scientists produce publications, data
and code
Dysfunctional scholarly literature
• Limited access
• Limited access
• Limited access
• No global search
• Limited access
• No global search
• No functional hyperlinks
First demonstration: 1968 WWW: 1989
Stanford Research Insti...
• Limited access
• No global search
• No functional hyperlinks
• No data visualization
• Limited access
• No global search
• No functional hyperlinks
• No data visualization
• No submission standards
• Limited access
• No global search
• No functional hyperlinks
• No data visualization
• No submission standards
• (Almost...
• Limited access
• No global search
• No functional hyperlinks
• No data visualization
• No submission standards
• (Almost...
• Limited access
• No global search
• No functional hyperlinks
• No data visualization
• No submission standards
• (Almost...
…it’s like the
web in 1995!
• Limited access
• No global search
• No functional hyperlinks
• No data visualization
• No su...
Scientific data in peril
Report on Integration of Data and Publications, ODE Report 2011
http://www.alliancepermanentaccess.org/wp-content/plugins/...
Non-existent software archives
• Institutional email
• Institutional webspace
• Institutional blog
• Library access card
• Open access repository
• Publi...
Don‘t let someone with orthogonal
interests touch your precious
http://www.sciencedirect.com/science/article/pii/S0092867415002421
http://science.sciencemag.org/content/351/6274/737.full
• 2013: US$86m of US$126m annual APA budget comes from publishing
“This was an unacceptable practice, and we regret that it took place.”
Michael Hansen, CEO Of Elsevier's Health Sciences D...
#researchparasites
Your tax dollars at work!
Modified from ARL: http://www.arl.org/bm~doc/arlstats06.pdf, http://www.arl.org/bm~doc/arlstat08.pdf
%Change
-50
0
50
100
...
(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010...
Costs[thousandUS$/article]
Legacy SciELO
(Sources: Van Noorden, R. (2013). Open access: The true cost of science publishin...
“The decision, based on market and competitor analysis, will bring Emerald’s
APC pricing in line with the wider market, ta...
Wasting billions on a parasitic industry
• Thomson Reuters: Impact Factor
• Eigenfactor (now Thomson Reuters)
• ScImago JournalRank (SJR)
• Scopus: SNIP, SJR
Sourc...
• Negotiable
• Irreproducible
• Mathematically
unsound
• PLoS Medicine, IF 2-11 (8.4)
(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291.
http://www.p...
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
• Rockefeller University Press bought their data from Thomson Reuters
• Up to 19% deviation from published records
• Secon...
• Left-skewed distributions
• Weak correlation of individual
article citation rate with journal IF
Seglen PO (1997): Why t...
https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
Is journal rank like astrology?
http://blog.frontiersin.org/2015/12/21/4782/
The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vi...
Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1...
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neur...
Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, ...
Brown, E. N., & Ramaswamy, S. (2007).
Quality of protein crystal structures. Acta
Crystallographica Section D Biological
C...
Journal Rank
Quality
Berghmans et al. (2002): doi: 10.1093/annonc/mdg203, Tressoldi et al. (2013) doi:10.1371/journal.pone...
Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
Data from: Fang, F., & Casadevall, A. (2011). RETRACTED SCIENCE AND THE RETRACTION INDEX Infection and Immunity DOI: 10.11...
Credit: Scott Edmund
“High-Impact” journals attract the
most unreliable research
Research questions:
True:
False:
Significant:
100
50
50
21
44
22
22
19
Quality & Productivity: Selecting the
sloppy scientists
Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1...
Open Science Collaboration Science 2015;349:aac4716
Open Science Collaboration Science 2015;349:aac4716
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neur...
“Do you trust scientists?”
an obscenely expensive anachronism
If we cannot even trust the scientific literature,
what information is trustworthy?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
What's wrong with our scholarly infrastructure?
Upcoming SlideShare
Loading in …5
×

What's wrong with our scholarly infrastructure?

1,732 views

Published on

First of a two-part series on the issues scientists face with their expensive, antiquated infrastructure and how to overcome these problems. First part on problems, second part (upcoming) on solutions.

Published in: Science
  • Be the first to comment

What's wrong with our scholarly infrastructure?

  1. 1. Björn Brembs Universität Regensburg http://brembs.net - @brembs
  2. 2. Scientists produce publications, data and code
  3. 3. Scientists produce publications, data and code
  4. 4. Scientists produce publications, data and code
  5. 5. Dysfunctional scholarly literature
  6. 6. • Limited access
  7. 7. • Limited access
  8. 8. • Limited access • No global search
  9. 9. • Limited access • No global search • No functional hyperlinks First demonstration: 1968 WWW: 1989 Stanford Research Institute: NLS Tim Berners-Lee: CERN
  10. 10. • Limited access • No global search • No functional hyperlinks • No data visualization
  11. 11. • Limited access • No global search • No functional hyperlinks • No data visualization • No submission standards
  12. 12. • Limited access • No global search • No functional hyperlinks • No data visualization • No submission standards • (Almost) no statistics
  13. 13. • Limited access • No global search • No functional hyperlinks • No data visualization • No submission standards • (Almost) no statistics • No text/data-mining
  14. 14. • Limited access • No global search • No functional hyperlinks • No data visualization • No submission standards • (Almost) no statistics • No text/data-mining • No effective way to sort, filter and discover
  15. 15. …it’s like the web in 1995! • Limited access • No global search • No functional hyperlinks • No data visualization • No submission standards • (Almost) no statistics • No text/data-mining • No effective way to sort, filter and discover • No scientific impact analysis • Lousy peer-review • No networking feature • Etc.
  16. 16. Scientific data in peril
  17. 17. Report on Integration of Data and Publications, ODE Report 2011 http://www.alliancepermanentaccess.org/wp-content/plugins/download-monitor/download.php?id=ODE+Report+on+Integration+of+Data+and+Publications
  18. 18. Non-existent software archives
  19. 19. • Institutional email • Institutional webspace • Institutional blog • Library access card • Open access repository • Publications? • Code? • Data?
  20. 20. Don‘t let someone with orthogonal interests touch your precious
  21. 21. http://www.sciencedirect.com/science/article/pii/S0092867415002421
  22. 22. http://science.sciencemag.org/content/351/6274/737.full
  23. 23. • 2013: US$86m of US$126m annual APA budget comes from publishing
  24. 24. “This was an unacceptable practice, and we regret that it took place.” Michael Hansen, CEO Of Elsevier's Health Sciences Division
  25. 25. #researchparasites
  26. 26. Your tax dollars at work!
  27. 27. Modified from ARL: http://www.arl.org/bm~doc/arlstats06.pdf, http://www.arl.org/bm~doc/arlstat08.pdf %Change -50 0 50 100 150 200 250 300 350 400 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 Subscription prices CPI/inflation Journals purchased
  28. 28. (Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126) Costs[thousandUS$/article] Legacy SciELO
  29. 29. Costs[thousandUS$/article] Legacy SciELO (Sources: Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature 495, 426–9; Packer, A. L. (2010). The SciELO Open Access: A Gold Way from the South. Can. J. High. Educ. 39, 111–126)
  30. 30. “The decision, based on market and competitor analysis, will bring Emerald’s APC pricing in line with the wider market, taking a mid-point position amongst its competitors.” Emerald spokesperson
  31. 31. Wasting billions on a parasitic industry
  32. 32. • Thomson Reuters: Impact Factor • Eigenfactor (now Thomson Reuters) • ScImago JournalRank (SJR) • Scopus: SNIP, SJR Source Normalized Impact per Paper
  33. 33. • Negotiable • Irreproducible • Mathematically unsound
  34. 34. • PLoS Medicine, IF 2-11 (8.4) (The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. http://www.plosmedicine.org/article/info:doi/10.1371%2Fjournal.pmed.0030291) • Current Biology IF from 7 to 11 in 2003 – Bought by Cell Press (Elsevier) in 2001…
  35. 35. https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
  36. 36. • Rockefeller University Press bought their data from Thomson Reuters • Up to 19% deviation from published records • Second dataset still not correct Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091
  37. 37. • Left-skewed distributions • Weak correlation of individual article citation rate with journal IF Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497http://www.bmj.com/cgi/content/full/314/7079/497
  38. 38. https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
  39. 39. Is journal rank like astrology?
  40. 40. http://blog.frontiersin.org/2015/12/21/4782/
  41. 41. The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328
  42. 42. Macleod MR, et al. (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. doi:10.1371/journal.pbio.1002273
  43. 43. Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
  44. 44. Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77
  45. 45. Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847
  46. 46. Journal Rank Quality Berghmans et al. (2002): doi: 10.1093/annonc/mdg203, Tressoldi et al. (2013) doi:10.1371/journal.pone.0056180, Brembs et al. (2013) doi:10.3389/fnhum.2013.00291, Fraley & Vazire (2014) doi:10.1371/journal.pone.0109019, Macleod et al. (2015) doi:10.1371/journal.pbio.1002273
  47. 47. Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
  48. 48. Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
  49. 49. Data from: Fang, F., & Casadevall, A. (2011). RETRACTED SCIENCE AND THE RETRACTION INDEX Infection and Immunity DOI: 10.1128/IAI.05661-11
  50. 50. Credit: Scott Edmund
  51. 51. “High-Impact” journals attract the most unreliable research
  52. 52. Research questions: True: False: Significant: 100 50 50 21 44 22 22 19
  53. 53. Quality & Productivity: Selecting the sloppy scientists
  54. 54. Freedman LP, Cockburn IM, Simcoe TS (2015) http://journals.plos.org/plosbiology/article?id=info:doi/10.1371/journal.pbio.1002165 61% (n=100) Open Science Collaboration
  55. 55. Open Science Collaboration Science 2015;349:aac4716
  56. 56. Open Science Collaboration Science 2015;349:aac4716
  57. 57. Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
  58. 58. “Do you trust scientists?”
  59. 59. an obscenely expensive anachronism
  60. 60. If we cannot even trust the scientific literature, what information is trustworthy?

×