Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scientific Interactions and Research Evaluation: From Bibliometrics to Altmetrics - Keynote ISI2015

2,566 views

Published on

Haustein, S. (2015). Scientific Interactions and Research Evaluation: From Bibliometrics to Altmetrics

Keynote at ISI2015 in Zadar, Croatia
http://isi2015.de/?session=keynote-c-i

Abstract. Since its creation 350 years ago, the scientific peer-reviewed journal has become the central and most important form of scholarly communication in the natural sciences and medicine. Although the digital revolution has facilitated and accelerated the publishing process by moving from print to online, it has not changed the scientific journal and scholarly communication as such. Today publications and citations in peer-reviewed journals are considered as indicators of scientific productivity and impact and used and misused in research evaluation. As scholarly communication is becoming more open and diverse and manuscripts, data, presentations and code are shared online, the altmetrics and open science movement demand the adaption of evaluation practices. Parallels are drawn between the early days of bibliometrics and current altmetrics research highlighting possibilities and limitations of various metrics and warning against adverse effects.

Published in: Data & Analytics
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Scientific Interactions and Research Evaluation: From Bibliometrics to Altmetrics - Keynote ISI2015

  1. 1. Scientific interactions and research evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan
  2. 2. Scientific Interactions Research Evaluation Opportunities & Challenges
  3. 3. Invisible Colleges Père Marin Mersenne (1588-1648) Henry Oldenburg (1619-1677) http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg
  4. 4. Invisible Colleges Père Marin Mersenne (1588-1648) Henry Oldenburg (1619-1677) http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg Information Hubs
  5. 5. Scientific Societies L’Académie royale des sciences 22 December 1666 The Royal Society 28 November 1660 http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg
  6. 6. Scientific Societies L’Académie royale des sciences 22 December 1666 The Royal Society 28 November 1660 http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg Institutionalization of Science
  7. 7. Scientific Journals Le journal des sçavans 5 January 1665 Philosophical Transactions 6 March 1665 http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
  8. 8. Scientific Journals Le Journal des sçavans 5 January 1665 Philosophical Transactions 6 March 1665 http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg Registration Certification Dissemination Archiving
  9. 9. Scientific Articles: 17th to Early 19th Century •  Experiments and descriptions of the natural world •  Avoiding “fine speaking” •  Various styles of arguing •  Qualitative and personal judgements Harmon, J.E. & Gross, A.G. (2007). The Scientific Literature. A Guided Tour. Chicago: University of Chicago Press.
  10. 10. Modern Scientific Articles •  Professionalized and highly specialized •  Increased focus on data, graphs, tables and theory •  Impersonal, technical and codified •  Style guides and gatekeeping •  Citations •  Introduction, Methods, Results and Discussion Larivière, V., Archambault, É. & Gingras, Y. (2008). Long-term variations in the aging of scientific literature: From exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2), 288-296. Sollaci, L.B. & Pereira, M.G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey. Journal of the Medical Library Association, 92(3), 364-371 Sollaci&Pereira(2004,p.365) Proportion of IMRaD adoption in medical journalsNumber of references 1900 to 2004 Larivière,Archambaul&Gingras(2008,p.293)
  11. 11. arXiv submission statistics from http://arxiv.org/stats/monthly_submissions Larivière, V., Lozano, G.A. & Gingras, Y. (2014). Are elite journals declining? Journal of the Association for Information Science and Technology, 65(4), 649-655. •  Improved access •  Acceleration •  Collaboration •  Peer review •  Distribution of preprints •  Decreasing importance of scientific journals •  Journal functions •  Diversification of publication venues •  Appearance and symbolic capital of journals unchanged Digital Revolution Submissions to arXiv Share of top 1% most cited papers Larivière,Lozano&Gingras(2014,p.652)
  12. 12. Academic Publishing Market Larivière, V., Haustein, S. & Mongeon, P. (in press). The oligopoly of academic publishers in the digital era. PLoS ONE. Larivière,Haustein&Mongeon(inpress) •  Aggravation of serials crisis •  Elsevier: €3,400 subscription price, 30% increase •  Profit margins of commercial publishers up to 40% •  Decline of scientific societies as publishers •  >50% of papers owned by five major publishers
  13. 13. Budapest Open Access Initiative Open Access “immediate, free availability on the public internet, permitting any users to read, download, copy, distribute, print, search or link to the full text of these articles” Budapest Open Access Initiative (2002) Archambaultetal.(2013,p.13) Freely available journal papers 2004 to 2011 •  Gold and Green •  Libre and Gratis •  Hybrid journals •  Elsevier: $500 to 5,000 •  Springer: $3,000 •  Wiley: $3,000 Archambault, É., Amyot, D., Deschamps, P., Nicol, A., Rebout, L. & Roberge, G. (2013). Proportion of Open Access Peer-Reviewed Papers at the European and World Levels 2004-2011. Report for the European Commission. http://www.science-metrix.com/pdf/SM_EC_OA_Availability_2004-2011.pdf
  14. 14. Open Science “opening up the research process by making all of its outcomes, and the way in which these outcomes were achieved, publicly available on the World Wide Web” Kraker et al. (2011, p. 645) Kraker, P., Leony, D., Reinhardt, W. & Beham, G. (2011). The case for an open science in technology enhanced learning. International Journal of Technology Enhanced Learning, 3(6), 643-654. •  Open Data •  Open Source •  Open Methodology •  Open Access •  Open Peer Review
  15. 15. Björneborn, L. & Ingwersen, P. (2004), Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227. Definition of Scholarly Metrics adaptedfrom:Björneborn&Ingwersen(2004,p.1217) informetrics scientometrics bibliometrics cybermetrics webometrics altmetrics
  16. 16. Otlet, P. (1934). Traité de documentation: le livre sur le livre, théorie et pratique. Pritchard, P. (1927). Statistical bibliography or bibliometrics? Journal of Documentation, 25, 348-349.. Definition of Scholarly Metrics informetrics scientometrics bibliometrics cybermetrics webometrics altmetrics “La «Bibliometrie» sera la partie définie de la Bibliologie qui s'occupe de la mesure ou quantité appliquée aux livres.” “the application of mathematics and statistical methods to books and other media of communication” Pritchard (1969, p. 348) Otlet (1934, p. 14)
  17. 17. Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press. Rousseau, R. & Ye, F. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 10–12. doi:10.1007/s11434-013-5939-3 informetrics scientometrics cybermetrics webometrics altmetrics “study and use of scholarly impact measures based on activity in online tools and environments” “a good idea but a bad name” Rousseau & Ye (2013, p. 2) Priem (2014, p. 266) bibliometrics Definition of Scholarly Metrics
  18. 18. Definition of Scholarly Metrics informetrics scientometrics bibliometrics cybermetrics webometrics altmetrics scholarly metrics Scholarly metrics are metrics based on acts and events (e.g., viewing, reading, saving, diffusing, mentioning, citing, reusing, modifying) related to scholarly documents (e.g., papers, blog posts, datasets, code, notes) or scholarly agents (e.g., researchers, universities, journals).
  19. 19. Gross, P.L.K. & Gross, E.M. (1927). College libraries and chemical education. Science, 66(1713), 385-389.. Citation analysis for objective collection management Bibliometrics for Library Mangement Gross&Gross,1927,p.386 Journals cited in the Journal of the American Chemical Society 1926
  20. 20. Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108-111. Bibliometrics for Information Retrieval Retrieval tool to handle information overload •  Institute for Scientific Information •  Science Citation Index •  Source Author Index •  Citation Index “It would not be excessive to demand that the thorough scholar check all papers that have cited or criticized such papers, if they could be located quickly. The citation index makes this check practicable.” Garfield, 1955, p. 108
  21. 21. Derek J. de Solla Price Science since Babylon (1961) Little Science – Big Science (1963) Bibliometrics for Sociology of Science Price, D. J. d. S. (1961). Science Since Babylon. New Haven / London: Yale University Press, Price, D. J. d. S. (1963). Little Science, Big Science. New York: Columbia University Press.
  22. 22. Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis, 79, 606–623. Robert K. Merton •  Social norms of science •  Communalism •  Universalism •  Disinterestedness •  Organized skepticism •  Matthew effect Bibliometrics for Sociology of Science “symbolically, [the reference] registers in the enduring archives the intellectual property of the acknowledged source by providing a pellet of peer recognition of the knowledge claim” Merton (1988, p. 621)
  23. 23. •  Performance measurement and policy instrument •  Commercialization Bibliometrics for Research Evaluation “When used properly, bibliometric indicators can provide a ‘monitoring device’ for university research-management and science policy. They enable research policy-makers to ask relevant questions of researchers on their scientific performance, in order to find explanations of the bibliometric results in terms of factors relevant to policy.” Moed et al. (1985, p. 131) Moed, H.F., Burger, W.J.M., Frankfort, J.G, van Raan, A.F.J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131-149.
  24. 24. •  Oversimplification publications = productivity citations = impact •  Uninformed use and misuse Impact Factor h-index •  Adverse effects “salami” publishing honorary authorship self-citations citation cartels Bibliometrics for Research Evaluation Hvistendahl, M. (2013). China’s publication bazaar. Science, 342(6162), 1035-1039. van Noorden, R. (2013). Brazilian citation scheme outed: Thomson Reuters suspends journals from its rankings for ‘citation stacking’, Nature, 500(7464), 510-511.
  25. 25. Criticism against current form of research evaluation Ø  Alternative forms of research ouput Ø  Alternative use and visibility of publications Altmetrics “We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem.” … Priem et al. (2010) Information overload … Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Alt-metrics: a manifesto. October. Retrieved from http://altmetrics.org/manifesto/ Piwowar, H. A. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.
  26. 26. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI) Altmetrics What do we know? %
  27. 27. Altmetrics What do we know? Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI) %
  28. 28. Altmetrics What do we know? Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI) %
  29. 29. Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. Altmetrics What do we know? Biomedical & Health Sciences Social Sciences & Humanities Life & Earth Sciences Natural Sciences & Engineering Mathematics & Computer Science Twittercoverage
  30. 30. Documenttype Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. Altmetrics What do we know? Biomedical & Health Sciences Social Sciences & Humanities Life & Earth Sciences Natural Sciences & Engineering Mathematics & Computer Science Twittercoverage
  31. 31. Altmetrics Highly tweeted
  32. 32. Altmetrics Highly tweeted
  33. 33. Altmetrics Highly tweeted
  34. 34. Altmetrics Highly tweeted
  35. 35. Altmetrics Altmetrics in the wild
  36. 36. Meaning of metrics Opportunities •  Heterogenous users •  Diverse motivations Challenges •  Understanding underlying processes •  Determining the meaning of metrics
  37. 37. Meaning of metrics Motivations and norms Saving to Mendeley Mentioning in News
  38. 38. Meaning of metrics Motivations and norms Recommending on F1000 Tweeting
  39. 39. Meaning of metrics Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V. (in press). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology. Bertin,Atanassova,Gingras&Larivière(inpress,p.13) Distribution of references along the IMRaD structure Citing in a journal article
  40. 40. Meaning of Metrics Haustein, S., Bowman, T.D. & Costas, R. (in press). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In Sugimoto , C.R. (Ed.). Theories of Informetrics: A Festschrift in Honor of Blaise Cronin. Acts leading to (online) events used for metrics Haustein,Bowman&Costas(inpress,p.13) RESEARCH OBJECT
  41. 41. Opportunities •  Detailed life cycle of scientific output •  Fine-grained indicators and adequate benchmarks Challenges •  Versions of research output •  Publication dates Time Analysis
  42. 42. Time Analysis Journal article •  Submitted manuscript •  Revised manuscript •  Accepted manuscript •  Version of Record •  Online publication •  Journal issue •  Online date •  Issue month Ø  Adjusting indicators 3 March 2014 15 July 2014 21 January 2015 February 2015
  43. 43. Time Analysis
  44. 44. Time Analysis Haustein, S., Bowman, T.D. & Costas, R. (in press). When is an article actually published? An analysis of online availability, publication, and indexation dates. Proceedings of the 15th International Conference on Scientometrics and Informetrics, Istanbul, Turkey. Tweets before publication?
  45. 45. Communities of Attention Opportunity •  Differentiating between types of use •  Measuring social impact Challenges •  Determining engagement •  Identifying users
  46. 46. Level of engagement on Twitter Communities of Attention !Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456
  47. 47. Communities of Attention Automated Twitter bots Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456 tweeting preprints from arXiv subject area tweeting topic- relevant papers selection of interesting papers
  48. 48. Communities of Attention exposure engagement median dissimilarity with paper title mediannumberoffollowers influencers / brokers orators / discussing disseminators / mumblers broadcasters tweet text differs from paper title tweet text is identical to paper title few followers many followers Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
  49. 49. Communities of Attention 708 of 125,083 users (0.6%) tweeting WoS papers published in 2012 (>100) Node size represents number of papers Network of users tweeting the same papers Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
  50. 50. Communities of Attention Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
  51. 51. Communities of Attention Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
  52. 52. Communities of Attention Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
  53. 53. Communities of Attention Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.
  54. 54. Outlook Sociological research. Scientometric studies. Avoid adverse effects.
  55. 55. Stefanie Haustein Thank you for your attention! Questions? stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan Thank you for your attention!

×