Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Practical applications of altmetrics

206 views

Published on

Seminar presented at the European Summer School for Scientometrics (ESSS 2018) held at Vienna on july 28, 2018

Published in: Education
  • Be the first to comment

  • Be the first to like this

Practical applications of altmetrics

  1. 1. Practical applications of altmetrics Nicolas Robinson-Garcia
  2. 2. Objectives ➢ Analyze differences by altmetric data provider and consequences from using one or the other ➢ Discuss critically which information of interest can be obtained from the different altmetrics ➢ Develop creative solutions to respond to current demands making use of altmetric data
  3. 3. Agenda I. The altmetric toolbox ➢ Indicators ➢ Data collection and processing II. Design of altmetric solutions ➢ Demands that can/not be addressed ➢ Interpreting altmetrics III. Altmetrics in practice ➢ Monitoring attention ➢ Supporting case studies
  4. 4. The altmetric toolbox WHICH ALTMETRIC PROVIDER SHOULD WE USE? • Altmetric providers cover a wide and heterogenenous set of indicators • This set of indicators varies by altmetric provider • Altmetric providers offer differing results for same indicators • These differences are due to the data collection process
  5. 5. The altmetric toolbox “Altmetrics is the study and use of scholarly impact measures based on activity in online tools and environments” Priem, Groth & Taraborelli, 2012
  6. 6. The altmetric toolbox
  7. 7. Citations News media Policy documents Open peer review (F1000, Publons) YouTube Open Syllabus Patents … The altmetric toolbox
  8. 8. Downloads-Usage Citations Library holdings Video plays Code forks News media Amazon reviews Goodreads … Citations News media Policy documents Open peer review (F1000, Publons) YouTube Open Syllabus Patents … The altmetric toolbox
  9. 9. The altmetric toolbox TYPES OF ALTMETRIC PROVIDERS • ALL IN ONE − Almetric.com ← SPRINGER NATURE − PlumX Analytics ← ELSEVIER − CrossRef Event Data − Lagotto ← PLOS • BY DATA SOURCE − Mendeley ← ELSEVIER − Twitter − Facebook − ResearchGate
  10. 10. The altmetric toolbox Social media • Twitter, Facebook, Google Plus… Tweets, retweets, likes, users Reference managers • Mendeley, Zotero, ReadCube… Bookmarks, collections, readers Research blogging • Publishers, institutional, personal… Mentions, comments, links OA content • PLOS journals, DataCite, Github… Downloads, comments, views SOURCES INDICATORS
  11. 11. ALTMETRIC.COM SOURCES AND INDICATORS • Public policy documents • Blogs • Mainstream media • Citations (web of science / scopus) • Online reference managers (mendeley) • Research highlights (F1000) • Post-publication peer-review platforms (pubpeer, publons) • Social Media (Facebook public pages, Twitter, Google+, LinkedIn, Sina, Weibo and Pinterest) • Open Syllabus Project • Multimedia y otros (YouTube, Reddit y Q&A)
  12. 12. PLUMX ANALYTICS SOURCES AND INDICATORS
  13. 13. LAGOTTO SOURCES AND INDICATORS Viewed • PubMed Central Usage Stats • Copernicus Discussed • Facebook, Twitter • Blogs • Wikipedia • Reddit • OpenEdition Saved • CiteULike • Mendeley Cited • CrossRef, PubMed • Scopus • WoS • DataCite • Europe PMC Recommended • F1000 Prime PLOS only • Usage • Figshare usage • Comments
  14. 14. CROSSREF EVENT DATA SOURCES AND INDICATORS
  15. 15. The altmetric toolbox Zahedi & Costas, 2018
  16. 16. The altmetric toolbox REASONS BEHIND THESE DIFFERENCES • Data collection. Use of different APIs to access primary sources • Aggregation and reporting. Differing strategies to identify mentions and aggregating identifiers (i.e., DOIs, URLs) • Data update. Update of data daily/weekly… ALTMETRIC INDICATORS ARE UNSTABLE!! Zahedi & Costas, 2018
  17. 17. The altmetric toolbox Three tweets linking to three different versions of the same publication
  18. 18. The altmetric toolbox Different versions of a publication merged by Altmetric.com
  19. 19. Design of altmetric solutions SOME CONSIDERATIONS 1. Goal 2. Altmetric sources ≈ Phenomenon analyzed 3. Selection of indicators 4. Selection of approach
  20. 20. Design of altmetric solutions MONITORING • The goal is to learn what is going on • Use of metrics should not ignore uncertainty • No direct consequences of its results • Superficial and exploratory EVALUATING • The goal is to assess if an expected behavior/results is taking place • Altmetrics used as support to other approaches • Useful when combining with qualitative analysis in case studies
  21. 21. Design of altmetric solutions INTERPRETING ALTMETRICS CASE I. PROXY AND INDICATOR ARE NOT ALIGNED Twitter mentions as a proxy for engagement with or consumption of scientific literature Robinson-Garcia et al., 2017
  22. 22. Design of altmetric solutions INTERPRETING ALTMETRICS CASE II. UNREPORTED CHANGES IN ALTMETRIC PROVIDERS Torres-Salinas, Robinson-Garcia & Gorraiz, Forthcoming
  23. 23. Design of altmetric solutions INTERPRETING ALTMETRICS CASE III. PROXY ALIGNED BUT FOR ALL THE WRONG REASONS Costas @ ESSS Granada 2016
  24. 24. Design of altmetric solutions INTERPRETING ALTMETRICS CASE IV. ERRORS IN DATA RETRIEVAL
  25. 25. Design of altmetric solutions INTERPRETING ALTMETRICS CASE V. MANY FALSE NEGATIVES Paper is not linked in the tweet but news media covering the paper, therefore, this mention is not tracked
  26. 26. Design of altmetric solutions SELECTING INDICATORS Bibliometric approach Altmetric approach
  27. 27. Design of altmetric solutions SELECTING INDICATORS Academics SMEs, Non-profits, Hospital, Gov. Agencies, Associations, Large firms SocietyAltmetrics
  28. 28. Design of altmetric solutions SELECTING INDICATORS Altmetric approach Relational approach
  29. 29. Design of altmetric solutions APPROACHES • Selection of data sources -> Primary sources vs. Aggregators • Reporting -> Unidimensional vs. Relational visualizations • Interpreting -> Numbers speak vs. Contextualization
  30. 30. Altmetrics in practice MONITORING http://knowmetrics.org/
  31. 31. Altmetrics in practice MONITORING Torres-Salinas et al. 2016
  32. 32. Altmetrics in practice MONITORING Wouters, Zahedi & Costas, 2018
  33. 33. Altmetrics in practice MONITORING
  34. 34. Altmetrics in practice EVALUATING PROFILING SCHOLARS HIGH SOC | HIGH SCI D’Este, Ramos-Vielba & Robinson-Garcia, 2018 Research area (topics) Nutrition and food science (food technology, viticulture & enology, Mediterranean-style diet) Publication record 74 pubs – 46% Top 10% Highly cited Altmetric record 63 pubs in social media – 21 policy briefs | 411 news | 186 blogs Her work appears on scientific news (e.g.): - Mediterranean diet - Effects of cooking time of tomato sauce on health
  35. 35. Altmetrics in practice EVALUATING PROFILING SCHOLARS Robinson-Garcia et al., 2018 Public sector NGO Unknown Academia Private sector Politicians Social spheres of interest of researchers
  36. 36. Altmetrics in practice EVALUATING ANALYZING RECEPTION OF RESEARCH INITIATIVES
  37. 37. Altmetrics in practice EVALUATING ANALYZING RECEPTION OF RESEARCH INITIATIVES
  38. 38. Practical applications of altmetrics Nicolas Robinson-Garcia @nrobinsongarcia
  39. 39. Design of altmetric solutions OTHER EXAMPLES http://almreports.plos.org
  40. 40. References D’Este, P., Ramos-Vielba, I., Robinson-Garcia, N. Reconciling scientific impact and societal relevance: The role of pro-social motivation and productive interactions. XVI Triple Helix Conference, Sept. 5-8, Manchester, United Kingdom Priem, J., Groth, P., & Taraborelli, D. (2012). The Altmetrics Collection. PLOS ONE, 7(11), e48753. https://doi.org/10.1371/journal.pone.0048753 Robinson-Garcia, N., van Leeuwen, T. N., & Rafols, I. (2018). Using Almetrics for Contextualised Mapping of Societal Impact: From Hits to Networks. Science and Public Policy. https://doi.org/10.1093/scipol/scy024 Robinson-Garcia, N., Costas, R., Isett, K., Melkers, J., & Hicks, D. (2017). The unbearable emptiness of tweeting—About journal articles. PLOS ONE, 12(8), e0183551. https://doi.org/10.1371/journal.pone.0183551 Torres-Salinas, D., Gumpenberger, C., & Gorraiz, J. (2017). PlumX As a Potential Tool to Assess the Macroscopic Multidimensional Impact of Books. Frontiers in Research Metrics and Analytics, 2. https://doi.org/10.3389/frma.2017.00005 Torres-Salinas, D., Robinson-Garcia, N., & Gorraiz, J. (2017). Filling the citation gap: measuring the multidimensional impact of the academic book at institutional level with PlumX. Scientometrics, 113(3), 1371–1384. Torres-Salinas, D., Robinson-Garcia, N., & Jiménez-Contreras, E. (2016). Can we use altmetrics at the institutional level? A case study analysing the coverage by research areas of four Spanish universities. In STI 2016 Conference: Peripheries, frontiers and beyond. Valencia. Retrieved from http://arxiv.org/abs/1606.00232 Wouters, P., Zahedi, Z., & Costas, R. (2018). Social media metrics for new research evaluation. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Handbook of Quantitative Science and Technology Research. Springer. Retrieved from https://arxiv.org/abs/1806.10541 Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators. PLOS ONE, 13(5), e0197326. https://doi.org/10.1371/journal.pone.0197326

×