This document discusses practical applications of altmetrics data. It begins by outlining different altmetrics data providers and how they may provide differing results. It then discusses how to design altmetric solutions by selecting appropriate indicators and approaches based on goals and target audiences. Several examples are given of how altmetrics can be used for monitoring attention to research and evaluating impact through case studies and researcher profiling. Interpreting altmetrics data requires consideration of proxy alignment, changes in data collection, and errors.
2. Objectives
➢ Analyze differences by altmetric data provider and
consequences from using one or the other
➢ Discuss critically which information of interest can be
obtained from the different altmetrics
➢ Develop creative solutions to respond to current demands
making use of altmetric data
3. Agenda
I. The altmetric toolbox
➢ Indicators
➢ Data collection and processing
II. Design of altmetric solutions
➢ Demands that can/not be addressed
➢ Interpreting altmetrics
III. Altmetrics in practice
➢ Monitoring attention
➢ Supporting case studies
4. The altmetric toolbox
WHICH ALTMETRIC PROVIDER SHOULD WE USE?
• Altmetric providers cover a wide and heterogenenous set of
indicators
• This set of indicators varies by altmetric provider
• Altmetric providers offer differing results for same indicators
• These differences are due to the data collection process
5. The altmetric toolbox
“Altmetrics is the study and use of scholarly
impact measures based on activity in online
tools and environments”
Priem, Groth & Taraborelli, 2012
9. The altmetric toolbox
TYPES OF ALTMETRIC PROVIDERS
• ALL IN ONE
− Almetric.com ← SPRINGER NATURE
− PlumX Analytics ← ELSEVIER
− CrossRef Event Data
− Lagotto ← PLOS
• BY DATA SOURCE
− Mendeley ← ELSEVIER
− Twitter
− Facebook
− ResearchGate
10. The altmetric toolbox
Social media
• Twitter, Facebook, Google Plus… Tweets, retweets, likes, users
Reference managers
• Mendeley, Zotero, ReadCube… Bookmarks, collections, readers
Research blogging
• Publishers, institutional, personal… Mentions, comments, links
OA content
• PLOS journals, DataCite, Github… Downloads, comments, views
SOURCES INDICATORS
11. ALTMETRIC.COM
SOURCES AND INDICATORS
• Public policy documents
• Blogs
• Mainstream media
• Citations (web of science / scopus)
• Online reference managers (mendeley)
• Research highlights (F1000)
• Post-publication peer-review platforms (pubpeer, publons)
• Social Media (Facebook public pages, Twitter, Google+, LinkedIn, Sina,
Weibo and Pinterest)
• Open Syllabus Project
• Multimedia y otros (YouTube, Reddit y Q&A)
16. The altmetric toolbox
REASONS BEHIND THESE DIFFERENCES
• Data collection. Use of different APIs to access primary sources
• Aggregation and reporting. Differing strategies to identify mentions
and aggregating identifiers (i.e., DOIs, URLs)
• Data update. Update of data daily/weekly…
ALTMETRIC INDICATORS ARE UNSTABLE!!
Zahedi & Costas, 2018
19. Design of altmetric solutions
SOME CONSIDERATIONS
1. Goal
2. Altmetric sources ≈ Phenomenon analyzed
3. Selection of indicators
4. Selection of approach
20. Design of altmetric solutions
MONITORING
• The goal is to learn what is going
on
• Use of metrics should not ignore
uncertainty
• No direct consequences of its
results
• Superficial and exploratory
EVALUATING
• The goal is to assess if an
expected behavior/results is
taking place
• Altmetrics used as support to
other approaches
• Useful when combining with
qualitative analysis in case
studies
21. Design of altmetric solutions
INTERPRETING ALTMETRICS
CASE I. PROXY AND INDICATOR
ARE NOT ALIGNED
Twitter mentions as a proxy for
engagement with or consumption of
scientific literature
Robinson-Garcia et al., 2017
22. Design of altmetric solutions
INTERPRETING ALTMETRICS
CASE II. UNREPORTED CHANGES
IN ALTMETRIC PROVIDERS
Torres-Salinas, Robinson-Garcia & Gorraiz, Forthcoming
23. Design of altmetric solutions
INTERPRETING ALTMETRICS
CASE III. PROXY ALIGNED BUT
FOR ALL THE WRONG REASONS
Costas @ ESSS Granada 2016
24. Design of altmetric solutions
INTERPRETING ALTMETRICS
CASE IV. ERRORS IN DATA
RETRIEVAL
25. Design of altmetric solutions
INTERPRETING ALTMETRICS
CASE V. MANY FALSE NEGATIVES
Paper is not linked in the tweet
but news media covering the
paper, therefore, this mention
is not tracked
29. Design of altmetric solutions
APPROACHES
• Selection of data sources -> Primary sources vs. Aggregators
• Reporting -> Unidimensional vs. Relational visualizations
• Interpreting -> Numbers speak vs. Contextualization
34. Altmetrics in practice
EVALUATING
PROFILING SCHOLARS
HIGH SOC | HIGH SCI
D’Este, Ramos-Vielba &
Robinson-Garcia, 2018
Research area (topics) Nutrition and food science
(food technology, viticulture & enology,
Mediterranean-style diet)
Publication record 74 pubs – 46% Top 10% Highly cited
Altmetric record 63 pubs in social media – 21 policy briefs | 411
news | 186 blogs
Her work appears on scientific news (e.g.):
- Mediterranean diet
- Effects of cooking time of tomato sauce on health
35. Altmetrics in practice
EVALUATING
PROFILING SCHOLARS
Robinson-Garcia et al.,
2018
Public sector
NGO
Unknown
Academia
Private sector
Politicians
Social spheres of
interest of researchers
40. References
D’Este, P., Ramos-Vielba, I., Robinson-Garcia, N. Reconciling scientific impact and societal relevance: The role of pro-social motivation
and productive interactions. XVI Triple Helix Conference, Sept. 5-8, Manchester, United Kingdom
Priem, J., Groth, P., & Taraborelli, D. (2012). The Altmetrics Collection. PLOS ONE, 7(11), e48753.
https://doi.org/10.1371/journal.pone.0048753
Robinson-Garcia, N., van Leeuwen, T. N., & Rafols, I. (2018). Using Almetrics for Contextualised Mapping of Societal Impact: From Hits
to Networks. Science and Public Policy. https://doi.org/10.1093/scipol/scy024
Robinson-Garcia, N., Costas, R., Isett, K., Melkers, J., & Hicks, D. (2017). The unbearable emptiness of tweeting—About journal articles.
PLOS ONE, 12(8), e0183551. https://doi.org/10.1371/journal.pone.0183551
Torres-Salinas, D., Gumpenberger, C., & Gorraiz, J. (2017). PlumX As a Potential Tool to Assess the Macroscopic Multidimensional
Impact of Books. Frontiers in Research Metrics and Analytics, 2. https://doi.org/10.3389/frma.2017.00005
Torres-Salinas, D., Robinson-Garcia, N., & Gorraiz, J. (2017). Filling the citation gap: measuring the multidimensional impact of the
academic book at institutional level with PlumX. Scientometrics, 113(3), 1371–1384.
Torres-Salinas, D., Robinson-Garcia, N., & Jiménez-Contreras, E. (2016). Can we use altmetrics at the institutional level? A case study
analysing the coverage by research areas of four Spanish universities. In STI 2016 Conference: Peripheries, frontiers and beyond.
Valencia. Retrieved from http://arxiv.org/abs/1606.00232
Wouters, P., Zahedi, Z., & Costas, R. (2018). Social media metrics for new research evaluation. In W. Glänzel, H. F. Moed, U. Schmoch, &
M. Thelwall (Eds.), Handbook of Quantitative Science and Technology Research. Springer. Retrieved from
https://arxiv.org/abs/1806.10541
Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four
major altmetric data aggregators. PLOS ONE, 13(5), e0197326. https://doi.org/10.1371/journal.pone.0197326