This document discusses a breakout group at the CERN Workshop on Innovations in Scholarly Communication focused on altmetrics. The group aims to make recommendations to improve acceptance of altmetrics. Key issues to be discussed include methodology, data quality, stability, and reliability of altmetrics. Under each issue, questions are posed about how altmetrics measure various dimensions like quality and relevance, how to ensure consistent and verifiable data, and what infrastructure is needed to provide a stable altmetrics service.

1. BG3 - Altmetrics
CERN Workshop on Innovations in Scholarly Communication (OAI8)
Daniel Beucke | beucke@sub.uni-goettingen.de
Ulrich Herb | u.herb@sulb.uni-saarland.de

2. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Objectives: from the Breakout Group’s Announcement
Possible outcomes:
Recommendations how to improve the acceptance of AltMetrics.
The session will focus on issues as methodology, data quality, stability
and reliability, tbc. …
Of course other determinants may be discussed too…
2

3. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Determinants: Methodology
“Methodology
is the systematic, theoretical analysis of the methods applied to a field of
study, or the theoretical analysis of the body of methods and principles
associated with a branch of knowledge.”
http://en.wikipedia.org/wiki/Methodology
3

4. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Methodology: Relevance for AltMetrics
• What is the theory behind AltMetrics?
• What is the meaning of the scores?
• How do the scores relate to dimensions as quality, relevance,
popularity, prominence ….?
Or is it sufficient:
• to collect numbers? … just as Citation Metrics do
• to analyze the data automatically? … in a data driven science
approach
4

5. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Determinants: Data Quality
“Data Quality
• The state of completeness, validity, consistency, timeliness and
accuracy that makes data appropriate for a specific use.”
http://en.wikipedia.org/wiki/Data_quality
5

6. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Data Quality: Relevance for AltMetrics
• How to detect fraud or gaming?
• How to identify and handle human and automated Social Media
Mentions / Events (e.g. via twitterfeed)?
• How to identify documents properly (via DOI, URL)?
• How to handle documents that are available under more than one
locator (DOI + URL, static HTML-URL + dynamic PHP-generated URL)
on a publication platform?
• How to handle documents that are available on more than one
publication platform (journal + repositories)?
6

7. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Determinants: Stability
Stability
Stable “means unchanging, permanent, firmly fixed or established.”
http://en.wikipedia.org/wiki/Stable_(disambiguation)
7

8. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Data Quality: Stability
• What infrastructure is needed to offer a stable AltMetrics service?
• Who (or what sort of institution/ agent) might/ should offer such
services? Publishers, Database Producers (e.g. Thomson Scientific),
Academics, Libraries, Infrastructure Organisations ..?
• How to finance such a service?
8

9. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Determinants
Reliability
“The quality of a measurement indicating the degree to which the
measure is consistent, that is, repeated measurements would give the
same result.”
http://en.wiktionary.org/wiki/reliability
9

10. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Reliability: Relevance for AltMetrics
• How to keep the data consistent?
• How to assure the data curation?
• Should AltMetrics Services offer the raw data they use in order to
allow re-analysis and verification (or falsification) of the scores?
• Should AltMetrics Services store the Social Media / Usage Events
(Tweets, Downloads, CiteULike Bookmarks, …) or should they
calculate the scores on the fly?
10

11. CERN Workshop on Innovations in Scholarly Communication (OAI8)
Discussion
What other determinants should be discussed?
http://bit.ly/11p1KVt
11