This document discusses metrics for evaluating scientific literature and research impact. It begins by describing traditional citation-based metrics like the impact factor. It then introduces alternative metrics (altmetrics) that measure impact through social media and online mentions. The document notes promises of altmetrics like assessing impact more quickly. It also outlines criticisms of metrics, including that citations take long to accumulate and most only consider articles, not other research outputs. It calls for more transparency and validation in developing metrics.

1. 1
b
b
www.know-center.at
Altmetrics
Open Science – Technologie und Wissenschaft
Wien, 13. Jänner
Dr. Peter Kraker

2. 2
AN EXPONENTIALLY
EXPANDING UNIVERSE
Part 1

3. 3
The Hitchhiker‘s Guide to the Galaxy
3Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

4. 4
How to Stay on Top of the Universe
4Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

5. 5
How to Stay on Top of Scientific Literature
5Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

6. 6
6Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
“One of the diseases of this age is the
multiplicity of books;
they doth so overcharge the world that it is not
able to digest the abundance of idle matter that
is every day hatched and brought forth into the
world.“
Attributed to Barnaby Rich in 1613

7. 7
7Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Price (1963)

8. 8
SCIENTOMETRICS:
THE QUANTITATIVE
ANALYSIS OF SCIENCE
Part 2

9. 9
Eugene Garfield: Pathways through Science
9Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Science Citation Index (Garfield 1955)
 Web of Science/Web of Knowledge
An index of incoming citations
Outgoing citations
Incoming citations

10. 10
10
Garfield et al.
(1964)
Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

11. 11
11Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

12. 12
12Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Van Eck and Waltman
(2010)

13. 13
Evaluative Scientometrics: Impact Factor
13Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
A measure to quantify the relative importance of a scientific
journal
The average number of citations in a given year y to papers of a
journal in the years y-1 and y-2
𝐼𝐹𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋, 2015 =
𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 2015 𝑡𝑜
𝑎𝑟𝑡𝑖𝑐𝑙𝑒𝑠 𝑝𝑢𝑏𝑙𝑖𝑠ℎ𝑒𝑑 𝑏𝑦
𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋 𝑖𝑛
2013 𝑎𝑛𝑑 2014
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑟𝑡𝑖𝑐𝑙𝑒𝑠
𝑝𝑢𝑏𝑙𝑖𝑠ℎ𝑒𝑑 𝑏𝑦 𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋
𝑖𝑛 2013 𝑎𝑛𝑑 2014

14. 14
14Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Source: Thomson Reuters

15. 15
15Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Journal X
Paper 1 Citations 15
Paper 2 Citations 17
Paper 3 Citations 14
Paper 4 Citations 18
Paper 5 Citations 15
Paper 6 Citations 15
Impact Factor 15.5
Median 15 2
Journal Y
100
2
1
2
1
2
18.0
Rank 2 1

16. 16
Criticisms of the Impact Factor
16Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Source: Amin & Mabe (2000)

17. 17
Antidotes
17Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
„Blockbuster“ papers can skew the IF
Use robust statistics (e.g. the median)
Report variation in the data (CWTS Journal Indicators)
Publication and citation behavior varies wildly between fields
Source-normalised impact factor (SNIP)
The IF window is too short
3- and 5-year Impact Factors

18. 18
Criticisms of citation-based metrics
18Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Citations take very long to appear in meaningful quantities
Citation-based metrics are dependent on the corpus that is used
for calculation
Citations are mostly assigned to articles, leaving other products
of research out of the picture (data, code etc.)
Citation-based metrics are often intransparent and irreproducible

19. 19
ALTMETRICS:
A NEW HOPE
Part 3

20. 20
Setting the Stage for Alternative Metrics
20Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Open Science and the publication of more research products
Increased use of online services in the scientific community
Source: figshare

21. 21
Altmetrics
21Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Altmetrics: metrics based on data generated in online systems
Seeing academic literature through the eyes of the readers (Rowlands
& Nicholas 2007)
Usage data (downloads, views, readership)
Links, likes and shares
Blogs and comments
Promises of altmetrics
Assess publications quicker and on a broader scale
Consider all outputs of research, not just papers
The altmetrics manifesto: http://altmetrics.org

22. 22
22Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://zenodo.org/record/34079

23. 23
23Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://www.altmetric.com/details/4822452

24. 24
24Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Source:
https://impactstory.org/Carl
Boettiger

25. 25
25Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

26. 26
26Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://openknowledgemaps.org
Kraker et al.
(2014)

27. 27
27Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://stellar.know-center.tugraz.at/umap
Evolution of a Knowledge Domain
Kraker et al.
(2014)

28. 28
The Emerging Open Digital Science Ecosystem
Bibliographic data – bibliometric data – full texts – open source software

29. 29
CAVEATS
Part 4

30. 30
30Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

31. 31
31Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics

32. 32
32Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Kraker & Lex
(2014)

33. 33
The Leiden Manifesto: Ten Principles for Research
Metrics (Hicks et al. 2015)
1) Quantitative evaluation should support qualitative, expert
assessment
…
4) Keep data collection and analytical processes open,
transparent and simple
5) Allow those evaluated to verify data and analysis
…
9) Recognize the systemic effects of assessment and indicators
10) Scrutinize indicators regularly and update them

34. 34
Further Reading
Blog: http://blog.wikimedia.de/2015/09/15/freies-wissen-und-
wissenschaft-teil-4-ueber-die-notwendigkeit-fuer-mehr-offenheit-
und-transparenz-in-der-qualitaetssicherung-und-evaluierung-
open-peer-review-und-open-metrics/
Publication: Kraker, P., & Lex, E. (2015). A Critical Look at the
ResearchGate Score as a Measure of Scientific Reputation.
In ASCW’15 Workshop at Web Science 2015.
doi:10.5281/zenodo.35401
Source Code: https://github.com/pkraker/Headstart

35. 35
b
Thank you for your attention!
Peter Kraker
pkraker@know-center.at
Twitter: @PeterKraker