Background image: http://commons.wikimedia.org/wiki/File:Water_drop_impact_on_a_water-surface_-_(5).jpg
Altmetrics
Measuring the impact of scientific activities
Kim Holmberg
Research Unit for the Sociology of Education
University of Turku
(e) kim.j.holmberg@utu.fi
(w3) http://kimholmberg.fi
@kholmber
Oulu,
8.5.2015

Bibliometrics
the study of quantitative aspects of
the production, dissemination, and
use of recorded information
Tague-Sufcliffe, 1992

0
1000
2000
3000
4000
5000
6000
7000
8000
9000
201420132012201120102009200820072006200520042003200220012000199919981997199619951994
The increase of publications about bibliometrics as indexed by Web of Science
between 1994 and 2014 (data retrieved from Web of Science on 9 February, 2015).

Goodhart’s Law:
When a measure becomes
the target, it ceases to be
a good measure
Goodhart, 1975

Outreach and Impact Leader, - Ref:1455401
“ We are seeking to recruit … to support, develop
and promote outreach and impact activities …
Our aim is to maximize the widest possible
impact of the research programme in whichever
way is most appropriate …
The salary range will be £41,430 - £48,873 per
annum...”
Taking impact seriously

Problems with
citation-based indicators

Problem 1:
Time

0 2 4 6 8 10 12 14 16 18 20
Impact factor window
Immediacy index window
Cited half-life
50% of
citations
50% of
citations
Time after publication (Years)
Citations

0 2 4 6 8 10 12 14 16 18 20
Time after publication (Years)
Citations
Time of research assessment
Option 1: Journal impact factor, Julkaisufoorumi, …
Potential
future
impact

0 2 4 6 8 10 12 14 16 18 20
Time after publication (Years)
Citations
Time of research assessment
Option 2: Citations (after n years)
Past
success and
impact

0 2 4 6 8 10 12 14 16 18 20
Time after publication (Years)
Citations How can we measure
impact in realtime?

Problem 2:
Journals

1
vs.

Fenner, M. & Lin, J. (2013). Altmetrics in Evolution: Defining & Redefining the Ontology of Article-Level Metrics.
http://dx.doi.org/10.3789/isqv25no2.2013.04
2

Problem 3:
Impact

Science
IMPACT ON:

Science
Society
Environment
Culture
Economy
Education
IMPACT ON:

SOLUTION?

Altmetrics
the study and use of scholarly
impact measures based on activity
in online tools and environments
Priem, J. (2014). Altmetrics. In Cronin, B. & Sugimoto, C.R.
(Eds.). Beyond Bibliometrics: Harnessing Multidimensional
Indicators of Scholarly Impact. MIT Press, 2014.

Created by
researchers
Created by
the public
Altmetrics

Created by
researchers
Indicating
future
scientific
impact?
Altmetrics

Indicating
other types of
impact, such
as societal
impact?
Created by
the public
Altmetrics

Researchers
The public

Rank Title Altmetric score News Blogs Tweets
1 Experimental evidence of massive-scale emotional
contagion through social networks
5044 301 130 3801
2 Variation in Melanism and Female Preference in Proximate
but Ecologically Distinct Environments
4823 4 11 6669
3 Artificial sweeteners induce glucose intolerance by
altering the gut microbiota
3500 127 34 3303
4 Stimulus-triggered fate conversion of somatic cells into
pluripotency
2986 159 92 3295
5 Dogs are sensitive to small variations of the Earth's
magnetic field
2956 54 33 2668
6 Christmas 2013: Research: The survival time of chocolates
on hospital wards: covert observational study
2734 3 4 3084
7 Epidemiological modeling of online social network
dynamics
2392 156 19 1649
8 Searching the Internet for evidence of time travelers 2246 34 22 2262
9 Conscious Brain-to-Brain Communication in Humans Using
Non-Invasive Technologies
2160 82 32 1660
10 Were James Bond's drinks shaken because of alcohol
induced tremor?
2146 127 14 1559
Top 10 articles according to the Altmetric Score (Source: http://www.altmetric.com/top100/2014/)

0
50
100
150
200
250
300
350
0
1000
2000
3000
4000
5000
6000
7000
8000
1 2 3 4 5 6 7 8 9 10
Altmetric score
Tweets
News
Blogs
The volume of tweets, blog posts, and news stories for the top ten research articles
from 2014 as measured with altmetrics, and their Altmetric Score (number of news
stories and blog posts are depicted on the secondary axis to the right in the figure)

Researchers
The public
Lowimpact
Highimpact

Altmetrics
Provide a more nuanced
image of the total impact
that research has made?

Citations Altmetrics
Altmetrics are not alternatives to citations

Citations Tweets
Older publications Newer publications
The effects of time on the volume of citations and tweets
(Haustein, Peters, Sugimoto, Thelwall, & Larivière, 2014)
The effects of time

Heterogeneity of altmetrics

doiDigital Object Identifier
Most altmetrics identified from:

Bollen J, Van de Sompel H, Hagberg A, Bettencourt L, Chute R, Rodriguez MA, et al. (2009) Clickstream
Data Yields High-Resolution Maps of Science. PLoS ONE 4(3): e4803. doi:10.1371/journal.pone.0004803
Disciplinary differences

Bots?
Haustein, S., Bowman, T.D., Holmberg, K., Tsou, A., Sugimoto, C.R. & Lariviére, V. (in press). Tweets as impact indicators: Examining the implications of
automated “bot” accounts on Twitter. To appear in JASIST. arXiv version available at: http://arxiv.org/ftp/arxiv/papers/1410/1410.4139.pdf.
Gaming altmetrics?

Stories behind the altmetrics?

Altmetrics and
open science?

Measuringthe Societal Impact of Open Science
AvoimenTieteenYhteiskunnallisenVaikuttavuudenMittarit
Project is financed by the Ministry of Education and Culture in Finland (2015-2016)

Goals of the research project:
1) Investigate altmetrics (at country-level, i.e.
Finland) and create a prototype to
demonstrate them
2) Develop methods to map societal impact of
research (and to analyze the stories behind
the altmetrics)

Shameless self
promotion:
Book
forthcoming
later in 2015

Kim Holmberg
kim.j.holmberg@utu.fi
http://kimholmberg.fi
@kholmber
Kiitos!
Slides available at
http://www.slideshare.net/kholmber