Altmetrics - Measuring the impact of scientific activities

5,448 views

Published on

An introduction to altmetrics, the complementary metrics of research impact. The presentation covers some of the challenges with more traditional measures, and the potential of and challenges with altmetrics. The presentation gives a brief overview of the background to a new research project about measuring the societal impact of open science.

Published in: Science

Altmetrics - Measuring the impact of scientific activities

  1. 1. Background image: http://commons.wikimedia.org/wiki/File:Water_drop_impact_on_a_water-surface_-_(5).jpg Altmetrics Measuring the impact of scientific activities Kim Holmberg Research Unit for the Sociology of Education University of Turku (e) kim.j.holmberg@utu.fi (w3) http://kimholmberg.fi @kholmber Oulu, 8.5.2015
  2. 2. Bibliometrics the study of quantitative aspects of the production, dissemination, and use of recorded information Tague-Sufcliffe, 1992
  3. 3. 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 201420132012201120102009200820072006200520042003200220012000199919981997199619951994 The increase of publications about bibliometrics as indexed by Web of Science between 1994 and 2014 (data retrieved from Web of Science on 9 February, 2015).
  4. 4. Goodhart’s Law: When a measure becomes the target, it ceases to be a good measure Goodhart, 1975
  5. 5. Outreach and Impact Leader, - Ref:1455401 “ We are seeking to recruit … to support, develop and promote outreach and impact activities … Our aim is to maximize the widest possible impact of the research programme in whichever way is most appropriate … The salary range will be £41,430 - £48,873 per annum...” Taking impact seriously
  6. 6. Problems with citation-based indicators
  7. 7. Problem 1: Time
  8. 8. 0 2 4 6 8 10 12 14 16 18 20 Impact factor window Immediacy index window Cited half-life 50% of citations 50% of citations Time after publication (Years) Citations
  9. 9. 0 2 4 6 8 10 12 14 16 18 20 Time after publication (Years) Citations Time of research assessment Option 1: Journal impact factor, Julkaisufoorumi, … Potential future impact
  10. 10. 0 2 4 6 8 10 12 14 16 18 20 Time after publication (Years) Citations Time of research assessment Option 2: Citations (after n years) Past success and impact
  11. 11. 0 2 4 6 8 10 12 14 16 18 20 Time after publication (Years) Citations How can we measure impact in realtime?
  12. 12. Problem 2: Journals
  13. 13. 1 vs.
  14. 14. Fenner, M. & Lin, J. (2013). Altmetrics in Evolution: Defining & Redefining the Ontology of Article-Level Metrics. http://dx.doi.org/10.3789/isqv25no2.2013.04 2
  15. 15. Problem 3: Impact
  16. 16. Science IMPACT ON:
  17. 17. Science Society Environment Culture Economy Education IMPACT ON:
  18. 18. SOLUTION?
  19. 19. Altmetrics the study and use of scholarly impact measures based on activity in online tools and environments Priem, J. (2014). Altmetrics. In Cronin, B. & Sugimoto, C.R. (Eds.). Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact. MIT Press, 2014.
  20. 20. Created by researchers Created by the public Altmetrics
  21. 21. Created by researchers Indicating future scientific impact? Altmetrics
  22. 22. Indicating other types of impact, such as societal impact? Created by the public Altmetrics
  23. 23. Researchers The public
  24. 24. Rank Title Altmetric score News Blogs Tweets 1 Experimental evidence of massive-scale emotional contagion through social networks 5044 301 130 3801 2 Variation in Melanism and Female Preference in Proximate but Ecologically Distinct Environments 4823 4 11 6669 3 Artificial sweeteners induce glucose intolerance by altering the gut microbiota 3500 127 34 3303 4 Stimulus-triggered fate conversion of somatic cells into pluripotency 2986 159 92 3295 5 Dogs are sensitive to small variations of the Earth's magnetic field 2956 54 33 2668 6 Christmas 2013: Research: The survival time of chocolates on hospital wards: covert observational study 2734 3 4 3084 7 Epidemiological modeling of online social network dynamics 2392 156 19 1649 8 Searching the Internet for evidence of time travelers 2246 34 22 2262 9 Conscious Brain-to-Brain Communication in Humans Using Non-Invasive Technologies 2160 82 32 1660 10 Were James Bond's drinks shaken because of alcohol induced tremor? 2146 127 14 1559 Top 10 articles according to the Altmetric Score (Source: http://www.altmetric.com/top100/2014/)
  25. 25. 0 50 100 150 200 250 300 350 0 1000 2000 3000 4000 5000 6000 7000 8000 1 2 3 4 5 6 7 8 9 10 Altmetric score Tweets News Blogs The volume of tweets, blog posts, and news stories for the top ten research articles from 2014 as measured with altmetrics, and their Altmetric Score (number of news stories and blog posts are depicted on the secondary axis to the right in the figure)
  26. 26. Researchers The public Lowimpact Highimpact
  27. 27. Altmetrics Provide a more nuanced image of the total impact that research has made?
  28. 28. Citations Altmetrics Altmetrics are not alternatives to citations
  29. 29. Citations Tweets Older publications Newer publications The effects of time on the volume of citations and tweets (Haustein, Peters, Sugimoto, Thelwall, & Larivière, 2014) The effects of time
  30. 30. Heterogeneity of altmetrics
  31. 31. doiDigital Object Identifier Most altmetrics identified from:
  32. 32. Bollen J, Van de Sompel H, Hagberg A, Bettencourt L, Chute R, Rodriguez MA, et al. (2009) Clickstream Data Yields High-Resolution Maps of Science. PLoS ONE 4(3): e4803. doi:10.1371/journal.pone.0004803 Disciplinary differences
  33. 33. Bots? Haustein, S., Bowman, T.D., Holmberg, K., Tsou, A., Sugimoto, C.R. & Lariviére, V. (in press). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. To appear in JASIST. arXiv version available at: http://arxiv.org/ftp/arxiv/papers/1410/1410.4139.pdf. Gaming altmetrics?
  34. 34. Stories behind the altmetrics?
  35. 35. Altmetrics and open science?
  36. 36. Measuringthe Societal Impact of Open Science AvoimenTieteenYhteiskunnallisenVaikuttavuudenMittarit Project is financed by the Ministry of Education and Culture in Finland (2015-2016)
  37. 37. Goals of the research project: 1) Investigate altmetrics (at country-level, i.e. Finland) and create a prototype to demonstrate them 2) Develop methods to map societal impact of research (and to analyze the stories behind the altmetrics)
  38. 38. Shameless self promotion: Book forthcoming later in 2015
  39. 39. Kim Holmberg kim.j.holmberg@utu.fi http://kimholmberg.fi @kholmber Kiitos! Slides available at http://www.slideshare.net/kholmber

×