Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scholarly communication
and evaluation:
from bibliometrics to altmetrics
Stefanie Haustein
stefanie.haustein@umontreal.ca
...
Scholarly Communication
• peer-reviewed journals
1665: Journal de Sçavans
Philosophical Transactions
replace personal corr...
Scholarly Communication
• citation analysis as retrieval tool to handle
information overload
“It would not be excessive to...
Scholarly Communication
• digital revolution
 electronic publishing
• acceleration, openness and diversification of schol...
Altmetrics
Criticism against current form of research evaluation:
• peer-reviewed publications in scholarly journals as th...
Altmetrics
• alternative use and visibility of publications
on social media:
more traditional forms of use:
• alternative ...
Definitions and terminology
• webometrics
“Polymorphous mentioning is likely to become a defining feature of Web-
based sc...
Definitions and terminology
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
adapted from: Bj...
Definitions and terminology
adapted from: Björneborn & Ingwersen (2004, p. 1217)
informetrics
scientometrics
bibliometrics...
Definitions and terminology
adapted from: Björneborn & Ingwersen (2004, p. 1217)
informetrics
scientometrics
bibliometrics...
Definitions and terminology
adapted from: Björneborn & Ingwersen (2004, p. 1217)
informetrics
scientometrics
bibliometrics...
Definitions and terminology
adapted from: Björneborn & Ingwersen (2004, p. 1217)
informetrics
scientometrics
bibliometrics...
Definitions and terminology
Acts leading to (online) events used for metrics
RESEARCH
OBJECT
Haustein,Bowman&Costas(2015)
Social media metrics: research
• Which social media metrics are valid impact indicators?
• What kind of impact do the vari...
Prevalence: social media uptake
• social media activity around scholarly articles grows
5% to 10% per month (Adie & Roe, 2...
Prevalence: coverage
Mendeley
93% of Science articles 2007 (Li, Thelwall & Giustini, 2012)
94% of Nature articles 2007 (Li...
Prevalence: density
Mean number of events per paper per document type
WoS papers 2012 with DOI
(Haustein, Costas & Lariviè...
Prevalence: density / intensity
Mean number of events per paper
WoS papers with DOIs 2012
all papers / papers with at leas...
Similarity: correlations
Spearman correlations with citations
WoS papers with DOIs 2012
all papers / papers with at least ...
Popularity: highly tweeted
Highly tweeted Physics paper
Popularity: highly tweeted
Highly tweeted paper
Popularity: highly tweeted
Highly tweeted paper
Communities of attention
Distinguishing between types of Twitter impact
• engagement = dissimilarity with paper title
• ex...
Communities of attention
• 660,149 original tweets
• 237,222 tweeted documents
• 125,083 unique users
• number of tweets t...
Communities of attention
exposure
engagement influencers /
brokers
orators /
discussing
disseminators /
mumblers
broadcast...
Communities of attention
mean tweets to papers
tp = 5.3
exposure
engagement
tp = 3.2tp = 1.7
tp = 11.5tp = 4.4
(Haustein, ...
Communities of attention
mean tweets per day
tpd = 5.9
exposure
engagement
tpd = 10.1tpd = 1.8
tpd = 9.4tpd = 1.7
(Haustei...
Communities of attention
mean relative citation rate
mncs = 2.3
exposure
engagement
mncs = 2.4mncs = 2.5
mncs = 2.1mncs = ...
Communities of attention
more than 100 tweeted papers
708 of 125,083 users (0.6%)
9 57
130 512
(Haustein, Bowman & Costas,...
Communities of attention
708 of 125,083 users (0.6%)
more than 100 tweeted papers
(Haustein, Bowman & Costas, submitted)
Some conclusions
• citations, Mendeley readers and tweets reflect different
kinds of impact on different social groups
• M...
Some conclusions
• fundamental differences between social media metrics
and citations:
• gatekeeping
• community
• engagem...
Some tips
When using altmetrics:
• time biases apply: don’t use for old papers!
• most metrics only captured for DOIs: rem...
Stefanie Haustein
Thank you for your attention!
Questions?
stefanie.haustein@umontreal.ca
@stefhaustein
crc.ebsi.umontreal...
Upcoming SlideShare
Loading in …5
×

Scholarly communication and evaluation: from bibliometrics to altmetrics

3,916 views

Published on

presentation at COAR-SPARC Conference 2015, Porto, Portugal, 16 April 2015

Session 4: Assessing Value
Chair: Lars Björnshauge

https://www.coar-repositories.org/community/events/annual-meeting-2015/programme/

Published in: Data & Analytics
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Scholarly communication and evaluation: from bibliometrics to altmetrics

  1. 1. Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan
  2. 2. Scholarly Communication • peer-reviewed journals 1665: Journal de Sçavans Philosophical Transactions replace personal correspondences • registration • certification • dissemination • archiving • “Little Science, Big Science” Derek J. de Solla Price (1963) exponential growth
  3. 3. Scholarly Communication • citation analysis as retrieval tool to handle information overload “It would not be excessive to demand that the thorough scholar check all papers that have cited or criticized such papers, if they could be located quickly. The citation index makes this check practicable.” • citation analysis as evaluation method  oversimplification of scientific work and success publications = productivity | citations = impact  adverse effects Garfield, 1955, p. 108
  4. 4. Scholarly Communication • digital revolution  electronic publishing • acceleration, openness and diversification of scholarly output and impact  open access and open science • altmetrics manifesto: Priem, Taraborelli, Groth and Neylon (2010) “No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem.”
  5. 5. Altmetrics Criticism against current form of research evaluation: • peer-reviewed publications in scholarly journals as the only form of output that “counts” • particularly against Journal Impact Factor • citations as the only form of impact that “counts” Altmetrics as alternatives: • including all research “products” • similar but more timely than citations  predicting scientific impact • different, broader impact than citations  measuring societal impact
  6. 6. Altmetrics • alternative use and visibility of publications on social media: more traditional forms of use: • alternative forms of research output  pragmatic development based on IT developments … … …
  7. 7. Definitions and terminology • webometrics “Polymorphous mentioning is likely to become a defining feature of Web- based scholarly communication.” “There will soon be a critical mass of web-based digital objects and usage statistics on which to model scholars’ communication behaviors […] and with which to track their scholarly influence and impact, broadly conceived and broadly felt.” • PLOS article level metrics (ALM) • altmetrics “study and use of scholarly impact measures based on activity in online tools and environments” “a good idea but a bad name” Priem (2014, p. 266) Cronin, Snyder, Rosenbaum, Martinson & Callahan (1998, p.1320) Cronin (2005, p. 196) Rousseau & Ye (2013, p. 2)
  8. 8. Definitions and terminology informetrics scientometrics bibliometrics cybermetrics webometrics altmetrics adapted from: Björneborn & Ingwersen (2004, p. 1217)
  9. 9. Definitions and terminology adapted from: Björneborn & Ingwersen (2004, p. 1217) informetrics scientometrics bibliometrics cybermetrics webometrics social media metrics social media metrics Haustein, Larivière, Thelwall, Amyot & Peters (2014)
  10. 10. Definitions and terminology adapted from: Björneborn & Ingwersen (2004, p. 1217) informetrics scientometrics bibliometrics cybermetrics webometrics social media metrics social media metrics “Although social media metrics seems a better fit as an umbrella term because it addresses the social media ecosystem from which they are captured, it fails to incorporate the sources that are not obtained from social media platforms (such as mainstream newspaper articles or policy documents) that are collected (for instance) by Altmetric.com.“ Haustein, Bowman & Costas (2015, p. 3)
  11. 11. Definitions and terminology adapted from: Björneborn & Ingwersen (2004, p. 1217) informetrics scientometrics bibliometrics cybermetrics webometrics social media metrics scholarly metrics
  12. 12. Definitions and terminology adapted from: Björneborn & Ingwersen (2004, p. 1217) informetrics scientometrics bibliometrics cybermetrics webometrics social media metrics scholarly metrics scholarly metrics “[T]he heterogeneity and dynamicity of the scholarly communication landscape make a suitable umbrella term elusive. It may be time to stop labeling these terms as parallel and oppositional (i.e., altmetrics vs bibliometrics) and instead think of all of them as available scholarly metrics— with varying validity depending on context and function.“ Haustein, Sugimoto & Larivière (2015, p. 3)
  13. 13. Definitions and terminology Acts leading to (online) events used for metrics RESEARCH OBJECT Haustein,Bowman&Costas(2015)
  14. 14. Social media metrics: research • Which social media metrics are valid impact indicators? • What kind of impact do the various metrics reflect? • What is the relationship between social media activity and bibliometric variables? • Which content receive the most attention on the platforms? • Who is engaging with scholarly material on social media sites? • What are the motivations behind this use?
  15. 15. Prevalence: social media uptake • social media activity around scholarly articles grows 5% to 10% per month (Adie & Roe, 2013) • Mendeley and Twitter largest sources for mentions of scholarly documents: Mendeley 521 million bookmarks 2.7 million users 32% increase of users from 9/2012 to 09/2013 (Haustein & Larivière, 2014) Twitter 500 million tweets per day 230 million active users 39% increase of users from 9/2012 to 09/2013 ca. 10% of researchers in professional context
  16. 16. Prevalence: coverage Mendeley 93% of Science articles 2007 (Li, Thelwall & Giustini, 2012) 94% of Nature articles 2007 (Li, Thelwall & Giustini, 2012) 80% of PLOS journals papers 2003-2010 (Priem, Piwowar & Hemminger, 2012) 66% of PubMed/WoS papers 2010-2012 (Haustein et al., 2014a) 63% of WoS papers with DOIs 2005-2011 (Zahedi, Costas & Wouters, 2014) 47% of Social Science WoS papers 2008 (Mohammadi et al., 2014) 35% of Engineering & Techn. WoS papers 2008 (Mohammadi et al., 2014) 31% of Physics WoS papers 2008 (Mohammadi et al., 2014) 13% of Humanities WoS papers 2008 (Mohammadi & Thelwall, 2014) Twitter 2% of WoS papers with DOIs 2005-2011 (Zahedi, Costas & Wouters, 2014) 9% of PubMed/WoS 2010-2012 (Haustein et al., 2014b) 13% of WoS papers with DOIs July-December 2011 (Costas, Zahedi & Wouters, 2014) 22% of WoS papers with DOIs 2012 (Haustein, Costas & Larivière, 2015)
  17. 17. Prevalence: density Mean number of events per paper per document type WoS papers 2012 with DOI (Haustein, Costas & Larivière, 2015
  18. 18. Prevalence: density / intensity Mean number of events per paper WoS papers with DOIs 2012 all papers / papers with at least one social media event 0.03 / 1.51 Blogs 0.78 / 3.65 Twitter 0.08 / 1.78 Facebook 0.01 / 1.66 Google+ 0.01 / 1.54 Mainstream media PubMed/WoS papers 2010-2012 6.43 / 9.71 Mendeley (Haustein et al., 2014a) (Haustein, Costas & Larivière, 2015)
  19. 19. Similarity: correlations Spearman correlations with citations WoS papers with DOIs 2012 all papers / papers with at least one social media event 0.124 / 0.191 Blogs 0.194 / 0.148 Twitter 0.097 / 0.167 Facebook 0.065 / 0.209 Google+ 0.083 / 0.199 Mainstream media PubMed/WoS papers 2011 0.386 / 0.456 Mendeley (Haustein et al., 2014a) (Haustein, Costas & Larivière, 2015)
  20. 20. Popularity: highly tweeted Highly tweeted Physics paper
  21. 21. Popularity: highly tweeted Highly tweeted paper
  22. 22. Popularity: highly tweeted Highly tweeted paper
  23. 23. Communities of attention Distinguishing between types of Twitter impact • engagement = dissimilarity with paper title • exposure = number of followers
  24. 24. Communities of attention • 660,149 original tweets • 237,222 tweeted documents • 125,083 unique users • number of tweets to 2012 papers • mean tweets per day • mean relative citation rate of tweeted papers • mean engagement • mean exposure • mean number of followers • mean number of following • tweeted document coupling user network (Haustein, Bowman & Costas, submitted)
  25. 25. Communities of attention exposure engagement influencers / brokers orators / discussing disseminators / mumblers broadcasters
  26. 26. Communities of attention mean tweets to papers tp = 5.3 exposure engagement tp = 3.2tp = 1.7 tp = 11.5tp = 4.4 (Haustein, Bowman & Costas, submitted)
  27. 27. Communities of attention mean tweets per day tpd = 5.9 exposure engagement tpd = 10.1tpd = 1.8 tpd = 9.4tpd = 1.7 (Haustein, Bowman & Costas, submitted)
  28. 28. Communities of attention mean relative citation rate mncs = 2.3 exposure engagement mncs = 2.4mncs = 2.5 mncs = 2.1mncs = 2.2 (Haustein, Bowman & Costas, submitted)
  29. 29. Communities of attention more than 100 tweeted papers 708 of 125,083 users (0.6%) 9 57 130 512 (Haustein, Bowman & Costas, submitted)
  30. 30. Communities of attention 708 of 125,083 users (0.6%) more than 100 tweeted papers (Haustein, Bowman & Costas, submitted)
  31. 31. Some conclusions • citations, Mendeley readers and tweets reflect different kinds of impact on different social groups • Mendeley seems to mirror use of broader but still academic audience, largely students and postdocs • Twitter seems to reflect popularity among general public and represents mix of societal impact, scientific discussion and dissemination and buzz • differences between disciplines, document types and age  reader counts and tweets cannot be directly compared without normalization
  32. 32. Some conclusions • fundamental differences between social media metrics and citations: • gatekeeping • community • engagement • quantitative and qualitative research needed: • determine biases and confounding factors • identify user groups • identify user motivations and types of use  meaning of social media metrics needs to be understood before they are applied to research evaluation
  33. 33. Some tips When using altmetrics: • time biases apply: don’t use for old papers! • most metrics only captured for DOIs: remember limitation! • social media metrics do not replace citations: don’t substitute! • social media metrics are heterogeneous: don’t blend! • document type: don’t compare! • disciplinary differences: don’t compare! • not all events reflect use or impact: differentiate! • motivations and confounding factors unknown: be careful!
  34. 34. Stefanie Haustein Thank you for your attention! Questions? stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan Thank you for your attention! Questions? Thank you for your attention! Questions?Obrigada! Special Issue “Social Media Metrics” Aslib Journal of Information Management 67(3) Early View: www.emeraldinsight.com/toc/ajim/67/3 Links to OA preprints: crc.ebsi.umontreal.ca/aslib/

×