Tweets and Mendeley readers: Two different types of article level metrics

3,368 views
3,106 views

Published on

Presentation at APE2014

Published in: Education, Technology, Business

Tweets and Mendeley readers: Two different types of article level metrics

  1. 1. Tweets and Mendeley readers Two different types of article level metrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein
  2. 2. Overview •  Altmetrics •  increasing use •  meaning? •  Aim of the studies •  Data sets and methods •  Results •  documents •  correlations •  disciplines •  Conclusions & outlook
  3. 3. Altmetrics: increasing use •  social media activity around scholarly articles growing by 5% to 10% per month (Adie & Roe, 2013) •  Mendeley and Twitter largest altmetrics sources •  Mendeley •  521 million bookmarks •  2.7 million users •  32% increase of users from 09/2012 to 09/2013 •  Twitter •  500 million tweets per day •  230 million active users •  39% increase of users from 09/2012 to 09/2013 Adie, E. & Roe, W. (2013). Altmetric: Enriching Scholarly Content with Article-level Discussion and Metrics. Learned Publishing, 26(1), 11-17. Mendeley statistics based on monthly user counts from 10/2010 to 01/2014 on the Mendeley website accessed through the Internet Archive Twitter statistics: https://business.twitter.com/whos-twitter and http://www.sec.gov/Archives/edgar/data/1418091/000119312513400028/d564001ds1a.htm
  4. 4. Altmetrics: meaning? •  ultimate goals •  similar to but more timely than citations Ø  predicting scientific impact •  different, broader impact than captured by citations Ø  measuring societal impact •  impact of various outputs Ø  “value all research products” Piwowar (2013) Piwowar, H. (2013). Value all research products. Nature, 493(7431), 159.
  5. 5. Altmetrics: meaning? •  Altmetrics are “representing very different things” (Lin & Fenner, 2013) •  unclear what exactly they measure: •  scientific impact •  social impact •  “buzz” Lin, J. & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20-26.
  6. 6. Altmetrics: meaning? ad-hoc classifications need to be supported by research
  7. 7. Altmetrics: meaning? scientist on Twitter tweeting scientific paper in non-scholarly manner: •  scientific impact? •  social impact? •  buzz?
  8. 8. Altmetrics: meaning?
  9. 9. Aim of the studies •  providing empirical evidence of Mendeley reader counts and tweets of scholarly documents for a large data set •  generate knowledge about factors influencing popularity of scholarly documents on Mendeley and Twitter •  analyzing the following research questions: •  •  •  •  What is the relationship between social-media and citation counts? How do social-media metrics differ? Which papers are highly tweeted or highly bookmarked? How do these aspects differ across scientific disciplines? Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (2014). Tweeting Biomedicine: An Analysis of Tweets and Citations in the Biomedical Literature. Journal of the Association for Information Sciences and Technology. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (submitted). Tweets vs. Mendeley readers: How do these two social media metrics differ? IT-Information Technology.
  10. 10. Aim of the studies •  large-scale analysis of tweets and Mendeley readers of biomedical papers •  Twitter and Mendeley coverage •  Twitter and Mendeley user rates •  correlation with citations •  discovering differences between: •  documents •  disciplines & specialties Ø  providing an empirical framework to compare coverage, correlations and user rates
  11. 11. Data sets & methods •  1.4 million PubMed papers covered by WoS •  publication years: 2010-2012 •  document types: articles & reviews •  matching of WoS and PubMed •  tweet counts collected by Altmetric.com •  collection based on PMID, DOI, URL •  matching WoS via PMID •  Mendeley readership data collected via API •  matching title and author names •  journal-based matching of NSF classification
  12. 12. Data sets & methods: framework
  13. 13. Data sets & methods: age biases Current biases influencing correlation coefficients Ø  compare documents of similar age Ø  normalize for age differences
  14. 14. Results: documents •  Twitter coverage is quite low but increasing •  correlation between tweets and citations is very low Publication year Twitter coverage Papers (T≥1) Spearman's ρ Mean Median Maximum T2010 C2010 2.4% 13,763 .104** 2.1 18.3 1 7 237 3,922 T2011 C2011 10.9% 63,801 .183** 2.8 5.7 1 2 963 2,300 T2012 C2012 20.4% 57,365 .110** 2.3 1.3 1 0 477 234 9.4% 134,929 .114** 2.5 5.1 1 1 963 3,922 T2010-2012 C2010-2012
  15. 15. Results: documents Top 10 tweeted documents: catastrophe & topical / web & social media / curious story scientific discovery / health implication / scholarly community Article Journal C T Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients is associated with exposure to low-dose irradiation PNAS 9 963 Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the Fukushima nuclear accident PNAS 30 639 Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips Science 11 558 Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator Journal of Physical Chemistry A -- 549 Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477 Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life expectancy: a prospective cohort study Lancet 51 419 Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations Journal of Sexual Medicine -- 392 Newman & Feldman (2011). Copyright and Open Access at the Bedside New England Journal of Medicine 3 332 Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323 Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA receptor expression in a mouse via the vagus nerve PNAS 31 297
  16. 16. Results: correlations PubMed papers covered by Web of Science (PY=2011) Spearman correlations between citations (C), Mendeley readers (R) and tweets (T) for all papers published in 2011 (A, n=586,600), for papers with respectively at least one citation (B, n=410,722), one Mendeley reader (C, n=390,190) or one tweet (D, n=63,800), one Mendeley reader and one tweet (E, n=45,229) and one citation, one Mendeley reader and one tweet (F, n=36,068). All results are significant at the 0.01 level (two-tailed).
  17. 17. Results: disciplines PubMed papers covered by Web of Science 2010-2012
  18. 18. Altmetrics: disciplinary biases x-axis: coverage of specialty on platform y-axis: correlation between social media counts and citations bubble size: intensity of use based on mean social media count rate
  19. 19. Results: disciplines General Biomedical Research papers 2011 Scatterplot of number of citations and number of tweets (A, ρ=0.181**) and Mendeley readers (B, ρ=0.677**), bubble size represents number of Mendeley readers (A) and tweets (B). The respective three most tweeted (A) and read (B) papers are labeled showing the first author.
  20. 20. Results: disciplines Public Health papers 2011 Scatterplot of number of citations and number of tweets (A, ρ=0.074**) and Mendeley readers (B, ρ=0.351**) for papers published in Public Health in 2011. The respective three most tweeted (A) and read (B) papers are labeled showing the first author.
  21. 21. Conclusions & outlook •  uptake, usage intensity and correlations differ between disciplines and research fields Ø  social media counts of papers from different fields are not directly comparable •  citations, Mendeley readers and tweets reflect different kind of impact on different social groups •  Mendeley seems to mirror use of a broader but still academic audience, largely students and postdocs •  Twitter seems to reflect the popularity among a general public and represents a mix of societal impact, scientific discussion and buzz Ø  the number of Mendeley readers and tweets are two distinct social media metrics
  22. 22. Conclusions & outlook •  before applying social media counts in information retrieval and research evaluation further research is needed: Ø  identifying different factors influencing popularity of scholarly documents on social media Ø  analyzing uptake and usage intensity in various disciplines Ø  differentiating between audiences and engagements
  23. 23. Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Sciences and Technology. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (submitted). Tweets vs. Mendeley readers: How do these two social media metrics differ? IT-Information Technology. Thank you for your attention! Questions? Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein

×