Disciplinary differences
and other biases
Exploring social media metrics in scholarly context
Stefanie Haustein

stefanie....
Overview
• Altmetrics: definitions
• Bibliometrics: in retrospect
• Altmetrics: present
• correlations
• publication age b...
Altmetrics: definitions
•
•

term coined by Jason Priem
introduced as a better filter
than and alternative to
citations an...
Altmetrics: definitions
• ultimate goals
•

similar to but more timely than citations
 predicting scientific impact

•

d...
Altmetrics: definitions
•

Altmetrics are “representing very different things”
(Lin & Fenner, 2013)

•

unclear what exact...
Altmetrics: definitions

ad-hoc
classifications
need to be
supported
by research
Altmetrics: definitions
scientist on
Twitter tweeting
scientific paper
in non-scholarly
manner:
• scientific impact?
• soc...
Altmetrics: definitions
• complex to define and classify tools and motivations
•
•

scientific and non-scientific audience...
Bibliometrics: in retrospect
• when Garfield created SCI, sociologists of science
analyzed meaning of publications and cit...
Bibliometrics: in retrospect
• empirical studies helped sociologists to understand
structure and norms of science
• for bi...
Bibliometrics: in retrospect
• similar to development of SCI in the 1960s, social
media metrics have to be analyzed:
• qua...
Altmetrics: correlations
e.g., Mendeley
• 793 Nature papers: ρ=0.559
•
•
•
•
•
•

•

Li, Thelwall, & Giustini (2012)

820 ...
Altmetrics: age biases
Current biases influencing correlation coefficients
Altmetrics: age biases
Current biases influencing correlation coefficients
Altmetrics: age biases
Current biases influencing correlation coefficients
Altmetrics: age biases
Current biases influencing correlation coefficients
 compare documents of similar age
 normalize ...
Altmetrics: disciplinary biases
PubMed papers covered by Web of Science 2010-2012
Altmetrics: disciplinary biases
PubMed papers covered by Web of Science 2010-2012
Altmetrics: disciplinary biases
x-axis:
coverage of
specialty on
platform
y-axis:
correlation
between social
media counts
...
Altmetrics: subject bias
General Biomedical Research papers 2011

Scatterplot of number of citations and number of tweets ...
Altmetrics: subject bias
Top 10 tweeted documents:

catastrophe & topical / web & social media / curious story
scientific ...
Altmetrics: future
• before applying social media counts in information
retrieval and research evaluation, we need:
 to u...
References
Bar-Ilan, J. (2011). Articles tagged by 'bibliometrics' on Mendeley and CiteULike. Paper presented at the Metri...
Thank you for your attention!

Questions?
Stefanie Haustein

stefanie.haustein@umontreal.ca
@stefhaustein
Upcoming SlideShare
Loading in...5
×

NISO Webinar: New Perspectives on Assessment How Altmetrics Measure Scholarly Impact

519

Published on

presentation at NISO webinar on altmetrics on November 13, 2013
http://www.niso.org/news/events/2013/webinars/altmetrics

Published in: Technology, Business

Transcript of "NISO Webinar: New Perspectives on Assessment How Altmetrics Measure Scholarly Impact"

  1. 1. Disciplinary differences and other biases Exploring social media metrics in scholarly context Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein
  2. 2. Overview • Altmetrics: definitions • Bibliometrics: in retrospect • Altmetrics: present • correlations • publication age biases • disciplinary biases • subject biases • Altmetrics: future • References
  3. 3. Altmetrics: definitions • • term coined by Jason Priem introduced as a better filter than and alternative to citations and peer-review http://altmetrics.org/manifesto/ • “…altmetrics is a good idea, but a bad name” “…we would like to propose the term influmetrics” Rousseau & Ye (2013) • • rather complementary than alternative to citations social media metrics
  4. 4. Altmetrics: definitions • ultimate goals • similar to but more timely than citations  predicting scientific impact • different, broader impact than captured by citations  measuring societal impact • impact of various outputs  “value all research products” Piwowar (2013)
  5. 5. Altmetrics: definitions • Altmetrics are “representing very different things” (Lin & Fenner, 2013) • unclear what exactly they measure: • scientific impact • social impact • “buzz” • all of the above?
  6. 6. Altmetrics: definitions ad-hoc classifications need to be supported by research
  7. 7. Altmetrics: definitions scientist on Twitter tweeting scientific paper in non-scholarly manner: • scientific impact? • social impact? • buzz?
  8. 8. Altmetrics: definitions • complex to define and classify tools and motivations • • scientific and non-scientific audiences cannot be determined on the platform used level of engagement differs not only between platforms but also within: saving paper to Mendeley library vs. tweeting about it saving vs. reading retweeting link vs. discussing content  differentiation between audiences and engagement needed to determine meaning of metrics
  9. 9. Bibliometrics: in retrospect • when Garfield created SCI, sociologists of science analyzed meaning of publications and citations (Merton, Zuckerman, Cole & Cole, etc.) • sociological research • What is it to publish a paper? • What are the reasons to cite? • empirical bibliometric research • disciplinary differences in publication • and citation behavior delay and obsolescence patterns
  10. 10. Bibliometrics: in retrospect • empirical studies helped sociologists to understand structure and norms of science • for bibliometricians, studies provided a theoretical framework and legitimation to use citation analysis in research evaluation • knowledge about disciplinary differences and obsolescence patterns helped to normalize statistics and create more appropriate indicators
  11. 11. Bibliometrics: in retrospect • similar to development of SCI in the 1960s, social media metrics have to be analyzed: • qualitative studies to analyze who, how and why • people use various social media platforms large-scale quantitative studies to determine differences and biases in terms of disciplines, topics, document types, publications years, publication types and sources, author age and affiliation, etc.  to find out what various social media metrics mean and what they can be used for
  12. 12. Altmetrics: correlations e.g., Mendeley • 793 Nature papers: ρ=0.559 • • • • • • • Li, Thelwall, & Giustini (2012) 820 Science papers: ρ=0.540 1,651 JASIST papers: ρ=0.458 Bar-Ilan (2012) 5,596 PLoS ONE papers: ρ=0.3 Priem, Piwowar, & Hemminger (2012) Bar-Ilan et al. (2011) 1,136 bibliometrics papers: ρ=0.448 Li, & Thelwall (2012) 1,389 F1000 papers: ρ=0.686 62,647 social science papers: ρ=0.516 Mohammadi & Thelwall (in press) 14,640 humanities papers: ρ=0.428 random sample of Zahedi, Costas, & Wouters (2013) 200,000 WoS papers: ρ=0.35 Haustein, et al.(submitted) 586,600 PubMed papers: ρ=0.386
  13. 13. Altmetrics: age biases Current biases influencing correlation coefficients
  14. 14. Altmetrics: age biases Current biases influencing correlation coefficients
  15. 15. Altmetrics: age biases Current biases influencing correlation coefficients
  16. 16. Altmetrics: age biases Current biases influencing correlation coefficients  compare documents of similar age  normalize for age differences
  17. 17. Altmetrics: disciplinary biases PubMed papers covered by Web of Science 2010-2012
  18. 18. Altmetrics: disciplinary biases PubMed papers covered by Web of Science 2010-2012
  19. 19. Altmetrics: disciplinary biases x-axis: coverage of specialty on platform y-axis: correlation between social media counts and citations bubble size: intensity of use based on mean social media count rate
  20. 20. Altmetrics: subject bias General Biomedical Research papers 2011 Scatterplot of number of citations and number of tweets (A, ρ=0.181**) and Mendeley readers (B, ρ=0.677**), bubble size represents number of Mendeley readers (A) and tweets (B). The respective three most tweeted (A) and read (B) papers are labeled showing the first author.
  21. 21. Altmetrics: subject bias Top 10 tweeted documents: catastrophe & topical / web & social media / curious story scientific discovery / health implication / scholarly community Article Journal C T Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients is associated with exposure to low-dose irradiation PNAS 9 963 Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the Fukushima nuclear accident PNAS 30 639 Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips Science 11 558 Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator Journal of Physical Chemistry A -- 549 Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477 Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life expectancy: a prospective cohort study Lancet 51 419 Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations Journal of Sexual Medicine -- 392 Newman & Feldman (2011). Copyright and Open Access at the Bedside New England Journal of Medicine 3 332 Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323 Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA receptor expression in a mouse via the vagus nerve PNAS 31 297
  22. 22. Altmetrics: future • before applying social media counts in information retrieval and research evaluation, we need:  to understand and define meaning of various social media metrics  to identify different biases  to differentiate between audiences and level of engagement  more transparency and reliability in data aggregation
  23. 23. References Bar-Ilan, J. (2011). Articles tagged by 'bibliometrics' on Mendeley and CiteULike. Paper presented at the Metrics 2011 Symposium on Informetric and Scientometric Research, New Orleans, Louisiana. Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars' visibility on the social web. In Proceedings of the 17th International Conference on Science and Technology Indicators, (pp. 98-109). Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the American Society for Information Science and Technology. Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of tweeting and scientific publication behavior. Aslib Proceedings. jasonpriem (2010, September 28). I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I'm liking #altmetrics. [Twitter post]. Li, X. & Thelwall, M. (2012). F1000, Mendeley and Traditional Bibliometric Indicators. In Proceedings of the 17th International Conference on Science and Technology Indicators, (pp. 541-551). Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461-471. Lin, J. & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20-26. Mohammadi, E., & Thelwall, M. (in press). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the American Society for Information Science and Technology. Piwowar, H. (2013). Value all research products. Nature, 493(7431), 159. Priem, J., Piwowar, H., & Hemminger, B.M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv. Priem, J., Taraborelli, D., Groth, P. & Neylon, C. (2010). Alt-Metrics: A Manifesto. Rousseau, R., & Ye, F.Y. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 58(26), 3288-3290. Zahedi, Z., Costas, R., & Wouters, P. (2013). What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact? Presentation held at the PLoS ALM Workshop 2013 in San Francisco.
  24. 24. Thank you for your attention! Questions? Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×