Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
June 18, 2014
NISO Virtual Conference: Transforming Assessment: Alternative Metrics and Other Trends
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics
- Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
Tananbaum "NISO Tech Summit: Reflections on Reflections Upon The Year of Open...
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
1. Identifying the Invisible Impact of Scholarly Publications:
Multi-Disciplinary Analysis Using Mendeley Data
Ehsan Mohammadi
@ehsanwlv
2. Traditional metrics
Why new metrics are needed?
Mendeley readership and altmetrics?
Current studies
One step forward: results of an empirical study
Next?
3. Traditionally, re-shelving statistics have been used for
research evaluation (Blecic, 1999).
Since 1960, the ISI has paved the way for a significant
change in research evaluation with citation analysis.
Citation-based metrics have been improved successfully.
Journal Impact Factor
H-index
And…
4. Limited to authors’ perspectives.
3-5 years needed to accumulate citations.
More appropriate for theoretical publications.
6. Sharing Culture
Conceptualisation
• Literature reviews
• Bibliographies
The “enhanced
publication”
Findings
• Books
• Journal articles
Openness and
diversity
Translation and
engagement
• Lectures
• Presentations
Data linked, and
shareable
Data
• Datasets
• Recorded interviews
(Czerniewicz, Kell, Willmers, & King, 2014)
7. The Higher Education Funding Council for England
(HEFCE), in the new Research Excellence Framework
(REF), considers all types of research impact outside
academia (HEFCE, 2011).
The National Science Foundation (NSF) will not
evaluate researchers based on their publications
only, but non-conventional products of research are
also considered (NSF, 2013).
8. Altmetrics is a new movement which tries to find
complementary measures for traditional metrics based
on scholars’ activities in social web platforms (Priem, Piwowar, &
Hemminger, 2012).
9. Scholarly Public
Recommended Faculty of 1000 Popular press
Cited Traditional citati
on
Wikipedia
Discussed Scholarly blogs Blogs, Twitter
Read /
Bookmarking
Mendeley,
Citeulike
Delicious
10. The early studies focused on availability of altmetrics
data.
Followed by investigating correlations between
altmetrics and citations.
Among all altmetrics, Mendeley readership counts
have the highest correlations with citations.
In all studies low/ medium correlations between
Mendeley readership counts and citations are
reported. (Li, Thelwall, & Giustini, 2012; Priem, Piwowar, & Hemminger, 2012; Bar-Ilan,
2012; Zahedi, Costas, & Wouters, 2013;Mohammadi & Thelwall, in press).
11. Interpretation of low/ medium
correlations between Mendeley
readership counts and
citations!??
12. What are the common types of readers for Clinical
Medicine, Engineering and Technology, Social
Science, Physics and Chemistry research articles in
Mendeley?
Does the academic or professional status of readers
in Mendeley affect the relationship between
Mendeley readership counts and citation counts?
13. • Journal articles of the disciplines for year 2008 were
downloaded from Web of Science (WoS). 480K
• Mendeley data for the WoS articles was collected
automatically via the Mendeley API.
• The data sets were Matched (title, author, year).
17. Most readers of papers in Mendeley are PhD
students.
Postgraduate students and postdoctoral researchers
are the two most common readers after PhD
students.
7.2% of Clinical Medicine papers were read by
people who are not academics.
Librarians were read a noticeable fraction of the
social science papers.
19. There were positive correlations between Mendeley
readership and citations for all occupations except
librarians.
The highest correlations revealed for users that are
also authors.
The lowest correlations were found for
undergraduates and non-academic users.
20. The majority of readers for all disciplines are PhD
students, postgraduates and postdocs but other
types of academics were also represented.
Younger researchers read more papers (Tenopir, King, Spencer, &
Wu, 2009) and also cite more resources in their
publications (Larivière, Sugimoto, & Bergeron, 2013) in comparison to
senior researchers.
Senior scholars are more reluctant to use social web
platforms (Mas-Bleda, Thelwall, Kousha, & Aguillo, in press)
21. Many Clinical Medicine papers were read by non-
academics which is an important issue because some
articles could be useful in clinical practice even if
they are not cited in the literature (Jones, Donovan, & Hanney, 2012)
22. Mendeley readers with authorship roles probably
reflect impact in a more similar way to traditional
citations.
The correlations for authors are not strong enough
to claim that Mendeley readership counts and
citation counts are interchangeable.
23. Students often benefit from articles that are not
highly cited and Mendeley provides an opportunity
to monitor impact on students, which probably
reflects the educational value of research articles.
Non-academic readership counts have among the
lowest correlations with citation counts, suggesting
that their readership counts could also help to
identify individual articles and types of article that
are valuable outside academia.
24. Mendeley readership is able to capture a dimension
of the impact of scientific documents on various
activities within academia such as reading without
subsequently citing, writing theses, doing
assignments or drafting research proposals.
Mendeley readership provides a little evidence of
using papers outside academia, such as medical
doctors and surgeons.
25. The meaning of Mendeley readership counts depends
upon the readers’ occupations may reflect:
Traditional citation impact .
Educational impact.
Impact on applied contexts.
27. Bar-Ilan, J. (2012). JASIST 2001–2010. Bulletin of Association for Information Science and Technology, 24–
28.
Burpee, K. J., & Fernandez, L. (2014). Scholarly Communication at Canadian Research Libraries:
Conversations with Librarians. Journal of Librarianship and Scholarly Communication, 2(2).
Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact
measurement. Scientometrics, 91(2), 461–471.
Mas Bleda, A., Thelwall, M., Kousha, K. & Aguillo, I. (2013). Do Highly Cited Researchers Successfully use
the Social Web. Scientometrics.
Mohammadi, E., & Thelwall, M. (in press). Mendeley readership altmetrics for the social sciences and
humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science
and Technology.
Mohammadi, E., Thelwall1, M., Haustein, S., & Larivière, V. (in press). Who Reads Research Articles? An
Altmetrics Analysis of Mendeley User Categories. Journal of the Association for Information Science and
Technology (JASIST).
Priem, Jason, Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to
explore scholarly impact. Arxiv preprint arXiv:1203.4745.
Zahedi, Z., Costas, R., & Wouters, P. (2013). How well developed are Altmetrics? Cross-disciplinary analysis
of the presence of “alternative metrics” in scientific publications. 14th International Society of
Scientometrics and Informetrics Conference (ISSI) (pp. 876–884). Vienna: Austrian Institute of Technology.
Editor's Notes
Sharing Culture
Open processes
Increased visibility
Increased collaboration
Earlier access
41.1% of all WoS papers had readership statistics in Mendeley
It is likely that academics use research articles in activities other than citing, for example in their teaching.