SlideShare a Scribd company logo
1 of 29
Download to read offline
Identifying the Invisible Impact of Scholarly Publications:
Multi-Disciplinary Analysis Using Mendeley Data
Ehsan Mohammadi
@ehsanwlv
 Traditional metrics
 Why new metrics are needed?
 Mendeley readership and altmetrics?
 Current studies
 One step forward: results of an empirical study
 Next?
 Traditionally, re-shelving statistics have been used for
research evaluation (Blecic, 1999).
 Since 1960, the ISI has paved the way for a significant
change in research evaluation with citation analysis.
 Citation-based metrics have been improved successfully.
Journal Impact Factor
H-index
And…
 Limited to authors’ perspectives.
 3-5 years needed to accumulate citations.
 More appropriate for theoretical publications.
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
Sharing Culture
Conceptualisation
• Literature reviews
• Bibliographies
The “enhanced
publication”
Findings
• Books
• Journal articles
Openness and
diversity
Translation and
engagement
• Lectures
• Presentations
Data linked, and
shareable
Data
• Datasets
• Recorded interviews
(Czerniewicz, Kell, Willmers, & King, 2014)
 The Higher Education Funding Council for England
(HEFCE), in the new Research Excellence Framework
(REF), considers all types of research impact outside
academia (HEFCE, 2011).
 The National Science Foundation (NSF) will not
evaluate researchers based on their publications
only, but non-conventional products of research are
also considered (NSF, 2013).
Altmetrics is a new movement which tries to find
complementary measures for traditional metrics based
on scholars’ activities in social web platforms (Priem, Piwowar, &
Hemminger, 2012).
Scholarly Public
Recommended Faculty of 1000 Popular press
Cited Traditional citati
on
Wikipedia
Discussed Scholarly blogs Blogs, Twitter
Read /
Bookmarking
Mendeley,
Citeulike
Delicious
 The early studies focused on availability of altmetrics
data.
 Followed by investigating correlations between
altmetrics and citations.
 Among all altmetrics, Mendeley readership counts
have the highest correlations with citations.
 In all studies low/ medium correlations between
Mendeley readership counts and citations are
reported. (Li, Thelwall, & Giustini, 2012; Priem, Piwowar, & Hemminger, 2012; Bar-Ilan,
2012; Zahedi, Costas, & Wouters, 2013;Mohammadi & Thelwall, in press).
Interpretation of low/ medium
correlations between Mendeley
readership counts and
citations!??
 What are the common types of readers for Clinical
Medicine, Engineering and Technology, Social
Science, Physics and Chemistry research articles in
Mendeley?
 Does the academic or professional status of readers
in Mendeley affect the relationship between
Mendeley readership counts and citation counts?
• Journal articles of the disciplines for year 2008 were
downloaded from Web of Science (WoS). 480K
• Mendeley data for the WoS articles was collected
automatically via the Mendeley API.
• The data sets were Matched (title, author, year).
0%
10%
20%
30%
40%
50%
60%
70%
Clinical Medicine Social Science Engineering and
Technology
Chemistry Physics
% 2008 WoS articles with readership statistics in Mendeley
(Based on NSF classification)
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
 Most readers of papers in Mendeley are PhD
students.
 Postgraduate students and postdoctoral researchers
are the two most common readers after PhD
students.
 7.2% of Clinical Medicine papers were read by
people who are not academics.
 Librarians were read a noticeable fraction of the
social science papers.
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
 There were positive correlations between Mendeley
readership and citations for all occupations except
librarians.
 The highest correlations revealed for users that are
also authors.
 The lowest correlations were found for
undergraduates and non-academic users.
 The majority of readers for all disciplines are PhD
students, postgraduates and postdocs but other
types of academics were also represented.
 Younger researchers read more papers (Tenopir, King, Spencer, &
Wu, 2009) and also cite more resources in their
publications (Larivière, Sugimoto, & Bergeron, 2013) in comparison to
senior researchers.
 Senior scholars are more reluctant to use social web
platforms (Mas-Bleda, Thelwall, Kousha, & Aguillo, in press)
 Many Clinical Medicine papers were read by non-
academics which is an important issue because some
articles could be useful in clinical practice even if
they are not cited in the literature (Jones, Donovan, & Hanney, 2012)
 Mendeley readers with authorship roles probably
reflect impact in a more similar way to traditional
citations.
 The correlations for authors are not strong enough
to claim that Mendeley readership counts and
citation counts are interchangeable.
 Students often benefit from articles that are not
highly cited and Mendeley provides an opportunity
to monitor impact on students, which probably
reflects the educational value of research articles.
 Non-academic readership counts have among the
lowest correlations with citation counts, suggesting
that their readership counts could also help to
identify individual articles and types of article that
are valuable outside academia.
 Mendeley readership is able to capture a dimension
of the impact of scientific documents on various
activities within academia such as reading without
subsequently citing, writing theses, doing
assignments or drafting research proposals.
 Mendeley readership provides a little evidence of
using papers outside academia, such as medical
doctors and surgeons.
The meaning of Mendeley readership counts depends
upon the readers’ occupations may reflect:
 Traditional citation impact .
 Educational impact.
 Impact on applied contexts.
Altmetrics =playing with numbers?
Stronger evidence is required
 Bar-Ilan, J. (2012). JASIST 2001–2010. Bulletin of Association for Information Science and Technology, 24–
28.
 Burpee, K. J., & Fernandez, L. (2014). Scholarly Communication at Canadian Research Libraries:
Conversations with Librarians. Journal of Librarianship and Scholarly Communication, 2(2).
 Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact
measurement. Scientometrics, 91(2), 461–471.
 Mas Bleda, A., Thelwall, M., Kousha, K. & Aguillo, I. (2013). Do Highly Cited Researchers Successfully use
the Social Web. Scientometrics.
 Mohammadi, E., & Thelwall, M. (in press). Mendeley readership altmetrics for the social sciences and
humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science
and Technology.
 Mohammadi, E., Thelwall1, M., Haustein, S., & Larivière, V. (in press). Who Reads Research Articles? An
Altmetrics Analysis of Mendeley User Categories. Journal of the Association for Information Science and
Technology (JASIST).
 Priem, Jason, Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to
explore scholarly impact. Arxiv preprint arXiv:1203.4745.
 Zahedi, Z., Costas, R., & Wouters, P. (2013). How well developed are Altmetrics? Cross-disciplinary analysis
of the presence of “alternative metrics” in scientific publications. 14th International Society of
Scientometrics and Informetrics Conference (ISSI) (pp. 876–884). Vienna: Austrian Institute of Technology.
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University
Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University

More Related Content

More from National Information Standards Organization (NISO)

More from National Information Standards Organization (NISO) (20)

Mattingly "Text and Data Mining: Building Data Driven Applications"
Mattingly "Text and Data Mining: Building Data Driven Applications"Mattingly "Text and Data Mining: Building Data Driven Applications"
Mattingly "Text and Data Mining: Building Data Driven Applications"
 
Mattingly "Text and Data Mining: Searching Vectors"
Mattingly "Text and Data Mining: Searching Vectors"Mattingly "Text and Data Mining: Searching Vectors"
Mattingly "Text and Data Mining: Searching Vectors"
 
Mattingly "Text Mining Techniques"
Mattingly "Text Mining Techniques"Mattingly "Text Mining Techniques"
Mattingly "Text Mining Techniques"
 
Mattingly "Text Processing for Library Data: Representing Text as Data"
Mattingly "Text Processing for Library Data: Representing Text as Data"Mattingly "Text Processing for Library Data: Representing Text as Data"
Mattingly "Text Processing for Library Data: Representing Text as Data"
 
Ross and Clark "Strategic Planning"
Ross and Clark "Strategic Planning"Ross and Clark "Strategic Planning"
Ross and Clark "Strategic Planning"
 
Mattingly "Data Mining Techniques: Classification and Clustering"
Mattingly "Data Mining Techniques: Classification and Clustering"Mattingly "Data Mining Techniques: Classification and Clustering"
Mattingly "Data Mining Techniques: Classification and Clustering"
 
Straza "Global collaboration towards equitable and open science: UNESCO Recom...
Straza "Global collaboration towards equitable and open science: UNESCO Recom...Straza "Global collaboration towards equitable and open science: UNESCO Recom...
Straza "Global collaboration towards equitable and open science: UNESCO Recom...
 
Lippincott "Beyond access: Accelerating discovery and increasing trust throug...
Lippincott "Beyond access: Accelerating discovery and increasing trust throug...Lippincott "Beyond access: Accelerating discovery and increasing trust throug...
Lippincott "Beyond access: Accelerating discovery and increasing trust throug...
 
Kriegsman "Integrating Open and Equitable Research into Open Science"
Kriegsman "Integrating Open and Equitable Research into Open Science"Kriegsman "Integrating Open and Equitable Research into Open Science"
Kriegsman "Integrating Open and Equitable Research into Open Science"
 
Mattingly "Ethics and Cleaning Data"
Mattingly "Ethics and Cleaning Data"Mattingly "Ethics and Cleaning Data"
Mattingly "Ethics and Cleaning Data"
 
Mercado-Lara "Open & Equitable Program"
Mercado-Lara "Open & Equitable Program"Mercado-Lara "Open & Equitable Program"
Mercado-Lara "Open & Equitable Program"
 
Ratner "Enhancing Open Science: Assessing Tools & Charting Progress"
Ratner "Enhancing Open Science: Assessing Tools & Charting Progress"Ratner "Enhancing Open Science: Assessing Tools & Charting Progress"
Ratner "Enhancing Open Science: Assessing Tools & Charting Progress"
 
Pfeiffer "Enhancing Open Science: Assessing Tools & Charting Progress"
Pfeiffer "Enhancing Open Science: Assessing Tools & Charting Progress"Pfeiffer "Enhancing Open Science: Assessing Tools & Charting Progress"
Pfeiffer "Enhancing Open Science: Assessing Tools & Charting Progress"
 
Hahnel “Mapping Progress: Reflections and Charting Future Pathways"
Hahnel “Mapping Progress: Reflections and Charting Future Pathways"Hahnel “Mapping Progress: Reflections and Charting Future Pathways"
Hahnel “Mapping Progress: Reflections and Charting Future Pathways"
 
Stall "Open Science: The Journey of a Scholarly Society"
Stall "Open Science: The Journey of a Scholarly Society"Stall "Open Science: The Journey of a Scholarly Society"
Stall "Open Science: The Journey of a Scholarly Society"
 
Hrynaszkiewicz "A Publisher's Perspective on Open Science"
Hrynaszkiewicz "A Publisher's Perspective on Open Science"Hrynaszkiewicz "A Publisher's Perspective on Open Science"
Hrynaszkiewicz "A Publisher's Perspective on Open Science"
 
Carter "Higher Education Leadership Initiative for Open Scholarship (HELIOS)"
Carter "Higher Education Leadership Initiative for Open Scholarship (HELIOS)"Carter "Higher Education Leadership Initiative for Open Scholarship (HELIOS)"
Carter "Higher Education Leadership Initiative for Open Scholarship (HELIOS)"
 
Webster "NISO Roundtable: Open Science and Libraries"
Webster "NISO Roundtable: Open Science and Libraries"Webster "NISO Roundtable: Open Science and Libraries"
Webster "NISO Roundtable: Open Science and Libraries"
 
Ratan "Are we there yet? Keeping the promise of open science"
Ratan "Are we there yet?  Keeping the promise of open science"Ratan "Are we there yet?  Keeping the promise of open science"
Ratan "Are we there yet? Keeping the promise of open science"
 
Tananbaum "NISO Tech Summit: Reflections on Reflections Upon The Year of Open...
Tananbaum "NISO Tech Summit: Reflections on Reflections Upon The Year of Open...Tananbaum "NISO Tech Summit: Reflections on Reflections Upon The Year of Open...
Tananbaum "NISO Tech Summit: Reflections on Reflections Upon The Year of Open...
 

Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics - Ehsan Mohammadi, Ph.D. Candidate, Wolverhampton University

  • 1. Identifying the Invisible Impact of Scholarly Publications: Multi-Disciplinary Analysis Using Mendeley Data Ehsan Mohammadi @ehsanwlv
  • 2.  Traditional metrics  Why new metrics are needed?  Mendeley readership and altmetrics?  Current studies  One step forward: results of an empirical study  Next?
  • 3.  Traditionally, re-shelving statistics have been used for research evaluation (Blecic, 1999).  Since 1960, the ISI has paved the way for a significant change in research evaluation with citation analysis.  Citation-based metrics have been improved successfully. Journal Impact Factor H-index And…
  • 4.  Limited to authors’ perspectives.  3-5 years needed to accumulate citations.  More appropriate for theoretical publications.
  • 6. Sharing Culture Conceptualisation • Literature reviews • Bibliographies The “enhanced publication” Findings • Books • Journal articles Openness and diversity Translation and engagement • Lectures • Presentations Data linked, and shareable Data • Datasets • Recorded interviews (Czerniewicz, Kell, Willmers, & King, 2014)
  • 7.  The Higher Education Funding Council for England (HEFCE), in the new Research Excellence Framework (REF), considers all types of research impact outside academia (HEFCE, 2011).  The National Science Foundation (NSF) will not evaluate researchers based on their publications only, but non-conventional products of research are also considered (NSF, 2013).
  • 8. Altmetrics is a new movement which tries to find complementary measures for traditional metrics based on scholars’ activities in social web platforms (Priem, Piwowar, & Hemminger, 2012).
  • 9. Scholarly Public Recommended Faculty of 1000 Popular press Cited Traditional citati on Wikipedia Discussed Scholarly blogs Blogs, Twitter Read / Bookmarking Mendeley, Citeulike Delicious
  • 10.  The early studies focused on availability of altmetrics data.  Followed by investigating correlations between altmetrics and citations.  Among all altmetrics, Mendeley readership counts have the highest correlations with citations.  In all studies low/ medium correlations between Mendeley readership counts and citations are reported. (Li, Thelwall, & Giustini, 2012; Priem, Piwowar, & Hemminger, 2012; Bar-Ilan, 2012; Zahedi, Costas, & Wouters, 2013;Mohammadi & Thelwall, in press).
  • 11. Interpretation of low/ medium correlations between Mendeley readership counts and citations!??
  • 12.  What are the common types of readers for Clinical Medicine, Engineering and Technology, Social Science, Physics and Chemistry research articles in Mendeley?  Does the academic or professional status of readers in Mendeley affect the relationship between Mendeley readership counts and citation counts?
  • 13. • Journal articles of the disciplines for year 2008 were downloaded from Web of Science (WoS). 480K • Mendeley data for the WoS articles was collected automatically via the Mendeley API. • The data sets were Matched (title, author, year).
  • 14. 0% 10% 20% 30% 40% 50% 60% 70% Clinical Medicine Social Science Engineering and Technology Chemistry Physics % 2008 WoS articles with readership statistics in Mendeley (Based on NSF classification)
  • 17.  Most readers of papers in Mendeley are PhD students.  Postgraduate students and postdoctoral researchers are the two most common readers after PhD students.  7.2% of Clinical Medicine papers were read by people who are not academics.  Librarians were read a noticeable fraction of the social science papers.
  • 19.  There were positive correlations between Mendeley readership and citations for all occupations except librarians.  The highest correlations revealed for users that are also authors.  The lowest correlations were found for undergraduates and non-academic users.
  • 20.  The majority of readers for all disciplines are PhD students, postgraduates and postdocs but other types of academics were also represented.  Younger researchers read more papers (Tenopir, King, Spencer, & Wu, 2009) and also cite more resources in their publications (Larivière, Sugimoto, & Bergeron, 2013) in comparison to senior researchers.  Senior scholars are more reluctant to use social web platforms (Mas-Bleda, Thelwall, Kousha, & Aguillo, in press)
  • 21.  Many Clinical Medicine papers were read by non- academics which is an important issue because some articles could be useful in clinical practice even if they are not cited in the literature (Jones, Donovan, & Hanney, 2012)
  • 22.  Mendeley readers with authorship roles probably reflect impact in a more similar way to traditional citations.  The correlations for authors are not strong enough to claim that Mendeley readership counts and citation counts are interchangeable.
  • 23.  Students often benefit from articles that are not highly cited and Mendeley provides an opportunity to monitor impact on students, which probably reflects the educational value of research articles.  Non-academic readership counts have among the lowest correlations with citation counts, suggesting that their readership counts could also help to identify individual articles and types of article that are valuable outside academia.
  • 24.  Mendeley readership is able to capture a dimension of the impact of scientific documents on various activities within academia such as reading without subsequently citing, writing theses, doing assignments or drafting research proposals.  Mendeley readership provides a little evidence of using papers outside academia, such as medical doctors and surgeons.
  • 25. The meaning of Mendeley readership counts depends upon the readers’ occupations may reflect:  Traditional citation impact .  Educational impact.  Impact on applied contexts.
  • 26. Altmetrics =playing with numbers? Stronger evidence is required
  • 27.  Bar-Ilan, J. (2012). JASIST 2001–2010. Bulletin of Association for Information Science and Technology, 24– 28.  Burpee, K. J., & Fernandez, L. (2014). Scholarly Communication at Canadian Research Libraries: Conversations with Librarians. Journal of Librarianship and Scholarly Communication, 2(2).  Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471.  Mas Bleda, A., Thelwall, M., Kousha, K. & Aguillo, I. (2013). Do Highly Cited Researchers Successfully use the Social Web. Scientometrics.  Mohammadi, E., & Thelwall, M. (in press). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology.  Mohammadi, E., Thelwall1, M., Haustein, S., & Larivière, V. (in press). Who Reads Research Articles? An Altmetrics Analysis of Mendeley User Categories. Journal of the Association for Information Science and Technology (JASIST).  Priem, Jason, Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. Arxiv preprint arXiv:1203.4745.  Zahedi, Z., Costas, R., & Wouters, P. (2013). How well developed are Altmetrics? Cross-disciplinary analysis of the presence of “alternative metrics” in scientific publications. 14th International Society of Scientometrics and Informetrics Conference (ISSI) (pp. 876–884). Vienna: Austrian Institute of Technology.

Editor's Notes

  1. Sharing Culture Open processes Increased visibility Increased collaboration Earlier access
  2. 41.1% of all WoS papers had readership statistics in Mendeley
  3. It is likely that academics use research articles in activities other than citing, for example in their teaching.