Next-generation Metrics for
Scholarly Performance
William Gunn, Ph.D.
Head of Academic Outreach
Mendeley
@mrgunn
https://o...
Open Access Policies
Open Access Has Benefits
• The perception that there were tradeoffs
when choosing open access slowed
adoption
• Only Open ...
King, Christopher (2012) Thomson Reuters Annual Report
http://ar.thomsonreuters.com/_files/pdf/MultiauthorPapers_ChrisKing...
Adams, Jonathan. "Collaborations: the fourth age of research."
Nature 497.7451 (2013): 557-560.
 Amgen: 47 of 53 “landmark” oncology publications could
not be reproduced
 Bayer: 43 of 67 oncology & cardiovascular pro...
Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph
publishing." new media & society 15.3 (2013...
Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph
publishing." new media & society 15.3 (2013...
AuthorsReaders Internet
Research is changing
Publishers
Repositories
Professional
Websites
Challenges for Repositories
• The world is leaning green
• Repositories thus have an
essential role
• Repositories aren’t ...
The Ouroboros problem
• repositories need content
deposited to get usage, and
they need usage to get
deposits
• Discovery ...
New forms of discovery
• personalized recommendations based on
reading history
• related articles
• recommender frameworks...
research impacts more than authors
http://altmetrics.org/manifesto/
...and aggregates research
data in the cloud
Mendeley extracts
research data…
Install
Mendeley Desktop
Collecting rich sig...
Home Work Mobile
Cloud Library
Shared Folder
Mendeley Research Catalog
Mendeley Research Catalog
Read papers + keep track of notes
470M documents
Taking some misery out of writing
Information Extraction
We are publishing this data to the LOD cloud
http://code-research.eu/
Defining readership
• Each document addition is a
“read”
• stamped with metadata
describing the context of the read
event
...
research impacts more than authors
http://altmetrics.org/manifesto/
Readership vs. citations
• it comes with a payload of
metadata
• it accrues faster
• it illuminates previously
hidden impa...
http://arxiv.org/html/1203.4745v1
altmetrics show broader impact
http://dx.doi.org/10.3789/isqv25no2.2013.04
http://www.asis.org/Bulletin/Apr-13/AprMay13_Lin_Fenner.html
altmetrics show broader impact
Issues To Be Addressed
• Identity
• Privacy
• Attribution
• Gaming
• Filtration
standards/
best practice
 Amgen: 47 of 53 “landmark” oncology publications could
not be reproduced
 Bayer: 43 of 67 oncology & cardiovascular pro...
Cultural skew is important
South America is weak on N.A. social media, strong on Mendeley
What would people build if they
could get the data?
• Impact Story – get credit for all your work
• PLOS ALM – article-lev...
www.mendeley.com
william.gunn@mendeley.com
@mrgunn
 OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley
 OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley
Upcoming SlideShare
Loading in …5
×

OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

1,075 views

Published on

Presentation at the OpenAIRE-COAR Conference: "Open Access Movement to Reality: Putting the Pieces Together", Athens - May 21-22, 2014.
Session 4: The impact of openness and how to evaluate research.
Next generation metrics of scholarly performance, by William Gunn - Head of Academic Outreach for Mendeley

Published in: Science, Business, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,075
On SlideShare
0
From Embeds
0
Number of Embeds
53
Actions
Shares
0
Downloads
14
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

  1. 1. Next-generation Metrics for Scholarly Performance William Gunn, Ph.D. Head of Academic Outreach Mendeley @mrgunn https://orcid.org/0000-0002-3555-2054
  2. 2. Open Access Policies
  3. 3. Open Access Has Benefits • The perception that there were tradeoffs when choosing open access slowed adoption • Only Open Access can fully provide the benefits of the web – interoperability, accessibility, discovery • An expanded definition of impact brings new kinds of content and new metrics
  4. 4. King, Christopher (2012) Thomson Reuters Annual Report http://ar.thomsonreuters.com/_files/pdf/MultiauthorPapers_ChrisKing.pdf
  5. 5. Adams, Jonathan. "Collaborations: the fourth age of research." Nature 497.7451 (2013): 557-560.
  6. 6.  Amgen: 47 of 53 “landmark” oncology publications could not be reproduced  Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results  Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible There is no gold standard
  7. 7. Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph publishing." new media & society 15.3 (2013): 359-378. Authors Publishers Printers Shippers Booksellers Readers
  8. 8. Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph publishing." new media & society 15.3 (2013): 359-378. Authors Publishers Printers Shippers Booksellers Readers 1770
  9. 9. AuthorsReaders Internet Research is changing Publishers Repositories Professional Websites
  10. 10. Challenges for Repositories • The world is leaning green • Repositories thus have an essential role • Repositories aren’t in the workflow of a researcher, neither in search for new ideas nor in presentation of their work
  11. 11. The Ouroboros problem • repositories need content deposited to get usage, and they need usage to get deposits • Discovery tools can address usage and usage will promote content – scholarly outputs are papers, datasets, code, and more.
  12. 12. New forms of discovery • personalized recommendations based on reading history • related articles • recommender frameworks –implement recommendations as a service • third-party recommender services –serve niche audiences
  13. 13. research impacts more than authors http://altmetrics.org/manifesto/
  14. 14. ...and aggregates research data in the cloud Mendeley extracts research data… Install Mendeley Desktop Collecting rich signals from domain experts.
  15. 15. Home Work Mobile Cloud Library
  16. 16. Shared Folder
  17. 17. Mendeley Research Catalog
  18. 18. Mendeley Research Catalog
  19. 19. Read papers + keep track of notes 470M documents
  20. 20. Taking some misery out of writing
  21. 21. Information Extraction
  22. 22. We are publishing this data to the LOD cloud http://code-research.eu/
  23. 23. Defining readership • Each document addition is a “read” • stamped with metadata describing the context of the read event • a read is like a citation, but faster and captures more
  24. 24. research impacts more than authors http://altmetrics.org/manifesto/
  25. 25. Readership vs. citations • it comes with a payload of metadata • it accrues faster • it illuminates previously hidden impact
  26. 26. http://arxiv.org/html/1203.4745v1 altmetrics show broader impact
  27. 27. http://dx.doi.org/10.3789/isqv25no2.2013.04
  28. 28. http://www.asis.org/Bulletin/Apr-13/AprMay13_Lin_Fenner.html altmetrics show broader impact
  29. 29. Issues To Be Addressed • Identity • Privacy • Attribution • Gaming • Filtration standards/ best practice
  30. 30.  Amgen: 47 of 53 “landmark” oncology publications could not be reproduced  Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results  Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible There is no gold standard
  31. 31. Cultural skew is important South America is weak on N.A. social media, strong on Mendeley
  32. 32. What would people build if they could get the data? • Impact Story – get credit for all your work • PLOS ALM – article-level metrics for papers • Plum Analytics – bespoke analytics for libraries (EBSCO) • Altmetric.com – altmetrics for publishers. (Digital Science)
  33. 33. www.mendeley.com william.gunn@mendeley.com @mrgunn

×