William Gunn discusses altmetrics from Mendeley's perspective. He outlines what Mendeley knows and still needs to understand about altmetrics, including how to predict impact, capture all mentions, and adjust for cultural differences. Gunn also discusses Mendeley's work with the Reproducibility Initiative to replicate highly cited papers, and their focus on improving recommendations, data quality, and building relationships with developers.

1. Altmetrics at Mendeley
William Gunn, Ph.D. Head of Academic Outreach Mendeley @mrgunn

2. Two audiences
•The information science community
–What we know & what we’re still trying to understand
–What we think important questions are
•The altmetrics community
–Where Mendeley is going
–What we think are the important things to address

3. What we think we know
•Where we are: discovery, but not assessment – we can describe, but not predict
•What it means to correlate with citations
•What we’re really measuring – attention
–who listens to you and who do you listen to
•minus the people who are listened to, but don’t listen well
•Lit derived metrics are not enough!

4. Amgen: 47 of 53 “landmark” oncology publications could not be reproduced
Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results
Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible
There is no gold standard

5. We didn’t see that a target is more likely to be validated if it was reported in ten publications or in two publications
NATURE REVIEWS DRUG DISCOVERY 10, 712 (SEPTEMBER 2011)

6. Either the results were reproducible and showed transferability in other models, or even a 1:1 reproduction of published experimental procedures revealed inconsistencies between published and in-house data
NATURE REVIEWS DRUG DISCOVERY 10, 712 (SEPTEMBER 2011)

7. Building a reproducibility dataset
•Mendeley and Science Exchange have started the Reproducibility Initiative
•$1.3M grant from LJAF to Initiative via Center for Open Science
•50 most highly cited & read papers from 2010, 2011, and 2012 will be replicated
•Figshare & PLOS to host data & replication reports

8. What we don’t know
•What we can predict
–need to understand intent, imported or derived reputation
•How to capture all mentions, even without direct identifiers
–what skew is there, and what does it mean
•How to adjust for regional or cultural differences

9. Cultural skew is important
South America is weak on N.A. social media, strong on Mendeley

10. How to understand sources of variability
•Collect the same set of metrics at different times, by different people, using different methods
•This will inform the standards process & assist IS people with capturing provenance, doing preservation, and giving advice

11. What are the important questions we aren’t asking yet?
•let’s get past the “ranking people by their Twitter followers” stuff
•Tell us what we should be looking at and how you would like to be involved

12. What people want to know about Mendeley
•We realize what we do makes a big difference
–RG/Academia began to do more once we showed the potential
–Researchers value our coverage and source neutrality
–Many consume our data, even when it’s crappy

13. Focusing on recommendations
•Mendeley Suggest
–personalized recommendations based on reading history
•related articles
–relatedness based on document similarity
•recommender frameworks
–implement recommendations as a service
•third-party recommender services
–serve niche audiences

14. improving data quality
•Research Catalog v2
–better duplicate detection
–readership numbers stable
•only increase
–canonical docs
•API v2
–exposing more information
•annotations
•other events (what do you want to see?)

15. Stability and Security
•We are serious
–adapting to and promoting changes in practice
•investing in building relationships with developers
•platform, not a silo

16. TEAM Project
academic knowledge management solutions
•Algorithms to determine the content similarity of academic papers
•Performing text disambiguation and entity recognition to differentiate between and relate similar in-text entities and authors of research papers.
•Developing semantic technologies and semantic web languages with the focus of metadata integration/validation
•Investigate profiling and user analysis technologies, e.g. based on search logs and document interaction.
•We will also improve folksonomies and through that, ontologies of text.
• Finally, tagging behaviour will be analysed to improve tag recommendations and strategies.
•http://team-project.tugraz.at/blog/

17. Code Project
Use case = mining research papers for facts to add to LOD repositories and light-weight ontologies.
•Crowd-sourcing enabled semantic enrichment & integration techniques for integrating facts contained in unstructured information into the LOD cloud
•Federated, provenance-enabled querying methods for fact discovery in LOD repositories
•Web-based visual analysis interfaces to support human based analysis, integration and organisation of facts
•http://code-research.eu/

21. Semantics vs. Syntax
•Language expresses semantics via syntax
•Syntax is all a computer sees in a research article.
•How do we get to semantics?
•Topic Modeling!

22. Distribution of Topics
0%
5%
10%
15%
20%
25%
30%
35%

23. Subcategories of Comp. Sci.
0%
5%
10%
15%
20%
AI
HCI
Info Sci
Software
Eng
Networks

24. www.mendeley.com
william.gunn@mendeley.com @mrgunn