Altmetrics trends - A survey of the #altmetrics landscape
•Download as PPTX, PDF•
0 likes•194 views
The document discusses the growing interest in altmetrics for measuring the impact of scholarly research. It provides an overview of various altmetric data aggregators and tools, such as Altmetric.com, Plum Analytics, Impact Story, and Crossref's DOI Event Tracker. The document also examines issues with altmetrics, such as concerns about the reliability, quality, and consistency of altmetric data.
Recommended
More Related Content
Viewers also liked
Viewers also liked (19)
Similar to Altmetrics trends - A survey of the #altmetrics landscape
Similar to Altmetrics trends - A survey of the #altmetrics landscape (20)
Recently uploaded
Recently uploaded (20)
Altmetrics trends - A survey of the #altmetrics landscape
- 1. Measuring Science William Gunn Director of Scholarly Communications, Mendeley http://orcid.org/0000-0002-3555-2054
- 2. Growing interest in altmetrics • 5+ million users • 1+ billion read events
- 3. Growing interest in altmetrics Bibliometrics Altmetrics Source: Google - https://www.google.com/trends/explore#q=bibliometrics%2Caltmetrics
- 5. Raw DataQuantitativeQualitative Aggregator Academia.edu, ResearchGate, and others don’t make their data available to third parties and thus aren’t considered here.
- 7. A quick tour of the altmetrics landscape
- 12. Altmetric.com
- 13. Plum Analytics Page Widgets Dashboards
- 14. Impact Story
- 16. Pubmed Commons
- 17. PubPeer
- 18. Publons
- 19. Crossref DOI Event Tracker
- 20. Issues and challenges •Trust –Is the data reliable? •Quality –Is the data complete, accurate, available? •Standards –Consistency in collecting and reporting the data
- 21. NISO ALMI Initiative 2013 – 2014 Phase I 3 meetings, 30 one-on-one interviews 1 White Paper 2014-2015 Phase II Working Groups: Definitions Sources Identifiers Quality
- 22. New metrics matter http://www.mendeley.com/groups/586171/altmetrics/ • Li X, Thelwall M, Giustini D (2012) Validating online reference managers for scholarly impact measurement. Scientometrics 91: 461–471. • Xuan Liang, et al. (2014) Building Buzz: (Scientists) Communicating Science in New Media Environments Journalism & Mass Communication Quarterly December 2014 91: 772-791 http://dx.doi.org/10.1177/1077699014550092 • Li X, Thelwall M (2012) F1000, Mendeley and traditional bibliometric indicators. In: Proceedings of the 17th International Conference on Science and Technology Indicators. Montréal, Canada. pp. 451–551. http://www.altmetricsconference.com/ http://altmetrics.org/altmetrics15/
Editor's Notes
- Hi, I’m William Gunn, Director of Scholarly Communications for Mendeley. In the next 30 minutes, I would like to briefly explain what altmetrics are, how they’re related to other metrics, and how they’re being used at Mendeley and other services you may have used or encountered researchers using.
- First, let me tell you why this topic is important enough that we chose to include it in this program. As information science research is starting to show and as the Doge of the Internet proclaims, altmetrics are growing in interest and importance. Scholars are increasingly adopting social media for the discussion and sharing of scholarly content, with Mendeley and Twitter being the largest sources in terms of coverage and volume of events. Mendeley has the largest coverage, which means, for any given article, you’re most likely to find some activity on Mendeley about it relative to other sources. Twitter, owing to sheer size, has the largest volume of events, though only a few percent of articles in any academic discipline are discussed on Twitter, mostly focusing around breaking news, such as the Fukushima nuclear disaster, natural disasters, or health and wellness news of general interest, like the recent news about processed meat being classified as a carcinogen.
- Relative to bibliometrics, a term with which most of you will be familiar, altmetrics has drawn more interest, as measured by Google search volume, for the past two years, and has really stolen the show from bibliometrics, which has been in decline for the past several years.
- So given that altmetrics is a component of the scholarly ecosystem that’s here to stay, we should probably understand what the alt- part of altmetrics means. One way to think about it is as an alternative to the traditional focus on citations, and another alt- perspective is the focus on a broader universe of scholarly content than just the paper. This schematic shows one view of such as expanded universe of both metrics and objects to which those metrics pertain. One could think of these boxes somewhat like buckets, into which fall various types of activities from which both qualitative and quantitative indicators are derived. Quantitative indicators are the metrics which are the focus on this talk, such as Mendeley readers, but the qualitative dimension, which includes things like blog posts written for a lay audience or tweets referencing a paper highlight the importance of being able to drill into the quantitative number of something into the content from which that event derived. A concrete example here may help: Counting the number of tweets containing a given paper title or DOI may or may not be all that useful, but looking at the actual substance of the tweets is always informative. A tweet from the account of a Nobel laureate is different from am automatically generated tweet when an article goes live, which is different from a jokey tweet about a paper with a funny title. In order to allow the consumer of any metric to make the best decision for them, the current best practice in how the qualitative and quantitative are reconciled in altmetrics is to always provide the raw data underlying any number. These best practices are being developed in a NISO working group, a link to which you can find at the end of this presentation.
- So in addition to the qualitative and quantitative dimension I mentioned earlier, there’s an additional dimension of raw data vs. aggregator. Mendeley, Scopus, and Figshare, for example, provide numbers that are as close to raw as possible. With citations, you can see both how many there are and exactly which papers are citing which. Due to the rapid time-scale and privacy concerns, Mendeley shows how many readers a paper has, but does not give detailed information on exactly who those readers are. Over time, this may shift to become more open like citations, but for the moment, we provide the academic status (phd, postdoc, etc), the country, and the discipline of the reader. Moving more in the qualitative direction, you see the post-publication peer review services such as Pubpeer and Publons, but also Pubmed Commons. Moving in the aggregator dimension, you see services such as Impactstory, Altmetric.com, and Plum Analytics, but also Elsevier which is now providing expanded article-level metrics in Scopus. Kudos is an interesting service which takes a hybrid approach. Underscoring the transparency aspect I mentioned earlier, Academia.edu and ResearchGate don’t fit into this schema, because they only provide a opaque number with no ability to drill into the underlying events, but also don’t allow their data to be harvested by aggregators.
- I do just want to briefly mention that non-citation based metrics are nothing new, but what is new is the integration of these into a unified framework with citation based metrics and the expanded universe of scholarly content, including datasets, code, presentations, blog posts, tweets, etc.
- Now let’s take a quick look at some of the major players individually.
- As the source with the greatest coverage, and as my privilege as presenter, I’m going to talk about Mendeley first. You can see the reader numbers I mentioned used in a few different ways on our site. On the catalog pages at mendeley.com, on the right, you can see this paper about the hot topic of next-gen sequencing has 32k readers. you can see a little bit of the demographic breakdown of these, and you can also see how journals stack up based on the number of articles being read from those journals. One thing to highlight here is that while you do see many of the usual suspects in the top 10 list, journals from which the articles are freely available do rank a bit higher, providing a bit of circumstantial evidence that open access publishing is actually important in getting your paper to everyone who would like to read it.
- In the Scopus catalog, Mendeley readers are also available, in addition to select items from other activity buckets. One important thing to note here is that in addition to the number, the percentile ranking of the number is shown. That helps a reader to put the number into context. The individual metrics shown here on the sidebar are linked to the metrics tab, which has further information, going all the way to seeing the individual tweets in the case of twitter, or demographics, in the case of Mendeley.
- Here’s an example of this further information – across the top of the page are the various activity buckets, with this overview page showing the numbers at a glance.
- Drilling further into the social media bucket, you can see the individual tweets, FB posts, and other items.
- Another major player in the altmetrics space is altmetric.com As a technical note, because this frequently confuses people, there’s no s on the company name – altmetrics refers to the topic, altmetric (no s).com refers to the company. If you try to go to altmetrics.com, you won’t get anywhere. It’s altmetrics.org or altmetric.com. OK, with that bit of detail out of the way, let’s talk about the company. The main place you’ll encounter them is on a publisher’s article pages, where their colorful little widget shows some of the numbers they aggregate. They don’t provide the percentile rankings, but you can get down to the raw data in many cases, which helps you understand what the numbers mean.
- Plum Analytics targets institutions, and they have a similar sort of widget that appears on some repository pages, again without much context, but the neat thing about this is that it works even for content that doesn’t have a DOI, whereas altmetric.com is DOI-based. I also like that they avoid the one-number-to-rule-them-all kind of approach that altmetric.com gives into. They also provide a dashboard so you can see analytics on content across various groupings such as by year, by researcher, or by collection.
- In contrast to publishers or institutions, Impact Story focuses on the researcher. It sells itself as a CV, but better. On the left you can see the range of scholarly content types we’ve talked about, with an automatically updated set of metrics appropriate for each type. The use of decals provides context and badges add a layer of behavioral nudges to promote good scholarly behaviors such as publishing open access or utilizing reproducible methodologies.
- The hybrid approach of Kudos is apparent here in a page from their site. On one tab there are lay summaries from the authors and on another tab there’s a summary of metrics. In the case of Kudos, the metrics are derived partly from activity on their site and partly from other sources, but are shown only to the authors, so I’ll say nothing further about those.
- Moving to more qualitative sources now, let’s take a look at Pubmed Commons. At Pubmed, if you are a published author, you can comment on abstracts on Pubmed. There’s a lot of potential for this to become a major source of commentary based solely on the brand recognition of pubmed and their huge amount of traffic relative to something like PubPeer or even Mendeley. One thing to note here is that the author of the comment is required to sign it with their real name, which is designed to prevent anonymous abuse, but which can also have the effect of suppressing certain voices, particularly those of younger researchers or in other cases where there’s a power imbalance between the commenter and the authors of the paper being commented upon.
- As a counterpoint to Pubmed Commons, we have Pubpeer. Comments on Pubpeer do not require registration or disclosure of identity. This has led to a robust culture of critique on the site, but it’s worthwhile noting that allegations of fraud don’t run free here. Commentary is still expected to be fact-based and well-supported and the PubPeer community tends to enforce such norms, providing a strong case that names are required to prevent abusive allegations. It’s also worth pointing out another project called Hypothesis, which isn’t strictly a altmetrics product but rather bills itself as a peer review layer for the web in general, has found a hybrid approach. They allow anonymous commenting, but it’s also not trivially easy for one person to create tons of “sockpuppet” accounts that pretend to have a conversation with each other to create false impressions or derail discussions.
- Publons is an effort which seeks to help researchers claim credit for the reviewing work that they do. You can see here that this researcher has claimed 100 reviews, pre-publication, which puts her in the 98% percentile relative to a base population which needs to be made apparent. You can also see some behavioral nudges in the openness score, but it’s a bit hard to understand at first what this means, and the merit score is an example of one of those derived numbers that are generally bad practice.
- Finally, if you’ve decided that altmetrics are for you but each of the efforts mentioned before are unusable for some reason, there’s software you can run yourself and freely modify to your liking. This software is called Lagotto and shown here implemented by Crossref as the DOI Event Tracker. The software receives inputs from sources about works registered with it and reports those as events – all of which are highly user-modifiable. For more on this see: http://lagotto.io
- So that’s it for the whistlestop tour of the altmetrics landscape. Many of you have questions in your mind about the quality of the data, the utility of it for your use case, the role of commercial players in the ecosystem, and how to address provenance and preservation of these data. Standards to address some of these issues are in the works and expected to, very roughly, follow the outlines of the COUNTER project.
- Starting in 2013, an Initiative was begun at NISO to discuss these and other issues, they have resulted in a white paper which I invite you to read for more information. Workgroups on Definitions, Sources, Identifiers, and Quality have been convened and there will be an invitation for public comment before the publication of recommendations and best practices by NISO.
- For further information, see the Mendeley Altmetrics group and get involved in the community, which primarily convenes around these two meetings.
- I hope this has been interesting and informative. Please feel free to get in touch with questions via email or tweet me @mrgunn. Thanks for your attention!