Measuring scholarly impact has traditionally been tied to the calculation of a scholarly article’s number of citations and the Impact Factor of its journal. Today, however, scholarly contributions take many forms: computer code, data sets, blog postings, tweets, practice guidelines and beyond. As the products of research evolve, so will the way in which credit is measured. This class will provide an overview of “altmetrics”, the movement to assess influence of both traditional and non-traditional scholarly contributions. We will define altmetrics, discuss why it is important in today’s digital scholarly environment, and demonstrate tools available to measure influence. After completing this course, the learner will be able to define altmetrics and compare it to traditional forms of measuring scholarly impact; name examples of scholarly contributions that are alternatives to traditional methods (e.g. datasets, blog postings, tweets, etc.); name examples of alternative means of measuring scholarly contributions (e.g. download counts, tweets about, etc.); discuss why today’s online, social environment necessitates a change in the way scholarly contributions are measured; name resources to learn more about altmetrics such as altmetrics.org; and name tools to measure alternative scholarly contributions such as Altmetric.com, Impact Story, Plum Analytics, etc.

1. Altmetrics: the movement, the
tools, and the implications
Kimberley R. Barker, MLIS
Andrea H. Denton, MILS
July 2017

2. Defining altmetrics
• J. Priem (@jasonpriem), I like the term
#articlelevelmetrics, but it fails to imply
*diversity* of measures. Lately, I'm liking
#altmetrics., 4:28 AM - 29 Sep 10, Tweet
• “…the creation and study of new metrics
based on the Social Web for analyzing, and
informing scholarship.”
– http://altmetrics.org/about/

3. Defining altmetrics
• Metrics that supplement or complement
traditional metrics

4. Awareness,
not
promotion!

5. From metrics to altmetrics
Measures
Traditional
Research
Products
Traditional
- Article
- Chapter
- Book
Times Cited
Impact Factor + Rank
H-index

6. From metrics to altmetrics
Measures
Traditional New
Research
Products
Traditional
- Article
- Chapter
- Book
Times Cited
Impact Factor +
Rank
H-index
Page Views
Downloads

7. From metrics to altmetrics
Measures
Traditional New
Research
Products
Traditional
- Article
- Chapter
- Books
Times Cited
Impact
Factor +
Rank
H-index
Page Views
New
- Dataset
- Blog post
- More
None

8. From metrics to altmetrics
Measures
Traditional New
Research
Products
Traditional
- Article
- Chapter
- Books
Times Cited
Impact Factor
+ Rank
H-index
Page Views
New
- Datasets
- Blog post
- More
None News stories
Blog mentions
Tweets

9. From metrics to altmetrics
Measures
Traditional New
Research
Products
Traditional
- Article
- Chapter
- Books
Times Cited
Impact Factor
+ Rank
H-index
Page Views
News stories
Blog mentions
Tweets
New
- Datasets
- Blog post
- More
None News stories
Blog mentions
Tweets

10. Additional scholarly contributions
• Blogs
• Invited Interviews
• Twitter
• Facebook postings
• Reddit
• Datasets
• Patents
• Software
• Copyrights

11. Examples of “altmeasuring”
• Downloads and page views
• Track-backs
• Tweets and retweets
• Links from review services (e.g. Facultyof1000)
• Sharing, social bookmarking
• News media

12. New metrics for traditional products

13. Newer metrics for traditional products
Br J Sports Med doi:10.1136/bjsports-2013-092417

14. New metrics for new products

15. Other influences
NSF “Publications” broadened to “Products
of Research” (Jan 2013)
• “citable and accessible including but not
limited to publications, data sets,
software, patents, and copyrights.“

16. Other influences
NIH Biosketch new format (Jan 2015)
• other non-publication research products,
including audio or video products; patents; data
and research materials; databases; educational
aids or curricula; instruments or equipment;
models; protocols; and software or netware…

17. Tools

18. • Measure web views and downloads
–Google Analytics
–Bit.ly
• Measure views and reads of articles
• Google Profiles
• ResearchGate
Early altmetric tools

19. Current Tools
• Impactstory
• Altmetric.com
• PlumX
• New! iCite (NIH Relative Citation Ratio)
• New! CiteScore (Elsevier)

20. Impactstory
• Create an online profile
– Discover and share how your research is read,
cited, tweeted, bookmarked, and more
– Help colleagues find and read your preprints,
articles, slides and other work by uploading
research products straight your profile
• Jason Priem and Heather Piwowar
• Free for 30 days, then $60 a year.

21. Sample Impactstory Profile

22. Altmetric.com
• London-based start-up
• Funding from Digital Science (LabGuru, FigShare)

23. Altmetric’s widget (“donut”)
• Used by publishers/journals
–Nature Publishing
–Cell Press
–Wiley ?
–BioMed Central
–Taylor & Francis
–BMJ Specialty journals

24. What sources does Altmetric track?
News outlets
• Over 1,300 sites
• Manually curated list
• Text mining
• Global coverage
Social media
and blogs
• Twitter, Facebook,
Google+, Sina Weibo
• Public posts only
• Manually curated list
Reference
managers
• Mendeley, CiteULike
• Reader counts
• Don’t count towards the
Altmetric score
Other sources
• Wikipedia
• YouTube
• Reddit
• F1000
• Pinterest
• Q&A
Post-publication
peer review
• Publons
• PubPeer
Policy documents
• NICE Evidence
• Intergovernmental Panel
on Climate Change
• Many more…

25. What is the Altmetric donut?
The Altmetric donut visualizes which are sources discussing an item.
In the Altmetric Explorer, hover the cursor over the donut visualization to
see the appropriate legend for an item.
Each source is colour coded:

26. What can the data tell you?
What type of attention is this research
receiving?
Where has this article received the most
traction?
Which countries are engaging most with
the content?
Has this article influenced policy, spurred
new research, or engaged a new audience?
Are reactions to this article positive or negative?
✓
✓
✓
✓
✓

27. Remember that the numbers don’t tell
you…
Quality of the paper
Quality of the researchers
Whole story

28. Nature Neuroscience

29. Altmetric Bookmarklet
• Free
• Reading a paper and
want to find out its
Altmetric details?
• Install the
bookmarklet in your
browser
• When viewing the
paper, “Altmetric it”

30. Altmetric Bookmarklet

31. Altmetric Bookmarklet

32. Plum Analytics
• PlumX is an institutional “impact dashboard”
that provides information on how research
output is being utilized, interacted with, and
talked about around the world
• Gathers metrics (altmetrics) about research
from more than thirty sources including PLOS,
PubMed and YouTube, and categorizes them

33. Not as “alt”, but new…
• Recognition that JIF not the only solution
– Relative Citation Ratio (NIH)
– CiteScore (Elsevier)
• Comparison within a subset
– NIH funded articles
– PLoS articles

34. Relative Citation Ratio
• From NIH
• For papers (biomedical (PubMed))
• Field-normalized; relative to the average NIH-
funded paper
• Use the iCite tool
https://icite.od.nih.gov/analysis

35. CiteScore
• From Elsevier
• For journal titles
• Part of a “basket of metrics”

36. However…

37. • Standards aren’t fully defined
– Definitions, calculations, etc.
– NISO effort
• Are altmetrics important for discovery?
For evaluation? Both?
Issues

38. Issues
• Impact vs. attention
–David C.’s Improbable Science… “Why you
should ignore altmetrics and other
bibliometric nightmares”
http://www.dcscience.net/?p=6369
• Popularity
–Popular topics get higher counts, quickly,
but then fade. How does this reflect
quality?

39. Issues
• Too much concern on metrics (“culture of
measurement” “yelpification”)
• Does social media help promote good
science? Or not? (e.g. anti-vaccine)

40. Altmetrics: where to start?
Altmetrics for Researchers (Duke University Medical Library)

41. What are your products?
• Paper, chapter, book?
• A clinical protocol?
• Software code?
• Conference poster?
• Teaching material?
• White paper?
• Data set

42. Where are your products?
• A repository?
• Website?
• Profile?
Are they well-described (findable)?
Are they accessible by others?
Are they citable?
Are they downloadable?
Are there metrics to tell you?

43. What metrics match those products?
Product Metric
Clinical protocol Adoption
Software code Downloads or forks
Conference poster Views
Teaching materials Adoption/adaptation
White paper Views, Tweets

44. What systems or tools can provide
those metrics?
• Journal’s website
–Views, downloads, comparisons
• Repository
–Views, downloads
• Altmetric.com; Impactstory

45. How will you explain these metrics?
• Contextualize
– “This paper was in the top 10% of all papers
downloaded in 2015.”
• Describe “broader impact”
–“This work was picked up by over 100 news
sources.”

46. Pulling it all together
• Online identity
• Personal branding
• Reputation management

47. Additional Readings
• Altmetrics for Researchers (Duke University
Medical Center Library & Archives)
• How to Track the Impact of Research Data
with Metrics (Digital Curation Centre)

48. Thank You!
Andrea Denton andrea@virginia.edu
Kimberley Barker kimberley@virginia.edu