The document discusses the challenges and opportunities in developing metrics for assessing the impact of scholarly publications from a librarian's perspective. It highlights the messy nature of existing data, the transition from product-based to service-oriented metrics, and the need for effective systems to analyze and communicate this data. Key trends include the rise of research information management systems and growing ethical considerations in the metrics landscape.

1. Developing New Metrics
A Pubrarian’s Perspective
NISO Training Series: Assessment Practices and Metrics
Charles Watkinson, AUL, Publishing, U-M Library
Director, University of Michigan Press
December 20, 2018

2. Pubrarian?

3. Why I am interested in (and scared of) assessment?
Interested in assessment because:
● We have a lot of data about use of our
products pouring in. It must be useful,
right? Commercial competitors think so.
● Our authors are increasingly asking us for
data about “impact,” partly because they
are interested but also because they are
being asked for it.
● We are part of a library that is very
interested in assessment (has just hired its
first assessment specialist, Craig Smith)
and we are trying to work out how to
benefit from this.
Scared of assessment because:
● The data we have coming in is very messy
and in many cases is very partial and
obfuscated.
● We don’t have good systems to
aggregate, analyze, or communicate this
data.
● We don’t have staff who know how to do
this work, nor do the staff who are
interested have the capacity to do much.
● We’re aware of the underlying
weaknesses of the data and worried about
unintended consequences.

4. Products vs. Services / Collections vs. Services
“Many organizations aspire to regain a competitive advantage by transforming
from a product manufacture into a service provider. This requires a substantial
reinvention . . . Moving from products to services sounds like an easy and obvious
value proposition shift, but it requires substantially reengineering the business
model.” Osterwalder, A., Pigneur, Y., Bernarda, G., & Smith, A. (2014). Value
proposition design: How to create products and services customers want. John
Wiley & Sons, p. 164
We have not made this shift yet in terms of assessment . . .

5. What are the measures of
“success” for a book?

6. Michigan Publishing: Books
We want a UMP book (OA or not) to:
● Make money
○ Sales
○ Licenses (e.g., for translations)
● Generate prestige for us/author
○ Peer reviews
○ Journal reviews
○ Awards and prizes
○ Rankings
○ Acceptance rate?
● Be used and useful
○ Downloads
○ Views
○ User experience improvement
○ Discovery improvement
● Have impact on “society”
○ Social media
○ Policy docs
○ Citations
○ Syllabus adoptions
○ Geographical spread of use
● Manifest our values?
○ Accessibility? Inclusion?

9. https://www.press.umich.edu/librarians
● Google Analytics
● COUNTER 5
● Altmetrics
● Citations
(Dimensions)

11. https://www.palgrave.com/us/book/9781137536624
https://www.springer.com/gp/authors-editors/bo
ok-authors-editors/author-services/bookmetrix

12. What are the measures of
“success” for a journal/article?

13. Michigan Publishing: Journals
We want an OA journal to:
● Generate prestige for us/sponsor
○ Citations
○ Reviews
○ Acceptance rate
○ “Impact factor” type things?
○ Peer perception
● Manifest our values?
○ Accessibility? Inclusion?
● Be used and useful
○ Downloads
○ Views
○ User Experience
○ Discovery
● Have impact on “society”
○ Social media
○ Policy docs
○ Citations
○ Spread of use
○ Syllabus adoptions
○ Accessibility? Inclusion?

14. http://hdl.handle.net/2027/spo.act2080.0049.111

15. https://doi.org/10.1371/journal.pone.0159614

16. What are the measures of
“success” for a repository object?

17. Michigan Publishing: Deep Blue docs and data
We want an OA repository object to:
● Be “recognized”
● Be used and useful
○ Downloads
○ Views
○ User experience
○ Discovery
○ Comparative downloads
(https://jusp.jisc.ac.uk/secure/irus-usa/about/)
● Have impact on “society”
○ Social media
○ Policy docs
○ Citations
○ Spread of use
○ Syllabus adoptions
● Manifest our values?
○ Accessibility? Inclusion?

19. https://docs.lib.purdue.edu/

20. What are some major trends in metrics
and assessment that
publishers are focused on?

21. Some trends worthy of attention
1. Rise of the RIMS -- Research Information Management Systems
2. Use and engagement as the currency of growing open access
3. Growing sophistication of altmetrics, measuring far more than just
social media activity
4. Ethical considerations about what is being measured and what it
means
5. More attention to “the plumbing” -- identifiers (DOIs, ORCIDs, etc.)
and how supply chains do(n’t) work

22. 1. Rise of the RIMs

23. https://experts.umich.edu/profiles_author/discover/publication

24. 2. Use and engagement = the currency of OA

25. 2016-09-01 - 2018-11-30
Knowledge Unlatched - KU Open Analytics
Title ISBN Platform Metric Compliance
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 oapen unique_title_requests counter-4
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 jstor total_item_investigations counter-4
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 jstor total_item_requests counter-4
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 project-muse total_item_investigations counter-4
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 project-muse total_item_requests counter-4
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 ebsco-host unique_title_requests counter-4
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 fulcrum total_item_requests counter-5
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 fulcrum unique_title_requests counter-5
Alienation Effects: Performance and Self-Management in
Yugoslavia, 1945-91
9780472053148, 9780472900589, 9780472121984, 9780472073146 bibliolabs unique_title_requests counter-4

26. http://bit.ly/monograph-data-trust
http://bit.ly/data-trust-survey
https://bisg.org/news/403482/Understanding-OA-Ebook-Usage-Toward-a-Common-Framework.htm
Understanding Open Access Ebook
Usage: Toward a Common Framework
● Structured community conversation
● June 2018 - end of May 2019
● Supported by Mellon Foundation
● KU Research draft white paper
(summer)
○ Landscape review
○ Data trust proposal
● Consultation and Summit (fall)
● Final BISG White Paper (spring)

27. 3. Growing sophistication of altmetrics
“How will Brexit affect health and health services in the UK? Evaluating three possible scenarios”
The Lancet, November 2017, https://doi.org/10.1016/S0140-6736(17)31926-8

28. 4. Ethical considerations (privacy, misuse, bias, etc.)
http://www.metrics-toolkit.org/
https://humetricshss.org
https://sfdora.org/
San Francisco Declaration on Research Assessment

29. 5. More attention to the plumbing
https://www.crossref.org/blog/the-article-nexus-linking-publications-to-associated-research-outputs/
ONIX vs. MARC

31. Thank You
watkinc@umich.edu
734 936 0452