Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scholarly Metrics in Specialized Settings


Published on

Presentation for the Bibliometric and Research Impact Community (BRIC) of Canada on case studies of research impact in specialized settings. Focus on Michigan Publishing by co-presenter Rebecca Welzenbach

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Scholarly Metrics in Specialized Settings

  1. 1. Scholarly Metrics in Specialized Settings: A View from the Trenches Elaine M. Lasda, University at Albany, SUNY Rebecca Welzenbach, University of Michigan Presentation for Bibliometric and Research Impact Community 15-16 May 2019 Laval University, Quebec City, QC, CAN
  2. 2. Merci de nous accueillir ici aujourd'hui!
  3. 3. Research Impact in the Academy
  4. 4. Metric "Tyranny" - Muller
  5. 5. Caution: Proceed With Care! Image by OpenClipart-Vectors from Pixabay
  6. 6. Specialized Contexts ● Impact of their research reaches beyond the academy ● Differing missions require differing metrics ● Purposes/benefits of metrics expand beyond employee promotion/tenure/institutional rankings etc. ● Demonstrating value in broader arenas, to broader constituencies
  7. 7. Book Genesis ● Meeting of the Minds with Emerald ● Special(ized) Libraries and Librarians ● Personal Network ● Variety is the Spice of Life
  8. 8. The Five Case Studies Image by PublicDomainPictures from Pixabay
  9. 9. Institute for Transportation Studies Kendra K. Levine
  10. 10. ITS & Research Impact The Challenge Tracking research through transportation project lifecycle and beyond (Justify funding from Legislature) Stakeholders Cali. State Legislature, High Level UC admins, Transportation community(public and private) Outputs/Content Assessed Technical reports and other grey literature, PRJAs, project reports, etc. Impact Deliverables Created a method for tracking impact using Google Scholar and liking research output to project codes. Librarian Roles Gathering, synthesizing data, communicating with field and administatiors, facilitating cooperation, technical expertise, subject matter knowledge
  11. 11. Environmental Protection Agency Taylor Abernethy Johnson & Anthony Holderied
  12. 12. EPA & Research Impact The Challenge Demonstrate meaningful value of scholarly output, researcher accountability, meet administrative and researcher requests Stakeholders Researchers, funders, award committees, agency administration Outputs/Content Assessed Mainly PRJAs and other traditional scholarly output using WoS/InCites, ImpactStory, PlumX, GS/PoP, Altmetric, news, etc. Impact Deliverables RIR/AIR <- impact reports with high quality visual appeal, context and data synthesis Librarian Roles Graphic design and visualization, data gathering, synthesizing, contexutalizing. Brought in on other data projects. Educating stakeholders.
  13. 13. University/ National Center for Atmospheric Research Keith Maull & Matthew Mayernik
  14. 14. NCAR/UCAR & Research Impact The Challenge Demonstrate impact in 3 arenas: annual report of activity, EarthCube “ecosystem”, and scientist use of supercomputer @ NCAR Stakeholders Funders (government/university members), researchers, UCAR/NCAR Directorate, Library Outputs/Content Assessed Largely PRJAs, but other scholarly output will be included Impact Deliverables Annual Report. Use/Impact beyond Journal- Author-Article levels. Impact of NCAR equipement/services. Used WoS/InCites, Altmetric Librarian Roles Software engineers! Gather data, technical expertise (developing an API), testing workfkow and delegating when appropriate
  15. 15. LA Museum of Natural History Richard P. Hulser
  16. 16. LA-MNH & Research Impact The Challenge Demonstrate interest and visibility of rsch & activity beyond citing references and media hits Stakeholders Museum admin, media & Marketing, donors, education program, resarchers Outputs/Content Assessed PRJAs, and nonscholarly publications Impact Deliverables Proof of concept of wide range of uses of altmetrics for a museum/humanities environment Librarian Roles Leading, project management/coordination, liaison with vendor, promotor/champion, educatio
  17. 17. Live, and In Person: Rebecca Welzenbach
  18. 18. Compliance & Defiance: Michigan Publishing's Early Encounters with Research Impact Metrics Rebecca Welzenbach Research Impact Librarian University of Michigan Library BRIC 2019, Laval University, Quebec City
  19. 19. What is Michigan Publishing doing? 1. Publishing and supporting excellent and innovative scholarship in the humanities and social sciences, including: a. Peer-reviewed scholarly monographs b. Independent, open access journals c. Institutional repository 2. Driving change in our industry to help university presses and library publishers adapt and thrive into the future, including: a. New business models b. Community-owned infrastructure c. Modeling leadership and collaboration
  20. 20. University of Michigan Press ● Founded in 1930 ● Part of the U-M Library since 2009 ● 80-100 monographs per year with disciplinary strength in Classical studies, performance studies, political science, African and Asian Studies, and more ● OA titles funded by Knowledge Unlatched, TOME, and more ● Acquisitions, Production, Marketing & Outreach, Business and Administration, technology ● Relies (mostly) on a sales model
  21. 21. Michigan Publishing Services ● Established as Library Unit ~2000 under the name Scholarly Publishing Office ● ~30 OA journals/serials and ~35 books ● Production and hosting support for external clients Lever Press & Humanities EBook ● Relies (mostly) on chargebacks
  22. 22. Deep Blue ● Launched in 2006 ● >124,000 objects in document repository (DSpace) ● 237 data sets in data repository (Samvera/Fedora) ● ⅓ of items in Deep Blue are not published/available anywhere else ● Sustained as a core service by U-M Library
  23. 23. Michigan Publishing and research impact metrics Historically: little involvement. But now, more is expected, requested, mandated. How best to engage effectively? Two options: ● Compliance: We can work to ensure that our publications are consistently recognized by and included in the systems and datasets upon which existing metrics are calculated. ● Defiance: We can articulate new (alternative) metrics that are meaningful for us and our stakeholders
  24. 24. University of Michigan Press (Monographs) Compliance ● Books have long been totally absent from the research impact metrics space. Where they’ve been indexed, the record is inconsistent: BKCI-SSH has indexed 194 UM Press titles while Scopus has indexed 916 ● Newer players Google Scholar and Dimensions Plus are changing what’s possible to know and show--but now we’re turning up a lot of gaps. ● Citation counts for a single title are interesting to authors--but only if accurate. Otherwise, distressing!
  25. 25. University of Michigan Press (Monographs) Defiance ● University presses tend to look to different metrics: ○ Financial (across press, not necessarily at book level) ○ Academic/disciplinary prestige/reputation (awards, reviews, repeat authors, attracting prominent authors) ○ Use and persistence (course adoptions, new printings/editions) ● Mapping the Free eBook Supply Chain study of OA ebook usage revealed that-- as presses have long known--usage is “spiky” and unpredictable. ● Altmetric Explorer pilot sheds light on long-term engagement with books, syllabus citations ● Lots of data from many sources, but difficult to analyze and share meaningfully
  26. 26. Michigan Publishing Services (Journals) Compliance ● For many of our journals (esp. In the humanities), JIF and comparable metrics are not meaningful--but that can change suddenly based on author demand ● As we expand into new disciplines--especially health sciences--the requirement for representation in indexes like WoS, Scopus, and Medline are a huge challenge, learning curve ● Student journals pose their own unique challenges ● Often our role is to educate, provide context, manage expectations, facilitate progress
  27. 27. Michigan Publishing Services (Journals) Defiance ● We’re interested in success, stability of our program and services ○ How many journals? ○ Are they publishing consistently? ○ What proportion of our service supports campus publications vs. off-campus? ○ Are we succeeding in getting them indexed? ● Altmetric pilot applied to journals in 2015. In 2017 shareable reports made this useful ● Google Analytics + Google Data Studio for sharing usage data with partners
  28. 28. Deep Blue (Institutional Repository) Compliance ● Many items in the repository were first published in scholarly journals with citation counts, JIF, and other bibliometrics. ● Integration of IR with Research Information System will ensure that these publications are preserved, accessible, and contextualized
  29. 29. Source: Byrne, Kate and Stephen Cawley. Connections, Collaborations, & Impact: Data-Driven Approaches to Understanding institutional research expertise. Digital Science case study. October 2018
  30. 30. Deep Blue (Institutional Repository) Defiance ● Visibility of informal and non-traditional forms of scholarship ● Download statistics ● Altmetric engagement data** ● Variety of types and forms of scholarship means that even the metrics we have don’t work the same way for everyone, everything
  31. 31. Leading for Change in our Industry New business models
  32. 32. Leading for Change in our Industry Community-Owned Infrastructure
  33. 33. Leading for Change in our Industry Leadership and Collaboration
  34. 34. Which is worse: To be represented inadequately, or not at all?
  35. 35. Which is better: To show up where we know others are counting? Or to count what matters to us?
  36. 36. Conclusion: Future Directions Compliance ● DOIs and ORCIDs ● Consistent capture & communication of data about usage and impact Defiance ● Interrogate what counts, and what we count ● HuMetricsHSS ● Responsible Metrics policies
  37. 37. Synthesis Image by Gerd Altmann from Pixabay
  38. 38. How the Cases Differ ● Range of library mission & purpose ● Funding sources ● Parent organization activities ● Relationships to stakeholders ● Evaluated subjects/objects ● Impact data output formats ● Technical resources & staff skill sets ● Maturity & level of services provided
  39. 39. Shared Challenges ● Labor intensive ● Lack of standardized identifiers for all output types ● “Out of the box” tools insufficient ● Metrics do not stand alone/speak for themselves ● Need for stakeholder education ● Measuring impact outside disciplinary boundaries/publications ● User education: “Metric Literacy”
  40. 40. Organizational Challenges/Opportunities ● “Canaries in the coal mine” ● Collaboration across the enterprise ● Embedded librarian/informationists ● Recognition of leadership ● Educational mission -> Information Literacy -> Metric Literacy
  41. 41. InfoPro/Librarian/(??) Challenges/Opportunities ● Project management ● Tech skills ● Negotiation ● Delegation ● Relationship building ● Valued team members
  42. 42. Benefits to the Greater Organization ● Confidence in the numbers ● Comparative advantage -> wise use of human resources ● Improved communication and breaking down silos ● Owning the story
  43. 43. Questions? Release Date: August 19, 2019