Levine-Clark, Michael, Jane Burke, and Henning Schönenberger, “Assessing the Value and Impact of Discovery Systems,” Invited, Special Libraries Association – Arabian Gulf Chapter, Kuwait City, April 20, 2016.

ASSESSING THE VALUE AND
IMPACT OF DISCOVERY SYSTEMS
Library Perspective
Michael Levine-Clark
University of Denver

Why do libraries implement a discovery system?
Discovery System Goals
• To improve the user experience
• To provide a Google-like experience
• One-stop shopping - too many resources, too hard to keep them straight
o for undergrads or beginners / for everyone
o for particular disciplines / for every discipline / for cross-discipline needs
o primarily for articles / for articles and books / for all library resources
• One starting point (when you don’t know where to search)
• To replace the catalog
• To reduce the number of individual A&I databases (to reduce costs?)
• To increase the number of users starting with the library / Compete w/
Google

Components of Value
Usage - (searches or full text downloads)
• Perfect example of variation in stakeholder
perspectives
• Publishers want their usage to go up
• Librarians want users to find relevant content
efficiently
o could lead to decreased usage
o in tension with usage-based resource evaluation (i.e. ⬆
CPU)
• Discovery vendors need to respond to both of those
needs
JSTOR as example

Components of Value
• Effectiveness – accuracy
• Efficiency – speed
• Comprehensiveness
• Integration with other library tools
• Reaching users wherever they might be
• Vary across stakeholders
• Others?

Components of Impact
Usage
• Goes up or down overall
• Proportion coming through library (vs Google)
• Proportion coming thru discovery system vs other
library sources
• Publisher-hosted vs Aggregator-hosted content
• Proportion in different library resource formats
o Print books vs ebooks vs articles
o Increased usage of previously siloed content?
 e.g. Digital library, special collections, or IR results?

Components of Impact
Other
• Bringing users (back) to library resources
o Directly or Indirectly
• Reduction in number of individual databases?
• Fewer questions about basic searching from users
o Users don’t have to wonder where/how to search - it’s
obvious
• Reference/Instruction librarians freed up to focus on
higher-level consultations/instruction
• Others?

Impact of Discovery Systems on Journal Usage
Our study compared usage before and after 2011 implementations at the
journal level, which is only one way of measuring impact.
Levine-Clark, McDonald, & Price (2014) “Discovery or Displacement? A
Large-Scale Longitudinal Study of the Effect of Discovery Systems on
Online Journal Usage,” UKSG Insights 27(3): 249-256.

Study Results
1.All Discovery systems (DS) increased usage relative to
control, and the degree of increase varied from one system to
another
2.There is as much variation within DS as across them
3.Discovery systems may sometimes shift usage toward the versions
of the content on their aggregated platform
4.DS impact is statistically significant, but explains only a small
portion of the variance

1. All Discovery Systems increased usage vs control group
Control DS1 DS 2 DS3 DS4

Study Results
1.All Discovery systems (DS) increased usage relative to control,
and the degree of increase varied from one system to another

2. There is as much variation within DS as across them
Control DS1 DS 2 DS3 DS4

Study Results
3.Discovery systems may sometimes shift usage toward the
versions of the content on their aggregated platform

DS may sometimes shift usage toward their
aggregated platform (unpublished data)
● 2 DS Vendors that also have aggregated journal collections
● Numbers indicate the number of journals that were hosted and used on both
the publisher & aggregator platform during 2013-14
● Different publishers for each DS, but same pattern
● Unlikely to be intentional; can be configured by library

Study Results
portion of the variance (η2 < 1% -- .0086)

Where do we go from here?
Ideal Questions -- Participation by multiple stakeholders
with an answer everyone wants to know
Two categories of questions
Broader Synthetic Questions
• How much of an impact do discovery services have?
Narrower Analytic Questions
• [maybe only meaningful if DS impact is compelling?]
• Address the why behind some aspects of impact

Do library discovery systems really make a difference?
Single publisher site referring URL data
But what about referrals that lead to Full Text use – aren’t these more
important?
• Are users finding/accessing more relevant
U of
Denver

Synthetic - Is Google Scholar a viable alternative to a
library discovery system?

Analytic - To what extent does library e-resource management and
linking configuration limit DS effectiveness?
Why to study
• Wide variation across libraries with same system in our study.
This could be one reason why
• If discovery implementation is only as effective as these other
decisions, important to know
How to study
• Compare libraries with same discovery system and different
configuration options
• Compare same library before/after changing configuration

Analytic - Does discovery system configuration affect the user
experience?
Why to study
• Wide variation in our study
• Many libraries appear not to think too carefully about this. An
afterthought or trial and error
How to study
• Compare libraries with different configurations
• Compare before/after in library after changing configurations
• Compare before/after across discovery system when systemwide
change made
Difficulty: Libraries often reconfigure gradually. Hard to link
specific configuration choice to impact.

Is there a future (in search & content) for libraries?
The even broader question that we rarely ask
• A variation on this question is, “what happens if we
cede discovery to Google Scholar and its ilk?”
• The promise (and threat) of Open Access
• Influence/Change User Behavior
o retain or grow dependence on library resources
o awareness of their use of library resources

22
Discovery and Metadata from a Publisher Perspective | Henning Schoenenberger | Gulf SLA | April 20, 2016
Discovery and Metadata
from a Publisher
Perspective
Henning Schoenenberger
Springer Nature | April 20, 2016

23
• Henning Schoenenberger
• Director Product Data and Metadata at Springer Nature
Personal Introduction

24
• Springer Nature is one of the world’s leading global research, educational
and professional publishers.
• Springer Nature is the world’s largest academic book publisher, publisher of
the world’s most influential journals and a pioneer in the field of open
research.
• 3,000+ journals and more than 210k eBooks available on link.springer.com.
• The company was formed in 2015 through the merger of Nature Publishing
Group, Palgrave Macmillan, Macmillan Education and Springer
Science+Business Media, with offices in over 50 countries.
Introduction Springer Nature

25
Data Source: 2014 Springer Global Academic and Government Customer Survey
Does your library use a web-scale discovery service,
e. g. Summon, EDS, Primo, or OCLC WorldCat/Local?
66%
34%

26
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Publishing Life Cycle
Researcher

27
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

28
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

29
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

30
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

31
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

32
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

33
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

34
Research/
Manuscript
Creation
Manuscript
Submission
Peer
Review/
Proposal
Stage
Planning
Production
Publication
Distribution/
Sales
Discovery
Researcher

35
Research Life Cycle

36
Research Life Cycle

37
• Discovery is key for our customers, researcher, students, graduates, professionals
who want to find and want to be found.
• Discovery is not as simple as Google – it is a complex, constantly changing
environment.
• Google and Google Scholar drive much traffic to our content platforms, but they
also drive many denials.
• Library-engineered tools drive more traffic AND downloads.
• Of course this requires attention and coordination between content provider,
discovery service provider and library.
The Importance of Discoverability
Source: Research conducted in 2015 by Bob Boissy, Director of Institutional Marketing and Account Development – Americas

38
The Information Food Chain
Content
Provider
Discovery
Service
Provider
Library
Catalog
Researcher /
End User
Discoverability Usage

39
SpringerLink User Survey
SpringerLink, including Springer for R&D as well as Springer for
Health & Hospitals, is one of the leading scientific websites in the
world. In 2013, the website recorded more than 220 million visits, and
is, according to the web traffic measurement companies Alexa.com
and Compete.com, among the 2,500 most popular global websites.
However, so far Springer had only little knowledge about who the
actual users are and for what purposes they use SpringerLink. The
goal of this survey was to obtain more information on these two
questions.
3,370 users responded to an online survey.
Background and
Objectives
Source: Research conducted in 2014 by Harald Wirsching, Director, Market Intelligence and Webanalytics

40
Users come from a wide range of organizations and
positions
 Roughly three quarters of all users are
from academic institutions, with strong
usage from all groups within academia:
Professors, lecturers, researchers,
Ph.D. candidates and students
 Further main user groups are
researchers in non-academic research
organizations, physicians & clinicians
as well as researchers and R&D
professionals in the industry
 The ‘other’ category includes librarians,
research and business consultants as
well as technicians, engineers and IT
professionals
Academic
researcher/
Professor/
Lecturer
29%
Ph.D.
candidate/
Doctoral
student
21%
Students
28%
Researcher
governm./
non-profit
7%
Physician/
clinician
5%
Researcher/
R&D prof.
industry
4%
Other
6%
What is your current primary position?
N=3,370

41
Use of main search tools by user group, age and medium
N=3,369
 Two-thirds of users started
their search on Google web
search or Google Scholar
 Other important referrers
include library websites, A&I
databases, CrossRef links,
recommendations from peers,
and TOC alerts
 Results are very much in line
with data from the Springer
Link web analytics tool
Google
Scholar, 33%
Google web
search, 30%
Library
website, 9%
Abstract &
Indexing, 8%
Reference
linking, 6%
Direct visit,
5%
Recommen-
dation, 3%
eMail alert,
3% Other, 4%
Where did you start this visit to the SpringerLink
website, or who directed you to SpringerLink?

42
How could publishers help you to better achieve your goals
(to get your job done)?
3%
3%
6%
10%
14%
30%
0% 10% 20% 30% 40%
Better e-mail alerts
Better abstracts
Lower prices
Provide related
content
Better search
functionalities
Free / open access to
content
N=1,749; open-ended question
• Two main topics were brought up. Firstly, the topic
about having access to all desired content, and related
to this, the perceived high prices of subscriptions and
PPV.
• Secondly, the whole search & discovery process, the
ability for an user to quickly identify and evaluate the
most relevant content to his research question. Here
the search functionalities on SpringerLink were
mentioned, but also a wish to identify ‘related’ content
to a piece of content that already has been found to be
useful. Other topics were better abstracts that help
users to more quickly understand the main points of a
document as well as better, keyword-based alerting
services.

43
• Springer Nature has agreements with discovery services to ensure that full text
data and metadata are being fully ingested.
• Springer Nature sends full text XML of our archival and frontlist content.
• Discovery services use this data for indexing and loading chapter/article level
bibliographic information.
• Springer Nature works closely with vendors to ensure completeness and
discoverability of our content.
Discovery Services – What Springer Nature Sends

44
• Springer Nature provides downloadable title lists in KBART format for all of our
sellable packages.
• Additionally, Springer Nature provides consortia and regional package title lists.
• Springer Nature works closely with link resolver vendors to keep collection
information up to date and to create clear targets.
Link Resolvers – What Springer Nature Provides

45
SpringerLink KBART Title Portal at http://link.springer.com/lists
• shows all sellable packages based on the KBART Phase II Recommendations
Downloader Tool at www.springer.com/marc
• Provides our MARC records and eBooks title lists for download
• Includes Palgrave content
Link Resolvers – How do they receive our data?

46
• Springer Nature provides free MARC records for Protocols and eBooks including the
Springer Book Archives and Springer References (downloader tool at
www.springer.com/marc) as well as for journals and journal articles. Contact
marc@springernature.com.
• Springer Nature MARC records are also available for free through OCLC.
MARC Records

47
• Query Springer Nature’s metadata (books, journals) or full-text content of Open Access
journal articles
• Different output formats such as XML, JSON, PAM/PRISM
• Access to the portal: https://dev.springer.com
API Portal

48
• Springer Nature works with just about every A&I service, over 400 in total, through
a dedicated A&I department.
• Notable A&I services include:
• Google Scholar
• ISI Databases
• PubMed
• PubMed Central
• Medline
• SCOPUS
Abstracting & Indexing

49
• Our Discovery Services Implementation
Guide outlines how Springer works with
third party vendors to increase the
discoverability of our content.
• PDF document available on
www.springer.com/discovery.
• Springer Nature Discovery Guide coming
soon!
Discovery Services Implementation Guide

50
• Perform “discovery reviews” to assess how Springer Nature interacts with library’s
online environment
• Guide customers on how Springer Nature works with discovery services and advise
on best practices for discovery
• Next generation systems combine the various layers (discovery, link resolver and
MARC records etc.) into one suite:
• Examples: OCLC’s WorldShare and ExLibris’ Alma
• Stay up-to-date on latest developments in our industry, e. g. research data,
BIBFRAME and RDA
Ongoing Activities

51
• One of the pain points coming out of the SpringerLink User Survey seems to be an
inefficient search process.
• It seems that users are overwhelmed with too many search results and would
appreciate every effort that would help them to identify the most ‘relevant’
content quickly.
• Quite a few users are unsatisfied with the way that our content is indexed. Here
semantic indexing with controlled vocabulary could add value, with the possibility
for users to filter content more precisely (incl. synonyms, additional tags etc.).
• Examples where Springer Nature collaborates with partners active in the field of
semantic technologies …
Semantic Search

52
Semantic Search

53
Semantic Search

54
Semantic Search

55
• With http://www.nature.com/ontologies and http://lod.springer.com Springer
Nature is already very active in the field of LOD.
• Project ongoing to combine both sides.
• The more your data is linked, the more it gets used: Discoverability and visibility.
Linked Open Data

56
Linked Open Data

57
Linked Open Data

58
Linked Open Data

59
Linked Open Data

60
Linked Open Data

61
Linked Open Data

62
Linked Open Data

63
Linked Open Data

64
Linked Open Data

65
Linked Open Data

66
Linked Open Data

67
Enriching metadata and linking it to other available
data
Linked Open
Data (LOD)
Cloud

68
• ORCID is a digital identifier for researchers and their work
ORCID: Open Researcher and Contributor ID
logo is
clickable
PDF metadata

69
• Persistent identifier for institutions
• Disambiguation: alternative institution names, acronyms, language variants
• Mapped to other standard identifiers (e. g. ISNI, OrgRef, Wikidata, FundRef)
GRID: Global Research Identifier Database

70
• Example: Kuwait University (grid.411196.a):
https://grid.ac/institutes/grid.411196.a
GRID: Global Research Identifier Database

71
• CrossRef’s Open Funder Registry provides a standard taxonomy for funding
organizations: standardized name + unique identifier
• Springer Nature participates in this initiative
• Funding information is given both in the PDF content (acknowledgements section)
and the metadata:
Funding Data: FundRef

72
• Global subject classification scheme for books hosted by EDItEUR
• Springer Nature among the participating organizations
• Has been recently added to our metadata formats
Thema Classification
SCY12005
Clinical
Psychology
Thema code:
MKM
Springer Nature
Subject
Classification Code
Thema term:
Clinical
Psychology

73
• Since mid 2014 new Springer journal articles are marked with the CrossMark logo.
• Example: http://link.springer.com/article/10.1186/s40638-014-0018-z
CrossMark

74
CrossMark

75
• Listening to what discovery services need from us as their scholarly publishing
partner
• Participating in professional conferences
• Abiding by industry standards co-developed by the various players
• Introducing new technologies to make online resources more visible, discoverable
and usable
• Staying engaged
Collaboration

76
Contact Us
For more information visit: springer.com/discovery
E-mail discovery@springernature.com
Visit springer.com/salescontacts to find your local
Springer Nature representative
Like us on Facebook: SpringerLibraryZone
Follow us on Twitter: @Library_Zone
Henning Schoenenberger:
henning.schoenenberger@springernature.com

Assessing the Value and
Impact of Discovery
Systems:
Discovery Provider Viewpoint
Jane Burke
Ex Libris, a ProQuest
Company
April 20, 2016

The reality is user expectations have evolved
and library systems have not kept up.
Meetings Users’ Expectations

• Self sufficient and do NOT ask librarians for help
• Expect everything to be online and immediately accessible
• Use multiple devices and are increasingly mobile
• Want to share and collaborate
Understanding today’s users

Presented by Alison Head. ACRL conference, April 11, 2013

• The importance of the role of the library as a
gateway for locating information has fallen over
time
• The library is increasingly disintermediated from
the actual research process
Faculty Survey
2009:
Key Strategic Insights for
Libraries, Publishers, and Societies
April 7, 2010
Authors:
Roger C. Schonfeld (Manager of Research) & Ross Housewright (Analyst)

Discovery Services Drive Growth

Discovery services drive growth in usage
Increased usage of publisher-hosted
journal content
Discovery or Displacement?: A Large Scale Longitudinal Study
of the Effect of Discovery Systems on Online Journal Usage
Michael Levine-Clark (University of Denver), John McDonald
(University of Southern California), Jason Price (SCELC
Consortium)
Increase usage of JSTOR content
for mutual US Higher Education
clients analyzed.
Plato’s Cave Revisited
Bruce Heterick (JSTOR)
Charleston Conference 2013

Discovery delivers proven value
0% 200% 400% 600% 800% 1000% 1200% 1400%
Academic Search Premier
APA PsycArticles
Business Source Complete
CINAHL Plus with Full Text
Education Full Text
Education Research Complete
Emerald Journals
General OneFile
Health Reference Center Academic
Journals@Ovid Ovid Full Text
JSTOR Biological Sciences Archive Collection
LexisNexis Academic
Literature Online (LION)
Oxford Journals
Print at GVSU Libraries
SAGE Complete A-Z List
ScienceDirect Journals
Wiley-Blackwell Journals (Frontfile Content)
Click Through Growth by Database from 2008 - 2011
486%
306%
475%
320%
429%
410%
1028%
359%
879%
722%
430%
991%
399%
1322%
840%
263%
288%
160%

Understanding user search behaviors
1 10 100
Number of terms per query
Summon query analysis –
• High frequency of
misspellings
• Natural language queries
are common
• Users often edit search
terms rather than seek new
strategies
• 85% of users never go past
first page of results
Erin Dorris Cassidy , Glenda Jones , Lynn McMain , Lisa Shen & Scott Vieira
(2014) Student Searching with EBSCO Discovery: A Usability Study, Journal of
Electronic Resources Librarianship, 26:1, 17-35, DOI:
10.1080/1941126X.2014.877331
45% of Searches
3 words or less

Image & Content Spotlighting
Reference
Content &
Related
topics
Contextual
Recommendations
Data-driven Related Search
Suggestion
Data-driven
Autocomplete
The Single Search Box is not enough

Over 50,000 Topics
7 languages and more comingAutomated Query Expansion™
Connects Users to Librarians

Librarian controlled recommendations
Data-driven search suggestions
Embedded reference chat

Newspaper Article Spotlighting
Image Spotlighting

Linking and access are critical to Discovery
• Users want immediate
access to full text
• Users expect the same
reliability of links that
Google provides
• Failed (dead-end) links have
real consequences, they:
• Frustrate users and lead to
dissatisfaction with the
library
• Negatively impact usage of
library resources
• Linking provides a bridge
from Google to Library
resources
Linking is a top complaint of
library end users…
“Bad links equal less users”
Steven Bell. “Resolving the Link Resolver Problem”
From the Bell Tower, August 7, 2014. http://bit.ly/1ly5cci

OpenURL Linking is unpredictable
“5-30% of OpenURL
Links commonly fail”
Jason S. Price and Cindi Trainor. "Digging into the Data:
Exposing the Causes of Resolver Failure."
Library Technology Reports 46.7 (October 2010): 15-26.

Link resolver menus are confusing
“25% of users failed to
click the appropriate
link to get them to the
desired content.”
Bonnie Imler and Michelle Eichenberger. “Do they ‘Get
it?’ Student Usage of SFX Citation Linking Software.”
College and Research Libraries. (September 2011) 454-
463.

Discovery
IEDL
Links
OpenURL-
based
Linkers
Knowledgebase
Rights Management

Transforming the Link Resolver
What we’ve done:
1. Integrated Index-Enhanced Direct Linking (IEDL)
technology into link resolver.
• Provides the most direct and reliable links to content
and moves libraries beyond sole reliance on OpenURL
2. Introduced a new Sidebar Helper frame giving
libraries a persistent presence in the research
workflow
• Offers improved user experience + opportunity for
libraries to be relevant and promote additional services
3. Launching new, modern user experience
• Eliminates most common confusion/failure points
• Offers library’s flexible configuration and customization
options

360 Link Sidebar Helper frame
New Sidebar Helper frame

Linking
embedded
in Google
Scholar
Link Resolver Embedded in Google

US Faculty Survey 2015
http://www.sr.ithaka.org/publications/research-data-management

Areas Of Focus
102
Outreach Instruction Research Data
“Interest in supporting students and their competencies and
learning outcomes shows signs of surging… increase in the
perceived importance of the role of the library in helping
undergraduate students develop research, critical analysis, and
information literacy skills”

And …
• Discovery starting points remain in flux.
“After faculty members expressed strongly
preferring starting their research with a specific
electronic research resource/database… they are
now reporting being equally as likely to begin with a
general purpose search engine as they are with a
specific electronic research resource/database.
Furthermore, the online library website/catalog has
become increasingly important for conducting
research since the previous cycle of the survey.”

Increasing Number of Data Sources
104
Administration Tools
& Normalization Rules
ILS, URM
Alma, Aleph, Voyager, Millennium,
Symphony, Unicorn, Talis, …
Library Resources
Course Reserves, LibGuides, Websites,
…
Digital Repositories
Rosetta, DSpace, CONTENTdm, Fedora
Commons, …
Online Databases
Subscription and open access
databases, newspapers articles, …
Supplementary Entities
Researchers, Librarians, …
Research Materials
Datasets, outputs, raw data, faculty
pages, …

Reach Out – Search Engines
105

Focus On Outreach
107
Collection
exposure and
publishing
tools

Support Reading List Life Cycle
108
Reading List
solutions such as
Leganto and SIPX

Instruction
109
Education and information literacy
tools:
• E.g. Reading lists
More focus on outcomes, such as
student success:
• Demonstrate correlation between
students use of library resources &
services and learning outcomes

Research Data Management
110
http://www.sr.ithaka.org/publications/research-data-management

Research Data Management
“The library is well situated to manage activities
such as outreach, data deposit, metadata creation, and preservation;
some university libraries are directed to do so,
while other proactively offer their services.”

Library as a Research Data Service Provider
112
Education Through Services’ Demonstration

Low hanging
fruit:
discoverability of
research data
Library as a Dissemination Service
113

Library Controlled Recommendations
Goal: Librarians embedded in research workflow
Integrated Reference Chat
Recommended Librarians & Research Guides

Levine-Clark, Michael, Jane Burke, and Henning Schönenberger, “Assessing the Value and Impact of Discovery Systems,” Invited, Special Libraries Association – Arabian Gulf Chapter, Kuwait City, April 20, 2016.

Levine-Clark, Michael, Jane Burke, and Henning Schönenberger, “Assessing the Value and Impact of Discovery Systems,” Invited, Special Libraries Association – Arabian Gulf Chapter, Kuwait City, April 20, 2016.

More Related Content

What's hot

Viewers also liked

Similar to Levine-Clark, Michael, Jane Burke, and Henning Schönenberger, “Assessing the Value and Impact of Discovery Systems,” Invited, Special Libraries Association – Arabian Gulf Chapter, Kuwait City, April 20, 2016.

More from Michael Levine-Clark

Recently uploaded

Levine-Clark, Michael, Jane Burke, and Henning Schönenberger, “Assessing the Value and Impact of Discovery Systems,” Invited, Special Libraries Association – Arabian Gulf Chapter, Kuwait City, April 20, 2016.