This document summarizes a major longitudinal study examining the effect of web-scale discovery services on online journal usage. The study analyzed usage data from 33 academic libraries across multiple discovery service platforms and publishers. Preliminary results found statistically significant differences in journal usage changes between libraries and across different discovery services. The study aims to help libraries and publishers better understand the impact of discovery services on accessing online content.

1. Discovery
or
Displacement?
A
Major
Longitudinal
Study
of
the
Effect
of
Web-­‐Scale
Discovery
Services
on
Online
(Journal)
Usage
SCELC
Colloquium
March
5,
2014
Michael
Levine-­‐Clark,
University
of
Denver
John
McDonald,
University
of
Southern
California
Jason
Price,
SCELC
ConsorNum

2. “…a steep increase in full text downloads
and link resolver click‐throughs suggests
Summon had a dramatic impact on user
behavior and the use of library collections
during this time period.”
The Impact ofWeb-scale Discovery on the Use of a Library
Collection
Doug Way (2010) http://scholarworks.gvsu.edu/
library_sp/9/

3. h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html
Vendor
marke5ng

4. Does
implementa4on
of
a
discovery
service
impact
usage
of
publisher-­‐
hosted
journal
content?

5. What
did
we
measure?
• Whether
there
is
an
effect
• NOT
why
that
effect
exists
(that’s
a
future
study!)

6. • “Society
will
need
to
shed
some
of
its
obsession
for
causality
in
exchange
for
simple
correla5ons:
not
knowing
why,
but
only
what”
• Cukier
&
Mayer-­‐Schonberger.
2013.
Big
data:
A
revolu4on
that
will
transform
how
we
live,
work,
and
think.

9. Data
collec5on
• List
of
libraries
with
discovery
services
> Searched
on
lib-­‐web-­‐cats
• Surveyed
Libraries
> Discovery
service
Implemented
> ImplementaNon
Date
(month/year)
> Search
box
locaNon
> MarkeNng
effort
• 149
Libraries
Gave
Approval
> 33
libraries
selected
for
this
phase
> 6
for
each
of
the
4
major
discovery
services
and
a
group
of
9
libraries
with
no
service

10. Dataset
• 33
Libraries
– 28
US,
2
CA,
1
each
from
UK,
AUS,
NZ
– WorldCat
book
holdings
> Average:
1,114,193
;
Range:
~300k
to
~2.6mil
• ImplementaNon
dates
(Discovery
Libraries):
> 2010
(3),
2011
(19),
2012
(2)
• 6
Publishers
• 9,206
Journals
• 163,545
Usable
ObservaNons

11. Methodology
Compared
COUNTER
JR1
total
full
text
arNcle
views
for
the
12
months
before
vs
12
months
aeer
implementaNon
date
June
2010
Start
ImplementaNon
May
2011
May
2012
End
Year
1
Year
2
Included
implementaNon
month
in
Year
1
to
ensure
that
both
periods
included
an
enNre
academic
year

13. Examine
Data
for
Outliers

14. Observa5ons
by
Library
&
Service

15. Observa5ons
by
Publisher

17. Average
Usage
Change
By
Discovery
&
Publisher

18. Analyzing
Usage
Change:
%
vs
Total
Use
12
months
before
Use
12
months
aRer
%
Change
Total
Change
Journal
A
500
600
20%
100
Journal
B
5
15
200%
10
Which
is
the
beier
measure?
Is
it
the
same
for
publisher-­‐
&
journal-­‐level
data?

19. Reducing
varia5on
due
to
ins5tu5on
size
Currently
converNng
to
change
per
FTE
Values
are
shown
as
x
1,000
to
bring
the
change
metric
back
per
journal-­‐library
combinaNon
to
a
minimum
of
0.1
2013
JISC
Discovery
study
took
a
similar
approach

20. Average
Usage
Change
By
Discovery
&
Publisher
Per
Journal
&
Per
10,000
FTE

21. Full
Model
Including
Discovery
Service,
Publisher,
and
Library
Including
Discovery
Service,
Publisher,
and
Library

22. Does the effect of discovery service differ across libraries?
Library
10-­‐15
Library
16-­‐21
Library
22-­‐27
Library
28-­‐33
Library
1-­‐9

23. Nested
ANOVA
Model
[all
three
factors
–
preliminary
results]

24. Does
usage
change
vary
across
libraries?
Institution (sorted by Mean Change)

25. Does
usage
change
vary
across
publishers?
Publisher (sorted by Mean Change)

26. Does
usage
change
vary
across
discovery
services?

27. Publisher
Does
the
effect
of
discovery
service
differ
across
publishers?

28. Results
Can
we
detect
differences
between
Discovery
Services,
Publishers,
and/or
Libraries
and/or
their
interac4ons?
• Library
–
Yes
• Publisher
–
No
• Discovery
Service
–
Yes
• DifferenNal
discovery
service
effect
by
publisher
–
Yes

29. Next
Steps
• Design
&
test
for
effects
of:
– Aggregator
full
text
availability
– Publisher
Size
– Journal
Subject
– Overall
usage
trends
(Requires
Disc
Srvc
‘control’)
– ConfiguraNon
opNons
in
Discovery
services
• Expand
pool
of
libraries
• Perhaps
explore
WHY

30. Sharing
Data
• With
par5cipa5ng
libraries
– Customized
reports
for
each
library
• With
par5cipa5ng
publishers
– Customized
reports
for
each
publisher
– PresentaNons
as
requested
• With
discovery
vendors
– PresentaNons
as
requested
• In
publica5ons
and
presenta5ons
– Maintaining
anonymity
of
data

31. Doing
“Resarch”,
SCELC
Style!
• Why
SCELC?
• SCELC
Funding
– staNsNcs
consultant
– research
&
wriNng
retreats
– See
hip://bit.ly/1dNMDL3
for
more
detail
• SCELC
libraries
encouraged
to
parNcipate
in
next
round
– Survey:
hip://bit.ly/DSparNcipaNon