This document summarizes a study on the impact of discovery services on usage of publisher-hosted journal content. The study analyzed usage data from 33 academic libraries before and after implementing a discovery service, finding that usage generally increased after implementation. However, the degree of impact varied between discovery services, publishers, and individual libraries. While discovery services were found to significantly increase usage overall, the effect differed depending on the specific publisher and service used. Further analysis is needed to understand the factors influencing these differences in impact.

1. Not always discovered: Phase two
of a study of the effect of discovery
systems on online (journal) usage
ER&L
March 18, 2014
Michael Levine-Clark, University of Denver
John McDonald, University of Southern California
Jason Price, SCELC Consortium
http://bit.ly/discovery-impact-erl2014

2. “…a steep increase in full text downloads
and link resolver click‐throughs suggests
Summon had a dramatic impact on user
behavior and the use of library collections
during this time period.”
The Impact ofWeb-scale Discovery on the Use of a Library
Collection
DougWay (2010)
http://scholarworks.gvsu.edu/library_sp/9/

3. http://www.oclc.org/partnerships/econtent/solutions.en.html
Vendor marketing

4. Does implementation of a discovery
service impact usage of publisher-
hosted journal content?

5. Publisher-hosted journals are
only part of the picture
eBooks, pBooks, aggregator journal content,
etc.
publisher journal content

6. What did we measure?
• Whether there is an
effect
• NOT why that effect
exists (that’s a future
study!)

7. • “Society will need to shed some of its obsession
for causality in exchange for simple correlations:
not knowing why, but only what”
• (Cukier & Mayer-Schonberger. 2013. Big data: A revolution that will
transform how we live, work, and think.)

10. Data collection
• List of libraries with discovery services
> Searched on lib-web-cats
• Surveyed Libraries
> Discovery service Implemented
> Implementation Date (month/year)
> Search box location
> Marketing effort
• 149 Libraries Gave Approval
> 33 libraries selected for this phase
> 6 for each of the 4 major discovery services and a
group of 9 libraries with no service

11. Dataset• 33 Libraries
– 28 US, 2 CA, 1 each from UK, AUS, NZ
– WorldCat book holdings
> Average: 1,114,193 ; Range: ~300k to ~2.6mil
• Implementation dates (Discovery Libraries):
> 2010 (3), 2011 (19), 2012 (2)
• 6 Publishers
• 9,206 Journals
• 163,545 Usable Observations

12. Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
June2010Start
Implementation
May2011
May2012
End
Year 1 Year 2
Included implementation month in Year 1 to ensure that
both periods included an entire academic year

13. Examine Data for Outliers

14. Observations by Library & Service

15. Observations by Publisher

17. Average Usage Change By Discovery & Publisher

18. Analyzing Usage Change: % vs Total
Use 12
months
before
Use 12
months
after
% Change
Total
Change
Journal A 500 600 20% 100
Journal B 5 15 200% 10
Which is the better measure?
Is it the same for publisher- & journal-level data?

19. Reducing variation due to institution size
Currently converting to change per FTE
Values are shown as x 1,000 to bring the change
metric back per journal-library combination to a
minimum of 0.1
2013 JISC Discovery study took a similar approach

20. Average Usage Change By Discovery & Publisher
Per Journal &
Per 10,000 FTE

22. Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library

23. Nested ANOVA Model
[all three factors – preliminary results]

24. Does usage change vary across libraries?
Institution (sorted by Mean Change)

25. Does usage change vary across libraries using the same service?
Library 10-15 Library 16-21 Library 22-27 Library 28-33Library 1-9

26. Does usage change vary across publishers?
Publisher (sorted by Mean Change)
B
AAA A A

27. Does usage change vary across discovery services?
A
BB
C
D

28. Publisher
Does the effect of discovery service differ across publishers?

29. Results
Can we detect differences between Discovery
Services, Publishers, and/or Libraries and/or
their interactions?
• Library – Yes
• Publisher – No
• Discovery Service – Yes
• Differential discovery service effect by
publisher – Yes

30. Next Steps
• Design & test for effects of:
– Aggregator full text availability
– Publisher Size
– Journal Subject
– Overall usage trends (Requires Disc Srvc ‘control’)
– Configuration options in Discovery services
• Expand pool of libraries
• Perhaps explore WHY

31. Sharing Data
• With participating libraries
– Customized reports for each library
• With participating publishers
– Customized reports for each publisher
– Presentations as requested
• With discovery vendors
– Presentations as requested
• In publications and presentations
– Maintaining anonymity of data

32. Past/Future Presentations
• Ithaka Sustainable Scholarship Conference (October 2013)
• Charleston Conference (November 2013)
• ER&L/Library Journal Webinar (December 2013)
• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)
• SCELC Colloquium (March 2014)
• ER&L (March 2014) http://bit.ly/discovery-impact-erl2014
• UKSG (April 2014)
• Presentations posted on slideshare :
– http://visualcv.com/lpq4t1s
michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org