Michael Levine-Clark, John McDonald, Jason Price

1. Not always discovered: Phase two
of a study of the effect of discovery
systems on online (journal) usage
ER&L
March 18, 2014
Michael Levine-Clark, University of Denver
John McDonald, University of Southern California
Jason Price, SCELC Consortium
http://bit.ly/discovery-impact-erl2014

2. “…a steep increase in full text downloads
and link resolver click‐throughs suggests
Summon had a dramatic impact on user
behavior and the use of library collections
during this time period.”
The Impact ofWeb-scale Discovery on the Use of a Library
Collection
DougWay (2010)
http://scholarworks.gvsu.edu/library_sp/9/

3. http://www.oclc.org/partnerships/econtent/solutions.en.html
Vendor marketing

4. Does implementation of a discovery
service impact usage of publisher-
hosted journal content?

5. Publisher-hosted journals are
only part of the picture
eBooks, pBooks, aggregator journal content,
etc.
publisher journal content

6. What did we measure?
• Whether there is an
effect
• NOT why that effect
exists (that’s a future
study!)

7. • “Society will need to shed some of its obsession
for causality in exchange for simple correlations:
not knowing why, but only what”
• (Cukier & Mayer-Schonberger. 2013. Big data: A revolution that will
transform how we live, work, and think.)

10. Data collection
• List of libraries with discovery services
> Searched on lib-web-cats
• Surveyed Libraries
> Discovery service Implemented
> Implementation Date (month/year)
> Search box location
> Marketing effort
• 149 Libraries Gave Approval
> 33 libraries selected for this phase
> 6 for each of the 4 major discovery services and a
group of 9 libraries with no service

11. Dataset• 33 Libraries
– 28 US, 2 CA, 1 each from UK, AUS, NZ
– WorldCat book holdings
> Average: 1,114,193 ; Range: ~300k to ~2.6mil
• Implementation dates (Discovery Libraries):
> 2010 (3), 2011 (19), 2012 (2)
• 6 Publishers
• 9,206 Journals
• 163,545 Usable Observations

12. Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
June2010Start
Implementation
May2011
May2012
End
Year 1 Year 2
Included implementation month in Year 1 to ensure that
both periods included an entire academic year

13. Examine Data for Outliers

14. Observations by Library & Service

15. Observations by Publisher

17. Average Usage Change By Discovery & Publisher

18. Analyzing Usage Change: % vs Total
Use 12
months
before
Use 12
months
after
% Change
Total
Change
Journal A 500 600 20% 100
Journal B 5 15 200% 10
Which is the better measure?
Is it the same for publisher- & journal-level data?

19. Reducing variation due to institution size
Currently converting to change per FTE
Values are shown as x 1,000 to bring the change
metric back per journal-library combination to a
minimum of 0.1
2013 JISC Discovery study took a similar approach

20. Average Usage Change By Discovery & Publisher
Per Journal &
Per 10,000 FTE

22. Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library

23. Nested ANOVA Model
[all three factors – preliminary results]

24. Does usage change vary across libraries?
Institution (sorted by Mean Change)

25. Does usage change vary across libraries using the same service?
Library 10-15 Library 16-21 Library 22-27 Library 28-33Library 1-9

26. Does usage change vary across publishers?
Publisher (sorted by Mean Change)
B
AAA A A

27. Does usage change vary across discovery services?
A
BB
C
D

28. Publisher
Does the effect of discovery service differ across publishers?

29. Results
Can we detect differences between Discovery
Services, Publishers, and/or Libraries and/or
their interactions?
• Library – Yes
• Publisher – No
• Discovery Service – Yes
• Differential discovery service effect by
publisher – Yes

30. Next Steps
• Design & test for effects of:
– Aggregator full text availability
– Publisher Size
– Journal Subject
– Overall usage trends (Requires Disc Srvc ‘control’)
– Configuration options in Discovery services
• Expand pool of libraries
• Perhaps explore WHY

31. Sharing Data
• With participating libraries
– Customized reports for each library
• With participating publishers
– Customized reports for each publisher
– Presentations as requested
• With discovery vendors
– Presentations as requested
• In publications and presentations
– Maintaining anonymity of data

32. Past/Future Presentations
• Ithaka Sustainable Scholarship Conference (October 2013)
• Charleston Conference (November 2013)
• ER&L/Library Journal Webinar (December 2013)
• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)
• SCELC Colloquium (March 2014)
• ER&L (March 2014) http://bit.ly/discovery-impact-erl2014
• UKSG (April 2014)
• Presentations posted on slideshare :
– http://visualcv.com/lpq4t1s
michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org