Levine-Clark, Michael, John McDonald, and Jason Price. Discovery or Displacement? A Large-Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage. July 23, 2014.

1. Discovery or Displacement?
A Large-Scale Longitudinal Study of the
Effect of Discovery Systems on Online
Journal Usage
July 23, 2014
Michael Levine-Clark, University of Denver
John McDonald, University of Southern California
Jason Price, SCELC Consortium

2. Why Do We Use Discovery Services?
• Too many sources of information
– Specialized
– Confusing
– What’s the right specialized database for my
subject?
• Why do you search in one place for an article,
another place for a different article, another
for a book?

3. Terminology
Discovery Tool = Discovery Layer =
Discovery Service = Discovery System

5. Web-scale discovery services
• Single source for
finding information
– Books
– Articles
– Local content
• Metadata and/or full
text
• Content is pre-indexed
and/or pre-harvested
• Single fast search
ILS
Publisher
Metadata
MLA
Bibliograph
y
Institutional
Repository
HathiTrust
Discovery Service

6. Does implementation of a discovery
service impact usage of publisher-hosted
journal content?

7. What did we measure?
• Whether there is an
effect
• NOT why that effect
exists (that’s a future
study!)

8. Publisher-hosted journals are
only part of the picture
eBooks, pBooks, newspaper articles, aggregator
journal content, etc.
publisher journal content
The six publishers in this study

9. Journals Traffic Sources
52.3
12.4
32.5
2.8
1
Open web search
Library referrals
Social media
Academic
N/A
(SAGE, Conrad ALPSP 2013)

10. An assumption
• At any given institution, given a relatively
stable user base, the total search effort will
remain roughly the same.

11. Discovery services
 Will take up an increasing amount of a finite
time for searching
 Will draw users from other (more or less
efficient) search tools
 Will alter the overall productivity of searches
(users will find more or less)
 Will alter the overall efficiency of users (users
will access more or less full-text)

12. • 33 Libraries Dataset
– 28 US, 2 CA, 1 each from UK, AUS, NZ
– WorldCat book holdings
> Average: 1,114,193 ; Range: ~300k to ~2.6mil
– 4 discovery groups, of 6 libraries each
– 1 control group, 9 libraries
• Implementation dates (Discovery Libraries):
> 2010 (3), 2011 (19), 2012 (2)
• 6 Publishers
• 9,206 Journals
• 163,545 Usable Observations

13. Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
Start June 2010
Implementation
May 2011
May 2012
End
Year 1 Year 2
Included implementation month in Year 1 to ensure that
both periods included an entire academic year

14. Examine Data for Outliers

15. Analyzing Usage Change: % vs Total
Use 12
months
before
Use 12
months
after
% Change
Total
Change
Journal A 500 600 20% 100
Journal B 5 15 200% 10
Which is the better measure?
Is it the same for publisher- & journal-level data?

16. Observations by Publisher

17. Journals by Library & Service
EDS Primo Summon WorldCat Control

18. Average Journal Usage by Library

19. Testable Effects
• Discovery Tool
– Implemented by multiple libraries
– Used to find content from all publishers
• Publisher
– Accessible in all discovery tools
– Accessible across multiple libraries
• Library
– Uses content from multiple publishers
– Uses only one discovery tool (so only within DT)

20. Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library

21. Nested ANOVA Model Results

22. How does usage change differ across discovery services?
Letters indicate statistically significant differences (Tukey multiple comparisons, p <
.05)
A
C
D
B B

23. How does usage change differ across publishers?
Letters indicate statistically significant differences (Tukey multiple comparisons, p <
.01)
C
Publisher (sorted by Mean Change)
D
B B B
A

24. How does usage change differ across publishers?

25. Does usage change vary across libraries?
Institution (sorted by Mean Change)

26. Usage Change Per Institution: All Journals
Control EDS Primo Summon WCL

27. 6.32
Usage Change By Institution: Pub 1
Control EDS Primo Summon WCL

28. Usage Change By Institution: Pub 2
Control EDS Primo Summon WCL

29. Usage Change By Institution: Pub 3
Control EDS Primo Summon WCL

30. Usage Change By Institution: Pub 4
Control EDS Primo Summon WCL

31. Usage Change Per Institution: Pub 5
Control EDS Primo Summon WCL

32. Usage Change by Institution: Pub 6
Control EDS Primo Summon WCL

33. Publisher 1 by Discovery System

39. Publisher 2 by Discovery System

45. Publisher 3 by Discovery System

51. Publisher 4 by Discovery System

57. Publisher 5 by Discovery System

63. Publisher 6 by Discovery System

69. Next Steps
• Design & test for effects of:
– Aggregator full text availability
– Journal age (archive vs current)
– Journal Subject
– Overall usage trends
– Configuration options in Discovery services
• Expand pool of libraries
• Perhaps explore WHY

70. Sharing Data
• With participating libraries
– Customized reports for each library
• With participating publishers
– Customized reports for each publisher
– Presentations as requested
• With discovery vendors
– Presentations as requested
• In publications and presentations
– Maintaining anonymity of data

71. Presentations
• Ithaka Sustainable Scholarship Conference (October 2013)
• Charleston Conference (Nov 2013) http://sched.co/17A3Kun
• ER&L/Library Journal Webinar (December 2013)
• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)
• SCELC Colloquium (March 2014) http://goo.gl/WmJoIw
• ER&L (Mar 2014) http://bit.ly/discovery-impact-erl2014
• UKSG (April 2014) https://www.youtube.com/watch?v=d_2ycMk_9fA
• Presentations to three publishers (Spring-Summer 2014)
– More to come