This presentation describes the problems of accurately counting file downloads from institutional repositories using commonly applied web analytics methods: page tagging and log file analysis. The presentation introduces a new prototype web service called RAMP (Repository Analytics and Metrics Portal) that provides a much more accurate method of counting file downloads.

1. Kenning Arlitsch, Dean - @kenning_msu
Patrick OBrien, Semantic Web Research Director - @sempob
Montana State University Library
Coalition for Networked Information
Washington, D.C.
December 13, 2016
Improving the Reported Use and
Impact of Institutional Repositories

2. Theme
Accurately assessing the impact of institutional repositories
Research funded by the Institute of Museum and Library Services (IMLS)
”Measuring Up: Assessing Accuracy of Reported Use and Impact of Digital Repositories”
http://scholarworks.montana.edu/xmlui/handle/1/8924

3. Agenda
❖A New Reporting Model
❖Accuracy of Analytics Reporting Tools
● Page Tagging
● Log file
❖RAMP
● New Prototype Web Service for IR reporting

4. A New Reporting Model
Page Type Definition Examples
Citable Content
Downloads
Non-HTML scholarly content
that may be formally cited in
the research process
● Publication (.pdf)
● Presentation (.ppt)
● Data Sets (.csv)
Item Summary
HTML pages to help user
decide to download the full
publication
● Title & Abstract
● Item Metadata
Ancillary
HTML pages that provide
general information or
navigation
● Search Results
● Browse by Author
● Statistics

8. Two Classes of Web Analytics
HTML
Analytics Service (SaaS)
1
Log Files
2Page Tagging
{JavaScript}

9. Google Analytics used by most academic
libraries
❖Tested 279 academic library websites
● ARL
● DLF
● OCLC-RLP
❖90% US libraries contained Google
tracking code

10. Page Tagging
Page Tagging does not track non-
HTML Citable Content Downloads
HTML
Non-HTML

11. Show Google Scholar direct link
graphic

12. Web Analytics Accuracy Risks
Risks Analytics Method
Area Types Page Tagging Log Files
OverCount
Visits Low High
Downloads Low High
Page Views Low Low
UnderCount
Visits Medium Medium
Downloads High Low
Page Views Low Low

13. Typical Page Tagging Data Sources
Page Type
Analytics
Pages Events
Citable Content
Downloads
- ✅
Item Summary ✅ -
Ancillary ✅ -

14. Four IR During 2016 Spring Semester January
5 to May 17 (n = 134 days)
Study Participant IR Platform URL
Montana State University ScholarWorks DSpace scholarworks.montana.edu
McMaster University MacSphere DSpace macsphere.mcmaster.ca
University of New Mexico LoboVault DSpace repository.unm.edu
University of Utah USpace CONTENTdm uspace.utah.edu

15. Page Tagging does not track non-
HTML Citable Content Downloads
Non-HTML
Search Analytics
Does!

16. Montana Method Includes Google Search
Console
Page Type
Analytics
Search
Console
Pages Events Search Analytics
Citable Content
Downloads
- ✅ ✅
Item Summary ✅ - -
Ancillary ✅ - -

17. All four IR using Google Analytics
Page Type
Analytics
Search
Console
Pages Events Search Analytics
Citable Content
Downloads
- 26,355 562,933
Item Summary 284,303 - -
Ancillary 201,793 - -
CCD Tracking Improvement
2,000%

18. 54.3%
46.2%
47.9%
42.5%
45.7%
53.8%
52.1%
57.5%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Most IR Activity is Unreported by Google Analytics
01/05/16 - 05/17/16 (days=134)
Reported Invisible
Most IR Activity Unreported by Google
Analytics.

19. 51%
54%
52%
63%
35%
27%
20%
20%
14%
19%
28%
17%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
~51% to 63% of Citable IR Activity is Unreported by Google Analytics
01/05/16 - 05/17/16 (days=134)
Citable Content Downloads Item Sumary PV Ancillary PV
Most IR activity consists of Citable Content
Downloads

20. Montana Method Challenges
❖ Missing non-Google direct link CCD
events
● Yahoo
● Bing
● Email
❖ Search Analytics limits time and access
● Moving 90 day window
● Granular data = programing skills to access
API

21. RAMP - Repository Analytics and Metrics
Portal
❖ Cloud Web service
❖ Currently accessible and free
● During grant period (December 2017)
❖ No training or configuration required
❖ Consistent method and terminology
❖ Benchmarking across time and
organization
❖ Request Access @ mixterj@oclc.org

22. Two New Publications
Published:
Patrick OBrien, Kenning Arlitsch, Leila Sterman, Jeff Mixter, Jonathan Wheeler, and
Susan Borda. “Undercounting File Downloads from Institutional Repositories,” Journal
of Library Administration, vol. 56, no. 7, 2016
Forthcoming:
Patrick OBrien, Kenning Arlitsch, Leila Sterman, Jeff Mixter, Jonathan Wheeler, and
Susan Borda. “RAMP: Repository Analytics and Metrics Portal: A Prototype Web
Service that Accurately Counts Item Downloads from Institutional Repositories,”
Submitted to Library Hi Tech, expected publication early 2017

23. Undercounting Research Team
❖Montana State University
●Kenning Arlitsch, Dean @kenning_msu
●Patrick OBrien, Semantic Web Research Director @sempob
●Leila Sterman, Scholarly Communication Librarian
@calamityleila
●Susan Borda, Digital Technologies Librarian @mutanthumb
❖OCLC Research
●Jeff Mixter, Software Engineer @jeffmixter
❖University of New Mexico
●Jonathan Wheeler, Data Curation Librarian