Discovery or Displacement?

A Large Scale Longitudinal Study of the
Effect of Discovery Systems on Online
Journal Usage
Ch...
“…a steep increase in full text
downloads and link resolver click‐
throughs suggests Summon had a
dramatic impact on user ...
Vendor marketing

http://www.oclc.org/partnerships/econtent/solutions.en.html
Does implementation of a discovery
service impact journal usage?
Web-scale discovery services
• Single source for
finding information
– Books
– Articles
– Local content

• Metadata and/or...
An assumption
• At any given institution, given a relatively
stable user base, the total search effort will
remain roughly...
Discovery services
 Will take up an increasing amount of a finite
time for searching
 Will draw users from other (more o...
Prior studies
• Some studies have indicated substantial
increases in usage after Discovery
implementation
– Descriptive st...
Data collection
• List of libraries with discovery services
> Searched on lib-web-cats

• Surveyed Libraries
> Discovery s...
Library demographics
• 20 US, 1 each from UK, AUS, NZ, CA
• 10 ARL Libraries included
• WorldCat book holdings
> Average: ...
Dataset
• 24 Libraries
• 4 Discovery services
• 6 Publishers
• 9,206 Journals
• 159,278 Observations
• 141,048 Usable Obse...
Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation ...
Collections notes
o Excluded journals that did not have 24
months of COUNTER reporting
o Limited ability to control for ch...
General trends
• Variation by institution within each
discovery service
• Variation by publisher within each
discovery ser...
Goals of our inferential statistics
Determine whether observed differences are
significant or resulted from chance effects...
ANOVA - Analyzing the data
Observation

=

Fit

+ Residual

Change
In = Library x + Publisher y + Disc Svc z + Residual Er...
ANOVA – F Ratio
Tests whether the means for levels within a
factor are distinguishable from each other
Average variability...
Does usage change vary across libraries?

Overall Average = 8.5

Institution (sorted by Mean Change)
Does usage change vary across libraries?

Overall Average = 8.5

Institution (sorted by Mean Change)
Does usage change vary across publishers?

Overall Average = 8.9

Publisher (sorted by Mean Change)
Does usage change vary across discovery services?

Overall Average = 8.9
Does the affect of discovery service differ across publishers?
Does the affect of discovery service differ across publishers?

Publishers (distinguished by color)
Do the discovery service means differ in the 2 way model?

15.0
12.3
4.5

3.7

Publishers (distinguished by color)
15.0
12.3
4.5

3.7

Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?

23.8

6.8-9.5

-3.9

Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?

23.8

6.8-9.5

-3.9

Publishers (distinguished by color)
Does the affect of discovery service differ across publishers?

Publishers (distinguished by color)
Publishers (distinguished by color)
Full Model –
including Discovery service, Publisher,
and Library
ANOVA Model including all three factors
Results - Can we detect differences between
Discovery Services, Publishers, and/or
Libraries and/or their interactions?
Di...
Interpretations & Conclusions
Analyzing usage is a complex task
No discovery service increased or decreased
usage across a...
A plethora of pending possible pursuits
• Design & test for effects of:
–Aggregator full text availability
–Institution Si...
Questions

michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Upcoming SlideShare
Loading in …5
×

Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage

1,488 views

Published on

Plenary session for Charleston Conference 2013. Authors: Michael Levine-Clark, John McDonald, Jason Price. In this first large scale study of the effect of discovery systems on electronic resource usage, the authors present initial findings on how these systems alter online journal usage by academic library researchers. The study examines usage of content hosted by four major academic journal publishers at 24 libraries that have implemented one of the major discovery systems, EBSCO's EDS, Ex Libris' Primo, OCLC's Worldcat Local, or SerialsSolutions’ Summon. A statistically rigorous comparison of COUNTER-compliant journal usage at each library from the 12 months before and after implementation will determine the degree to which usage rises or falls after discovery tool implementation and address rumors that discovery tools differ in their impact on electronic resource usage.

Published in: Education, Business, Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,488
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
1
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Selecting the libraries:
    Maximized number of libraries balancing the number and size of libraries representing each discovery service and keeping implementation dates within a year or two of each other
  • Mention that there were only 3 institutions + publisher groupings that we had to leave out
    Usable observations = journals at institutions that had counter report values for individ
  • Explain concept of outliers and why they are removed.
    Z-Score on the Y axis is calculated as the number of standard deviations from the mean.
    Of the 141,048 obs, we eliminated only ~100 outliers.
  • Although we were asked to keep the identity of the publishers confidential, we have used a consistent color scheme to identify results that pertain to each publisher
  • Summary of all journals from a publisher to the libraries having that discovery service. Many of the bars include total journal observations in excess of 10,000 while some of those, particularly for the smallest two publishers (in green an orange) include between 500-1000 observations.
  • Analysis of Variance (or ANOVA ) allows us to determine whether the dffferences we observed are significant
    The analysis Breaks down the influences on each observation into the effect of levels of the factor of interest (I.e. Lib, Pub, Disc) and error
    The numbers show an analysis of one observation – each value expresses the difference from the mean of 1 level (Lib x, Pub Y, and Disc Srvc Z) as well as the portion of the value that is not determined by these factors (i.e. the residual error)
    If we imagine the hundreds of observations for each combination, we can understand that the size of the residual error relative to the value determines whether the effects are significant predictors of the observed values
  • Determines whether means (or averages) of each level within a factor are distinguishable from each other
    or put another way it assesses the likelihood that journal change values sampled from different levels of a factor (say libraries) are actually estimating the same population…
    or statistically different populations
  • So, in response to our first question: Does usage change vary across libraries?
    We see the 24 libraries sorted by mean change along the x axis and
    mean change in usage plus or minus two standard errors on the y axis
    Standard errors are a measure of the variability around each individual library
    In general, when 2xSE bars overlap, those means are not distinguishable
    The overall average change was 8.5….
    And our F-ratio of about 32 tells us that institution alone is a significant predictor of mean change in usage after discovery service implementation.
  • Whenever the p value (shown in the significance column) is less than 0.05, it indicates that we can reject the null hypothesis that there is no differences among levels of the factor (in this case, libraries)
    But for our single factor anovas, this ignores the impact of different discovery services and journal publishers on mean change in usage
  • Click The grand mean for change in usage across publishers is 8.9
    One publisher appears to have a mean change that isnt distinguishable from zero, whereas…
    And the significant F value and non overlapping error bars suggest that the mean change varied across publishers, BUT
    As we will see, when we add the other factors we’ll find that these differences are actually explained by discovery service and institution effects rather than publisher differences
  • This data shows that the mean usage increase was positive for all discovery services, ….
    Although we cant distinguish these from industry wide increases since we didn’t examine usage change in libraries that did NOT implement a discovery service
    So when we just look at discovery srvc & ignore the variation due to publisher or library we do see differences
  • When we include both Discovery Service and Publisher in a Two way model,
    We can ask whether we can detect differences when we take both Discovery Means & Publisher means into account
    In addition to asking whether we can detect a difference across discovery service alone and publisher alone
    The two way model addresses whether the impact of discovery service is equivalent for each publisher
    We can think of it as asking whether these lines are parallel (statistically) or whether they cross
  • These are the same data in the previous slide separated into panels by discovery service.
    The following slides will step through testing for an effect of discovery service, publisher, and their interaction
  • Do the discovery service means we see here differ significantly in the two way model?
  • Do the discovery service means we see here differ significantly in the two way model?
    No– they do not.
  • Do the publisher means we see here differ significantly in the two way model?
  • Do the publisher means we see here differ significantly in the two way model?
    No–they do not.
  • Does the affect of discovery service differ across publishers?
  • Does the affect of discovery service differ across publishers?
    Yes, it does. Statistically these lines are not parallel.
  • Mixed Nested Parially-crossed model
    Acknowledges that each library can only implement one discovery tool
    Takes all three variables into account in the same test.
  • To reiterate the full model results,
  • Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage

    1. 1. Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage Charleston Conference November 7, 2013 Michael Levine-Clark, University of Denver John McDonald, University of Southern California Jason Price, SCELC Consortium
    2. 2. “…a steep increase in full text downloads and link resolver click‐ throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/
    3. 3. Vendor marketing http://www.oclc.org/partnerships/econtent/solutions.en.html
    4. 4. Does implementation of a discovery service impact journal usage?
    5. 5. Web-scale discovery services • Single source for finding information – Books – Articles – Local content • Metadata and/or full text • Content is pre-indexed and/or pre-harvested • Single fast search ILS ILS Publisher Publisher Metadata Metadata MLA MLA Bibliography Bibliography Institutional Institutional Repository Repository HathiTrust HathiTrust Discovery Service Discovery Service
    6. 6. An assumption • At any given institution, given a relatively stable user base, the total search effort will remain roughly the same. – X students will have Y assignments and Z hours per day to search – X faculty will publish Y papers and have Z hours per day to search
    7. 7. Discovery services  Will take up an increasing amount of a finite time for searching  Will draw users from other (more or less efficient) search tools  Will alter the overall productivity of searches (users will find more or less)  Will alter the overall efficiency of users (users will access more or less full-text)
    8. 8. Prior studies • Some studies have indicated substantial increases in usage after Discovery implementation – Descriptive statistics only – Single institution studies only • Some publishers report decreased usage of content – Anecdotal, may affect some and not others
    9. 9. Data collection • List of libraries with discovery services > Searched on lib-web-cats • Surveyed Libraries > Discovery service Implemented > Implementation Date (month/year) > Search box location > Marketing effort • 149 Libraries Gave Approval > 24 libraries selected for this phase > 6 for each of the 4 major discovery services
    10. 10. Library demographics • 20 US, 1 each from UK, AUS, NZ, CA • 10 ARL Libraries included • WorldCat book holdings > Average: 1,114,193 > Median: 1,044,153 > High: 2,665,796 > Low: 298,365 • Implementation dates: > 2010 (3), 2011 (19), 2012 (2)
    11. 11. Dataset • 24 Libraries • 4 Discovery services • 6 Publishers • 9,206 Journals • 159,278 Observations • 141,048 Usable Observations
    12. 12. Methodology Compared COUNTER JR1 total full text article views for the 12 months before vs 12 months after implementation date Year 1 Year 2 Included implementation month in Year 1 to ensure that both periods included an entire academic year
    13. 13. Collections notes o Excluded journals that did not have 24 months of COUNTER reporting o Limited ability to control for changes in aggregator, backfile access, or expanded holdings o Outliers removed from analysis
    14. 14. General trends • Variation by institution within each discovery service • Variation by publisher within each discovery service • Some publishers saw overall net increase, while some experienced a decrease in usage
    15. 15. Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level Start with an exploratory analysis and end with a comprehensive model
    16. 16. ANOVA - Analyzing the data Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8) After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
    17. 17. ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other Average variability due to the factor F-ratio = --------------------------------------------------Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
    18. 18. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
    19. 19. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
    20. 20. Does usage change vary across publishers? Overall Average = 8.9 Publisher (sorted by Mean Change)
    21. 21. Does usage change vary across discovery services? Overall Average = 8.9
    22. 22. Does the affect of discovery service differ across publishers?
    23. 23. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
    24. 24. Do the discovery service means differ in the 2 way model? 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
    25. 25. 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
    26. 26. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
    27. 27. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
    28. 28. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
    29. 29. Publishers (distinguished by color)
    30. 30. Full Model – including Discovery service, Publisher, and Library
    31. 31. ANOVA Model including all three factors
    32. 32. Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes Differential library effect by publisher -- Yes
    33. 33. Interpretations & Conclusions Analyzing usage is a complex task No discovery service increased or decreased usage across all libraries and/or all publishers > Discovery service and publisher as variables on their own were significant predictors of usage change > Interaction of Discovery service & Publisher was significant > Some control needed for no discovery service and for size of institution. > >
    34. 34. A plethora of pending possible pursuits • Design & test for effects of: –Aggregator full text availability –Institution Size / Enrollment Profile –Publisher Size –Journal Subject –Overall usage trends (Requires Disc Srvc ‘control’) –Configuration options in Discovery services • Follow-up presentation at UKSG (April 2014) –Including Control group & Additional libraries –Add Additional variables & further analysis
    35. 35. Questions michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org

    ×