Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage

1,210 views

Published on

Plenary session for Charleston Conference 2013. Authors: Michael Levine-Clark, John McDonald, Jason Price. In this first large scale study of the effect of discovery systems on electronic resource usage, the authors present initial findings on how these systems alter online journal usage by academic library researchers. The study examines usage of content hosted by four major academic journal publishers at 24 libraries that have implemented one of the major discovery systems, EBSCO's EDS, Ex Libris' Primo, OCLC's Worldcat Local, or SerialsSolutions’ Summon. A statistically rigorous comparison of COUNTER-compliant journal usage at each library from the 12 months before and after implementation will determine the degree to which usage rises or falls after discovery tool implementation and address rumors that discovery tools differ in their impact on electronic resource usage.

Published in: Education, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,210
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage

  1. 1. Discovery  or  Displacement?   A  Large  Scale  Longitudinal  Study  of  the   Effect  of  Discovery  Systems  on  Online   Journal  Usage   Charleston  Conference   November  7,  2013   Michael  Levine-­‐Clark,  University  of  Denver   John  McDonald,  University  of  Southern  California   Jason  Price,  SCELC  ConsorJum  
  2. 2. “…a steep increase in full text downloads and link resolver click‐ throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/ library_sp/9/
  3. 3. Vendor  marke5ng   h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html  
  4. 4. Does implementation of a discovery  service impact journal usage?
  5. 5. Web-­‐scale  discovery  services   •  Single  source  for   finding  informaJon   –  Books   –  ArJcles   –  Local  content   •  Metadata  and/or  full   text   •  Content  is  pre-­‐indexed   and/or  pre-­‐harvested   •  Single  fast  search     ILS   Publisher   Metadata   MLA   Bibliography   InsJtuJonal   Repository   HathiTrust   Discovery  Service  
  6. 6. An  assump5on   •  At  any  given  insJtuJon,  given  a  relaJvely   stable  user  base,  the  total  search  effort  will   remain  roughly  the  same.   –  X  students  will  have  Y  assignments  and  Z  hours   per  day  to  search   –  X  faculty  will  publish  Y  papers  and  have  Z  hours   per  day  to  search  
  7. 7. Discovery  services   §  Will  take  up  an  increasing  amount  of  a  finite   Jme  for  searching   §  Will  draw  users  from  other  (more  or  less   efficient)  search  tools   §  Will  alter  the  overall  producJvity  of  searches   (users  will  find  more  or  less)   §  Will  alter  the  overall  efficiency  of  users  (users   will  access  more  or  less  full-­‐text)  
  8. 8. Prior  studies   •  Some  studies  have  indicated  substanJal   increases  in  usage  a]er  Discovery   implementaJon   –  DescripJve  staJsJcs  only   –  Single  insJtuJon  studies  only   •  Some  publishers  report  decreased  usage  of   content   –  Anecdotal,  may  affect  some  and  not  others  
  9. 9. Data  collec5on   •  List  of  libraries  with  discovery  services   > Searched  on  lib-­‐web-­‐cats   •  Surveyed  Libraries   > Discovery  service  Implemented   > ImplementaJon  Date  (month/year)   > Search  box  locaJon   > MarkeJng  effort   •  149  Libraries  Gave  Approval   > 24  libraries  selected  for  this  phase   > 6  for  each  of  the  4  major  discovery  services  
  10. 10. Library  demographics   •  20  US,  1  each  from  UK,  AUS,  NZ,  CA   •  10  ARL  Libraries  included   •  WorldCat  book  holdings   > Average:  1,114,193   > Median:  1,044,153   > High:  2,665,796   > Low:  298,365   •  ImplementaJon  dates:     > 2010  (3),  2011  (19),  2012  (2)  
  11. 11. Dataset   •  24  Libraries   •  4  Discovery  services   •  6  Publishers   •  9,206  Journals   •  159,278  ObservaJons   •  141,048  Usable  ObservaJons    
  12. 12. Methodology   Compared  COUNTER  JR1  total  full  text  arJcle  views  for  the                 Included  implementaJon  month  in  Year  1  to  ensure  that   both  periods  included  an  enJre  academic  year   End   Year  2   May  2012   Year  1   ImplementaJon   May  2011   Start   June  2010   12  months  before  vs  12  months  a]er  implementaJon  date  
  13. 13. Collec5ons  notes                 o  Excluded  journals  that  did  not  have  24   months  of  COUNTER  reporJng   o  Limited  ability  to  control  for  changes  in   aggregator,  backfile  access,  or  expanded   holdings   o  Outliers  removed  from  analysis  
  14. 14. General  trends   •  VariaJon  by  insJtuJon  within  each   discovery  service   •  VariaJon  by  publisher  within  each   discovery  service   •  Some  publishers  saw  overall  net   increase,  while  some  experienced  a   decrease  in  usage  
  15. 15. Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level Start with an exploratory analysis and end with a comprehensive model
  16. 16. ANOVA - Analyzing the data Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8) After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
  17. 17. ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other Average variability due to the factor F-ratio = --------------------------------------------------Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
  18. 18. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
  19. 19. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
  20. 20. Does usage change vary across publishers? Overall Average = 8.9 Publisher (sorted by Mean Change)
  21. 21. Does usage change vary across discovery services? Overall Average = 8.9
  22. 22. Does the affect of discovery service differ across publishers?
  23. 23. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
  24. 24. Do the discovery service means differ in the 2 way model? 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
  25. 25. 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
  26. 26. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
  27. 27. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
  28. 28. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
  29. 29. Publishers (distinguished by color)
  30. 30. Full Model – including disc srvc, publisher, and library
  31. 31. ANOVA Model including all three factors
  32. 32. Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes Differential library effect by publisher -- Yes
  33. 33. Interpreta5ons  &  Conclusions   >  >  >  >  >  Analyzing  usage  is  a  complex  task   No  discovery  service  increased  or  decreased   usage  across  all  libraries  and/or  all  publishers   Discovery  service  and  publisher  as  variables  on   their  own  were  significant  predictors  of  usage   change   InteracJon  of  Discovery  service  &  Publisher  was   significant   Some  control  needed  for  no  discovery  service   and  for  size  of  insJtuJon.  
  34. 34. A  plethora  of  pending  possible  pursuits   •  Design  &  test  for  effects  of:     –  –  –  –  –  –  Aggregator  full  text  availability   InsJtuJon  Size  /  Enrollment  Profile   Publisher  Size       Journal  Subject   Overall  usage  trends  (Requires  Disc  Srvc  ‘control’)   ConfiguraJon  opJons  in  Discovery  services   •  Follow-­‐up  presenta5on  at  UKSG  (April  2014)   –  Including  Control  group  &  AddiJonal  libraries   –  Add  AddiJonal  variables  &  further  analysis  
  35. 35. Ques5ons   michael.levine-­‐clark@du.edu  |  johndmcd@usc.edu  |  jason@scelc.org    

×