Your SlideShare is downloading. ×
0
Discovery	
  or	
  Displacement?	
  

A	
  Large	
  Scale	
  Longitudinal	
  Study	
  of	
  the	
  
Effect	
  of	
  Discove...
“…a steep increase in full text
downloads and link resolver click‐
throughs suggests Summon had a
dramatic impact on user ...
Vendor	
  marke5ng	
  

h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html	
  
Does implementation of a discovery 
service impact journal usage?
Web-­‐scale	
  discovery	
  services	
  
•  Single	
  source	
  for	
  
finding	
  informaJon	
  
–  Books	
  
–  ArJcles	
...
An	
  assump5on	
  
•  At	
  any	
  given	
  insJtuJon,	
  given	
  a	
  relaJvely	
  
stable	
  user	
  base,	
  the	
  t...
Discovery	
  services	
  
§  Will	
  take	
  up	
  an	
  increasing	
  amount	
  of	
  a	
  finite	
  
Jme	
  for	
  searc...
Prior	
  studies	
  
•  Some	
  studies	
  have	
  indicated	
  substanJal	
  
increases	
  in	
  usage	
  a]er	
  Discove...
Data	
  collec5on	
  
•  List	
  of	
  libraries	
  with	
  discovery	
  services	
  
> Searched	
  on	
  lib-­‐web-­‐cats...
Library	
  demographics	
  
•  20	
  US,	
  1	
  each	
  from	
  UK,	
  AUS,	
  NZ,	
  CA	
  
•  10	
  ARL	
  Libraries	
 ...
Dataset	
  
•  24	
  Libraries	
  
•  4	
  Discovery	
  services	
  
•  6	
  Publishers	
  
•  9,206	
  Journals	
  
•  15...
Methodology	
  
Compared	
  COUNTER	
  JR1	
  total	
  full	
  text	
  arJcle	
  views	
  for	
  the	
  

	
  
	
  
	
  
	...
Collec5ons	
  notes	
  

	
  
	
  
	
  
	
  	
  	
  
	
  

o  Excluded	
  journals	
  that	
  did	
  not	
  have	
  24	
  ...
General	
  trends	
  
•  VariaJon	
  by	
  insJtuJon	
  within	
  each	
  
discovery	
  service	
  
•  VariaJon	
  by	
  p...
Goals of our inferential statistics
Determine whether observed differences are
significant or resulted from chance effects...
ANOVA - Analyzing the data
Observation

=

Fit

+ Residual

Change
In = Library x + Publisher y + Disc Svc z + Residual Er...
ANOVA – F Ratio
Tests whether the means for levels within a
factor are distinguishable from each other
Average variability...
Does usage change vary across libraries?

Overall Average = 8.5

Institution (sorted by Mean Change)
Does usage change vary across libraries?

Overall Average = 8.5

Institution (sorted by Mean Change)
Does usage change vary across publishers?

Overall Average = 8.9

Publisher (sorted by Mean Change)
Does usage change vary across discovery services?

Overall Average = 8.9
Does the affect of discovery service differ across publishers?
Does the affect of discovery service differ across publishers?

Publishers (distinguished by color)
Do the discovery service means differ in the 2 way model?

15.0
12.3
4.5

3.7

Publishers (distinguished by color)
15.0
12.3
4.5

3.7

Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?

23.8

6.8-9.5

-3.9

Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?

23.8

6.8-9.5

-3.9

Publishers (distinguished by color)
Does the affect of discovery service differ across publishers?

Publishers (distinguished by color)
Publishers (distinguished by color)
Full Model –
including disc srvc, publisher, and library
ANOVA Model including all three factors
Results - Can we detect differences between
Discovery Services, Publishers, and/or
Libraries and/or their interactions?
Di...
Interpreta5ons	
  &	
  Conclusions	
  
> 
> 
> 

> 
> 

Analyzing	
  usage	
  is	
  a	
  complex	
  task	
  
No	
  discove...
A	
  plethora	
  of	
  pending	
  possible	
  pursuits	
  
•  Design	
  &	
  test	
  for	
  effects	
  of:	
  	
  
– 
– 
– ...
Ques5ons	
  

michael.levine-­‐clark@du.edu	
  |	
  johndmcd@usc.edu	
  |	
  jason@scelc.org	
  	
  
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Upcoming SlideShare
Loading in...5
×

Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage

2,349

Published on

Levine-Clark, Michael, John McDonald, and Jason Price, "Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage," Charleston Conference, November 7, 2013.

Plenary session for Charleston Conference 2013. Authors: Michael Levine-Clark, John McDonald, Jason Price. In this first large scale study of the effect of discovery systems on electronic resource usage, the authors present initial findings on how these systems alter online journal usage by academic library researchers. The study examines usage of content hosted by four major academic journal publishers at 24 libraries that have implemented one of the major discovery systems, EBSCO's EDS, Ex Libris' Primo, OCLC's Worldcat Local, or SerialsSolutions’ Summon. A statistically rigorous comparison of COUNTER-compliant journal usage at each library from the 12 months before and after implementation will determine the degree to which usage rises or falls after discovery tool implementation and address rumors that discovery tools differ in their impact on electronic resource usage.

Published in: Education, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,349
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
62
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage"

  1. 1. Discovery  or  Displacement?   A  Large  Scale  Longitudinal  Study  of  the   Effect  of  Discovery  Systems  on  Online   Journal  Usage   Charleston  Conference   November  7,  2013   Michael  Levine-­‐Clark,  University  of  Denver   John  McDonald,  University  of  Southern  California   Jason  Price,  SCELC  ConsorJum  
  2. 2. “…a steep increase in full text downloads and link resolver click‐ throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/ library_sp/9/
  3. 3. Vendor  marke5ng   h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html  
  4. 4. Does implementation of a discovery  service impact journal usage?
  5. 5. Web-­‐scale  discovery  services   •  Single  source  for   finding  informaJon   –  Books   –  ArJcles   –  Local  content   •  Metadata  and/or  full   text   •  Content  is  pre-­‐indexed   and/or  pre-­‐harvested   •  Single  fast  search     ILS   Publisher   Metadata   MLA   Bibliography   InsJtuJonal   Repository   HathiTrust   Discovery  Service  
  6. 6. An  assump5on   •  At  any  given  insJtuJon,  given  a  relaJvely   stable  user  base,  the  total  search  effort  will   remain  roughly  the  same.   –  X  students  will  have  Y  assignments  and  Z  hours   per  day  to  search   –  X  faculty  will  publish  Y  papers  and  have  Z  hours   per  day  to  search  
  7. 7. Discovery  services   §  Will  take  up  an  increasing  amount  of  a  finite   Jme  for  searching   §  Will  draw  users  from  other  (more  or  less   efficient)  search  tools   §  Will  alter  the  overall  producJvity  of  searches   (users  will  find  more  or  less)   §  Will  alter  the  overall  efficiency  of  users  (users   will  access  more  or  less  full-­‐text)  
  8. 8. Prior  studies   •  Some  studies  have  indicated  substanJal   increases  in  usage  a]er  Discovery   implementaJon   –  DescripJve  staJsJcs  only   –  Single  insJtuJon  studies  only   •  Some  publishers  report  decreased  usage  of   content   –  Anecdotal,  may  affect  some  and  not  others  
  9. 9. Data  collec5on   •  List  of  libraries  with  discovery  services   > Searched  on  lib-­‐web-­‐cats   •  Surveyed  Libraries   > Discovery  service  Implemented   > ImplementaJon  Date  (month/year)   > Search  box  locaJon   > MarkeJng  effort   •  149  Libraries  Gave  Approval   > 24  libraries  selected  for  this  phase   > 6  for  each  of  the  4  major  discovery  services  
  10. 10. Library  demographics   •  20  US,  1  each  from  UK,  AUS,  NZ,  CA   •  10  ARL  Libraries  included   •  WorldCat  book  holdings   > Average:  1,114,193   > Median:  1,044,153   > High:  2,665,796   > Low:  298,365   •  ImplementaJon  dates:     > 2010  (3),  2011  (19),  2012  (2)  
  11. 11. Dataset   •  24  Libraries   •  4  Discovery  services   •  6  Publishers   •  9,206  Journals   •  159,278  ObservaJons   •  141,048  Usable  ObservaJons    
  12. 12. Methodology   Compared  COUNTER  JR1  total  full  text  arJcle  views  for  the                 Included  implementaJon  month  in  Year  1  to  ensure  that   both  periods  included  an  enJre  academic  year   End   Year  2   May  2012   Year  1   ImplementaJon   May  2011   Start   June  2010   12  months  before  vs  12  months  a]er  implementaJon  date  
  13. 13. Collec5ons  notes                 o  Excluded  journals  that  did  not  have  24   months  of  COUNTER  reporJng   o  Limited  ability  to  control  for  changes  in   aggregator,  backfile  access,  or  expanded   holdings   o  Outliers  removed  from  analysis  
  14. 14. General  trends   •  VariaJon  by  insJtuJon  within  each   discovery  service   •  VariaJon  by  publisher  within  each   discovery  service   •  Some  publishers  saw  overall  net   increase,  while  some  experienced  a   decrease  in  usage  
  15. 15. Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level Start with an exploratory analysis and end with a comprehensive model
  16. 16. ANOVA - Analyzing the data Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8) After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
  17. 17. ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other Average variability due to the factor F-ratio = --------------------------------------------------Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
  18. 18. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
  19. 19. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
  20. 20. Does usage change vary across publishers? Overall Average = 8.9 Publisher (sorted by Mean Change)
  21. 21. Does usage change vary across discovery services? Overall Average = 8.9
  22. 22. Does the affect of discovery service differ across publishers?
  23. 23. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
  24. 24. Do the discovery service means differ in the 2 way model? 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
  25. 25. 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
  26. 26. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
  27. 27. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
  28. 28. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
  29. 29. Publishers (distinguished by color)
  30. 30. Full Model – including disc srvc, publisher, and library
  31. 31. ANOVA Model including all three factors
  32. 32. Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes Differential library effect by publisher -- Yes
  33. 33. Interpreta5ons  &  Conclusions   >  >  >  >  >  Analyzing  usage  is  a  complex  task   No  discovery  service  increased  or  decreased   usage  across  all  libraries  and/or  all  publishers   Discovery  service  and  publisher  as  variables  on   their  own  were  significant  predictors  of  usage   change   InteracJon  of  Discovery  service  &  Publisher  was   significant   Some  control  needed  for  no  discovery  service   and  for  size  of  insJtuJon.  
  34. 34. A  plethora  of  pending  possible  pursuits   •  Design  &  test  for  effects  of:     –  –  –  –  –  –  Aggregator  full  text  availability   InsJtuJon  Size  /  Enrollment  Profile   Publisher  Size       Journal  Subject   Overall  usage  trends  (Requires  Disc  Srvc  ‘control’)   ConfiguraJon  opJons  in  Discovery  services   •  Follow-­‐up  presenta5on  at  UKSG  (April  2014)   –  Including  Control  group  &  AddiJonal  libraries   –  Add  AddiJonal  variables  &  further  analysis  
  35. 35. Ques5ons   michael.levine-­‐clark@du.edu  |  johndmcd@usc.edu  |  jason@scelc.org    
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×