Measuring Value and ROI of Academic Libraries: The IMLS Lib Value Project


Published on

Presented by Carol Tenopir and Rachel Fleming-May, University of Tennessee, and Tina Chrzastowski, University of Illinois Urbana Champaign

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Also University of Pittsburgh, Drexel, 4 universities in NY
  • Academics read a lot—they report on average reading 25 articles, 8 books or book chapters, and almost 12 other publications. This appears to be pretty close to reading in the US and Australia. On average across all disciplines. 295-300 per year compares to 280 per year 23.3 per month in 2005 in the U.S.Books or book chaptersOther publications include web sites, conference proceedings, gov docs
  • Source of book readings is different—purchased or from publisher is main source; library is second.
  • <1% responded they read the article “elsewhere”For ALL responses (n=1162): Library was only 1.7%, Office/Lab was 60.2%, Home was 27.9%, and traveling was 10%
  • Value of virtual visits to the library.
  • Question: Did you obtain your article through a print or electronic source?2011 Frequencies: Electronic: 658 (print=49)2005: Electronic: 394
  • Outcomes from reading are another important explicit value that go beyond just sheer amount of reading. From nearly 900 respondents in the United States (Tenopir et al, 2009a), readings were found to have many important outcomes. Faculty members told us that their Readings have profound outcomes. They:• Inspire new thinking (55%/54 of readings)• Improve results (40%/38 of readings)• Narrow, broaden, or change the focus (27%/28 of readings)• Resolve technical problems (12%10 of readings)• Save time (12%/10 of readings)• Lead to faster completion of the task at hand (7% /5 of readings)• Foster collaborations (6%/4 of readings)• Waste the time of the reader (<1%/<1 of readings)
  • Our data will not match reported ARL ebook data since we report titles acquired to ARL and publisher use stats record volumes acquired.Also data for 2008 is sketchy for use data.
  • **Note #New Ebooks for 2008 is a rougher estimate. Cost per PRINT book in the CARLI consortia 2003-2008 was $64.55.Spiro and Henry note that while “the initial costs of acquiring electronic contents may be higher (than print), the long-term costs will be lower” IN fact, my data point to lower e content initial costs AND low long-term costs.
  • ** Some vendors have only one way of reporting use – downloads/views of the entire book rather than individual sections. This is rare, only a few vendors, but it means that that use is a view of the entire book. **Since use data gathering could only get to 82%, it’s not 100%!! Not all use was able to be counted.
  • 7 of the 40 ebook vendors we purchase ebooks from were not able to supply ebook use data. Also note that Open Access ebooks are not currently providing use data (just like open access ejournal publishers). One of the many problems with these data is that we often buy huge ebook packages at the end of a fiscal year, or they get turned on then – those are counted as ebooks available or purchased in that year, but actually can’t really be used until the next fiscal year. AND, total use for earlier years are less reliable (undercounting) due to unavailable data.*Total uses per year generally undercounted because only publishers for which we have data--which actually excludes most Ebooks. According to ARL data, our largest Ebook collections come from free resources that don't track usage for us: Early English Books Online, ECCO, Archive of Americana, Monograph print study (CARLI Consortia) found a range of cost/use from $18.00 to $35.00 for print monos (five years, FY2003-2008)
  • Ebsco only started providing ebook use data in 2011,and very spotty too! Not useful. No password, despite a few months of trying, from Archives of Americana. EEBO is used mostly as a database for locating where print copies of the books are located. American Council of Learned Societies had only a few titles that were used a LOT. Over 7,000 uses of one book in a few months, obviously used for a text for a class. E-reserves basically. Again, return users not unique users (but not that there’s anything wrong with that).
  • So 11% of ebook holdings account for nearly 50% of use; not exactly the 80/20 rule, but close! Wiley switched to normal COUNTER data in calendar year 2009, and the way they report usage prior to that means that the #Ebooks Total for FY2008 is incomplete. So very likely if we had the full number you wouldn't see that 20% to 10% dip in %Ebooks Used by Publisher from 2008-2009 that you see on the graph now--the %used in 2008 for Wiley is realistically lower than 20%.
  • Large proportion of unused ebooks is due to buying large package deals; more selective buying, or PDA ebooks might result in a more thoroughly used collection. Very much “just in case” instead of “just in time.” However, in a different, broader study of monograph use, print monographs had a “not used” rate of 33% (CARLI consortia) over a 5-year period. Other studies have shows that “not used” rates of up to Approval books at Penn State had a non-circ rate of 31% and UIUC’s was 40% (approximately a 2.5-year study period).we continue to buy “Just in Case” books in e format, just like we do in print. BUT, lower costs overall for e books. We can afford to have a broader collection since there are lower costs associated with ebooks. More on that to come later in this presentation.
  • This is a high number of AVERAGE uses per ebook. And trending upwards. Due perhaps to multiple users possible per ebook, or returning users who “check out” the ebook each time they use it. ???
  • Are these repeat users or new users? Probably some of both, but the lack of a distinct “comet tail” of single uses per single book means that ebooks are used differently than print books. Repeat users equivalent to renewals in print circulation?
  • Paul N. Courant and Matthew “Buzzy” Nielsen in: The Idea of Order: Transforming Research Collections for 21st Century Scholarship. Council on Library and Information Resources (CLIR), Washington, D.C. Accessed online at:
  • To summarize this section,
  • IRB APPROVAL RECEIVED FIRST! Elsevier generously offered an incentive of a $100 Amazon gift voucher to each participant. The response was overwhelming. Within a week nearly 400 UIUC volunteers had signed up to participate. Elsevier asked us to stop recruiting on October 15, and they eventually weeded the number of UIUC participants to 129 faculty and Ph.D. students.
  • Very few Humanists and Social Scientists. Self-selecting? Lots of chemists (and other physical scientists and engineers) who have been widely exposed to ebooks for the past 5-6 years.Social Science + Humanities + Interdisciplinary = 8 0r less than 1 percent. So looking at results by discipline is not valid. I included a few just to see how those 7 responded!
  • Monthly or weekly or daily basis = about 76%. Somewhat surprising for scientists? I don’t think so, but there is a sense out there that scientists and engineers don’t use books. Wrong!Daily or weekly basis: 47.3% or nearly half of users are “frequent users”
  • Nothing new here! Can cite other studies with similar results.
  • Asked to check as many boxes as apply. Not wildly different uses and needs from the different disciplines – in fact very much the same across the disciplines. Ebook use behaviour is not determined by discipline or subject of the ebook.
  • Note here that the earlier, self-reported “Behavior” pretty closely matches what they value. So we may be able to say that in the future behavior is a predictor of value, which makes sense. People don’t spend time doing things they don’t value. Even when you are rewarding them for their participation.
  • 67.4% say they either “Need to have” or it would be “nice to have” the ebook they viewed. Based on 129 persons reading over 800 ebooks
  • Comment box following the valuation question elicited the most responses, 435 comments. That comment box was simply labeled “Please elaborate” So we saw in the previous slide that the response was mostly positive in terms of value (70/30 split basically). However, more comments that followed the question were negative, although by a small margin. Because the overwhelming negative comment was about the lack of relevant material, not about format, we can see that this model has “legs” and can keep running. Content and access needs to improve. But users are ready.More negatives than positives, most focusing on lack of relevant material and not having access when neededTotal responses totaled 435
  • YMMD Your Mileage May Differ! USE IS LOCALLY INFLUENCED!
  • “Non-Standard” here means “confusing to users!”
  • Measuring Value and ROI of Academic Libraries: The IMLS Lib Value Project

    1. 1. Measuring Value and ROI of Academic Libraries:The IMLS Lib-Value Project Carol Tenopir University of Tennessee Charleston 2011 Center for Information and Communication Studies
    2. 2. Multiple institutions usingLIB-VALUE: multiple methods to measure multiple values for multiple stakeholders Center for Information and Communication Studies
    3. 3. Measuring value Special Information Ebooks Collections Commons Journal Teaching and All ServicesCollections Learning Website andReading and Building Tools ValueScholarship Bibliography Center for Information and Communication Studies
    4. 4. Return on investment in a strict sense……is a quantitative measure expressed as a ratio of the value returned to the institution for each monetary unit invested in the library. For every $/€/£ spent on the library, the university received ‘X’ $/€/£ in return. Demonstrate that library collections contribute to income-generating activities Center for Information and Communication Studies
    5. 5. Return on investment is also……values of all types that come to stakeholders and the institution from the library’s collections, services, and contribution to its communities. Center for Information and Communication Studies
    6. 6. In the information context economistMachlup described 2 types of value:1.purchase or exchange value: what one is willing to pay for information in money and/or time, and2. use value: the favorable consequences derived from reading and using the information. Center for Information and Communication Studies
    7. 7. Lib-Value Comprehensive Library Value Study (Bruce Kingma)• Economic (private) – What is the value to an individual to use the library resources?• Social (public) – What is the value to the institution of the library?• Environmental (externality) – What is the value of the environmental savings of library provision of electronic resources? – Have libraries gone green without knowing it? Center for Information and Communication Studies
    8. 8. Readings for work related purposes Center for Information and Communication Studies
    9. 9. Average readings per month: U.K. faculty, 201130 25252015 1110 8 5 0 Article Book Other Publicationn=2117, June 9 2011, 6 UKuniversities Center for Information and Communication Studies
    10. 10. Source of article readings 100 90 80 70 65 60 Percent 50 40 30 20 14 9 6 6 10 0 Library Free Web Dept. or Personal Colleagues Copy Other Subscription Journal, Website Subscriptionn=1189, June 9, 2011, 6 U.K universities Center for Information and Communication Studies
    11. 11. Where did you read this last article? (Articles from library only) Home 26% Travelling 10% Library 2% Office, Lab 62%n=764, June 9, 2011, 6 Center for Information and Communication StudiesU.K universities
    12. 12. Preliminary faculty survey results Average last 30 % of Activity days respondents Physical Visits 2.9 visits 73% Remote Visits 14.2 visits 88%Average Total Resources Used: in-person visit to the library 7.3 uses 80% remotely online 14.9 uses 89% Center for Information and Communication Studies
    13. 13. Use of library collections for articles US, 2005 UK, 2011Print Print28% 7% Electronic Electronic 72% 93% n=562 n=775June 9, 2011, 6 U.K universities Center for Information and Communication Studies
    14. 14. Outcomes of journal article reading 2004-06 20111st Inspire new thinking or ideas 55% 54%2nd Improve results 40% 38%3rd Narrow/broaden/change the focus 27% 28%4th Resolve technical problems 12% 10%5th Save time or other resources 12% 10%6th Aid in faster completion 7% 5%7th Assist or result in 6% 4%collaboration/joint research Center for Information and Communication Studies
    15. 15. Portrait of a successful faculty member… ” •Publishes more •Wins awards •Reads more •Reads more from the library •For every article cited, reads 27-40 additional articles Center for Information and Communication Studies
    16. 16. More details and searchable Lib-Valuebibliographic database available on the project website: Center for Information and Communication Studies
    17. 17. LIB-VALUE:TEACHING & LEARNINGRachel A. Fleming-May, Assistant ProfessorSchool of Information Sciences,The University of Tennessee
    18. 18. • Value of academic library resources &services in support of teaching • Instructors: •Survey, to be followed by “real time” conversation
    19. 19. • All UTK Constituents with instructional responsibilities:Survey:  Tenured/tenure-track faculty  “Clinical” faculty  Part-time faculty  GTA’s  Administrators (e.g., Dean of Students’ Office) • Materials used for teaching support, whether or not provided by UTK Libraries  Readings, etc., for students, print, electronic, other formats  Reading to support own pedagogical development
    20. 20. Perceived Benefits of Support provided byUTK Libraries: • Savings… of own time of own money of other resources • Improvements… teaching course-relatedmaterials student performance
    21. 21. Determining the Focus of Inquiry: Institutional priorities VolVision 2015
    22. 22. …And FederalHigher Education Opportunity ActRegulations
    23. 23. Communication & InformationColleges:Agricultural Sciencesand Natural Education, HeResources alth, HumanArchitecture andDesign ServicesArts and SciencesBusinessAdministrationCommunication and A&SInformationEducation, Health, and Human SciencesEngineeringLaw Agricultural Sciences &Nursing NaturalSocial Work ResourcesVeterinary Medicine
    24. 24. Has your approach to identifying readings for your classeschanged in the past 3-5 years?• They are more likely to search or browse subscription databases for readings. True/ Somewhat True : 59% browse electronic journals to identify readings for my students. True/ Somewhat True : 66%• …and less likely to browse print journals to identify course readings. True/ Somewhat True: 50% (Only 7% more likely to browse print journals)
    25. 25. Has your approach to collecting and distributing readings foryour classes changed in the past 3-5 years?• I require my students to purchase fewer printed textbooks. True or Somewhat True: 34%• I require my students to purchase more printed textbooks. Untrue: 60.0%• I require my students to purchase fewer course packets of printed materials. True or Somewhat True: 31% N/A: 44.4%
    26. 26. As a result of using the Libraries services, collections, or facilities, do you feel that your teaching has improved in any of these ways?• The readings I assign are more up-to-date and/or varied True or Somewhat True: 70%• I read more/more widely to prepare for teaching True/Somewhat True: 63%• My assignments are more creative True/Somewhat True: 48%
    27. 27. Do you feel that yourperformance has improved as a result ofyour using the Librariesservices, collections, or facilities to supportyour teaching?My students are...
    28. 28. True or Somewhat True:• citing sources that are more appropriatefor academic work. 52%• writing citations that are more completeand/or correct. 40%• accessing information from a widervariety of sources. 66%
    29. 29. ≤5 hours6-10 In a typicalhours semester, I save11-15hours time by using the library to support≥16 my teaching.hoursI do notsave time.
    30. 30. In a typicalsemester, using $50 orthe library less…saves memoney that Imight havespent onmaterials (suchas books,journals, orphotocopying)to support myteaching or my +$1001studentslearning. $0
    31. 31. I save…paper and inkpaper because I do not make as many copies as I used to.Money and paper formerly expended on printing/photocopying.Time!Time: I can access services from home so do not need to travel to campus. Can work by my own scheduletime in terms of accessing readings on timely topicstime!!!!
    32. 32. Successes…• “Over the years, the library has been a great support in diverse ways: library orientation is mandatory in our first year; researching precedents is expected of all architecture students from first year on; DMS, Reserve, Studio, Map Library, and Archives, have all assisted me in a great way.”• “My students are also composing information in a wider variety of sources.”• “[My students are] more able to distinguish between valid sources and ‘junk.’”
    33. 33. …and “opportunities”:• “We need clear help with proper citations, especially for images. I am confused about faculty access to image data bases too.”• “Just never thought about [using the library services to support teaching] - especially with distance education students.”• “The level of difficulty of the materials at the library is much more advanced than the subjects I teach.”• “I teach primarily doctoral students. If they need advice about the library they wont make it through the program.”
    34. 34. Caveats Next Steps• Having the survey vetted • Targeted email invitations by librarians was • Conversations with essential. faculty• Be prepared for • Student learning and “backseat driving” about experience: the instrument itself.  Multi-phase instructional• Identify a program for at-risk students comprehensive  Studies of Commons use (and non-use) distribution strategy  In-library survey  In-class survey
    35. 35. THANK YOU!
    36. 36. Assessing the Value of Ebooks toAcademic Libraries and Users
    37. 37. Assessing the Value of Ebooks to Academic Libraries and Users• –
    38. 38. Assessing the Value of Ebooks to Academic Libraries and Users• – –
    39. 39. Assessing the Value of Ebooks to Academic Libraries and Users• – –•
    40. 40. Assessing the Value of Ebooks to Academic Libraries and Users• – –• –
    41. 41. What is Value? (from–––
    42. 42. What is Value? (from–––
    43. 43. What is Value? (from–––
    44. 44. What is Value? (from–––
    45. 45. Data Sets Employed to Answer Research Questions•••• – – –
    46. 46. Overall Ebook Growth at UIUC 2008-2011Fiscal #Ebooks Added Per Cumulative Ebook Percent Increase Per Year Year* Total Year2007 292,002 NA2008 27,531 345,186 9%2009 66,178 411,364 19%2010 73,404 484,768 18%2011 129,435 614,203 27% *Counts are per volume, not per title
    47. 47. UIUC Cost Per Ebook 2008-2011Fiscal Year $ Spent #New Ebooks $ per Ebook 2008 $224,047 27,531 $8.13 2009 $204,678 66,178 $3.09 2010 $383,167 73,404 $5.22 2011 $732,725 129,435 $5.66
    48. 48. Definition of an Ebook “Use”••••
    49. 49. Cost and Use Data for UIUC Ebooks 2008-2011 #Ebooks Added #Ebooks Amount Avg. $ per new Cost PerFiscal Year from Previous Total Uses (Cumulative) Spent Ebook Use Year 2007 292,002 $185,991 2008 345,186 $224,047 27,531 $8.14 151,089 $1.48 2009 411,364 $204,678 66,178 $3.09 251,273 $0.81 2010 484,768 $383,167 73,404 $5.22 563,871 $0.68 2011 614,203 $732,725 129,435 $5.66 709,944 $1.05 * Use data available from 82% of ebook publishers
    50. 50. Top 10 FY2011 Ebook Publishers byNumber of Volumes Available at UIUC Publisher #Ebooks in FY2011 FY2011 Uses Gale (includes Eighteenth CenturyCollections Online, Making of American Law, Making of Modern Mind) 332,609 69,769 Early English Books Online 106,853 94 Archives of Americana 66,892 Unknown Springer 45,924 206,740 EBSCO 11,936 Unknown Wiley 10,448 88,875 CRCNetBase 6,857 6,768Organisation for Economic Co-operation and Development 6,615 545 Netlibrary 6,182 1,234 American Council of Learned Societies 3,368 33,123 Total 597,684 (97%) 407,148 (57%)
    51. 51. UIUC Number of Ebook Uses/Year for Four Publishers These four publishers’ total downloads represent 49% of total uses for ebooks at UIUC in 2011and 11% of total ebook holdings
    52. 52. UIUC Ebooks Used and Unused Wiley, Elsevier, RSC and Springer70,000 36.9% Used 34.0% Used 41,54360,000 31.1% Used 38,57850,000 34,465 20.2% Used40,000 Ebooks Unused 29,26830,000 Ebooks Used20,000 24,260 19,857 15,56410,000 7,405 0 2008 2009 2010 2011
    53. 53. Use Frequency: Average uses per ebook
    54. 54. RSC Ebooks at UIUC by Use Frequency 500 450 400 350 300 1 Use 250 2-20 Uses 200 20-100 Uses 101+ Uses 150 100 50 0 2009 2010 2011 149 Uses in 2009 of “Molecular Biology and Biotechnology” 130 Uses in 2010 of “Handbook of Surface Plasmon Resonance” 317 Uses in 2011 of “Food Flavors and Chemistry”
    55. 55. Comparison of per-object cost of print versus electronic storage (relative to print cost). From: Courant and Nielsen, 2010, “On the Cost of Keeping a Book.”
    56. 56. From the perspective of library value, ebooks: • Have a low cost-per-ebook purchase • Have a low cost-per-use • Are more cost effective to lend, store and preserve than print • Offer greater accessibility to users (24/7 anywhere) • Offer greater availability to users (higher uses per ebook than print) • Can provide broader collection variety due to low cost, package purchases and lower facilities and staffing costs; also • Often no need to purchase multiple copies • But….
    57. 57. From the perspective of library value, ebooks: • Have a low cost-per-ebook purchase • Have a low cost-per-use • Are more cost effective to lend, store and preserve than print • Offer greater accessibility to users (24/7 anywhere) • Offer greater availability to users (higher uses per ebook than print) • Can provide broader collection variety due to low cost, package purchases and lower facilities and staffing costs; also • Often no need to purchase multiple copies • But…. What do Users Think?
    58. 58. The Value of Ebooks to Users••• – – – –
    59. 59. Survey Methodology••••••
    60. 60. The Value of Ebooks to Users In which field are you working? N=129 Social Sciences, 3 Engineering and Computer Science, 22 Humanities, 4 Interdisciplinary, 1 Life Sciences, 21 Engineering and Computer SciencePhysical Sciences, 77 Humanities Interdisciplinary Life Sciences Not applicable Physical Sciences Social Sciences Not applicable, 1
    61. 61. The Value of Ebooks to Users My position is best described as: Professor/ Researcher/Other 15 (12%) PhD student 114 (88%)
    62. 62. The Value of Ebooks to UsersWhat is your preferred form at this moment for a scholarly book? n = 114 (PhD students) do not know / no opinion / does not apply, 1 print, 45 electronic, 45 no preferred form, 23
    63. 63. The Value of Ebooks to UsersWhat is your preferred form at this moment for a scholarly book? n = 15 (Professor/Researcher/Other ) print, 6 electronic, 7 no preferred form, 2
    64. 64. The Value of Ebooks to Users
    65. 65. The Value of Ebooks to UsersMy usage of (printed or electronic) books for research purposes is characterized by: 100% 90% 80% 70% 60% 50% Humanities and Social Sciences 40% Life Sciences 30% 20% Physical Sciences, Engineering and Computer Science 10% 0% I use bits of I read one or I read most or all information two chapters chapters from a from a book from a book book
    66. 66. The Value of Ebooks to Users What are the main advantages of E-books from your perspective? [PLEASE TICK A MAXIMUM OF 3 BOXES] # Responses Percent24 hours/7 days per week access 82 63.6%online access 79 61.2%easy to search and navigate 52 40.3%downloading to laptop 39 30.2%easy storage 36 27.9%off campus access 33 25.6%copying and pasting 16 12.4%downloading to e-reader 9 7.0%easy to share with colleagues 8 6.2%easy to use in an electronic learning environment 6 4.7%easy to use multiple documents at once 5 3.9%use of multimedia in the E-book 4 3.1%
    67. 67. The Value of Ebooks to UsersMy online behavior includes the following characteristics:
    68. 68. How do you value the information from this Elsevier E-book? Based on over 800 ebook uses
    69. 69. How do you value the information from this Elsevier E-book? Based on over 800 ebook uses
    70. 70. Comment Box Responses Following Value Questions Comment Description Not Clear Negative PositiveLack of relevant material 2 89 0Did not have access to material 10 57 5Did no better providing information than other resources(Google, journal articles) 5 27 0Ebook was convenient and easy to access 2 0 6Not clear 55 0 1Obtained relevant/useful information 2 0 79Liked ability to search within the ebooks 0 0 2Would serve as a nice additional resource 0 1 13Liked the ebook because it gave background information 4 1 39Liked the ebook because it gave good detail 0 0 1Search is good 0 0 9Search results similar to other sources 1 1 0Did not like search 0 6 0Contained current information 0 0 1E-book was not up-to-date enough 0 8 0Information in E-book was too general 0 3 0Liked platorm search 0 0 2E-book content was too specific 0 3 0Totals 81 196 158
    71. 71. From the perspective of the user, ebooks: • Offer impressive 24/7 accessibility from anywhere • Are found using multiple search engines • Are “Nice to Have” (12.6%) or “Need to Have” (54.8%) • Are not likely to be shared with colleagues, printed out, or be used in “cut and paste” • Are most likely to be read from the screen or briefly reviewed • Need to have the ability to be downloaded in some format • Are often difficult to access, despite strong interest in the title.
    72. 72. Conclusions• Ebooks offer value to the library in both a monetary way and through documented “usefulness” to patrons (although YMMD).• As noted by Courant and Neilsen, ebooks are less expensive to own, circulate, maintain and preserve than print books.• Ebooks offer value to patrons, who “esteem” them due to – Accessibility and availability (24/7, literally anywhere in the world) – Portability – Search and navigation capabilities• There is still plenty of room for improvement!
    73. 73. Conclusions• Despite documented value to both users and libraries, some ebook issues remain to be solved, including: – Non-standard downloading policies – Non-standard cut and paste capabilities – Confusion about what ebooks are locally available – Inadequate and non-standard discovery tools – Digital rights management – Perpetual access “guarantees” – Sparse content in some disciplines – However… all predictions are for these issues, and others that have yet to arise, to be solved within the next 5-10 years.
    74. 74. Thank you! Special thanks to UIUC Graduate Assistants Dan Tracy and William Weathers and to Wendy Shelburne, Michael Norman and Elsevier. This research is a part ofValues, Outcomes, and Return on Investment of Academic Libraries ("Lib-Value"), a three year study funded by the Institute of Museum and Library Services IMLS grant # LG-06-09-0152-09.