A workbook on How to Identify Your Impact: The Value of Libraries
1A Workbook forA Half-Day WorkshopHow To Identify Your Impact!Association of Christian LibrariansJoe MatthewsJune 10, 2013
2Table of ContentsPagePerformance Measures 1The How Questions 2Lack of a Connection 3Criteria for Judging Value 4Tools for Measuring Value 6Value 7Value of Information 10Astin’s IEO Model 11Perspectives on Value 14Value of a Library 15Personal Value 15Organizational Value 17Academic Library Organizational Value 17Student Enrollment 20Student Learning 20Student Retention 25Student Engagement 27Student Career Success 28Faculty 29Institutional Ranking 30Financial ValueWhat is ROI? 31Cost Benefit Analysis 31ROI in Academic Libraries 32Communicating Value 36
3Working Together1Performance MeasuresRichard Orr in 19732Input measures – budget, staff, space, collections,Output measures – actual use of library services and collections (counts)Process measures – time or cost or quality to perform a task or activityOutcome measures –change in attitude, behavior, knowledge, skill, status orcondition. Outcomes occur first in people and then in organizations (or society)Efficiency – are we doings things right?Effectiveness – are we doing the right things?Outcomes accrue first to the individual and then to the organization and/or societyat large.Performance MeasuresInput Process Output Outcomes OutcomesLibrary Services Individual SocietyEfficiency Effec venessCost Effec veness ImpactVALUECost benefit
4The How QuestionsLibrary controlLibrary & Customers decideCustomers decide“If you live by the numbers, you die by the numbers.”3Outcome tutorial, seehttp://www.shapingoutcomes.orgChallenges with Performance Measures1. Lack of consensus about what should be measured and how2. Lack of understanding of performance measurement and metrics3. Organizational structural issues4. Lack of precision in measuring performance, and5. Alignment issues6. Determining the “bottom line” is too far away7. Majority of stakeholders are too far away8. Library staff find it difficult to see the “big” pictureAnd the survey said? Two-thirds of managers who are responsible for approvinglibrary budgets – no idea of value of the library4
5Lack of a Connection• Budget and outputs (and outcomes) are separated• No “bottom line” measure for libraries• Decision-making process is bigger than the library• Library has neither champions nor foes• Library benefits are not widely self-evidentMooers’ Law – “An information retrieval systems will tend not to be used wheneverit is more painful and troublesome for a customer to have information than for himnot to have it.”5S. R. Ranganathan’s Fourth Law of Library Science - “Save the time of the reader.”6
6Criteria for judging value of an information service7Customer Criterion Value Added by the ServiceEase of use Browsing, formatting, mediation service,orientation service, ordering, physicalaccessibilityNoise reduction Access (item identification, subjectdescription, subject summary), linkage,precision, selectivityQuality Accuracy, comprehensiveness, currency,reliability, validityAdaptability Closeness to problem, flexibility,simplicity, stimulatoryTime savings Response speedCost savings Cost savings
7Calls for accountability and transparencyKey Question – It is not how much an information resource and/or service is used,but rather what is the impact or benefit of the information servicein the life of the library customer.Key Insight - Value is determined from the perspective of the user.Carol Tenopir and Don King81. Implicit measures that imply value, but do not directly measure value2. Explicit measures that directly describe purchase or use values.The nature of information is changingOrr’s Fundamental Questions How What How
8Tools for Assessment - EvaluationLevels of Assessment• Individual student• Course• Departmental/Program• College or UniversityTypes of Measures• Direct– Provide tangible, visible and self-explanatory evidence of whatstudents have & have not learned• Indirect– Capture students’ perceptions of their knowledge & skills; supplementdirect measures; sometimes called surrogatesQualitative Tools• Provides in-depth understanding of user responses and interactions• Represents part of a long-term strategy of formative evaluativeQuantitative Tools• Surveys• Transaction logs• Statistics from systems• Observations (count)Triangulation is importantCorrelation does not equal causation - careful
9Definitions of ValueIts own philosophical discipline – axiology or Value TheoryA nouno Exchange for or equivalenceo Monetary or material wortho Usefulness, utilityo Principle, standard, or qualityo Toll, cost or priceo Darkness or lightness of colorA verbo Estimate the worth of something (appraise)o Regard highly (esteem)o Assign a value to somethingOther definitions depending on the fieldQualify other terms
10Adam SmithValue-in-exchangeThe price paid is the accepted indicator of valueValue-in-use or “utility theory”Benefits to the user define the value (of information)o Normative value – models to assess risk in decision makingo Realistic value – before and after consequences of information on theperformance of decision makerso Perceived value – Users can recognize (and articulate) the direct andintangible values of informationIndividuals determine or attribute valueDefinitions of Information1. Information as subjective knowledge2. Information as useful data3. Information as a resource4. Information as a commodity5. Information as a constitutive force in society9Information may, or may not, reduce uncertainty10
11Quality of informationThis fast food approach to information consumption drives librarians crazy.“Our information is healthier and tastes better too” they shout. But nobodylistens. We’re too busy Googling.”~ Peter MorvilleConvenience trumps everything!Herb Simon Satisficing11 Good enoughDifferent conceptions of informationEpistemic information – within the context of human knowledge andunderstandingSystemic information–information as a part of transmission –Shannon-Weaver model of communicationLibrary services are:Nonexcludible - use by one individual does not reduce their availability (andpotential value) to anotherNonrival – individuals are not excluded form using the libraryKey characteristics of information12Uncertainty KnowledgeAmbiguity IndeterminacyRedundancy System dependencySharing TimelinessCompression PresentationStability Multiple life cyclesLeakability Substitutability
12Value of InformationInformation needs an expected value-in-use to arouse the interest of the user.Information in a library’s collection represents a “potential value” until used.The collection also represents a “future value” since it will be available for futuregenerations of students, faculty, and researchers.The value of the local collection is, however, declining each year (dramatically).Collections are being disrupted as we move from atoms (objects) to bits.We are not talking about insurance value or replacement value.Historically, information has been embedded in physical modes of delivery.13ReachRichnesso Bandwidtho Degree of Customizationo Amount of InteractivityThe Internet changes everything!
13Astin’s Input-Environment-Output Model14Meta-analysis of 109 higher education studies showed that for:15Entering student characteristics–Socioeconomic status (SES), high school GPA, andACT/SAT are the best predictors of student successPsychosocial and study skill factors – Academic goals, academic self-efficacy, andacademic-related skills are the best predictors of college retention. In addition,social support and social engagement are good predictors of college retention.Financial support and institutional selectivity are correlated with retention.Achievement motivation is the strongest predictor for GPA.EnteringStudentCharacteris csGradua ngStudentCharacteris csEnvironmentProgramsIns tu onalCharacteris csFellow StudentsPlace of ResidenceFacultyLibrary Services
14Framework for Student Learning Outcomes16IntelligenceGeneral Fluid CrystallizedGeneral ReasoningVerbal Quantitative SpatialExample: Graduate Record ExaminationBroad AbilitiesReasoning Critical Thinking Problem SolvingDecision Making CommunicatingIn Broad DomainsDisciplines - Humanities, Social Services, SciencesAnd Responsibility – Personal, Social, Moral, and CivicExample: Collegiate Learning AssessmentKnowledge, Understanding, and ReasoningIn Major Fields and Professions (Business, Law, Medicine)Example: ETS’s Major Field TestsAbstract,ProcessOrientedConcrete,Content-OrientedInheritance xAccumulatedExperienceDirectExperience
15Define, develop, and measure outcomesthat contribute toinstitutional effectivenessACRL Standards for Libraries in Higher EducationPerspectives on ValueValueOrganizationalPersonalFinancial
16Value of a LibraryPersonal PerspectiveTefko Saracevic and Paul Kantor17Impact categories1. Cognitive results. Use of the library may have an impact in the mind of theuser. “What was learned?”2. Affective results. Use of the library may have an emotional impact on theuser.3. Meeting expectations. Users may be getting what they needed, sought, orexpected; be getting too much; be getting nothing4. Accomplishments in relation to tasks5. Time aspects. Information provided by a library may lead to saving time6. Money aspects. Using the library may result in saving money or generatingnew revenues.
17PersonalGates Foundation Generic Learning OutcomesKnowledge & UnderstandingKnowing what or aboutsomethingLearning facts or informationMaking sense of somethingDeepening understandingMaking links & relationshipsbetween thingsSkillsKnowing how to do somethingBeing able to do new thingsIntellectual skillsInformation management skillsSocial skillsCommunication skillsPhysical skillsAttitudes & ValuesFeelingsPerceptionsSelf-esteemAttitudes towards othersIncreased capacity for toleranceEmpathyIncreased motivationAttitudes towards anorganizationAttitudes related to anexperienceEnjoyment, Inspiration, CreativityHaving funBeing surprisedInnovative thoughtsCreativityExploration, experimentation andmakingBeing inspiredActivity, Behavior, ProgressionWhat people doWhat people intend to doWhat people have doneReported or observed actionsA change in the way peoplemanage their lives
18Organizational PerspectiveStudent Learning is Affected by … (NSSE)• Full-time students• Live on campus• Interact more with faculty• Study more• Collaborate with their peersIs the Academic Library Used?• 50% never used the library 18• Use of libraries at small, academically challenging liberal artscolleges are correlated with other purposeful activities• Library use less intensive at larger universities• Students who work harder use library resourcesCaution – Halo error19Academically Adrift20• Gains in student performance are quite low• Individual learning is characterized by persistence• Notable variation within and across institutionsWabash National Study21Bibliographic InstructionInformation Literacy
21Student Enrollment23Student LearningMeta-analysisStudent Learning occurs …Direct measuresCapstone experienceUse of a portfolioA standardized exam (e.g., the Collegiate Learning Assessment).Indirect measuresGrade point averageStudent retention ratesCollegiate experience surveys - NSSESuccess in graduate school examsGraduate student publicationsFellowshipsPost-doctoratesTime to first jobSalary of first jobAnd so forth
22In England, the Library Impact Data Project248 Universities analyzing data from the last 6 yearsVisit library buildingsBorrow materialsDownload eResourcesResults of the Library Impact Data Project showA correlation between borrowing materials and downloading eResourcesand a student’s grade point average.About half of all undergraduate students did not use ANY library serviceLargest group of library non-users are part-time and distance studentsSome library non-users achieve high GPAsMajority of library non-users did poorly – low GPAsRemember: Correlation does not = CausalitySee http://library.hud.ac.uk/blogs/projects/lidp/See also, the JISC EBEAM Project – Evaluating the Benefits of Electronic AssessmentManagement http://library.hud.ac.uk/blogs/projects/ebeam/See also, the JISC Copac Activity Data Project – Sharing and reusing HE librarycirculation activity data http://copac.ac.uk/innovations/activity-data/?tag=copacad
24Grade Point AverageIn Australia, the University of Wollongong25Foreign studentsIn Hong Kong, the Hong Kong Baptist University library26Weak support for library instruction and GPAHope CollegeImplication? Library MUST COMBINE its data with other data from the University inorder to prepare an analysis that focuses on outcomes!The library MUST develop partnerships!Note that customer satisfaction ratings are NOT an indicator of value.University of MinnesotaGym Bags and MortarboardsThe Library Study27• 5,368 first-year non-transfer students• Use of library was associated with a .23 increase in students GPA• More use of the library, GPA also goes up
25Library Instruction and GPAHong King Baptist University28• 45 sample groups – N=31 to 1,223, study majors• Pairs of data• One-fourth (11) had a positive relationship• Results:– 1 or 2 workshops – little impact on GPA– 3 or 4 workshops – ½ show a positive impact– 5 workshops (1 sample group) – 100% had a higher GPAUniversity of Wyoming Libraries29• Analysis of 4,489 transcripts• Slight positive relationship between upper-level library instruction coursesand GPA –0.075 GPA difference – that’s less than 1/10th of 1 percent
26Student Retention/GraduationWhy important: greater revenues, which mean lower costs per degree conferredRetention Concepts• Institutional retention– Enrolling & graduating from the same institution• Program retention– Enrolling & graduating with the same major/department/school• System retention– Students who leave one university yet continue and complete post-secondary studies elsewhere• Institutional retention– Enrolling & graduating from the same institution• Program retention– Enrolling & graduating with the same major/department/school• System retention– Students who leave one university yet continue and complete post-secondary studies elsewhere• Persistence– From first to second year? Entry to graduation?• Completion– From entry to graduation (Student goals?)• Graduation rates– Are transfers included? Time period?• Attrition– Leaving university? Leaving higher ed?
27• Stopout– Leave university with the intention (and action) of returning later tocomplete a program• Dropout– Leave university with intention (and action) of NOT returning• Transfer– Change institutions yet persist in higher education– May change type of institution– Voluntary vs. involuntary attrition? Tinto’s “Model of Student Integration” Bean’s “Model of Student Attrition” Carroll et al Graduate Distant Education Student Model of Retention30Retention strategies focus on people – not physical resourcesCurricular and behavioral integrationFrequent contact with faculty (and other people on campus)Accessible and responsive staffConvenient and responsive librariesHigh impact educational experiencesCalls for librarians to increase contact with studentsStudy by Hamrick, Schuh & Shelley - increased library funding led to increasedgraduation rates31Early studies – weak support for use of library and retentionLibrary orientations have weak support for student retentionLibraries that spend more on materials and on staff have greater retention ratesTotal library expenditures may be related to higher graduation rates
28Student Engagement/ExperiencesSurveysNational Survey of Student Engagement (NSSE)Community College Survey of Student Engagement (CCSSE)Beginning College Survey of Student Engagement (BCSSE)Faculty Survey of Student Engagement (FSSE)Student self-report their experiences and over-report their experiencesLibraries can add questions to national surveysAnalysis of NSSE data from 1984 to 2002 (N= 380,000 respondents)32More than 50% of the respondents never visited the library or use a libraryservice during their undergraduate careerAcademic libraries and their services at small, academically challengingliberal arts colleges are strongly correlated with other educationallypurposeful activities (note that such institutions are usually residential innature, the library is closely located to student residences making accesseasier)Library use is less frequent in larger doctoral/research-intensive – perhapsdue to the readily availability of other alternativesIndividual students who frequently use library resources are more likely towork harder to meet faculty academic expectationsLibrary experiences do not lead to gains in information literacyLibrary experiences do not lead to gains in student satisfactionLibrary experiences do not lead to what students gain overall from college.Goal: Gain insight into the relationship between engagement, library outcomes, andstudent success.
29Student Career Success• Job placement rates• First-year job salaries• Professional/graduate school acceptance• Internship success• Marketable skills
30Faculty TeachingPartner with faculty to integrate information literacy in their classesDevelop and implement new and/or improve curriculaImprove faculty research productivitySave time in preparing for classes (Simmel 2007, Dickenson 2006)Faculty ResearchPrincipal contributions were collections but that is shifting as faculty increasinglyrelies on online resources (Ithaka’s studies).Weak support for: size of collections, reference queries are related to facultyproductivity. Personal characteristics of the researcher are more important thaninstitutional characteristics.Faculty productivity and award recognition related to the amount of time reading(Tenopir and King). Library collections (physical and virtual) provide convenienceand ease of access, which saves the time of the faculty member/researcher.• Library is the source for most journal articles (individual subscriptions areway down)• If library subscriptions were unavailable – productivity would decrease 17%• Library is not the source of book readings• 42% of reading material is library providedAltmetrics – Manifesto Jason Priem – Univ of North Carolina – Chapel Hillhttp://altmetrics.org/manifesto/
31Institutional RankingInstitutional rank is not related to quality of education.For example, no link between NSSE and US News and World Report rankingsHard to isolate the impact of library services from other institutional activities whenanalyzing institutional rankings. Universities that spend more per student arebetter in many areas and no one area can take “credit.”In England, the Research Assessment Exercise (RAE) – the best institutions haveboth the best RAE ratings and the best libraries.Large library collections contribute (20 to 40%) to the prestige of the university(Liu 2003, 2009).Group of EightLibrary provides access to information resources that are:• Indispensable for their research• Maintain a high-level overview of their field• Value for money is good• Library not available costs would increase 40%• Take 31% longer to locate same information
32Financial ValueThree possible economic measuresEconomic ValueEconomic ActivityEconomic Benefits1. Direct Use Benefits –a tangible benefit.2. Indirect Use Benefits or economic impact3. Non-use Benefits.What is ROI?ROI is a tool to help with decision-making in either planning or evaluation ofservices.ROI ApplicationsProjectsServicesOrganizational – Library Valuation studiesCost-Benefit Analysis (CBA)• Maximize the benefits for given costs• Minimize the costs for a given level of benefits• Maximize the ratio of benefits over costs• Maximize the net benefits (present value of benefits minus the present valueof costs)• Maximize the internal rate of returnFor libraries, CBA is the value of benefits divided by the costs.Challenge: Estimating the value of the benefits
33ROI in Academic LibrariesDrexel University – Don King and Carol Montgomery33University of Pittsburgh ROILibrary journal collections (physical and electronic) had an ROI of 2.9:1 (King et al2004)A similar study reported an ROI of 5.35:134The Portuguese electronic scientific information consortium B-on (Willingness topay – contingent Valuation)35Another study indicated that students were willing to pay $5.59 per semester tomaintain the current hours of the reference desk.36University of Illinois at Urbana-Champaign37Connected citations to resources in the library’s collection to successful grantproposals and the resulting grant income. Results: 4.38:1 ROIProcess:Survey of faculty membersData collection (covering a ten year period)Interviews with senior faculty and administratorsIn simple terms the formula isThe actual formula
348 other universities located in different countries38Result: ROI ranged from 0.27:1 to 15.54:1Variability in grant funding, characteristics of the university, country of studyAnalysis based on the assumption that the best proposals (those with citations) arewinning the competitive grant application process.James Neal, University Librarian at Columbia University, suggested that ROI inacademic libraries are a “miscalculated, defensive and risky strategy.”39
35• Comprehensive assessment of the library• ROI of the journal collection & readership• ROI for support of teaching & learning• ROI of digitized special collections• ROI of eBooks• Value of library commons• BibliographyProject Web site - http://libvalue.cci.utk.edu/Comprehensive Assessment of the Library – Bryant University40Value of articles provided by the libraryo Use value or the favorable consequences from readingo Purchase value or what readers pay for the information in their timeor money to process and read articlesValue of books, other materials, reference, instruction, photocopiers, AVequipment, etc.Syracuse University ROI = 4.49:1 Bruce KingmaROI of the Journal Collection & Readership - Tenopir41421. Scholarly reading is essential to academic work2. ECollections are making a difference3. Library plays an essential role in academic work and success4. Booking readings is different from article reading5. Successful academic read more
36ROI for Support of Teaching & LearningPerceived Benefits –43• Savings …– Of own time 65% of faculty – Yes saves time– Of own money 63% of faculty – Yes saves $– Of other resources – printing, copier• Improvements …– Teaching– Course-related materials– Student performanceROI of Digitized Special Collections44Value of Library Commons
37Communicating ValueResonateValue PropositionAlignmentBe intentional – planPartner, collaborateROI is just one piece of the value puzzleNumbers and storiesWritten and spoken, elevator speeches
38The Value of the Library Collection – InsuranceGreat resource -http://www.loc.gov/preservation/emergprep/insurancevaluation.html
401 Claire Creaser and Valerie Spezi. Working Together: Evolving Value for AcademicLibraries. UK: Loughborough University, June 2012.2 Richard Orr. Mesuring the Goodness of Library Services. Journal ofDocumentation, 29 (3), 1973, 315-52.3 Allan Pratt and Ellen Altman. Live by the Numbers, Die by the Numbers. LibraryJournal, 122 (7), April 15, 1997, 48-49.4 James Matrazzo and Lawrence Prusak. Valuing Corporate Libraries. SpecialLibraries, 81 (2), 1990, 102-110.5 Calvin Mooers. Mooers’ Law or, Why Some Retrieval Systems Are Used and OthersAre Not. American Documentation, 11, 1960, 201-209.6 S. R. Ranganathan. The Five Laws of Library Science. New Delhi, India: Ess EssPublications, 1931.7 Robert Taylor. Value-added Processes in Information Systems. Norwood, NJ: AblexPublishing, 1986.8 Allan Pratt and Ellen Altman. Live by the Numbers, Die by the Numbers. LibraryJournal, 122 (7), April 15, 1997, 48-49.9 Jennifer Rowley. Promotion and Marketing Communications in the InformationMarketplace. Library Review, 47 (8), 1998, 383-388.10 Michael Buckland. Information as Thing. Journal of the American Society forInformation Science, 42 (5), 1991, 351-360.11 Herbert Simon. Models of Man. New York: Wiley, 1957.12 Douglas Badenoch, Christine Reid, Paul Burton, Forbes Gibb, and CharlesOppenheim. The Value of Information, in Mary Feeney and Maureen Grieves (Eds.).The Value and Impact of Information. London: Bowker Saur, 1994, 9-78.13 Philip Evans and Thomas Wurster. Strategy and the New Economics ofInformation. Harvard Business Review, September-October 1997, 71-82.14 Alexander Astin. Achieving Education Excellence: A Critical Assessment ofPriorities and Practices in Higher Education. San Francisco, CA: Jossey-Bass, 1985.15 Steven Robbins et al. Do Psychosocial and Study Skill Factors Predict CollegeOutcomes?: A Meta-Analysis. Psychological Bulletin, 130 (2), March 2004, 261-288.
4116 Richard Shavelson. Measuring College Learning Responsibility: Accountability in aNew Era. Stanford, CA: Stanford University Press, 2010, 13.17 Tefko Saracevic and Paul Kantor. Studying the Value of Library and InformationServices. Part I. Establishing a Theoretical Framework. The Journal of the AmericanSociety for Information Science, 48 (6), June 1997a, 527-542.Tefko Saracevic and Paul Kantor. Studying the Value of Library and InformationServices. Part II. Methodology and Taxonomy. The Journal of the American Societyfor Information Science, 48 (6), June 1997b, 543-563.18George Kuh and Robert Gonyea. The Role of the Academic Library in PromotingStudent Engagement in Leaning. College & Research Libraries, July 2003, 256-282.19 Gary Pike. The Constant Error of the Halo in Educational Outcomes Research.Research in Higher Education, 40 (1), 1999, 61-86.20Richard Arum and Josipa Roksa. Academically Adrift: Limited Learning on CollegeCampuses. Chicago: University of Chicago Press, 2011.21Pascarella et al. How Robust Are the Findings of Academically Adrift? Change,May-June 2011.22 Megan Oakleaf. The Value of Academic Libraries: A Comprehensive ResearchReview and Report. Chicago: Association of College & Research Libraries, 2010, p.19.Available at http://www.acrl.ala.org/value/23Gary Reynolds. The Impact of Facilities on Recruitment & Retention of Students.New Directions for Institutional Research, 135, Fall 2007.24 Deborah Goodall and David Pattern. Academic Library Non/Low use andUndergraduate Achievement: A Preliminary Report of Research in Progress. LibraryManagement, 32 (3), 2011, 159-170.See also, Graham Stone, David Pattern and Bryony Ramsden. Looking for the LinkBetween Library Usage and Student Attainment. Aridne, 67, July 2011.25 Margie Jantti and Brian Cox. Capturing Business Intelligence Required forTargeted Marketing, Demonstrating Value, and Driving Process Innovation, in 9thNorthumbria International Conference on Performance Measurement in Libraries andInformation Services: Proving Value in Challenging Times, 22-25 August 2011,University of York, in press.
4226 Shun Han Rebekah Wong and T.D. Webb. Uncovering Meaningful CorrelationBetween Student Academic Performance and Library Material Usage. College &Research Libraries, 74 (4), July 2011, 361-70.Shun Han Rebekah Wong and Dianne Cmor. Measuring Association BetweenLibrary Instruction and Graduation GPA. College & Research Libraries, 74 (5),September 2011, 464-473.27 Jan Fransen et al. Library Data and Student Success. A Presentation at the LibraryTechnology Conference, Macalester College, Minnesota March 14-15, 2012.Available athttp://digitalcommons.macalester.edu/libtech_conf/2012/sessions/28/28Wong and Cmor CR&L Sept 201129Melissa Bowles-Terry. Library Instruction and Academic Success: A Mixed-Methods Assessment of a Library Instruction Program. Evidence Based Library andInformation Practice, 7(1), 2012.30 David Carroll, Eric Ng, and Dawn Birch. Retention and Progression ofPostgraduate Business Students: an Australian Perspective. Open Learning, 24 (3),November 2009, 197-209.31 Florence Hamrick, John Schuh & Mark Shelley. Predicting Higher EducationGraduation Rates from Institutional Characteristics and Resource Allocation.Education Policy Analysis Archives, 19 (12), May 2004, 1-23.32 George Kuh and Robert Gonyea. The Role of the Academic Library in PromotingStudent Engagement in Leaning. College & Research Libraries, July 2003, 256-282.33 King, D.W., Tenopir, C., Montgomery, C.H., and Aerni, S.E. Patterns of Journal Useby Faculty at Three Diverse Universities. D-Lib Magazine, 9:10, October 2003.Available at http://www.dlib.org/dlib/october03/king/10king.html34 Luiza Melo and Cesaltina Pires. Measuring the Economic Value of the ElectronicScientific Information Services in Portuguese Academic Libraries. Journal ofLibrarianship and Information Science, 43 (3), 2011, 146-156.35Melo, Luiza Baptista and Pires, Cesaltina Electronic academic libraries servicesvaluation: a case study of the Portuguese electronic scientific information consortiumb-on., 2010 . In 2nd QQML - International Conference on Qualitative andQuantitative Methods in Libraries,, Chania, Crete, Greece, 25-28 May 2010.[Conference Paper]36 Harless, D.W. and Allen, F.R. Using the Contingent Valuation Method to MeasurePatron Benefits of Reference Desk Service in an Academic Library, College &
43Research Libraries, 60 (1), 1999, 56–69.37 Judy Luther. University Investment in the Library: What’s the Return? A Case Studyat the University of Illinois at Urbana-Champaign. San Diego: Elsevier, 2008.38 Carol Tenopir, Amy Love, Joseph Park, Lei Wu, Bruce Kigma, and Donald King.Return on Investment in Academic Libraries: An International Study of the Value ofResearch Libraries to the Grants Process. San Diego: Elsevier, 2009.39 James Neal. Stop the Madness: The Insanity of ROI and the Need for NewQualitative Measures of Academic Library Success. ACRL Conference, March 3-April2, 2011, Philadelphia, Pennsylvania, p. 424. Available athttps://www.ala.org/ala/mgrps/divs/acrl/conferences/confsandpreconfs/national/2011/papers/stop_the_madness.pdf40 Donald King. Demonstration of Methods to Assess the Use, Value, and ROI of AllAcademic Library Services. 2012. Available at LibValue Project Web site.41Tenopir, C., Mays, R., and Wu, L. (2011). Journal Article Growth and ReadingPatterns. New Review of Information Networking, 16(1), 4-22.42 Carol Tenopir, Regina Mays and Lei Wu. Journal Article Growth and ReadingPatterns. New Review of Information Networking, 16, 2011, 4–22.43Fleming-May, R. (2011 Nov). Lib-Value: Teaching & Learning. XXXI AnnualCharleston Conference. Charleston, SC.44Wise, K. and G. Baker (2012 Jan). Assessing the Return on Investment in DigitizedSpecial Collections. ALA Midwinter. Dallas, Texas.