LIBRARYASSESSMENT Jennifer Rutner, Senior Analyst, Ithaka S+R Pratt Institute Human Information Behavior September 22, 2011
Senior Analyst, Ithaka S+R School: BA in Religious Studies, 2002 MLIS from Pratt, 2005 Enrolled in Quantitative Methods for the Social Sciences at Columbia, 2010-? Work: Assessment & Planning Librarian, Columbia University Libraries, 2006-2011 Chair, ACRL Assessment Committee, 2011
Assessment“To assess, in general, is to determine the importance, size, or value of; to evaluate. Library staff assess operations by collecting, interpreting, and using data to make decisions and improve customer service.”ARL Spec Kit #303, Library Assessment, December 2007
What we talk about when we talk about “assessment” Quality assurance User ROI/Value researchEvaluation assessment Impact
Culture of AssessmentA Culture of Assessment is an organizational environment in which decisions are based on facts, research and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for library clients.A Culture of Assessment exists in organizations where staff care to know what results they produce and how those results relate to customer expectations.Amos Lakos: www.usc.edu/.../locations/leavey/news/conference/presentations/presentations_9-16/Assessment/UCLA_Lakos.ppt
Why assess? Stuff I know Stuff I know I dont know Stuff I dont know I dont know
How is data used in libraries?• Strategic planning + management• Decision making• Program evaluation• Advocacy• Budgeting• Regular service improvements
Assessment Mission (CUL)“Serve library users and staff through the gathering, analysis, and application of high-quality, actionable information to guide library decision making.”
The Research ProcessEstablish Gather Information Environmentalresearch available Needs scanquestions information (unknowns)* Assign EstablishTest tools Develop tools methodology priorities Conduct Decision IRB Analysis assessment making
The Assessment ProcessBrainstorming Information Priority Methodology Formal Outcome Need Question “I want to know…”
SurveysUse random sampling to generalize to the broader population.Set questions, with no opportunity for follow-up.Allows for statistical analysis.
QuestionnairesEvaluation tool: How was it?Exploratory tool: What do you do?Preferences tool: What do you want?Reporting tool: What did you do?Analysis stops at descriptive statistics.
Response Rate vs. Representativeness “It ain’t response rate.” 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% % of population % of respondents 10.00% 0.00%
Focus Groups“Researchers attempt to capture peoples explanations”• Answering “how?” “why?” and “what?” questions – open ended question• Look at a topic up-close, rather than get a panoramic view of the entire issueFocus groups can uncover insights andperspectives that are not retrievableby other methods, from a small groupof participants.
Focus GroupsStandard – Uses a rigid set of questionsGuided – Uses a set of topics to explore, but the question wording is flexibleExploratory – The most informal, questions arise through the course of the conversation about the topic
EthnographyMethodologies – Work study – Photo essays – Dream-catcher workshops – Mapping diaries – Pilot projects – Interviews
ObservationWait, watch, write. Head Count+ • Sections of the library • Type of seating • Group vs. individual • Food/drink • Technology
Way Finding: 3D UsabilityHow do users navigate space? – Provide task – Record steps to completion – Record completion rate – Record challenges
Learning Outcomes• Rubrics What are we teaching?• Pre- and post-tests• Minute Papers – What’s one useful thing What are they learning? you learned today? – What’s one thing you’re still confused about? How does this impact their academic success?
Usability StudiesMethods Evaluates• Card sort • Ease of use• Heuristiv evaluation • Efficiency of use• Paper prototyping • Memorability• Personas • Error frequency and• Task analysis severity• Work-study • Subjective satisfaction www.usability.gov
ROI “For every dollar invested in the library, the college/university/school/ community/business received X dollars in return.” “For every dollar invested in the library, the library produces X services, which can be valued at Y.”
Institutional Review Board (IRB)• Protect human subjects. Protocols include:• Ensure research is • Research questions and ethical. hypothesis• Ensure research • Subject profiles complies with federal • Study procedures and state laws. • Recruitment materials • Report on findings
LibQual+ Survey 2009www.libqual.org“22 questions and a box” Affect of Service Information Control Library as Place Comments?
LibQual+ ItemsAffect of Service Information ControlAS-1 Employees who instill confidence in users IC-1 Making electronic resources accessible from myAS-2 Giving users individual attention home or officeAS-3 Employees who are consistently courteous IC-2 A library Web site enabling me to locate informationAS-4 Readiness to respond to users questions on my ownAS-5 Employees who have the knowledge to answer user IC-3 The printed library materials I need for my work questions IC-4 The electronic information resources I needAS-6 Employees who deal with users in a caring fashion IC-5 Modern equipment that lets me easily accessAS-7 Employees who understand the needs of their users needed informationAS-8 Willingness to help users IC-6 Easy-to-use access tools that allow me to find things on my ownAS-9 Dependability in handling users service problems IC-7 Making information easily accessible for independent useLibrary as Place IC-8 Print and/or electronic journal collections I requireLP-1 Library space that inspires study and learning for my workLP-2 Quiet space for individual activitiesLP-3 A comfortable and inviting location Local QuestionsLP-4 A getaway for study, learning, or research Providing help when and where I need itLP-5 Community space for group learning and group Making me aware of library services study Availability of subject assistance Ability to navigate library Web pages Access to archives, special collections
Response: Representativeness Response by status across Status % of % of the University matches the responses population population distribution Undergraduates 40.03% 32.38% very closely. Graduates 53.21% 55.21% Faculty 6.78% 12.40% Greatest difference: 8% This is representative data!http://www.libqual.org/documents/admin/Representativeness.pdf
Response: Representativeness Response by discipline across the University matches the population distribution nearly perfectly. E.g. We’re not missing anyone!http://www.libqual.org/documents/admin/Representativeness.pdf
Reading LibQual+ Charts DesiredSuperiority Gap Perceived/Reality Zone of Tolerance Adequacy Gap Minimum
Is there a difference in scores from year to year? (ANOVA)• 2006-2009 adequacy gaps from each ARL institution.• P-value = 0.119, which is not deemed statistically significant.Faculty were no more or less dissatisfied with journal collections in 2009.
Journals and Overall Satisfaction Figure 8: LibQUAL+ 2009, Correlation of Faculty Satisfaction with Journal Collections (IC-8) and Overall Library Service, 21 Libraries 9 r =0.71 8 7 6 5-1.2 -1 -0.8 -0.6 -0.4 -0.2 0
What else should we be watching? Figure 6: LibQUAL+ 2006-09, Information Control Adequacy Gaps Over Time0.20.1 0 IC1 IC2 IC3 IC4 IC5 IC6 IC7 IC8 2006 (n=37 )-0.1 2007 (n=19)-0.2 2008 (n=14) 2009 (n=21)-0.3-0.4-0.5-0.6-0.7
What do our faculty say?Support“What would be great for faculty would be if when things are not available, there was one source in the library, extraordinarily skilled at tracking down items. […] These people would be specialists in working the electronic and journal capabilities.”
What do our faculty say?Search and Online Access“I think just having free text search, like Google book search, would be something that would be very, very useful to have. I still feel like we are living 20 years behind where the rest of the world is in terms of being able to search these databases and large collections of books that we have.”
What do our faculty say?Work-Arounds“I just buy them individually from my research funds, so it’s coming out of my research money.”
What do our faculty say?Quick List“If I was to give a suggestion, maybe to have discipline-specific pointers that could help each discipline find things. […] It’s more of an interface issue than a collections issue.”
What do our faculty say?Resources“The size of the collection is not as important as getting the current collection working as smooth as possible. Before, when we used to go to the library, we got service.”
What do our faculty say?Print vs. Electronic“A few years ago, I wouldn’t have said that. But, I guess things have changed.”
The 2CUL Project“Columbia and Cornell University Libraries are pleased to join forces in a transformative and enduring partnership between our two great library systems that enables us to pool resources to provide content, expertise, and services that are impossible to accomplish acting alone.” http://2cul.org/
Research ProceduresSpring 2010• Ethnographic Training• Focus groups (5 total)Summer + Fall 2010• Interviews (45 total) – 90 minutes, individual• Post-questionnaire (paper)Winter 2010-11 $• Analysis and reporting
Code Tree Code Level The Research + The FirstThe Student Institution + The Library Writing Profession Department Institutional Space Coursework Personal Space Librarians Funding Exams + Preparation Job Search Previous Experience Collections Dept. Requirements + Expectations Prospectus + Preparation Second Personal Expectations Culture + Community Discovery Services Writing Process + Advising Revision Self-determination Library Space Publishing Teaching Defense + Preparation Personal Life Information Technology Attrition Management Challenges Third Successes Opportunities Fourth
Time from BA graduation Through Expected PhD CompletionGraduation from undergrad to start of PhD Start of PhD to candidacy Candidacy to PhD expected graduation
Humanities PhD Student Study Questionaire Please rate your overall satisfaction with the following at CU: Library Spaces 3 6 8 5 1 1 (Columbia only)Library Collections 24 16 2 3 Library Services 21 19 3 2 Very Satisfied Satisfied Neutral Dissatisfied Advising Very Dissatisfied 10 8 2 1 3 (Columbia only) n/a Funding 14 22 2 2 5 Grad Program 14 24 2 3 2 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Time Spent in the Library by Frequency and Duration 10 9 8 7 6No. of Students 5 4 3 2 1 0 1< 1 -2 hrs 2 - 4 hrs 4 - 6 hrs 6 - 8 hrs 8+ Daily Weekly Monthly
Activities in the Library by Status Post-Exam Pre-Exam other meet colleagues consult librarian computersresearch w. non-library… research w. library… office hours read write browse 0% 20% 40% 60% 80% 100% Percentage of Students
Humanities PhD Student Study QuestionnaireHave you visited any non-CU libraries to use their collections for dissertation research? 5 10 Yes No n/a 30
Humanities PhD Student Study QuestionnaireDo you receive financial support from CU for the academic year? 4 2 Yes No n/a 39
Humanities PhD Student Study QuestionnaireDo you have an outside job that provides income? 5 6 Yes No n/a 34
Findings: Provide Space"The thing that has been the best for me is having a space to work. I got more done last year when I had my locked carrel than I had gotten done in years before or since, because it was a dedicated space in which I could keep all of my sources [...]."
Findings: Foster Community“It’s having community. Belonging to your community, having friends that are doing this and feeling that you have something worthwhile to say that other people are recognizing it.”
Findings: Provide Access to Deep Research Collections“I have to say that I have had every resource that I have needed from the library. I really can’t say, ‘Here I am in the sixth year because you didn’t buy that set of resources for me and I don’t have the materials to work with, so how can you expect me produce work?’”
Findings: Provide Research, InformationManagement, and Teaching Expertise Assistance“[…] maybe sitting down with an advanced research reference librarian . . . might be in my best interest as I go into the writing stage of my paper, just so that I can make sure I am not saying something that has already been said or duplicating research, or that I am not missing something that is cutting-edge and thats really important to my argument.”
Findings: Developing Scholarly Identity“I had to tell my committee in an email, I plan on having a draft of the first chapter to you by June. If I dont, please get on my case… So, I actually found that I needed to make deadlines for myself and then tell them so that they knew, and even though they wouldnt care, my pride was at stake at that point.”
MissionITHAKA is a not-for-profit organization that helps the academiccommunity use digital technologies to preserve the scholarly record andto advance research and teaching in sustainable ways.We pursue this mission by providing innovative services that aid in theadoption of these technologies and that create lasting impact..
Our Services• Ithaka S+R is a strategic consulting and research service that focuses on the transformation of scholarship and teaching in an online environment, with the goal of identifying the critical issues facing our community and acting as a catalyst for change.• JSTOR is a research platform that enables discovery, access, and preservation of scholarly content.• Portico is a digital preservation service for e-journals, e-books, and other scholarly e-content.
Ithaka S+R Surveys 2009 Faculty Survey 2010 Library Director Survey 3,025 Faculty responses 239 Library Directors responded 8.6% of population 13% of population Focus on research institutions 9 Carnegie Classification Levels: – 79 responses are doctoral Conducted 2000, 2003, 2006, – 66 are master’s 2009, 2012 – 94 are baccalaureateIthaka S+R Staff: Roger Schonfeld, Ross Housewright, Matt Long
The Role of the Library: Comparison with Faculty How important to you is it that your college or university library provide each of the functions below? (Percentage answering very important.) Teaching Facilitator Undergraduate Information Literacy Teacher Research Supporter Buyer Archive Gateway 0% 20% 40% 60% 80% 100% Faculty Members Library DirectorsNote: Faculty member data are from Ithaka S+R Faculty Survey 2009.
Digital vs. Print Spending “What percentage of your library’s materials budget is spent on the following items?” NowFive years from Now 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Journals Books All other items
Digital vs. Print Spending Journals Books Directors predict a Directors predict a 106% 46% drop in spending on drop in spending on print journals in the print books in the next 5 years… next 5 years.…bringing budget shares to: …bringing budget shares to:12% Print / 88% Digital 54% Print / 46% Digital
Print to Electronic Transition: Publishing “I am completely comfortable with journals (that I use regularly/ that my library subscribes to) ceasing their print versions and publishing in electronic-only form.” Library Directors Faculty Members Agree 70% 39% Agree Neither agree Neither agree nor disagree 22% 30% nor disagree Disagree 8% 32% Disagree
Print to Electronic Transition: Existing Collections“Within the next five years, the use of (online or digitizedjournals/ electronic versions of scholarly monographs) willbe so prevalent among faculty and students that it will notbe necessary to maintain library collections of hard-copy… … Journals.” … Books.” Agree 54% 7% Agree Disagree 13% 59% Disagree
Library Spending PrioritiesIf you received a 10% increase in your librarysbudget next year in addition to the funds youalready expect to receive, in which of thefollowing areas would you allocate the money?(Please check up to three areas in the following listthat you would invest in.)
Library Spending Priorities Online or digital journals Tools for discovery (OPACs, indices,… Staff for reference and user services/… Facilities expansions and renovations Other digital resources Electronic versions of scholarly…Staff in management/administration of… 0% 20% 40% 60% 80% 100% If you received a 10% increase in your librarys budget next year in addition to the funds you already expect to receive, in which of the following areas would you allocate the money?
Library Staffing Priorities Supporting faculty instruction and student… Providing reference Percentage services ranking this Purchasing/ licensing item as 1 digital resources Percentage Building or maintaining ranking thislocal discovery resources item as 2 Supporting the research projects of faculty… Developing and maintaining special… 0% 20% 40% 60% 80% 100% Ideally, how would you prioritize your staff resources in the following areas? Please rank the items by order of importance.
Discovery: A Declining Role for the Library?80%70%60%50%40% Library Directors Faculty Members30%20%10%0% 2006 "Now" "5 years from now" Percentage answering that it is very important that the library serve as a "gateway"
User Needs AssessmentIn the past 2 years, has your library regularly solicited feedbackabout services or collections from library users in any of thefollowing ways? (Please check all that apply.) 94% Informal discussions with faculty and students or emails soliciting feedback 71% Locally developed polls or surveys 49% Focus groups or test sessions 37% Cross-institutional polls or survey (such as Libqual+) 16% Structured Interviews 10% Ethnographic studies 8% With the help of outside consultants 6% Other Only 13% do not have a formal means to assess user needs.
Library Strategies35% of respondents agreed that “My library has a well-developed strategy to meet changing user needs and research habits.”
challenges• Lack tradition of using data for improvement• No assessment advocate within organization• Library staff lack research methodology abilities• Weak analysis and presentation of data• Inability to identify actionable data• Library “culture” is skeptical of data• Leadership does not view as priority/provide resources• Library organizational structure is “silo-based”• Staff do not have sufficient timeTurning Results into Action: Using Assessment Information to Improve Library Performance, Steve Hiller (University of Washington) , Stephanie Wright (University of Washington)
ReferencesAbout Assessment• “Value of Academic Libraries: A Comprehensive Review and Report,” Megan Oakleaf: http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf• Focus Groups: A Practical Guide for Applied Research by Richard Krueger and Mary Anne Casey• ARL SPEC Kit #303 on Library Assessment, December 2007• Keys to Effective, Sustainable, and Practical Assessment Steve Hiller, Martha Kyrillidou, and Jim Self http://www.arl.org/arldocs/stats/statsevents/laconf/2006/HillerSelf.ppt• www.libraryassessment.info (blog)• Columbia Assessment Program: https://www1.columbia.edu/sec/cu/libraries/bts/assessment/index.htmlInteresting Studies• Studying Students, The Undergraduate Research Project at the University of Rochester http://www.ala.org/ala/mgrps/divs/acrl/publications/digital/Foster-Gibbons_cmpd.pdf• For fun: http://www.coolinfographics.com/• Cornell University Libraries ROI: http://research.library.cornell.edu/value• Ellsevier/UIUC ROI: http://www.slidefinder.net/t/library_strategic_investment/illinois_roi_study/1313459• University of Arizona), Learning in an Online Environment: Assessment of an Online Information Literacy Credit• Course, Yvonne Mery, Jill Newby, Ke Peng: http://libraryassessment.org/bm~doc/2010_lac_poster.pdf• University of Chicago: Wayfinding Revisited: Improved Techniques for Solving Usability Problems in Physical Spaces Agnes Tatarka, David K. Larsen• LibQual+ Survey Literature: www.libqual.org/Publications• 2CUL Humanities PhD Study:Conferences• Library Assessment Conference Proceedings: http://libraryassessment.org• Northumbria Conference Proceedings: http://www.lib.sun.ac.za/Northumbria7/Programme.htmIthaka S+R• http://www.ithaka.org/ithaka-s-r/research/faculty-surveys-2000-2009• http://www.ithaka.org/ithaka-s-r/research/ithaka-s-r-library-survey-2010