Forging Forward: Using Evaluation as a Stepping Stone Introduces the importance of evaluation (often times called assessment in academic libraries)) for all types of libraries. Getting Started with Evaluation, Managing with Data

1. Forging Forward:
Using Evaluation as a
Stepping Stone
Joe Matthews
SLA – San Diego Fall Seminar
October 30, 2015

2. Evaluation is the process of …
• Identifying and collecting data
• Focusing on specific services or
activities
• Establishing criteria to gauge success
• Determining the degree to which
stated goals and objectives are
achieved

4. Types of Evaluation Activities
• Program or service planning
• Program or service monitoring
• Impact assessment (outcomes)
• Economic efficiency

5. Evaluation Questions
How much? How many? How economical? How prompt?
Magnitude
% of change
% of overall change
Costs
Magnitude
Change
Resources used
Units processed
Cycle times
Turnaround time
Anticipatory
Library Controls

6. Evaluation Questions
How valuable? How reliable? How accurate? How well?
Effort expended
Cost-benefit
obtained
Dependability
Access
Accuracy
Currency
Completeness
Comprehensiveness
Accuracy
Expertise
Performance
meeting
expectations
Library & Customer Decide

7. Evaluation Questions
How courteous? How responsive? How satisfied?
Welcoming
Attentive
Anticipatory
Helpful
Empathetic
Expectations met
Materials obtained
Personal interaction
Ease of use
Equipment used
Facilities
Willingness to return
Customers Decide

8. Evidence-Based Library and
Information Practice
1. Formulate a clearly defined question
2. Find the best evidence to answer the
question
3. Critically appraise the evidence
4. Combine the appraisal of the evidence with
professional judgment to make a decision
5. Evaluate the effectiveness of the results
6. Disseminate the results

9. Library Metrics
Resources Capability Outputs Outcomes Impacts
Input
Measures
Process
Measures
Output
Measures
Outcome
Measures
Impact
Measures

10. Library Metrics
• National Center for Education
Statistics Academic Library Survey
• Integrated Postsecondary Educational
Data System (IPEDS)
• ACRL Academic Library Trends and
Statistics survey
• ACRLMetrics
• IMLS Public Libraries Survey
• PLAMetrics

12. Internal Evaluation
• Facilities
• Technologies
• Library Personnel
• Collections
• Services
• Other
Input, Process, and Output Measures

13. External Evaluation
Library’s Perspective
• Demographics and use
• Collection strengths
• Library personnel
• Library deployed technologies
• Facilities
• Services provided
• Unique programs and services

14. External Evaluation
Customer Perspective – Outcomes
• ACRL – Megan Oakleaf’s Value report
• ACRL – Assessment in Action
• The LibValue Project – Carol Tenopir
• PLA – Project Outcome

15. Measuring Satisfaction
Customer
Satisfaction
In-person interview
Focus group
interview
Exit interview
Comment cards
Surveys
Complaint/
Compliment analysis
Mystery shopping

16. Satisfaction Surveys
• Gaps Model of Service Quality –
LibQUAL
– Affect of service
– Information control
– Library as place
• Counting Opinions – LibSat
• DIY

17. Customer Satisfaction
• Customer comments
• Net Promoter Score (NPS)
• Opportunity index
• Resolution rate

18. Value of the Library
Value of the
Library
Personal &
Organizational
Benefits
Economic
Benefits
Direct
Direct
Indirect
Nonuse
Indirect

19. Value of the Special Library
• Cognitive results
• Affective results
• Meeting expectations
• Task accomplishments
• Time aspects
• Money aspects

20. Value of the Academic Library
Student Faculty Institution
Recruitment/Enrollment
Retention & Graduation
Success
Achievement
Learning
Experience, Attitude &
Perception of Quality
Research Productivity
Grants
Teaching
Reputation & Prestige

21. Value of the Public Library
Generic
Learning
Outcomes
Knowledge & Understanding
Attitudes & Values
Activity, Behavior
& ProgressionEnjoyment, Inspiration
& Creativity
Skills

22. Communicating the Results
• Library staff members
• Friends groups
• Prospective donors
• Board of Trustees
• Community governing
boards
• Institutional stakeholders
• Library administrative
agencies
• State legislatures
• Manager/Partner
• Customers
– Faculty
– Graduate students
– Undergraduate students
– Others

24. 1. Accountability is part of librarianship
2. Austerity increases the need for
evaluation
3. Librarians are engaged in
organizational change management
4. Evaluation is not brain surgery

25. 5. Meaningful evaluation focuses on
customers and demonstrates the
value of the library to them
6. The hardest thing is to start –
“Just do it”
7. Play with the various concepts

26. 8. Make evaluation a part of your
regular work
9. Share the results
10. Implement a results reporting
system

27. Joe AT JoeMatthews.Org