J. Stephen Town, Cranfield University. Evaluation of Library & Information Services: Does it lead to innovation and effectiveness? November 16-17 Vilnius, Lithuania

1. Performance and innovation culture in
academic libraries: the role of LibQUAL+
in enhancing quality
J. Stephen Town
Cranfield University
Evaluation of Library & Information Services: Does it
lead to innovation and effectiveness?
November 16-17
Vilnius, Lithuania

2. Summary
• Thoughts on performance and innovation
• LibQUAL+ as a source of innovation and
effectiveness
– The UK national experience
– Institutional experiences (Two case studies)
• Innovation culture
– Development issues

3. Performance and innovation
Is there a connection?

4. Some dichotomies?
• Performance or measurement?
• Evaluation or innovation?
• Indicators or improvements?
• Accountability or activity?
• Processes or projects?
• Equity or equivocation?
• Quality or acquiescence?

5. The UK HE Quality & measurement
context
• Quality as QA
– Teaching Quality Assessment
• Quality as Peer Review
– Research Assessment Exercise
• Quality as batteries of performance
indicators
– Statistical collection and league tables
• Quality as culture
– TQM, IIP, Charter Mark

6. Measurement Progress (Lancour, 1951)
Three Phases of Academic Library development
• “Storehouse”
• “Service”
• “Educational”

7. The University Context
Universities have two “bottom lines”
1. Financial (as in business)
2. Academic, largely through reputation in
• Research (the priority in leading Universities)
• Teaching (& maybe Learning)

8. Library Pressures for Accountability
The need is therefore to demonstrate the Library
contribution in these two dimensions:
1. Financial, through “value for money” or related
measures
2. Impact on research, teaching and learning
This also implies that “competitive” data will be highly
valued

9. The UK & Ireland Experience
The SCONUL Working Group on Performance
Improvement
• Ten years of “toolkit” development to assist in
performance measurement and improvement
• SCONUL ‘Top concern survey’ 2005
• The SCONUL Value and Impact Measurement
Programme (VAMP)

10. Examples of tools developed 1
• Integration
• Efficiency &
Comparability
Quality assurance
Guidelines
SCONUL Statistics &
interactive service
HELMS national
performance indicators
E-measures project
Benchmarking Manual

11. Examples of tools developed 2
• Satisfaction
• Impact
SCONUL Satisfaction
Survey
SCONUL LibQUAL+
Consortium
LIRG/SCONUL Impact
Initiative
Information Literacy
Success Factors

12. VAMP Objectives
• New missing measurement instruments &
frameworks
• A full coherent framework for performance,
improvement and innovation
• Persuasive data for University Senior
Managers, to prove value, impact,
comparability, and worth

13. Missing methods?
• An impact tool or tools, for both teaching &
learning and research
• A robust Value for Money/Economic Impact
tool
• Staff measures
• Process & operational costing tools

14. Member Survey Findings
• 38 respondents; 27% of population
• 70% undertaken value or impact measurement
• Main rationales are advocacy, service improvement,
comparison
• Half used in-house methodologies; half used standard
techniques
• Main barrier is lack of tools, making time an issue
• Buy-in of stakeholders is an issue

15. Some Conclusions …
• There is a need to demonstrate value and that
libraries make a difference
• Measurement needs to show ‘real’ value
• Need to link to University mission
• Libraries are, and intend to be, ahead of the game
• Impact may be difficult or impossible to measure
– A pedagogic project needed?
• Measurement is not innovation, and may inhibit it
– Re-engineering needed first?

16. Next Steps 1
“Content” Products
2.1 Value & Impact Guidelines
2.1.1 Institutional Value (eg VFM & Economic Impact)
2.1.2 Impact on Teaching & Learning
2.1.3 Impact on Research

17. Next Steps 2
“Content” Products
2.2 Staffing & Operational Measures Guidelines
2.2.1 Staff Costing
2.2.2 Staff Added Value measures
2.2.3 Other operational costing methods
2.3 Re-branding & packaging of existing tools

18. Next Steps 3
“Process” Products
3.1 Web Site
3.2 Community of practice establishment
3.3 Maintenance & sustainability strategy

19. LibQUAL+ for enhancing quality
A source of data to support
performance and innovation

20. What is LibQUAL+?
• A web-based survey tool designed to
measure Library quality
• Provides comparable data with other
institutions to help benchmark services
• Provides detailed data to suggest a service
improvement agenda, and longitudinal data
to test improvement actions

21. LibQUAL+ History
• ARL New Measures initiative
• Developed by Texas A&M University
• Based on SERVQUAL
• Piloted in 2000
• Now used by over 850 libraries worldwide

23. LibQUAL+ in Europe
• SCONUL (UK & Ireland)
– 2003: Pilot with 20 member libraries
– 2004: 17 participants
– 2005: 17 participants
– 2006: 22 participants
– 55 different institutions over the 4 years
• European Business Schools Librarians’ Group
– 2004: Pilot with 5 member libraries
– 2006: 12 participants in 7 European countries
• National Health Service (UK)
– 2006: Pilot with 12 member libraries

24. Benefits of LibQUAL+
• Managed service
– for delivery & analysis
– cost
• Web-based
• Gap analysis
• Permits benchmarking
– Peers, nationally & internationally

25. Time frame
• Surveys can be run for a chosen duration in:
– Session 1: January – June
– Session 2: July – December
• January / February
– Training for Session 1 Participants
– Results meeting for Session 2 Participants
• July / August
– Training for Session 2 Participants
– Results meeting for Session 1 Participants

26. Dimensions of
Library Service Quality
Empathy
Information
Control
Responsiveness
Symbol
Utilitarianspace
Assurance
Scope of Content
Ease of Navigation
Self -Reliance
Library asPlace
Library
Service
Quality
Model 3
Refuge
Affectof Service
Reliability
Convenience
Timeliness
Equipment
F. Heath, 2005

27. The Survey Comprises of
• 22 Core questions
• 5 Local questions (selected by the
institution)
• 5 Information Literacy questions
• 3 General Satisfaction questions
• Demographic questions
• A free-text comments box

28. Sample Survey

29. How it works
• For the 22 “core” questions and 5 “local”
questions users rate out of 1 – 9 their:
– Minimum service level
– Desired service level
– Perceived service performance
• This gives us a “Zone of Tolerance” for each
question, and an “Adequacy Gap”

31. Benefits of gap analysis
AverageRating
9.0
8.0
7.0
6.0
5.0
4.0
3.0
2.0
Affect of Service Information Control Library as Place
Perceived
Range of
Minimum to
Desired
Range of
Minimum to
Perceived
(“Gap”)

32. Comments box
• Free-Text comments box at the end of the survey
• About 40% of participants provide open-ended
comments, and these are linked to demographics
and quantitative data
• Users elaborate the details of their concerns
• Users feel the need to be constructive in their
criticisms, and offer specific suggestions for action

33. The SCONUL Experience

34. LibQUAL+ Participants 2003
• University of Bath
• Cranfield University
• Royal Holloway & Bedford
New College
• University of Lancaster
• University of Wales, Swansea
• University of Edinburgh
• University of Glasgow
• University of Liverpool
• University of London Library
• University of Oxford
• University College
Northampton
• University of Wales College
Newport
• University of Gloucestershire
• De Montfort University
• Leeds Metropolitan
University
• Liverpool John Moores
University
• Robert Gordon University
• South Bank University
• University of the West of
England, Bristol
• University of Wolverhampton

35. LibQUAL+ Participants 2004
• Brunel University
• Loughborough University
• University of Strathclyde
• University of York
• Glasgow University
• Sheffield University
• Trinity College, Dublin
• UMIST + University of
Manchester
• University of Liverpool
• Anglia Polytechnic
University
• University of Westminster
• London South Bank
University
• Napier University
• Queen Margaret University
College
• University College
Worcester
• University of East London

36. LibQUAL+ Participants 2005
• University of Exeter
• University of Edinburgh
• University of Dundee
• University of Bath
• University of Ulster
• University College
Northampton
• University of Birmingham
• Roehampton University
• University of Glasgow
• University of Surrey
• Royal Holloway UoL
• City University
• Cranfield University
• University of Luton
• Dublin Institute of
Technology
• London South Bank
University
• Coventry University

37. LibQUAL+ Participants 2006
• Cambridge University
• Cranfield University
• Goldsmiths College
• Institute of Education
• Institute of Technology
Tallaght*
• Queen Mary, University of
London
• Robert Gordon University
• St. George's University of
London
• University of Aberdeen
• University College for the
Creative Arts
• University of Central
Lancashire
• University of
Gloucestershire
• University of Leeds
• University of Leicester
• University of Liverpool
• University of the West of
England
• University of Warwick
• University of Westminster
• London South Bank
University
• Scottish Royal Agricultural
College
• University of Birmingham
• University of Glasgow

38. Overall Potential UK Sample to 2006
• Full variety of institutions
• 43% of institutions
• 38% of HE students (>800,000)
• 42% of Libraries
• 48% of Library expenditure

39. SCONUL Overall Results 2005

40. Aims & purposes
• Analysis compilation
• Comparison to existing
survey methods
• A library focused survey
• Benchmarking
• Charter Mark
application
• Strategic planning aid
• Real data as opposed to
lobbying
• To make adjustments
where needed
• To test improvement
• “User satisfaction - as
simple as that”

41. Process Feedback
• Straightforward
• Publicity requires the
most effort
• Difficulty in obtaining
email addresses
• Difficulty in obtaining
demographic data
• Very simple to
administer
• Results as expected
• More in-depth detail
obtained
• More ‘discriminatory’
than other surveys
• Helped to strengthen
Library’s case
• Comments very specific
& helpful

42. Case Studies

43. Cranfield University at DCMT
• Cranfield’s Library services at the Defence
College of Management & Technology
• Contract situation demanding high quality
services
• Military and civilian education and research
in defence, management & technology
• About 1000 students, almost all postgraduate
and post-experience

44. DCMT Library Surveys
• Student perspective
(1993)
• Exit questionnaires
(1994-)
• Information Services
(Priority Search 1996)
• DTC MSc & MA Students
(1997)
• Researchers Survey
(Web based 1998)
• SCONUL Survey Pilot
(1999)
• SCONUL Template
(2001)
• LibQUAL+ (2003, 2005,
2006)

45. DCMT LibQUAL+ Surveys
• 2003, 2005, 2006
• Increasing responses
– 11%, 16%, 22%
– Year on year 40% up
• Increasing comments
– 83, 153, 205 (almost 60% of respondents)
• Improved performance across three years

46. DCMT Overall 2006

47. Agenda for Action 2003
• Information skills training
• Improving staff specialist skills
• Access to electronic resources
• Customer care to different users

48. DCMT Survey aims for 2005-06
• Test new Library building
• Test launch of the new Library Web site
• Test maintenance of other progress
– Improved capability in data analysis & presentation
• Develop a new strategy in line with changing
academic needs

49. Changes over three years

50. Internal Benchmarking
Affect of Service
Information
Control
Library as
Place Overall

51. National Average External Benchmarking
-1.6
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
Superiority Mean
DCMT Superiority Mean -0.07 -0.83 -0.33 -0.41
UK Superiority Mean -0.91 -1.32 -1.49 -1.20
US Superiority Mean -0.73 -1.37 -1.17 -1.09
Affect of Service Information Control Library as Place Overall

52. Peer-to-Peer Benchmarking
Dimension Summary
4.00
4.50
5.00
5.50
6.00
6.50
7.00
7.50
8.00
8.50
Affect of
Service
Affect of
Service
Information
Control
Information
Control
Library as
Place
Library as
Place
Overall Overall
DCMT Library 2 DCMT Library 2 DCMT Library 2 DCMT Library 2

53. University of Glasgow
• Founded in 1451
• Large research-led institution
• About 20,000 students in 10 Faculties, and
about 6,000 staff
• Member of the Russell Group of major UK
research-led Universities
• Founder member of Universitas 21

54. Survey Participation
• Participation in LibQUAL+ 2006 will be
Glasgow’s 4th successive year in the SCONUL
Consortium
• 2006 – 1,535 responses
• 2005 – 1,423 responses
• 2004 – 2,212 responses, 920 comments
• 2003 – 502 responses, 402 comments

55. Aims of Use of the Data
• Strategic Service Developments
– Data to support service development
– Ability to identify where not meeting expectations
– Measure if change has met need
• Budget Discussions
– Data to support bid for increased funding
– Data to support case for change in emphasis (towards e-provision)
• Marketing Position
– Status of the library within the University
– Importance of national & international benchmarking

56. LibQUAL+ Outcomes
• New Web Services Administrator
• Increased opening Hours
– Earlier Saturday morning opening
– Sunday morning opening
– Increased late opening hours
(From January 2006 Mon-Thurs 08:00 – 02:00)
• Now providing 222,578 seat hours per week

57. Library Refurbishment Programme reinstated at costs
in excess of £8 million
From: To:

58. Conclusions
• LibQUAL+ is now a market leading survey tool for UK
& Irish Academic & Research Libraries, and growing
use in Europe
• Use of the technique can strongly support and help
develop an innovative culture, and provide evidence
of impact as well as satisfaction
• Some significant advantages over other survey
methods
• Additional support and data analysis is now available
in Europe through ARL/Cranfield contract

59. Innovation culture?
• Customer Focus
– Requires a satisfaction instrument
– Requires further analysis & consultation
• Systematic improvement
– Requires success factors (& measures) for each
– Requires project management alongside BAU processes
• Total Involvement
– Requires not only a quality culture, but also a meta-quality
culture in which the culture itself can be assessed
– Supported by other techniques eg situational leadership

60. A Quality Maturity Model (Wilson & Town)

61. The Level One Library
5 Optimising
4 Managed
3 Defined
2 Repeatable
1 Initial
2
3
4
5
1
The improvement process is characterised as ad hoc, and
occasionally even chaotic. Innovation and evaluation
tends to be one-off and success depends
on individual efforts and heroics.

62. Level Two
5 Optimising
4 Managed
3 Defined
2 Repeatable
1 Initial
2
3
4
5
1
Basic project management approaches are established to
improve performance & service. The necessary
discipline is in place to repeat earlier successes on
innovation projects.

63. Level Five: an innovative culture
5 Optimising
4 Managed
4 Defined
2 Repeatable
1 Initial
2
3
4
5
1
The Library has a fully defined, organised and managed
approach to innovation and improvement, and is able to
optimise fully its evaluation and improvement project effort
to the benefit of customers. All staff understand and share
the appropriate cultural assumptions.

64. Overall Conclusions
• There is an overall management challenge in
creating an innovative culture for which we still lack
a full range of measures
• Measures are an essential aid, but not the culture
itself
• Effectiveness requires proof of impact, which may
be impossible to measure
• Building an innovative culture is unlikely to result
from evaluation alone, but it is a necessary step
• Staff development is more important than
measurement, but they can be linked

65. LibQUAL+
If you would like to know more about LibQUAL+, or are
considering participating as a consortium or
independently see:
www.libqual.org
Or contact:
Selena Lock
email: s.a.lock@cranfield.ac.uk
Telephone: +44 (0) 1793 785561

66. Acknowledgements
• SCONUL and its Working Group on Performance
Improvement
• Selena Lock, R&D Officer, Cranfield University
• Jacqui Dowd, Management Information Officer,
Glasgow University
• Bruce Thompson, Distinguished Professor of
Educational Psychology and CEHD Distinguished
Research Fellow, and Distinguished Professor of
Library Science, Texas A&M University
• Association of Research Libraries

67. Seminar Questions?
• Do measurement and evaluation guarantee
performance or quality?
• Does evaluation lead to innovation?
• What are the key measures for
accountability in an academic context?
• What professional development
requirements arise from these evaluation
methods?