This document summarizes Selena Killick's presentation on evaluating big deal journal packages through qualitative and quantitative methods at Cranfield University. Cranfield University spends over £150 million annually, with 68% of its information expenditure going towards journal subscriptions in 2010-2011. Killick developed an approach using both quantitative metrics like downloads, costs, and usage trends, as well qualitative measures involving academic liaisons and analyzing reading lists and REF preferred journals. She created an Excel template to automate calculations from COUNTER and subscription data. The results helped inform collection decisions while justifying expenditures and demonstrating the value of resources.

1. Qualitative &
Quantitative Methods in
Electronic Collection
Management
4th International Conference on Qualitative
and Quantitative Methods in Libraries
Limerick, Ireland
24 May 2012
Selena Killick
Library Quality Officer

2. Cranfield University
• The UK's only wholly postgraduate university focused
on science, technology, engineering and
management
• One of the UK's top five research intensive
universities
• Annual turnover £150m
• 40% of our students study whilst in employment
• We deliver the UK Ministry of Defence's largest
educational contract

3. Expenditure on
Journals
Journal Spend
2006-07 2007-08 2008-09 2009-10

4. Information Expenditure
by Format 2010-11
Books eBooks
4% 4%
Databases
24%
Journals
68%

5. Information Expenditure
by Format 2010-11
Books eBooks
4% 4%
Databases
24%
Journals
31%
Big Deals
37%

6. Evaluating the Big Deals:
Requirements
• Systematic
• Sustainable
• Internal benchmarking
• Elevator pitch
• So what?
• Enable informed decision making
• Demonstrate smart procurement

7. New Approach
Quantitative: Qualitative:
• Size • Academic Liaison
• Usage • REF Preferred
• Coverage • Reading Lists Review
• Value for Money
CONYERS, A., 2007. What do publisher usage statistics tell us? The Analysing Publisher Deal project from
Evidence Base. SCONUL Focus, no.40, pp.72-76, Available at:
http://www.sconul.ac.uk/publications/newsletter/40/25.pdf.

8. Quantitative Metrics
Including:
• Average number of downloads per title
• % of titles with zero downloads
• Average cost per title
• Usage of core titles
• Cost per full-text download
• Overall
• For each core title
• Three-year trends for most popular titles
• Number of core titles in Top 30 most popular titles

9. Subscribed Titles
• For each core title the cost, downloads, and cost-per-
download categorised:
• Zero
• Low
• Medium
• High
• Cancel?

10. Considerations
• When to measure from/to?
• calendar, financial/academic, or contract year?
• Which titles make up our core collection?
• Do we have access to all of the ‘zero use’ titles?
• What constitutes Low/Medium/High?
• What about the aggregator usage statistics?
• Do we trust the usage statistics?
• What is the size of the target population?

11. Excel Template
• Two main data sources:
• COUNTER JR1
• Subscription agent financial report
• Automated as much as possible
• Match formulas working with ISSNs to link title price to
usage/holdings
• All calculations are completed automatically when the
data sources are added
• Results fitted onto a one-page printout

12. Quantitative
Reporting
• Systematic 
• Sustainable 
• Internal benchmarking 
• Elevator pitch 
• So what? 
• Enable informed decision making 
• Demonstrate smart procurement 

13. Qualitative Measures:
Academic Liaison
• Who’s using it?
• Why?
• How?
• Who’s recommending it?
• How valuable is it?
• What will be the impact if we cancel?
• Teaching?
• Research?

14. REF Preferred Journals
• Each academic school compiled a list of journals
seen as being REF Critical
• Lists combined & analysed by Library staff
• Number of overlapping titles
• Number & cost of titles we do not hold in our
collection
• Usage of titles we do hold in our collection

15. Reading List Review
Analysis of course reading lists:
• What are our academic recommending?
• Where is it published?
• How often is it recommended?

16. Quantitative &
Qualitative
Reporting
• Systematic 
• Sustainable 
• Internal benchmarking 
• Elevator pitch 
• So what? 
• Enable informed decision making 
• Demonstrate smart procurement 

17. Using the results.
What they can do:
• Both qualitative and quantitative measures tell the
story of the resource
• Aid decision making
• Justify procurement
• Safeguard budgets…?

18. What they can’t do:

19. Closing thoughts
• Is it worth investing in this?
• Qualitative & Quantitative
• Danger of relying on cost-per-download

20. Looking Ahead
• Journal Usage Statistics Portal
• New reports = new internal processes
• Review of all budgets
• All Resources
• Systems
• Staff
• Services
• Demonstrating Value and Impact
• Resources
• Services

21. References
• CONYERS, A., 2010. Usage Statistics and Online Behaviour (2). In: G.
STONE, R. ANDERSON and J. FEINSTEIN, eds, The E-Resources
Management Handbook – UKSG. Burford: UKSG. Available at:
http://uksg.metapress.com/link.asp?id=084t98646x2rn62k
• CONYERS, A., 2007. What do publisher usage statistics tell us? The
Analysing Publisher Deal project from Evidence Base. SCONUL Focus,
no.40, pp.72-76, Available at:
http://www.sconul.ac.uk/publications/newsletter/40/25.pdf.
• Lib-Stats discussion list and archive: http://www.jiscmail.ac.uk/LIB-
STATS
• TAYLOR-ROE, J. and SPENCER, C., 2005. A librarian's view of usage
metrics: through a glass darkly? Serials: The Journal for the Serials
Community, 18(2), pp. 124-131.
• Joint Usage Statistics Portal (JUSP): http://jusp.mimas.ac.uk/

22. Thank You
Selena Killick
Cranfield University
s.a.killick@cranfield.ac.uk
Tel: 01793 785561