Evaluating the Big Deal: Usage Statistics for Decision Making
Evaluating the Big Deal:What metrics matter?Usage Statistics for Decision MakingLondon, 2 February 2012Selena KillickLibrary Quality Officer
Introduction • Institutional, financial and strategic context • Evaluating the Big Deals • Quantitative Reporting • Qualitative Measures • Using the Results • Conclusions
Cranfield University • The UKs only wholly postgraduate university focused on science, technology, engineering and management • One of the UKs top five research intensive universities • Annual turnover £150m • 40% of our students study whilst in employment • We deliver the UK Ministry of Defences largest educational contract
Key Drivers • Financial realities • Demonstrating value for money • Strategic alignment • Research Excellence Framework (REF) • Income Mission critical • Reputation
Expenditure on Journals Journal Spend 2006-07 2007-08 2008-09 2009-10
Information Expenditureby Format 2010-11 Books eBooks 4% 4% Databases 24% Journals 68%
Information Expenditureby Format 2010-11 Books eBooks 4% 4% Databases 24% Journals 31% Big Deals 37%
Previous TechniquesUsed: Annual journals review using the follow data • Circulation figures – issues and renewals • “Sweep survey” to capture in-house use • Journal contents page requests • Download figures • Journal prices v the cost of ILL requests More recent focus on “cost per download”
New ApproachQuantitative: Qualitative:• Size • Academic Liaison• Usage • Reading Lists Review• Coverage • REF Preferred• Value for Money
Requirements • Systematic • Sustainable • Internal benchmarking • Elevator pitch • So what? • Enable informed decision making • Demonstrate smart procurement
Our Approach • What has everyone else done? • Analysing Publisher Deals Project • Storage centre • Excel training • Template design
Basic Metrics • Number of titles within a package • Total annual full-text downloads • Cost: • Core titles • e-Access Fee • Total costs
Value Metrics • Average number of requests per title • Average cost per title • Total cost as % of information provision expenditure • Cost per full-text download • Average download per FTE student/staff/total • Average cost per FTE student/staff/total
The Long TailDownloads Titles Titles Titles Long Tail Short Tail No Tail
Subscribed Titles • Reviewing performance of core collection • REF Preferred? • Popular? • Three year trends in cost / downloads / CPD • Cost / Downloads / CPD categorised: • Zero • Low • Medium • High • Cancel?
Popular Titles • Which titles are the most popular? • Top 30 titles in the package • Three year trends in downloads • REF Preferred? • Subscribed title?
Considerations • When to measure from/to? • calendar, financial/academic, or contract year? • Which titles make up our core collection? • Do we have access to all of the „zero use‟ titles? • What constitutes Low/Medium/High? • What about the aggregator usage statistics? • Do we trust the usage statistics? • What is the size of the target population?
Downloading Statistics • Get organised • Gather your usernames and passwords • Create local files to save and store usage reports • Software now on the market to manage this for you • Joint Usage Statistics Portal
Reporting Period • Calendar, financial/academic, or contract year? • COUNTER Reports = Calendar year • Converted using Vlookup on ISSNs • Manual • Problematic • Automatically converted using the JUSP
Aggregator UsageStatistics • Combining usage from publishers and aggregators at a title level • Combined using Pivot Tables • Manual • Problematic • Where possible combined using JUSP
Excel Template • Two main data sources: • COUNTER JR1 • Subscription agent financial report • Automated as much as possible • Match formulas working with ISSNs to link title price to usage/holdings • All calculations are completed automatically when the data sources are added
Academic Liaison • Who‟s using it? • Why? • How? • What will be the impact if we cancel? • Teaching? • Research? • How valuable is it?
Quantitative on theQualitative: Analysis on the five REF Preferred Recommended Journals Lists: • Overlapping titles • Unsubscribed titles • Financial shortfall • Current recommended subscribed titles • Usage data
Reading List Review Qualitative analysis on course reading lists: • What are our academic recommending? • Where is it published? • How often is it recommended? • Are there alternatives?