Getting the Most Out of Your E-Resources: Measuring Success Todd Carpenter Managing Director, NISO

Where are we headed this afternoon? A bit about NISO Overview of usage measurement of e-resources COUNTER & SUSHI The application of usage data Issues and concerns with use data A glimpse into the future

A bit about NISO

Overview of usage measurement of e-resources

COUNTER & SUSHI

The application of usage data

Issues and concerns with use data

A glimpse into the future

What is NISO? NISO - National Information Standards Organization NISO is the only ANSI-accredited organization tasked with the development of standards in the field of Information and Documentation Work with publishers, libraries, agents and other systems vendors to develop community consensus Develop wide range of standards Paper permanence and steal shelving Accessibility issues Bibliographic formats and exchange Web-based delivery, OpenURL, Metasearch, SUSHI

NISO - National Information Standards Organization

NISO is the only ANSI-accredited organization tasked with the development of standards in the field of Information and Documentation

Work with publishers, libraries, agents and other systems vendors to develop community consensus

Develop wide range of standards

Paper permanence and steal shelving

Accessibility issues

Bibliographic formats and exchange

Web-based delivery, OpenURL, Metasearch, SUSHI

Big Challenges, Modest Resources Revenue: $900K, up 20% in 2007 Primary income: Member dues (60%) Other income: Seminars, Publishing (20%) New sources of revenue in 2007 - Grants Mellon $196K, IMLS - $24K (20%) Staff: 4 Professional full-time Virtual staff: 10+ (Consultants, Partners) 83 Voting Members, 25 LSA members as of 2007 Maintenance Agencies: 12 Volunteers: 300+ spread out across the world

Revenue: $900K, up 20% in 2007

Primary income: Member dues (60%)

Other income: Seminars, Publishing (20%)

New sources of revenue in 2007 - Grants

Mellon $196K, IMLS - $24K (20%)

Staff: 4 Professional full-time

Virtual staff: 10+ (Consultants, Partners)

83 Voting Members, 25 LSA members as of 2007

Maintenance Agencies: 12

Volunteers: 300+ spread out across the world

Standards – Why should I care? Standards accelerate production, ordering/sales, dissemination, locating, storing and preserving information Key standards which NISO has developed and helping to bring consensus around ISSN, OpenURL, Z39.50, NCIP In development: DOI, SUSHI, SERU, LEWG In planning: Institutional ID, Performance Measures, OpenURL Expansion,

Standards accelerate production, ordering/sales, dissemination, locating, storing and preserving information

Key standards which NISO has developed and helping to bring consensus around

ISSN, OpenURL, Z39.50, NCIP

In development: DOI, SUSHI, SERU, LEWG

In planning: Institutional ID, Performance Measures, OpenURL Expansion,

Meaningless? Certainly, there’s a lot of data The difference between meaningless and meaningful data is APPLICATION

Certainly, there’s a lot of data

The difference between meaningless and meaningful data is APPLICATION

The Early Days of E-Resources Thinking back to 1998 - Many open questions How do you record traffic? What is a hit? Is a hit different than a download? What about reloading? What about images and links? What should a report include? Eventually, counting different versions of texts How often do you need to provide stats Traffic to abstracts, TOCs or other elements? Content over multiple pages?

Thinking back to 1998 - Many open questions

How do you record traffic?

What is a hit?

Is a hit different than a download?

What about reloading?

What about images and links?

What should a report include?

Eventually, counting different versions of texts

How often do you need to provide stats

Traffic to abstracts, TOCs or other elements?

Content over multiple pages?

Slow development of consensus ICOLC - Guidelines for Statistical Measurement of Usage of Web-Based Information Resources Released in 1998, updated in 2000 Minimum Requirements– Data elements, timeframe, etc Confidentiality Access Delivery Definitions Formats National Commission on Libraries and Information Science (NCLIS) Electronic Access and Use-Related Measures Released in 2001

ICOLC - Guidelines for Statistical Measurement of Usage of Web-Based Information Resources

Released in 1998, updated in 2000

Minimum Requirements– Data elements, timeframe, etc

Confidentiality

Access

Delivery

Definitions

Formats

National Commission on Libraries and Information Science (NCLIS) Electronic Access and Use-Related Measures

Released in 2001

Toward Formalization ANSI/NISO Z39.7: 2004 - Information Services and Use Metrics & Statistics for Libraries and Information Providers -- Data Dictionary ONLINE: www.niso.org/emetrics/current/index.html Technical Committee 46 - Information and Documentations, SC 8 - Statistics and Performance Indicators ISO 2789: 2006 Information and documentation -- International library statistics ISO 11620: 1998 Information and documentation -- Library performance indicators

ANSI/NISO Z39.7: 2004 - Information Services and Use Metrics & Statistics for Libraries and Information Providers -- Data Dictionary

ONLINE: www.niso.org/emetrics/current/index.html

Technical Committee 46 - Information and Documentations, SC 8 - Statistics and Performance Indicators

ISO 2789: 2006 Information and documentation -- International library statistics

ISO 11620: 1998 Information and documentation -- Library performance indicators

Project COUNTER COUNTER (Counting Online Usage of NeTworked Electronic Resources) Formed in 2002 Membership organization (as of 3/15/08) Industry Organizations - 13 Library Consortia - 62 Libraries - 84 Publishers - 66 Establishes Codes of Practice on the gathering, compiling and storage of publishing usage data

COUNTER (Counting Online Usage of NeTworked Electronic Resources)

Formed in 2002

Membership organization (as of 3/15/08)

Industry Organizations - 13

Library Consortia - 62

Libraries - 84

Publishers - 66

Establishes Codes of Practice on the gathering, compiling and storage of publishing usage data

COUNTER Codes of Practice Definitions Specifications for Usage Reports What they should include What they should look like How and when they should be delivered Data processing guidelines Auditing (New in 2006) Compliance Maintenance and development of the Codes of Practice Governance of COUNTER

Definitions

Specifications for Usage Reports

What they should include

What they should look like

How and when they should be delivered

Data processing guidelines

Auditing (New in 2006)

Compliance

Maintenance and development of the Codes of Practice

Governance of COUNTER

COUNTER: Current Codes of Practice 1) Journals and databases Release 1 Code of Practice launched January 2003 Release 2 replaced Release 1 in January 2006 Release 3 under consideration - to include SUSHI compliance and consortia reporting Now a widely adopted standard by publishers and librarians 60%+ of Science Citation Index articles now covered 2) Books and reference works Code of Practice for Books was launched March 2006 Relevant usage metrics less clear than for journals Different issues than for journals Direct comparisons between books less relevant Understanding how different categories of book are used is more relevant

1) Journals and databases

Release 1 Code of Practice launched January 2003

Release 2 replaced Release 1 in January 2006

Release 3 under consideration - to include SUSHI compliance and consortia reporting

Now a widely adopted standard by publishers and librarians

60%+ of Science Citation Index articles now covered

2) Books and reference works

Code of Practice for Books was launched March 2006

Relevant usage metrics less clear than for journals

Different issues than for journals

Direct comparisons between books less relevant

Understanding how different categories of book are used is more relevant

Journal Report 1 Example

Need for easier access to usage data As a community, we need to reduce the time and effort necessary to collect, format and compile usage data “ Time for meaningful analysis is compromised by the time required just to gather and record the statistics.” Median percentage of time spent on analysis is only 25 percent More than half of the time is spent on gathering and formatting Average number of hours spent working on usage data is 96 hours, but ranged on the high end up to 1-2 FTEs entirely focused on data Usage reports to help them make subscription decisions (94%) and justify expenditures (86%) for their electronic resources DATA FROM: Gayle Baker, Eleanor J. Read, Vendor Usage Data for Electronic Resources: A Survey of Libraries http://smartech.gatech.edu/handle/1853/13611

As a community, we need to reduce the time and effort necessary to collect, format and compile usage data

“ Time for meaningful analysis is compromised by the time required just to gather and record the statistics.”

Median percentage of time spent on analysis is only 25 percent

More than half of the time is spent on gathering and formatting

Average number of hours spent working on usage data is 96 hours, but ranged on the high end up to 1-2 FTEs entirely focused on data

Usage reports to help them make subscription decisions (94%) and justify expenditures (86%) for their electronic resources

DATA FROM: Gayle Baker, Eleanor J. Read, Vendor Usage Data for Electronic Resources: A Survey of Libraries http://smartech.gatech.edu/handle/1853/13611

One example of the need One university with more than 75 online resources from which they draw usage data They have a 80-page booklet containing the details of how to access and gather usage data! How much time does it take not just compiling and maintaining this notebook, but even just going through it?

One university with more than 75 online resources from which they draw usage data

They have a 80-page booklet containing the details of how to access and gather usage data!

How much time does it take not just compiling and maintaining this notebook, but even just going through it?

Briefly: SUSHI Need: Simplify and automate the gathering of usage data for librarians Librarians spending months gathering data Solution Server/Client system to exchange COUNTER reports Easily incorporated into usage systems (on publisher side) or into ERM (on library side) Client calls to server, asks for report, and server runs the report and sends it on Data exchange is taking place by machine talking with machine

Need: Simplify and automate the gathering of usage data for librarians

Librarians spending months gathering data

Solution

Server/Client system to exchange COUNTER reports

Easily incorporated into usage systems (on publisher side) or into ERM (on library side)

Client calls to server, asks for report, and server runs the report and sends it on

Data exchange is taking place by machine talking with machine

Content Provider Library SUSHI Server Usage Data SUSHI Client Internet ERM SUSHI is a Web Service which sends an XML request to a content provider to obtain an XML response containing the usage report. ? Response COUNTER Request SOAP Slide courtesy of Oliver Pesch, EBSCO Information Services, Co-Chair SUSHI

SUSHI: Where now, where to? Passed unanimously by NISO membership in September, 2007 Formally approved as ANSI/NISO Z39.93:2007 Working toward broad ADOPTION Ask that it be included in your ERM solution Demand your content providers become SUSHI and COUNTER (rev 3) compliant Talk to your vendors and your consortia

Passed unanimously by NISO membership in September, 2007

Formally approved as ANSI/NISO Z39.93:2007

Working toward broad ADOPTION

Ask that it be included in your ERM solution

Demand your content providers become SUSHI and COUNTER (rev 3) compliant

Talk to your vendors and your consortia

Careful the conclusions you draw

What is the most used resource? BioOne’s most viewed article The nest architecture of the Florida harvester ant, Pogonomyrmex badius Walter R. Tschinkel “ Coolest images on the net”

BioOne’s most viewed article

The nest architecture of the Florida harvester ant, Pogonomyrmex badius

Walter R. Tschinkel

“ Coolest images on the net”

Link Prefectching affecting data? Link prefetching is a browser mechanism, which utilizes browser idle time to download or prefetch documents that the user might visit in the near future. Based on previous use data, the site provides a set of prefetching hints to the browser, and after the browser is finished loading the page, it begins silently prefetching specified documents and stores them in its cache. When the user visits a prefetched document, it can be served up quickly out of the browser's cache. Source: Mozzila .org

Link prefetching is a browser mechanism, which utilizes browser idle time to download or prefetch documents that the user might visit in the near future.

Based on previous use data, the site provides a set of prefetching hints to the browser, and after the browser is finished loading the page, it begins silently prefetching specified documents and stores them in its cache.

When the user visits a prefetched document, it can be served up quickly out of the browser's cache.

Source: Mozzila .org

User Interface Issues Affecting Data? How a publisher system is designed could affect reported usage For example: If you have to visit the HTML page to get the PDF Source: Price & Davis, JASIST, 2006 arxiv.org/pdf/cs/0602060

How a publisher system is designed could affect reported usage

For example:

If you have to visit the HTML page to get the PDF

Source: Price & Davis, JASIST, 2006 arxiv.org/pdf/cs/0602060

Metasearch affecting data? Metasearch engines conduct multiple searches simultaneously Retrieve, consolidate and ranks results based on algorithmic and semantic analysis Provide users with resource selections However, in many cases this search and retrieve results in hits and downloads

Metasearch engines conduct multiple searches simultaneously

Retrieve, consolidate and ranks results based on algorithmic and semantic analysis

Provide users with resource selections

However, in many cases this search and retrieve results in hits and downloads

Applying Usage Data Data - All facts Information - Facts within context Knowledge - Interrelationships among relevant facts Wisdom - Actionable knowledge

Basic Measures Cost-per use - Are we getting comparative value from this resource? Are my systems working? Are there barriers to use - why are similar products experiencing different use patterns? Expressing value to administration, contributors or government sponsors

Cost-per use - Are we getting comparative value from this resource?

Are my systems working?

Are there barriers to use - why are similar products experiencing different use patterns?

Expressing value to administration, contributors or government sponsors

Using Usage as a Quality Measure The amount of traffic an item receives is separate, but valuable, metric for assessing quality Citation measures capture only one type of use - scholarly citation, not necessarily quality Teaching or clinical use is extremely valuable

The amount of traffic an item receives is separate, but valuable, metric for assessing quality

Citation measures capture only one type of use - scholarly citation, not necessarily quality

Teaching or clinical use is extremely valuable

Comparing usage measures Development project underway within COUNTER in partnership with UKSG Goal: Derive a meaningful calculation for assessing quality through COUNTER data Usage of items / Period of time Questions primarily related to the denominator used in calculation

Development project underway within COUNTER in partnership with UKSG

Goal: Derive a meaningful calculation for assessing quality through COUNTER data

Usage of items / Period of time

Questions primarily related to the denominator used in calculation

Two methods for assessing quality Impact Factor Established, understood and generally accepted Funding agencies, researchers rely on its data Limitation in the fields of scholarship it covers Reflects value of journals to researchers, but not all users Over-emphasis on IF distorts the behaviour of authors Over-used, mis-used and over-interpreted Usage Factor Usage-based alternative perspective Would cover all online journals Would reflect value of journals to all categories of user Would be easy to understood

Impact Factor

Established, understood and generally accepted

Funding agencies, researchers rely on its data

Limitation in the fields of scholarship it covers

Reflects value of journals to researchers, but not all users

Over-emphasis on IF distorts the behaviour of authors

Over-used, mis-used and over-interpreted

Usage Factor

Usage-based alternative perspective

Would cover all online journals

Would reflect value of journals to all categories of user

Would be easy to understood

What’s in store in the future MESUR - MEtrics from Scholarly Usage of Resources Mellon funded project to study assessment of the impact of scholarly communication items, and hence of scholars, with metrics that derive from usage data Data analysis imaese from Johan Bollen

MESUR - MEtrics from Scholarly Usage of Resources

Mellon funded project to study assessment of the impact of scholarly communication items, and hence of scholars, with metrics that derive from usage data

MESUR - Some Examples Discerning methods of citation networks Describing journal usage comparisons Describing potential connectedness measures Relatedness of items based on use patterns “ Readers who viewed this also views…” Image source: www.mesur.org

Discerning methods of citation networks

Describing journal usage comparisons

Describing potential connectedness measures

Relatedness of items based on use patterns

“ Readers who viewed this also views…”

Privacy Concerns If you have enough data, you can pinpoint exact people Say you have domain expertise You see person X looks at this article, then that article, and onto this series of articles You’ll probably be able to figure who the person is and what they’re working on

If you have enough data, you can pinpoint exact people

Say you have domain expertise

You see person X looks at this article, then that article, and onto this series of articles

You’ll probably be able to figure who the person is and what they’re working on

Other Areas for Development Non-Journal content, A&I, bibliographies Content on multiple platforms Different versions existing on the network Institutional repository systems Limited usage tracking and consistent reporting Multimedia content, streaming content Mash-ups and multi-feed content Research data and visualization tools

Non-Journal content, A&I, bibliographies

Content on multiple platforms

Different versions existing on the network

Institutional repository systems

Limited usage tracking and consistent reporting

Multimedia content, streaming content

Mash-ups and multi-feed content

Research data and visualization tools

Thank you! Todd Carpenter, Managing Director [email_address] One North Charles Street Suite 1905 Baltimore, MD 21201 USA (301) 654-2512 (410) 685-5278 www.niso.org

Todd Carpenter, Managing Director

[email_address]

One North Charles Street

Suite 1905

Baltimore, MD 21201 USA

(301) 654-2512

(410) 685-5278

www.niso.org