IOTA @ NASIG 2011: Measuring the Quality of OpenURL Links

NISO’s IOTA Initiative Measuring the Quality of OpenURL Links NASIG Annual Conference St. Louis, MO June 2 – 5, 2011 Rafal Kasprowski, Rice University

Agenda
In the Beginning: Full-text linking and Advent of OpenURL

IOTA: Created in response to OpenURL linking problems

IOTA’s analytical approach

Community-derived reports comparing quality of vendor OpenURLs

Concept of the OpenURL Quality Index
IOTA & KBART: relationship & joint initiative

Community involvement in IOTA: necessary for best outcomes

Before OpenURL: Proprietary Linking

Certain A&I database providers (e.g., CSA, PubMed) offered full-text linking option for a select number of content providers.

Libraries manually activated full-text linking with providers they had subscriptions with.

A&I --> Full Text

Proprietary Linking: Pros and Cons
Linking had to be activated manually by libraries for each full-text provider.

A&I providers offering this option were few.

Selection of full-text providers was limited.

But...
Once set up, the static links to full texts were accurate.
Problem source pinpointed easily: A&I --> Full Text

Advent of OpenURL
Objective: Deliver full texts unrestrained by proprietary silos.

Open standard generating dynamic links at time of request.

A-Z list (e.g., e-journal, e-books):
Knowledge base (KB) with library's holdings.
Replaces librarian as intermediary in linking.
Indicates provider of "appropriate copy"

A&I ("Source") --> A-Z list ("KB") --> Full Text ("Target")

A, Bernand , et al. " A versatile nanotechnology to connect individual nano-objects for the fabrication of hybrid single-electron devices. " Nanotechnology 21 , no. 44 ( November 5, 2010 ): 445201 . Academic Search Complete , EBSCO host (accessed October 24, 2010).
OpenURL: resolver, syntax, linking nodes http://ps4ps6lm2r.search.serialssolutions.com/?issn=0957-4484&volume= 21 &issue= 44 &date= 20101105 &spage= 445201 &title= Nanotechnology &atitle= A+ versatile+nanotechnology+to+connect+individual+nano-objects+for+the+ fabrication+of+hybrid+single-electron+devices. &aulast= A++Bernand Source Citation Target OpenURL (Source OpenURL structured similarly)

Pros & Cons of OpenURL
Pros:
KB/Resolver vendors took over most of the linking setup: Less work for libraries and providers.
Participation by A&I platforms and full-text providers exceeded proprietary linking: OpenURL scales better.

Cons:
Dynamic linking less predictable than static linking: more difficult to pinpoint cause of link failures.
OpenURL linking not improved significantly last 10 years.
No systematic method exists to benchmark OpenURLs.

Identifying source of problem…
"72% of respondents to the online survey either agreed or strongly agreed that a significant problem for link resolvers is the generation of incomplete or inaccurate OpenURLs by databases (for example, A&I products)."
Culling, James. 2007. Link Resolvers and the Serials Supply Chain: Final Project Report for UKSG, p.33. http://www.uksg.org/sites/uksg.org/files/uksg_link_resolvers_final_report.pdf.

Defining methodology for addressing problem…

Recently, researchers have indicated the need for metadata quality metrics , including:
completeness;
accuracy;
conformance to expectations;
logical consistency and coherence.
Bruce, Thomas R. and Hillmann, Diane I. 2004. The Continuum of Metadata Quality: Defining, Expressing, Exploiting. In Metadata in Practice . Ed. Diane I. Hillmann and Elaine L. Westbrooks. Chicago: American Library Association, pp. 238-256.

Année philologique OpenURL Study
2008 Cornell study led by Adam Chandler*
Problem: Too often links sent from Aph did not successfully resolve to requested resource.
Objective: Examine quality of OpenURLs offered to users by Aph in order to improve the linking.

Aph Study investigated:
Faulty citation metadata from source database.
Method to evaluate the OpenURLs.
*Chandler, Adam. 2009. Results of L’Année philologique online OpenURL Quality Investigation: Mellon Planning Grant Final Report. http://metadata.library.cornell.edu/oq/files/200902%20lannee-mellonreport-openurlquality-final.pdf.

Scoring System & Aph Study Outcomes
Concept of scoring in Aph study (based on B. Hughes study)*
establish a baseline for comparison;
results to be shared with data providers;
develop a best practice.

Problem analysis in Aph study limited to:
source link
presence/absence of citation metadata elements

Results:
OpenURL quality model: compares elements in Aph OpenURLs to those of other providers.
No scoring was achieved for Aph , but model is first step towards scoring system.
*Hughes, Baden. 2004. Metadata Quality Evaluation: Experience from the Open Language Archives Community. In Digital Libraries: International Collaboration and Cross-Fertilization. Ed. Zhaoneng Chen et al. Berlin: Springer-Verlag, 2004, pp. 320-329.

Creation of IOTA
NISO:
Accepts proposal to take Aph Study to wider community.
New OpenURL quality metrics initiative formed in Jan. 2011.
Branded as: I mproving O penURLs T hrough A nalytics.

Basic Assumptions:
Results are achieved through an analytical investigation of how OpenURL links work.
Practical Goal: Not the OpenURL standard is addressed, but the links (OpenURLs) generated by standard.
Selective changes to OpenURLs will lead to significant improvement in linking success rate.
Motto: "small changes. big improvements"

IOTA Desired Outcomes
Produce qualitative reports that will help OpenURL providers quickly compare their OpenURL quality to that of their peers.
Develop community-recognized index for measuring the quality of OpenURL links generated by content providers.

Method:
fair;
transparent;
scalable across all OpenURLs and their providers.

How is comparing OpenURLs useful?
Content providers generating OpenURLs can:
compare their OpenURLs with other providers;
make improvements to their OpenURLs.
Institutions can:
compare OpenURL providers;
make local adjustments to OpenURL setup.
Link resolver vendors can:
compare OpenURL providers;
change their OpenURL provider settings:
Link resolvers;
Web-scale discovery products.

Report types
Source reports
Viewing how a particular (1) vendor or (2) database
A. uses OpenURL elements (element frequency)
B. formats OpenURL elements (pattern frequency)
Element / Pattern reports
Viewing how a particular (1) element or format
A. is used across vendors
B. is used across databases

Running reports

Reports: Log file providers

Report: source (vendor or database)

Report: source = vendor

Report: element, source = vendor

Report: pattern, source = vendor

Report: source = database

Report: element, source = database

Report: pattern, source = database

Report: element and pattern frequency

Report: element & pattern frequency: Vendor option

Report: element/pattern frequency: Choosing Metric

Report: element/pattern by vendor

Report: element & pattern frequency: Database option

Report: element/pattern by database

Reporting System: improvements underway
Consolidating variant instances of databases and vendors if the same;
Separating article-like requests from book-like requests
Either/Or situation: most resources do not offer both formats
Once separation is completed, users will be given corresponding options to select OpenURL data by format: ARTICLE or BOOK
These improvements will also benefit the accuracy of the OpenURL scoring system.

OpenURL Quality Index: initial version
1. Core elements:
Any element contained in IOTA's OpenURL reporting system;
13M OpenURLs already obtained from libraries content providers.

2. Scoring system based on assumption :
Correlation exists between
# of core elements ("OpenURL completeness") &
ability of OpenURLs to link to specific content.

3. Weighting assigned to core elements:
Based on relative importance
spage vs atitle
issn vs jtitle
doi/pmid vs date, etc.

OpenURL Quality Index: vendor rating

Work in Progress
Element weighting still in progress:
E.g., importance of identifiers (doi, pmid) vs bibliographic data (issn, volume, spage).

Currently, IOTA focuses on OpenURLs from citation sources only. OpenURL quality is also influenced by:
knowledge base,
resolver,
full-text provider (target).

High "completeness" score of OpenURLs not always indicative of "success" in linking to full texts
Combination of multiple indexes along linking nodes may provide more complete picture.

IOTA & KBART: NISO working groups
IOTA
Deals with issues specific to OpenURL linking;
Seeks improvements in OpenURL elements used by:
OpenURL providers.
KBART
“ Knowledge Bases And Related Tools”
Deals with data issues at the KB level
Seeks improvements in data exchange practices between:
content providers (e.g. OpenURL providers);
product vendors (e.g. link resolver vendors);
subscription agents.

IOTA & KBART: related through OpenURL
IOTA node:
analyzing data sent from OpenURL source to link resolver.
KBART node:
creating formatting best practices for data sent from content providers to knowledge base (and link resolver) vendors.

KBART-IOTA joint initiative
KBART-IOTA node :
Exploring together the third source of failures:
link-to (or target) syntax and behavior which couples link resolvers to content providers
Collaboration begun in March 2011 is meant to address OpenURL quality in a broader context.

How can I get involved?
If you are an OpenURL provider:
Contribute data to IOTA
Review the IOTA data
This data is meant to help you make improvements in your OpenURL linking

If you are a librarian:
Contribute data to IOTA
Spread the word to vendors about IOTA
OpenURL data contact:
Adam Chandler, alc28@cornell.edu

IOTA Web Presence

Questions?
http://openurlquality.niso.org
http://www.niso.org/workrooms/openurlquality
@nisoiota on twitter
Rafal Kasprowski Electronic Resources Librarian Rice University, Fondren Library MS 44 Houston, TX 77005 [email_address] http://www.slideshare.net/rkaspro

IOTA @ NASIG 2011: Measuring the Quality of OpenURL Links

by Rafal Kasprowski on Jun 11, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

IOTA @ NASIG 2011: Measuring the Quality of OpenURL Links Presentation Transcript