This section of the presentation on IOTA will focus on the Completeness Index, its validation and optimization.
We will start with an overview of the premise behind the completeness scores and completeness index and the corresponding element weights. Next we will review how we went about validating the concept, followed by the technique used to optimize the element weights. And finally we will look at the results of this work, our conclusions and the next steps for IOTA
Lets start with a couple definitions…The Completeness Score is the measure of the completeness of a single OpenURL. And by completeness we mean the number of metadata elements provided in the OpenURL out of a desired or core number.The Completeness Index is a number the is attributed to a content provider (OpenURL “referrer”) to measure the completeness of their OpenURLs in aggregate. It is essentially an average of the completeness scores of OpenURLs coming from that content provider.
The Completeness Score is a number between 0 and 1 that measures the “completeness” of the elements in the OpenURL, with 0 indicating no elements provided and 1 indicating all desired elements were there – a perfect OpenURL.It is a weighted score with some elements carrying more weight than others –the element’s weight is based on its importance in target links.<CLICK>When you look at the syntaxes for target links, you will see some elements like ISSN, Volume and Issue appearing more often than elements like article title or author. Common sense tells us that, if an OpenURL is missing an that is needed in many Target Links its probability of failure is much greater than if it was missing an element needed by only a few targets.<CLICK>So each element is assigned a value based on its “importance”…we call this the element weight. Calculating the Completeness Score is done by taking a sum of the “weights” for each core element found in the OpenURL and dividing that sum by the maximum possible score. Basically comparing the data in a OpenURL to a “perfect OpenURL”
Lets run through a quick example. This table shows the core elements for an article link… and for the simplicity of this example we will assume all elements are equally important so each gets a weight of 1 – a perfect OpenURL will get the maximum score of 8.
Now lets look at some OpenURL elements…. In this OpenURL we have…<CLICK>Date … so we add one point<CLICK>ISSN… add another point<CLICK>Volume… another point<CLICK>ISSUE… another<CLICK>And Article Title… and another point<CLICK>… the result is a total of 5 points.<CLICK>The calculation is Sum of the weights for this OpenURL divided by the total for all weights<CLICK>Which is five divided by 8<CLICK>Or .625
Since not all elements are equally important, giving each a weight of 1 is not reasonable. So how did we come up with the initial set of weights? We looked at the link syntaxes for over 300 target link and did an occurrence count of each of the core element and used that to influence the weights. Then tweaked based on what looked about right.
Here is what we came up with…
Individual elements got weights from 1 to 5 with identifiers getting a higher score of 8.
A couple of comments… Most link resolvers will handle either or both print and online ISSN – if you have one, the resolver will look up both -- so score each separately isn’t a reflection of reality.
Also, identifiers, like PubMed ID and DOI are used by most link resolvers to look-up the full citation in PubMed or CrossRef –thus if you have one of these identifiers in an OpenURL you really have access to all the core data elements. Giving an OpenURL with just a DOI fewer points than an OpenURL with DOI and other elements isn’t really valid since at the end of the day, the link resolver data enhancement will make them equal (in fact the DOI-only one might work better if there were any conflicts in metadata values.)
But that is what we started with and we needed to see if we could statistically justify these numbers. So we ran some tests…Aron Wolf from Serials Solution and I each ran a series of tests using the same OpenURLs on our respective link resolvers. The first thing we did was eliminate library holdings as a variable by using the entire knowledge base as our “collection”. Remember we were testing whether there areenough of the right data elements to find full text, not if the library had a subscription.<CLICK>We then took a sample of some 15,000 OpenURLs – picked at random. We ran each through our link resolvers to see if the OpenURL would populate a full text target. We gave the OpenURL a “success score” of 1 for finding a full text target and “0” for not.<CLICK> The theory is that if the completeness of the OpenURL affects its success, which means there should be a relationship between “success” and the “completeness score”… so we also calculated the “completeness score” for each OpenURL.<CLICK>Then we looked for a statistical correlation between the two.
After running the numbers this is what come out of the LinkSource test… The graph shows average Completeness Score and average Success score by “referrer” – (OpenURL source). You can see some tracking but not overwhelming.When we calculated the Correlation Coefficient between these two scores across all 15,000 OpenURLs in the test, the result was .43. Which indicates some level or relationship but, again, not particularly strong.
We needed a better way of determining the element weights, so we sought help from Phil Davis – a researcher with some experience in statistical modeling. Phil’s suggestion was to perform stepwise regression to see the effect of individual elements on a sample of OpenURLs. And that is what we did…We started with a set of “perfect” OpenURLs – ones that not only included all core data elements, but that also resolved to match a full text target on both LinkSource and 360 Link… we used a set of 1500.<CLICK>We then ran several series of tests where we ran the OpenURL past the link resolver with a different element removed for each test series.<CLICK>We recorded the success (or rather failure rates) associated with each element. The elements with the higher failure rates are more important to the success of the OpenURL than the ones with lower failure rates.<CLICK>We then used the failure rates as a basis for weights.<CLICK>Then we used the weights and re-ran our 15,000 sample test.
So how’d it turn out? Again, here are numbers for LinkSource.<Click>You can see Volume was a key element with 74% of OpenURLs failing when it was removed.<Click>Author last name was not very important with less than a 10th of a percent failure rate<Click>Date was surprising low too. This could be for a few reasons – the level of forgiveness in the holdings matching logic (e.g. treat no date as “any date”), the ability for the link resolver to discover the date by looking up the article citation in the knowledge base using volume/issue/start page coupled with the fact that a lot of full text providers don’t use date explicitly in the outbound links.
We created article weights. <Click>Rather than use raw failure rates, we used logarithmic values of the failure rates – the number of failures per 10,000.
Then we ran our 15,000 record sample again. You can see from the graph that average completeness score and average success score for the OpenURL providers align very closely, and the Correlation Coefficient of these two values across all 15,000 test OpenURLs is .80 – which indicates a strong correlation. Good news for the test.This tells us that the Completeness Index can be used as a predictor of OpenURL success from a particular content provider – a low Completeness Index is a good indicator there is a problem.
As I mentioned, similar test were conducted by Aron Wolf using 360 Link. The same OpenURLs were used and Aron was able to validate the concept of the Completeness Score; however, the failure rates were not exactly the same as for the LinkSource test; therefore, Aron’s weights were not the same… So why the difference?Some possible explanations would be the variations in metadata enhancement -- the ability to “correct” weaknesses in an incoming OpenURL.Strictness in the target link rules. For example, some targets will display just the journal home page if insufficient metadata is available – in real life this is probably better than no link because the user can navigate to the full text from there. For the LinkSource test, for example, we had removed a lot of the forgiveness because we wanted “success” as getting directly to the full text.And link syntaxes is another possible variable. For example, JSTOR has an OpenURL-based syntax and another based on SICI code. The data you need to each is different therefore depending on the syntax used, the weights for corresponding elements would be different.
96.3 percent of the OpenURLs in the LinkSource test were successful in populating either a full text target or an ILL form (with a sufficient data that the ILL department could complete the request). This seems pretty good, so why is OpenURL viewed as problematic?<CLICK>Part of it is perception -- what constitutes success in the eyes of a link resolver may not constitute success in the eyes of an end user. Remember our tests were done where holdings were not a factor (as if our test library subscribed to everything in the world). If you add specific library holdings into the mix, then more of the links will go to ILL forms –many end user consider a link to an ILL form a broken link – this is the perception problem.<CLICK>A couple thoughts to combat this… if your discovery provider systems (the source of the OpenURL) allow, use their holdings features to change the wording of the OpenURL link. Use “Link to full text” if item is from a journal in the library’s online collection; use, “check library collection”, if it is the print collection; or, use “request from library” for everything else. This way the end user’s expectations are controlled and the perception of the technology improved.
Our conclusions…The step-wise approach to determining element weights works. The completeness index scores do correlate to OpenURL success.However, Knowledge Base contents and resolver technology influence the results and thus prevent a universal set of weights and scores.<Click>The Completeness Index is a mechanism individual link resolver vendors can use to provide metrics to help improve their service quality by identifying those OpenURL sources that are more problematic.
Other take-aways…(as indicated on the screen)
And what’s next for IOTA is…(review what is on the screen)
NISO's IOTA OpenURL Quality Initiative @ ALA & SLA 2012
NISOS IOTA Initiative:Fixing OpenURL Links Using Data Analysis Improving OpenURLs Through Analytics Special Libraries Association, Annual Conference, Chicago, IL, July 17, 2012 American Library Association, Annual Meeting, Anaheim, CA, June 24, 2012 Rafal Kasprowski, Electronic Resources Librarian, Rice University Susan Marcin, Licensed Electronic Resources Librarian, Columbia University Oliver Pesch, Chief Strategist, EBSCO Information Services
HISTORY OF OPENURL / OVERVIEW OF IOTA Rafal Kasprowski Rice University email@example.com
What is IOTA?• Initiative that measures the relative importance of the elements that make up OpenURL links to help vendors improve their OpenURL strings so that the maximum number of OpenURL requests resolve to a correct record.Elements:• journal title book title ISBN• ISSN start page DOI• volume author PMID• issue date …
Presentation• Part I (Rafal Kasprowski, Electronic Resources Librarian, Rice University) • History of OpenURL and IOTA • IOTA’s Deliverables: • OpenURL Reports (comparing vendors’ OpenURL links) • OpenURL Quality Index (preliminary version)• Part II (Susan Marcin, Licensed Electronic Resources Librarian, Columbia University) • Usefulness of IOTA’s OpenURL Reports in improving OpenURL links for e-books• Part III (Oliver Pesch, Chief Strategist, EBSCO Information Services) • Improvements to IOTA’s OpenURL Quality Index and its limitations
Part I - Agenda• Full-text linking: from proprietary linking to OpenURL• From Cornell to NISO: IOTA created in response to OpenURL linking problems• IOTA in the context of ERM best practices: IOTA and KBART• IOTA’s analytical approach• Reports comparing vendor OpenURLs• Weighting OpenURL elements• Concept of the OpenURL Quality Index and preliminary version.
Before OpenURL: Proprietary Linking• Certain A&I database providers (e.g., CSA, PubMed) offered full-text linking options for a select number of content providers.• Libraries manually activated full-text linking with providers they had subscriptions with.• A&I --> Full Text
Proprietary Linking: Pros and Cons• Linking had to be activated manually by libraries for each full-text provider.• A&I providers offering this option were few.• Selection of full-text providers was limited.But...• Once set up, the static links to full texts were accurate.• Problem source pinpointed easily: A&I --> Full Text
Advent of OpenURL• Objective: Deliver full texts unrestrained by proprietary silos.• Open standard generating dynamic links at time of request.• A-Z list (e.g., e-journals, e-books): o Knowledge base (KB) with librarys holdings. o Replaces librarian as intermediary in linking. o Indicates provider of "appropriate copy"• A&I ("Source") --> A-Z list ("KB") --> Full Text ("Target")
OpenURL: syntax, resolver, linking nodesSource CitationA, Bernand, et al. "A versatile nanotechnology to connect individual nano-objectsfor the fabrication of hybrid single-electron devices." Nanotechnology 21, no. 44(November 5, 2010): 445201. Academic Search Complete, EBSCOhost (accessedOctober 24, 2010).Target Link (example using OpenURL syntax, similar to Source OpenURL)http://www.anytarget.com/?issn=0957-4484&volume=21&issue=44&date=20101105&spage=445201&title=Nanotechnology&atitle=A+versatile+nanotechnology+to+connect+individual+nano-objects+for+the+ fabrication+of+hybrid+single-electron+devices.&aulast=A++Bernand
Example of Resolver Menu PageMatthew Reidsma, “jQuery for Customizing Hosted Library Services", http://matthew.reidsrow.com/articles/11 (accessed July 23, 2012)
Pros & Cons of OpenURLPros:• KB/Resolver vendors took over most of the linking setup: Less work for libraries and providers.• Participation by A&I platforms and full-text providers exceeded proprietary linking: OpenURL scales better.Cons:• Dynamic linking less predictable than static linking: more difficult to pinpoint cause of link failures.• OpenURL linking not improved significantly in last 10 years.• No systematic method exists to benchmark OpenURLs.
Identifying source of problem…“72% of respondents to the online survey either agreed or stronglyagreed that a significant problem for link resolvers is the generationof incomplete or inaccurate OpenURLs by databases (for example,A&I products).”Culling, James. 2007. Link Resolvers and the Serials Supply Chain: Final Project Report for UKSG, p.33.http://www.uksg.org/sites/uksg.org/files/uksg_link_resolvers_final_report.pdf.Defining methodology for approaching problemRecently, researchers have indicated the need for metadata qualitymetrics, including: • completeness; • accuracy; • conformance to expectations; • logical consistency and coherence.Bruce, Thomas R. and Hillmann, Diane I. 2004. The Continuum of Metadata Quality: Defining, Expressing, Exploiting. In Metadata inPractice. Ed. Diane I. Hillmann and Elaine L. Westbrooks. Chicago: American Library Association, pp. 238-256.
Année philologique OpenURL Study 2008 Cornell study led by Adam Chandler* • Problem: Too often links sent from Aph did not successfully resolve to requested resource. • Objective: Examine quality of OpenURLs offered to users by Aph in order to improve the linking. Aph Study investigated: • Faulty citation metadata from source database. • Method to evaluate the OpenURLs.*Chandler, Adam. 2009. Results of L’Année philologique online OpenURL Quality Investigation: Mellon Planning Grant Final Report.http://metadata.library.cornell.edu/oq/files/200902%20lannee-mellonreport-openurlquality-final.pdf.
Scoring System & Aph Study OutcomesConcept of scoring in Aph study (based on B. Hughes study)* • establish a baseline for comparison; • results to be shared with data providers; • develop a best practice.Problem analysis in Aph study limited to: • source link • presence/absence of citation metadata elementsResults: • OpenURL quality model: compares elements in Aph OpenURLs to those of other providers. • No scoring was achieved for Aph, but model is first step towards scoring system.*Hughes, Baden. 2004. Metadata Quality Evaluation: Experience from the Open Language Archives Community. In Digital Libraries:International Collaboration and Cross-Fertilization. Ed. Zhaoneng Chen et al. Berlin: Springer-Verlag, 2004, pp. 320-329.
Creation of IOTA IOTA is formed in January 2010• NISO accepts proposal to take Aph Study to wider community.• URL: openurlquality.org
IOTA & KBART: complementary NISO working groupsIOTA• Deals with issues specific to OpenURL linking;• Seeks improvements in OpenURL elements used by: • OpenURL providers.KBART• “Knowledge Bases And Related Tools”• Deals with data issues at the KB level• Seeks improvements in data exchange practices between: • content providers (e.g. OpenURL providers); • product vendors (e.g. link resolver vendors). • subscription agents;
IOTA & KBART: related through OpenURL• IOTA: • analyzing data sent from OpenURL source to link resolver.• KBART: • creating best practices for data formats sent from content providers to knowledge base (and link resolver) vendors.
IOTA’s Basic Assumptions• Results are achieved through an analytical investigation of how OpenURL links work.• Practical: Not the OpenURL standard is addressed, but links (OpenURLs) generated by standard.• Selective changes to OpenURLs will lead to significant improvement in linking success rate. o Motto: "small changes. big improvements"
IOTA’s Desired Outcomes……a continuation of Aph StudyA. Produce qualitative reports that will help OpenURL providers quickly compare their OpenURL quality to that of their peers.B. Develop community-recognized index for measuring the quality of OpenURL links generated by content providers: scalable across all OpenURLs and their providers
Usefulness of comparing OpenURLs• Content providers that generate OpenURLs can: • compare their OpenURLs with other providers; • make improvements to their OpenURLs.• Institutions can: • compare OpenURL providers; • make local adjustments to OpenURL setup.• Resolver vendors can: • compare OpenURL providers; • Change their settings for OpenURL providers: • Link resolvers; • Web-scale discovery products.
Report types• Source reports • Viewing how a particular (1) vendor or (2) database • A. uses OpenURL elements (element frequency) • B. formats OpenURL elements (pattern frequency)• Element / Pattern reports • Viewing how a particular (1) element or format • A. is used across vendors • B. is used across databases• Vendor Completeness Report • Viewing vendors’ OpenURL quality score
OpenURL Quality Index:Rating vendors by their OpenURLs1. Core Elements: • Any element contained in IOTAs OpenURL reporting system; • 20M OpenURLs obtained from libraries & content providers.2. Scoring System: • Assumption: Correlation exists between o # of core elements ("OpenURL completeness") & o ability of OpenURLs to link to specific content.3. Element Weighting: • Assigned based on their relative importance: o spage vs atitle o issn vs jtitle o doi/pmid vs date, etc.
Further investigation was needed• Element weighting needed to be adjusted in a more systematic way: o Importance of identifiers (doi, pmid) vs bibliographic data (issn, volume, spage, etc.) o Relative importance of bib. data (issn vs volume vs spage, etc.)• IOTA focused on OpenURLs from citation sources only. How is OpenURL linking impacted by other factors? o knowledge base, o resolver, o full-text provider (target).• High "completeness" score of OpenURLs not always indicative of "success" in linking to full texts o Combination of indexes, incl. “success index”, developed by IOTA and/or other groups may lead to more precise metrics.
Presentation: Parts II and III• Part II (Susan Marcin, Columbia University) • Usefulness of IOTA’s OpenURL Reports in improving OpenURL links for e-books• Part III (Oliver Pesch, EBSCO Information Services) • Improvements to IOTA’s OpenURL Quality Index and its limitations
E-Books & OpenURL Linking A collaborative study by the 2CUL E-Books Task Force Susan Marcin Columbia University Libraries firstname.lastname@example.org
What is 2CUL?2CUL is a transformative partnership between two majoracademic research libraries, the Columbia UniversityLibraries and the Cornell University Library, based on abroad integration of resources, collections, services, andexpertise. http://2cul.org/
2CUL E-Books Task ForceE-books represent a large, diverse, and rapidly growing groupof library materials whose acquisition, description,management, and use touch many parts of our libraries’organization. Moreover, models affecting virtually every aspectof e-books are still evolving, leaving a host of issues in flux,with many options and no perfect solutions.• Survey e-book landscape in more detail• Recommend steps that our libraries should take in the short term to improve e-book access and management• Make recommendations for action
E-Books Linking Group• Examine, evaluate and compare the quality of E-Book OpenURLs• Focus on what works, what doesn’t and whyGroup members:• Adam Chandler, E-Resources & Database Management Research Librarian, Cornell University• Susan Marcin, Licensed Electronic Resources Librarian, Columbia University
OpenURL :: NISO Standard Z39.88URL strings generated “on the fly.”The OpenURL path from citation to full text consists of databeing generated and passed through the following places: 1. An OpenURL is sent from the citation resource to the OpenURL resolver 2. The data is matched against a knowledge base to generate content on the OpenURL resolver page 3. A second proprietary URL is sent from the resolver page to the full text
What the OpenURL string looks like (invisible to user)http://rd8hp6du2b.search.serialssolutions.com/?ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info:sid/summon.serialssolutions.com&rft_val_fmt=info:ofi/fmt:kev:mtx:dc&rft.title=Apple+Developer+Programs&rft.date=2010-01-01&rft.pub=Apress&rft.isbn=1430229314&rft.spage=179&rft.epage=190&rft_id=info:doi/10.1007%2F978-1-4302-2932-2_9&rft.externalDBID=n%2Fa&rft.externalDocID=978-1-4302-2932-2_209124_Chap9link
Why does improved OpenURL linkingmatter?Users• Better user experience: Users expect the provision of easy connection between library resources• Increased patron satisfaction with library e-resourcesLibrary• Enhanced discovery of and cross-linking into subscription e-resources
How did the task force assess the quality of E-BookOpenURLs?• Looked at NISO IOTA data (IOTA = Improving OpenURL through Analytics)• IOTA tracks the OpenURLs sent from citations to link resolvers http://openurlquality.niso.org/
Question: How many patron requests fore-books resulted in full text?To find the answer --Three weeks of Columbia Us openURL log data on the NISOservers analyzed: • one week from January 2011 • one week from February 2011 • one week from March 201178,540 total openURL requests in these 3 weeksHow many of these OpenURLs are for e-books?• 1474 Requests that contained an ISBN• 781 Requests that lacked an ISBN but contained genre=bookWe analyzed 2255 e-book OpenURLs.
ISBNs in OpenURL Strings – Do theymatter? 1474 OpenURL Requests contained an ISBN Yes NoFull text link offered? 47% (698/1474) 47% (698/1474) "Get Book" link Successful? Yes (book link No (book link N/A (links to journal, rather leads to full-text fails) than book or cannot successfully) process information)"Get Book" 94.9% (663/698) 1.1% (8/698) 1.1% (8/698)linkSuccessful?
Including "Genre=Book" in OpenURLStringsLacks ISBN, but found full text anyway? Yes No Found full text? 28% (208/781) 73% (573/781)
OpenURL linking failure :: "Bad" Data • Correctly identified as "genre=book," but article title "atitle" passed in openURL string.http://rd8hp6du2b.search.serialssolutions.com/?SS_Page=refiner&sid=sersol%3ARefinerQuery&rft.aulast=Mihailovic&url_ver=Z39.88-2004&l=RD8HP6DU2B&SS_ReferentFormat=BookFormat&rft.genre=book&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.atitle=corporeal+words&citationsubmit=Search&SS_LibHash=RD8HP6DU2B&rfr_id=info%3Asid%2Fsersol%3ARefinerQuery&rft.aufirst=Alexandar&SS_Errors=RequiredDataMissinglink
A better OpenURL might look like thishttp://rd8hp6du2b.search.serialssolutions.com/?rft.au=Mihailovic%2C+Alexandar&sid=sersol%3ARefinerQuery&SS_authors=Mihailovic%2C+Alexandar&rft.aulast=Mihailovic&url_ver=Z39.88-2004&l=RD8HP6DU2B&SS_ReferentFormat=BookFormat&rft.genre=book&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.title=Corporeal+Words&rft.isbn=0810114593&citationsubmit=Search¶mdict=en-US&rfr_id=info%3Asid%2Fsersol%3ARefinerQuery&SS_LibHash=RD8HP6DU2B&SS_isbnh=0-8101-1459-3&rft.aufirst=Alexandarlink
IOTA Metric Report: ISBN Vendor Percentage of Total number ofDrawing from our OpenURL OpenURLsobservations requests that analyzed contain thisabout the elementimportance of summon.serialssolutions.com 98 17575ISBN for full text ebsco 82 27023linking, we pqil 82 1170analyzed the csa 79 12952IOTA metric hww 75 1428"ISBN" to refworks 68 1897highlight the firstsearch.oclc.org 65 10309differences unknown 63 3662across a sample sersol 30 4140of OpenURL oup 7 2071providers.
Detailed Comparison of Summon vs. OUP metric summon.serialso oupThe lutions.com (percentage ofdifferences in (percentage of OpenURLsthe presence OpenURLs containing thisof the ISBN containing this element)element are element)very important aulast 0 90in connecting date 100 96the user to thefull text. doi 11 12 isbn 98 7 title 100 99
Criteria for improved OpenURL linkingSending only a title in an OpenURL string is not alwayssufficient to find a match. If there is an ISBN associated with abook, then the OpenURL provider should ideally include theISBN in the OpenURL request for better results.The inclusion of certain criteria do appear to promote OpenURLlinking success, such as: • ISBN • DOI • TitleIf one includes additional data, such as an author last name,with the title, the precision of the results should improve.
Can we rank the OpenURL linkingsuccess of e-book providers?IOTA Vendor Rating OpenURLsCompleteness AnalyzedReport (Draft) summon.serialssolutions.co 0.64 17862 m pqil 0.61 2237metric weight csa 0.59 27483aulast 1 ebsco 0.57 50927date 2 hww 0.57 2448doi 8 refworks 0.55 3078isbn 7 firstsearch.oclc.org 0.54 23444 unknown 0.43 5079title 5 oup 0.4 3382 sersol 0.29 8227 gale 0.26 3019
Links & Contact• 2CUL -- http://2cul.org/• NISO IOTA -- http://openurlquality.niso.org/• 2011 2CUL E-Books Task Force: Linking Subgroup Report http://tinyurl.com/linkingreport Susan Marcin Columbia University Libraries email@example.com
NISOS IOTA INITIATIVE: COMPLETENESS INDEX AND IMPROVING ELEMENT WEIGHTS Oliver Pesch EBSCO Information Services firstname.lastname@example.org
Overview• Premise for IOTA completeness score and element weights• Proving the theory through real-life tests• Using statistical approach to determine weights• Test results• Conclusions• Next steps for IOTA
The premise behind IOTA• Completeness Score is the measure of the “completeness” of a single OpenURL• Completeness Index is attributed to the content provider as an overall measure of the completeness of their OpenURLs
The premise behind IOTA• The Completeness Score is calculated by “weighing” the elements provided in the OpenURL based on their importance in target links• Some elements are more important than others and will have a higher weight• Completeness Score equals the sum of weights of elements found divided by the maximum score possible
The premise behind IOTA• Simple example assuming equal element weights Element Description Weight This OpenURL ATitle Article title 1 AuLast Author’s last name 1 Date Date of publication 1 ISSN ISSN 1 Issue Issue number 1 SPage Start page 1 Title Journal Title 1 Volume Volume number 1 TOTAL 8
SAMPLE OPEN URL DATAThe premise behind IOTA ?date=2/4/2008 &issn=1083-3013 &volume=13• Simple example assuming equal element weights &issue=20 &atitle=the+casualties+of+war Completeness Score... Element Description Weight This OpenURL (Total for This OpenURL) Total Weights ATitle Article title 1 1 AuLast 5 / Author’s last name 8 1 Date Date of publication 1 1 ISSN = .625 ISSN 1 1 Issue Issue number 1 1 SPage Start page 1 Title Journal Title 1 Volume Volume number 1 1 TOTAL 8 5
Determining the weights• Initial approach • Frequency of element occurrence in target link templates • Combined with reasoning
Initial WeightsOpenURL data element Description WeightATitle Article title 1AuLast Author’s last name 1Date Date of publication 5eISSN Online ISSN 3ISSN Print ISSN 3Issue Issue number 3Jtitle Journal Title 1Pmid PubMed ID 8SPage Start page 3Title Journal Title 1Volume Volume number 3DOI Digital Object Identifier 8
Initial WeightsOpenURL data element Description WeightATitle Article title 1AuLast Author’s last name 1Date Date of publication 5eISSN Initial weights were Online ISSN 3somewhat subjective.ISSN Print ISSN 3Issue Issue number 3Jtitle Journal Title 1Pmid PubMed ID 8SPage Start page 3Title Journal Title 1Volume Volume number 3DOI Digital Object Identifier 8
Initial WeightsOpenURL data element Description WeightATitle Article title 1AuLast Author’s last name 1Date Date of publication 5eISSN Online ISSN 3ISSN Print ISSN 3Issue Issue number 3Jtitle Most link resolver Journal Title 1 knowledge bases canPmid PubMed ID 8handle look-ups by eitherSPage Start page 3Print ISSN or Online ISSNTitle Journal Title 1 (both are not needed)Volume Volume number 3DOI Digital Object Identifier 8
Initial Weights OpenURL data element Description Weight ATitle Article title 1 Most link resolvers will AuLast enhance identifiers like Author’s last name 1 PubMed ID and DOI; Date Date of publication 5 therefore, having an eISSN Online ISSN 3identifier is like having all ISSN Print ISSN 3 metadata elements. Issue Issue number 3 Jtitle Journal Title 1 Pmid PubMed ID 8 SPage Start page 3 Title Journal Title 1 Volume Volume number 3 DOI Digital Object Identifier 8
Validating the Completeness Score• Use real OpenURLs and a commercial link resolver. (tested with LinkSource and 360-Link) • Remove institutional holdings as a limit to resolution• Process each OpenURL through the link resolver to determine “Success” • Score one point for finding at least one full text target• Calculate the completeness score for each OpenURL• Look for a statistical correlation between the completeness score and the success score
Results: Original Weights 1.2000 1.0000 0.8000 Average of 0.6000 Completeness 0.4000 Score 0.2000 0.0000 Average of Success Score Correlation Coefficient .43Tests conducted on sample of 15,000 OpenURLs randomly pulled from IOTA database
A Statistical Approach to DeterminingElement Weights• Select a set of “perfect” OpenURLs • include all key data elements and resolve to full text• Perform step-wise regression • Test failure rates for each element by removing that element• Use failure rates as basis for weights• Use new weights to test for correlation between weights and success for larger sample
Failure Rates from 1500 OpenURL test sampleAuthor’s last name is least Element removed Description Failure Percentage importantOpenURL from the ATitle Article title .74% Date is AuLast surprisingly low Author’s last name .07% Date Date of publication .4% ISSN ISSN (either online or 22.02% print ISSN) Issue Issue number 20.27% SPage Volume is most critical Start page 33.27% Title Journal Title (either .61% Title or Jtitle) Volume Volume number 74.14%
Calculated Element Weights Element Description Weight* ATitle Article title 1.87 AuLast Author’s last name 0.83 Date Date of publication 1.61 ISSN ISSN (either online or 3.34 print ISSN) Issue Issue number 3.31 SPage Start page 3.52 Title Journal Title (either Title 1.78 or Jtitle) Volume Volume number 3.87 *Element weight calculation: log10 (failure-rate-per-10,000 OpenURLs)
Results: New Weights 1.2000 1.0000 Average of 0.8000 Completeness 0.6000 Score 0.4000 0.2000 Average of Success Score 0.0000 Correlation Coefficient .80Tests conducted on sample of 15,000 OpenURLs randomly pulled from IOTA database
NotesTesting the same OpenURLs on 360-Link results indifferent numbers but consistent trends. Differences maybe attributed to: • Variations in metadata enhancement techniques • Strictness in target link rules (e.g. required elements before link shows – tied to level of forgiveness of target) • Link syntax used for target
Notes96.3 of OpenURLs in the test were able to populate afull text target of credible ILL form… • Perception of high failure rate of OpenURL may be attributed to library holdings and user expectations • Suggestion: set link text to control expectations • Link to full text (for items in the online collection) • Check library collection (for things in print collection) • Request from library (for everything else)
Conclusions• Step-wise regression approach to element weights works• Completeness Index scores can be correlated to actual OpenURL “success”• KB and resolver technology influence results and prevent a universal set of element weights The Completeness Index is a mechanismindividual link resolver vendors can use to provide metrics to help improve their service quality
Other takeawaysSeveral factors involved in perceived “link failure”:1. Bad or missing metadata in the OpenURL link2. Inaccurate holdings data within the resolver’s knowledge base3. Flexibility of syntax to the target - e.g., target supports at least two: OpenURL syntax, DOI link, proprietary link structure4. Flexibility of resolution logic at the target - i.e., target finds way to create link using available data when some data missing or wrong5. User expectations - e. g., link resolver provided link to OPAC or ILL form, but user was expecting full text- IOTA focused on (1)- KBART working on (2)- Education of content providers could address (4)- Displaying OpenURL button only if full text available could address (5)
What’s next for IOTA• Continue offering public access to reports on element frequency• Publish technical report on work to date• Publish recommended practice for calculation and use of completeness scores for link quality assessment by link resolver vendors• Continue work as a NISO standing committee for at least one more year