Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IOTA @ NASIG 2011: Measuring the Quality of OpenURL Links


Published on

Published in: Technology, Education
  • Be the first to comment

IOTA @ NASIG 2011: Measuring the Quality of OpenURL Links

  1. 1. NISO’s IOTA Initiative Measuring the Quality of OpenURL Links NASIG Annual Conference St. Louis, MO June 2 – 5, 2011   Rafal Kasprowski, Rice University
  2. 2. Agenda <ul><ul><li>In the Beginning: Full-text linking and Advent of OpenURL </li></ul></ul><ul><li>  </li></ul><ul><ul><li>IOTA: Created in response to OpenURL linking problems </li></ul></ul><ul><li>  </li></ul><ul><ul><li>IOTA’s analytical approach </li></ul></ul><ul><li>  </li></ul><ul><ul><li>Community-derived reports comparing quality of vendor OpenURLs </li></ul></ul><ul><li>  </li></ul><ul><ul><li>Concept of the OpenURL Quality Index </li></ul></ul><ul><ul><li>IOTA & KBART: relationship & joint initiative </li></ul></ul><ul><li>  </li></ul><ul><ul><li>Community involvement in IOTA: necessary for best outcomes </li></ul></ul>
  3. 3. Before OpenURL: Proprietary Linking <ul><li>  </li></ul><ul><ul><li>Certain A&I database providers (e.g., CSA, PubMed) offered full-text linking option for a select number of content providers. </li></ul></ul><ul><li>  </li></ul><ul><ul><li>Libraries manually activated full-text linking with providers they had subscriptions with. </li></ul></ul><ul><li>  </li></ul><ul><ul><li>A&I --> Full Text </li></ul></ul><ul><li>  </li></ul>
  4. 4. Proprietary Linking: Pros and Cons <ul><ul><li>Linking had to be activated manually by libraries for each full-text provider. </li></ul></ul><ul><li>  </li></ul><ul><ul><li>A&I providers offering this option were few. </li></ul></ul><ul><li>  </li></ul><ul><ul><li>Selection of full-text providers was limited. </li></ul></ul><ul><li>  </li></ul><ul><li>But... </li></ul><ul><ul><li>  Once set up, the static links to full texts were accurate. </li></ul></ul><ul><ul><li>Problem source pinpointed easily: A&I --> Full Text </li></ul></ul>
  5. 5. Advent of OpenURL <ul><ul><li>Objective: Deliver full texts unrestrained by proprietary silos. </li></ul></ul><ul><li>  </li></ul><ul><ul><li>Open standard generating dynamic links at time of request. </li></ul></ul><ul><li>  </li></ul><ul><ul><li>A-Z list (e.g., e-journal, e-books): </li></ul></ul><ul><ul><ul><li>Knowledge base (KB) with library's holdings. </li></ul></ul></ul><ul><ul><ul><li>Replaces librarian as intermediary in linking. </li></ul></ul></ul><ul><ul><ul><li>Indicates provider of &quot;appropriate copy&quot; </li></ul></ul></ul><ul><li>  </li></ul><ul><ul><li>A&I (&quot;Source&quot;) --> A-Z list (&quot;KB&quot;) --> Full Text (&quot;Target&quot;) </li></ul></ul>
  6. 6. <ul><li>A, Bernand , et al. &quot; A versatile nanotechnology to connect individual nano-objects for the fabrication of hybrid single-electron devices. &quot; Nanotechnology 21 , no. 44 ( November 5, 2010 ): 445201 . Academic Search Complete , EBSCO host (accessed October 24, 2010). </li></ul>OpenURL: resolver, syntax, linking nodes 21 &issue= 44 &date= 20101105 &spage= 445201 &title= Nanotechnology &atitle= A+ versatile+nanotechnology+to+connect+individual+nano-objects+for+the+ fabrication+of+hybrid+single-electron+devices. &aulast= A++Bernand Source Citation Target OpenURL (Source OpenURL structured similarly)
  7. 7. Pros & Cons of OpenURL <ul><li>Pros: </li></ul><ul><ul><li>KB/Resolver vendors took over most of the linking setup: Less work for libraries and providers. </li></ul></ul><ul><ul><li>Participation by A&I platforms and full-text providers exceeded proprietary linking: OpenURL scales better. </li></ul></ul><ul><li>  </li></ul><ul><li>Cons: </li></ul><ul><ul><li>Dynamic linking less predictable than static linking: more difficult to pinpoint cause of link failures. </li></ul></ul><ul><ul><li>OpenURL linking not improved significantly last 10 years. </li></ul></ul><ul><ul><li>No systematic method exists to benchmark OpenURLs. </li></ul></ul>
  8. 8. Identifying source of problem… <ul><li>&quot;72% of respondents to the online survey either agreed or strongly agreed that a significant problem for link resolvers is the generation of incomplete or inaccurate OpenURLs by databases (for example, A&I products).&quot; </li></ul><ul><li>Culling, James. 2007. Link Resolvers and the Serials Supply Chain: Final Project Report for UKSG, p.33. </li></ul><ul><li>  </li></ul><ul><li>Defining methodology for addressing problem… </li></ul><ul><li>  </li></ul><ul><li>Recently, researchers have indicated the need for metadata quality metrics , including: </li></ul><ul><ul><li>completeness; </li></ul></ul><ul><ul><li>accuracy; </li></ul></ul><ul><ul><li>conformance to expectations; </li></ul></ul><ul><ul><li>logical consistency and coherence. </li></ul></ul><ul><li>Bruce, Thomas R. and Hillmann, Diane I. 2004. The Continuum of Metadata Quality: Defining, Expressing, Exploiting. In Metadata in Practice . Ed. Diane I. Hillmann and Elaine L. Westbrooks. Chicago: American Library Association, pp. 238-256. </li></ul>
  9. 9. Année philologique OpenURL Study <ul><li>2008 Cornell study led by Adam Chandler* </li></ul><ul><ul><li>Problem: Too often links sent from Aph did not successfully resolve to requested resource. </li></ul></ul><ul><ul><li>Objective: Examine quality of OpenURLs offered to users by Aph in order to improve the linking. </li></ul></ul><ul><li>  </li></ul><ul><li>Aph Study investigated: </li></ul><ul><ul><li>Faulty citation metadata from source database. </li></ul></ul><ul><ul><li>Method to evaluate the OpenURLs. </li></ul></ul>*Chandler, Adam. 2009. Results of L’Année philologique online OpenURL Quality Investigation: Mellon Planning Grant Final Report.
  10. 10. Scoring System & Aph Study Outcomes <ul><li>Concept of scoring in Aph study (based on B. Hughes study)* </li></ul><ul><ul><li>establish a baseline for comparison; </li></ul></ul><ul><ul><li>results to be shared with data providers; </li></ul></ul><ul><ul><li>develop a best practice. </li></ul></ul><ul><li>  </li></ul><ul><li>Problem analysis in Aph study limited to: </li></ul><ul><ul><li>source link </li></ul></ul><ul><ul><li>presence/absence of citation metadata elements </li></ul></ul><ul><li>  </li></ul><ul><li>Results: </li></ul><ul><ul><li>OpenURL quality model: compares elements in Aph OpenURLs to those of other providers. </li></ul></ul><ul><ul><li>No scoring was achieved for Aph , but model is first step towards scoring system. </li></ul></ul>*Hughes, Baden. 2004. Metadata Quality Evaluation: Experience from the Open Language Archives Community. In Digital Libraries: International Collaboration and Cross-Fertilization. Ed. Zhaoneng Chen et al. Berlin: Springer-Verlag, 2004, pp. 320-329.
  11. 11. Creation of IOTA <ul><li>NISO: </li></ul><ul><ul><li>Accepts proposal to take Aph Study to wider community. </li></ul></ul><ul><ul><li>New OpenURL quality metrics initiative formed in Jan. 2011. </li></ul></ul><ul><ul><li>Branded as: I mproving O penURLs T hrough A nalytics. </li></ul></ul><ul><li>  </li></ul><ul><li>Basic Assumptions: </li></ul><ul><ul><li>Results are achieved through an analytical investigation of how OpenURL links work. </li></ul></ul><ul><ul><li>Practical Goal: Not the OpenURL standard is addressed, but the links (OpenURLs) generated by standard. </li></ul></ul><ul><ul><li>Selective changes to OpenURLs will lead to significant improvement in linking success rate. </li></ul></ul><ul><ul><ul><li>Motto: &quot;small changes. big improvements&quot; </li></ul></ul></ul>
  12. 12. IOTA Desired Outcomes <ul><ul><li>Produce qualitative reports that will help OpenURL providers quickly compare their OpenURL quality to that of their peers. </li></ul></ul><ul><ul><li>Develop community-recognized index for measuring the quality of OpenURL links generated by content providers. </li></ul></ul><ul><li>   </li></ul><ul><ul><li>Method: </li></ul></ul><ul><ul><ul><li>fair; </li></ul></ul></ul><ul><ul><ul><li>transparent; </li></ul></ul></ul><ul><ul><ul><li>scalable across all OpenURLs and their providers. </li></ul></ul></ul>
  13. 13. How is comparing OpenURLs useful? <ul><li>Content providers generating OpenURLs can: </li></ul><ul><ul><li>compare their OpenURLs with other providers; </li></ul></ul><ul><ul><li>make improvements to their OpenURLs. </li></ul></ul><ul><li>Institutions can: </li></ul><ul><ul><li>compare OpenURL providers; </li></ul></ul><ul><ul><li>make local adjustments to OpenURL setup. </li></ul></ul><ul><li>Link resolver vendors can: </li></ul><ul><ul><li>compare OpenURL providers; </li></ul></ul><ul><ul><li>change their OpenURL provider settings: </li></ul></ul><ul><ul><ul><li>Link resolvers; </li></ul></ul></ul><ul><ul><ul><li>Web-scale discovery products. </li></ul></ul></ul>
  14. 14. Report types <ul><li>Source reports </li></ul><ul><ul><li>Viewing how a particular (1) vendor or (2) database </li></ul></ul><ul><ul><ul><li>A. uses OpenURL elements (element frequency) </li></ul></ul></ul><ul><ul><ul><li>B. formats OpenURL elements (pattern frequency) </li></ul></ul></ul><ul><li>Element / Pattern reports </li></ul><ul><ul><li>Viewing how a particular (1) element or format </li></ul></ul><ul><ul><ul><li>A. is used across vendors </li></ul></ul></ul><ul><ul><ul><li>B. is used across databases </li></ul></ul></ul>
  15. 15. Running reports
  16. 16. Reports: Log file providers
  17. 17. Report: source (vendor or database)
  18. 18. Report: source = vendor
  19. 19. Report: element, source = vendor
  20. 20. Report: pattern, source = vendor
  21. 21. Report: source = database
  22. 22. Report: element, source = database
  23. 23. Report: pattern, source = database
  24. 24. Report: element and pattern frequency
  25. 25. Report: element & pattern frequency: Vendor option
  26. 26. Report: element/pattern frequency: Choosing Metric
  27. 27. Report: element/pattern by vendor
  28. 28. Report: element & pattern frequency: Database option
  29. 29. Report: element/pattern by database
  30. 30. Reporting System: improvements underway <ul><li>Consolidating variant instances of databases and vendors if the same; </li></ul><ul><li>Separating article-like requests from book-like requests </li></ul><ul><ul><li>Either/Or situation: most resources do not offer both formats </li></ul></ul><ul><ul><li>Once separation is completed, users will be given corresponding options to select OpenURL data by format: ARTICLE or BOOK </li></ul></ul><ul><li>These improvements will also benefit the accuracy of the OpenURL scoring system. </li></ul>
  31. 31. OpenURL Quality Index: initial version <ul><li>1. Core elements: </li></ul><ul><ul><li>Any element contained in IOTA's OpenURL reporting system; </li></ul></ul><ul><ul><li>13M OpenURLs already obtained from libraries content providers. </li></ul></ul><ul><li>  </li></ul><ul><li>2. Scoring system based on assumption : </li></ul><ul><ul><li>Correlation exists between </li></ul></ul><ul><ul><ul><li># of core elements (&quot;OpenURL completeness&quot;) & </li></ul></ul></ul><ul><ul><ul><li>ability of OpenURLs to link to specific content. </li></ul></ul></ul><ul><li>  </li></ul><ul><li>3.  Weighting assigned to core elements: </li></ul><ul><ul><li>Based on relative importance </li></ul></ul><ul><ul><ul><li>spage vs atitle </li></ul></ul></ul><ul><ul><ul><li>issn vs jtitle </li></ul></ul></ul><ul><ul><ul><li>doi/pmid vs date, etc. </li></ul></ul></ul>
  32. 32. OpenURL Quality Index: vendor rating
  33. 33. Work in Progress <ul><ul><li>Element weighting still in progress: </li></ul></ul><ul><ul><ul><li>E.g., importance of identifiers (doi, pmid) vs bibliographic data (issn, volume, spage). </li></ul></ul></ul><ul><li>  </li></ul><ul><ul><li>Currently, IOTA focuses on OpenURLs from citation sources only. OpenURL quality is also influenced by: </li></ul></ul><ul><ul><ul><li>knowledge base, </li></ul></ul></ul><ul><ul><ul><li>resolver, </li></ul></ul></ul><ul><ul><ul><li>full-text provider (target). </li></ul></ul></ul><ul><li>  </li></ul><ul><ul><li>High &quot;completeness&quot; score of OpenURLs not always indicative of &quot;success&quot; in linking to full texts </li></ul></ul><ul><ul><ul><li>Combination of multiple indexes along linking nodes may provide more complete picture. </li></ul></ul></ul>
  34. 34. IOTA & KBART: NISO working groups <ul><li>IOTA </li></ul><ul><li>Deals with issues specific to OpenURL linking; </li></ul><ul><li>Seeks improvements in OpenURL elements used by: </li></ul><ul><ul><li>OpenURL providers. </li></ul></ul><ul><li>KBART </li></ul><ul><li>“ Knowledge Bases And Related Tools” </li></ul><ul><li>Deals with data issues at the KB level </li></ul><ul><li>Seeks improvements in data exchange practices between: </li></ul><ul><ul><li>content providers (e.g. OpenURL providers); </li></ul></ul><ul><ul><li>product vendors (e.g. link resolver vendors); </li></ul></ul><ul><ul><li>subscription agents. </li></ul></ul>
  35. 35. IOTA & KBART: related through OpenURL <ul><li>IOTA node: </li></ul><ul><ul><li>analyzing data sent from OpenURL source to link resolver. </li></ul></ul><ul><li>KBART node: </li></ul><ul><ul><li>creating formatting best practices for data sent from content providers to knowledge base (and link resolver) vendors. </li></ul></ul>
  36. 36. KBART-IOTA joint initiative <ul><li>KBART-IOTA node : </li></ul><ul><ul><li>Exploring together the third source of failures: </li></ul></ul><ul><ul><ul><li>link-to (or target) syntax and behavior which couples link resolvers to content providers </li></ul></ul></ul><ul><li>Collaboration begun in March 2011 is meant to address OpenURL quality in a broader context. </li></ul>
  37. 37. How can I get involved? <ul><li>If you are an OpenURL provider: </li></ul><ul><ul><li>Contribute data to IOTA </li></ul></ul><ul><ul><li>Review the IOTA data </li></ul></ul><ul><ul><li>This data is meant to help you make improvements in your OpenURL linking </li></ul></ul><ul><li>  </li></ul><ul><li>If you are a librarian: </li></ul><ul><ul><li>Contribute data to IOTA </li></ul></ul><ul><ul><li>Spread the word to vendors about IOTA </li></ul></ul><ul><ul><li>OpenURL data contact: </li></ul></ul><ul><ul><li>Adam Chandler, </li></ul></ul>
  38. 38. IOTA Web Presence
  39. 39. Questions? <ul><li> </li></ul><ul><li> </li></ul><ul><li>@nisoiota on twitter </li></ul>Rafal Kasprowski Electronic Resources Librarian Rice University, Fondren Library MS 44 Houston, TX 77005 [email_address]