Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NISO Webinar: Discovery & Delivery: Innovations & Challenges


Published on

Today’s library discovery services are primarily based upon indexes derived from journals, e-books and other electronic information of a scholarly nature. The content comes from a range of information providers and products--commercial, open access, institutional, etc. By indexing the content in advance, discovery services have the ability to deliver more sophisticated services with instant performance, compared to the federated search techniques used previously. Libraries increasingly rely on index-based discovery services as their strategic interfaces through which their patrons gain access to the rapidly growing breadth of information that may be available to them.

This webinar will discuss the challenges of operating a centralized index-based discovery system. Learn about their strengths, and their weaknesses, the needs for standards and best practices in this arena, how libraries and providers can assess the usage, and how libraries can satisfy audiences with different needs--ranging from undergraduates to faculty across every discipline.

Published in: Education, Technology
  • Be the first to comment

NISO Webinar: Discovery & Delivery: Innovations & Challenges

  1. 1. [insert web address for NISO webinar page]Discovery and Delivery: Innovations and Challenges September 26, 2012 Speakers: Lucy Harrison, Timothy Babbitt, David Bietila
  2. 2. Introducing the Open Discovery Initiative NISO Webinar: Discovery and Delivery: Innovations and ChallengesLucy Harrison, Florida Virtual Campus, D2D Liaison September 26, 2012 2
  3. 3. Topics• What are centralized indexes?• What are their strengths and weaknesses?• What is the NISO ODI initiative?• How will it help improve the discovery landscape? 3
  4. 4. Evolution of Library Search• Card Catalogs• Online Catalogs• Federated search tools• Next-generation library catalogs (discovery interfaces)• Index-based discovery services 4
  5. 5. Discovery Interfaces ILS Data Digital Search: Local Collections Index ProQuest Search Results EBSCOhost Federated Search Engine … MLA Bibliography ABC-CLIO Real-time query and responses
  6. 6. Index-based Discovery ILS Data Digital Search: Collections Consolidated Index ProQuest Search Results EBSCOhost … MLA Bibliography ABC-CLIO Harvesting and indexing performed in advance
  7. 7. Strengths of Index-basedDiscovery• Fast response time (vs. federated search)• Structured metadata: – Improves search & retrieval – Faceted navigation – Improves integration of search results• Indexing full-text of content amplifies access 7
  8. 8. Issues with Index-basedDiscovery• Important to understand depth of indexing – Currency, dates covered, full-text or citation, quality of metadata• Uneven participation diminishes impact• Ecosystem dominated by private agreements• Complexity and uncertainty poses barriers for participation
  9. 9. Need to Bring Order to Chaos• Libraries need the ability to understand and evaluate tools, content, providers• Information providers need the confidence that their content is being treated fairly• Service providers need the ability to more efficiently integrate content 9
  10. 10. Key Areas for Libraries• Strategic investments (in subscriptions and discovery solutions)• Expect comprehensive representation of resources in discovery indexes• Need to be able to evaluate the depth and quality of these index-based discovery products• Usage reporting
  11. 11. Collection Coverage Questions • How well does the index cover the body of scholarly content? • Why do some publishers not participate? • How can libraries understand the differences in coverage among competing services? • How are your library’s content packages represented by the discovery service? • Which resources are not represented in index? • Is content indexed at the citation or full-text level? • What is the quality of the metadata? • What are the restrictions for non-authenticated users?
  12. 12. Key Areas for Service Providers• Encourage information providers to participate• Lower thresholds of technical involvement• Clarify the business rules associated with involvement• Common industry standards and definitions• Usage reporting
  13. 13. Key Areas for InformationProviders• Discovery brings uncertainty• Want to expose content widely (increase usage), but• There are trust issues – With Access / Authentication – With ―Fair‖ Linking• Private agreements• Usage reporting
  14. 14. Need healthy ecosystemamong discovery service providers, libraries and information providers
  15. 15. OPEN DISCOVERY INITIATIVEPromoting Transparency in Discovery 15
  16. 16. ODI Pre-History• June 26, 2011: Exploratory meeting @ ALA Annual• July 2011: NISO expresses interest• Aug 7, 2011: Proposal drafted by participants submitted to NISO 16
  17. 17. ODI ProposalDefine standards and/or best practices forindex-based discovery services – Evaluate the breadth and depth of content – Evaluate availability of content to different institutions and to different users – Streamline workflows – Define models for fair linking – Determine what usage statistics should be collected and disseminated 17
  18. 18. ODI Pre-History• June 26, 2011: Exploratory meeting @ ALA Annual• July 2011: NISO expresses interest• Aug 7, 2011: Proposal drafted by participants submitted to NISO• Aug 2011: Proposal accepted by D2D• Vote of approval by NISO membership• Oct 2011: ODI launched• Feb 2012: ODI Workgroup Formed 18
  19. 19. ODI Charge and Objectives• Improve information services to end users as mediated through index-based discovery services• Create an environment that broadens stakeholder participation and ensures confidence• Foster development of best practices and effective means of assessment 19
  20. 20. Specific Benefits Librarians – Can offer their users as wide a range of content as possible via their discovery service of choice – Can better evaluate discovery services to address their needs 20
  21. 21. Specific Benefits Information providers – Have the confidence that the discovery service providers are handling their content in an appropriate manner – Are therefore encouraged to make available the widest range of content— in terms of breadth and depth – for indexing by the discovery service providers 21
  22. 22. Specific Benefits Discovery service providers – Receive more standardized and efficient integration with the information providers through common industry definitions and communications 22
  23. 23. Balance of Constituents Libraries Marshall Breeding, Co-Chair Michele Newberry, Florida Virtual Campus Jamene Brooks-Kieffer, Kansas State University Sara Brownmiller, University of Oregon Laura Morse, Harvard University Lucy Harrison, Florida Virtual Campus (D2D Ken Varnum, University of Michigan liaison/observer) Information Providers Lettie Conrad, SAGE Publications Linda Beebe, American Psychological Assoc Roger Schonfeld, ITHAKA/JSTOR/Portico Aaron Wood, Alexander Street Press Jeff Lang, Thomson Reuters Peter Noerr, MuseGlobal Service Providers Jenny Walker, Ex Libris Group (Co-Chair) David Lindahl, University of Rochester (XC) John Law, Serials Solutions Jeff Penka, OCLC (D2D liaison/observer) Michael Gorrell, EBSCO Information Services 23
  24. 24. Organization• Reports in NISO through Document to Delivery topic committee (D2D)• Staff support from NISO (Nettie Lagace)• Co-Chairs – Jenny Walker (Ex Libris) – Marshall Breeding (Library Consultant)• D2D Observers: – Jeff Penka (OCLC), Lucy Harrison (FLVC) 24
  25. 25. ODI Project Goals1. Identify, possibly through surveys or other questionnaires, the needs and requirements of the three stakeholder groups in this area of work• Created subgroups for information gathering: – Level of Indexing – Library Rights – Technical formats – Usage Statistics – Fair Linking 25
  26. 26. ODI Project Goals1. Identify, possibly through surveys or other questionnaires, the needs and requirements of the three stakeholder groups in this area of work• Created subgroups for information gathering• Conducted interviews with stakeholders• Created survey with input from all sub groups• Survey is currently live (closes October 4) 26
  27. 27. 27
  28. 28. ODI Project Goals1. Identify, possibly through surveys or other questionnaires, the needs and requirements of the three stakeholder groups in this area of work• Created subgroups for information gathering• Conducted interviews with stakeholders• Created survey with input from all sub groups• Survey is currently live (closes October 4)• Analyze results as input to Goal 2 28
  29. 29. ODI Project Goals:2. Create recommendations and tools to streamline the process by which information providers, discovery service providers, and librarians work together to better serve libraries and their users
  30. 30. ODI Project Goals:3. Provide effective means for librarians to assess the level of participation by information providers in discovery services, to evaluate the breadth and depth of content indexed and the degree to which this content is made available to the user
  31. 31. Specific deliverables• Standard Vocabulary• NISO Recommended Practice: – Data format & transfer – Communicating content rights – Levels of indexing, content availability – Linking to content – Usage statistics – Evaluate compliance• Inform and Promote Adoption 31
  32. 32. TimelineMilestone Target Date StatusAppointment of working group December 2011Approval of charge and initial work plan March 2012Agreement on process and tools June 2012Survey completed Oct 4, 2012Completion of information gathering October 2012Completion of initial draft January 2013Completion of final draft May 2013 32
  33. 33. Connect with ODI• ODI Project website:• Interest group mailing list:• Email ODI: 33
  34. 34. Seeing Discovery Through User Colored GlassesNISO Webinar: Discovery and Delivery: Innovations and Challenges Timothy Babbitt, Senior Vice President, Platform Management, ProQuest
  35. 35. Understanding What Is Valuable to UsersFoundation of Librarianship S.R. Ranganathan’s The Five Laws of Library Science (Madras India: Madras Library Association, 1931) 1. Books are for use. 2. Every reader his [or her] book. 3. Every book its reader. 4. Save the time of the reader. 5. The library is a growing organism. Even at a time when the emphasis was entirely on physical media the focus was on the individual goals and needs each type of user (the reader)
  36. 36. The Five Laws Updated Ranganathan Updated Laws 1. Books are for use. 1. Information in all of its forms is for 2. Every reader his [or her] book. use. 2. Every researcher their 3. Every book its reader. information. 4. Save the time of the reader. 3. Every medium and delivery 5. The library is a growing platform its user. organism. 4. Enable efficient discovery by the user. 5. The library is part of an evolving research ecosystem. The proliferation of web based information tools allows us to track users and their behavior with increasing precision but understanding the unique needs and behaviors of different researchers requires deep analysis and interpretation of a variety of data – traditional usage data does not tell the whole story.
  37. 37. How do we measure value in the evolvingresearch landscape? Traditional Model  Usage data to measure value (searches and retrievals) – more usage = more value Growing Trend  Web based analytics – behavioral and attitudinal dimensions segmented by types of users Key Question  How can we combine both approaches and what can they provide that traditional usage data cannot?  Usage Data = what they did  Other Analytics = what they were trying to do; did they succeed; what was the context; and who was doing it?
  38. 38. Value Differs by UserWho is doing the research matters Librarians  Making their patrons successful  Promoting services of library to students and faculty with confidence  Delivering services that are convenient for their users Faculty  Extending their influence and reach in their discipline through published research  Efficiently directing students to materials that meet learning objectives  Obtaining research grants Students  Completing coursework in accordance with faculty directives  Remaining in compliance with source attribution policies  Accessing information conveniently
  39. 39. Conventional Wisdom Might Tell Us  High use = high value  High satisfaction = high value  Use (search and retrievals) is homogenous Including Attitudinal and Behavioral dimensions to traditional usage data allows us to challenge many long held assumptions
  40. 40. Three Dimensions of Analysis Focus of Analysis: The discovery and consumption of documents (COUNTER Usage Reporting e.g. Searches and Retrieves) etc.Focus of Analysis: Focus of Analysis: What is the user’sWhat is the user Attitudinal Behavioral context?satisfaction? (Page Views per visit, Time on Site, and Clickstream etc.)
  41. 41. Three Dimensions of Analysis Focus of Analysis: The discovery and consumption of documents (COUNTER Usage Reporting e.g. Searches and Retrieves) etc. Attitudinal Behavioral
  42. 42. Usage in the new age of discovery A case study Earlier this year, a library contacted us about a large increase in search usage in one of their databases: Month 11-Jan 11-Feb 11-Mar 11-Apr … Searches Run 50 250 43 199 Month 12-Jan 12-Feb 12-Mar 12-Apr … Searches Run 265 616 1176 847
  43. 43. Web scale discovery and workflow By default, the user runs an all database search (thus showing as a ―hit‖ against all DB’s in the library’s subscription.)In this scenario, the user starts their search for ―space tourism‖ via a discovery service such as Summon. Finding a document they like, they click through to the ProQuest platform and land at the document level. Then decide to run another search, which ―hits‖ all databases.
  44. 44. Usage in the new age of discovery  The cause? Heavily utilization of a new web scale discovery service.  Users now used our website differently than users who started at the library ―Databases A-Z‖ page. The change in how their users were coming to our site led to a change in usage.  Usage data showed the effect of the change of their patrons’ research environment. Determining the cause required looking beyond the usage data. For more details see the white paper ―Usage in the New Age of Discovery‖
  45. 45. Three Dimensions of Analysis Usage Focus of Analysis: What is the user’s Attitudinal Behavioral context? (Page Views per visit, Time on Site, and Clickstream etc.)
  46. 46. Behavioral Analysis:Precise Measure of Visits
  47. 47. Behavioral AnalyticsPages Viewed/ Visits
  48. 48. Three Dimensions of Analysis UsageFocus of Analysis:What is the user Attitudinal Behavioralsatisfaction?
  49. 49. Attitudinal – Satisfaction Comparison Survey Methodology  Over 19,500 Surveys  Data from November 2011 through September 2012  Continuous monitoring  Predictive modeling Librarians Faculty Students
  50. 50. Satisfaction Comparison: Role
  51. 51. Satisfaction Comparison: RoleNumber of Respondents
  52. 52. Attitudinal Analysis:Student Satisfaction Trend-line
  53. 53. Student Satisfaction: By Week
  54. 54. Satisfaction Comparison: Primary Purpose
  55. 55. Satisfaction Comparison: Primary PurposeNumber of Respondents
  56. 56. Attitudinal AnalysisSatisfaction Comparison: School Type
  57. 57. Attitudinal AnalysisSatisfaction Comparison: School Type
  58. 58. Cross Tab:Primary Purpose and School TypeData from November 8, 2011 through August 12, 2012
  59. 59. “Deeper Dive” with Segment Analysis:Data: Nov. 8, 2011 through Aug 7, 2012
  60. 60. Seeing Value from a UserPerspective  We need the triangulation of  Usage data  Behavioral data  Attitudinal data  Drive understanding of  What did the users do?  What was the context of use?  What were they trying to do and were they successful?  Important because the research ecosystem is changing  i.e. Web scale discovery
  61. 61. Toward understanding what isvaluable to users Updated Laws 1. Information in all of its forms is for use. 2. Every researcher their information. 3. Every medium and delivery platform its user. 4. Enable efficient discovery by the user. 5. The library is part of an evolving research ecosystem.
  62. 62. Next steps  Complete integration and analysis of  Usage data  Behavioral data  Attitudinal data  Give libraries a deep understanding of value from user segments  Next stop…content and search analysis!
  63. 63. Collecting Patron Perspectives onDiscovery ToolsNISO WebinarDiscovery and Delivery: Innovations andChallengesSeptember 26, 2012David BietilaWeb Program DirectorThe University of Chicago Library
  64. 64. Data Sources Finding Aids Databases Digital & Indexes Collections Discovery Catalog Tool Web Pages
  65. 65. Assessment Questions Product Characteristics  Technical  Functional  Interface Relevance to Users  Define use case  Novice users of databases and electronic resources who need to find articles on a topic.  Assess product’s applicability to this use case
  66. 66. 1. User CommentsTopic User attitudes toward the product, and specific aspects valued by users.Method Posted link to a comment form in the header. Set up an info table for three days. Required minimal effort to collect data.Sample Questions What type of resource were you looking for with the Articles Plus? Please share any positive or negative comments about your experience.Results Unprecedented proportion of positive comments. Users valued the speed of this search, and the ability to search both books and articles in one place. Functional & Interface Assessment
  67. 67. Resource sought Journal articleSuccessful? YesAffiliation Graduate or Professional School StudentComment This was a quick search and gave me exactly what I needed.Resource sought Academic journal articlesSuccessful? NoAffiliation Graduate or Professional School Student Record included books, needed a way to filter this outComment results from Time magazine, not what I was looking for at all Both books and articles on certain topics (ones related to philosophy, psychology andResource sought literature).Successful? PartialAffiliation Graduate or Professional School Student Its wonderful to have ONE place where you can search for both articles and books! However, it seems like more books should show up because some books relevant to my search showed upComment in lens but not in Articles Plus. If you dont choose this search tool, please do adopt some search tool that allows comprehensive searching of both books and articles!
  68. 68. 2. Usage StatisticsTopic Usage relative to major databasesMethod Examined data presented in the discovery tool’s admin interface. Compared usage with that of high traffic databases (Web of Science, JSTOR, Academic Search Premier).Results Observed increasing usage, reaching parity in the initial year with Academic Search Premier. Verified that users were finding and using the tool. Interface Assessment
  69. 69. Searches per Month - Articles Plus and Major Databases500004500040000 EDS Foundation Index35000 (ArticlesPlus)30000 Web of Science25000 JSTOR2000015000 Academic Search10000 Premier 5000 0
  70. 70. 3. Usability TestingTopic Clarity of search interface, including functional elements and labels. User ability to complete article discovery tasks.Method Assigned representative article searching tasks. Recording screen activity and spoken comments for analysis.Sample Questions Can you locate full text of the following article? Mary L. Dudziak, “Desegregation as a Cold War Imperative,” 41 Stanford Law Review 61 (1988) You are working to prepare a summary of developments in the field of astronomy. Can you locate five articles on astronomy that were published in Nature in 2010? Functional & Interface Assessment
  71. 71. Task Participant ParticipantSession Status Department 1 2 3 4 5 6 7 8 9 10 1 Graduate Public Policy         X  2 Graduate CMES ~  X   X  X X  3 Undergrad Undeclared     ~  X  X  4 Undergrad Anthro       ~    5 Graduate MAPH           6 Undergrad Physics ~     ~     7 Undergrad English      ~   X ~ 8 Graduate MAPSS ~      ~   ~
  72. 72. 3. Usability Testing Results • Cut or relabeled certain facets • Made heavily used features more prominent. • Removed unused or confusing features • Determined collections to be retained or removed from coverage. Functional & Interface Assessment
  73. 73. Iterative Evaluation Refine Evaluate Use Tool Case
  74. 74. Other Use Cases Experienced researchers  Searching interdisciplinary topics  Searching outside their primary area  Searching within their primary area
  75. 75. 4. Subject AssessmentsTopic Applicability of the discovery tool for use by advanced researchers in disciplines with a variety of requirements.Method Enlisted bibliographers and subject specialists as proxies for different user constituencies. Created an evaluation rubric to ensure comparable results across over 40 disciplines assessed.Results Determined which disciplines would be best served by the tool. Identified strength of coverage in numerous disciplines. Functional Assessment
  76. 76. Subject Evaluation Form
  77. 77. Compared to Compared toSubjects – Social First 25 Results Core Database JSTOR/ GoogleSciences Results Scholar ResultsAnthropology/Geography/Maps ~ ~ ~Gender Studies ~ ~ -History + + +International Political Economy ~ - ~Psychology + - ~Sexuality Studies + + +
  78. 78. Outcomes & Future Assessment Recommendation to purchase Improvements  Local configuration  Enhancement requests Marketing & Communication Benefit of Iterative Assessment Future assessment  Applicability of new coverage  Revisiting usability  Overlap with other search tools
  79. 79. THANK YOU Thank you for joining us today.Please take a moment to fill out the brief online survey. We look forward to hearing from you!