• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
1. Introduction
 

1. Introduction

on

  • 576 views

 

Statistics

Views

Total Views
576
Views on SlideShare
576
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft Word

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    1. Introduction 1. Introduction Document Transcript

    • Technical Evaluation of Sirsi RFP Written Responses and RFP Demonstration System This report is submitted in partial fulfillment of contract # 306-03-8165. William E. Moen, Ph.D. Kathleen R. Murray, Ph.D. Irene Lopatovska, M.L.S Texas Center for Digital Knowledge University of North Texas Denton, Texas March 26, 2003 William E. Moen, Ph.D. Texas Center for Digital Knowledge Principal Investigator University of North Texas
    • RFP: TECHNICAL EVALUATION FOR SIRSI The following research assistants contributed to this report: Alexis Linoski, Fatih Oguz, Rich Reed, Barb Schultz-Jones, and Scott Simon. ZLOT Project Page 2 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI Contents 1. Introduction..............................................................................................................................................4 2. Evaluation Results for Sirsi......................................................................................................................4 2.1 Understanding the Project..................................................................................................................4 2.2 Demonstration System: Z-Interop Search Target...............................................................................8 2.2.1 Questionnaire.............................................................................................................................8 2.2.2 Test Searches & Scans..............................................................................................................9 2.3 Demonstration System: TexShare Database Search Targets............................................................9 2.3.1 Questionnaire...........................................................................................................................10 2.3.2 Test Searches...........................................................................................................................11 2.4 Demonstration System: Database Interaction..................................................................................11 ZLOT Project Page 3 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI 1. Introduction This document contains the results of evaluating Sirsi’s response to the Request for Proposal (RFP) for Library of Texas Resource Discovery Service Software (RFP number 306-03-8200). The following sections address the four parts of the RFP assessed by the ZLOT RFP evaluation team. 2. Evaluation Results for Sirsi 2.1 Understanding the Project In all, 70 requirements were included in the RFP. The requirements were classified by three priority levels: must have, should have, and desirable. Table 1 identifies the number of requirements evaluated in each category. Priority Level Number Must Have 32 Should Have 34 Desirable 4 Total 70 Table 1. Requirements by Priority Level Sirsi indicated they could meet all but two of the requirements, TRS-SRI-C1 (Allow users to create default user profiles) and functional requirement SI-C3 (Support queries in Spanish). Both of these requirements are in the ‘Desirable’ priority category. Table 2 lists the number and percent of responses to the first question analyzed in this section: Does the vendor state they provide the functionality in their RDS? Vendor Response Number Percent Yes 68 97% No 2 3% No Response 0 0% Total 70 100% Table 2. Number of Requirements Vendor Meets The second question analyzed in this section is: How does the vendor provide the functionality? There were five possible choices: 1. Vendors’ current system 2. Third-party product 3. Custom development for LOT 4. Outsourcing 5. Other Sirsi indicated that 74% (n = 52) of the requirements are satisfied by their proposed system. The vendor will develop 23% (n = 16) of the requirements for the LOT RDS. (See Table 3.) ZLOT Project Page 4 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI Vendor Response Number Percent Current System 52 74% Third Party 0 0% Custom Development 16 23% Outsourcing 0 0% Other 0 0% Total 68 97% Table 3. How Vendor Meets Requirements Included below are the ZLOT RFP evaluation team’s comments regarding the vendor's response to each of the requirements. These are presented in priority groupings. The requirements are identified by the numbers in the RFP, with the exception that the four functional requirements are preceded by ‘F-‘. Requirements: Priority 1 – RDS Software Must Have Requirement ZLOT Review Comments 1 TRS-SRI-A1.1 Vendor indicates satisfying this requirement “based on the commercial databases currently licensed for state wide access in Texas.” 2 TRS-SRI-A1.2 Vendor indicates satisfying this requirement “based on the commercial databases currently licensed for state wide access in Texas.” 3 TRS-SRI-A2 There was no indication in the vendor’s comments regarding how target resources are administered. An “automated update service” is included in the proposed system for the plugins/connectors that harvest content. 4 TRS-SRI-A3.1 No details provided by vendor for this requirement. 5 TRS-SRI-A3.2 No details provided by vendor for this requirement. 6 TRS-SRI-A4 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. Vendor “assumes a single central database of user authentication which is maintained by a central agency.” 7 TRS-SRI-A5 No details provided by vendor for this requirement. 8 TRS-SRI-A6 Vendor indicates satisfying this requirement by using plugins/connectors so as not to “interrupt the user’s session experience.” 9 TRS-SRI-A7 Vendor plans to release this functionality by Summer 2003. Vendor will document the format of their history transaction log so that third party database vendors can map their statistical data. 10 TRS-SRI-A8.1 No details provided by vendor for this requirement. 11 TRS-SRI-A8.2 No details provided by vendor for this requirement. 12 TRS-SRI-A9 No details provided by vendor for this requirement. 13 TRS-SI-A1.1 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 14 TRS-SI-A1.2 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 15 TRS-SI-A1.3 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 16 TRS-SI-A2.1 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 17 TRS-SI-A2.2 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 18 TRS-SI-A2.3 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 19 TRS-SI-A3.1 Vendor claims to offer a protocol neutral/multi-protocol technology. 20 TRS-SI-A3.2 Vendor claims to offer a protocol neutral/multi-protocol technology. ZLOT Project Page 5 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI Requirement ZLOT Review Comments 21 TRS-SI-A4.1 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 22 TRS-SI-A4.2 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 23 TRS-SI-A5 Vendor did not provide any details about their development plans other than saying that they will work with TSLAC “to formulate a plan for providing this feature as an option in a future release.” 24 TRS-SI-A6.1 No details provided by vendor for this requirement. 25 TRS-SI-A6.2 No details provided by vendor for this requirement. 26 TRS-RI-A1.1 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 27 TRS-RI-A1.2 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 28 TRS-RI-A1.3 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 29 TRS-RI-A2 Vendor indicates satisfying this requirement depends on the ability of the target to provide the specific data and functionality. 30 TRS-RI-A3 Vendor indicates satisfying this requirement by providing direct or indirect linking services. 31 TRS-RI-A4 No details provided by vendor for this requirement. 32 TRS-RI-A5 No details provided by vendor for this requirement. Requirements: Priority 2 – RDS Software Should Have Requirement ZLOT Review Comments 33 F-SRI-B1 “Separate logins can trigger different ‘rooms’ to appear.” 34 TRS-SRI-B2 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 35 F-SRI-B3 No details provided by vendor for this requirement. 36 TRS-SRI-B4 The details provided by the vendor for this requirement do not address the requirement directly. 37 TRS-SRI-B5 Vendor indicates satisfying this requirement by providing a database of databases to assist the user in locating the resources they need. Vendor considers this capability to be at the heart of their product concept. 38 TRS-SRI-B6 The details provided by vendor for this requirement do not address the requirement directly. 39 TRS-SRI-B7 No details provided by vendor for this requirement. 40 TRS-SI-B1.1 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 41 TRS-SI-B1.2 No details provided by vendor for this requirement. 42 TRS-SI-B1.3 No details provided by vendor for this requirement. 43 TRS-SI-B1.4 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. The vendor notes that they will be using definitions from Functional Area C of the Bath Profile for non-catalog searches. 44 TRS-SI-B1.5 The vendor states “the Bath/US profiles do not provide a specification for limiting a search to items available in electronic form or full-text.” Satisfying this requirement for non-catalog resource collections such as the TexShare databases is dependent on the database vendor implementing this functionality. ZLOT Project Page 6 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI Requirement ZLOT Review Comments 45 TRS-SI-B2.1 Vendor indicates satisfying this requirement depends on the ability of the target to provide the specific data and functionality. 46 TRS-SI-B2.2 Vendor indicates satisfying this requirement depends on the ability of the target to provide the specific data and functionality. 47 TRS-SI-B3.1 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 48 TRS-SI-B3.2 Vendor claims to offer a protocol neutral/multi-protocol technology to search against “a theoretically unlimited number of targets simultaneously.” 49 TRS-SI-B4.1 Vendor will address this requirement by creating mappings to Bath and non-Bath compliant targets. 50 TRS-SI-B4.2 Vendor does not address the issue of supporting administratively defined sets of targets. 51 TRS-SI-B5.1 No details provided by vendor for this requirement. 52 TRS-SI-B5.2 No details provided by vendor for this requirement. 53 TRS-SI-B6 No details provided by vendor for this requirement. 54 TRS-SI-B7 Vendor indicates satisfying this requirement by allowing the organization of resource collections on many levels, including (1) topic and purpose, (2) a search of multiple resources, and (3) professional skills as a subject matter expert. 55 TRS-SI-B8 Vendor indicates satisfying this requirement by providing a batch loader and documentation to enable interaction with “the state’s locally created and managed relational database management system.” 56 TRS-SI-B9.1 No details provided by vendor for this requirement. 57 TRS-SI-B9.2 Vendor indicates satisfying this requirement by including a fully integrated Open URL Resolver for which “rules can be established that determine the scope addressed by the Resolver.” 58 TRS-SI-B10 No details provided by vendor for this requirement. 59 TRS-SI-B11 This requirement is being evaluated for the vendor's RFP demonstration system in another aspect of the ZLOT technical evaluation. 60 TRS-RI-B1 Vendor indicates satisfying this requirement depends on the ability of the target to provide the specific data and functionality. 61 TRS-RI-B2 Vendor indicates satisfying this requirement depends on the ability of the target to provide the specific data and functionality. 62 TRS-RI-B3 Vendor indicates satisfying this requirement “through a linking service such as the Sirsi Open URL Resolver.” 63 TRS-RI-B4.1 No details provided by vendor for this requirement. 64 TRS-RI-B4.2 No details provided by vendor for this requirement. 65 TRS-RI-B4.3 Vendor did not provide any details about their development plans. 66 TRS-RI-B5 Vendor indicates satisfying this requirement through their integrated Open URL Resolver. Requirements: Priority 3 – Desirable Features of the RDS Software Requirement ZLOT Review Comments 67 TRS-SRI-C1 Vendor addressed this requirement as a future product development feature in their two-year plan. “This functionality will be available to Texas as a no-cost upgrade when available.” 68 F-SI-C2 No details provided by vendor for this requirement. 69 F-SI-C3 Vendor “has no plans to provide an on-the-fly machine translation.” 70 TRS-RI-C1 No details provided by vendor for this requirement. ZLOT Project Page 7 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI System Support Customer service assistance is available Monday – Friday between 7:00am and 7:00 p.m. CST. Emergency assistance is available 24 hours a day, 7 days a week. Sirsi provides a full one-year warranty for all hardware and software and will pass through the full manufacturer’s warranty for all products to the customer. The maintenance contract for any server equipment supplied by Sirsi is between them and the manufacturer. For hardware problems, Sirsi will deal directly with a maintenance subcontractor. Major system upgrades are developed and released on an annual basis. Sirsi also provides frequent interim maintenance updates downloadable from an ftp server via the web. Software support is available on an annually renewable contract. Most software maintenance is performed online. 2.2 Demonstration System: Z-Interop Search Target The evaluation of the demonstration system's interaction with the Z-Interop search target consisted of a 13-item questionnaire (Appendix C in Technical Evaluation Methodology document), 39 test searches, and 3 Scans (Appendix D in Technical Evaluation Methodology document). 2.2.1 Questionnaire The items on the questionnaire are associated with the following TRS: • TRS-SI-A1.1 • TRS-RI-A1.2 • TRS-RI-A2 The demonstration system’s search and retrieval interface provided functionality for 12 (92%) items identified in the technical requirement specifications (see Table 4). Questionnaire Response Number Percent Yes 12 92% No 1 8% Total 13 100% Table 4. Number of Questionnaire Items The following list is limited to items that did not meet the technical requirements addressed in the questionnaire (Appendix C in Technical Evaluation Methodology document): Language of resource (TRS-RI-A1.2): The data type was not included in the retrieved results in either the brief or the full record view. Language type appeared the MARC21 record view. Quality Assessment The demonstration system received a quality level Good (3 of 5). Comments regarding the quality assessment: • The interface format was initially confusing; My Room should have been named login; • After logging in there was trouble finding the search page ZLOT Project Page 8 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI • Font size may be too small. 2.2.2 Test Searches & Scans The second part of the evaluation of the demonstration system's interaction with the Z-Interop search target consisted of test searches issued against the Z-Interop search target (Appendix D in Technical Evaluation Methodology document). The following is a description of the results. All thirty-nine searches (100%) conformed to the profile specification. (See Table 5.) Z-Interop Conformance Number Percent Yes 39 90% No 10% 0 Total 39 100% Table 5. Conformance for Searches For all thirty-nine of the test searches, the attribute combinations for each query as defined in the Bath Profile or the U.S. National Profile were correct. There is also a requirement for issuing Scan requests (TRS-SI-B11). Three Scan requests were issued through the Advanced Search interface as part of the evaluation of the demonstration system, with success for each Scan. (See Table 6.) Scan Successful Number Percent 100% Yes 3 0% No 0 Total 3 100% Table 6. Number of Scan Requests Each Scan retrieved a set of index entries. Follow-on searches from the browse list result of the Scans, however, did not have expected results: • Author Exact Match Scan follow-on search resulted in "No records retrieved!" • Title Exact Match Scan follow-on search resulted in "No records retrieved!" • Subject Exact Match Scan follow-on search resulted in "No records retrieved!" 2.3 Demonstration System: TexShare Database Search Targets This section describes the results that ZLOT RFP evaluation team members received when assessing the demonstration system's Z39.50 client's conformance to Profile Level 0 and Level 1 queries to two TexShare database targets: 1. EBSCO Academic Search Premier 2. Gale Health Reference Center Academic. The ability to concurrently perform Profile Level 0 searches of both TexShare databases and of multiple Library of Texas (LOT) resource collection types was also required in this part of the technical evaluation. The Bath compliant Z-Interop search target and the two TexShare database targets constitute the multiple resource collection types. ZLOT Project Page 9 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI The TexShare Database evaluation consisted of a 6-item questionnaire (Appendix E in Technical Evaluation Methodology document) and 39 test searches (Appendix F in Technical Evaluation Methodology document). The results of these evaluations follow. 2.3.1 Questionnaire Questions 1, 2, and 3 are associated with the following TRS and include 18 items that were assessed (see Table 7): • TRS-SI-A1.1 • TRS-RI-A1.3 Questionnaire Response Number Percent Yes 10 56% No 8 44% Total 18 100% Table 7. Number of Questionnaire Items Question 2 evaluated whether specific data elements were presented to the user in the retrieved record. The Sirsi demonstration system did not successfully access the Gale TexShare database. For that reason, there was no way to see how the demonstration system would present retrieved records from that database. Records retrieved from the Ebsco TexShare database indicate the ability to present the data elements when available in the record, e.g., if item was available in full text (i.e. digital). In some cases, the data elements were only available in the MARC record view. Question 4 provides comments regarding the search process: • It was not possible to successfully search against the GALE database; the Search Details shows the search failed; the results page states: "No records retrieved!" • In Expert Search the default number of Retrieved results and Show results is set to 10. The maximum number of Retrieved/Show records is 100. Every time a New Search is performed the Retrieved/Show numbers return to the default. • If the Expert search interface is not used for a while it times out and a different search interface appears and the TexShare search targets no longer appear. They reappear if the Expert Search tab is clicked again. Question 5 provides comments regarding the search results: • The Save As button did not work. We were unable to locate it easily. • The Retrieved/Show results (search interface) are set to a minimum default number of 10. It limits the number of Retrieved/Show records in the retrieve interface. A user has to go back to the search interface and change the default Retrieve/Show to a greater number. Quality Assessment The demonstration system received a quality level of Good (3 of 5). Comments regarding the quality assessment: • The results page has many buttons and other features that make it seem very busy. The user has many choices to make from the page, and not all of the buttons may be intuitive to the user. ZLOT Project Page 10 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI 2.3.2 Test Searches In all, 39 search requirements were included in the assessment of the demonstration system (Appendix F in Technical Evaluation Methodology document). The Gale TexShare database was not accessible through the vendor’s demonstration system. We know that Gale does not accept both username and password for authentication via Z39.50, and the vendor may not have tested this completely. As a result, we were unable to assess the vendor’s demonstration system against Gale. (See Table 8.) Details about the missing functionality are provided below for search results against the EBSCO database and the Z- Interop target only. System Response Number Percent Yes 32 82% No 7 8% Total 39 100% Table 8. Number of Search Requirements The following list is limited to search results that did not meet the technical requirements. Title search – keyword with right truncation (TRS-SI-B1.4): Truncation on the last letter did not return consistent results with the native interface and the Z-Interop Z-client. However, truncation after 4 letters did return consistent results from all three search interfaces. ISBN (TRS-SI-B1.4): The vendor does not provide a search option specifically for ISBN but does have an area for Standard Identifier where ISBN can be entered. This search option is available in the Expert Search interface. ISSN (TRS-SI-B1.4): The vendor does not provide a search option specifically for ISSN but does have an area for Standard Identifier where ISSN can be entered. This search option is available in the Expert Search interface. Any, author, title, subject keyword (TRS-SI-A4.1): Concurrent searches to the TexShare databases were not performed because access to Gale was not provided. 2.4 Demonstration System: Database Interaction This section presents the results of assessing the vendor demonstration systems use of the Texas State Library and Archives Commission (TSLAC) authentication data to control access to licensed databases (TRS-SRI-A4) and the TSLAC library profile data to select local library information for use in the application (TRS-SRI-B2). The vendor's system provided a place to log into the application using one of the user ids and user passwords associated with a particular library. For the six items of local library information that were to be presented in the user interface, 3 of the 6 were present after login. (See Table 9.) Interface Customization For Local Library Number Percent Yes 3 67% No 3 33% Total 6 100% ZLOT Project Page 11 March 26, 2003
    • RFP: TECHNICAL EVALUATION FOR SIRSI Table 9. Number of Local Library Information Items The My Room icon is the login, which is not immediately intuitive. The About icon hyperlinks to the library's homepage and when hovering over the About icon, the URL is visible, which was counted as addressing the "URL to the library" requirement. Even though the vendor did not display the logo provided in the Texas Library Directory database, the interface prominently displays a picture or logo of local Library. Interface does include E-mail address through the Contact icon. Physical address and telephone number were not provided in the interface, although these could be found by clicking on About icon and the information found on the library's homepage. Nicely personalized for the local library through use of links to the library's catalog and the library's calendar of events. Quality Assessment The demonstration system received a quality level of Very Good (4 of 5). Comments regarding the quality assessment: Navigational controls very consistent - icon driven; basically intuitive. Nice looking layout and colors. All pages viewed had a similar look and feel. Limited evaluation of the site. ZLOT Project Page 12 March 26, 2003