Federated Search Webinar for SLA (Special Libraries Assoc.)

Federated Search in a Disparate Environment PREPARED FOR: SLA Webinar Series Evidence-Based Practice in Libraries 2040 Corbett Rd Monkton, Md 21111 (410.472.4631 Helen L. Mitchell Curtis * hmitchell5@gmail.com Principal, Enterprising Solutions September 9, 2009

Enterprising Solutions Biography Helen L. Mitchell Curtis – Principal, Enterprising Solutions  32+ years at FDA leading one of the largest enterprise search implementations among Civilian Federal Agencies  Develop enterprise-wide search strategies & solutions  Integrate search technologies across IT applications and disparate document repositories  Build governance, management and end user buy-in  Promote collaboration, standards, findability and improved organization of data and document assets  Passion – to help clients to reduce costs, improve quality and efficiency, reduce 'pain points' and achieve a positive search experience 2

Enterprising Solutions Polling Question • What is Your Role? (select all that apply, if group participants) • CIO, Executive Director • Library Director (Corporate, Gov’t, Academia, Solo) • Librarian/Information Management Professional • IT Professional or Consultant • Project/Product Manager • Sales/Marketing/Communications • End User (i.e., Scientist, Researcher, Engineering Professional) • Federated Search Vendor • Other 3

Enterprising Solutions Agenda 1. Terms Clarified 2. Types of Federated Search (FS) 3. FS Challenges & Benefits 4. FDA Case Study 5. FS Evaluation Criteria 6. Examples of FS Solutions 7. Live Federated Search Demo 8. Best Practices 9. Future Vision 10. Questions & Answers 4

1. Definition by AIIM Market IQ 2. Definition by CMS Watch Enterprising Solutions Clarify Terms 3. A Federated Search Primer – Part II 4. Deep Web Technologies 5. Federated Search Rpt & Toolkit-Jill Hurst-Wahl • Reliable and complete retrieval of content based on user need, i.e. everything relevant is recalled (recall) while simultaneously Findability returning only that content relevant to the user’s focus (precision), thus eliminating the review of irrelevant content by the 1 user. • Systems…within an organization…seeking information held Enterprise internally…in a variety of formats and locations, including Search databases, document management systems, and other 2 repositories. Content is pre-indexed, simultaneously searched, (ES) and displayed to authorized users. • The process of performing a simultaneous real-time search of Federated multiple diverse and distributed sources from a single search 3 page, with the federated search engine acting as intermediary. Search (FS) • The set of web-sites and their documents that cannot be accessed via crawler-type search engines such as Google. Deep web content Deep Web typically lives inside of databases, and is accessed through search 4 forms. It is also referred to as the Hidden or Invisible Web. • SW written to access a content source that must know the URL of Connector the source, how to send search commands, its search syntax, & 5 how to process the search results returned from a source. 5

Enterprising Solutions Polling Question  Information Accessibility (select all that apply) 1. I can easily find information to do my job 2. Less than 50% of our organization’s info is searchable online 3. More than 50% of our organization's info is searchable online 4. I reference less than 5 systems (info sources) in any given week 5. I reference 5 or more systems (info sources) in any given week 6

Enterprising Solutions Findability Issues  AIIM Market IQ Research on Findability (of 528 end users):  50% believe Findability in their organization is ―Worse to Much Worse‖ than their consumer-facing web sites  49% have no formal goal for Enterprise Findability within their organizations  49% ―Agreed or Strongly Agreed‖ that finding the information to do their job is difficult and time consuming  69% believe less than 50% of their organization's information is searchable online  36% reference five or more systems in any given week 7 Source: AIIM Market Intelligence, 2008

Enterprising Solutions Why Use Federated Search To increase findability to better accomplish business objectives. To issue a single query across multiple content sources through a common search interface. When not feasible to re-index all of the content available from large public sites like PubMed. To increase user awareness of all content sources such as deep web for scientific, technical and business content. To eliminate using multiple database search protocols & passwords. When don‘t have the rights to index the content (e.g. subscription sites). Real-time search: for content constantly being updated & impractical to 8 keep the data as timely as it needs to be.

Federated Search Sources Enterprising Solutions (examples) Reason Corporate Academic Gov’t Public Library Subscription Databases X X X X Internal or External Repositories X X Library Catalog(s) X X X X News X X Digitized Material X X X Blogs & Wikis X X X Intranet/Internet Sites X X Industry Specific Sources X DB‘s available to customers X X Historical Collections X 9

Enterprising Solutions Typical Non-Federated Search 10 Courtesy of MuseGlobal, Inc.

Enterprising Solutions Typical Federated Search 11 Courtesy of MuseGlobal, Inc.

Enterprising Solutions Federated „Master Index‟ Search  Index multiple data sources content into a single master index  Queries & results come from that one master index  Many Enterprise Search products integrate FS via ‗connectors‘ to accomplish this (ex., FAST, Autonomy, Endeca) 12 Source: New Idea Engineering, Inc.

Enterprising Solutions Federated „Data Silos‟ Search ‗Search Federator‘ processes queries for each data source silo Transforms search terms to match each content source requirements Submits query to each of the sources simultaneously Merges each source‘s results together - single look & feel Maintains no indices of its own, relies on linked systems capabilities 13 Source: New Idea Engineering, Inc.

Enterprising Solutions Surface vs. Deep Web Search Popular search engines (Google, Yahoo…) ―crawl‖ surface web FS can drill down to the deep web where specialized content (i.e., scientific and technical databases) reside Deep Web FS Examples: www.completeplanet.com - 70,000+ searchable DBs & specialty search engines www.science.gov- federates U.S. federal agency science info http://imlsdcc.grainger.uiuc.edu/ - Institute of Museum & Library Services (IMLS) - Digital Collections & Content w/descriptions of digital resources developed by IMLS grantees 14 Source: Juanico-Environmental Consultants, Ltd.

Enterprising Solutions Vertical Search Engine  Closely related to Deep Web – searches for a particular niche i.e., a specific industry, topic, type of content (e.g., scientific research, travel, movies, images, blogs)  Example: www.vetseek.info - is a search engine focusing on veterinary science and related topics 15

Enterprising Solutions Polling Question Federated Search Solutions (select one) 1. We are currently conducting an evaluation to procure a Federated Search Product 2. We currently have a Federated Search Solution installed that satisfies our requirements 3. We have a Federated Search Solution by are considering replacing it or enhancing its capabilities & features 16

Enterprising Solutions Challenges  Authentication  Showing each record‘s branding and copyright information  Licensed or subscription databases  True De-duplication  Virtually impossible because DBs return 10-20 results at a time  Vendors usually just de-dupe the first results set returned  Security  Mapping user credentials and access rights to each repository security model  Speed  Limited by slowest search engine‘s performance 17

Enterprising Solutions Challenges (continued)  Lack of data standardization  Each source has a unique access method & needs translation  Metadata mapping between FSS and underlying systems  Access methods to sources may change  Requires an interface rewrite or modification  Rules for error handling  Ex. Query term not available—exclude the query, the repository, or proceed without the term?  Ex. Timeouts or connection problem  Complex searches usually not available  Fielded searches  Known Items, i.e. Article Name  Best to directly search database 18

Enterprising Solutions Challenges (continued)  Relevancy scores  Can‘t identify a single relevancy ranking model  Relevancy rankings for repository‘s results refers to its own  May be not be useful when comparing the results with those from another system  Access to content stored in a variety of places  Results page may not let user obtain identified documents  This may involve a built-in viewer or invoking the owning product‘s interface.  Combining navigators from each result set  i.e., faceted search, taxonomies and auto-generate clusters  Selecting the right FS engine  Depends on business goals, type of content sources – structured vs. unstructured, licensed/subscriptions 19

Enterprising Solutions Benefits • Single master index • Quicker response times • No need to access original data sources • Relevancy algorithms applied uniformly • Dynamic navigators are available for all documents • Time savings • Searches many sources at one time • Combines results into a single results page • Quality of results • Client selects the sources to search • Minimum impact on the data silos • Only accessed when a user performs a query • Eliminates increased load crawling/indexing the data source 20

Enterprising Solutions Benefits (continued) • Improve productivity • Reduces number of searches executed to find relevant results • Save, reuse, schedule, and share effective search queries • Leverage security controls at queried source • Access repositories secured against crawls but can be accessed by search queries • Reduce costs • No additional capacity requirements for content index since its not crawled by search server • Most current content • Real time searches - as soon as the source is updated, the info is available to the searcher on the very next query • Increase awareness • Identify most relevant sources to search based on # of results each source produced 21

Enterprising Solutions FDA Case Study Success (Federated „Master Index‟ Search System) ACTIONS RESULT Started small with high ‘pain Increased productivity & popularity. points’. Modified business processes. Standardized nomenclature improved efficiencies. Users across organization Produced more timely & QUALITY could find content in silos. work products. Indexed structured & Grew from 1 repository of 500 docs unstructured content with to 50 with 30 million docs. Accessed document level security. on ‘need to know’ basis. Introduced standardized Reduced development time & costs. search web services into Increased mgmt & user acceptance. applications. Integrated in more applications. Increased user awareness Used more & content added. Search with training, newsletters & requirements now captured at meetings. BEGINNING of project development. 22

Enterprising Solutions Evaluation Criteria Overview  Identify Goals  Create an Effective Search Strategy  Collect Business Requirements  Conduct needs assessment  Work Closely with User Community 23

Evaluation Criteria Overview Enterprising Solutions (continued)  Define Features and Functions  Eliminate emotional decisions re: product, company or others using the product  High Precision  Return content relevant to user‘s focus  High Recall  Recall everything relevant to user‘s need  Thoroughly Research Products, Users & Product Reviewers 24

Enterprising Solutions Sample Evaluation Criteria Rating Criteria Importance Product #1 Product #1 Product #2 Product #2 (Rank 1-5) Score Weighted Score Score Weighted Score (0-100) (Rank x Score) (0-100) (Rank x Score) Ease of Use 5 85 425 70 350 Ability to Customize UI 1 80 80 65 65 Speed 5 90 450 85 425 De-duplication 4 75 300 75 300 Clustering 4 85 340 80 320 Help Functionality 3 70 210 0 0 Alerts 4 90 360 50 200 # of Searchable Sources 3 90 270 80 240 Save Selections/Citations 2 85 170 0 0 Security 4 90 360 85 340 Product Cost 5 75 375 85 425 Vendor Credibility 4 95 380 85 340 Total Weighted Score 1010 3720 760 3005 25 -Courtesy of Federated Search Report & Tool Kit

FSS Example Enterprising Solutions (uses FAST ESP – Vertical Search) Features of Interest 26

FSS Example Enterprising Solutions (uses MS & Vivisimo) Features of Interest 27

Enterprising Solutions FSS Example (uses Deep Web Technologies) Features of Interest 28

Enterprising Solutions FSS Example (uses Webfeat) Features of Interest 29

Digital Library FSS Example Enterprising Solutions http://www.calisphere.universityofcalifornia.edu/ Features of Interest 30

Digital Library FSS Example Enterprising Solutions http://www.calisphere.universityofcalifornia.edu 1 2 3 31

Enterprising Solutions FSS Example (LibraryFind® developed by Oregon State Univ Libraries) Features of Interest 32

Enterprising Solutions Semantic Federated Search (prototype by Collexis & Deep Web Technologies) SOURCES: •PubMed •NCI=Nat‘l Cancer Inst DeepWeb Technologies (a federated search provider) and •DTIC=Defense Tech. Info Ctr •PMC=PubMed Central Collexis (a developer of semantic search & knowledge •ScrDOEIB=DOE Info Bridge discovery solutions) teamed up to deliver the world’s first •Eurekalert=Science News semantic federated search. THESAURI Used: •MeSH •DTIC=Defense Tech. Info Ctr •How does semantic federated search work? •All results from your initial query are processed through one or more thesauri. (i.e., MeSH & DTIC.) •The system then returns terms that are found both in the top results and in the thesauri. 33

Enterprising Solutions Collexis & Deep Web Technologies (Search Results – screenshot 1) Unlike clustering, which simply lumps together words that are frequently found near each other, these terms are being suggested from an expert- developed thesaurus (taxonomy) in which 2429 hits terms are meaningfully & consistently organized. The longer the Semantic terms. blue bar, the more semantic evidence found for that term. 34

Enterprising Solutions Collexis & Deep Web Technologies (Search Results – screenshot 2) •Clicking on term “Mental Recall” from prior screen added term to search, reduced relevant hits to 3; & terms suggested are organized. •Thesaurus-based search will consistently suggest terms in the same organized way. •Clustering changes the way it organizes suggestions with every query. • Clustering tends to be useful for very broad, general or unpredictable content. *Thesaurus-based semantic search tends to be better when you are working consistently in knowledge domains, such as medicine, physics or electronics. 35

Enterprising Solutions Best Practices Strategically plan how to deliver your mission and just DO IT! Do proof of concept – demos can be deceiving Establish common set of standards & governance model Measure results by establishing key performance indicators Leverage lessons learned to reduce project cycles, increase trust and empower communities 36

Enterprising Solutions Future Vision Personalized Search • A simple, persistent box on a users‘ browser, cell, or entertainment screen that initiates a search based on what the user was doing, their previous keystrokes, & perhaps using historical data. Better Quality of Search Results • Number of results retrieved, Relevance Ranking, De-Duplication Enterprise Mashups • Combine real-time searching with social networking tools, maps, etc. Users build the index by their searching • Know Web pages people display, what‘s on them & what apps are showing up on users' computers 37

Enterprising Solutions Future Vision (continued) Query analysis & predictive modeling on the fly • Business users expect to access info behind company firewalls & from the larger web world using the same tools and consistency Improved Navigators, Facets, Clustering • Filter result sets dynamically for more relevant results Web of Interconnected Data • Automate analysis of database structures and cross-reference results. Ex.- Health site cross-references data from pharmaceutical companies with the latest findings from medical researchers Visualization Technologies 38 • Enable extreme-scale knowledge discovery

Enterprising Solutions Resources 1. Great resource for many Federated Search topics: www.federatedsearchblog.com – Author: Sol Lederman 2. Open Source & commercial search components & tools list: http://tinyurl.com/l3w8of 3. Federated Search Vendors: http://tinyurl.com/92s8qv 4. Deep Web Databases: http://tinyurl.com/yam3sw 5. Deep Web resources: http://www.internettutorials.net/deepweb.asp 6. Digital Image Resources on the Deep Web: http://tinyurl.com/46vcqp 7. Info on Vertical Search Engines: http://tinyurl.com/lpcufw 8. 50 Niche Search Engines: http://tinyurl.com/lukxwx 9. Library of Congress FS Portal Products/Vendors list: http://tinyurl.com/l6mdy8 10. Resources to Research & Mine the Deep Web: http://tinyurl.com/6g5768 39

Enterprising Solutions References 1) ―What‟s in a Name: Federated Search‖ – Miles Kehoe, New Idea Engineering, Inc,Vol. 4 No.4 8/07 2) “Federated Search Engine Article” - Online (Weston, Conn.) 28 no2 16-19 Mr/Ap 2004 (Reprint of article by Donna Fryer www.SearchitRight.com ) 3) “Growing Up With Federated Search” - by Walt Warnick, OSTI 4) “Sophisticated Yet Simple - The Technology Behind OSTI's E-print Network: Part 3” – Walt Warnick, OSTI 5) “Vertical Search Engines & the Deep Web” - Laura B. Cohen http://www.internettutorials.net/ 6) Blog: www.federatedsearchblog.com – by Sol Lederman 7) “Exploring a „Deep Web‟ that Google can‟t Grasp” - NYT 2-23-09 http://tinyurl.com/mvt42f 8) “Federated Search Primer, Part I-III” – by Sol Lederman 9) www.searchdoneright.com – by Vivisimo –Raoul – CEO & Cofounder 10) “Enterprise Search Grows Up‟”- Podcast from BizTalk 11) “Federation: Big Need, Still a Challenge” – Stephen Arnold, 4/25/08 12) “The Future of Federated Search or What Will the World Look Like in 10 Years” – Rich Turner 13) “Federated Search Report & Tool Kit” – Jill Hurst-Wahl, 10/08, © Free Pint Limited 2008 40

Enterprising Solutions QUESTIONS 41

Enterprising Solutions THANK YOU! Helen L. Mitchell Curtis Principal Enterprising Solutions hmitchell5@gmail.com 410-472-4631(w) 410-259-7766(m) 42 42

Enterprising Solutions Enterprising Solutions “Results Driven…Exceeding Expectations” 43

http://federatedsearchblog.com	145
http://bblearn.missouri.edu	63
http://www.slideshare.net	8
http://feeds.feedburner.com	1
http://www.lmodules.com	1

Federated Search Webinar for SLA (Special Libraries Assoc.)

by Helen Mitchell on Sep 13, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 218

Statistics

Federated Search Webinar for SLA (Special Libraries Assoc.) — Presentation Transcript