MetaSearch vs Harvesting and Indexing

MetaSearch vs Harvesting and Indexing Lukas Koster Library of the University of Amsterdam -- http://commonplace.net 2009 http://www.flickr.com/photos/donpezzano/3044975399/

So many databases to search

MetaSearch – Federated Search Z39.50 SRU Proprietary Search Translate search syntax MARC21 MARCXML DC Conversion Merging Deduplication Ranking (First 30 per DB) Results Database Connectors MetaSearch tool Databases Searching and Data fetching: One integrated interdependent on-the-fly procedure Search Engine

Technical bottlenecks Z39.50 SRU Proprietary Search Translate search syntax MARC21 MARCXML DC Conversion Merging Deduplication Ranking (First 30 per DB) Results Database Connectors MetaSearch tool Databases Connection Access Authorisation Search Engine

Technical bottlenecks
Changes in
Remote database server IP address
Remote database server hostname
Remote database server configuration
Remote database authentication
Firewall
Database system
Network

MetaSearch limitations
Differences in searches, indexes
Author
Subject
Multiple languages
Speed (slowness)
Limited number of searchable databases
Not all results in first set
Relevance

Author searches
Variations in author name storage formats
Henry James
James, Henry
James, H.
H.James
Which Henry James?
Or is it: Henry, J./James Henry ?
Variations in supported search formats
Only one?
All of the above?

Variations in author names

Subject searches
Different qualification, keyword schemes per database
LoC subject Headings
Dutch Basic Classification
Local subject schemes
Different use of subjects per database
Cooking
Cookery
Food
Different use of subjects within one database
Errors

Multilingual searches
All words searches
Subject searches
English “cooking”
Japanese “???”
Title searches
Translations (We need FRBR!)
Author searches (historical names)
See: Erasmus

All processing on the fly
Issues, dependent on each other:
Speed (slowness)
Limited number of searchable databases
Relevance

Speed (slowness)
Dependent on
Search term transformation
Response time of external databases
Speed of internet connection
Conversion of results to presentation format
Merging of results
Deduplication of results
Relevance ranking

Limited number of databases
Searching too many databases takes too long
Local processing time influenced by
Merging ( takes time )
Deduplication ( takes time )
Ranking ( takes time )

Merging, deduplication, ranking of all results takes too long
Only first 30 or so of each database are processed initially
Get more: next 30 per database are fetched and processed

Relevance
Dependent on default sort order (relevance?, date?) of each external database
Dependent on default ranking mechanism of each database
Local ranking initially performed on first batches of 30 records per database
After additional fetching records, ranking is done again:
Initial top results may go down

Solution?
Don’t rank
Don’t deduplicate
Don’t merge (in advance)
If you don’t merge, there is no point in deduplicating or ranking!!
“ Does not make much sense anyway”
“ Does not work always anyway”
“ So, you have separate lists that you can merge later on”

Search with MetaSearch

Translate search syntax on the fly

Fetching results

Conversion of results on the fly

Results with MetaSearch

Harvesting and indexing Search Normalising Indexing Ranking Results Central index H&I tool Databases Harvesting Searching and Data fetching: Two completely separate procedures Search Engine

Advantages of H&I
Speed
No maximum number of searchable databases
All results in first set
No differences in searches, indexes
Relevance
Fewer technical bottlenecks
Central index always available in case of connection problem

H&I: Aquabrowser

H&I: Primo

MetaSearch = “Just in time”
Bookshop – Central Book Deposit
Always order on request
Risk of logistics problems
http://www.flickr.com/photos/stijnnieuwendijk/125159282/

H&I = “just in case”
Bookshop with large stock
Customers always find something
Maybe not the most recent stuff
http://www.flickr.com/photos/brewbooks/2131521680/

Images
http://www.flickr.com/photos/donpezzano/3044975399/
http://www.flickr.com/photos/halighalie/663414371/
http://www.flickr.com/photos/notionscapital/2280408255/
http://www.flickr.com/photos/giveawayboy/2691195763/
http://www.flickr.com/photos/joshb/444529511/
http://www.flickr.com/photos/eaglelover2006/3168378578/
http://www.flickr.com/photos/robbie73/3387189144/
http://www.flickr.com/photos/saralparker/2602254206/
http://www.flickr.com/photos/manchesterlibrary/2034771121/
http://www.flickr.com/photos/bk/158637798/
http://www.flickr.com/photos/saamiam/3802869384/
http://www.flickr.com/photos/roboppy/37024023/

http://commonplace.net	445
http://www.slideshare.net	21
http://www.bibnet.lu	15
http://translate.googleusercontent.com	6
http://paper.li	2
http://commonplace.net.	1
http://webcache.googleusercontent.com	1

MetaSearch vs Harvesting and Indexing

by Lukas Koster on Oct 16, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 491

Statistics

MetaSearch vs Harvesting and Indexing — Presentation Transcript