MetaSearch vs Harvesting and Indexing

MetaSearch vs Harvesting and Indexing Lukas Koster Library of the University of Amsterdam -- http://commonplace.net 2009 http://www.flickr.com/photos/donpezzano/3044975399/

So many databases to search

MetaSearch – Federated Search Z39.50 SRU Proprietary Search Translate search syntax MARC21 MARCXML DC Conversion Merging Deduplication Ranking (First 30 per DB) Results Database Connectors MetaSearch tool Databases Searching and Data fetching: One integrated interdependent on-the-fly procedure Search Engine

Technical bottlenecks Z39.50 SRU Proprietary Search Translate search syntax MARC21 MARCXML DC Conversion Merging Deduplication Ranking (First 30 per DB) Results Database Connectors MetaSearch tool Databases Connection Access Authorisation Search Engine

Technical bottlenecks

Changes in

Remote database server IP address

Remote database server hostname

Remote database server configuration

Remote database authentication

Firewall

Database system

Network

MetaSearch limitations

Differences in searches, indexes

Author

Subject

Multiple languages

Speed (slowness)

Limited number of searchable databases

Not all results in first set

Relevance

Author searches

Variations in author name storage formats

Henry James

James, Henry

James, H.

H.James

Which Henry James?

Or is it: Henry, J./James Henry ?

Variations in supported search formats

Only one?

All of the above?

Variations in author names

Subject searches

Different qualification, keyword schemes per database

LoC subject Headings

Dutch Basic Classification

Local subject schemes

Different use of subjects per database

Cooking

Cookery

Food

Different use of subjects within one database

Errors

Multilingual searches

All words searches

Subject searches

English “cooking”

Japanese “???”

Title searches

Translations (We need FRBR!)

Author searches (historical names)

See: Erasmus

All processing on the fly

Issues, dependent on each other:

Speed (slowness)

Limited number of searchable databases

Relevance

Speed (slowness)

Dependent on

Search term transformation

Response time of external databases

Speed of internet connection

Conversion of results to presentation format

Merging of results

Deduplication of results

Relevance ranking

Limited number of databases

Searching too many databases takes too long

Local processing time influenced by

Merging ( takes time )

Deduplication ( takes time )

Ranking ( takes time )

Merging, deduplication, ranking of all results takes too long

Only first 30 or so of each database are processed initially

Get more: next 30 per database are fetched and processed

Relevance

Dependent on default sort order (relevance?, date?) of each external database

Dependent on default ranking mechanism of each database

Local ranking initially performed on first batches of 30 records per database

After additional fetching records, ranking is done again:

Initial top results may go down

Solution?

Don’t rank

Don’t deduplicate

Don’t merge (in advance)

If you don’t merge, there is no point in deduplicating or ranking!!

“ Does not make much sense anyway”

“ Does not work always anyway”

“ So, you have separate lists that you can merge later on”

Search with MetaSearch

Translate search syntax on the fly

Fetching results

Conversion of results on the fly

Results with MetaSearch

Harvesting and indexing Search Normalising Indexing Ranking Results Central index H&I tool Databases Harvesting Searching and Data fetching: Two completely separate procedures Search Engine

Advantages of H&I