Using metadata repositories with search Enterprise Search Summit 5/14/2007 Jean Graef The Montague Institute Jean.graef at montague.com (413) 367-0245

Topics Role of metadata in search & discovery Where is metadata stored? What is a metadata repository? How to create a metadata repository Using metadata within search Commercial products How to sell metadata repositories

Role of metadata in search & discovery

Where is metadata stored?

What is a metadata repository?

How to create a metadata repository

Using metadata within search

Commercial products

How to sell metadata repositories

1. Metadata in search & discovery Smarter full text search Search by attribute Faceted navigation Fielded search Legacy navigation tools Card catalogs & subject guides A – Z indexes, tables of contents, glossaries, lists Bibliographic databases Content networks

Smarter full text search

Search by attribute

Faceted navigation

Fielded search

Legacy navigation tools

Card catalogs & subject guides

A – Z indexes, tables of contents, glossaries, lists

Bibliographic databases

Content networks

Smarter full text search Results summaries “Best Bets” Synonyms Topic browse “See also” references

Results summaries

“Best Bets”

Synonyms

Topic browse

“See also” references

Out of the box Results summaries

Topics, thesaurus

Search by attribute

Legacy retrieval tools: Before

Legacy retrieval tools: After

Content networks

2. Where is metadata stored? Embedded in document Embedded in application In XML file In external database

Embedded in document

Embedded in application

In XML file

In external database

Embedded in document

Embedded in application

Embedded in application

In spreadsheet or database

Why isolate metadata? Easier to standardize & localize Easier to update Easier to share with multiple applications Full text search ERP/transaction applications CMS and DMS applications Legacy retrieval tools Insurance against technological change

Easier to standardize & localize

Easier to update

Easier to share with multiple applications

Full text search

ERP/transaction applications

CMS and DMS applications

Legacy retrieval tools

Insurance against technological change

Who uses metadata? Programs Search engines Other applications Humans Authors Site administrators Indexers Readers/visitors

Programs

Search engines

Other applications

Humans

Authors

Site administrators

Indexers

Readers/visitors

3. A metadata repository is… A data storage structure that describes the characteristics of information objects as an aid in identification, discovery, assessment, and management. Able to be read and updated by both humans and computers. Example: <Title>Web site makeover</title>

A data storage structure that describes the characteristics of information objects as an aid in identification, discovery, assessment, and management.

Able to be read and updated by both humans and computers.

Example: <Title>Web site makeover</title>

Two familiar examples Library card catalog Data = book, journal title Metadata = author, title, subject, call number Bibliographic database Data = journal article Metadata = author, title, pub date, publisher, keywords

Library card catalog

Data = book, journal title

Metadata = author, title, subject, call number

Bibliographic database

Data = journal article

Metadata = author, title, pub date, publisher, keywords

Library catalog (Endeca)

Bibliographic database

Two kinds Data management tools Catalog of business definitions, data processing systems, & application components. ER diagrams Rochade, Informatica Classification (semantic) tools Reference for names, terms, topics, and other data used to classify content objects Collaboration, display of terms & relationships Products we discuss here

Data management tools

Catalog of business definitions, data processing systems, & application components.

ER diagrams

Rochade, Informatica

Classification (semantic) tools

Reference for names, terms, topics, and other data used to classify content objects

Collaboration, display of terms & relationships

Products we discuss here

Data management tool

Classification tool

Metadata repository for search Controlled vocabulary Thesaurus Link to content object (e.g. URL, file path) Other attributes Language Geographic region Industry Search is more than a search engine!

Controlled vocabulary

Thesaurus

Link to content object (e.g. URL, file path)

Other attributes

Language

Geographic region

Industry

Search is more than a search engine!

4. Creating a metadata repository Search terms URL term Thesaurus Term BT/NT RT Use Controlled Vocabulary term Card catalog Author Title Subject Contact database Name Password Address

Search terms

Controlled vocabulary

Thesaurus

Metadata repository segment

Where does data come from? System (unique ID, date saved, user ID) User input (free text) User input (selected from list) Program generated (assigned from rules) Database lookup (e.g. employee directory) Licensed from creator or vendor

System (unique ID, date saved, user ID)

User input (free text)

User input (selected from list)

Program generated (assigned from rules)

Database lookup (e.g. employee directory)

Licensed from creator or vendor

Metadata sources & uses

5. Using metadata in search Export as XML Access via ODBC Vendor API Web services

Export as XML

Access via ODBC

Vendor API

Web services

Export XML Metadata repository XSL Style Sheet Search engine Related terms Use terms Preferred names Use names XML File

XML Style sheet

XML thesaurus data

Access via ODBC indexes Metadata Repository Search Engine Index Search Results List Search Engine

Access via ODBC

Access via ODBC

Product checklist Easy to use for both indexers & laymen Basic thesaurus fields & relationships BT, NT, Use/Use For, RT Definitions, scope notes, source Multiple vocabularies Polyhierarchy Error checking Duplicates No x-refs to dead-end terms

Easy to use for both indexers & laymen

Basic thesaurus fields & relationships

BT, NT, Use/Use For, RT

Definitions, scope notes, source

Multiple vocabularies

Polyhierarchy

Error checking

Duplicates

No x-refs to dead-end terms

Product checklist Workflow features Candidate terms, approvals Import/export formats Statistics/reports Terms used in queries Add new fields & relationships Robust daabase search features Boolean, truncated, & phrase search

Workflow features

Candidate terms, approvals

Import/export formats

Statistics/reports

Terms used in queries

Add new fields & relationships

Robust daabase search features

Boolean, truncated, & phrase search

6. Enterprise-grade products Data Harmony Schemalogic Factiva Synaptica Wordmap

Data Harmony

Schemalogic

Factiva Synaptica

Wordmap

Data Harmony Thesaurus tool + rules-based indexer Create & manage thesaurus terms Call indexer to assign terms to documents Has interfaced with: Documentum Sharepoint Index document from within Sharepoint MarkLogic Verity & Ultraseek

Thesaurus tool + rules-based indexer

Create & manage thesaurus terms

Call indexer to assign terms to documents

Has interfaced with:

Documentum

Sharepoint

Index document from within Sharepoint

MarkLogic

Verity & Ultraseek

Data Harmony Multi-lingual capabilities Time-limited trial download $100,000 + for both thesaurus & indexer pricing based on the number of servers Customer base: government agencies, corporations

Multi-lingual capabilities

Time-limited trial download

$100,000 + for both thesaurus & indexer

pricing based on the number of servers

Customer base: government agencies, corporations

Data Harmony file formats

Schemalogic Thesaurus, vocabularies, schemas Manage terms & vocabularies Interfaces for both indexers & business people Has interfaced with: Auto-categorization tools (Teragram, Nstein) Search engines: OmniFind, FAST, Verity, Autonomy Sharepoint

Thesaurus, vocabularies, schemas

Manage terms & vocabularies

Interfaces for both indexers & business people

Has interfaced with:

Auto-categorization tools (Teragram, Nstein)

Search engines: OmniFind, FAST, Verity, Autonomy

Sharepoint

Schemalogic Multi-lingual capabilities $50,000 - $750,000 for software + services pricing based on the number of seats & servers Customers: Commercial publishers, corporations

Multi-lingual capabilities

$50,000 - $750,000 for software + services

pricing based on the number of seats & servers

Customers: Commercial publishers, corporations

Schemalogic: Indexer’s view

Schemalogic: Business person’s view

Factiva Synaptica Thesaurus, vocabularies, “warehouse” Manage terms & vocabularies Index Management Service (classification) Multi-lingual capabilities $100,000 + Customers: Commercial publishers, corporations

Thesaurus, vocabularies, “warehouse”

Manage terms & vocabularies

Index Management Service (classification)

Multi-lingual capabilities

$100,000 +

Customers: Commercial publishers, corporations

Factiva Synaptica

Wordmap Thesaurus manager, tagger, auto-classifier, topic browse (navigator) Manage terms & vocabularies Assign terms Clusters documents into categories Yellow-pages style directory with x-refs Multi-lingual capability

Thesaurus manager, tagger, auto-classifier, topic browse (navigator)

Manage terms & vocabularies

Assign terms

Clusters documents into categories

Yellow-pages style directory with x-refs

Multi-lingual capability

Wordmap: Indexer’s view

Wordmap: User’s view

7. Selling metadata repositories Time saved by users in finding information Staff time saved by self service Staff time saved in preparing & publishing content Increased revenues from information products & services

Time saved by users in finding information

Staff time saved by self service

Staff time saved in preparing & publishing content

Increased revenues from information products & services

Metadata repository lab Enter your own data Thesaurus Map terms in two different thesauri Digital assets (documents, images, etc) Names: people, products, organizations Relationships Authored by Subject of Made by Acquired by Used by

Enter your own data

Thesaurus

Map terms in two different thesauri

Digital assets (documents, images, etc)

Names: people, products, organizations

Relationships

Authored by

Subject of

Made by

Acquired by

Used by

Metadata repository Lab Manual data entry or import Navigation tools Fielded (faceted) search Bibliography Glossary Back-of-the-book style A – Z index Subject hierarchy (table of contents) Custom lists & export formats (XML)

Manual data entry or import

Navigation tools

Fielded (faceted) search

Bibliography

Glossary

Back-of-the-book style A – Z index

Subject hierarchy (table of contents)

Custom lists & export formats (XML)

A – Z index

More info Montague Institute Review http://www.montague.com/review/review.html