Semantic Search using RDF Metadata (SemTech 2005)

Semantic Search Using RDF Metadata Semantic Technology Conference 2005 8 March 2005 Bradley P. Allen Siderean Software, Inc.

Overview
Semantic search
Motivation
Enterprise adoption
Semantic search using RDF
Examples
Lessons
Directions

Problem
“ We have to understand what information we have and organize it,’ says [Santa Clara Co. CIO] Ajmani, who estimates that saving each employee an hour a month spent looking for information would save millions of dollars.” [Information Week, 1/19/04]
“… typical enterprise floundering in a sea of information … too many repositories, each with its own set of applications.” [IDC, 2004]
“ The search capabilities on most company and content-oriented Web sites are as bad now as they were several years ago.” [eWeek, 1/26/04]

Portal-driven demand for a better solution
“ A portal provides an integrated information source for our internal process users or external customers”
“ Now we have to architect the information related to business processes differently to search across multiple repositories”
But they lack tools and applications that support this

Current solutions
Enterprise search, portals, knowledge management and content management systems lashed up in ad hoc architectures
Doesn’t unify data and content
Doesn’t provide context or scope
Too many results (requires searching the answer to the original search)

Why semantic search?
Explicitly represented knowledge can
Unify access to both content and data
Create context and frames of reference
Intellectual contributions that inform the search process must be captured
The answer should include the question

Semantic search – some definitions
Search: the process of retrieving objects matching a given query
Semantic search:
Search that uses an explicit representation of knowledge to retrieve, organize or display objects matching a query
Search that transparently renders human insight into the nature of matches

Benefits in the enterprise
Addresses pervasive frustration with enterprise search
Let users
Find high-value information quickly
Add more value to it, and
Share it with others
Aligns information to business needs

Roots
Parametric search
Query by example
Retrieval by reformulation
Rabbit, Argon
Work in existing enterprise search and knowledge management
Autonomy, Semio

Semantic search requires metadata
Ontologies
Specifications of how to represent classes, instances and their properties
Sometimes called “vocabularies”
Controlled vocabularies
Terms for saying what something is about
Also called “taxonomies” and “thesauri”
Instances
Descriptions of resources
Application profiles
Specifications of which classes and properties are useful and how they are to be used in an application

Current metadata solutions are costly
Much custom development done
Not easy to tag or incorporate content into the desired structures
No easy way for groups creating the vocabularies to deliver them to production environments
Perceived lack of tools
Point solutions not well integrated
Existing platform solutions closed

Metadata in today’s enterprises
From thirty interviews conducted with Fortune 1000 organizations during Fall 2004
Use of metadata not yet widespread but emerging
Understanding varies widely across enterprises
Three basic approaches
Top down, bottom up, and give up

Approach: top down
CEO says “We must be an information-driven company”
“ Corporate controlled vocabulary that all divisions will use”
Typically based on Dublin Core
Used for subject tagging
The effort is multi-year, ROI hard to track, and may not be implemented or adopted widely

Approach: bottom up
Groups determine their vocabulary while describing their process
Often in a collaboration environment
Light tagging of content when it is created or when the content is published to a portal
Again, based on Dublin Core and their own controlled vocabularies

Approach: give up
Assumption: too difficult to create metadata from existing content
“ We can’t ever hope to organize this morass of content, so let’s put in a search appliance like Google”
“ Our internal needs are like the public internet and users are familiar with Google searches”
But still feel that metadata would improve matters, particularly within business units

Don’t give up!
RDF can make metadata use easier and less costly
An open standard for metadata reduces cost and avoids technology and vendor lock-in
A “universal solvent” for data and content
A platform for reuse and sharing

Building semantic search systems with RDF
Define/reuse ontologies expressed in RDF(S)
Classes for defining instances and controlled vocabularies
Properties for facets and additional attributes
Import/transform instances into an RDF representation
Resources referred to via URIs
Content and controlled vocabularies
Write application profiles in terms of RDF

Types of semantic search in RDF
Searching for RDF
Swoogle
Adding value to search using RDF
TAP, FOAFNaut
Searching resources using RDF
Edutella, Seamark

Swoogle: Searching for RDF
Crawling for SW documents
Leverages Google indexing
And structure of key document types
Searching for ontologies and instance data
Mostly relevant to people bulding semantic applications rather than general users

TAP: Adding value to search using RDF
Layering “related items” on top of traditional Web search
Arm’s length integration and value-add for traditional Web search

FOAFNaut: Adding value to search using RDF
Specialized search and visualization over FOAF networks
Introducing the notion of social aspects of finding information

Edutella: Searching resources using RDF
P2P architecture federating collections of learning objects
Work on distributing RDF queries using schema information
RDF as a more natural representation for learning objects than IEEE LOM

Seamark: Searching resources using RDF
Using ontologies and taxonomies to define navigation over specific collections
First implementation of faceted navigation using RDF

Faceted navigation as a type of semantic search
Metadata may be faceted, i.e., includes properties whose ranges form a near-orthogonal set of controlled vocabularies
Creator: Dickens, Charles
Subject: Arsenic, Antimony
Location: World > U.S. > California > Venice
Facets form a frame of reference for information overview, access and discovery
Other properties serve as landmarks and cues

Case study: DC 2003 Online Proceedings
Further the goals of the Dublin Core Metadata Initiative (DCMI) by providing DC-centric faceted navigation of online proceedings

Project timeline
July 2003
Initial experiment using DC 2002 site
August 2003
Initial proposal to DCMI
Iterative prototyping involving
Selection and development of ontologies
Generation of instance metadata
Specification of application profile
Conversion of DC2003 dataset into navigable RDF
Elapsed time to implement: 1 day
September 2003
Design and editing of controlled vocabulary
Final iterations on site pages
Launch at conference

Ontology
Reused ontologies and metadata vocabularies
Papers and posters: Dublin Core
Creators: Friend Of A Friend (FOAF)
Subjects: Thesaurus Interchange Format (TIF)
Added relatively few properties and classes in a conference ontology
Events
Tracks

Ontology for conferences
<s:Class rdf:about="&dcconf;Event">
<s:label>Presentation</s:label>
</s:Class>
<s:Class rdf:about="&dcconf;Paper">
<s:label>Paper</s:label>
<s:subClassOf rdf:resource="&dcconf;Event"/>
</s:Class>
<s:Class rdf:about="&dcconf;Track">
<s:label>Conference Track</s:label>
</s:Class>
<rdf:Property rdf:about="&dcconf;track">
<s:label>Track</s:label>
<s:comment>The track that the given paper is in.</s:comment>
<s:domain rdf:resource="&dcconf;Event" />
<s:range rdf:resource="&dcconf;Track" />
</rdf:Property>

Controlled vocabulary
Author-assigned keywords used as source materials
Combined author-assigned with editorial judgment about the CV terms and structure

Seed thesaurus

Wrapping author-assigned keywords
<tif:Term rdf:about="&dcconf2003;Relational_Database">
<tif:value>Relational Database</tif:value>
<tifs:USE rdf:resource="&dcconf2003;Relational_Databases" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;relationship_metadata">
<tif:value>Relationship metadata</tif:value>
<tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;requirements">
<tif:value>Requirements</tif:value>
</tif:Term>
<tif:Term rdf:about="&dcconf2003;resource_discovery">
<tif:value>Resource discovery</tif:value>
<tifs:BT rdf:resource="&dcconf2003;Discovery" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;resource-level_metadata">
<tif:value>Resource-level metadata</tif:value>
</tif:Term>
<tif:Term rdf:about="&dcconf2003;SCORM">
<tif:value>SCORM</tif:value>
<tifs:USE rdf:resource="&dcconf2003;Sharable_Content_Object_Reference_Model_SCORM" />
</tif:Term>

Adding editorial control
<tif:Term rdf:about="&dcconf2003;Domain_Metadata">
<tif:value>Domain Metadata</tif:value>
<tifs:BT rdf:resource="&dcconf2003;Applications" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;Governments">
<tif:value>Governments</tif:value>
<tifs:BT rdf:resource="&dcconf2003;Organizations_and_Domains" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;Federal_Geographic_Data_Committee_Metadata">
<tif:value>Federal Geographic Data Committee Metadata</tif:value>
<tifs:RT rdf:resource="&dcconf2003;Governments" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;Geospatial_Metadata">
<tif:value>Geospatial Metadata</tif:value>
<tifs:RT rdf:resource="&dcconf2003;Organizations_and_Domains" />
</tif:Term>
<tif:Term rdf:about="&dcconf2003;Government_Agency_Metadata">
<tif:value>Government Agency Metadata</tif:value>
<tifs:RT rdf:resource="&dcconf2003;Governments" />
</tif:Term>

Instance metadata
Paper and poster metadata automatically extracted from author submissions
Ad hoc Perl script
Manual review and cleanup of generated RDF
Mostly Dublin Core with some application-specific properties
Creator and organization metadata manually collated from paper and poster metadata
Represented in FOAF (but not in the manner in which FOAF is typically used)

Papers and posters
<dcconf:Paper rdf:about="http://www.siderean.com/dc2003/103_paper-22.pdf"> <seamark:texturl>http://www.siderean.com/dc2003/103_paper-22.pdf</seamark:texturl>
<rdf:type rdf:resource="&dcconf;Event"/>
<dcconf:track rdf:resource="&dcconf;Interoperability" />
<dc:title>Two Paths to Interoperable Metadata</dc:title>
<dc:creator rdf:resource="&dcconf;Godby_Carol" />
<dc:creator rdf:resource="&dcconf;Smith_Devon" />
<dc:creator rdf:resource="&dcconf;Childress_Eric" />
<dc:description> This paper describes a prototype for a Web service that translates between pairs of metadata schemas. Despite a current trend toward encoding in XML and XSLT, we present arguments for a design that features a more distinct separation of syntax from semantics. The result is a system that auomates routine processes, has a well-defined place for human input, and achieves a clean separation of the document data model, the document translations, and the machinery of the application. </dc:description>
<dc:subject rdf:resource="&dcconf2003;metadata_schema_translation" />
<dcconf:authorKeyword rdf:resource="&dcconf2003;metadata_schema_translation" />
<dc:subject rdf:resource="&dcconf2003;Web_services" />
<dcconf:authorKeyword rdf:resource="&dcconf2003;Web_services" />
<dc:subject rdf:resource="&dcconf2003;communities_of_practice" />
<dcconf:authorKeyword rdf:resource="&dcconf2003;communities_of_practice" />
</dcconf:Paper>

Creators and organizations
<foaf:Person rdf:about="&dcconf;Greenberg_Jane">
<foaf:name>Greenberg, Jane</foaf:name>
<foaf:mbox rdf:resource="mailto:janeg@ils.unc.edu" />
<foaf:memberOf rdf:resource="&dcconf;University_of_North_Carolina_at_Chapel_Hill" />
<foaf:publication rdf:resource="http://www.siderean.com/dc2003/202_Paper82-color-NEW.pdf" />
</foaf:Person>
<foaf:Organization rdf:about="&dcconf;University_of_North_Carolina_at_Chapel_Hill">
<foaf:name>University of North Carolina at Chapel Hill, USA</foaf:name>
<foaf:member rdf:resource="&dcconf;Greenberg_Jane" /> <foaf:member rdf:resource="&dcconf;Crystal_Abe" />
</foaf:Organization>

Application profile
Expressed in XRBR (XML For Retrieval By Reformulation)
Specifies a view over (possibly heterogeneous) RDF schemas with hints as to its interpretation and use for faceted navigation
Provides a language for query reformulation and refinement in the context of navigation
Query: “give me all resources where…” + advice
Response: result set + suggested query refinements + original query

Application profile: specifying facets
<xrbr:query xmlns:xrbr="http://www.siderean.com/2001/10/xrbr/" item-type="http://www.dcmi.org/dcconf/objects#Event" sort-dimension="title" >
<xrbr:hint flattenresults="yes" startpagecolumns="4"/>
<xrbr:dimensions>
<xrbr:dimension name="title"
predicate="http://purl.org/dc/elements/1.1/title">
<xrbr:hint textsearch="yes" label="Title" function="itemlabel"/>
<xrbr:return />
</xrbr:dimension>
<xrbr:dimension name="description"
predicate="http://purl.org/dc/elements/1.1/description">
<xrbr:hint textsearch="yes" label="Description"
function="itemdescription"/>
<xrbr:return />
</xrbr:dimension>
…
</xrbr:dimensions>
</xrbr:query>

Application profile: specifying hierarchical facets
…
<xrbr:dimension name="BT1" predicate="http://purl.org/dc/elements/1.1/subject"
display-predicate="http://www.w3c.rl.ac.uk/2003/07/31-tif#value"
root-resource="http://www.dcmi.org/dcconf/2003#Organizations_and_Domains"
ancestor-predicate="http://www.w3c.rl.ac.uk/2003/07/31-tif-simple#BT" >
<xrbr:hint label="Organizations and Domains"
facet="yes“
scopenote="Sectors, languages, special literatures or communities that use metadata" />
<xrbr:suggestions count="7" />
</xrbr:dimension>
…

Application profile: flattening graphs
…
<xrbr:structure name="creator" predicate="http://purl.org/dc/elements/1.1/creator">
<xrbr:dimension name="creatorname"
predicate="http://xmlns.com/foaf/0.1/#name">
<xrbr:hint label="Author" textsearch="yes"/>
<xrbr:return />
</xrbr:dimension>
<xrbr:dimension name="creatororg“
predicate="http://xmlns.com/foaf/0.1/#memberOf"
display-predicate="http://xmlns.com/foaf/0.1/#name">
<xrbr:hint label="Author Affiliation" />
<xrbr:return />
</xrbr:dimension>
</xrbr:structure>
…

Automatically generated interface

Alternate view: creators

Alternate view: subjects

Site start page

Site drilldown

Case study: Environmental Health News
Aggregating news stories from the Web
Semi-automated metadata creation by a team of subject matter experts and editors
Semantic search to design custom feeds

Case study: Gateway to Educational Materials
Aggregating learning objects from members of the GEM Consortium
Embedding semantic search into a portal

Case study: NASA JPL
Project information aggregated from content and data repositories
Using and extending taxonomies
Exploiting document type/genre

Related work in RDF
OCLC
Metadata Switch
MIT
Simile
Longwell
Haystack
Aduna
Sesame
Ontoprise
OntoSeek
Nature Publishing Group
Urchin

Issues
Scale: must be commensurate with expectations and requirements from traditional web and enterprise search
Number of objects, feeds: 10 6 to 10 9
Ingest rates: ~ 10 3 – 10 4 triples/sec, how many per resource?
Tagging: where and when?
Latency: < 0.5 sec user time regardless of application
Retrieval algorithms: many alternatives still being explored
Federated services vs. centralized servers
Relationship to relevance ranking
Support for aggregate and text search operators in RDF query
Usability: lots of work to be done to validate benefits
Navigation
Precision and recall
Visualization
Security, trust and provenance: just beginning to understand

Lessons
Balanced incremental approach
Leverage metadata and indices at hand
Exploit statistics where desirable
But layer a framework on top to structure the statistics
Significant mileage from very simple frameworks

Lessons: ontologies
Don’t do: assume you have to build elaborate OWL ontologies
Don’t have to boil the ocean to get the benefits
OWL DL, are OWL Full are overkill for this class of application
Do: Tiny Ontologies Stitched Together
RDF Schema with a smattering of RDF/OWL properties (e.g., owl:inverse)
Start with DC + SKOS + FOAF

Lessons: controlled vocabularies
Don’t do: huge monolithic taxonomies
Unless they are ready at hand and can be reused largely without modification
Do: bite-sized controlled vocabularies that exploit faceted approaches
4 facets x 10 terms per facet versus 10 4 terms in a single taxonomy
Start with flat term lists
Add BT/NT/RT relationships over time

Lessons: instances
Manual creation
Don’t do: exhaustive author creation of metadata
Do: community annotation and tagging
(Semi-)automated creation
Don’t do: assume elaborate information extraction based on NLP, subject tagging and categorization
Do: quick and dirty NEE or better yet, stick to readily available asset and relational metadata (date, creator, document type/genre)
Much of the benefit at a fraction of the effort

Application profiles
Metadata is increasingly pervasive
The way to leverage existing information infrastructure
Exploit “on-demand” information integration feature of RDF
DB + XML -> XLST - > RDF(S)

The big question: statistics vs. knowledge
Statistics can’t deliver everything
Alan Kay’s puppy analogy
Vitanyi work on “Google learning”
On the other hand, knowledge is dearly won
CYC
Need a balance that enables adoption without losing the benefits
Lessons from
Statistics vs. knowledge in NLP
Expert systems

Future directions
User tagging + RDF: the killer SW application?
The rehabilitation of metadata in the social software community
The re-emergence of RSS/RDF
“ Folksonomy”-driven collaborative search
Del.icio.us, Flickr, CiteULike
Growth of the SW compared to historical growth of the Web: it’s 1994 all over again

Summary
Semantic search has a role in today’s enterprises
RDF provides a framework that can ease adoption and encourage innovation in semantic search
The future for enterprise and consumer use looks bright

Semantic Search using RDF Metadata (SemTech 2005)

by Bradley Allen on Oct 03, 2011

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics

Semantic Search using RDF Metadata (SemTech 2005) Presentation Transcript