Making the Web Searchable Peter Mika Researcher, Data Architect Yahoo! Research

Yahoo! Research (research.yahoo.com)

Yahoo! Research Barcelona Established January, 2006 Led by Ricardo Baeza-Yates Research areas Web Mining content, structure, usage Distributed Web retrieval Multimedia retrieval NLP and Semantics

Established January, 2006

Led by Ricardo Baeza-Yates

Research areas

Web Mining

content, structure, usage

Distributed Web retrieval

Multimedia retrieval

NLP and Semantics

Yahoo! by numbers (April, 2007) There are approximately 500 million users of Yahoo! branded services, meaning we reach 50 percent – or 1 out of every 2 users – online, the largest audience on the Internet (Yahoo! Internal Data). Yahoo! is the most visited site online with nearly 4 billion visits and an average of 30 visits per user per month in the U.S. and leads all competitors in audience reach, frequency and engagement (comScore Media Metrix, US, Feb. 2007). Yahoo! accounts for the largest share of time Americans spend on the Internet with 12 percent (comScore Media Metrix, US, Feb. 2007) and approximately 8 percent of the world’s online time (comScore WorldMetrix, Feb. 2007). Yahoo! is the #1 home page with 85 million average daily visitors on Yahoo! homepages around the world, an increase of nearly 5 million visitors in a month (comScore WorldMetrix, Feb. 2007). Yahoo!’s social media properties (Flickr, delicious, Answers, 360, Video, MyBlogLog, Jumpcut and Bix) have 115 million unique visitors worldwide (comScore WorldMetrix, Feb. 2007). Yahoo! Answers is the largest collection of human knowledge on the Web with more than 90 million unique users and 250 million answers worldwide (Yahoo! Internal Data). There are more than 450 million photos in Flickr in total and 1 million photos are uploaded daily. 80 percent of the photos are public (Yahoo! Internal Data). Yahoo! Mail is the #1 Web mail provider in the world with 243 million users (comScore WorldMetrix, Feb. 2007) and nearly 80 million users in the U.S. (comScore Media Metrix, US, Feb. 2007) Interoperability between Yahoo! Messenger and Windows Live Messenger has formed the largest IM community approaching 350 million user accounts (Yahoo! Internal Data). Yahoo! Messenger is the most popular in time spent with an average of 50 minutes per user, per day (comScore WorldMetrix, Feb. 2007). Nearly 1 in 10 Internet users is a member of a Yahoo! Groups (Yahoo! Internal Data). Yahoo! is one of only 26 companies to be on both the Fortune 500 list and the Fortune’s “Best Place to Work” List (2006).

There are approximately 500 million users of Yahoo! branded services, meaning we reach 50 percent – or 1 out of every 2 users – online, the largest audience on the Internet (Yahoo! Internal Data).

Yahoo! is the most visited site online with nearly 4 billion visits and an average of 30 visits per user per month in the U.S. and leads all competitors in audience reach, frequency and engagement (comScore Media Metrix, US, Feb. 2007).

Yahoo! accounts for the largest share of time Americans spend on the Internet with 12 percent (comScore Media Metrix, US, Feb. 2007) and approximately 8 percent of the world’s online time (comScore WorldMetrix, Feb. 2007).

Yahoo! is the #1 home page with 85 million average daily visitors on Yahoo! homepages around the world, an increase of nearly 5 million visitors in a month (comScore WorldMetrix, Feb. 2007).

Yahoo!’s social media properties (Flickr, delicious, Answers, 360, Video, MyBlogLog, Jumpcut and Bix) have 115 million unique visitors worldwide (comScore WorldMetrix, Feb. 2007).

Yahoo! Answers is the largest collection of human knowledge on the Web with more than 90 million unique users and 250 million answers worldwide (Yahoo! Internal Data).

There are more than 450 million photos in Flickr in total and 1 million photos are uploaded daily. 80 percent of the photos are public (Yahoo! Internal Data).

Yahoo! Mail is the #1 Web mail provider in the world with 243 million users (comScore WorldMetrix, Feb. 2007) and nearly 80 million users in the U.S. (comScore Media Metrix, US, Feb. 2007)

Interoperability between Yahoo! Messenger and Windows Live Messenger has formed the largest IM community approaching 350 million user accounts (Yahoo! Internal Data).

Yahoo! Messenger is the most popular in time spent with an average of 50 minutes per user, per day (comScore WorldMetrix, Feb. 2007).

Nearly 1 in 10 Internet users is a member of a Yahoo! Groups (Yahoo! Internal Data).

Yahoo! is one of only 26 companies to be on both the Fortune 500 list and the Fortune’s “Best Place to Work” List (2006).

Agenda Publishing metadata on the Semantic Web A brief history of publishing metadata in HTML Semantic Search research and applications

Publishing metadata on the Semantic Web

A brief history of publishing metadata in HTML

Semantic Search research and applications

The many faces of the Semantic Web Six ways of publishing RDF Linked RDF files (linked data) Metadata inside webpages SPARQL endpoints Feeds XSLT/GRDDL Automated tools Non-exclusive but in practice most publisher choose one

Six ways of publishing RDF

Linked RDF files (linked data)

Metadata inside webpages

SPARQL endpoints

Feeds

XSLT/GRDDL

Automated tools

Non-exclusive but in practice

most publisher choose one

Option 1: Standalone RDF documents RDF documents linked to other RDF documents Use rdfs:seeAlso to point to a related document It says: Go and look at that document if you want to know more Advantages: No change to the publishing of the HTML documents Data can be published by third party Tools RDB-to-RDF mappers such as D2RQ or Triplify Linked Data browsers Examples: Most datasets in the Linked Data cloud . . . #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population

RDF documents linked to other RDF documents

Use rdfs:seeAlso to point to a related document

It says: Go and look at that document if you want to know more

Advantages:

No change to the publishing of the HTML documents

Data can be published by third party

Tools

RDB-to-RDF mappers such as D2RQ or Triplify

Linked Data browsers

Examples: Most datasets in the Linked Data cloud

Option 1: cntd. For discovery, the metadata is often linked from HTML pages < link rel=&quot;meta&quot; type=&quot;application/rdf+xml&quot; title=&quot;FOAF&quot; href=&quot;http://www.cs.vu.nl/~pmika/foaf.rdf&quot; /> Additional advantages: Discovery from the webpage It’s clear that the metadata is a machine representation of the human-targeted content of the page Examples: FOAF profiles, BestBuy . Peter Mika was born in Budapest. #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population

For discovery, the metadata is often linked from HTML pages

< link rel=&quot;meta&quot; type=&quot;application/rdf+xml&quot; title=&quot;FOAF&quot; href=&quot;http://www.cs.vu.nl/~pmika/foaf.rdf&quot; />

Additional advantages:

Discovery from the webpage

It’s clear that the metadata is a machine representation of the human-targeted content of the page

Examples: FOAF profiles, BestBuy

Option 2: Metadata inside web pages Using microformats, RDFa, MicroData (more later) Advantages: No content negotiation required No separate database export required Browser plug-in friendly Search engine friendly Copy-paste friendly Tools: XML editors (e.g. Oxygen) Triplr RDFa Distiller RDFa bookmarklet Ubiquity RDFa plugin Optimus microformat parser Examples: many, including SlideShare, YouTube, LinkedIn, Digg, Myspace, Facebook… Peter Mika was born in Budapest. Peter Mika was born in Budapest. #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population

Using microformats, RDFa, MicroData (more later)

Advantages:

No content negotiation required

No separate database export required

Browser plug-in friendly

Search engine friendly

Copy-paste friendly

Tools:

XML editors (e.g. Oxygen)

Triplr

RDFa Distiller

RDFa bookmarklet

Ubiquity RDFa plugin

Optimus microformat parser

Examples: many, including SlideShare, YouTube, LinkedIn, Digg, Myspace, Facebook…

Option 3: SPARQL endpoints Query access to your RDF database Similar to exposing your database on the Web and giving someone read-only SQL access Advantages: Most flexible and best performing access from a consumer perspective Tools: Triple stores (Oracle, Virtuoso, Sesame, Jena, OWLIM etc.) RDB-to-RDF mappers such as D2RQ and Triplify #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population

Query access to your RDF database

Similar to exposing your database on the Web and giving someone read-only SQL access

Advantages:

Most flexible and best performing access from a consumer perspective

Tools:

Triple stores (Oracle, Virtuoso, Sesame, Jena, OWLIM etc.)

RDB-to-RDF mappers such as D2RQ and Triplify

Option 4: feeds The equivalent of a database dump No standard feed format for RDF Advantages Submit your data without making it public Yahoo! consumes: DataRSS GoogleBase feeds NewsML Submit your feed using SiteExplorer . #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population #PeterM #Bud born “ Peter Mika” label “ Budapest” label #Hun capital-of “ 2,000,000” population

The equivalent of a database dump

No standard feed format for RDF

Advantages

Submit your data without making it public

Yahoo! consumes:

DataRSS

GoogleBase feeds

NewsML

Submit your feed using SiteExplorer

Option 5: XSLT Publish the transformation from HTML to structured data GRDDL is a standard for linking an HTML page to a transformation that produces RDF data Advantages No change to the page Disadvantages Transformation needs to be executed to get to the data Tools Intel MashMaker Dapper Glue API from AdaptiveBlue <XSLT> xx yy 1 2

Publish the transformation from HTML to structured data

GRDDL is a standard for linking an HTML page to a transformation that produces RDF data

Advantages

No change to the page

Disadvantages

Transformation needs to be executed to get to the data

Tools

Intel MashMaker

Dapper

Glue API from AdaptiveBlue

Option 6: Automatic markup Restricted mostly to tagging entities with identifiers Advantages Less manual effort Disadvantages Limited to finding relevant entities in text Tools OpenCalais Zemanta API Peter Mika was born in Budapest. <person>Peter Mika</person> was born in <location>Budapest</location>.

Restricted mostly to tagging entities with identifiers

Advantages

Less manual effort

Disadvantages

Limited to finding relevant entities in text

Tools

OpenCalais

Zemanta API

Example: Zemanta A personal writing assistant for bloggers Plugin for popular blogging platforms and web mail clients Analyzes text as you type and suggests hyperlinks, tags, categories, images and related articles API available with the same functionality

A personal writing assistant for bloggers

Plugin for popular blogging platforms and web mail clients

Analyzes text as you type and suggests hyperlinks, tags, categories, images and related articles

API available with the same functionality

Metadata in HTML

Brief history of the Annotated Web 1995: HTML meta tags 1996: Simple HTML Ontology Extensions (SHOE) 1998: RDF/XML RDF/XML in HTML RDF linked from HTML 2003: Web 2.0 Tagging Microformats Metadata in Wikipedia Machine tags in Flickr 2005: eRDF 2008: RDFa 2009: Microdata (?)

1995: HTML meta tags

1996: Simple HTML Ontology Extensions (SHOE)

1998: RDF/XML

RDF/XML in HTML

RDF linked from HTML

2003: Web 2.0

Tagging

Microformats

Metadata in Wikipedia

Machine tags in Flickr

2005: eRDF

2008: RDFa

2009: Microdata (?)

HTML meta tags <HTML> <HEAD profile=&quot;http://dublincore.org/documents/dcq-html/&quot;> <META name=&quot;DC.author &quot; content=&quot; Peter Mika &quot;> <LINK rel=&quot;DC.rights copyright&quot; href=&quot; http://www.example.org/rights.html &quot; /> <LINK rel=&quot;meta&quot; type=&quot;application/rdf+xml&quot; title=&quot;FOAF&quot; href= &quot; http://www.cs.vu.nl/~pmika/foaf.rdf &quot;> </HEAD> … </HTML>

<HTML>

<HEAD profile=&quot;http://dublincore.org/documents/dcq-html/&quot;>

<META name=&quot;DC.author &quot; content=&quot; Peter Mika &quot;>

<LINK rel=&quot;DC.rights copyright&quot; href=&quot; http://www.example.org/rights.html &quot; />

<LINK rel=&quot;meta&quot; type=&quot;application/rdf+xml&quot; title=&quot;FOAF&quot;

href= &quot; http://www.cs.vu.nl/~pmika/foaf.rdf &quot;>

</HEAD>

…

</HTML>

SHOE example (Hefflin & Hendler, 1996) <ONTOLOGY &quot;our-ontology&quot; VERSION=&quot;1.0&quot;> <ONTOLOGY-EXTENDS &quot;organization-ontology&quot; VERSION=&quot;2.1&quot; PREFIX=&quot;org&quot; URL=&quot;http://www.ont.org/orgont.html&quot;> <ONTDEF CATEGORY=&quot;Person&quot; ISA=&quot;org.Thing&quot;> <ONTDEF RELATION=&quot;lastName&quot; ARGS=&quot;Person STRING&quot;> <ONTDEF RELATION=&quot;firstName&quot; ARGS=&quot;Person STRING&quot;> <ONTDEF RELATION=&quot;marriedTo&quot; ARGS=&quot;Person Person&quot;> <ONTDEF RELATION=&quot;employee&quot; ARGS=&quot;org.Organization Person&quot;> </ONTOLOGY > <HEAD> <META HTTP-EQUIV=&quot;Instance-Key&quot; CONTENT=&quot;http://www.cs.umd.edu/~george&quot;> <USE-ONTOLOGY &quot;our-ontology&quot; VERSION=&quot;1.0&quot; PREFIX=&quot;our&quot; URL=&quot;http://ont.org/our-ont.html&quot;> </HEAD> <BODY> <CATEGORY &quot;our.Person&quot;> <RELATION &quot;our.marriedTo&quot; TO=&quot;http://www.cs.umd.edu/~helena&quot;> <RELATION &quot;our.employee&quot; FROM=&quot;http://www.cs.umd.edu&quot;> My name is <ATTRIBUTE &quot;our.firstName&quot;> George </ATTRIBUTE> <ATTRIBUTE &quot;our.lastName&quot;> Cook </ATTRIBUTE> and I live at...

<ONTOLOGY &quot;our-ontology&quot; VERSION=&quot;1.0&quot;>

<ONTOLOGY-EXTENDS &quot;organization-ontology&quot; VERSION=&quot;2.1&quot; PREFIX=&quot;org&quot; URL=&quot;http://www.ont.org/orgont.html&quot;>

<ONTDEF CATEGORY=&quot;Person&quot; ISA=&quot;org.Thing&quot;>

<ONTDEF RELATION=&quot;lastName&quot; ARGS=&quot;Person STRING&quot;>

<ONTDEF RELATION=&quot;firstName&quot; ARGS=&quot;Person STRING&quot;>

<ONTDEF RELATION=&quot;marriedTo&quot; ARGS=&quot;Person Person&quot;>

<ONTDEF RELATION=&quot;employee&quot; ARGS=&quot;org.Organization Person&quot;>

</ONTOLOGY >

SHOE system

SHOE Text-based query interface

SHOE Graphical Query Interface

Example: Creative Commons Embedding CC license in HTML (now deprecated): <HTML> <HEAD>… </HEAD> <BODY> … <!–- <rdf:RDF xmlns=&quot;http://creativecommons.org/ns#&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot; xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot;> <Work rdf:about=&quot;http://www.yergler.net/averages/&quot;> <dc:title>The Law of Averages</dc:title> <dc:description>...because eventually i&apos;ll be right...</dc:description> <license rdf:resource=&quot;http://creativecommons.org/licenses/by-nc/1.0/&quot; /> </Work> <License rdf:about=&quot;http://creativecommons.org/licenses/by-nc/1.0/&quot;> <requires rdf:resource=&quot;http://web.resource.org/cc/Notice&quot; /> <permits rdf:resource=&quot;http://web.resource.org/cc/Reproduction&quot; /> <permits rdf:resource=&quot;http://web.resource.org/cc/Distribution&quot; /> <prohibits rdf:resource=&quot;http://web.resource.org/cc/CommercialUse&quot; /> </License> </rdf:RDF> -->

Embedding CC license in HTML (now deprecated):

Example: Creative Commons Current: rel attribute (HTML4) This work is licensed under a <a rel=&quot;license&quot; href=&quot;http://creativecommons.org/licenses/by/3.0/us/&quot;>Creative Commons Attribution 3.0 United States License</a>. Use of the “rel” attribute for semantic annotation is the birth of the microformat…

Current: rel attribute (HTML4)

Use of the “rel” attribute for semantic annotation is the birth of the microformat…

Microformats (μf) Community centered around microformats.org Specifications and discussions are hosted there Agreements on the way to encode certain kinds metadata in HTML Reuse of semantic-bearing HTML elements Based on existing standards Minimality Microformats exist for a limited set of objects hCard (persons and organizations) hCalendar (events) hResume hProduct hRecipe Varying degrees of support and stability hCard and rel-tag are widely supported

Community centered around microformats.org

Specifications and discussions are hosted there

Agreements on the way to encode certain kinds metadata in HTML

Reuse of semantic-bearing HTML elements

Based on existing standards

Minimality

Microformats exist for a limited set of objects

hCard (persons and organizations)

hCalendar (events)

hResume

hProduct

hRecipe

Varying degrees of support and stability

hCard and rel-tag are widely supported

Example: microformats <cite class=&quot; vcard &quot;> <a class=&quot; fn url &quot; rel=&quot;friend colleague met&quot; href=&quot;http://meyerweb.com/&quot;> Eric Meyer </a> </cite> wrote a post ( <cite> <a href=&quot;http://meyerweb.com/eric/thoughts/2005/12/16/tax-relief/&quot;> Tax Relief </a></cite> ) about an unintentionally humorous letter he received from the <span class=&quot; vcard &quot;> <a class=&quot; fn org url &quot; href=&quot;http://irs.gov/&quot;> Internal Revenue Service </a> </span>. <div class=&quot; vcard &quot;> <a class=&quot; email fn &quot; href=&quot;mailto:jfriday@host.com&quot;> Joe Friday </a> <div class=&quot; tel &quot;> +1-919-555-7878 </div> <div class=&quot; title &quot;> Area Administrator, Assistant </div> </div>

Microformats: limitations No shared syntax Each microformat has a separate syntax tailored to the vocabulary No formal schemas Limited reuse, extensibility of schemas Unclear which combinations are allowed No datatypes No namespaces, unique identifiers (URIs) no interlinking mapping between instances is required Relationship to page context is often unclear

No shared syntax

Each microformat has a separate syntax tailored to the vocabulary

No formal schemas

Limited reuse, extensibility of schemas

Unclear which combinations are allowed

No datatypes

No namespaces, unique identifiers (URIs)

no interlinking

mapping between instances is required

Relationship to page context is often unclear

RDFa RDF-in-attributes World Wide Web Consortium (W3C) recommendation for encoding RDF triples in HTML Full RDF support Recommendation specifies the algorithm for parsing the triples out of HTML Requires XHTML in principle In practice, no one cares

RDF-in-attributes

World Wide Web Consortium (W3C) recommendation for encoding RDF triples in HTML

Full RDF support

Recommendation specifies the algorithm for parsing the triples out of HTML

Requires XHTML in principle

In practice, no one cares

RDFa in a slide <p xmlns:rdfs=&quot;http://www.w3.org/2000/01/rdf-schema#&quot; xmlns:foaf=&quot;http://xmlns.com/foaf/0.1/” typeof=”foaf:Person &quot; about=&quot;http://example.org/staff/jo&quot; > <span property= &quot; rdfs:label foaf:name &quot;> Jo Smith </span>. <span property= &quot; foaf:title &quot;> Web hacker </span> at <a rel=”vcard:org&quot; property= &quot; foaf:name &quot; href=&quot;http://example.org&quot;> Acme Corp </a>. You can contact me <a rel= &quot; foaf:mbox&quot; href=&quot;mailto:jo@example.org&quot;> via email </a>. </p> ... Assign the prefixes rdfs and foaf to the RDFS and FOAF namespaces (as in XML, RDF/XML etc.) Create a new resource of type foaf:Person Assign a value to a property Give it a URI Link to another resource and assign a name to it

Microformats vs. RDFa Choose microformats when you find a microformat that fits your needs (and supported by your favorite tool or search engine) Microformats are first option because they are simple We support all major microformats, see the documentation It’s a common misconception that RDFa requires XHTML: it doesn’t If you find none that perfectly fits your needs then you need RDFa Microformats have a fixed schema: you can not add your own attributes Example: a social networking site with user profiles VCard is a good candidate, but for example it doesn’t have a way to express the user’s social connections You either live without this, or go with RDFa

Choose microformats when you find a microformat that fits your needs (and supported by your favorite tool or search engine)

Microformats are first option because they are simple

We support all major microformats, see the documentation

It’s a common misconception that RDFa requires XHTML: it doesn’t

If you find none that perfectly fits your needs then you need RDFa

Microformats have a fixed schema: you can not add your own attributes

Example: a social networking site with user profiles

VCard is a good candidate, but for example it doesn’t have a way to express the user’s social connections

You either live without this, or go with RDFa

Keep an eye on HTML5 Currently under standardization at the W3C Last Call in Fall 2009 Introduces Microdata Similar to microformats Some predefined vocabularies with central registration Some of the flexibility of RDFa Introduce new terms using reverse domain names or full URIs Semantic HTML elements such as <time>, <video>, <article>…

Currently under standardization at the W3C

Last Call in Fall 2009

Introduces Microdata

Similar to microformats

Some predefined vocabularies with central registration

Some of the flexibility of RDFa

Introduce new terms using reverse domain names or full URIs

Semantic HTML elements such as <time>, <video>, <article>…

Microdata example <div item=“http://www.yahoo.com/resource/person”> <p>My name is <span itemprop=&quot; name &quot;> Neil </span>.</p> <p>My band is called <span itemprop=&quot; band &quot;> Four Parts Water </span>. I was born on <time itemprop=&quot; birthday &quot; datetime=&quot; 2009-05-10 &quot;>May 10th 2009</time>. <img itemprop=&quot; image &quot; src=” me.png &quot; alt=”me”> </p> </div

The process of annotating with RDFa Invest in familiarizing with the RDFa syntax by reading the RDFa Primer It is also highly recommended that you read the RDF Primer . RDF is the data model used by RDFa. Choose a vocabulary from the SearchMonkey documentation that fits your needs A vocabulary describes a set of types and attributes within a given domain If you don’t fin d a good candidate , extend an existing one or create a new one Annotate your page. Before you start, you might want to validate your page for (X)HTML conformance using the W3C’s (X)HTML Validator to reduce the chance of errors. Choose Document Type XHTML + RDFa. No specific tool support. If you have an HTML or XML editor that supports DTDs, you will have syntax checking and highlighting. Use the RDFa Distiller to validate which data can be extracted from your page. If you fancy, use the RDF Validator to graphically visualize the RDF graph that is outputted. Put the annotated page online. The data will extracted the next time your page is crawled No need to explicitly submit anything No notification when your site is crawled See http://rdfa.info/rdfa-implementations for new tools and APIs

Invest in familiarizing with the RDFa syntax by reading the RDFa Primer

It is also highly recommended that you read the RDF Primer . RDF is the data model used by RDFa.

Choose a vocabulary from the SearchMonkey documentation that fits your needs

A vocabulary describes a set of types and attributes within a given domain

If you don’t fin d a good candidate , extend an existing one or create a new one

Annotate your page.

Before you start, you might want to validate your page for (X)HTML conformance using the W3C’s (X)HTML Validator to reduce the chance of errors. Choose Document Type XHTML + RDFa.

No specific tool support. If you have an HTML or XML editor that supports DTDs, you will have syntax checking and highlighting.

Use the RDFa Distiller to validate which data can be extracted from your page.

If you fancy, use the RDF Validator to graphically visualize the RDF graph that is outputted.

Put the annotated page online. The data will extracted the next time your page is crawled

No need to explicitly submit anything

No notification when your site is crawled

See http://rdfa.info/rdfa-implementations for new tools and APIs

Choosing a vocabulary Look at SearchMonkey objects Video, Games, Presentations, Events, News, Businesses, Products, Discussion Search the Web or ask for advice on mailing lists [email_address] [email_address] Wikis semanticweb.org vocamp.org Beware of people who claim to have the vocabulary of everything Preferably you want something small and targeted Never a 100% fit  you will need to introduce vocabulary terms (classes and properties) Do not introduce new classes/properties in existing namespaces Example: the namespace http://xmlns.com/foaf/0.1/ is used by the FOAF project. Try not to introduce a new term without contacting the owner, i.e. the membership of the FOAF mailing list.

Look at SearchMonkey objects

Video, Games, Presentations, Events, News, Businesses, Products, Discussion

Search the Web or ask for advice on mailing lists

[email_address]

[email_address]

Wikis

semanticweb.org

vocamp.org

Beware of people who claim to have the vocabulary of everything

Preferably you want something small and targeted

Never a 100% fit  you will need to introduce vocabulary terms (classes and properties)

Do not introduce new classes/properties in existing namespaces

Example: the namespace http://xmlns.com/foaf/0.1/ is used by the FOAF project. Try not to introduce a new term without contacting the owner, i.e. the membership of the FOAF mailing list.

Advanced topic: creating a vocabulary Get advice on methodology vocamp.org and semanticweb.org Choose a namespace and a prefix Give sensible names, e.g. name it after your site, but don’t call it searchmonkey Namespace ends either with a slash or a hash Create an RDF or OWL document describing your classes and properties Use an ontology editor such as Protégé 4.0 Follow naming conventions Publish your vocabulary Make sure the URIs of your properties and classes are resolvable E.g. myvocab:digicam should resolve to a document containing the definition of myvocab:digicam Convince others to adopt your vocabulary If you are in fishing, convince other fishing businesses

Get advice on methodology

vocamp.org and semanticweb.org

Choose a namespace and a prefix

Give sensible names, e.g. name it after your site, but don’t call it searchmonkey

Namespace ends either with a slash or a hash

Create an RDF or OWL document describing your classes and properties

Use an ontology editor such as Protégé 4.0

Follow naming conventions

Publish your vocabulary

Make sure the URIs of your properties and classes are resolvable

E.g. myvocab:digicam should resolve to a document containing the definition of myvocab:digicam

Convince others to adopt your vocabulary

If you are in fishing, convince other fishing businesses

Exercise (slideshare.net/pmika) Explore data on the Web Microformats Search for pages on Yahoo using searchmonkey:com.yahoo.page.uf.hcard Try Operator Firefox Plug-in Try Optimus RDFa Create yourself or search for pages on Yahoo using searchmonkey:com.yahoo.page.rdf.rdfa Try RDFa bookmarklet to highlight RDFa Try RDFa Distiller to extract RDF from HTML Try RDF Validator to visualize your RDF data Mark up your webpage using RDFa Use RDFa Distiller to test Try automated annotation using Zemanta or OpenCalais

Explore data on the Web

Microformats

Search for pages on Yahoo using searchmonkey:com.yahoo.page.uf.hcard

Try Operator Firefox Plug-in

Try Optimus

RDFa

Create yourself or search for pages on Yahoo using searchmonkey:com.yahoo.page.rdf.rdfa

Try RDFa bookmarklet to highlight RDFa

Try RDFa Distiller to extract RDF from HTML

Try RDF Validator to visualize your RDF data

Mark up your webpage using RDFa

Use RDFa Distiller to test

Try automated annotation using Zemanta or OpenCalais

Semantic Search

Why semantic search? (P. Raghavan) Old battles are won Main driver of user perception of search quality used to be: precision of navigational queries The technical prowess was about crawling and spam The hard core was indexing and retrieval Currently, the biggest bottlenecks in IR not computational, but in modeling user cognition If only we could find a computationally expensive way to solve the problem then we should be able to make it go faster

Old battles are won

Main driver of user perception of search quality used to be: precision of navigational queries

The technical prowess was about crawling and spam

The hard core was indexing and retrieval

Currently, the biggest bottlenecks in IR not computational, but in modeling user cognition

If only we could find a computationally expensive way to solve the problem

then we should be able to make it go faster

Searches that show a ‘semantic gap’ Ambiguous searches Paris Hilton Multimedia search Images of Paris Hilton Imprecise or overly precise searches Publications by Jim Hendler Find images of strong and adventurous people (Lenat) Searches for descriptions Search for yourself without using your name Product search (ads!) Searches that require aggregation Size of the Eiffer tower (Lenat) Public opinion on Britney Spears World temperature by 2020 Queries that require a deeper understanding of the query, the content and/or the world at large

Ambiguous searches

Paris Hilton

Multimedia search

Images of Paris Hilton

Imprecise or overly precise searches

Publications by Jim Hendler

Find images of strong and adventurous people (Lenat)

Searches for descriptions

Search for yourself without using your name

Product search (ads!)

Searches that require aggregation

Size of the Eiffer tower (Lenat)

Public opinion on Britney Spears

World temperature by 2020

Not just search

Semantic Search Def. matching the user’s query with the Web’s content at a conceptual level, often with the help of world knowledge R. Guha, R. McCool: Semantic Search, WWW2003 Related disciplines Semantic Web, IR, Databases, NLP, IE As a field ISWC/ESWC/ASWC, WWW, SIGIR Exploring Semantic Annotations in Information Retrieval (ECIR08, WSDM09) Semantic Search Workshop (ESWC08, WWW09) Future of Web Search: Semantic Search (FoWS09)

Def. matching the user’s query with the Web’s content at a conceptual level, often with the help of world knowledge

R. Guha, R. McCool: Semantic Search, WWW2003

Related disciplines

Semantic Web, IR, Databases, NLP, IE

As a field

ISWC/ESWC/ASWC, WWW, SIGIR

Exploring Semantic Annotations in Information Retrieval (ECIR08, WSDM09)

Semantic Search Workshop (ESWC08, WWW09)

Future of Web Search: Semantic Search (FoWS09)

Semantics at every step of the IR process bla bla bla? q=“bla” * 3 Document processing bla bla bla Ranking Query processing Search interface The IR engine The Web The Semantic Web bla bla bla bla bla bla “ bla” θ (q,d)

Document processing Goal: provide a higher level representation of text in some conceptual space Diverse methods Document classification Information Extraction Named-entity recognition, word-sense disambiguation, semantic role labeling, wrapper induction, form filling, etc. Previously: open source toolkits such as GATE, OpenNLP Require expert user to operate and train NEW: Online tools For non-expert users Embed metadata inside documents and/or link entities in documents to the cloud Often using existing metadata as background knowledge

Goal: provide a higher level representation of text in some conceptual space

Diverse methods

Document classification

Information Extraction

Named-entity recognition, word-sense disambiguation, semantic role labeling, wrapper induction, form filling, etc.

Previously: open source toolkits such as GATE, OpenNLP

Require expert user to operate and train

NEW: Online tools

For non-expert users

Embed metadata inside documents and/or link entities in documents to the cloud

Often using existing metadata as background knowledge

Examples: online NLP OpenCalais (Thomson Reuters) Online interface and API for named-entity recognition, (partial) disambiguation and relationship extraction HTML annotator (microformats), Firefox plug-in, WordPress plug-in, Yahoo Pipes service, etc. Works best on the news domain Zemanta Zemanta is a ‘personal writing assistant’ that recognizes and disambiguates named entities and suggests related content A Firefox-plugin that extends the functionality of popular blogging and online email platforms, o nline interface and API Broad coverage but partial recognition

OpenCalais (Thomson Reuters)

Online interface and API for named-entity recognition, (partial) disambiguation and relationship extraction

HTML annotator (microformats), Firefox plug-in, WordPress plug-in, Yahoo Pipes service, etc.

Works best on the news domain

Zemanta

Zemanta is a ‘personal writing assistant’ that recognizes and disambiguates named entities and suggests related content

A Firefox-plugin that extends the functionality of popular blogging and online email platforms, o nline interface and API

Broad coverage but partial recognition

Examples: information extraction from templated HTML Intel MashMaker Create mashups based on information extracted from the page you are browsing (Firefox plugin) Wrapper induction trained using manual annotations Export the XSLT to be used in Yahoo’s SearchMonkey Dapper Semantic Advertising company Wrapper induction trained using manual annotations (online tool) API to the dynamic extraction Glue Social interaction around objects on the Web (Firefox plugin) API provides access to the information extracted from popular sites

Intel MashMaker

Create mashups based on information extracted from the page you are browsing (Firefox plugin)

Wrapper induction trained using manual annotations

Export the XSLT to be used in Yahoo’s SearchMonkey

Dapper

Semantic Advertising company

Wrapper induction trained using manual annotations (online tool)

API to the dynamic extraction

Glue

Social interaction around objects on the Web (Firefox plugin)

API provides access to the information extracted from popular sites

Query Interpretation Provide a higher level representation of queries in some conceptual space Ideally, the same space in which documents are represented Queries may be keywords, questions, semi-structured, structured etc. Interpretation treated as a separate step from ranking Required for federation, i.e. determine where to send the query Choosing between interpretations could be left to the user Due to performance requirements You cannot execute the query to determine what it means and then query again General world knowledge (e.g. DBpedia), domain ontologies or the schema of the actual data can be used

Provide a higher level representation of queries in some conceptual space

Ideally, the same space in which documents are represented

Queries may be keywords, questions, semi-structured, structured etc.

Interpretation treated as a separate step from ranking

Required for federation, i.e. determine where to send the query

Choosing between interpretations could be left to the user

Due to performance requirements

You cannot execute the query to determine what it means and then query again

General world knowledge (e.g. DBpedia), domain ontologies or the schema of the actual data can be used

Example: Semantic Search Assist Observation: the same type of objects often have the same query context Users asking for the same aspect of the type Could we make query suggestions based on the type of the entity? Improvement for infrequent queries apple ipod nano review sony plasma tv review jerry yang biography biography tim berners lee tim berners lee blog peter mika yahoo britney spears shaves her head

Observation: the same type of objects often have the same query context

Users asking for the same aspect of the type

Could we make query suggestions based on the type of the entity?

Improvement for infrequent queries

Models Desirable properties: P1: Fix is frequent within type P2: Fix has frequencies well-distributed across entities P3: Fix is infrequent outside of the type Models: apple ipod nano review entity fix type: product

Desirable properties:

P1: Fix is frequent within type

P2: Fix has frequencies well-distributed across entities

P3: Fix is infrequent outside of the type

Models:

Demo

Ranking Goal: match the query representation to content representation Again, possibly with the use of background knowledge Methods depend on the actual representation Bag of words, NL parse trees, SPARQL… the qualities of the queries and content the use case e.g. time available for ranking may vary from milliseconds to days

Goal: match the query representation to content representation

Again, possibly with the use of background knowledge

Methods depend on

the actual representation

Bag of words, NL parse trees, SPARQL…

the qualities of the queries and content

the use case

e.g. time available for ranking may vary from milliseconds to days

Search interface Goal is to facilitate the interaction between the user and the system helping the user to formulate queries present the results in an intelligent manner Semantic-based representations allow novel interfaces Adaptive presentation presentation adapts to the kind of query and results presented Aggregated search Grouping similar items, summarizing results in various ways Possibilities for filtering, possibly across different dimensions Task completion Help the user to fulfill the task by placing the query in a task context

Goal is to facilitate the interaction between the user and the system

helping the user to formulate queries

present the results in an intelligent manner

Semantic-based representations allow novel interfaces

Adaptive presentation

presentation adapts to the kind of query and results presented

Aggregated search

Grouping similar items, summarizing results in various ways

Possibilities for filtering, possibly across different dimensions

Task completion

Help the user to fulfill the task by placing the query in a task context

Putting it all together: Semantic Search Engines Natural Language search engines Hakia Powerset (now built into Bing) TrueKnowledge Structured data search engines Searching open web data Sindice Sigma Searching closed world data Wolfram Alpha (closed structured data + computation) Web search engines Yahoo’s SearchMonkey BOSS and YQL Google’s Rich Snippets

Natural Language search engines

Hakia

Powerset (now built into Bing)

TrueKnowledge

Structured data search engines

Searching open web data

Sindice

Sigma

Searching closed world data

Wolfram Alpha (closed structured data + computation)

Web search engines

Yahoo’s SearchMonkey

BOSS and YQL

Google’s Rich Snippets

An open platform for using structured data to build more useful and relevant search results Creating an ecosystem of publishers, developers and end-users Helping publishers to implement semantic annotation and motivating them by allowing to customize their search results Providing tools and APIs for developers to create compelling applications Improving search result presentation for our end-users Semantic Web technology Support for a number of microformats, RDFa RDF based data representation Industry standard vocabularies (or use any of your own) SearchMonkey

An open platform for using structured data to build more useful and relevant search results

Creating an ecosystem of publishers, developers and end-users

Helping publishers to implement semantic annotation and motivating them by allowing to customize their search results

Providing tools and APIs for developers to create compelling applications

Improving search result presentation for our end-users

Semantic Web technology

Support for a number of microformats, RDFa

RDF based data representation

Industry standard vocabularies (or use any of your own)

image deep links name/value pairs or abstract Enhanced Result

A difficult problem! What if one would try to do this automatically? Document summarization Page structure detection Information Extraction Image recognition Link classification and ranking But one of those cases where a little semantics can go a long way…

What if one would try to do this automatically?

Document summarization

Page structure detection

Information Extraction

Image recognition

Link classification and ranking

But one of those cases where a little semantics can go a long way…

SearchMonkey Acme.com’s database Index RDF/Microformat Markup site owners/publishers share structured data with Yahoo!. 1 consumers customize their search experience with Enhanced Results or Infobars 3 site owners & third-party developers build SearchMonkey apps. 2 DataRSS feed Web Services Page Extraction Acme.com’s Web Pages

Example apps LinkedIn hCard plus feed data Creative Commons by Ben Adida CC in RDFa

LinkedIn

hCard plus feed data

Creative Commons by Ben Adida

CC in RDFa

Example apps. II. Other me by Dan Brickley Google Social Graph API wrapped using a Web Service

Other me by Dan Brickley

Google Social Graph API wrapped using a Web Service

Google’s Rich Snippets Shares a subset of the features of SearchMonkey Encourages publishers to embed certain microformats and RDFa into webpages Currently reviews, people, products, business & organizations These are used to generate richer search results SearchMonkey is customizable Developers can develop applications themselves SearchMonkey is open Wide support for standard vocabularies API access

Shares a subset of the features of SearchMonkey

Encourages publishers to embed certain microformats and RDFa into webpages

Currently reviews, people, products, business & organizations

These are used to generate richer search results

SearchMonkey is customizable

Developers can develop applications themselves

SearchMonkey is open

Wide support for standard vocabularies

API access

BOSS: Build your Own Search Service Ability to re-order results and blend-in addition content No restrictions on presentation No branding or attribution Access to multiple verticals (web search, image, news) 40+ supported language and region pairs Pricing (BOSS) Pay-by-usage 10,000 queries a day still free Serve any ads you want For more info, http://developer.yahoo.com/search/boss/

Ability to re-order results and blend-in addition content

No restrictions on presentation

No branding or attribution

Access to multiple verticals (web search, image, news)

40+ supported language and region pairs

Pricing (BOSS)

Pay-by-usage

10,000 queries a day still free

Serve any ads you want

For more info, http://developer.yahoo.com/search/boss/

BOSS API to structured data Simple HTTP GET calls, no authentication You need an Application ID: register at developer.yahoo.com/search/boss/ http://boss.yahooapis.com/ysearch/web/v1/{query}?appid={appid}&format=xml&view=searchmonkey_feed Restrict your query using special words searchmonkey:com.yahoo.page.uf.{format} {format} is one of hcard, hcalendar, tag, adr, hresume etc. searchmonkey:com.yahoo.page.rdf.rdfa

Simple HTTP GET calls, no authentication

You need an Application ID: register at developer.yahoo.com/search/boss/

http://boss.yahooapis.com/ysearch/web/v1/{query}?appid={appid}&format=xml&view=searchmonkey_feed

Restrict your query using special words

searchmonkey:com.yahoo.page.uf.{format}

{format} is one of hcard, hcalendar, tag, adr, hresume etc.

searchmonkey:com.yahoo.page.rdf.rdfa

Demo: resume search Search pages with resume data and given keywords {keyword} searchmonkey:com.yahoo.page.uf.hresume Parse the results as DataRSS (XML) Extract information and display using YUI

Search pages with resume data and given keywords

{keyword} searchmonkey:com.yahoo.page.uf.hresume

Parse the results as DataRSS (XML)

Extract information and display using YUI

Demo

Yahoo Query Language (YQL) Query web APIs as virtual tables Mash-up data by joining tables Add an API by adding a table definition Example: select my friends and sort by nickname

Query web APIs as virtual tables

Mash-up data by joining tables

Add an API by adding a table definition

Example: select my friends and sort by nickname

PHP example : select the last 100 photos from Flickr with the word Austin <?php $url = &quot;http://query.yahooapis.com/v1/public/yql?q=&quot;; $q = &quot;select * from flickr.photos.search(100) where text=’Austin'&quot;; $fmt = &quot;xml&quot;; $x = simplexml_load_file($url.urlencode($q).&quot;&format=$fmt&quot;); foreach($x->attributes('http://www.yahooapis.com/v1/base.rng') as $k=>$v) { $$k=(string)$v; } echo <<<EOB $count photos fetched from {$x->diagnostics->url} in {$x->diagnostics->url['execution-time']} seconds<br> EOB; $flickr = &quot;http://static.flickr.com/&quot;; foreach($x->results->photo as $p) { echo &quot;<img src=&quot;$flickr{$p['server']}/{$p['id']}_{$p['secret']}_s.jpg&quot;/>
&quot;; } ?>

<?php

$url = &quot;http://query.yahooapis.com/v1/public/yql?q=&quot;;

$q = &quot;select * from flickr.photos.search(100) where text=’Austin'&quot;;

$fmt = &quot;xml&quot;;

$x = simplexml_load_file($url.urlencode($q).&quot;&format=$fmt&quot;);

foreach($x->attributes('http://www.yahooapis.com/v1/base.rng') as $k=>$v) {

$$k=(string)$v;

}

echo <<<EOB

$count photos fetched from

{$x->diagnostics->url} in

{$x->diagnostics->url['execution-time']} seconds<br>

EOB;

$flickr = &quot;http://static.flickr.com/&quot;;

foreach($x->results->photo as $p) {

echo &quot;<img src=&quot;$flickr{$p['server']}/{$p['id']}_{$p['secret']}_s.jpg&quot;/>
&quot;;

}

?>

YQL example ( source )

That’s all there is to it! <?php $root = 'http://query.yahooapis.com/v1/public/yql?q='; $city = 'Barcelona'; $loc = 'Barcelona'; $yql = 'select * from html where url = 'http://en.wikipedia.org/wiki/'.$city.'' and xpath=&quot;//div[@id='bodyContent']/p&quot; limit 3'; $url = $root . urlencode($yql) . '&format=xml'; $info = getstuff($url); $info = preg_replace(&quot;/.*<results>|</results>.*/&quot;,'',$info); $info = preg_replace(&quot;/<?xml version=&quot;1.0&quot;&quot;. &quot; encoding=&quot;UTF-8&quot;?>/&quot;,'',$info); $info = preg_replace(&quot;//&quot;,'',$info); $info = preg_replace(&quot;/&quot;/wiki/&quot;,'&quot;http://en.wikipedia.org/wiki',$info); $yql = 'select * from upcoming.events.bestinplace(5) where woeid in '. '(select woeid from geo.places where text=&quot;'.$loc.'&quot;)'. ' | unique(field=&quot;description&quot;)'; $url = $root . urlencode($yql) . '&format=json'; $events = getstuff($url); $events = json_decode($events); foreach($events->query->results->event as $e){ $evHTML.='<li><h3><a href=&quot;'.$e->ticket_url.'&quot;>'.$e->name.'</a></h3><p>'. substr($e->description,0,100).'&hellip;</p></li>'; } $yql = 'select * from flickr.photos.info where photo_id in '. '(select id from flickr.photos.search where woe_id in '. '(select woeid from geo.places where text=&quot;'.$loc.'&quot;)) limit 16'; $url = $root . urlencode($yql) . '&format=json'; $photos = getstuff($url); $photos = json_decode($photos); foreach($photos->query->results->photo as $s){ $src = &quot;http://farm{$s->farm}.static.flickr.com/{$s->server}/&quot;. &quot;{$s->id}_{$s->secret}_s.jpg&quot;; $phHTML.='<li><a href=&quot;'.$s->urls->url->content.'&quot;><img alt=&quot;'. $s->title.'&quot; src=&quot;'.$src.'&quot;></a></li>'; } $yql='select description from rss where '. ' url=&quot;http://weather.yahooapis.com/forecastrss?p=SPXX0015&u=c&quot;'; $url = $root . urlencode($yql) . '&format=json'; $weather = getstuff($url); $weather = json_decode($weather); $weHTML = $weather->query->results->item->description; function getstuff($url){ $curl_handle = curl_init(); curl_setopt($curl_handle, CURLOPT_URL, $url); curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2); curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1); $buffer = curl_exec($curl_handle); curl_close($curl_handle); if (empty($buffer)){ return 'Error retrieving data, please try later.'; } else { return $buffer; } }?>

<?php

$root = 'http://query.yahooapis.com/v1/public/yql?q=';

$city = 'Barcelona';

$loc = 'Barcelona';

$yql = 'select * from html where url = 'http://en.wikipedia.org/wiki/'.$city.'' and xpath=&quot;//div[@id='bodyContent']/p&quot; limit 3';

$url = $root . urlencode($yql) . '&format=xml';

$info = getstuff($url);

$info = preg_replace(&quot;/.*<results>|</results>.*/&quot;,'',$info);

$info = preg_replace(&quot;/<?xml version=&quot;1.0&quot;&quot;.

&quot; encoding=&quot;UTF-8&quot;?>/&quot;,'',$info);

$info = preg_replace(&quot;//&quot;,'',$info);

$info = preg_replace(&quot;/&quot;/wiki/&quot;,'&quot;http://en.wikipedia.org/wiki',$info);

$yql = 'select * from upcoming.events.bestinplace(5) where woeid in '.

'(select woeid from geo.places where text=&quot;'.$loc.'&quot;)'.

' | unique(field=&quot;description&quot;)';

$url = $root . urlencode($yql) . '&format=json';

$events = getstuff($url);

$events = json_decode($events);

foreach($events->query->results->event as $e){

$evHTML.='<li><h3><a href=&quot;'.$e->ticket_url.'&quot;>'.$e->name.'</a></h3><p>'.

substr($e->description,0,100).'&hellip;</p></li>';

}

$yql = 'select * from flickr.photos.info where photo_id in '.

'(select id from flickr.photos.search where woe_id in '.

'(select woeid from geo.places where text=&quot;'.$loc.'&quot;)) limit 16';

$url = $root . urlencode($yql) . '&format=json';

$photos = getstuff($url);

$photos = json_decode($photos);

foreach($photos->query->results->photo as $s){

$src = &quot;http://farm{$s->farm}.static.flickr.com/{$s->server}/&quot;.

&quot;{$s->id}_{$s->secret}_s.jpg&quot;;

$phHTML.='<li><a href=&quot;'.$s->urls->url->content.'&quot;><img alt=&quot;'.

$s->title.'&quot; src=&quot;'.$src.'&quot;></a></li>';

}

$yql='select description from rss where '.

' url=&quot;http://weather.yahooapis.com/forecastrss?p=SPXX0015&u=c&quot;';

$url = $root . urlencode($yql) . '&format=json';

$weather = getstuff($url);

$weather = json_decode($weather);

$weHTML = $weather->query->results->item->description;

function getstuff($url){

$curl_handle = curl_init();

curl_setopt($curl_handle, CURLOPT_URL, $url);

curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);

curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);

$buffer = curl_exec($curl_handle);

curl_close($curl_handle);

if (empty($buffer)){

return 'Error retrieving data, please try later.';

} else {

return $buffer;

}

}?>

Exercise Build a SearchMonkey application Try a few YQL queries Find data on the Web using search or by browsing Linked Data Build your own search engine using BOSS

Build a SearchMonkey application

Try a few YQL queries

Find data on the Web using search or by browsing Linked Data

Build your own search engine using BOSS

Challenges Future work in Semantic Web (Semi-)automated ways of metadata creation How do we go from 5% to 95%? Data quality Can we trust statements people make about each other’s data? Reasoning To what extent is reasoning useful? For example, how much would entity resolution or taxonomic reasoning help? Scale How do we exploit cluster computing techniques? What is between databases and IR engines? Fostering social agreements How do we get people to reuse vocabularies?

Future work in Semantic Web

(Semi-)automated ways of metadata creation

How do we go from 5% to 95%?

Data quality

Can we trust statements people make about each other’s data?

Reasoning

To what extent is reasoning useful?

For example, how much would entity resolution or taxonomic reasoning help?

Scale

How do we exploit cluster computing techniques?

What is between databases and IR engines?

Fostering social agreements

How do we get people to reuse vocabularies?

Challenges Future work in IR Query interpretation Ranking with metadata Evaluation of semantic search Personalization Semantic ads Constraints Users still want to see a document Keyword-based search cannot suffer Whole page relevance, monetization can only increase Established expectations Query entry Result presentation

Future work in IR

Query interpretation

Ranking with metadata

Evaluation of semantic search

Personalization

Semantic ads

Constraints

Users still want to see a document

Keyword-based search cannot suffer

Whole page relevance, monetization can only increase

Established expectations

Query entry

Result presentation

Contact Peter Mika [email_address] Come to Barcelona and stop by Ask about our internship program SearchMonkey developer.yahoo.com/searchmonkey/ mailing lists [email_address] [email_address] forums http://suggestions.yahoo.com/searchmonkey

Peter Mika

[email_address]

Come to Barcelona and stop by

Ask about our internship program

SearchMonkey

developer.yahoo.com/searchmonkey/

mailing lists

[email_address]

[email_address]

forums

http://suggestions.yahoo.com/searchmonkey

the monkey is out!