Making the Web Searchable Peter Mika Researcher, Data Architect Yahoo! Research

Yahoo! Research (research.yahoo.com)

Yahoo! Research Barcelona Established January, 2006 Led by Ricardo Baeza-Yates Research areas Web Mining content, structure, usage Distributed Web retrieval Multimedia retrieval NLP and Semantics

Established January, 2006

Led by Ricardo Baeza-Yates

Research areas

Web Mining

content, structure, usage

Distributed Web retrieval

Multimedia retrieval

NLP and Semantics

Yahoo! by numbers (April, 2007) There are approximately 500 million users of Yahoo! branded services, meaning we reach 50 percent – or 1 out of every 2 users – online, the largest audience on the Internet (Yahoo! Internal Data). Yahoo! is the most visited site online with nearly 4 billion visits and an average of 30 visits per user per month in the U.S. and leads all competitors in audience reach, frequency and engagement (comScore Media Metrix, US, Feb. 2007). Yahoo! accounts for the largest share of time Americans spend on the Internet with 12 percent (comScore Media Metrix, US, Feb. 2007) and approximately 8 percent of the world’s online time (comScore WorldMetrix, Feb. 2007). Yahoo! is the #1 home page with 85 million average daily visitors on Yahoo! homepages around the world, an increase of nearly 5 million visitors in a month (comScore WorldMetrix, Feb. 2007). Yahoo!’s social media properties (Flickr, delicious, Answers, 360, Video, MyBlogLog, Jumpcut and Bix) have 115 million unique visitors worldwide (comScore WorldMetrix, Feb. 2007). Yahoo! Answers is the largest collection of human knowledge on the Web with more than 90 million unique users and 250 million answers worldwide (Yahoo! Internal Data). There are more than 450 million photos in Flickr in total and 1 million photos are uploaded daily. 80 percent of the photos are public (Yahoo! Internal Data). Yahoo! Mail is the #1 Web mail provider in the world with 243 million users (comScore WorldMetrix, Feb. 2007) and nearly 80 million users in the U.S. (comScore Media Metrix, US, Feb. 2007) Interoperability between Yahoo! Messenger and Windows Live Messenger has formed the largest IM community approaching 350 million user accounts (Yahoo! Internal Data). Yahoo! Messenger is the most popular in time spent with an average of 50 minutes per user, per day (comScore WorldMetrix, Feb. 2007). Nearly 1 in 10 Internet users is a member of a Yahoo! Groups (Yahoo! Internal Data). Yahoo! is one of only 26 companies to be on both the Fortune 500 list and the Fortune’s “Best Place to Work” List (2006).

There are approximately 500 million users of Yahoo! branded services, meaning we reach 50 percent – or 1 out of every 2 users – online, the largest audience on the Internet (Yahoo! Internal Data).

Yahoo! is the most visited site online with nearly 4 billion visits and an average of 30 visits per user per month in the U.S. and leads all competitors in audience reach, frequency and engagement (comScore Media Metrix, US, Feb. 2007).

Yahoo! accounts for the largest share of time Americans spend on the Internet with 12 percent (comScore Media Metrix, US, Feb. 2007) and approximately 8 percent of the world’s online time (comScore WorldMetrix, Feb. 2007).

Yahoo! is the #1 home page with 85 million average daily visitors on Yahoo! homepages around the world, an increase of nearly 5 million visitors in a month (comScore WorldMetrix, Feb. 2007).

Yahoo!’s social media properties (Flickr, delicious, Answers, 360, Video, MyBlogLog, Jumpcut and Bix) have 115 million unique visitors worldwide (comScore WorldMetrix, Feb. 2007).

Yahoo! Answers is the largest collection of human knowledge on the Web with more than 90 million unique users and 250 million answers worldwide (Yahoo! Internal Data).

There are more than 450 million photos in Flickr in total and 1 million photos are uploaded daily. 80 percent of the photos are public (Yahoo! Internal Data).

Yahoo! Mail is the #1 Web mail provider in the world with 243 million users (comScore WorldMetrix, Feb. 2007) and nearly 80 million users in the U.S. (comScore Media Metrix, US, Feb. 2007)

Interoperability between Yahoo! Messenger and Windows Live Messenger has formed the largest IM community approaching 350 million user accounts (Yahoo! Internal Data).

Yahoo! Messenger is the most popular in time spent with an average of 50 minutes per user, per day (comScore WorldMetrix, Feb. 2007).

Nearly 1 in 10 Internet users is a member of a Yahoo! Groups (Yahoo! Internal Data).

Yahoo! is one of only 26 companies to be on both the Fortune 500 list and the Fortune’s “Best Place to Work” List (2006).

Agenda The Annotated Web SearchMonkey Demo Technology DataRSS format Query language Lessons learned Toward Semantic Search BOSS Build your Own Search Service Y!OS 1.0 Yahoo! Open Strategy

The Annotated Web

SearchMonkey

Demo

Technology

DataRSS format

Query language

Lessons learned

Toward Semantic Search

BOSS

Build your Own Search Service

Y!OS 1.0

Yahoo! Open Strategy

The Annotated Web

Previously in search Horizontal search Yahoo… Keyword-based indexing Minimal natural language processing Limited experiments with ontologies (query expansion) Vertical search e.g. shopping.com, Kelkoo Faceted search, browsing Fixed ontology Combinations Google Base, Google Co-op Web-scale, but fixed ontologies Proprietary technology Can we do better with the Semantic Web? Address the long tail of queries (88% of queries) Use standard technology Not a new question. But the answer may be new.

Horizontal search

Yahoo…

Keyword-based indexing

Minimal natural language processing

Limited experiments with ontologies (query expansion)

Vertical search

e.g. shopping.com, Kelkoo

Faceted search, browsing

Fixed ontology

Combinations

Google Base, Google Co-op

Web-scale, but fixed ontologies

Proprietary technology

Can we do better with the Semantic Web?

Address the long tail of queries (88% of queries)

Use standard technology

Not a new question. But the answer may be new.

Which Semantic Web? Two visions Data Web Bringing the content of databases to the Web (linkeddata.org) Rich data, heavyweight semantics Deep Web Annotated Web Annotating the content of Web resources (documents, mm) Simple data, lightweight semantics Shallow Web This presentation is about the Annotated Web.

Two visions

Data Web

Bringing the content of databases to the Web (linkeddata.org)

Rich data, heavyweight semantics

Deep Web

Annotated Web

Annotating the content of Web resources (documents, mm)

Simple data, lightweight semantics

Shallow Web

This presentation is about the Annotated Web.

Brief history of the Annotated Web 1995: HTML meta tags 1996: Simple HTML Ontology Extensions (SHOE) 1998: RDF/XML RDF/XML in HTML RDF linked from HTML 2003: Web 2.0 Tagging Microformats Metadata in Wikipedia Machine tags in Flickr 2005: eRDF 2008: RDFa

1995: HTML meta tags

1996: Simple HTML Ontology Extensions (SHOE)

1998: RDF/XML

RDF/XML in HTML

RDF linked from HTML

2003: Web 2.0

Tagging

Microformats

Metadata in Wikipedia

Machine tags in Flickr

2005: eRDF

2008: RDFa

HTML meta tags <HTML> <HEAD profile=&quot;http://dublincore.org/documents/dcq-html/&quot;> <META name=&quot;DC.author &quot; content=&quot; Peter Mika &quot;> <LINK rel=&quot;DC.rights copyright&quot; href=&quot; http://www.example.org/rights.html &quot; /> <LINK rel=&quot;meta&quot; type=&quot;application/rdf+xml&quot; title=&quot;FOAF&quot; href= &quot; http://www.cs.vu.nl/~pmika/foaf.rdf &quot;> </HEAD> … </HTML>

<HTML>

<HEAD profile=&quot;http://dublincore.org/documents/dcq-html/&quot;>

<META name=&quot;DC.author &quot; content=&quot; Peter Mika &quot;>

<LINK rel=&quot;DC.rights copyright&quot; href=&quot; http://www.example.org/rights.html &quot; />

<LINK rel=&quot;meta&quot; type=&quot;application/rdf+xml&quot; title=&quot;FOAF&quot;

href= &quot; http://www.cs.vu.nl/~pmika/foaf.rdf &quot;>

</HEAD>

…

</HTML>

SHOE example (Hefflin & Hendler, 1996) <ONTOLOGY &quot;our-ontology&quot; VERSION=&quot;1.0&quot;> <ONTOLOGY-EXTENDS &quot;organization-ontology&quot; VERSION=&quot;2.1&quot; PREFIX=&quot;org&quot; URL=&quot;http://www.ont.org/orgont.html&quot;> <ONTDEF CATEGORY=&quot;Person&quot; ISA=&quot;org.Thing&quot;> <ONTDEF RELATION=&quot;lastName&quot; ARGS=&quot;Person STRING&quot;> <ONTDEF RELATION=&quot;firstName&quot; ARGS=&quot;Person STRING&quot;> <ONTDEF RELATION=&quot;marriedTo&quot; ARGS=&quot;Person Person&quot;> <ONTDEF RELATION=&quot;employee&quot; ARGS=&quot;org.Organization Person&quot;> </ONTOLOGY > <HEAD> <META HTTP-EQUIV=&quot;Instance-Key&quot; CONTENT=&quot;http://www.cs.umd.edu/~george&quot;> <USE-ONTOLOGY &quot;our-ontology&quot; VERSION=&quot;1.0&quot; PREFIX=&quot;our&quot; URL=&quot;http://ont.org/our-ont.html&quot;> </HEAD> <BODY> <CATEGORY &quot;our.Person&quot;> <RELATION &quot;our.marriedTo&quot; TO=&quot;http://www.cs.umd.edu/~helena&quot;> <RELATION &quot;our.employee&quot; FROM=&quot;http://www.cs.umd.edu&quot;> My name is <ATTRIBUTE &quot;our.firstName&quot;> George </ATTRIBUTE> <ATTRIBUTE &quot;our.lastName&quot;> Cook </ATTRIBUTE> and I live at...

<ONTOLOGY &quot;our-ontology&quot; VERSION=&quot;1.0&quot;>

<ONTOLOGY-EXTENDS &quot;organization-ontology&quot; VERSION=&quot;2.1&quot; PREFIX=&quot;org&quot; URL=&quot;http://www.ont.org/orgont.html&quot;>

<ONTDEF CATEGORY=&quot;Person&quot; ISA=&quot;org.Thing&quot;>

<ONTDEF RELATION=&quot;lastName&quot; ARGS=&quot;Person STRING&quot;>

<ONTDEF RELATION=&quot;firstName&quot; ARGS=&quot;Person STRING&quot;>

<ONTDEF RELATION=&quot;marriedTo&quot; ARGS=&quot;Person Person&quot;>

<ONTDEF RELATION=&quot;employee&quot; ARGS=&quot;org.Organization Person&quot;>

</ONTOLOGY >

SHOE system

SHOE Text-based query interface

SHOE Graphical Query Interface

Example: Creative Commons Embedding CC license in HTML (now deprecated): <HTML> <HEAD>… </HEAD> <BODY> … <!–- <rdf:RDF xmlns=&quot;http://creativecommons.org/ns#&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot; xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot;> <Work rdf:about=&quot;http://www.yergler.net/averages/&quot;> <dc:title>The Law of Averages</dc:title> <dc:description>...because eventually i&apos;ll be right...</dc:description> <license rdf:resource=&quot;http://creativecommons.org/licenses/by-nc/1.0/&quot; /> </Work> <License rdf:about=&quot;http://creativecommons.org/licenses/by-nc/1.0/&quot;> <requires rdf:resource=&quot;http://web.resource.org/cc/Notice&quot; /> <permits rdf:resource=&quot;http://web.resource.org/cc/Reproduction&quot; /> <permits rdf:resource=&quot;http://web.resource.org/cc/Distribution&quot; /> <prohibits rdf:resource=&quot;http://web.resource.org/cc/CommercialUse&quot; /> </License> </rdf:RDF> -->

Embedding CC license in HTML (now deprecated):

Example: Creative Commons Current: rel attribute (HTML4) This work is licensed under a <a rel=&quot;license&quot; href=&quot;http://creativecommons.org/licenses/by/3.0/us/&quot;>Creative Commons Attribution 3.0 United States License</a>. Use of the “rel” attribute for semantic annotation is the birth of the microformat…

Current: rel attribute (HTML4)

Use of the “rel” attribute for semantic annotation is the birth of the microformat…

Example: microformats <cite class=&quot; vcard &quot;> <a class=&quot; fn url &quot; rel=&quot;friend colleague met&quot; href=&quot;http://meyerweb.com/&quot;> Eric Meyer </a> </cite> wrote a post ( <cite> <a href=&quot;http://meyerweb.com/eric/thoughts/2005/12/16/tax-relief/&quot;> Tax Relief </a></cite> ) about an unintentionally humorous letter he received from the <span class=&quot; vcard &quot;> <a class=&quot; fn org url &quot; href=&quot;http://irs.gov/&quot;> Internal Revenue Service </a> </span>. <div class=&quot; vcard &quot;> <a class=&quot; email fn &quot; href=&quot;mailto:jfriday@host.com&quot;> Joe Friday </a> <div class=&quot; tel &quot;> +1-919-555-7878 </div> <div class=&quot; title &quot;> Area Administrator, Assistant </div> </div>

microformats microformats.org Originated by Tantek Celik and others Agreements on the way to encode certain kinds metadata in HTML Reuse of semantic-bearing HTML elements Based on existing standards Community process Persons, events , listings etc. but also syntactic metadata: licenses , tags Microformats have no shared syntax Each microformat has a separate syntax tailored to the vocabulary Microformats are not ontologies No formal descriptions of schema , only text Limited reuse, extensibility of schemas No datatypes No namespaces, unique identifiers (URIs) no interlinking mapping between instances is required Relationship to page context is unclear Widely used in millions of documents User-generated as well as automatically generated

microformats.org

Originated by Tantek Celik and others

Agreements on the way to encode certain kinds metadata in HTML

Reuse of semantic-bearing HTML elements

Based on existing standards

Community process

Persons, events , listings etc. but also syntactic metadata: licenses , tags

Microformats have no shared syntax

Each microformat has a separate syntax tailored to the vocabulary

Microformats are not ontologies

No formal descriptions of schema , only text

Limited reuse, extensibility of schemas

No datatypes

No namespaces, unique identifiers (URIs)

no interlinking

mapping between instances is required

Relationship to page context is unclear

Widely used in millions of documents

User-generated as well as automatically generated

Example: tags and machine tags

Example: Tags and machine tags Tags User defined keywords Minimal agreement Is ‘rock’ on Flickr same as ‘rock’ on myspace? Is ‘rock’ by me on Flickr is the same as ‘rock’ by you on Flickr? Is ‘rock’ by me on Flickr today the same as ‘rock’ by me on myspace tomorrow? Machine tags User defined values for user defined properties Possibility to define the namespace (but not enforced) Limited use

Tags

User defined keywords

Minimal agreement

Is ‘rock’ on Flickr same as ‘rock’ on myspace?

Is ‘rock’ by me on Flickr is the same as ‘rock’ by you on Flickr?

Is ‘rock’ by me on Flickr today the same as ‘rock’ by me on myspace tomorrow?

Machine tags

User defined values for user defined properties

Possibility to define the namespace (but not enforced)

Limited use

RDF-based annotation #1: eRDF eRDF Ian Davis (Talis) Embedding RDF in HTML Straightforward mapping to RDF triples (XSLT available) HTML4 compatible More complex than microformats Use any RDF/OWL vocabulary Reuse of semantic-bearing HTML elements is limited More limited than RDF No blank nodes No data types No statements about subjects other than the current document Limited usage

eRDF

Ian Davis (Talis)

Embedding RDF in HTML

Straightforward mapping to RDF triples (XSLT available)

HTML4 compatible

More complex than microformats

Use any RDF/OWL vocabulary

Reuse of semantic-bearing HTML elements is limited

More limited than RDF

No blank nodes

No data types

No statements about subjects other than the current document

Limited usage

RDF-based annotation #2: RDFa RDFa World Wide Web Consortium (W3C) last call document Similar intent as eRDF, but full RDF support Requires XHTML Big question: user complexity (  data quality) <p typeof=&quot;contact:Info&quot; about=&quot;http://example.org/staff/jo&quot;> <span property=&quot;contact:fn&quot;> Jo Smith </span>. <span property=&quot;contact:title&quot;> Web hacker </span> at <a rel=&quot;contact:org&quot; href=&quot;http://example.org&quot;> Example.org </a>. You can contact me <a rel=&quot;contact:email&quot; href=&quot;mailto:jo@example.org&quot;> via email </a>. </p> ...

RDFa

World Wide Web Consortium (W3C) last call document

Similar intent as eRDF, but full RDF support

Requires XHTML

Big question: user complexity (  data quality)

SearchMonkey

Creating an ecosystem of publishers, developers and end-users Motivating and helping publishers to implement semantic annotation Providing tools for developers to create compelling applications Focusing on end-user experience Rich abstracts as a first application Addressing the long tail of query and content production Standard Semantic Web technology dataRSS = Atom + RDFa Industry standard vocabularies http://developer.yahoo.com/searchmonkey/ SearchMonkey

Creating an ecosystem of publishers, developers and end-users

Motivating and helping publishers to implement semantic annotation

Providing tools for developers to create compelling applications

Focusing on end-user experience

Rich abstracts as a first application

Addressing the long tail of query and content production

Standard Semantic Web technology

dataRSS = Atom + RDFa

Industry standard vocabularies

http://developer.yahoo.com/searchmonkey/

Before After an open platform for using structured data to build more useful and relevant search results What is SearchMonkey?

image deep links name/value pairs or abstract Enhanced Result

YAHOO! CONFIDENTIAL | Infobar

SearchMonkey Acme.com’s database Index RDF/Microformat Markup site owners/publishers share structured data with Yahoo!. 1 consumers customize their search experience with Enhanced Results or Infobars 3 site owners & third-party developers build SearchMonkey apps. 2 DataRSS feed Web Services Page Extraction Acme.com’s Web Pages

Developer tool

Developer tool

Developer tool

Developer tool

Developer tool

Gallery

Example apps LinkedIn hCard plus feed data Creative Commons by Ben Adida CC in RDFa

LinkedIn

hCard plus feed data

Creative Commons by Ben Adida

CC in RDFa

Example apps. II. Other me by Dan Brickley Google Social Graph API wrapped using a Web Service

Other me by Dan Brickley

Google Social Graph API wrapped using a Web Service

DataRSS <?profile http://search.yahoo.com/searchmonkey-profile ?> <feed xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://www.w3.org/2005/Atom ../latest/xsd/datarss.xsd“> <id>http://www.linkedin.com/datarss/</id> <author> <name>Peter Mika (pmika@yahoo-inc.com)</name> </author> <title>Example data feed for social</title> <updated>2007-11-14T04:05:06+07:00</updated> <entry> <!-- title field of entry is not used for anything --> <title>Peter Mika</title> <!--URL of the webpage extracted from --> <id>http://www.linkedin.com/ppl/webprofile?id=5054019</id> <updated>2007-11-14T04:05:06+07:00</updated> <content type=&quot;application/xml&quot;> <y:adjunct version=&quot;1.0&quot; name=&quot;social-simple&quot; xmlns:y=&quot;http://search.yahoo.com/datarss/&quot;> <y:item rel=&quot;dc:subject&quot;> <y:type typeof=&quot;foaf:Person&quot;> <y:meta property=&quot;foaf:name&quot;>John Doe</y:meta> <y:meta property=&quot;foaf:gender&quot;>male</y:meta> <y:item rel=&quot;foaf:homepage&quot; resource=&quot;http://www.joeisageek.com&quot;/> <y:item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:johndoe@example.org&quot;/> <y:item rel=&quot;foaf:weblog&quot; resource=&quot;http://johnblog.example.org&quot;/> <y:item rel=&quot;foaf:knows&quot;> <y:type typeof=&quot;foaf:Person&quot;> <y:meta property=&quot;foaf:name&quot;>Jane Doe</y:meta> <y:meta property=&quot;foaf:gender&quot;>female</y:meta> <y:item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:janedoe@example.org&quot;/> </y:type> </y:item> </y:type> </y:item> </y:adjunct> </entry> </feed> Atom 1.0 XML + RDFa

<?profile http://search.yahoo.com/searchmonkey-profile ?>

<feed xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;

xsi:schemaLocation=&quot;http://www.w3.org/2005/Atom ../latest/xsd/datarss.xsd“>

<id>http://www.linkedin.com/datarss/</id>

<author>

<name>Peter Mika (pmika@yahoo-inc.com)</name>

</author>

<title>Example data feed for social</title>

<updated>2007-11-14T04:05:06+07:00</updated>

<entry>

<!-- title field of entry is not used for anything -->

<title>Peter Mika</title>

<!--URL of the webpage extracted from -->

<id>http://www.linkedin.com/ppl/webprofile?id=5054019</id>

<updated>2007-11-14T04:05:06+07:00</updated>

<content type=&quot;application/xml&quot;>

<y:adjunct version=&quot;1.0&quot; name=&quot;social-simple&quot; xmlns:y=&quot;http://search.yahoo.com/datarss/&quot;>

<y:item rel=&quot;dc:subject&quot;>

<y:type typeof=&quot;foaf:Person&quot;>

<y:meta property=&quot;foaf:name&quot;>John Doe</y:meta>

<y:meta property=&quot;foaf:gender&quot;>male</y:meta>

<y:item rel=&quot;foaf:homepage&quot; resource=&quot;http://www.joeisageek.com&quot;/>

<y:item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:johndoe@example.org&quot;/>

<y:item rel=&quot;foaf:weblog&quot; resource=&quot;http://johnblog.example.org&quot;/>

<y:item rel=&quot;foaf:knows&quot;>

<y:type typeof=&quot;foaf:Person&quot;>

<y:meta property=&quot;foaf:name&quot;>Jane Doe</y:meta>

<y:meta property=&quot;foaf:gender&quot;>female</y:meta>

<y:item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:janedoe@example.org&quot;/>

</y:type>

</y:item>

</y:type>

</y:item>

</y:adjunct>

</entry>

</feed>

The data part <adjunct version=&quot;1.0&quot; id=“com.yahoo.page.rdfa&quot; xmlns=&quot;http://search.yahoo.com/datarss/“ updated=“2007-11-14T04:05:06+07:00”> <item rel=&quot;dc:subject&quot;> <type typeof =&quot;foaf:Person&quot;> <meta property =&quot;foaf:name&quot;>John Doe</meta> <meta property=&quot;foaf:gender&quot;>male</meta> <item rel =&quot;foaf:homepage&quot; resource =&quot;http://www.joeisageek.com&quot;/> <item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:johndoe@example.org&quot;/> <item rel=&quot;foaf:weblog&quot; resource=&quot;http://johnblog.example.org&quot;/> <item rel=&quot;foaf:knows&quot;> <type typeof=&quot;foaf:Person&quot;> <meta property=&quot;foaf:name&quot;>Jane Doe</meta> <meta property=&quot;foaf:gender&quot;>female</meta> <item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:janedoe@example.org&quot;/> </type> </item> </type> </item> </adjunct>

<adjunct version=&quot;1.0&quot; id=“com.yahoo.page.rdfa&quot; xmlns=&quot;http://search.yahoo.com/datarss/“

updated=“2007-11-14T04:05:06+07:00”>

<item rel=&quot;dc:subject&quot;>

<type typeof =&quot;foaf:Person&quot;>

<meta property =&quot;foaf:name&quot;>John Doe</meta>

<meta property=&quot;foaf:gender&quot;>male</meta>

<item rel =&quot;foaf:homepage&quot; resource =&quot;http://www.joeisageek.com&quot;/>

<item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:johndoe@example.org&quot;/>

<item rel=&quot;foaf:weblog&quot; resource=&quot;http://johnblog.example.org&quot;/>

<item rel=&quot;foaf:knows&quot;>

<type typeof=&quot;foaf:Person&quot;>

<meta property=&quot;foaf:name&quot;>Jane Doe</meta>

<meta property=&quot;foaf:gender&quot;>female</meta>

<item rel=&quot;foaf:mbox&quot; resource=&quot;mailto:janedoe@example.org&quot;/>

</type>

</item>

</type>

</item>

</adjunct>

DataRSS An Atom extension for structured data Why a new format? A feed format is required by publishers Exclusive content (e.g. partnerships, paid inclusion) No changes necessary to the web page No standard named graph format for the Semantic Web Needed to capture meta-metadata such as source and timestamp of information Not really a new format An Atom extension Use any RDFa parser to get the triples out cf. Google Base feeds

An Atom extension for structured data

Why a new format?

A feed format is required by publishers

Exclusive content (e.g. partnerships, paid inclusion)

No changes necessary to the web page

No standard named graph format for the Semantic Web

Needed to capture meta-metadata such as source and timestamp of information

Not really a new format

An Atom extension

Use any RDFa parser to get the triples out

cf. Google Base feeds

What happened since the launch? It’s starting to work! Click rates improve  Publishers are willing to invest  More structured data  More applications  More users  Click rates improve Increasing excitement all around Standardization of RDFa is bringing new energy Good market for companies that help publishers to ‘semantify’ or support developers in extracting structured from web pages OpenCalais, Dapper, AdaptiveBlue, Intel MashMaker, Zemanta… There have been some lessons learned…

It’s starting to work!

Click rates improve  Publishers are willing to invest  More structured data  More applications  More users  Click rates improve

Increasing excitement all around

Standardization of RDFa is bringing new energy

Good market for companies that help publishers to ‘semantify’ or support developers in extracting structured from web pages

OpenCalais, Dapper, AdaptiveBlue, Intel MashMaker, Zemanta…

There have been some lessons learned…

Data quality Publishers/developers want the quick and dirty answer, not the long and clean one Resource or literal? <meta property=“vcard:url”>http://www.example.org</meta> <meta property=“vcard:tel”>0034691792522</meta> Webpage or resource? Should we allow a resource have the same URI as an existing webpage? This is the default in eRDF/RDFa! <div class=“foaf-name”>Peter Mika</div> Types vs. datatypes <meta property=“use:email”>pmika@cs.vu.nl</meta> Extensibility rdfs:movies Complexity of the formalism = Data quality down

Publishers/developers want the quick and dirty answer, not the long and clean one

Resource or literal?

<meta property=“vcard:url”>http://www.example.org</meta>

<meta property=“vcard:tel”>0034691792522</meta>

Webpage or resource?

Should we allow a resource have the same URI as an existing webpage?

This is the default in eRDF/RDFa!

<div class=“foaf-name”>Peter Mika</div>

Types vs. datatypes

<meta property=“use:email”>pmika@cs.vu.nl</meta>

Extensibility

rdfs:movies

Vocabularies Coverage is small Books, movies, stuff people care about… Competing proposals Versions floating around Not maintained I cannot maintain your vocabulary for you Vocabularies for microformats A must The role of the W3C Ontologies as member submission…. Vocabularies not designed for the annotated Web Distributed ontology development = Mess

Coverage is small

Books, movies, stuff people care about…

Competing proposals

Versions floating around

Not maintained

I cannot maintain your vocabulary for you

Vocabularies for microformats

A must

The role of the W3C

Ontologies as member submission….

Vocabularies not designed for the annotated Web

eRDF Difficult for complex pages and dangerous in non-expert hands Serious limitations No datatypes No subjects other than identifiers within the current page Reuse of the id attribute <div id=“blue_panel”> <span class=“foaf-name”>Peter Mika</span> <div id=“white_clickable_thingy_on_the right”> <span class=“foaf-mbox”>pmika@hotmail.com</span> … .

Difficult for complex pages and dangerous in non-expert hands

Serious limitations

No datatypes

No subjects other than identifiers within the current page

Reuse of the id attribute

<div id=“blue_panel”>

<span class=“foaf-name”>Peter Mika</span>

<div id=“white_clickable_thingy_on_the right”>

<span class=“foaf-mbox”>pmika@hotmail.com</span>

… .

RDFa A huge improvement E.g. no repurposing of HTML attributes Still, not everything is intuitive to the uninitiated: <div about=“#id”> <span property=“foaf:name“>Peter Mika</span> <span rel=“foaf:img“ typeof=“foaf:Image”> <span property=“dc:format”>jpg</span> … </span </div> <div about=“#id”> <span property=“foaf:name“>Peter Mika</span> <span rel=“foaf:img“ resource=“http://www.example.org/photo.jpg”> <span typeof=“foaf:Image”> <span property=“dc:format”>jpg</span> </span </span </div>

A huge improvement

E.g. no repurposing of HTML attributes

Still, not everything is intuitive to the uninitiated:

<div about=“#id”>

<span property=“foaf:name“>Peter Mika</span>

<span rel=“foaf:img“ typeof=“foaf:Image”>

<span property=“dc:format”>jpg</span>

…

</span

</div>

<div about=“#id”>

<span property=“foaf:name“>Peter Mika</span>

<span rel=“foaf:img“ resource=“http://www.example.org/photo.jpg”>

<span typeof=“foaf:Image”>

<span property=“dc:format”>jpg</span>

</span

</span

</div>

Semantic Search

Semantics and IR Hard searches that cannot be solved with purely syntactical approach Ontologies in IR shown to work in limited domains In Web IR most attempts (e.g. query expansion) have failed What is new? The scale and breadth. Growth in annotations, all domains (Web 2.0) Data Web vs. Deep Web

Hard searches that cannot be solved with purely syntactical approach

Ontologies in IR shown to work in limited domains

In Web IR most attempts (e.g. query expansion) have failed

What is new? The scale and breadth.

Growth in annotations, all domains (Web 2.0)

Data Web vs. Deep Web

Hard searches Ambiguous searches Paris Hilton Multimedia search Images of Paris Hilton Imprecise or overly precise searches Publications by Jim Hendler Find images of strong and adventurous people (Lenat) Searches for descriptions Search for yourself without using your name Product search (ads!) Searches that require aggregation Size of the Eiffer tower (Lenat) Public opinion on Britney Spears Queries that require a deeper understanding of the query, the content and/or the world at large Note: some of these are so hard that users don’t even try them any more

Ambiguous searches

Paris Hilton

Multimedia search

Images of Paris Hilton

Imprecise or overly precise searches

Publications by Jim Hendler

Find images of strong and adventurous people (Lenat)

Searches for descriptions

Search for yourself without using your name

Product search (ads!)

Searches that require aggregation

Size of the Eiffer tower (Lenat)

Public opinion on Britney Spears

Queries that require a deeper understanding of the query, the content and/or the world at large

Note: some of these are so hard that users don’t even try them any more

Example…

Application: query intent Paris Hilton is a person!

Application: query intent #2 Hugo is a person!

Time to experiment! Future work in IR Ranking documents Ranking resources/triples Ranking resources in the presence of documents New interfaces Query interface Result presentation Metrics Precision/recall CTR ROI

Future work in IR

Ranking documents

Ranking resources/triples

Ranking resources in the presence of documents

New interfaces

Query interface

Result presentation

Metrics

Precision/recall

CTR

ROI

Time to experiment! #2 Future work in Semantic Web (Semi-)automated ways of metadata creation (NLP!) Entity resolution Scale Data quality We allow providing metadata for other people’s sites! Reasoning To the extent that it’s useful Constraints Keyword-based search can not suffer Preference for very conservative solutions Metadata in the context of documents Assumption is still that the user will want to see/visit the source (what to do with linked data?)

Future work in Semantic Web

(Semi-)automated ways of metadata creation (NLP!)

Entity resolution

Scale

Data quality

We allow providing metadata for other people’s sites!

Reasoning

To the extent that it’s useful

Constraints

Keyword-based search can not suffer

Preference for very conservative solutions

Metadata in the context of documents

Assumption is still that the user will want to see/visit the source

(what to do with linked data?)

Going even further… NLP, Information Extraction New types of application Aggregators Stateful apps Intent-driven apps Mobile apps … Beyond search Analysis, design, diagnosis etc. on top of aggregated data Personalization Building rich user profiles Re-ranking results based on interests Monetization No more “buy virgins on eBay”

NLP, Information Extraction

New types of application

Aggregators

Stateful apps

Intent-driven apps

Mobile apps

…

Beyond search

Analysis, design, diagnosis etc. on top of aggregated data

Personalization

Building rich user profiles

Re-ranking results based on interests

Monetization

No more “buy virgins on eBay”

BOSS: Build your Own Search Service Unlimited queries per day Ability to re-order results and blend-in addition content No restrictions on presentation No branding or attribution Access to multiple verticals (web search, image, news) Ability to monetize 40+ supported language and region pairs http://developer.yahoo.com/search/boss/

Unlimited queries per day

Ability to re-order results and blend-in addition content

No restrictions on presentation

No branding or attribution

Access to multiple verticals (web search, image, news)

Ability to monetize

40+ supported language and region pairs

http://developer.yahoo.com/search/boss/

Y!OS 1.0 Yahoo! Open Strategy Yahoo! Social Platform Profiles, Connections, Updates, Contacts and Status Yahoo! Query Language (YQL) Access (other) web services using a SQL-like language Yahoo! Application Platform (YAP) Developer hosted execution of applications with access to Yahoo's Social APIs and YQL; Support for OpenSocial's JavaScript API ; and Support for server-side YML tags. Future: run applications on Y! sites OpenID, OAuth

Yahoo! Open Strategy

Yahoo! Social Platform

Profiles, Connections, Updates, Contacts and Status

Yahoo! Query Language (YQL)

Access (other) web services using a SQL-like language

Yahoo! Application Platform (YAP)

Developer hosted execution of applications with access to Yahoo's Social APIs and YQL;

Support for OpenSocial's JavaScript API ; and

Support for server-side YML tags.

Future: run applications on Y! sites

OpenID, OAuth

Contact Peter Mika [email_address] Come to Barcelona and stop by SearchMonkey developer.yahoo.com/searchmonkey/ mailing lists [email_address] [email_address] forums http://suggestions.yahoo.com/searchmonkey http://suggestions.yahoo.com/datarss Semantic Web FAQ http://devel.yahoo.com/searchmonkey/smguide/faq.html

Peter Mika

[email_address]

Come to Barcelona and stop by

SearchMonkey

developer.yahoo.com/searchmonkey/

mailing lists

[email_address]

[email_address]

forums

http://suggestions.yahoo.com/searchmonkey

http://suggestions.yahoo.com/datarss

Semantic Web FAQ

http://devel.yahoo.com/searchmonkey/smguide/faq.html

the monkey is out!