Year of the Monkey Peter Mika Researcher and Data Architect Yahoo!

SearchMonkey Acme.com’s database Index RDF/Microformat Markup site owners/publishers share structured data with Yahoo!. 1 consumers customize their search experience with Enhanced Results or Infobars 3 site owners & third-party developers build SearchMonkey apps. 2 DataRSS feed Web Services Page Extraction Acme.com’s Web Pages

March, 2008: The news is out! “ Yahoo’s support for semantic web standards like RDF and microformats is exactly the incentive websites need to adopt them. Instead of semantic silos scattered across the Web (think Twine), Yahoo will be pulling all the semantic information together when available, as a search engine should. Until now, there were few applications that demanded properly structured data from third parties. That changes today.” – TechCrunch

“ Yahoo’s support for semantic web standards like RDF and microformats is exactly the incentive websites need to adopt them. Instead of semantic silos scattered across the Web (think Twine), Yahoo will be pulling all the semantic information together when available, as a search engine should. Until now, there were few applications that demanded properly structured data from third parties. That changes today.” – TechCrunch

May, 2008: Time to party!

What happened since the launch? It’s working in and out We are contributing significantly to the growth of the Semantic Web Users are delighted  Publishers are willing to invest  More structured data  More applications  More users are delighted Increasing excitement all around A year later, even Google gets excited This presentation is about the things we learned along the way How far are we? Lessons about technology Lessons about our communities Moving ahead

It’s working in and out

We are contributing significantly to the growth of the Semantic Web

Users are delighted  Publishers are willing to invest  More structured data  More applications  More users are delighted

Increasing excitement all around

A year later, even Google gets excited

This presentation is about the things we learned along the way

How far are we?

Lessons about technology

Lessons about our communities

Moving ahead

How far are we?

The Web is getting more structured

But just how far are we: lots of data or very little? Percentage of URLs with embedded metadata in various formats Sep, 2008 Mar, 2009 >400% increase in RDFa data

The Semantic Gap The real question is whether this data serves a purpose Our purpose: fulfilling the information needs of our users It’s not about the size! Consider Wikipedia. Demand= needs Supply = information

The real question is whether this data serves a purpose

Our purpose: fulfilling the information needs of our users

It’s not about the size! Consider Wikipedia.

Analysis through query logs Research questions: How much of this data would ever be encountered by a user through search? What categories of queries can be answered? What’s the role of large sites? Method Imitating the average search behavior of users through web search query log analysis Reproducible experiments (given query log data) BOSS web search API Returns metadata for search result URLs in RDF/XML or DataRSS

Research questions:

How much of this data would ever be encountered by a user through search?

What categories of queries can be answered?

What’s the role of large sites?

Method

Imitating the average search behavior of users through web search query log analysis

Reproducible experiments (given query log data)

BOSS web search API

Returns metadata for search result URLs in RDF/XML or DataRSS

Data Microformats, eRDF, RDFa data Query log data US query log Random sample of 7k queries Recent query log covering over a month period Query classification data US query log 1000 queries classified into various categories

Microformats, eRDF, RDFa data

Query log data

US query log

Random sample of 7k queries

Recent query log covering over a month period

Query classification data

US query log

1000 queries classified into various categories

Caveats For us, and for the time being, search = document search For this experiment, we assume current bag-of-words document retrieval is a reasonable approximation of semantic search For us, search = web search We are dealing with the average search user There are many queries the users have learned not to ask Volume is a rough approximation of value There are rare information needs with high pay-offs, e.g. patent search, financial data, biomedical data…

For us, and for the time being, search = document search

For this experiment, we assume current bag-of-words document retrieval is a reasonable approximation of semantic search

For us, search = web search

We are dealing with the average search user

There are many queries the users have learned not to ask

Volume is a rough approximation of value

There are rare information needs with high pay-offs, e.g. patent search, financial data, biomedical data…

Number of queries with a given number of results with particular formats (N=7081) Impressions Average impressions per query Notes: Queries with 0 results with metadata not shown You cannot add numberss in columns: a query may return documents with different formats Assume queries return more than 10 results On average, a query has at least one result with metadata. Are tags as useful as hCard? That’s only 1 in every 16 queries. 1 2 3 4 5 6 7 8 9 10 ANY 2127 1164 492 244 85 24 10 5 3 1 7623 1.08 hcard 1457 370 93 11 3 0 0 0 0 0 2535 0.36 rel-tag 1317 350 95 44 14 8 6 3 1 1 2681 0.38 adr 456 77 21 6 1 0 0 0 0 0 702 0.10 hatom 450 52 8 1 0 0 0 0 0 0 582 0.08 license 359 21 1 1 0 0 0 0 0 0 408 0.06 xfn 339 26 1 1 0 0 0 1 0 0 406 0.06

Notes:

Queries with 0 results with metadata not shown

You cannot add numberss in columns: a query may return documents with different formats

Assume queries return more than 10 results

The influence of head sites (N=7081) Impressions Average impressions per query If YouTube came up with a microformat, it would be the fifth most important. 1 2 3 4 5 6 7 8 9 10 ANY 2127 1164 492 244 85 24 10 5 3 1 7623 1.08 hcard 1457 370 93 11 3 0 0 0 0 0 2535 0.36 rel-tag 1317 350 95 44 14 8 6 3 1 1 2681 0.38 wikipedia.org 1676 1 0 0 0 0 0 0 1 0 1687 0.24 adr 456 77 21 6 1 0 0 0 0 0 702 0.10 hatom 450 52 8 1 0 0 0 0 0 0 582 0.08 youtube.com 475 1 0 0 0 0 0 2 0 0 493 0.07 license 359 21 1 1 0 0 0 0 0 0 408 0.06 xfn 339 26 1 1 0 0 0 1 0 0 406 0.06 amazon.com 345 3 0 0 0 0 1 0 0 0 358 0.05

Restricted by category: local queries (N=129) Impressions Average impressions per query The query category largely determines which sites are important. 1 2 3 4 5 6 7 8 9 10 ANY 36 16 10 0 4 1 0 0 0 0 124 0.96 hcard 31 7 5 1 0 0 0 0 0 0 64 0.50 adr 15 8 2 1 0 0 0 0 0 0 41 0.32 local.yahoo.com 24 0 0 0 0 0 0 0 0 0 24 0.19 en.wikipedia.org 24 0 0 0 0 0 0 0 0 0 24 0.19 rel-tag 19 2 0 0 0 0 0 0 0 0 23 0.18 geo 16 5 0 0 0 0 0 0 0 0 26 0.20 www.yelp.com 16 0 0 0 0 0 0 0 0 0 16 0.12 www.yellowpages.com 14 0 0 0 0 0 0 0 0 0 14 0.11

Summary It’s not how much, it’s how useful For us, it’s a matter of who is looking for it We would need to break down usefulness Usefulness for improving presentation Usefulness for ranking Usefulness for disambiguation … Usefulness by how much someone is willing to pay for it Linked Data will need to be studied separately

It’s not how much, it’s how useful

For us, it’s a matter of who is looking for it

We would need to break down usefulness

Usefulness for improving presentation

Usefulness for ranking

Usefulness for disambiguation

…

Usefulness by how much someone is willing to pay for it

Linked Data will need to be studied separately

Lessons about technology

Publisher’s dilemma: choosing the syntax I. microformats Cover the most common types of objects microformats.org, but also ‘unofficial’ such as Facebook Share Great for the publisher As long as it exactly fits: microformats are not extensible Various degrees of adoption and readiness Not so great from a parsing perspective Combinations of microformats are a problem Low end of semantics Very little structure

microformats

Cover the most common types of objects

microformats.org, but also ‘unofficial’ such as Facebook Share

Great for the publisher

As long as it exactly fits: microformats are not extensible

Various degrees of adoption and readiness

Not so great from a parsing perspective

Combinations of microformats are a problem

Low end of semantics

Very little structure

Publisher’s dilemma: choosing the syntax II. eRDF The dying breed, is rapidly replaced by RDFa RDFa When microformats are not enough Full RDF expressiveness Bring your own vocabulary Syntax is difficult with many caveats e.g. the notion of unfinished or ‘dangling’ triples e.g. combination of the rel and typeof attributes Still, RDFa is on the rise and now the default choice

eRDF

The dying breed, is rapidly replaced by RDFa

RDFa

When microformats are not enough

Full RDF expressiveness

Bring your own vocabulary

Syntax is difficult with many caveats

e.g. the notion of unfinished or ‘dangling’ triples

e.g. combination of the rel and typeof attributes

Still, RDFa is on the rise and now the default choice

Publisher’s dilemma: choosing the syntax III. DataRSS feeds Good for private data and not having to modify the page XSLT Often a good initial step in experimentation Writing XSLTs is hard, but visual tools such as MashMaker help No support for GRDDL yet Linked Data using RDF/XML Relationship of Linked Data and HTML pages is unclear Linked data is often published for someone else, e.g. Dbpedia Not supported by our infrastructure yet

DataRSS feeds

Good for private data and not having to modify the page

XSLT

Often a good initial step in experimentation

Writing XSLTs is hard, but visual tools such as MashMaker help

No support for GRDDL yet

Linked Data using RDF/XML

Relationship of Linked Data and HTML pages is unclear

Linked data is often published for someone else, e.g. Dbpedia

Not supported by our infrastructure yet

The language tradeoff: complexity vs. simplicity Complexity means more mistakes and less adoption Resource or literal? <meta property=“vcard:url”>http://www.example.org</meta> <meta property=“vcard:tel”>0034691792522</meta> Webpage or resource? Should we allow a resource have the same URI as an existing webpage? This is the default in eRDF/RDFa! Types vs. datatypes <meta datatype=“mytype:email”>pmika@cs.vu.nl</meta> Extensibility rdfs:movies But simplicity results in loss of semantics <span class=“open”> <b>Mon-Wed</b> 10am-5pm </span> Not the last word on the subject, see HTML5

Complexity means more mistakes and less adoption

Resource or literal?

<meta property=“vcard:url”>http://www.example.org</meta>

<meta property=“vcard:tel”>0034691792522</meta>

Webpage or resource?

Should we allow a resource have the same URI as an existing webpage?

This is the default in eRDF/RDFa!

Types vs. datatypes

<meta datatype=“mytype:email”>pmika@cs.vu.nl</meta>

Extensibility

rdfs:movies

But simplicity results in loss of semantics

<span class=“open”> <b>Mon-Wed</b> 10am-5pm </span>

Not the last word on the subject, see HTML5

Where are the vocabularies? No vocabularies in many domains Books, movies, stuff people care about… Too many competing proposals in other domains Often versions of the same proposal Example: vocabularies for microformats Not maintained I cannot maintain your vocabulary for you Limited tool support Too many expert tools until now Many vocabularies are not designed for annotation Missing meeting point and social process An ontology is a shared , formal representation of a domain

No vocabularies in many domains

Books, movies, stuff people care about…

Too many competing proposals in other domains

Often versions of the same proposal

Example: vocabularies for microformats

Not maintained

I cannot maintain your vocabulary for you

Limited tool support

Too many expert tools until now

Many vocabularies are not designed for annotation

Missing meeting point and social process

An ontology is a shared , formal representation of a domain

How do we build communities? www.vocamp.org

Learning about our communities

Publishers, developers, users and marketers Most applications are developed by the site owner Exposing only the data that is required for the application How to encourage other types of applications and more data? Typical developer is a front-end engineer Mostly new to semantic technologies, but motivated However, ramp-up is steep. Learning RDF plus SearchMonkey. How to simplify development? Users have little interest in customizing their search experience They attach less value to customization than we thought How to give our users more value without the need for customization?

Most applications are developed by the site owner

Exposing only the data that is required for the application

How to encourage other types of applications and more data?

Typical developer is a front-end engineer

Mostly new to semantic technologies, but motivated

However, ramp-up is steep. Learning RDF plus SearchMonkey.

How to simplify development?

Users have little interest in customizing their search experience

They attach less value to customization than we thought

How to give our users more value without the need for customization?

Helping our publishers What if we could remove the need for programming? SearchMonkey objects Generate enhanced results based on markup in common formats Copy-paste code Validator

What if we could remove the need for programming?

SearchMonkey objects

Generate enhanced results based on markup in common formats

Copy-paste code

Validator

LATE BREAKING NEWS Five new objects: Product, Local, News, Event, Discussion

Opening up new ways of accessing structured data BOSS (Build your Own Search Service) API Full service web search API Access metadata with search results view=searchmonkey_rdf&format=xml Use magic words to restrict search to results with certain kinds of metadata e.g. searchmonkey:com.yahoo.page.uf.hcard YQL Query web services as virtual relational tables Create mashups by joining tables The microformats ‘table’ allows similar access as with BOSS

BOSS (Build your Own Search Service) API

Full service web search API

Access metadata with search results

view=searchmonkey_rdf&format=xml

Use magic words to restrict search to results with certain kinds of metadata

e.g. searchmonkey:com.yahoo.page.uf.hcard

YQL

Query web services as virtual relational tables

Create mashups by joining tables

The microformats ‘table’ allows similar access as with BOSS

New default on applications for our users

Moving ahead

Summary CELEBRATING 1 YEAR ANNIVERSARY OF SEARCHMONKEY In 23 markets around the world 70 million enhanced results viewed daily (US) >15% increase in click-through rates 200 people enter dev tool to start creating an app a day >15,000 developers registered to build apps >400 applications in gallery Amount of RDFa structured data increased by 413%

What we’ve done Opened up SearchMonkey via BOSS and YQL Significantly simplified the work of publishers and developers Improved user experience through a number of new applications Working with the community to establish standards and best practices Vocabulary developers (microformats, FOAF, SIOC, GoodRelations etc.) Standard organizations (e.g. providing feedback for HTML5) Semantic data consumers and producers (e.g. Common Tag) The Semantic Web community (e.g. VoCamps)

Opened up SearchMonkey via BOSS and YQL

Significantly simplified the work of publishers and developers

Improved user experience through a number of new applications

Working with the community to establish standards and best practices

Vocabulary developers (microformats, FOAF, SIOC, GoodRelations etc.)

Standard organizations (e.g. providing feedback for HTML5)

Semantic data consumers and producers (e.g. Common Tag)

The Semantic Web community (e.g. VoCamps)

The Future Supporting the community with new tools We need your feedback! Creating entirely new search experiences based on structured data Intent-driven Task/goal oriented Stateful Delightful

Supporting the community with new tools

We need your feedback!

Creating entirely new search experiences based on structured data

Intent-driven

Task/goal oriented

Stateful

Delightful

Contact Peter Mika [email_address] SearchMonkey developer.yahoo.com/searchmonkey/ mailing lists [email_address] [email_address] forums http://suggestions.yahoo.com/searchmonkey

Peter Mika

[email_address]

SearchMonkey

developer.yahoo.com/searchmonkey/

mailing lists

[email_address]

[email_address]

forums

http://suggestions.yahoo.com/searchmonkey

Hear more about what we do Wednesday 8:30AM: Executive Round Table on Semantic Search with Andrew Tomkins, Chief Scientist, Yahoo Search Wednesday 11:45AM: SearchMonkey and the Semantic Web with Kevin Haas, Senior Engineering Manager, Yahoo! Search Wednesday 2:30PM: Year of the Monkey: Lessons from the first year of SearchMonkey with Peter Mika, Yahoo! Research Thursday 11AM: The Semantic Web Gang looks back at SemTech 2009 with Peter Mika, Yahoo! Research

Wednesday 8:30AM: Executive Round Table on Semantic Search with Andrew Tomkins, Chief Scientist, Yahoo Search

Wednesday 11:45AM: SearchMonkey and the Semantic Web with Kevin Haas, Senior Engineering Manager, Yahoo! Search

Wednesday 2:30PM: Year of the Monkey: Lessons from the first year of SearchMonkey with Peter Mika, Yahoo! Research

Thursday 11AM: The Semantic Web Gang looks back at SemTech 2009 with Peter Mika, Yahoo! Research