Semantic Search on the Public Web with Creative Commons

Semantic Search on the Public Web with Creative Commons

2006.03.07

Mike Linksvayer

Billion$ (0)

Let's get the hype out of the way....

Billion$ (1)

Billion$ (2)

Billion$ (3)

This calls for a mashup...

Billion$ (4)

Billion$ (5)

Fortunately CC's founders thought of that from the beginning...

Billion$ (6)

Billion$ (7)

About Creative Commons

Core Licensing Suite: Creator/Licensor chooses license options NonCommercial No Derivatives ShareAlike Every Creative Commons licenses allows the world to copy and distribute a work provided that the licensee credits the creator/licensor In addition creator/licensor may apply the following conditions:

Simple License Generator

Internet Archive Free Hosting for CC works http://www.archive.org/

Creative Commons Metadata

Creative Commons Metadata Example

<rdf:RDF xmlns="http://web.resource.org/cc/"

xmlns:dc="http://purl.org/dc/elements/1.1/"

xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">

<Work rdf:about="http://example.com/article.html">

<dc:title>An Example Article</dc:title>

<dc:date>2003-10-01</dc:date>

<dc:type rdf:resource="http://purl.org/dc/dcmitype/Text" />

<license rdf:resource="http://creativecommons.org/licenses/by-nc-sa/2.5/" />

</Work>

<License rdf:about="http://creativecommons.org/licenses/by-nc-sa/2.5/">

<permits rdf:resource="http://web.resource.org/cc/Reproduction" />

<permits rdf:resource="http://web.resource.org/cc/Distribution" />

<requires rdf:resource="http://web.resource.org/cc/Notice" />

<requires rdf:resource="http://web.resource.org/cc/Attribution" />

<prohibits rdf:resource="http://web.resource.org/cc/CommercialUse" />

<permits rdf:resource="http://web.resource.org/cc/DerivativeWorks" />

<requires rdf:resource="http://web.resource.org/cc/ShareAlike" />

</License>

</rdf:RDF>

Rights Description Use Cases Discovery Expression Commerce Management(1)

Rights Description vs. Rights Management(2) Copy/Use promotion vs. Copy/Use protection Encourage fans vs. Discourage casual pirates Resource management vs. Customer management Web content model vs. 20 th century content model Not mutually exclusive in theory.

Why Semantic Web? Small organization, no central registration for every license Decentralization: Let a thousand search engines bloom; web as API Existing RDF tools could take advantage of CC RDF

Why RDF-in-HTML comments? (yuck)

Considered:

Robots.txt-like

HTML meta tags

LINK to external RDF file

RDF-in-HTML comments wins because

Metadata colocated with human visible HTML, only single copy & paste for licensors

Full power of RDF

CC Search History I

Postgresql/tsearch2/python prototype (early 2004)

Sloooowwwww, but did what a prototype should do

CC Search History II

CC-Nutch (late 2004)

Nutch aims to be open source search engine comparable to commercial web scale search engines

Built on top of Lucene full text index

CC plugin only ~500 lines of code (not counting UI, CC-required additions to Nutch core)

http://search.creativecommons.org uses Nutch, >1m CC-licensed pages indexed

CC Search History III

Yahoo! Search for Creative Commons (early 2005)

Search CC-licensed subset of Yahoo!’s index (~15m* pages)

*very rough guesstimate

CC Search History IV

Google CC search (November 2005)

Search CC-licensed subset of Google’s index (~45m* pages)

*very rough guesstimate

CC Search History V (the future) Better metadata formats Image and Video search Derivatives search Content commerce search “ Live” web search “ Management” (desktop, workgroup) Semantic mashups

Future CC metadata formats

“ Semantic XHTML” AKA “lowercase semantic web” AKA “microformats” (now)

<a rel=“license” href=“ http://creativecommons.org/licenses/by/2.5/ ”>

RDF/A AKA XHTML2 metadata (in working group)

GRDDL (gleaning resource descriptions from dialects of languages)

Image and Video search Better metadata formats Image and Video search Derivatives search Content commerce search “ Live” web search “ Management” (desktop, workgroup) Semantic mashups

Searching for Derivative Works

Creative Commons (0)

Derivatives search RDF/XML snippet: <dc:source rdf:resource=”http://ccmixter.org/media/files/victor/3385”/> Query like Yahoo! link: search or Technorati Cosmos search source:http://ccmixter.org/media/files/victor/3385 “ Who sampled this” as the new “who linked to this”

Content commerce search Transaction costs should be low even if rights are reserved Commercial terms and other commerce described by metadata associated with a work Find me work I can use at a price I can pay for usage rights warranty/paper trail (even if rights not reserved) Reintermediate consumer and creator

“ Live” web search (feeds) Feeds are explicitly metadata-rich (unlike typical web page) Existing blog search ignores metadata Web search will become more like blog search, vice versa?

“ Management” (desktop, workgroup) Desktop search (OS-level) Content creation and media player integration XMP Semantic Wikis

Semantic mashups

Issues for Semantic Search on the Public Web Metadata quality Trust Scalability Usability Compatibility Critical mass State of the art IR works very well – high expectations!