Open Hack 2008 Searchmonkey

Feeding the Monkey: the SearchMonkey data layer, presentation applications, and you

SearchMonkey Presentation by: Evan Goer, SearchMonkey Community Manager (goer@yahoo-inc.com) Paul Tarjan, SearchMonkey Devtool Developer (ptarjan@yahoo-inc.com) http://www.slideshare.net/searchmonkey/open-hack-2008-searchmonkey-presentation

What is SearchMonkey? an open platform for using structured data to build more useful and relevant search results Before After

Enhanced Result Key/value Pairs or abstract Links Image

Infobar Summary Blob

How to get data to SearchMonkey?

Humans see:

name

picture of a person

current job

industry, …

Computers see:

an undifferentiated

blob of HTML

Can we make computers smarter?

Artificial intelligence is hard. Plus…

How does it Work? How does it work? Acme.com’s DB Index RDF/Microformat Markup site owners/publishers share structured data with Yahoo! 1 consumers customize their search experience with Enhanced Results or Infobars 3 site owners & third-party developers build SearchMonkey apps 2 DataRSS feed Web Services Page Extraction Acme.com’s Site

Data Sources: RDF and Microformats Name Cached Open Mode Notes Yahoo! Index yes yes Passive Old-School Y! Index data RDFa, eRDF yes yes Passive Vocab + markup decoupled Microformats yes yes Passive Vocab + markup coupled DataRSS feed yes no Active Atom + metadata XSLT no no Active Good for prototyping Web Service no no Active Brings in remote data

Approach #1: Embedded RDF <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN” "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd"> <html xmlns= http://www.w3.org/1999/xhtml xmlns:dc= http://purl.org/dc/elements/1.1/ xmlns:foaf= http://xmlns.com/foaf/0.1/ lang="en" xml:lang="en"> <head> <title>The Amazing Home Page of Joe Smith</title> </head> <body> <h1 property="dc:title">Joe's Home Page</h1> <div rel="foaf:maker"> <h2 property="foaf:name">Joe Smith</h2> <div rel="foaf:depiction" resource="http://joesmith.org/images/jsmith.png"> <img src="/images/jsmith.png" alt="Smiling headshot of Joe" /> <p property="dc:rights">Creative Commons Attribution 3.0 Unported</p> </div> </div> …

Cached data

allows Enhanced Results

but not for dynamic data

Reuse existing markup

but requires site redesign

Open approach

everyone can use

Passive, crawled by Y!

less bureaucracy to set up

Approach #2: Embedded Microformats <div id="hcard-Joe-Smith" class="vcard"> <span class="fn">Joe Smith</span> <div class="adr"> <div class="street-address">123 Murphy Avenue</div> <span class="locality">Sunnyvale</span>, <span class="region">California</span> <span class="postal-code">94086</span> </div> <div class="tel">(408) 555-1234</div> </div> …

Cached data

Reuse existing markup

but requires site redesign

Open approach

everyone can use

Passive, crawled by Y!

less bureaucracy to set up

Data Sources: DataRSS Feed Name Cached Open Mode Notes Yahoo! Index yes yes Passive Old-School Y! Index data RDFa, eRDF yes yes Passive Vocab + markup decoupled Microformats yes yes Passive Vocab + markup coupled DataRSS feed yes no Active Atom + metadata XSLT no no Active Good for prototyping Web Service no no Active Brings in remote data

Approach #3: DataRSS Feed <?profile http://search.yahoo.com/searchmonkey-profile ?> <feed xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.w3.org/2005/Atom ../xsd/datarss.xsd" xmlns:dc="http://purl.org/dc/terms/” xmlns="http://www.w3.org/2005/Atom" xmlns:commerce="http://search.yahoo.com/searchmonkey/commerce/" xmlns:y="http://search.yahoo.com/datarss/"> <id>http://local.yahoo.com/datarss/</id> <author><name>Peter Mika (pmika@yahoo-inc.com)</name></author> <title>Example data feed for Local</title> <updated>2008-07-16T04:05:06+07:00</updated> <entry> <title>Parcel 104</title> <id>http://local.yahoo.com/info-21583016-parcel-104-santa-clara</id> <updated>2008-07-16T04:05:06+07:00</updated> <content type="application/xml"> <y:adjunct version="1.0" name="com.yahoo.local”> <y:item rel="dc:subject"> <y:type typeof="vcard:VCard commerce:Restaurant”> <y:meta property="commerce:hoursOfOperation"> Breakfast daily, Lunch Mon.-Fri., Dinner Mon.-Sat.

Cached data

Generate feed from DB

and maintain afterwards

Closed approach

only Yahoo! gets data

Actively provide a feed

coord w/Yahoo! to set up

Building with Structured Data

Structured data -> easy app building

Relies on RDF, microformats, DataRSS

That was the hard part – whew, you’re done 

PHP in a typical app

Mostly simple assignments, Data::get()

Possibly strings, XML, math

Use if statements to check whether fields exist

Need to punt? Just return an empty array

Data Sources: XSLT Extractors Name Cached Open Mode Notes Yahoo! Index yes yes Passive Old-School Y! Index data RDFa, eRDF yes yes Passive Vocab + markup decoupled Microformats yes yes Passive Vocab + markup coupled DataRSS feed yes no Active Atom + metadata XSLT no no Active Good for prototyping Web Service no no Active Brings in remote data

Approach #4: Extract with XSLT <?xml version="1.0"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"> <xsl:template match="/"> <adjunctcontainer> <adjunct id="smid:{$smid}" version="1.0"> <item rel="rel:Photo” resource="{//div[@class='hresume']//div[@class='image']/img/@src}"/> <item rel="rel:Card"> <meta property="vcard:fn"> <xsl:value-of select="//div[@class='hresume']//span[contains(@class,'fn')]"/> </meta> <meta property="vcard:title"> <xsl:value-of select="//div[@class='hresume']//ul[@class='current']/li"/> </meta> </item> </adjunct> </adjunctcontainer> </xsl:template> </xsl:stylesheet>

Generally not cached

too slow, infobar only

but good for dynamic data

Scrape page with XSLT

operates on cleaned up version of the DOM

watch out for template changes

Easy to prototype

Prototyping with XSLT

What if I don’t have structured data?

I don’t own the site

I do own the site, but I want to prototype first

Build an XSLT custom data service first

Write some XSLT to extract the data and transform it into DataRSS

Mostly about finding the right Xpath (use Firebug or Xpather )

Quick to implement, but brittle

Can’t do a good Enhanced Result

Creating an Infobar

Infobar advantages

Annotate someone else’s site

Use links and images from other domains

Mash up info from multiple sites

Affiliate / coupon links? Hmmm…

Can act on *, all websites

But these apps can be annoying if poorly designed

Key design principles

Put something useful in the summary

Be creative with the HTML

Data Sources: Web Services Name Cached Open Mode Notes Yahoo! Index yes yes Passive Old-School Y! Index data RDFa, eRDF yes yes Passive Vocab + markup decoupled Microformats yes yes Passive Vocab + markup coupled DataRSS feed yes no Active Atom + metadata XSLT no no Active Good for prototyping Web Service no no Active Brings in remote data

Approach #5: Call a Web Service <?xml version="1.0"?> <xsl:stylesheet xmlns:xsi= http://www.w3.org/2001/XMLSchema-instance xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0” xmlns:h= http://www.w3.org/1999/xhtml xmlns:y="urn:yahoo:srch” xsi:schemaLocation="urn:yahoo:srch http://api.search.yahoo.com/SiteExplorerService/V1/PageDataResponse.xsd "> <xsl:template match="/"> <adjunctcontainer xmlns:my="http://example.com/ns/1.0"> <adjunct id="smid:{$smid}" version="1.0"> <meta property="my:link1"> <xsl:value-of select="//y:Result[1]/y:Url"/> </meta> <meta property="my:result1"> <xsl:value-of select="//y:Result[1]/y:Title"/> </meta> </adjunct> </adjunctcontainer> </xsl:template> </xsl:stylesheet>

Generally not cached

too slow, infobar only

but good for dynamic data

Call a Remote Web Service

allows SearchMonkey apps to glue together

can handle OpenSearch XML natively

Resources

Main:

http://developer.yahoo.com/searchmonkey

Lists and Forums:

[email_address]

http://suggestions.yahoo.com/searchmonkey

RDF and Microformats:

http://microformats.org

http://www.w3.org/TR/xhtml-rdfa-primer

Next Steps

Identify content to use in SearchMonkey

Weigh the strengths and drawbacks of each method for providing data:

RDF

Microformats

DataRSS feed

Custom data services

Go build your data layer and app!

Come talk to us for help. 

Win Fabulous Prizes!

FINIS questions?

Open Hack 2008 Searchmonkey

1 comments

Notes on slide 1

Favorites, Groups & Events