Build your Own Search Service
Chris Heilmann
Saurabh Sahni
Open Hack Day 2009 - Bangalore
http://www.slideshare.net/saurabhsahni/

Outline
• Search engines using BOSS
• About BOSS API
– What?
– Why?
– Features
• How to use it
– BOSS API
– Code example
– BOSS Mashup framework
-2-

Search engines
using BOSS
-3-

hakia: http://hakia.com/
-4-

hakia: http://hakia.com/
-5-

hakia: http://hakia.com/
-6-

Cluuz: http://cluuz.com
-7-

Cluuz: http://cluuz.com
-8-

Cluuz: http://cluuz.com
-9-

Keyword finder - http://keywordfinder.org/
- 10 -

askBOSS: http://ask-boss.appspot.com/
- 11 -

askBOSS: http://ask-boss.appspot.com/
- 12 -

askBOSS: http://ask-boss.appspot.com/
- 13 -

askBOSS: http://ask-boss.appspot.com/
- 14 -

askBOSS: http://ask-boss.appspot.com/
- 15 -

About BOSS API
- 16 -

What?
• Open Yahoo’s core search features via web services to
let 3rd parties revolutionize Search
http://developer.yahoo.com/search/boss
- 17 -

Opening the search technology stack
Rank
Assist
EXTRACT
Retrieve
SPAM <-> Gold Usage
CRAWL
Web Map
Analyze
Index Index
50B pages * 20ms page download = 31 years
- 18 -

Opening the search technology stack
Your App here
WEB API
Rank
Assist
EXTRACT
Retrieve
SPAM <-> Gold Usage
CRAWL
Web Map
Analyze
Index Index
50B pages * 20ms page download = 31 years
- 19 -

Why?
• Removes entry barriers
• Asset to Innovate
– Develop new relevance models
– Change presentation style
• Search anywhere
– Improve Vertical Quality w/ Web comprehensiveness
- 20 -

BOSS API features
• No branding or attribution
• Ability to change presentation stlye
• Ability to re-order results and blend-in additional content
• Access to multiple verticals (web search, image, news)
• Keyword suggestions, spell checks
• Semantic data, in-links, abstracts
• Ability to monetize
- 21 -

How to use it?
- 22 -

Get Started
• Register for an application id
http://developer.yahoo.com/wsregapp/
• Documentation
http://developer.yahoo.com/search/boss/boss_guide/
• Code samples: Javascript, PHP and Python
http://www.saurabhsahni.com/boss-examples.zip
- 23 -

BOSS API
Searching
Slumdog
Millionaire
(Source: http://en.wikipedia.org/wiki/File:Slumdog_Millionaire_poster.jpg)
- 24 -

BOSS API
• Search for slumdog millionaire:
– http://boss.yahooapis.com/ysearch/web
/v1/slumdog+millionaire
?appid=xyz&format=xml
- 25 -

BOSS API: XML response
http://boss.yahooapis.com/ysearch/web/v1/slumdog+millionaire?appid=xyz&format=xml
- 26 -

Site Restrict Search
• Search for slumdog millionaire on selected movie sites
– Add param sites=indiatimes.com,movies.yahoo.com,imdb.com
– http://boss.yahooapis.com/ysearch/web/v1/slumdog
+millionaire?appid=xyz&sites=indiatimes.co
m%2Cmovies.yahoo.com&format=xml
- 27 -

http://boss.yahooapis.com/ysearch/web/v1/slumdog+millionaire?
appid=xyz&sites=indiatimes.com%2Cmovies.yahoo.com&format=xml
- 28 -

Search images
• http://boss.yahooapis.com/ysearch/images/v1
/slumdog +millionaire?dimensions=large
- 29 -

http://boss.yahooapis.com/ysearch/images/v1/
slumdog +millionaire
- 30 -

Search News
• http://boss.yahooapis.com/ysearch/news/v1
/slumdog +millionaire?age=15d
- 31 -

http://boss.yahooapis.com/ysearch/news/v1/
slumdog + millionaire?age=15d
- 32 -

Movie Search
Code Example
- 33 -

- 34 -

Movie Search Code Example
- 35 -

http://www.saurabhsahni.com/boss-examples.zip
- 36 -

More with
BOSS API
- 37 -

Related keywords
Add parameter view=keyterms
– http://boss.yahooapis.com/ysearch/web/v1/slumdog
+millionaire?appid=xyz&view=keyterms&format=xml
- 38 -

http://boss.yahooapis.com/ysearch/web/v1/slumdog +millionaire?appid=xyz&view=keyterms&format=xml
- 39 -

Semantic Data
• Access structured data acquired through SearchMonkey
- 40 -

Semantic Data
view=searchmonkey_rdf
view=searchmonkey_feed
http://developer.yahoo.com/search/boss/stuctureddata.html
- 41 -

http://boss.yahooapis.com/ysearch/web/v1/slumdog +millionaire?appid=xyz&
view=searchmonkey_feed&format=xml
- 42 -

Long abstracts
• Add parameter abstract=long
– get up to 300 characters instead of 130
- 43 -

Spell Check
http://boss.yahooapis.com/ysearch/spelling/v1/
milionare?format=xml
Response
- 44 -

BOSS Search API REST Interface
http://boss.yahooapis.com/ysearch/{vert}/v1/{query}
• {query}: term to look for (url-encoded)
• {vert} := {web, news, images, spelling}
• @ required
– appid
• @ optional
– start, count, lang, region, format, callback, sites, view
- 45 -

Site Explorer
• Get page inlinks
– http://boss.yahooapis.com/ysearch/se_inlink/v1/{URL}
?appid={APPID}
• Page data: collection of subpages in a domain
– http://boss.yahooapis.com/ysearch/se_pagedata/v1/{URL}
?appid={APPID}
- 46 -

BOSS Mashup Framework
• Python (v2.5+) library
• BOSS Search SDK plus …
• SQL for remixing arbitrary XML/JSON sources
http://developer.yahoo.com/search/boss/mashup.html
- 47 -

BMF + Google App Engine
• Enhanced version of BMF to GAE platform
• http://zooie.wordpress.com/2008/08/04/yahoo-boss-google-app-engine-integrated/
• Enables quick deployment of BOSS applications online
- 48 -

More BOSS Implementations
• http://mashable.com/boss/
• http://delicious.com/tag/bossmashup
• Add yours by tagging it with “bossmashup” on
Del.icio.us!
- 49 -

One more thing…
- 50 -

BOSS Custom
Your App here
WEB API
Rank
Assist
EXTRACT
Retrieve
SPAM <-> Gold Usage
CRAWL
Web Map
Analyze
Index
50B pages * 20ms page download = 31 years
- 51 -

Thank You
Questions?
More: http://developer.yahoo.com/search/boss/
Slides: http://www.slideshare.net/saurabhsahni/
- 52 -

Appendix
- 53 -

Search UI Templates are Included in the
BOSS Mashup Framework
http://www.yahoo.com
BOSS Mashup Framework simplifies aggregating and presenting multiple data sources
- 54 -

BMF Features
• select, group, sort, union, joins, udfs, where
• Text normalization and duplicate removal
• Auto-transformation of resource-oriented API results
into tables w/o parsing
• All-in-memory storage and retrieval operations
• Ability to join lists of tables via an arbitrary predicate
function (map-like)
• Search UI template framework
• Single search function provides total access to
BOSS REST API
- 55 -

BOSS in Academic Research
• The biggest dataset available on web
• Very useful for Web-mining research experiments
– Natural language processing
– Semantic extraction
– Related keywords
– Similarity detection
– Clustering algorithms
– Spelling corrections
- 56 -