Finding things on the
web with BOSS
Christian Heilmann | http://wait-till-i.com | http://scriptingenabled.org
Open Hack Day 2009 - Bangalore, India

What is innovation?

Innovation is improving the
current state of affairs.

In the best of all cases this
means that it gets easier for
the person using a product
to reach a goal.

This can be achieved by
connecting several different
products and turning them
into one. (pssst... Mashup).

I’ve seen this followed
cleverly on several levels
here in India.

BOSS is your chance to make
the web an easier to navigate
space.

You can help us turn a search
engine into a find engine.

Web search is broken!
http://luckyrobot.com/2009/02/06/search-is-broken-%E2%80%93-really-broken/
http://www.borthwick.com/weblog/2009/02/05/creative-destruction-google-slayed-by-the-notificator/

Back in the
days the
web was
small and
largely static
documents.

Nowadays it
is huge and
its content
and the
publication
speed
increased a
lot.

This is one challenge of
search engines these days.

The others are old, but also
increasing

Now, say you have
the most awesome
idea of a search
engine that works
around these
issues.

Where you will get stuck is
the overhead of indexing,
storing and filtering the data
of the web.

And this is where BOSS
comes in.

It is an API interface to our
data stores for search.

BOSS is Build Your Own
Search Service:
http://developer.yahoo.com/search/boss/

To use it, you need a
Application ID:
https://developer.yahoo.com/wsregapp/

And there is full
documentation available:
http://developer.yahoo.com/search/boss/boss_guide/

Happy Hacking!

... oh alright then ...

You can get the code
examples I will show here:
http://isithackday.com/hacks/
bangalore/bosscode.zip

Say you want to search the
web for donkeys.

Because
Donkeys alright then ...
... oh
Rock!

Using BOSS you can do this
with a REST API and display
the results any way you want!

The REST API:
boss.yahooapis.com/ysearch/{type}/v1/
{search}

The REST API:
boss.yahooapis.com/ysearch/{type}/v1/
{search}
type is what you want to
search:
web: the interwebs
news: new stuff
images: pictures

The REST API:
boss.yahooapis.com/ysearch/{type}/v1/
{search}
search is the term to look for
(url-encoded)
Put “” around terms to ensure the right order, f.e. “donkey
fur” (you don’t want to see cats, do you?)
Filter with a -, f.e. donkey -shrek
Restrict to a site using site:, f.e. donkey site:flickr.com

The REST API:
boss.yahooapis.com/ysearch/{type}/v1/
{search}
Other parameters:
appid: your app ID (needed)
count: amount of results
start: where to start the counting
region / lang: country and language
format: xml or json
sites: restrict to certain sites (comma separated)

Web search REST API:
boss.yahooapis.com/ysearch/web/v1/{search}
Extra parameters:
filter: To filter out nasties, use filter=-porn-hate
type: to search different types. You can use html, text, pdf,
xl, msword, ppt or groups like msoffice and nonhtml. You
can also do a type=msoffice,-xl

Image search REST API:
boss.yahooapis.com/ysearch/images/v1/
{search}
Extra parameters:
filter: no nudies
dimensions: all, small, medium, large, wallpaper,
widewallpaper
refererurl: all images in that url
url: image at that url

News search REST API:
boss.yahooapis.com/ysearch/news/v1/{search}
Extra parameters:
age: how old the news are in days. Last five days would be
“5d”

There are restrictions how to
display results and
information as to what data
comes back.

For this, read the guide!
http://developer.yahoo.com/search/boss/boss_guide/

Everybody Duck!

There will be code

The easiest way to use BOSS
is using JavaScript.

http://boss.yahooapis.com/
ysearch/web/v1/donkeys?
format=json&appid={id}

{\"ysearchresponse\":
{\"responsecode\":\"200\",\"nextpage\":\"\\/ysearch\\/web\\/
v1\\/donkeys?
format=json&appid=[...]&start=10\",\"totalhits\":\"492
215\",\"deephits\":\"15700000\",\"count\":\"10\",\"start\":\"0
\",\"resultset_web\":[{\"abstract\":\"Hyperlinked
description of the domesticated mammal discussing
its appearance, relationship to horses, economic
<b>...<\\/b> horses and <b>donkeys<\\/b> were
brought back <b>...<\\/b>\",\"clickurl\":\"http:\\/\\/
lrd.yahooapis.com\\/
_ylc=X3oDMTU4b2NoaDR2BF9TAzIwMjMxNTI3MDIEYXBwaWQDV
Fg2YjRYSFYzNEVuUFhXMHNZRXI1MWhQMXBuNU84S0FHcy5MUVN
YZXIxWjdSbW1WclpvdXo1U3Z5WGtXc1ZrLQRwb3MDMARzZXJ2a
WNlA1lTZWFyY2hXZWIEc2xrA3RpdGxlBHNyY3B2aWQDR3lDaEg
wU081cTlmSktUNG1ndTVUUUJNdlNjaS4wa1ZUVndBQVF5Sw--
\\/SIG=11820sato\\/**http%3A\\/\\/en.wikipedia.org\\

To use this across domains,
simply define a callback
parameter:

http://boss.yahooapis.com/
ysearch/web/v1/donkeys?
format=json&callback=foundd
onkeys&appid={id}

founddonkeys({\"ysearchresponse\":
{\"responsecode\":\"200\",\"nextpage\":\"\\/ysearch\\/web
\\/v1\\/donkeys?
format=json&callback=founddonkeys&appid=TX6b4XHV3
4EnPXW0sYEr51hP1pn5O8KAGs.LQSXer1Z7RmmVrZouz5SvyX
kWsVk-
&start=10\",\"totalhits\":\"492215\",\"deephits\":\"15700
000\",\"count\":\"10\",\"start\":\"0\",\"resultset_web\":
[{\"abstract\":\"Hyperlinked description of the
domesticated mammal discussing its appearance,
relationship to horses, economic <b>...<\\/b>
horses and <b>donkeys<\\/b> were brought back
<b>...<\\/b>\",\"clickurl\":\"http:\\/\\/
lrd.yahooapis.com\\/
_ylc=X3oDMTU4cG05cXJwBF9TAzIwMjMxNTI3MDIEYXBwaWQD
VFg2YjRYSFYzNEVuUFhXMHNZRXI1MWhQMXBuNU84S0FHcy5MU
VNYZXIxWjdSbW1WclpvdXo1U3Z5WGtXc1ZrLQRwb3MDMARzZX

All you then need to do is put
this url in a script node and
write the founddonkeys
function:

<div id=\"searchresults\"></div>
<script type=\"text/javascript\">
function founddonkeys(o){
var donkeys = o.ysearchresponse.resultset_web;
var results = document.createElement('ul');
for(var i=0,j=donkeys.length;i<j;i++){
var item = document.createElement('li');
var link = document.createElement('a');
var abstract = document.createElement('p');
link.setAttribute('href',donkeys[i].clickurl);
link.innerHTML = donkeys[i].title;
item.appendChild(link);
abstract.innerHTML = donkeys[i].abstract;
item.appendChild(abstract);
results.appendChild(item);
}

document.getElementById('searchresults').appendChild(r
esults);
}
</script>
<script type=\"text/javascript\" charset=\"utf-8\"
src=\"http://boss.yahooapis.com/ysearch/web/v1/donkeys?
format=json&callback=founddonkeys&appid=xxx\"></script>

Two problems though:

First of all - without
JavaScript there are no
donkeys!

Secondly - you can only find
donkeys!

The solution: Event Handling
and dynamic script
generation.

<p>Warning: this is terrible code, USE A LIBRARY INSTEAD!</p>
<ul id=\"searches\">
<li><a href=\"http://search.yahoo.com/search?va=donkeys\">
Search for Donkeys
</a>
</li>
<li><a href=\"http://search.yahoo.com/search?va=kittens\">
Search for kittens
</a>
</li>
</ul>
<div id=\"searchresults\"></div>

<script type=\"text/javascript\" charset=\"utf-8\">
function founddonkeys(o){
var donkeys = o.ysearchresponse.resultset_web;
var results = document.createElement('ul');
for(var i=0,j=donkeys.length;i<j;i++){
var item = document.createElement('li');
var link = document.createElement('a');
var abstract = document.createElement('p');
link.setAttribute('href',donkeys[i].clickurl);
link.innerHTML = donkeys[i].title;
item.appendChild(link);
abstract.innerHTML = donkeys[i].abstract;
item.appendChild(abstract);
results.appendChild(item);
}
var resultsbox = document.getElementById('searchresults');
resultsbox.innerHTML = '';
resultsbox.appendChild(results);
}
var APIkey = 'TX6b4XHV34EnPXW0sYEr51hP1pn5O8KAGs'+
'.LQSXer1Z7RmmVrZouz5SvyXkWsVk-';
var searchlinks =
document.getElementById('searches').getElementsByTagName('a');
for(var i=0;searchlinks[i];i++){

var APIkey = 'TX6b4XHV34EnPXW0sYEr51hP1pn5O8KAGs'+
'.LQSXer1Z7RmmVrZouz5SvyXkWsVk-';
var searchlinks = document.getElementById('searches').
getElementsByTagName('a');
for(var i=0;searchlinks[i];i++){
searchlinks[i].onclick = function(){
var searchterm = this.href.split('va=')[1];
var url = 'http://boss.yahooapis.com/ysearch/web/v1/' +
searchterm + '?format=json&' +
'callback=founddonkeys' + '&appid=' + APIkey
var s = document.createElement('script');
s.setAttribute('type','text/javascript');
s.setAttribute('src',url);
document.getElementsByTagName('head')[0].appendChild(s);
return false;
}
}
</script>

*click*

Using the YUI library (YUI3
JavaScript and CSS grids) you
can easily make this much
cooler:

To make using BOSS with
JavaScript easier, I’ve written
a BOSS wrapper called
YBOSS:
http://icant.co.uk/sandbox/yboss/

<div id=\"results\"></div>
<script type=\"text/javascript\" src=\"yboss-lib.js\">
</script>
<script type=\"text/javascript\">
YBOSS.get({
searches:'search,news',
query:'obama',
count:10,
callback:seedpics
});
function seedpics(o){
var all = '<h4>Web Sites</h4>' + o.webHTML +
'<h4>News</h4>' + o.newsHTML;
var out = document.getElementById('results');
out.innerHTML = all;
}
</script>

For server side code there are
code examples available at:
http://www.saurabhsahni.com/boss-
examples.zip

There’s also the Python
mashup framework that
allows for SQL for remixing
arbitrary XML/JSON sources:
http://developer.yahoo.com/search/boss/mashup.html

And there’s an easy way to
deploy BOSS apps at Google
App Engine:
http://zooie.wordpress.com/2008/08/04/yahoo-boss-
google-app-engine-integrated/

All this
has been
around for
a while.

Here are some new things
added lately:

People are trying to make the
web a less messier place by
adding semantic data to
HTML documents.

Using
SearchMonkey
technology
BOSS now
lists this
information in
the results.
http://www.flickr.com/photos/glenscott/3273401181/

view=searchmonkey_feed
view=searchmonkey_rdf
http://developer.yahoo.com/search/boss/
structureddata.html

Using the view=keyterms
parameter you can get
keywords associated with
each result.

http://keywordfinder.org

In order to get longer
descriptions of each result
you can now use the
abstract=long parameter to
get up to 300 characters
instead of 130.

Another thing we’ve done is
using the Yahoo Site Explorer
and bundle it with BOSS.

This way you can now get
page information and page
inlink information with two
new BOSS services.

http://boss.yahooapis.com/
ysearch/se_inlink/v1/{URL}?
appid={APPID}
Gets you a list of pages that
linked to the URL provided.

http://boss.yahooapis.com/
ysearch/se_pagedata/v1/
{URL}?appid={APPID}
Gets you a list of all pages
belonging to a domain in the
Yahoo! index.

So what has been done using
BOSS?

http://ask-boss.appspot.com/

http://ask-boss.appspot.com/

http://ask-boss.appspot.com/

http://hakia.com/

http://www.oneriot.com/

And on a more
lighter note:

The client side is where
strange things happen.
http://isithackday.com/hacks/web-the-adventure/

The motherlode of BOSS
implementations:
http://mashable.com/boss/
http://delicious.com/tag/bossmashup
Add yours by tagging it with
“bossmashup” on Del.icio.us!

THANKS!
Keep in touch:
Christian Heilmann
http://wait-till-i.com
http://scriptingenabled.org
http://twitter.com/codepo8