Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Remixing web data for your hacks
                   the easy way
                YQL – the can opener of the web
Hacking together systems
in 24 hours is a lot of fun.
But you want to spend that
time thinking about the
interface.
And not how you get to the
right data in the right
format.
The web is full of juicy and
long lasting data.




                     hquot;p://www.flickr.com/photos/clspeace/162336973/
And there is a lot of it
around.




                      hquot;p://www.flickr.com/photos/tudor/2981410947/
However, our attempts to
get to it can be clumsy.




                  hquot;p://www.flickr.com/photos/lumachrome/21403687...
What we need is an easy
way to get to that data.




                   hquot;p://www.flickr.com/photos/carey@lden/11543522...
We had a way to do that
for quite a while now.
http://pipes.yahoo.com/
Pipes, however is high end
technology…




                    hquot;p://www.flickr.com/photos/axio/2346342672/
We’re developers, not
interface users.




                   hquot;p://www.flickr.com/photos/codepo8/2278641937/
So for a long time people
asked Yahoo for a
command line version of
pipes.

Can we have one?
Yes, we can!
The Yahoo Query
Language, or short YQL is a
unified interface language
to the web.
http://developer.yahoo.com/yql/
Using YQL, accessing the
web and its services
becomes as easy as SQL:

select {what} from
{service} where
{condition}
Say you want kittens in
your hack (who doesn’t?)

select * from
flickr.photos.search
where text=quot;kittenquot;
Say you want kittens in
your hack (who doesn’t?)
Say you only want 5 kittens

select * from
flickr.photos.search
where text=quot;kittenquot;
limit 5
Say you only want 5 kittens
Nice, but where can you
get this?
YQL is a REST API in itself,
and it has two endpoints.
The public endpoint does
not need any
authentication.

http://query.yahooapis.com/v1/
public/yql?q={query}
&format={format}
The private endpoint needs
oauth authentication.

http://query.yahooapis.com/v1/
yql?q={query}
&format={format}
hquot;p://...
Output formats are XML or
JSON.
JSON also allows for a
callback parameter to use
the output directly as
JSON-P.
http://query.yahooapis.com/v1/public/yql?
q=select%20*%20from flickr.photos.search
%20where%20text=%22kitten%22limit
%205&...
You can mix and match
several web services using
the in() command.
Guess what this does:

select * from
flickr.photos.info where
photo_id in (select id from
flickr.photos.search where
woe_i...
Find London, UK, without a
doubt.
select * from
flickr.photos.info where
photo_id in (select id from
flickr.photos.search ...
Find Photos of London, UK

select * from
flickr.photos.info where
photo_id in (select id from
flickr.photos.search where
w...
Find Photos of London, UK
with a “By” CC license
select * from
flickr.photos.info where
photo_id in (select id from
flickr...
Give me everything you
know about these photos.
select * from
flickr.photos.info where
photo_id in (select id from
flickr....
Using a command like this
and some out-of-the-box UI
elements like Yahoo Maps
and the YUI carousel, you
can build somethin...
http://isithackday.com/hacks/cantine/
index.php?loc=covent+garden
Instead of selecting all the
information you can also
limit the results:
select name,url from
upcoming.venue where metro_i...
select name,url from
upcoming.venue where metro_id in
(select id from upcoming.metro
where search_text=quot;stokeyquot;)
The diagnostics part of the returned
data shows you what happened and
how long it took
Here’s what Yahoo offers you in this
  format:
flickr.photos.exif             local.search            mybloglog.community.fi...
Here’s what Yahoo offers you in this
  format:
search.images                 search.images                 search.images
s...
You want even more?




             http://www.flickr.com/photos/verylastexcitingmoment/3123597774/
Alright, how about this?


              json
atom
              microformats
csv
              rss
feed
              xml...
Telegraph’s headlines
anyone?
select * from html where
url='http://www.telegraph.co.uk/'
and xpath='//h3/a'
Telegraph’s headlines
anyone?
You can dynamically create
YQL queries to collate
several sources…
Then use cURL to pull them
off the web with one single
http request!
YQL caches and
compresses the results for
you.

Pretty easy, isn’t it?
Here’s another fun part:
Anyone can be part of the
YQL interface.
All we need the data
provider to do is to create a
schema that explains their
data structure.

http://developer.yahoo.com/...
And people do…
amazon              nmm
bitly               nyt
delicious           opensocial
dopplr              search
e...
For example the national
maritime museum:
select * from
nmm.archive.search where
searchterm=‘horatio nelson'
select * from
nmm.archive.search where
searchterm=‘horatio nelson'
That’s pretty cool –
allowing anyone to be part
of this interface.
The only shame is that you
can’t do all the things in
YQL that you can do in
Pipes – for example string
manipulation.
We wondered how to make
this possible.

One thing we didn’t want
to sacrifice is the simplicity
of the language itself.
So instead of inventing an
own language, we decided
to piggy-back on one you
already know.
YQL execute allows you to
embed JavaScript in the
open table schema that
runs on the YQL server and
converts the data for ...
For example you can
augment an existing service
to do something different.
The following example
adds a rank to search
results.
http://www.yqlblog.net/samples/searchrank.xml
Stored as XML this can be
used in a YQL query:
use 'http://yqlblog.net/
samples/searchrank.xml' as
searchrank;
select * fr...
Anyways, the *easiest* way
to start with YQL is to use
the console.
http://developer.yahoo.com/yql/console/
Of course, you can also
spend half the hack day
reading API docs 
Check out some code
examples on.

http://isithackday.com/hacks/ohd-london
THANKS!
Chris Heilmann
http://twitter.com/codepo8
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Open Hack London - Introduction to YQL
Upcoming SlideShare
Loading in …5
×

Open Hack London - Introduction to YQL

9,647 views

Published on

An introduction to YQL for hackers at the open hack day in London, talking about code demos at http://isithackday.com/hacks/ohd-london/

Published in: Education, Technology, Sports

Open Hack London - Introduction to YQL

  1. 1. Remixing web data for your hacks the easy way YQL – the can opener of the web
  2. 2. Hacking together systems in 24 hours is a lot of fun.
  3. 3. But you want to spend that time thinking about the interface.
  4. 4. And not how you get to the right data in the right format.
  5. 5. The web is full of juicy and long lasting data. hquot;p://www.flickr.com/photos/clspeace/162336973/
  6. 6. And there is a lot of it around. hquot;p://www.flickr.com/photos/tudor/2981410947/
  7. 7. However, our attempts to get to it can be clumsy. hquot;p://www.flickr.com/photos/lumachrome/2140368742/
  8. 8. What we need is an easy way to get to that data. hquot;p://www.flickr.com/photos/carey@lden/115435226/
  9. 9. We had a way to do that for quite a while now.
  10. 10. http://pipes.yahoo.com/
  11. 11. Pipes, however is high end technology… hquot;p://www.flickr.com/photos/axio/2346342672/
  12. 12. We’re developers, not interface users. hquot;p://www.flickr.com/photos/codepo8/2278641937/
  13. 13. So for a long time people asked Yahoo for a command line version of pipes. Can we have one?
  14. 14. Yes, we can!
  15. 15. The Yahoo Query Language, or short YQL is a unified interface language to the web. http://developer.yahoo.com/yql/
  16. 16. Using YQL, accessing the web and its services becomes as easy as SQL: select {what} from {service} where {condition}
  17. 17. Say you want kittens in your hack (who doesn’t?) select * from flickr.photos.search where text=quot;kittenquot;
  18. 18. Say you want kittens in your hack (who doesn’t?)
  19. 19. Say you only want 5 kittens select * from flickr.photos.search where text=quot;kittenquot; limit 5
  20. 20. Say you only want 5 kittens
  21. 21. Nice, but where can you get this?
  22. 22. YQL is a REST API in itself, and it has two endpoints.
  23. 23. The public endpoint does not need any authentication. http://query.yahooapis.com/v1/ public/yql?q={query} &format={format}
  24. 24. The private endpoint needs oauth authentication. http://query.yahooapis.com/v1/ yql?q={query} &format={format} hquot;p://developer.yahoo.com/yql/guide/ authoriza@on‐access.html
  25. 25. Output formats are XML or JSON. JSON also allows for a callback parameter to use the output directly as JSON-P.
  26. 26. http://query.yahooapis.com/v1/public/yql? q=select%20*%20from flickr.photos.search %20where%20text=%22kitten%22limit %205&format=json&callback=meow
  27. 27. You can mix and match several web services using the in() command.
  28. 28. Guess what this does: select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  29. 29. Find London, UK, without a doubt. select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  30. 30. Find Photos of London, UK select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  31. 31. Find Photos of London, UK with a “By” CC license select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  32. 32. Give me everything you know about these photos. select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  33. 33. Using a command like this and some out-of-the-box UI elements like Yahoo Maps and the YUI carousel, you can build something *very* quickly.
  34. 34. http://isithackday.com/hacks/cantine/ index.php?loc=covent+garden
  35. 35. Instead of selecting all the information you can also limit the results: select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text=quot;stokeyquot;)
  36. 36. select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text=quot;stokeyquot;)
  37. 37. The diagnostics part of the returned data shows you what happened and how long it took
  38. 38. Here’s what Yahoo offers you in this format: flickr.photos.exif local.search mybloglog.community.find flickr.photos.info music.ar0st.id mybloglog.member flickr.photos.interes0ngness music.ar0st.popular mybloglog.member.contacts flickr.photos.recent music.ar0st.search mybloglog.member.newwithcontacts flickr.photos.search music.ar0st.similar mybloglog.member.newwithme flickr.photos.sizes music.release.ar0st mybloglog.members.find flickr.places music.release.id mybloglog.stats.find flickr.places.info music.release.popular upcoming.category geo.places music.release.search upcoming.country geo.places.ancestors music.track.id upcoming.events geo.places.belongtos music.track.popular upcoming.events.bes0nplace geo.places.children music.track.search upcoming.groups geo.places.neighbors music.video.category upcoming.metro geo.places.parent music.video.id upcoming.state geo.places.siblings music.video.popular upcoming.user geo.placetypes music.video.search upcoming.venue gnip.ac0vity music.video.similar
  39. 39. Here’s what Yahoo offers you in this format: search.images search.images search.images search.news search.news search.news search.siteexplorer.inlinks search.siteexplorer.inlinks search.siteexplorer.inlinks search.siteexplorer.pages search.siteexplorer.pages search.siteexplorer.pages search.spelling search.spelling search.spelling search.suggest search.suggest search.suggest search.termextract search.termextract search.termextract search.web search.web search.web social.connec0ons social.connec0ons social.connec0ons social.contacts social.contacts social.contacts social.presence social.presence social.presence social.profile social.profile social.profile social.updates social.updates social.updates weather.forecast weather.forecast weather.forecast yahoo.iden0ty yahoo.iden0ty yahoo.iden0ty yap.setsmallview yap.setsmallview yap.setsmallview
  40. 40. You want even more? http://www.flickr.com/photos/verylastexcitingmoment/3123597774/
  41. 41. Alright, how about this? json atom microformats csv rss feed xml html
  42. 42. Telegraph’s headlines anyone? select * from html where url='http://www.telegraph.co.uk/' and xpath='//h3/a'
  43. 43. Telegraph’s headlines anyone?
  44. 44. You can dynamically create YQL queries to collate several sources…
  45. 45. Then use cURL to pull them off the web with one single http request!
  46. 46. YQL caches and compresses the results for you. Pretty easy, isn’t it?
  47. 47. Here’s another fun part: Anyone can be part of the YQL interface.
  48. 48. All we need the data provider to do is to create a schema that explains their data structure. http://developer.yahoo.com/yql/guide/ yql-opentables-chapter.html
  49. 49. And people do… amazon nmm bitly nyt delicious opensocial dopplr search etsy shopping friendfeed social github twi?er Greader update.groovy guardian weather imdb wesabe iploca0on whitepages las>m yahoo nestoria yelp ne>lix zillow http://github.com/spullara/yql-tables/tree/master
  50. 50. For example the national maritime museum: select * from nmm.archive.search where searchterm=‘horatio nelson'
  51. 51. select * from nmm.archive.search where searchterm=‘horatio nelson'
  52. 52. That’s pretty cool – allowing anyone to be part of this interface.
  53. 53. The only shame is that you can’t do all the things in YQL that you can do in Pipes – for example string manipulation.
  54. 54. We wondered how to make this possible. One thing we didn’t want to sacrifice is the simplicity of the language itself.
  55. 55. So instead of inventing an own language, we decided to piggy-back on one you already know.
  56. 56. YQL execute allows you to embed JavaScript in the open table schema that runs on the YQL server and converts the data for you. http://developer.yahoo.com/yql/guide/yql- execute-chapter.html
  57. 57. For example you can augment an existing service to do something different.
  58. 58. The following example adds a rank to search results. http://www.yqlblog.net/samples/searchrank.xml
  59. 59. Stored as XML this can be used in a YQL query: use 'http://yqlblog.net/ samples/searchrank.xml' as searchrank; select * from searchrank where query='pizza' and dispurl like '%pizzahut%'
  60. 60. Anyways, the *easiest* way to start with YQL is to use the console. http://developer.yahoo.com/yql/console/
  61. 61. Of course, you can also spend half the hack day reading API docs 
  62. 62. Check out some code examples on. http://isithackday.com/hacks/ohd-london
  63. 63. THANKS! Chris Heilmann http://twitter.com/codepo8

×