Remixing web data for your hacks the easy way YQL – the can opener of the web
Hacking together systems in 24 hours is a lot of fun.
But you want to spend that time thinking about the interface.
And not how you get to the right data in the right format.
The web is full of juicy and long lasting data. http://www.flickr.com/photos/clspeace/162336973/
And there is a lot of it around. http://www.flickr.com/photos/tudor/2981410947/
However, our attempts to get to it can be clumsy. http://www.flickr.com/photos/lumachrome/2140368742/
What we need is an easy way to get to that data. http://www.flickr.com/photos/careytilden/115435226/
We had a way to do that for quite a while now.
http://pipes.yahoo.com/
Pipes, however is high end technology… http://www.flickr.com/photos/axio/2346342672/
We’re developers, not interface users. http://www.flickr.com/photos/codepo8/2278641937/
So for a long time people asked Yahoo for a command line version of pipes. Can we have one?
Yes, we can!
The Yahoo Query Language, or short YQL is a unified interface language to the web. http://developer.yahoo.com/yql/
Using YQL, accessing the web and its services becomes as easy as SQL: select {what} from {service} where {condition}
Say you want kittens in your hack (who doesn’t?) select * from flickr.photos.search where text="kitten"
Say you want kittens in your hack (who doesn’t?)
Say you only want 5 kittens select * from flickr.photos.search where text="kitten" limit 5
Say you only want 5 kittens
Nice, but where can you get this?
YQL is a REST API in itself, and it has two endpoints.
The public endpoint does not need any authentication. http://query.yahooapis.com/v1/public/yql?q={query} &format={format}
The private endpoint needs oauth authentication. http://query.yahooapis.com/v1/yql?q={query} &format={format} http://devel...
Output formats are XML or JSON.  JSON also allows for a callback parameter to use the output directly as JSON-P.
http://query.yahooapis.com/v1/public/yql?q=select%20*%20from flickr.photos.search%20where%20text=%22kitten%22limit%205&for...
 
You can mix and match several web services using the in() command.
Guess what this does: select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id...
Find photos in London, UK with a Creative Commons “By” license and give me all the information you have about them.
select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from...
Using a command like this and some out-of-the-box UI elements like Yahoo Maps and the YUI carousel, you can build somethin...
http://isithackday.com/hacks/cantine/index.php?loc=covent+garden
Instead of selecting all the information you can also limit the results: select name,url from upcoming.venue where metro_i...
select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")
The diagnostics part of the returned data shows you what happened and how long it took
Here’s what Yahoo offers you in this format: flickr.photos.exif flickr.photos.info flickr.photos.interestingness flickr.ph...
Here’s what Yahoo offers you in this format: search.images search.news search.siteexplorer.inlinks search.siteexplorer.pag...
You want even more? http://www.flickr.com/photos/verylastexcitingmoment/3123597774/
Alright, how about this? atom csv feed html json microformats rss xml
Telegraph’s headlines anyone? select * from html where url='http://www.telegraph.co.uk/' and xpath='//h3/a'
Telegraph’s headlines anyone?
You can dynamically create YQL queries to collate several sources…
Then use cURL to pull them off the web with one single http request!
YQL caches and compresses the results for you. Pretty easy, isn’t it?
Here’s another fun part: Anyone can be part of the YQL interface.
All we need the data provider to do is to create a schema that explains their data structure. http://developer.yahoo.com/y...
And people do… http://github.com/spullara/yql-tables/tree/master amazon bitly delicious dopplr etsy friendfeed github Grea...
For example the national maritime museum: select * from nmm.archive.search where searchterm=‘horatio nelson'
select * from nmm.archive.search where searchterm=‘horatio nelson'
That’s pretty cool – allowing anyone to be part of this interface.
The only shame is that you can’t do all the things in YQL that you can do in Pipes – for example string manipulation.
We wondered how to make this possible. One thing we didn’t want to sacrifice is the simplicity of the language itself.
So instead of inventing an own language, we decided to piggy-back on one you already know.
YQL execute allows you to embed JavaScript in the open table schema that runs on the YQL server and converts the data for ...
For example you can augment an existing service to do something different. The following example adds a rank to search res...
 
 
Stored as XML this can be used in a YQL query: use 'http://yqlblog.net/samples/searchrank.xml' as searchrank;  select * fr...
 
Anyways, the *easiest* way to start with YQL is to use the console. http://developer.yahoo.com/yql/console/
 
 
 
 
 
 
 
 
 
Of course, you can also spend half the hack day reading API docs  
Check out some code examples on. http://isithackday.com/hacks/ohd-london
THANKS! Chris Heilmann http://twitter.com/codepo8
Upcoming SlideShare
Loading in …5
×

YQL - Christian Heilmann Open Hack London presentation

4,323 views

Published on

Presentation about YQL

Published in: Technology, Sports
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
4,323
On SlideShare
0
From Embeds
0
Number of Embeds
2,006
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

YQL - Christian Heilmann Open Hack London presentation

  1. 1. Remixing web data for your hacks the easy way YQL – the can opener of the web
  2. 2. Hacking together systems in 24 hours is a lot of fun.
  3. 3. But you want to spend that time thinking about the interface.
  4. 4. And not how you get to the right data in the right format.
  5. 5. The web is full of juicy and long lasting data. http://www.flickr.com/photos/clspeace/162336973/
  6. 6. And there is a lot of it around. http://www.flickr.com/photos/tudor/2981410947/
  7. 7. However, our attempts to get to it can be clumsy. http://www.flickr.com/photos/lumachrome/2140368742/
  8. 8. What we need is an easy way to get to that data. http://www.flickr.com/photos/careytilden/115435226/
  9. 9. We had a way to do that for quite a while now.
  10. 10. http://pipes.yahoo.com/
  11. 11. Pipes, however is high end technology… http://www.flickr.com/photos/axio/2346342672/
  12. 12. We’re developers, not interface users. http://www.flickr.com/photos/codepo8/2278641937/
  13. 13. So for a long time people asked Yahoo for a command line version of pipes. Can we have one?
  14. 14. Yes, we can!
  15. 15. The Yahoo Query Language, or short YQL is a unified interface language to the web. http://developer.yahoo.com/yql/
  16. 16. Using YQL, accessing the web and its services becomes as easy as SQL: select {what} from {service} where {condition}
  17. 17. Say you want kittens in your hack (who doesn’t?) select * from flickr.photos.search where text="kitten"
  18. 18. Say you want kittens in your hack (who doesn’t?)
  19. 19. Say you only want 5 kittens select * from flickr.photos.search where text="kitten" limit 5
  20. 20. Say you only want 5 kittens
  21. 21. Nice, but where can you get this?
  22. 22. YQL is a REST API in itself, and it has two endpoints.
  23. 23. The public endpoint does not need any authentication. http://query.yahooapis.com/v1/public/yql?q={query} &format={format}
  24. 24. The private endpoint needs oauth authentication. http://query.yahooapis.com/v1/yql?q={query} &format={format} http://developer.yahoo.com/yql/guide/authorization-access.html
  25. 25. Output formats are XML or JSON. JSON also allows for a callback parameter to use the output directly as JSON-P.
  26. 26. http://query.yahooapis.com/v1/public/yql?q=select%20*%20from flickr.photos.search%20where%20text=%22kitten%22limit%205&format=json&callback=meow
  27. 28. You can mix and match several web services using the in() command.
  28. 29. Guess what this does: select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  29. 30. Find photos in London, UK with a Creative Commons “By” license and give me all the information you have about them.
  30. 31. select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text='london,uk') and license=4)
  31. 32. Using a command like this and some out-of-the-box UI elements like Yahoo Maps and the YUI carousel, you can build something *very* quickly.
  32. 33. http://isithackday.com/hacks/cantine/index.php?loc=covent+garden
  33. 34. Instead of selecting all the information you can also limit the results: select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")
  34. 35. select name,url from upcoming.venue where metro_id in (select id from upcoming.metro where search_text="stokey")
  35. 36. The diagnostics part of the returned data shows you what happened and how long it took
  36. 37. Here’s what Yahoo offers you in this format: flickr.photos.exif flickr.photos.info flickr.photos.interestingness flickr.photos.recent flickr.photos.search flickr.photos.sizes flickr.places flickr.places.info geo.places geo.places.ancestors geo.places.belongtos geo.places.children geo.places.neighbors geo.places.parent geo.places.siblings geo.placetypes gnip.activity local.search music.artist.id music.artist.popular music.artist.search music.artist.similar music.release.artist music.release.id music.release.popular music.release.search music.track.id music.track.popular music.track.search music.video.category music.video.id music.video.popular music.video.search music.video.similar mybloglog.community.find mybloglog.member mybloglog.member.contacts mybloglog.member.newwithcontacts mybloglog.member.newwithme mybloglog.members.find mybloglog.stats.find upcoming.category upcoming.country upcoming.events upcoming.events.bestinplace upcoming.groups upcoming.metro upcoming.state upcoming.user upcoming.venue
  37. 38. Here’s what Yahoo offers you in this format: search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connections social.contacts social.presence social.profile social.updates weather.forecast yahoo.identity yap.setsmallview search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connections social.contacts social.presence social.profile social.updates weather.forecast yahoo.identity yap.setsmallview search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connections social.contacts social.presence social.profile social.updates weather.forecast yahoo.identity yap.setsmallview
  38. 39. You want even more? http://www.flickr.com/photos/verylastexcitingmoment/3123597774/
  39. 40. Alright, how about this? atom csv feed html json microformats rss xml
  40. 41. Telegraph’s headlines anyone? select * from html where url='http://www.telegraph.co.uk/' and xpath='//h3/a'
  41. 42. Telegraph’s headlines anyone?
  42. 43. You can dynamically create YQL queries to collate several sources…
  43. 44. Then use cURL to pull them off the web with one single http request!
  44. 45. YQL caches and compresses the results for you. Pretty easy, isn’t it?
  45. 46. Here’s another fun part: Anyone can be part of the YQL interface.
  46. 47. All we need the data provider to do is to create a schema that explains their data structure. http://developer.yahoo.com/yql/guide/yql-opentables-chapter.html
  47. 48. And people do… http://github.com/spullara/yql-tables/tree/master amazon bitly delicious dopplr etsy friendfeed github Greader guardian imdb iplocation lastfm nestoria netflix nmm nyt opensocial search shopping social twitter update.groovy weather wesabe whitepages yahoo yelp zillow
  48. 49. For example the national maritime museum: select * from nmm.archive.search where searchterm=‘horatio nelson'
  49. 50. select * from nmm.archive.search where searchterm=‘horatio nelson'
  50. 51. That’s pretty cool – allowing anyone to be part of this interface.
  51. 52. The only shame is that you can’t do all the things in YQL that you can do in Pipes – for example string manipulation.
  52. 53. We wondered how to make this possible. One thing we didn’t want to sacrifice is the simplicity of the language itself.
  53. 54. So instead of inventing an own language, we decided to piggy-back on one you already know.
  54. 55. YQL execute allows you to embed JavaScript in the open table schema that runs on the YQL server and converts the data for you. http://developer.yahoo.com/yql/guide/yql-execute-chapter.html
  55. 56. For example you can augment an existing service to do something different. The following example adds a rank to search results. http://www.yqlblog.net/samples/searchrank.xml
  56. 59. Stored as XML this can be used in a YQL query: use 'http://yqlblog.net/samples/searchrank.xml' as searchrank; select * from searchrank where query='pizza' and dispurl like '%pizzahut% '
  57. 61. Anyways, the *easiest* way to start with YQL is to use the console. http://developer.yahoo.com/yql/console/
  58. 71. Of course, you can also spend half the hack day reading API docs 
  59. 72. Check out some code examples on. http://isithackday.com/hacks/ohd-london
  60. 73. THANKS! Chris Heilmann http://twitter.com/codepo8

×