select * from internet<br />Yahoo Query Language:<br />Jonathan Trevor<br />
Application ?<br />
+<br />data visualization<br />data manipulation<br />Application =<br />
data<br />
data<br />
var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon );  map.a...
var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon );  map.a...
var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon );  map.a...
Why YQL?<br />Thousands of Web Services and sources that provide valuable data<br />Require Developers to read documentati...
YQL<br />Cloud web service and SQL-Like Language <br />Familiar to developers<br />Synonymous with Data access<br />Expres...
YQL Statement Basics<br />show: lists the supported tables<br />desc: describes the structure of a table<br />select: fetc...
Demo: developer.yahoo.com/yql/<br />
Get any data:<br />Open Data Tables<br />
Open Data Tables<br />Bindings<br />YQL<br />
Open Data Tables<br />Map YQL to Endpoint<br />YQL<br />
USE Verb and environment files<br />use ‘http://url-to-def.xml’ as tableName;<br />select * from tableNamewhere key=‘value...
Open Developer Community<br />Leverage the work of others<br />github repository<br />http://github.com/yql/yql-tables/tre...
Insert / Update / Delete<br />With INSERT/ UPDATE/ DELETESQL verbs you can modify data on the internet or cloud<br />creat...
INSERT INTO Cloud<br />
yql.storage / yql.storage.admin tables<br />Allow developers to store and work with data in YQL itself<br />	share tables,...
yql.storage / yql.storage.admin tables<br />insert into yql.storage.admin(url) values (&quot;http://dom.com/table.xml&quot...
Building your own open data tables<br />
Open Data Tables<br />Brings the power of the table model to new APIs<br />Schema defines mapping between YQL and Endpoint...
YQL<br />Open Data Table example<br />
Open Data Table example<br />
Open Data Table example<br />
&lt;execute&gt; element<br />Extends Open Data Tables with server-side code<br />Javascript server-side scripting<br />No ...
YQL &lt;execute&gt;<br />access APIs that require authentication<br />Netflix OAuth, FlickrAuth, Google AuthSub<br />join ...
hello world<br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt;<br />&lt;meta&gt;<br />   ...
Scraping a page<br />&lt;select produces=&quot;XML&quot; itemPath=&quot;deals.deal&quot;&gt;<br />      &lt;urls&gt;&lt;ur...
Inner join (unified web+image search)<br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt;<...
select *from twitter.statuswhere id=&quot;2108869549&quot;andusername=&quot;xxx&quot;and password=&quot;yyy”<br />&lt;sele...
insert into twitter.status(status,username,password)values (&quot;new tweet from YQL&quot;,”xxx”,”yyy”);<br />&lt;insert i...
delete from twitter.statuswhere id=&quot;2108869549&quot;and username=&quot;xxx&quot;and password=&quot;yyy”<br />&lt;dele...
Conclusion<br />More easily build applications<br />fewer lines of code<br />data processing done away from app<br />consi...
Conclusion<br />Build applications that run faster<br /><ul><li>service in the cloud does the work
store tables, environments, js, text files…
conversion, filtering, parallel requests...</li></ul>With smaller network footprint<br />fewer (client) calls<br />smaller...
Upcoming SlideShare
Loading in...5
×

Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!

4,254

Published on

From Jonathan Trevor's presentation "Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!" at the 2009 Cloud Computing Expo in Santa Clara, CA, USA. Here's the talk description on the Expo's site: http://cloudcomputingexpo.com/event/session/505

Published in: Technology
1 Comment
13 Likes
Statistics
Notes
  • Hebat sekali cara penerangan anda. Saya berminat untuk embed ke dalam website saya, iaitu www.ringtonesz.org/ dan www.lyrics-search.org/

    Ahmad Said.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
4,254
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
1
Likes
13
Embeds 0
No embeds

No notes for slide

Transcript of "Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!"

  1. 1. select * from internet<br />Yahoo Query Language:<br />Jonathan Trevor<br />
  2. 2. Application ?<br />
  3. 3. +<br />data visualization<br />data manipulation<br />Application =<br />
  4. 4. data<br />
  5. 5. data<br />
  6. 6.
  7. 7. var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…<br />?<br />…<br />if (flickcurl_prepare(fc, &quot;flickr.photos.geo.correctLocation&quot;, parameters, count)) { … }<br />…<br />?<br />?<br />http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2<br />?<br />http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c<br />
  8. 8. var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…<br />?<br />…<br />if (flickcurl_prepare(fc, &quot;flickr.photos.geo.correctLocation&quot;, parameters, count)) { … }<br />…<br />?<br />?<br />http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2<br />?<br />http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c<br />
  9. 9. var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…<br />…<br />if (flickcurl_prepare(fc, &quot;flickr.photos.geo.correctLocation&quot;, parameters, count)) { … }<br />…<br />http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2<br />http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c<br />Bindings<br />YQL<br />YQL stmt<br />select<br />desc<br />show<br />
  10. 10. Why YQL?<br />Thousands of Web Services and sources that provide valuable data<br />Require Developers to read documentation and form URLs/queries.<br />Data is isolated<br />Needs filtering, combining, tweaking, shaping even after it gets to the developer.<br />
  11. 11. YQL<br />Cloud web service and SQL-Like Language <br />Familiar to developers<br />Synonymous with Data access<br />Expressive enough to get the right data.<br />Self describing - show, desc table<br />Allows you to query, filter, join and update data across any structured data on the web / web services<br />And Yahoo’s sherpa cloud storage<br />Real time engine<br />
  12. 12. YQL Statement Basics<br />show: lists the supported tables<br />desc: describes the structure of a table<br />select: fetches data<br />insert/update/delete: modify data<br />use: use an Open Data Table<br />set: define key-values across Open Data Tables<br />
  13. 13. Demo: developer.yahoo.com/yql/<br />
  14. 14. Get any data:<br />Open Data Tables<br />
  15. 15. Open Data Tables<br />Bindings<br />YQL<br />
  16. 16. Open Data Tables<br />Map YQL to Endpoint<br />YQL<br />
  17. 17. USE Verb and environment files<br />use ‘http://url-to-def.xml’ as tableName;<br />select * from tableNamewhere key=‘value’...<br />Imports the Open Data Table definition <br />Makes it available as tableName<br />Acts just like other YQL Tables<br />Open tables can be collected together into environment files<br />
  18. 18. Open Developer Community<br />Leverage the work of others<br />github repository<br />http://github.com/yql/yql-tables/tree/master<br />datatables.orgmirrors the master and builds a single environment with all tables (faster serving)<br />Console with community tables<br />
  19. 19. Insert / Update / Delete<br />With INSERT/ UPDATE/ DELETESQL verbs you can modify data on the internet or cloud<br />create tweets, add blog entries, update DBs etc<br />store tables or pieces of code in sherpa<br />Some tables with I/U/D<br />Twitter Status, Bit.ly shortener<br />Yahoo profile status, application update stream<br />Amazon SimpleDB<br />
  20. 20. INSERT INTO Cloud<br />
  21. 21. yql.storage / yql.storage.admin tables<br />Allow developers to store and work with data in YQL itself<br /> share tables, environments and JavaScript with anyone<br /> hosted by YQL <br />Layered on Yahoo! Sherpa DHT cloud storage platform<br /> globally available, quick everywhere<br />Storage mapped to INSERT/UPDATE/DELETE<br />
  22. 22. yql.storage / yql.storage.admin tables<br />insert into yql.storage.admin(url) values (&quot;http://dom.com/table.xml&quot;)<br />Token based capabilities and access control<br />Using a stored piece of data<br />&lt;inserted&gt;    &lt;execute&gt;store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193&lt;/execute&gt;    &lt;select&gt;store://08fd2c74-dxxx-48c4-9ee1-3bf7e5a1ec92&lt;/select&gt;    &lt;update&gt;store://3cc85a99-6xxx-4600-ade1-f7f83ecc4b83&lt;/update&gt; &lt;/inserted&gt; <br />use “store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193” as mytable;select * from mytable where …<br />
  23. 23. Building your own open data tables<br />
  24. 24. Open Data Tables<br />Brings the power of the table model to new APIs<br />Schema defines mapping between YQL and Endpoint<br />Keys can either be query, path or matrix parameters<br />Keys can be marked as required<br />YQL Compiler validates existence of required keys <br />
  25. 25. YQL<br />Open Data Table example<br />
  26. 26. Open Data Table example<br />
  27. 27. Open Data Table example<br />
  28. 28. &lt;execute&gt; element<br />Extends Open Data Tables with server-side code<br />Javascript server-side scripting<br />No DOM, events etc, pure JS scripting<br />E4X support - XML literals, filtering ...<br />YQL no longer performs a GET request to the table binding URL<br />YQL provides additional useful global objects<br />request, response, y.rest, y.include, y.query...<br />Execute tag must return the data that will be the output of the YQL select<br />Move business logic of your application to the cloud<br />
  29. 29. YQL &lt;execute&gt;<br />access APIs that require authentication<br />Netflix OAuth, FlickrAuth, Google AuthSub<br />join data across services<br />grab New York Times article tags and generate find associated flickr photos<br />combine multiple searches into a single result (twitter, web, news and images)<br />augment data:<br />city/state to APIs that just return zip code<br />create APIs from web page<br />celebrity birthdays scraped from imdb, caltrain<br />data transformation<br />convert the result from xml to Google&apos;s kml format<br />
  30. 30. hello world<br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt;<br />&lt;meta&gt;<br /> &lt;sampleQuery&gt;select * from {table} where a=&apos;cat&apos; and b=&apos;dog&apos;;&lt;/sampleQuery&gt;<br />&lt;/meta&gt;<br />&lt;bindings&gt;<br />&lt;select itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br />&lt;urls&gt;&lt;url&gt;http://fake.url/{a}&lt;/url&gt;&lt;/urls&gt;<br />&lt;inputs&gt;<br />&lt;key id=&apos;a&apos; type=&apos;xs:string&apos; paramType=&apos;path&apos; required=&quot;true&quot; /&gt;<br />&lt;key id=&apos;b&apos; type=&apos;xs:string&apos; paramType=&apos;variable&apos; required=&quot;true&quot; /&gt;<br />&lt;/inputs&gt;<br />&lt;execute&gt;&lt;![CDATA[<br />// Your javascript goes here. We will run it on our servers<br />response.object = &lt;item&gt;<br />&lt;url&gt;{request.url}&lt;/url&gt;<br />&lt;a&gt;{a}&lt;/a&gt;<br /> &lt;b&gt;{b}&lt;/b&gt;<br /> &lt;/item&gt;;<br /> ]]&gt;&lt;/execute&gt;<br /> &lt;/select&gt;<br /> &lt;/bindings&gt;<br />&lt;/table&gt;<br />
  31. 31. Scraping a page<br />&lt;select produces=&quot;XML&quot; itemPath=&quot;deals.deal&quot;&gt;<br /> &lt;urls&gt;&lt;url&gt;http://deals.yahoo.com&lt;/url&gt;&lt;/urls&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />vardealhtml = y.query(&quot;select * from html where url=&apos;http://deals.yahoo.com&apos; and xpath=&apos;//div[contains(@class, &quot;product-display&quot;)]&apos;&quot;).results;<br />var deals = &lt;deals/&gt;;<br />function compress(object) {<br /> if (!object) return &quot;&quot;;<br /> return object.toString().replace(/s+/, &quot; &quot;)<br />}<br />for each (var div in dealhtml.div) {<br />var deal = &lt;deal/&gt;;<br />deal.name += &lt;name&gt;{compress(div.h3.a.text())}&lt;/name&gt;;<br />deal.link += &lt;link&gt;{div.div[0].a.@href}&lt;/link&gt;;<br />…<br />deal.description += &lt;description&gt;{compress(div.p[0].*)}&lt;/description&gt;;<br />deal.logo += &lt;logo&gt;{compress(div.div[1].*)}&lt;/logo&gt;;<br />deals.deal += deal;<br />}<br />response.object = deals;<br /> ]]&gt;&lt;/execute&gt;<br /> &lt;/select&gt;<br />
  32. 32. Inner join (unified web+image search)<br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt;<br /> &lt;bindings&gt;<br /> &lt;select itemPath=&quot;results.result&quot; produces=&quot;XML&quot;&gt;<br /> &lt;inputs&gt;<br /> &lt;key id=&quot;query&quot; type=&quot;xs:string&quot; paramType=&quot;query&quot; required=&quot;true&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />varqs = query;<br />var search = y.query(&apos;select * from search.web(50) where query=@query&apos;, {query: qs}).results;<br />var images = [];<br /> default xml namespace=&apos;http://www.inktomi.com/&apos;;<br /> for each (var result in search.result) {<br />images.push(y.query(&apos;select * from search.images(1) where query=@query and url=@url&apos;, {url:result.url, query:qs}));<br /> }<br />vari = 0;<br /> for each (var result in search.result) {<br />var image = images[i++].results.result;<br /> if (image) { result.image = &lt;image&gt;{image}&lt;/image&gt;; }<br /> }<br />response.object = search;<br /> ]]&gt;&lt;/execute&gt;<br /> &lt;/select&gt;<br /> &lt;/bindings&gt;<br />&lt;/table&gt;<br />
  33. 33. select *from twitter.statuswhere id=&quot;2108869549&quot;andusername=&quot;xxx&quot;and password=&quot;yyy”<br />&lt;select itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br /> &lt;urls&gt;&lt;url&gt;http://twitter.com/statuses/show/{id}.xml&lt;/url&gt;&lt;/urls&gt;<br /> &lt;inputs&gt;<br /> &lt;key id=&quot;username&quot; type=&quot;xs:string&quot; required=&quot;false&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;password&quot; type=&quot;xs:string&quot; required=&quot;false&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;id&quot; type=&quot;xs:integer&quot; required=&quot;true&quot; paramType=&quot;path&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />varr = null;<br /> if (username && password) {<br /> y.include(&quot;http://yqlblog.net/samples/base64.js&quot;);<br />varauthheader = &quot;Basic &quot;+Base64.encode(username+&quot;:&quot;+password);<br />r = request.header(&quot;Authorization&quot;,authheader).get().response;<br /> } else {<br />r = request.get().response;<br /> }<br />response.object = r;<br /> ]]&gt;&lt;/execute&gt;<br />&lt;/select&gt;<br />
  34. 34. insert into twitter.status(status,username,password)values (&quot;new tweet from YQL&quot;,”xxx”,”yyy”);<br />&lt;insert itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br /> &lt;urls&gt;<br /> &lt;url&gt;http://twitter.com/statuses/update.xml&lt;/url&gt;<br /> &lt;/urls&gt;<br /> &lt;inputs&gt;<br /> &lt;value id=&quot;username&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;value id=&quot;password&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;value id=&quot;status&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br /> y.include(&quot;http://yqlblog.net/samples/base64.js&quot;);<br />varauthheader = &quot;Basic &quot;+Base64.encode(username+&quot;:&quot;+password);<br />var content = &quot;status=&quot;+status;<br />response.object = request.header(&quot;Authorization&quot;,authheader).post(content).response;<br /> ]]&gt;&lt;/execute&gt;<br />&lt;/insert&gt;<br />
  35. 35. delete from twitter.statuswhere id=&quot;2108869549&quot;and username=&quot;xxx&quot;and password=&quot;yyy”<br />&lt;delete itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br /> &lt;urls&gt;<br /> &lt;url&gt;http://twitter.com/statuses/destroy/{id}.xml&lt;/url&gt;<br /> &lt;/urls&gt;<br /> &lt;inputs&gt;<br /> &lt;key id=&quot;username&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;password&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;id&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;path&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />y.include(&quot;http://yqlblog.net/samples/base64.js&quot;);<br />varauthheader = &quot;Basic &quot;+Base64.encode(username+&quot;:&quot;+password);<br />response.object = request.header(&quot;Authorization&quot;,authheader).del().response;<br /> ]]&gt;&lt;/execute&gt;<br />&lt;/delete&gt;<br />
  36. 36. Conclusion<br />More easily build applications<br />fewer lines of code<br />data processing done away from app<br />consistent and familiar syntax for all data access<br />iterative environment for developing the query<br />
  37. 37. Conclusion<br />Build applications that run faster<br /><ul><li>service in the cloud does the work
  38. 38. store tables, environments, js, text files…
  39. 39. conversion, filtering, parallel requests...</li></ul>With smaller network footprint<br />fewer (client) calls<br />smaller data amounts<br />closer to the data, fatter pipes<br />
  40. 40. data<br />YQL<br />
  41. 41. YQL: select * from internet<br />Jonathan Trevor<br />Questions?<br />http://developer.yahoo.com/yql<br />

×