Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!

4,970 views

Published on

From Jonathan Trevor's presentation "Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!" at the 2009 Cloud Computing Expo in Santa Clara, CA, USA. Here's the talk description on the Expo's site: http://cloudcomputingexpo.com/event/session/505

Published in: Technology
  • Hebat sekali cara penerangan anda. Saya berminat untuk embed ke dalam website saya, iaitu www.ringtonesz.org/ dan www.lyrics-search.org/

    Ahmad Said.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!

  1. 1. select * from internet<br />Yahoo Query Language:<br />Jonathan Trevor<br />
  2. 2. Application ?<br />
  3. 3. +<br />data visualization<br />data manipulation<br />Application =<br />
  4. 4. data<br />
  5. 5. data<br />
  6. 6.
  7. 7. var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…<br />?<br />…<br />if (flickcurl_prepare(fc, &quot;flickr.photos.geo.correctLocation&quot;, parameters, count)) { … }<br />…<br />?<br />?<br />http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2<br />?<br />http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c<br />
  8. 8. var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…<br />?<br />…<br />if (flickcurl_prepare(fc, &quot;flickr.photos.geo.correctLocation&quot;, parameters, count)) { … }<br />…<br />?<br />?<br />http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2<br />?<br />http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c<br />
  9. 9. var map = new YMap(document.getElementById(&apos;map&apos;));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…<br />…<br />if (flickcurl_prepare(fc, &quot;flickr.photos.geo.correctLocation&quot;, parameters, count)) { … }<br />…<br />http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2<br />http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c<br />Bindings<br />YQL<br />YQL stmt<br />select<br />desc<br />show<br />
  10. 10. Why YQL?<br />Thousands of Web Services and sources that provide valuable data<br />Require Developers to read documentation and form URLs/queries.<br />Data is isolated<br />Needs filtering, combining, tweaking, shaping even after it gets to the developer.<br />
  11. 11. YQL<br />Cloud web service and SQL-Like Language <br />Familiar to developers<br />Synonymous with Data access<br />Expressive enough to get the right data.<br />Self describing - show, desc table<br />Allows you to query, filter, join and update data across any structured data on the web / web services<br />And Yahoo’s sherpa cloud storage<br />Real time engine<br />
  12. 12. YQL Statement Basics<br />show: lists the supported tables<br />desc: describes the structure of a table<br />select: fetches data<br />insert/update/delete: modify data<br />use: use an Open Data Table<br />set: define key-values across Open Data Tables<br />
  13. 13. Demo: developer.yahoo.com/yql/<br />
  14. 14. Get any data:<br />Open Data Tables<br />
  15. 15. Open Data Tables<br />Bindings<br />YQL<br />
  16. 16. Open Data Tables<br />Map YQL to Endpoint<br />YQL<br />
  17. 17. USE Verb and environment files<br />use ‘http://url-to-def.xml’ as tableName;<br />select * from tableNamewhere key=‘value’...<br />Imports the Open Data Table definition <br />Makes it available as tableName<br />Acts just like other YQL Tables<br />Open tables can be collected together into environment files<br />
  18. 18. Open Developer Community<br />Leverage the work of others<br />github repository<br />http://github.com/yql/yql-tables/tree/master<br />datatables.orgmirrors the master and builds a single environment with all tables (faster serving)<br />Console with community tables<br />
  19. 19. Insert / Update / Delete<br />With INSERT/ UPDATE/ DELETESQL verbs you can modify data on the internet or cloud<br />create tweets, add blog entries, update DBs etc<br />store tables or pieces of code in sherpa<br />Some tables with I/U/D<br />Twitter Status, Bit.ly shortener<br />Yahoo profile status, application update stream<br />Amazon SimpleDB<br />
  20. 20. INSERT INTO Cloud<br />
  21. 21. yql.storage / yql.storage.admin tables<br />Allow developers to store and work with data in YQL itself<br /> share tables, environments and JavaScript with anyone<br /> hosted by YQL <br />Layered on Yahoo! Sherpa DHT cloud storage platform<br /> globally available, quick everywhere<br />Storage mapped to INSERT/UPDATE/DELETE<br />
  22. 22. yql.storage / yql.storage.admin tables<br />insert into yql.storage.admin(url) values (&quot;http://dom.com/table.xml&quot;)<br />Token based capabilities and access control<br />Using a stored piece of data<br />&lt;inserted&gt;    &lt;execute&gt;store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193&lt;/execute&gt;    &lt;select&gt;store://08fd2c74-dxxx-48c4-9ee1-3bf7e5a1ec92&lt;/select&gt;    &lt;update&gt;store://3cc85a99-6xxx-4600-ade1-f7f83ecc4b83&lt;/update&gt; &lt;/inserted&gt; <br />use “store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193” as mytable;select * from mytable where …<br />
  23. 23. Building your own open data tables<br />
  24. 24. Open Data Tables<br />Brings the power of the table model to new APIs<br />Schema defines mapping between YQL and Endpoint<br />Keys can either be query, path or matrix parameters<br />Keys can be marked as required<br />YQL Compiler validates existence of required keys <br />
  25. 25. YQL<br />Open Data Table example<br />
  26. 26. Open Data Table example<br />
  27. 27. Open Data Table example<br />
  28. 28. &lt;execute&gt; element<br />Extends Open Data Tables with server-side code<br />Javascript server-side scripting<br />No DOM, events etc, pure JS scripting<br />E4X support - XML literals, filtering ...<br />YQL no longer performs a GET request to the table binding URL<br />YQL provides additional useful global objects<br />request, response, y.rest, y.include, y.query...<br />Execute tag must return the data that will be the output of the YQL select<br />Move business logic of your application to the cloud<br />
  29. 29. YQL &lt;execute&gt;<br />access APIs that require authentication<br />Netflix OAuth, FlickrAuth, Google AuthSub<br />join data across services<br />grab New York Times article tags and generate find associated flickr photos<br />combine multiple searches into a single result (twitter, web, news and images)<br />augment data:<br />city/state to APIs that just return zip code<br />create APIs from web page<br />celebrity birthdays scraped from imdb, caltrain<br />data transformation<br />convert the result from xml to Google&apos;s kml format<br />
  30. 30. hello world<br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt;<br />&lt;meta&gt;<br /> &lt;sampleQuery&gt;select * from {table} where a=&apos;cat&apos; and b=&apos;dog&apos;;&lt;/sampleQuery&gt;<br />&lt;/meta&gt;<br />&lt;bindings&gt;<br />&lt;select itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br />&lt;urls&gt;&lt;url&gt;http://fake.url/{a}&lt;/url&gt;&lt;/urls&gt;<br />&lt;inputs&gt;<br />&lt;key id=&apos;a&apos; type=&apos;xs:string&apos; paramType=&apos;path&apos; required=&quot;true&quot; /&gt;<br />&lt;key id=&apos;b&apos; type=&apos;xs:string&apos; paramType=&apos;variable&apos; required=&quot;true&quot; /&gt;<br />&lt;/inputs&gt;<br />&lt;execute&gt;&lt;![CDATA[<br />// Your javascript goes here. We will run it on our servers<br />response.object = &lt;item&gt;<br />&lt;url&gt;{request.url}&lt;/url&gt;<br />&lt;a&gt;{a}&lt;/a&gt;<br /> &lt;b&gt;{b}&lt;/b&gt;<br /> &lt;/item&gt;;<br /> ]]&gt;&lt;/execute&gt;<br /> &lt;/select&gt;<br /> &lt;/bindings&gt;<br />&lt;/table&gt;<br />
  31. 31. Scraping a page<br />&lt;select produces=&quot;XML&quot; itemPath=&quot;deals.deal&quot;&gt;<br /> &lt;urls&gt;&lt;url&gt;http://deals.yahoo.com&lt;/url&gt;&lt;/urls&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />vardealhtml = y.query(&quot;select * from html where url=&apos;http://deals.yahoo.com&apos; and xpath=&apos;//div[contains(@class, &quot;product-display&quot;)]&apos;&quot;).results;<br />var deals = &lt;deals/&gt;;<br />function compress(object) {<br /> if (!object) return &quot;&quot;;<br /> return object.toString().replace(/s+/, &quot; &quot;)<br />}<br />for each (var div in dealhtml.div) {<br />var deal = &lt;deal/&gt;;<br />deal.name += &lt;name&gt;{compress(div.h3.a.text())}&lt;/name&gt;;<br />deal.link += &lt;link&gt;{div.div[0].a.@href}&lt;/link&gt;;<br />…<br />deal.description += &lt;description&gt;{compress(div.p[0].*)}&lt;/description&gt;;<br />deal.logo += &lt;logo&gt;{compress(div.div[1].*)}&lt;/logo&gt;;<br />deals.deal += deal;<br />}<br />response.object = deals;<br /> ]]&gt;&lt;/execute&gt;<br /> &lt;/select&gt;<br />
  32. 32. Inner join (unified web+image search)<br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt;<br /> &lt;bindings&gt;<br /> &lt;select itemPath=&quot;results.result&quot; produces=&quot;XML&quot;&gt;<br /> &lt;inputs&gt;<br /> &lt;key id=&quot;query&quot; type=&quot;xs:string&quot; paramType=&quot;query&quot; required=&quot;true&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />varqs = query;<br />var search = y.query(&apos;select * from search.web(50) where query=@query&apos;, {query: qs}).results;<br />var images = [];<br /> default xml namespace=&apos;http://www.inktomi.com/&apos;;<br /> for each (var result in search.result) {<br />images.push(y.query(&apos;select * from search.images(1) where query=@query and url=@url&apos;, {url:result.url, query:qs}));<br /> }<br />vari = 0;<br /> for each (var result in search.result) {<br />var image = images[i++].results.result;<br /> if (image) { result.image = &lt;image&gt;{image}&lt;/image&gt;; }<br /> }<br />response.object = search;<br /> ]]&gt;&lt;/execute&gt;<br /> &lt;/select&gt;<br /> &lt;/bindings&gt;<br />&lt;/table&gt;<br />
  33. 33. select *from twitter.statuswhere id=&quot;2108869549&quot;andusername=&quot;xxx&quot;and password=&quot;yyy”<br />&lt;select itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br /> &lt;urls&gt;&lt;url&gt;http://twitter.com/statuses/show/{id}.xml&lt;/url&gt;&lt;/urls&gt;<br /> &lt;inputs&gt;<br /> &lt;key id=&quot;username&quot; type=&quot;xs:string&quot; required=&quot;false&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;password&quot; type=&quot;xs:string&quot; required=&quot;false&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;id&quot; type=&quot;xs:integer&quot; required=&quot;true&quot; paramType=&quot;path&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />varr = null;<br /> if (username && password) {<br /> y.include(&quot;http://yqlblog.net/samples/base64.js&quot;);<br />varauthheader = &quot;Basic &quot;+Base64.encode(username+&quot;:&quot;+password);<br />r = request.header(&quot;Authorization&quot;,authheader).get().response;<br /> } else {<br />r = request.get().response;<br /> }<br />response.object = r;<br /> ]]&gt;&lt;/execute&gt;<br />&lt;/select&gt;<br />
  34. 34. insert into twitter.status(status,username,password)values (&quot;new tweet from YQL&quot;,”xxx”,”yyy”);<br />&lt;insert itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br /> &lt;urls&gt;<br /> &lt;url&gt;http://twitter.com/statuses/update.xml&lt;/url&gt;<br /> &lt;/urls&gt;<br /> &lt;inputs&gt;<br /> &lt;value id=&quot;username&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;value id=&quot;password&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;value id=&quot;status&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br /> y.include(&quot;http://yqlblog.net/samples/base64.js&quot;);<br />varauthheader = &quot;Basic &quot;+Base64.encode(username+&quot;:&quot;+password);<br />var content = &quot;status=&quot;+status;<br />response.object = request.header(&quot;Authorization&quot;,authheader).post(content).response;<br /> ]]&gt;&lt;/execute&gt;<br />&lt;/insert&gt;<br />
  35. 35. delete from twitter.statuswhere id=&quot;2108869549&quot;and username=&quot;xxx&quot;and password=&quot;yyy”<br />&lt;delete itemPath=&quot;&quot; produces=&quot;XML&quot;&gt;<br /> &lt;urls&gt;<br /> &lt;url&gt;http://twitter.com/statuses/destroy/{id}.xml&lt;/url&gt;<br /> &lt;/urls&gt;<br /> &lt;inputs&gt;<br /> &lt;key id=&quot;username&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;password&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;variable&quot;/&gt;<br /> &lt;key id=&quot;id&quot; type=&quot;xs:string&quot; required=&quot;true&quot; paramType=&quot;path&quot;/&gt;<br /> &lt;/inputs&gt;<br /> &lt;execute&gt;&lt;![CDATA[<br />y.include(&quot;http://yqlblog.net/samples/base64.js&quot;);<br />varauthheader = &quot;Basic &quot;+Base64.encode(username+&quot;:&quot;+password);<br />response.object = request.header(&quot;Authorization&quot;,authheader).del().response;<br /> ]]&gt;&lt;/execute&gt;<br />&lt;/delete&gt;<br />
  36. 36. Conclusion<br />More easily build applications<br />fewer lines of code<br />data processing done away from app<br />consistent and familiar syntax for all data access<br />iterative environment for developing the query<br />
  37. 37. Conclusion<br />Build applications that run faster<br /><ul><li>service in the cloud does the work
  38. 38. store tables, environments, js, text files…
  39. 39. conversion, filtering, parallel requests...</li></ul>With smaller network footprint<br />fewer (client) calls<br />smaller data amounts<br />closer to the data, fatter pipes<br />
  40. 40. data<br />YQL<br />
  41. 41. YQL: select * from internet<br />Jonathan Trevor<br />Questions?<br />http://developer.yahoo.com/yql<br />

×