Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    4 Favorites

    Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo! - Presentation Transcript

    1. select * from internet
      Yahoo Query Language:
      Jonathan Trevor
    2. Application ?
    3. +
      data visualization
      data manipulation
      Application =
    4. data
    5. data
    6. var map = new YMap(document.getElementById('map'));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
      ?

      if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }

      ?
      ?
      http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
      ?
      http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
    7. var map = new YMap(document.getElementById('map'));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
      ?

      if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }

      ?
      ?
      http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
      ?
      http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
    8. var map = new YMap(document.getElementById('map'));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…

      if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }

      http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
      http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
      Bindings
      YQL
      YQL stmt
      select
      desc
      show
    9. Why YQL?
      Thousands of Web Services and sources that provide valuable data
      Require Developers to read documentation and form URLs/queries.
      Data is isolated
      Needs filtering, combining, tweaking, shaping even after it gets to the developer.
    10. YQL
      Cloud web service and SQL-Like Language
      Familiar to developers
      Synonymous with Data access
      Expressive enough to get the right data.
      Self describing - show, desc table
      Allows you to query, filter, join and update data across any structured data on the web / web services
      And Yahoo’s sherpa cloud storage
      Real time engine
    11. YQL Statement Basics
      show: lists the supported tables
      desc: describes the structure of a table
      select: fetches data
      insert/update/delete: modify data
      use: use an Open Data Table
      set: define key-values across Open Data Tables
    12. Demo: developer.yahoo.com/yql/
    13. Get any data:
      Open Data Tables
    14. Open Data Tables
      Bindings
      YQL
    15. Open Data Tables
      Map YQL to Endpoint
      YQL
    16. USE Verb and environment files
      use ‘http://url-to-def.xml’ as tableName;
      select * from tableNamewhere key=‘value’...
      Imports the Open Data Table definition
      Makes it available as tableName
      Acts just like other YQL Tables
      Open tables can be collected together into environment files
    17. Open Developer Community
      Leverage the work of others
      github repository
      http://github.com/yql/yql-tables/tree/master
      datatables.orgmirrors the master and builds a single environment with all tables (faster serving)
      Console with community tables
    18. Insert / Update / Delete
      With INSERT/ UPDATE/ DELETESQL verbs you can modify data on the internet or cloud
      create tweets, add blog entries, update DBs etc
      store tables or pieces of code in sherpa
      Some tables with I/U/D
      Twitter Status, Bit.ly shortener
      Yahoo profile status, application update stream
      Amazon SimpleDB
    19. INSERT INTO Cloud
    20. yql.storage / yql.storage.admin tables
      Allow developers to store and work with data in YQL itself
      share tables, environments and JavaScript with anyone
      hosted by YQL
      Layered on Yahoo! Sherpa DHT cloud storage platform
      globally available, quick everywhere
      Storage mapped to INSERT/UPDATE/DELETE
    21. yql.storage / yql.storage.admin tables
      insert into yql.storage.admin(url) values ("http://dom.com/table.xml")
      Token based capabilities and access control
      Using a stored piece of data
      <inserted>    <execute>store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193</execute>    <select>store://08fd2c74-dxxx-48c4-9ee1-3bf7e5a1ec92</select>    <update>store://3cc85a99-6xxx-4600-ade1-f7f83ecc4b83</update> </inserted>
      use “store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193” as mytable;select * from mytable where …
    22. Building your own open data tables
    23. Open Data Tables
      Brings the power of the table model to new APIs
      Schema defines mapping between YQL and Endpoint
      Keys can either be query, path or matrix parameters
      Keys can be marked as required
      YQL Compiler validates existence of required keys
    24. YQL
      Open Data Table example
    25. Open Data Table example
    26. Open Data Table example
    27. <execute> element
      Extends Open Data Tables with server-side code
      Javascript server-side scripting
      No DOM, events etc, pure JS scripting
      E4X support - XML literals, filtering ...
      YQL no longer performs a GET request to the table binding URL
      YQL provides additional useful global objects
      request, response, y.rest, y.include, y.query...
      Execute tag must return the data that will be the output of the YQL select
      Move business logic of your application to the cloud
    28. YQL <execute>
      access APIs that require authentication
      Netflix OAuth, FlickrAuth, Google AuthSub
      join data across services
      grab New York Times article tags and generate find associated flickr photos
      combine multiple searches into a single result (twitter, web, news and images)
      augment data:
      city/state to APIs that just return zip code
      create APIs from web page
      celebrity birthdays scraped from imdb, caltrain
      data transformation
      convert the result from xml to Google's kml format
    29. hello world
      <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
      <meta>
      <sampleQuery>select * from {table} where a='cat' and b='dog';</sampleQuery>
      </meta>
      <bindings>
      <select itemPath="" produces="XML">
      <urls><url>http://fake.url/{a}</url></urls>
      <inputs>
      <key id='a' type='xs:string' paramType='path' required="true" />
      <key id='b' type='xs:string' paramType='variable' required="true" />
      </inputs>
      <execute><![CDATA[
      // Your javascript goes here. We will run it on our servers
      response.object = <item>
      <url>{request.url}</url>
      <a>{a}</a>
      <b>{b}</b>
      </item>;
      ]]></execute>
      </select>
      </bindings>
      </table>
    30. Scraping a page
      <select produces="XML" itemPath="deals.deal">
      <urls><url>http://deals.yahoo.com</url></urls>
      <execute><![CDATA[
      vardealhtml = y.query("select * from html where url='http://deals.yahoo.com' and xpath='//div[contains(@class, "product-display")]'").results;
      var deals = <deals/>;
      function compress(object) {
      if (!object) return "";
      return object.toString().replace(/s+/, " ")
      }
      for each (var div in dealhtml.div) {
      var deal = <deal/>;
      deal.name += <name>{compress(div.h3.a.text())}</name>;
      deal.link += <link>{div.div[0].a.@href}</link>;

      deal.description += <description>{compress(div.p[0].*)}</description>;
      deal.logo += <logo>{compress(div.div[1].*)}</logo>;
      deals.deal += deal;
      }
      response.object = deals;
      ]]></execute>
      </select>
    31. Inner join (unified web+image search)
      <table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
      <bindings>
      <select itemPath="results.result" produces="XML">
      <inputs>
      <key id="query" type="xs:string" paramType="query" required="true"/>
      </inputs>
      <execute><![CDATA[
      varqs = query;
      var search = y.query('select * from search.web(50) where query=@query', {query: qs}).results;
      var images = [];
      default xml namespace='http://www.inktomi.com/';
      for each (var result in search.result) {
      images.push(y.query('select * from search.images(1) where query=@query and url=@url', {url:result.url, query:qs}));
      }
      vari = 0;
      for each (var result in search.result) {
      var image = images[i++].results.result;
      if (image) { result.image = <image>{image}</image>; }
      }
      response.object = search;
      ]]></execute>
      </select>
      </bindings>
      </table>
    32. select *from twitter.statuswhere id="2108869549"andusername="xxx"and password="yyy”
      <select itemPath="" produces="XML">
      <urls><url>http://twitter.com/statuses/show/{id}.xml</url></urls>
      <inputs>
      <key id="username" type="xs:string" required="false" paramType="variable"/>
      <key id="password" type="xs:string" required="false" paramType="variable"/>
      <key id="id" type="xs:integer" required="true" paramType="path"/>
      </inputs>
      <execute><![CDATA[
      varr = null;
      if (username && password) {
      y.include("http://yqlblog.net/samples/base64.js");
      varauthheader = "Basic "+Base64.encode(username+":"+password);
      r = request.header("Authorization",authheader).get().response;
      } else {
      r = request.get().response;
      }
      response.object = r;
      ]]></execute>
      </select>
    33. insert into twitter.status(status,username,password)values ("new tweet from YQL",”xxx”,”yyy”);
      <insert itemPath="" produces="XML">
      <urls>
      <url>http://twitter.com/statuses/update.xml</url>
      </urls>
      <inputs>
      <value id="username" type="xs:string" required="true" paramType="variable"/>
      <value id="password" type="xs:string" required="true" paramType="variable"/>
      <value id="status" type="xs:string" required="true" paramType="variable"/>
      </inputs>
      <execute><![CDATA[
      y.include("http://yqlblog.net/samples/base64.js");
      varauthheader = "Basic "+Base64.encode(username+":"+password);
      var content = "status="+status;
      response.object = request.header("Authorization",authheader).post(content).response;
      ]]></execute>
      </insert>
    34. delete from twitter.statuswhere id="2108869549"and username="xxx"and password="yyy”
      <delete itemPath="" produces="XML">
      <urls>
      <url>http://twitter.com/statuses/destroy/{id}.xml</url>
      </urls>
      <inputs>
      <key id="username" type="xs:string" required="true" paramType="variable"/>
      <key id="password" type="xs:string" required="true" paramType="variable"/>
      <key id="id" type="xs:string" required="true" paramType="path"/>
      </inputs>
      <execute><![CDATA[
      y.include("http://yqlblog.net/samples/base64.js");
      varauthheader = "Basic "+Base64.encode(username+":"+password);
      response.object = request.header("Authorization",authheader).del().response;
      ]]></execute>
      </delete>
    35. Conclusion
      More easily build applications
      fewer lines of code
      data processing done away from app
      consistent and familiar syntax for all data access
      iterative environment for developing the query
    36. Conclusion
      Build applications that run faster
      • service in the cloud does the work
      • store tables, environments, js, text files…
      • conversion, filtering, parallel requests...
      With smaller network footprint
      fewer (client) calls
      smaller data amounts
      closer to the data, fatter pipes
    37. data
      YQL
    38. YQL: select * from internet
      Jonathan Trevor
      Questions?
      http://developer.yahoo.com/yql

    + Yahoo Developer NetworkYahoo Developer Network, 2 weeks ago

    custom

    369 views, 4 favs, 1 embeds more stats

    From Jonathan Trevor's presentation "Yahoo! Query L more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 369
      • 367 on SlideShare
      • 2 from embeds
    • Comments 0
    • Favorites 4
    • Downloads 0
    Most viewed embeds
    • 2 views on http://webtweets.blogspot.com

    more

    All embeds
    • 2 views on http://webtweets.blogspot.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories