YQL talk at OHD Jakarta


Published on

Shreeni from Y! Sing does a deep dive on YQL

Published in: Technology, Design
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

YQL talk at OHD Jakarta

  1. 1. Standardizing web querying - Using YQL for accessing and exposing APIs<br />
  2. 2. Agenda<br />Introduction to YQL<br />How to get started<br />What’s supported through YQL<br />Theme of the day – mim.yahoo.com<br />Lots of examples using mim API <br />Mashup using YQL<br />Introduction to more YQL goodies<br />Server side javascript execution<br />YQL hosted storage<br />Concluding remarks<br />..lets keep it interactive<br />
  3. 3. YQL – Yahoo Query Language<br />Enables you to access Internet data with SQL-like commands<br />YQL hides the complexity of Web service APIs by presenting data as simple tables, rows, and columns.<br />YQL includes pre-defined tables for popular Yahoo! Web services such as Flickr, Search, Geo andMim.<br />YQL can access services on the Internet that output data in the following formats: HTML, XML, JSON, RSS, Atom, and microformat.<br />Create fast mashups reducing latency, code<br />
  4. 4. How to get started<br />Explore the APIs - http://developer.yahoo.com/everything.html<br />Get a developer key - https://developer.apps.yahoo.com/wsregapp/<br />Read the documentation - http://developer.yahoo.com/yql/guide/<br />Visit the YQL Console - http://developer.yahoo.com/yql/console/<br />The YQL Console enables you to run YQL statements interactively from your browser. <br />Hack away..<br />
  5. 5. Y! APIs supported by YQL<br />Show tables<br />Flickr<br />Y! geo<br />Y! local search<br />Y! maps<br />Y! mim<br />Y! music<br />Y! MyBlogLog<br />Y! search<br />Y! social<br />Y! upcoming<br />
  6. 6. External APIs supported by YQL<br />Through datatables.org<br />Show tables<br />Amazon ECS<br />AWS<br />Bit.ly<br />Comicvine<br />Craigslist<br />Del.icio.us<br />Dopplr<br />Facebook<br />Friendfeed<br />Github<br />Google (News, Search, Translate)<br />Guardian<br />IMDB<br />LastFm<br />MediaWikiAPI<br />Microsoft (Bing)<br />Twitter<br />Zillow<br />
  7. 7. Theme of the day – mim.yahoo.com<br />Mim has an YQL based API for everything – reading, searching, updating, deleting and so on.<br />Profile information ofmimusers<br />Followers/following information<br />Query for users<br />Listing mim posts of a user<br />Popular posts<br />Search for posts<br />Inserting (posting to mim) through YQL<br /><ul><li>Reposting, commenting deleting</li></li></ul><li>Getting information about self (logged in)<br />select * from meme.info where owner_guid=me<br />&lt;query xmlns:yahoo=&quot;http://www.yahooapis.com/v1/base.rng&quot; yahoo:count=&quot;1” yahoo:lang=&quot;en-US” yahoo:uri=&quot;http://query.yahooapis.com/v1/yql?q=select+*+from+meme.info+where+owner_guid%3Dme&quot;&gt;<br />&lt;results&gt;<br /> &lt;meme&gt;<br /> &lt;guid&gt;NKADUBBZPUQQG6GF6AHTDTABSQ&lt;/guid&gt;<br /> &lt;name&gt;shreeni&lt;/name&gt;<br /> &lt;title&gt;Shreeni&amp;#39;sMim&lt;/title&gt;<br /> &lt;description/&gt;<br /> &lt;url&gt;http://meme.yahoo.com/shreeni/&lt;/url&gt;<br /> &lt;avatar_url&gt;http://a.com/b.jpeg&lt;/avatar_url&gt;<br /> &lt;language&gt;en&lt;/language&gt;<br /> &lt;followers&gt;18&lt;/followers&gt;<br /> &lt;/meme&gt;<br /> &lt;/results&gt;<br />&lt;/query&gt;<br />
  8. 8. In JSON<br />select * from meme.info where owner_guid=me<br />{&quot;query&quot;: {<br /> &quot;count&quot;: &quot;1&quot;, &quot;created&quot;: &quot;2009-11-19T10:33:11Z&quot;,<br /> &quot;lang&quot;: &quot;en-US&quot;, &quot;updated&quot;: &quot;2009-11-19T10:33:11Z&quot;,<br /> &quot;uri&quot;: &quot;http://query.yahooapis.com/v1/yql?q=select+*+from+meme.info+where+owner_guid%3Dme&quot;,<br /> &quot;results&quot;: {<br /> &quot;meme&quot;: {<br /> &quot;guid&quot;: &quot;NKADUBBZPUQQG6GF6AHTDTABSQ&quot;,<br /> &quot;name&quot;: &quot;shreeni&quot;, &quot;title&quot;: &quot;Shreeni'sMim&quot;,<br /> &quot;description&quot;: null,<br /> &quot;url&quot;: &quot;http://mim.yahoo.com/shreeni/&quot;,<br /> &quot;avatar_url&quot;: &quot;http://d.yimg.com/gg/shreeni/avatars/7a0d55f7d0aab9e326a7906a110f8027d46bd49b.jpeg&quot;,<br /> &quot;language&quot;: &quot;en&quot;, &quot;followers&quot;: &quot;18&quot;<br />}}}}<br />
  9. 9. How to run YQL queries<br /><ul><li>Convert to URL: http://query.yahooapis.com/v1/public/yql?q=<yql_query>&format=<json/xml>
  10. 10. Use the YQL Console - http://developer.yahoo.com/yql/console/
  11. 11. Use a PHP SDK provided by Y! - http://developer.yahoo.com/social/sdk/#php</li></li></ul><li>Selecting specific fields<br />select title, url, avatar_url, language from meme.info where name=&apos;shreeni&apos;<br />&lt;results&gt;<br /> &lt;meme&gt;<br /> &lt;title&gt;Shreeni&amp;#39;sMim&lt;/title&gt;<br /> &lt;url&gt;http://mim.yahoo.com/shreeni/&lt;/url&gt;<br /> &lt;avatar_url&gt;http://d.yimg.com/gg/shreeni/avatars/7a0d55f7d0aab9e326a7906a110f8027d46bd49b.jpeg&lt;/avatar_url&gt;<br /> &lt;language&gt;en&lt;/language&gt;<br /> &lt;/meme&gt;<br /> &lt;/results&gt;<br />
  12. 12. Selecting specific fields (also Followers API)<br />select name from meme.followers where owner_guid = “NKADUBBZPUQQG6GF6AHTDTABSQ”<br /> &lt;results&gt;<br /> &lt;meme&gt;<br /> &lt;name&gt;herry&lt;/name&gt;<br /> &lt;/meme&gt;<br /> &lt;meme&gt;<br /> &lt;name&gt;grahamhills&lt;/name&gt;<br /> &lt;/meme&gt;<br /> &lt;meme&gt;<br /> &lt;name&gt;fizzycitrus&lt;/name&gt;<br /> &lt;/meme&gt;<br /> &lt;/results&gt;<br />
  13. 13. Sub-select and Limits<br />select name from meme.followers where owner_guid in (select guid from meme.info where name=&apos;shreeni’) limit 2<br /> &lt;results&gt;<br /> &lt;meme&gt;<br /> &lt;name&gt;herry&lt;/name&gt;<br /> &lt;/meme&gt;<br /> &lt;meme&gt;<br /> &lt;name&gt;grahamhills&lt;/name&gt;<br /> &lt;/meme&gt;<br />&lt;/results&gt;<br />
  14. 14. Functions<br />select name from meme.followers where owner_guid in (select guid from meme.info where name=&apos;shreeni&apos;) | sort(field=&quot;name&quot;, descending=&quot;true”)<br />
  15. 15. Functions<br />Sort<br />Tail<br />Truncate<br />Reverse<br />unique<br />Sanitize<br />
  16. 16. Following API inmim<br />select name from meme.following where owner_guid in (select guid from meme.info where name=&apos;bangwinissimo&apos;)<br />
  17. 17. Searching for people<br />select guid, name, title, language from meme.people where query=&quot;Fajar”<br />select guid, name, title, language from meme.people where query=&quot;Fajar&quot; and language=”id&quot;<br />
  18. 18. Finding specific Posts of amimuser<br />select * from meme.posts where owner_guid in (select guid from meme.info where name=&apos;bangwinissimo&apos;)<br />select * from meme.posts where owner_guid in (select guid from meme.info where name=&apos;bangwinissimo&apos;) and type=&apos;text&apos;<br />
  19. 19. Popular Posts<br />select * from meme.popular<br />select * from meme.popular where locale=&quot;id&quot;<br />
  20. 20. Search inmim<br />SELECT * FROM meme.search WHERE query=&quot;jakarta”<br />
  21. 21. Inserting/Deletingmimposts<br />INSERT INTO meme.user.posts (type, content) VALUES (&quot;text&quot;, &quot;This is a text post from YQL!”)<br />DELETE FROM meme.user.posts where pubid = &quot;Nswwwmv&quot;<br />
  22. 22. Reposting, Commenting<br />INSERT INTO meme.user.posts (guid, pubid, comment) VALUES (&quot;S5R44PGJRBLKNEE5GYSRQPTXLQ&quot;, &quot;rGCOBCK&quot;, &quot;this is the repost comment”)<br />INSERT INTO meme.user.comments (guid, pubid, comment) VALUES (&quot;S5R44PGJRBLKNEE5GYSRQPTXLQ&quot;, &quot;rGCOBCK&quot;, &quot;meme rocks&quot;)<br />
  23. 23. YQL Cheat Sheet <br />http://bit.ly/yql-cheat-sheet<br />
  24. 24. Exposing own API through YQL<br />&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt; <br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt; <br /> &lt;meta&gt; <br /> &lt;sampleQuery&gt;select * from {table} where query=’q&apos;&lt;/sampleQuery&gt; <br /> &lt;/meta&gt; <br /> &lt;bindings&gt; <br /> &lt;select itemPath=&quot;results&quot; produces=&quot;JSON&quot; &gt; <br /> &lt;urls&gt; <br /> &lt;url env=&quot;all&quot;&gt;http://query-url&lt;/url&gt; <br /> &lt;/urls&gt; <br /> &lt;inputs&gt; <br /> &lt;key id=”q&quot; type=&quot;xs:string&quot; paramType=&quot;query&quot; required=&quot;true&quot; /&gt; <br /> &lt;/inputs&gt;<br /> &lt;/select&gt; <br /> &lt;/bindings&gt; <br />&lt;/table&gt; <br />
  25. 25. Now access it through YQL<br />USE &quot;http://definition-file.xml&quot; AS mytable;<br />SELECT * FROM mytable WHERE query=&quot;whatever”<br />
  26. 26. Mashup - introduction<br /><ul><li>Finding the Sunrise/Sunset time for Jakarta
  27. 27. Earthtoolsapi for sunrise-sunset information
  28. 28. XML based api
  29. 29. Needs latitude-longitude of the location
  30. 30. Yahoo Geo API to convert location to latitute-longitude
  31. 31. Would need 2 different API calls and effort from your side
  32. 32. Assuming Yahoo API was in JSON would have to deal with 2 formats</li></li></ul><li>Solution – Exposing 3rd party API in YQL<br />&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt; <br />&lt;table xmlns=&quot;http://query.yahooapis.com/v1/schema/table.xsd&quot;&gt; <br /> &lt;meta&gt; <br /> &lt;author&gt;Earthtools.org&lt;/author&gt; <br /> &lt;documentationURL&gt;http://www.earthtools.org/&lt;/documentationURL&gt;<br /> &lt;sampleQuery&gt;select * from {table} where lat=&apos;lat&apos; and lon=&apos;lon&apos; and day=&apos;day&apos; and month=&apos;month&apos; &lt;/sampleQuery&gt; <br /> &lt;/meta&gt; <br /> &lt;bindings&gt; <br /> &lt;select itemPath=&quot;sun&quot; produces=&quot;XML&quot; &gt; <br /> &lt;urls&gt; <br /> &lt;urlenv=&quot;all&quot;&gt;http://www.earthtools.org/sun/{lat}/{lon}/{day}/{month}/99/1&lt;/url&gt; <br /> &lt;/urls&gt; <br /> &lt;inputs&gt; <br /> &lt;key id=&quot;lat&quot; type=&quot;xs:string&quot; paramType=&quot;path&quot; required=&quot;true&quot; /&gt;&lt;!– 3 others --&gt;<br /> &lt;/inputs&gt; &lt;/select&gt;&lt;/bindings&gt; &lt;/table&gt;<br />
  33. 33. Querying for the data<br /># Get the Lat Long information for Jakarta<br />select * from geo.places where text=&quot;Jakarta&quot; limit 1;<br /># Now get the sunrise/sunset from earthtools using a <br /># YQL query<br />USE &quot;http://rahukalamcalculator.appspot.com/yql/earthtools-definition.xml&quot; AS earthtools;<br />SELECT morning, evening FROM earthtoolsWHERE lat=&quot;1.365580&quot; and lon=&quot;103.827713&quot; and day=&quot;21&quot; and month=&quot;11”<br />
  34. 34. Combine the queries<br /># One awesome call!<br />select * from mytable where (lat,lon) in (select centroid.latitude, centroid.longitude from geo.places where text=&quot;jakarta&quot;) and day=&quot;21&quot; and month=&quot;11”<br />
  35. 35. Mashup – solution highlights<br /><ul><li>Exposed a third party API through YQL
  36. 36. Expressed business logic in Query – readable, maintainable
  37. 37. Used server side for the actual join
  38. 38. Less latency
  39. 39. Use the format you like</li></li></ul><li>More goodies – YQL server side execution<br /><ul><li>Executing JavaScript allows you to use conditional logic and to format data in a granular manner.
  40. 40. Better data shaping and parsing
  41. 41. Better support for calling external Web service
  42. 42. Better support for adding, modifying, and deleting data using external Web services
  43. 43. Lots of examples and detailed documentation at http://developer.yahoo.com/yql/guide/yql-execute-examples.html</li></li></ul><li>Server Side execution - example<br />&lt;bindings&gt; <br /> &lt;select itemPath=&quot;&quot; produces=&quot;XML&quot;&gt; <br /> &lt;urls&gt; <br /> …<br /> &lt;/inputs&gt; <br /> &lt;execute&gt;&lt;![CDATA[ <br /> // Your javascript goes here. We will run it on our servers <br />response.object = &lt;item&gt; <br /> &lt;url&gt;{request.url}&lt;/url&gt; <br /> &lt;a&gt;{a}&lt;/a&gt; <br /> &lt;b&gt;{b}&lt;/b&gt; <br /> &lt;/item&gt;; <br /> ]]&gt;&lt;/execute&gt; <br /> &lt;/select&gt; <br /> &lt;/bindings&gt; <br />
  44. 44. More goodies – YQL Hosted Storage<br /><ul><li>Size Limit: You can have up to 1000 records, with each record being up to 100KB.
  45. 45. Retention Limit: Records not read, updated, or executed at least once every thirty days may be removed.
  46. 46. Record Format: Records must be in a text-based format. Examples include JavaScript code, XML files, Open Data Tables, or YQL environment files.
  47. 47. Authentication for New Records: All connections to yql.storage.admin must be authorized using two-legged OAuth. Alternatively, you can create new records using the YQL console. However, connections to yql.storage do not require authentication.</li></li></ul><li>Storing into YQL<br /># Just the value<br />insert into yql.storage.admin (value) values (&quot;example text content&quot;)<br /># Contents of a URL<br />insert into yql.storage.admin (name,url) values (&quot;newrecord&quot;,&quot;http://hostingdomain.com/mytable.xml&quot;)<br /># response looks like<br /> &lt;results&gt; <br /> &lt;inserted&gt; <br /> &lt;execute&gt;store://35ad2c72-e353-41c4-9d21-4a7e5a1ec92&lt;/execute&gt; <br /> &lt;select&gt;store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92&lt;/select&gt; <br /> &lt;update&gt;store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83&lt;/update&gt; <br /> &lt;/inserted&gt; <br /> &lt;/results&gt;<br />
  48. 48. Querying from hosted storage<br /># Uses select-id<br />select * from yql.storage where name=&quot;store://08fd2c74”<br /># Uses update-id<br />delete from yql.storage where name=&quot;store://3cc85a99”<br />update yql.storage set value=&quot;new value&quot; where name=&quot;store://3cc85a99”<br /># Use contents as env file (uses execute-id)<br />use &quot;store://35ad2c72&quot; as mytable; <br />select * from mytable;<br /># Use contents as JS for server side execute (uses execute-id)<br />y.include(&apos;store://35ad2c72&apos;); <br />response.object = &lt;success&gt;{success}&lt;/success&gt;;<br />
  49. 49. Concluding remarks<br /><ul><li>YQL is an extensible gateway to use APIs across and beyond Yahoo!
  50. 50. Allows standardization of APIs across the web
  51. 51. Familiar and concise language for expressing business and data logic
  52. 52. Powerful tool to build mashups
  53. 53. You can build upon existing APIs and expose third party APIs
  54. 54. Sound server side support, including javascript execution and storage to enable easy development</li></li></ul><li>Questions?<br />
  55. 55. Thank You!<br /><ul><li>http://developer.yahoo.com/yql
  56. 56. shreeni@yahoo-inc.com
  57. 57. http://tech.shreeni.info</li>