YQL talk at OHD Jakarta

Standardizing web querying - Using YQL for accessing and exposing APIs

Agenda
Introduction to YQL
How to get started
What’s supported through YQL
Theme of the day – mim.yahoo.com
Lots of examples using mim API
Mashup using YQL
Introduction to more YQL goodies
Server side javascript execution
YQL hosted storage
Concluding remarks
..lets keep it interactive

YQL – Yahoo Query Language
Enables you to access Internet data with SQL-like commands
YQL hides the complexity of Web service APIs by presenting data as simple tables, rows, and columns.
YQL includes pre-defined tables for popular Yahoo! Web services such as Flickr, Search, Geo andMim.
YQL can access services on the Internet that output data in the following formats: HTML, XML, JSON, RSS, Atom, and microformat.
Create fast mashups reducing latency, code

How to get started
Explore the APIs - http://developer.yahoo.com/everything.html
Get a developer key - https://developer.apps.yahoo.com/wsregapp/
Read the documentation - http://developer.yahoo.com/yql/guide/
Visit the YQL Console - http://developer.yahoo.com/yql/console/
The YQL Console enables you to run YQL statements interactively from your browser.
Hack away..

Y! APIs supported by YQL
Show tables
Flickr
Y! geo
Y! local search
Y! maps
Y! mim
Y! music
Y! MyBlogLog
Y! search
Y! social
Y! upcoming

External APIs supported by YQL
Through datatables.org
Show tables
Amazon ECS
AWS
Bit.ly
Comicvine
Craigslist
Del.icio.us
Dopplr
Facebook
Friendfeed
Github
Google (News, Search, Translate)
Guardian
IMDB
LastFm
MediaWikiAPI
Microsoft (Bing)
Twitter
Zillow

Theme of the day – mim.yahoo.com
Mim has an YQL based API for everything – reading, searching, updating, deleting and so on.
Profile information ofmimusers
Followers/following information
Query for users
Listing mim posts of a user
Popular posts
Search for posts
Inserting (posting to mim) through YQL
Reposting, commenting deleting

Getting information about self (logged in)
select * from meme.info where owner_guid=me
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="1” yahoo:lang="en-US” yahoo:uri="http://query.yahooapis.com/v1/yql?q=select+*+from+meme.info+where+owner_guid%3Dme">
<results>
<meme>
<guid>NKADUBBZPUQQG6GF6AHTDTABSQ</guid>
<name>shreeni</name>
<title>Shreeni'sMim</title>
<description/>
<url>http://meme.yahoo.com/shreeni/</url>
<avatar_url>http://a.com/b.jpeg</avatar_url>
<language>en</language>
<followers>18</followers>
</meme>
</results>
</query>

In JSON
select * from meme.info where owner_guid=me
{"query": {
"count": "1", "created": "2009-11-19T10:33:11Z",
"lang": "en-US", "updated": "2009-11-19T10:33:11Z",
"uri": "http://query.yahooapis.com/v1/yql?q=select+*+from+meme.info+where+owner_guid%3Dme",
"results": {
"meme": {
"guid": "NKADUBBZPUQQG6GF6AHTDTABSQ",
"name": "shreeni", "title": "Shreeni'sMim",
"description": null,
"url": "http://mim.yahoo.com/shreeni/",
"avatar_url": "http://d.yimg.com/gg/shreeni/avatars/7a0d55f7d0aab9e326a7906a110f8027d46bd49b.jpeg",
"language": "en", "followers": "18"
}}}}

How to run YQL queries
Convert to URL: http://query.yahooapis.com/v1/public/yql?q=&format=
Use the YQL Console - http://developer.yahoo.com/yql/console/
Use a PHP SDK provided by Y! - http://developer.yahoo.com/social/sdk/#php

Selecting specific fields
select title, url, avatar_url, language from meme.info where name='shreeni'
<results>
<meme>
<title>Shreeni'sMim</title>
<url>http://mim.yahoo.com/shreeni/</url>
<avatar_url>http://d.yimg.com/gg/shreeni/avatars/7a0d55f7d0aab9e326a7906a110f8027d46bd49b.jpeg</avatar_url>
<language>en</language>
</meme>
</results>

Selecting specific fields (also Followers API)
select name from meme.followers where owner_guid = “NKADUBBZPUQQG6GF6AHTDTABSQ”
<results>
<meme>
<name>herry</name>
</meme>
<meme>
<name>grahamhills</name>
</meme>
<meme>
<name>fizzycitrus</name>
</meme>
</results>

Sub-select and Limits
select name from meme.followers where owner_guid in (select guid from meme.info where name='shreeni’) limit 2
<results>
<meme>
<name>herry</name>
</meme>
<meme>
<name>grahamhills</name>
</meme>
</results>

Functions
select name from meme.followers where owner_guid in (select guid from meme.info where name='shreeni') | sort(field="name", descending="true”)

Functions
Sort
Tail
Truncate
Reverse
unique
Sanitize

Following API inmim
select name from meme.following where owner_guid in (select guid from meme.info where name='bangwinissimo')

Searching for people
select guid, name, title, language from meme.people where query="Fajar”
select guid, name, title, language from meme.people where query="Fajar" and language=”id"

Finding specific Posts of amimuser
select * from meme.posts where owner_guid in (select guid from meme.info where name='bangwinissimo')
select * from meme.posts where owner_guid in (select guid from meme.info where name='bangwinissimo') and type='text'

Popular Posts
select * from meme.popular
select * from meme.popular where locale="id"

Search inmim
SELECT * FROM meme.search WHERE query="jakarta”

Inserting/Deletingmimposts
INSERT INTO meme.user.posts (type, content) VALUES ("text", "This is a text post from YQL!”)
DELETE FROM meme.user.posts where pubid = "Nswwwmv"

Reposting, Commenting
INSERT INTO meme.user.posts (guid, pubid, comment) VALUES ("S5R44PGJRBLKNEE5GYSRQPTXLQ", "rGCOBCK", "this is the repost comment”)
INSERT INTO meme.user.comments (guid, pubid, comment) VALUES ("S5R44PGJRBLKNEE5GYSRQPTXLQ", "rGCOBCK", "meme rocks")

YQL Cheat Sheet
http://bit.ly/yql-cheat-sheet

Exposing own API through YQL
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<meta>
<sampleQuery>select * from {table} where query=’q'</sampleQuery>
</meta>
<bindings>
<select itemPath="results" produces="JSON" >
<urls>
<url env="all">http://query-url</url>
</urls>
<inputs>
<key id=”q" type="xs:string" paramType="query" required="true" />
</inputs>
</select>
</bindings>
</table>

Now access it through YQL
USE "http://definition-file.xml" AS mytable;
SELECT * FROM mytable WHERE query="whatever”

Mashup - introduction
Finding the Sunrise/Sunset time for Jakarta
Earthtoolsapi for sunrise-sunset information
XML based api
Needs latitude-longitude of the location
Yahoo Geo API to convert location to latitute-longitude
Would need 2 different API calls and effort from your side
Assuming Yahoo API was in JSON would have to deal with 2 formats

Solution – Exposing 3rd party API in YQL
<?xml version="1.0" encoding="UTF-8"?>
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<meta>
<author>Earthtools.org</author>
<documentationURL>http://www.earthtools.org/</documentationURL>
<sampleQuery>select * from {table} where lat='lat' and lon='lon' and day='day' and month='month' </sampleQuery>
</meta>
<bindings>
<select itemPath="sun" produces="XML" >
<urls>
<urlenv="all">http://www.earthtools.org/sun/{lat}/{lon}/{day}/{month}/99/1</url>
</urls>
<inputs>
<key id="lat" type="xs:string" paramType="path" required="true" /><!– 3 others -->
</inputs> </select></bindings> </table>

Querying for the data
# Get the Lat Long information for Jakarta
select * from geo.places where text="Jakarta" limit 1;
# Now get the sunrise/sunset from earthtools using a
# YQL query
USE "http://rahukalamcalculator.appspot.com/yql/earthtools-definition.xml" AS earthtools;
SELECT morning, evening FROM earthtoolsWHERE lat="1.365580" and lon="103.827713" and day="21" and month="11”

Combine the queries
# One awesome call!
select * from mytable where (lat,lon) in (select centroid.latitude, centroid.longitude from geo.places where text="jakarta") and day="21" and month="11”

Mashup – solution highlights
Exposed a third party API through YQL
Expressed business logic in Query – readable, maintainable
Used server side for the actual join
Less latency
Use the format you like

More goodies – YQL server side execution
Executing JavaScript allows you to use conditional logic and to format data in a granular manner.
Better data shaping and parsing
Better support for calling external Web service
Better support for adding, modifying, and deleting data using external Web services
Lots of examples and detailed documentation at http://developer.yahoo.com/yql/guide/yql-execute-examples.html

Server Side execution - example
<bindings>
<select itemPath="" produces="XML">
<urls>
…
</inputs>
<execute><![CDATA[
// Your javascript goes here. We will run it on our servers
response.object = <item>
<url>{request.url}</url>
<a>{a}</a>
<b>{b}</b>
</item>;
]]></execute>
</select>
</bindings>

More goodies – YQL Hosted Storage
Size Limit: You can have up to 1000 records, with each record being up to 100KB.
Retention Limit: Records not read, updated, or executed at least once every thirty days may be removed.
Record Format: Records must be in a text-based format. Examples include JavaScript code, XML files, Open Data Tables, or YQL environment files.
Authentication for New Records: All connections to yql.storage.admin must be authorized using two-legged OAuth. Alternatively, you can create new records using the YQL console. However, connections to yql.storage do not require authentication.

Storing into YQL
# Just the value
insert into yql.storage.admin (value) values ("example text content")
# Contents of a URL
insert into yql.storage.admin (name,url) values ("newrecord","http://hostingdomain.com/mytable.xml")
# response looks like
<results>
<inserted>
<execute>store://35ad2c72-e353-41c4-9d21-4a7e5a1ec92</execute>
<select>store://08fd2c74-d393-48c4-9ee1-3bf7e5a1ec92</select>
<update>store://3cc85a99-6a89-4600-ade1-f7f83ecc4b83</update>
</inserted>
</results>

Querying from hosted storage
# Uses select-id
select * from yql.storage where name="store://08fd2c74”
# Uses update-id
delete from yql.storage where name="store://3cc85a99”
update yql.storage set value="new value" where name="store://3cc85a99”
# Use contents as env file (uses execute-id)
use "store://35ad2c72" as mytable;
select * from mytable;
# Use contents as JS for server side execute (uses execute-id)
y.include('store://35ad2c72');
response.object = <success>{success}</success>;

Concluding remarks
YQL is an extensible gateway to use APIs across and beyond Yahoo!
Allows standardization of APIs across the web
Familiar and concise language for expressing business and data logic
Powerful tool to build mashups
You can build upon existing APIs and expose third party APIs
Sound server side support, including javascript execution and storage to enable easy development

Questions?

Thank You!
http://developer.yahoo.com/yql
shreeni@yahoo-inc.com
http://tech.shreeni.info

YQL talk at OHD Jakarta

by Michael Smith on Dec 01, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 1

Statistics

YQL talk at OHD Jakarta — Presentation Transcript