Yahoo! Query Language - YQL: Select * from Internet - Including Live Demo!

select * from internet
Yahoo Query Language:
Jonathan Trevor

Application ?

+
data visualization
data manipulation
Application =

data

var map = new YMap(document.getElementById('map'));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
?
…
if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }
…
?
?
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
?
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c

var map = new YMap(document.getElementById('map'));…varcurrentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…
…
if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }
…
http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2
http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c
Bindings
YQL
YQL stmt
select
desc
show

Why YQL?
Thousands of Web Services and sources that provide valuable data
Require Developers to read documentation and form URLs/queries.
Data is isolated
Needs filtering, combining, tweaking, shaping even after it gets to the developer.

YQL
Cloud web service and SQL-Like Language
Familiar to developers
Synonymous with Data access
Expressive enough to get the right data.
Self describing - show, desc table
Allows you to query, filter, join and update data across any structured data on the web / web services
And Yahoo’s sherpa cloud storage
Real time engine

YQL Statement Basics
show: lists the supported tables
desc: describes the structure of a table
select: fetches data
insert/update/delete: modify data
use: use an Open Data Table
set: define key-values across Open Data Tables

Demo: developer.yahoo.com/yql/

Get any data:
Open Data Tables

Open Data Tables
Bindings
YQL

Open Data Tables
Map YQL to Endpoint
YQL

USE Verb and environment files
use ‘http://url-to-def.xml’ as tableName;
select * from tableNamewhere key=‘value’...
Imports the Open Data Table definition
Makes it available as tableName
Acts just like other YQL Tables
Open tables can be collected together into environment files

Open Developer Community
Leverage the work of others
github repository
http://github.com/yql/yql-tables/tree/master
datatables.orgmirrors the master and builds a single environment with all tables (faster serving)
Console with community tables

Insert / Update / Delete
With INSERT/ UPDATE/ DELETESQL verbs you can modify data on the internet or cloud
create tweets, add blog entries, update DBs etc
store tables or pieces of code in sherpa
Some tables with I/U/D
Twitter Status, Bit.ly shortener
Yahoo profile status, application update stream
Amazon SimpleDB

INSERT INTO Cloud

yql.storage / yql.storage.admin tables
Allow developers to store and work with data in YQL itself
share tables, environments and JavaScript with anyone
hosted by YQL
Layered on Yahoo! Sherpa DHT cloud storage platform
globally available, quick everywhere
Storage mapped to INSERT/UPDATE/DELETE

yql.storage / yql.storage.admin tables
insert into yql.storage.admin(url) values ("http://dom.com/table.xml")
Token based capabilities and access control
Using a stored piece of data
<inserted> <execute>store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193</execute> <select>store://08fd2c74-dxxx-48c4-9ee1-3bf7e5a1ec92</select> <update>store://3cc85a99-6xxx-4600-ade1-f7f83ecc4b83</update> </inserted>
use “store://4cd56a99-7xxx-4600-ade1-3bf7f5a11193” as mytable;select * from mytable where …

Building your own open data tables

Open Data Tables
Brings the power of the table model to new APIs
Schema defines mapping between YQL and Endpoint
Keys can either be query, path or matrix parameters
Keys can be marked as required
YQL Compiler validates existence of required keys

YQL
Open Data Table example

Open Data Table example

<execute> element
Extends Open Data Tables with server-side code
Javascript server-side scripting
No DOM, events etc, pure JS scripting
E4X support - XML literals, filtering ...
YQL no longer performs a GET request to the table binding URL
YQL provides additional useful global objects
request, response, y.rest, y.include, y.query...
Execute tag must return the data that will be the output of the YQL select
Move business logic of your application to the cloud

YQL <execute>
access APIs that require authentication
Netflix OAuth, FlickrAuth, Google AuthSub
join data across services
grab New York Times article tags and generate find associated flickr photos
combine multiple searches into a single result (twitter, web, news and images)
augment data:
city/state to APIs that just return zip code
create APIs from web page
celebrity birthdays scraped from imdb, caltrain
data transformation
convert the result from xml to Google's kml format

hello world
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<meta>
<sampleQuery>select * from {table} where a='cat' and b='dog';</sampleQuery>
</meta>
<bindings>
<select itemPath="" produces="XML">
<urls><url>http://fake.url/{a}</url></urls>
<inputs>
<key id='a' type='xs:string' paramType='path' required="true" />
<key id='b' type='xs:string' paramType='variable' required="true" />
</inputs>
<execute><![CDATA[
// Your javascript goes here. We will run it on our servers
response.object = <item>
<url>{request.url}</url>
<a>{a}</a>
<b>{b}</b>
</item>;
]]></execute>
</select>
</bindings>
</table>

Scraping a page
<select produces="XML" itemPath="deals.deal">
<urls><url>http://deals.yahoo.com</url></urls>
<execute><![CDATA[
vardealhtml = y.query("select * from html where url='http://deals.yahoo.com' and xpath='//div[contains(@class, "product-display")]'").results;
var deals = <deals/>;
function compress(object) {
if (!object) return "";
return object.toString().replace(/s+/, " ")
}
for each (var div in dealhtml.div) {
var deal = <deal/>;
deal.name += <name>{compress(div.h3.a.text())}</name>;
deal.link += <link>{div.div[0].a.@href}</link>;
…
deal.description += <description>{compress(div.p[0].*)}</description>;
deal.logo += <logo>{compress(div.div[1].*)}</logo>;
deals.deal += deal;
}
response.object = deals;
]]></execute>
</select>

Inner join (unified web+image search)
<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd">
<bindings>
<select itemPath="results.result" produces="XML">
<inputs>
<key id="query" type="xs:string" paramType="query" required="true"/>
</inputs>
<execute><![CDATA[
varqs = query;
var search = y.query('select * from search.web(50) where query=@query', {query: qs}).results;
var images = [];
default xml namespace='http://www.inktomi.com/';
for each (var result in search.result) {
images.push(y.query('select * from search.images(1) where query=@query and url=@url', {url:result.url, query:qs}));
}
vari = 0;
for each (var result in search.result) {
var image = images[i++].results.result;
if (image) { result.image = <image>{image}</image>; }
}
response.object = search;
]]></execute>
</select>
</bindings>
</table>

select *from twitter.statuswhere id="2108869549"andusername="xxx"and password="yyy”
<select itemPath="" produces="XML">
<urls><url>http://twitter.com/statuses/show/{id}.xml</url></urls>
<inputs>
<key id="username" type="xs:string" required="false" paramType="variable"/>
<key id="password" type="xs:string" required="false" paramType="variable"/>
<key id="id" type="xs:integer" required="true" paramType="path"/>
</inputs>
<execute><![CDATA[
varr = null;
if (username && password) {
y.include("http://yqlblog.net/samples/base64.js");
varauthheader = "Basic "+Base64.encode(username+":"+password);
r = request.header("Authorization",authheader).get().response;
} else {
r = request.get().response;
}
response.object = r;
]]></execute>
</select>

insert into twitter.status(status,username,password)values ("new tweet from YQL",”xxx”,”yyy”);
<insert itemPath="" produces="XML">
<urls>
<url>http://twitter.com/statuses/update.xml</url>
</urls>
<inputs>
<value id="username" type="xs:string" required="true" paramType="variable"/>
<value id="password" type="xs:string" required="true" paramType="variable"/>
<value id="status" type="xs:string" required="true" paramType="variable"/>
</inputs>
<execute><![CDATA[
var content = "status="+status;
response.object = request.header("Authorization",authheader).post(content).response;
]]></execute>
</insert>

delete from twitter.statuswhere id="2108869549"and username="xxx"and password="yyy”
<delete itemPath="" produces="XML">
<urls>
<url>http://twitter.com/statuses/destroy/{id}.xml</url>
</urls>
<inputs>
<key id="username" type="xs:string" required="true" paramType="variable"/>
<key id="password" type="xs:string" required="true" paramType="variable"/>
<key id="id" type="xs:string" required="true" paramType="path"/>
</inputs>
<execute><![CDATA[
response.object = request.header("Authorization",authheader).del().response;
]]></execute>
</delete>

Conclusion
More easily build applications
fewer lines of code
data processing done away from app
consistent and familiar syntax for all data access
iterative environment for developing the query

Conclusion
Build applications that run faster