YQL: Hacking on steroids - Yahoo! Open Hack Day 2012

HACKING ONSTEROIDS WITH YQLSaurabh SahniYDN Product Guy & HackerTwitter: @saurabhsahni

Hacking togethersystems in 24 hours islot of fun.

Data manipulationHacks = + Data visualization

The web has lot of data around

ProgrammableWeb.Com – 6831 APIs

Yahoo! has opened up its data

http://developer.yahoo.com/everything.html

THE TROUBLE WITH DATA •  You need to find data API •  Get Access – Signup for key •  Find data endpoint •  Read docs to learn what parameters you have •  Get data in obscure format •  Use data after converting and filtering •  More APIs you use, more is your annoyance

To make dataaccess easy on theweb, Yahoo!created YQL

YQL turns webservices and data onthe web intodatabases.

select {what} from {where} where {conditions}

You can select, filter,sort and limit

You can even insert,update and deletefrom it.

FINDING VIDEOS ABOUT BANGALORESELECT * FROM youtube.search wherequery=bangalore

SELECTING PHOTOS OF HACKDAYSELECT * FROM flickr.photos.search wheretext="hackday” and api_key=“b5a60b2a…”

INSERTING DATAINSERT INTO bitly.shorten (login, apiKey, longUrl)VALUES (ME, API_KEY, http://yahoo.com)

UPDATING DATA UPDATE social.profile.status SET status="Using YQL UPDATE” WHERE guid="NJFIDHVPVVISDX7UKED2WHU"

RETRIEVING MY CONTACTSSELECT * FROM social.contacts WHEREguid=me

ACCESSING PRIVATE DATA http://query.yahooapis.com/v1/yqlUses OAuth 1.0 for authorizationOAuth is complicated – use one of our SDKs athttps://github.com/yahoo

You can also mix andmatch several webservices using the in()command.

select * from search.termextractwhere context in (selectdescription from rss whereurl=http://rss.news.yahoo.com/rss/topstories)

Almost all the topAPIs on web areaccessible from YQL

Some of them amazon foursquare peerindexapple geo salesforcebbc github slidesharebible google themovidebboss hackernews tumblrcampfire ign twittercontentanalysis intuit vimeocraigslist kiva weatherdelicious klout yahoodopplr lastfm youtubeetsy netflix zillowfacebook paypal

You want evenmore?

Alright, how about this? atom json csv microformats feed rss html xml

The easiest way tostart with YQL is touse the consolehttp://developer.yahoo.com/yql/console

YQL: http://developer.yahoo.com/yql/console

How to get this datain your app?

YQL is a REST APIin itself and has twoendpoints

The public endpoint does not needany authentication.http://query.yahooapis.com/v1/public/yql?q={query} &format={format}

The private endpoint needs oauthauthentication.http://query.yahooapis.com/v1/yql?q={query}&format={format}

Output formats are XMLor JSON

LET’S SEE IT

QUERY EXAMPLES select * from yahoo.ﬁnance.quotes where symbol in ("^IXIC","^DJI","YHOO ","AAPL")

QUERY EXAMPLES select * from weather.bylocaHon where locaHon in ("bangalore, in", ”new york, us")

QUERY EXAMPLESFind hackday tweets:SELECT * FROM twitter.search where q=hackday’Search Yahoo! Answers for resolved questions about cars:select * from answers.search where query="cars" and type="resolved”Find distance between Bangalore and Mumbai:select * from geo.distance where place1="bangalore" andplace2="mumbai”Extract important terms from top stories on Yahoo! news:select * from search.termextract where context in (select descriptionfrom rss where url=http://rss.news.yahoo.com/rss/topstories)

QUERY EXAMPLESGet Olympic medal listselect * from html where url=http://sports.yahoo.com/olympics/medals.html and xpath=//*[@id=mediasportsoverallmedalcount]/div[2]/table/tbody/tr/td/aShorten a URL:insert into yahoo.y.ahoo.it (url, keysize) values (http://www.javarants.com, 5)Search apartments in criagslist:select * from craigslist.search where location="bangalore" andtype="apa" and query="indiranagar”

QUERY EXAMPLESScrape news from Yahoo! Finance:select * from html where url="http://finance.yahoo.com/q?s=yhoo" and xpath=//div[@id="yfi_headlines"]/div[2]/ul/li/a’Select, filter data from google spreadsheets:select * from csv where url="https://spreadsheets.google.com/pub?key=0ArYndzim-lbrdF8wc3A5QWl1ZGRpdkxRZk80SU9zUXc&output=csv"and col5 like Bangalore%’ ;

Let’s find hackdayphotos on flickr

How about limitingto those clicked inBangalore

MAKING REQUESTS: FLICKR URLS <photo farm="3" id="5708163920" isfamily="0" isfriend="0" ispublic="1" owner="31832337@N04" secret="0075137487" server="2496" title="San Francisco"/>

MAKING REQUESTS: FLICKR URLS Photo URL http://farm{$farm}.static.flickr.com/{$server}/ {$id}_{$secret}.jpg Photo Page URL http://www.flickr.com/photos/{$owner}/{$id} Photo Owner Profile URL http://www.flickr.com/photos/{$owner}

https://github.com/saurabhsahni/Hacks/

Finding Recent Photos from ﬂickr

Some YQL Hacks

ChromYQLip is achrome extension forpage scraping via YQLOpen Hack Bangalore 2010 Winner

http://bit.ly/chromeYQL

VIDEO CLIP

http://www.webmeme.in

WEBMEME.INFetch multiple feeds in different formats like atom, RSS andtransform them into consistent RSS formatSelect * from rss where url in (‘http://feeds.feedburner.com/pluggd’,‘http://quatrainman.blogspot.com/atom.xml’, ‘…’)Filter news containing “india” from multiple feeds:select * from rss where url in (http://feeds.feedburner.com/TechCrunch, http://www.readwriteweb.com/rss.xml,http://gigaom.com/feed/) and description like %india%’

YQL is open – youcan get your datatables in our system

All you need to do iswrite an XMLschema and put iton Github.

http://github.com/yql/yql-tables

Here is the craigslistsearch tablehttps://github.com/yql/yql-tables/tree/master/craigslist/craigslist.search.xml

USE INSTANTLY BY UPLOADING ON YOURSITE USE http://www.mysite.com/my_table.xml AS mytable; SELECT * FROM mytable WHERE user=saurabh

You can even writeserver sidejavascript to build awebservice oraugment one.hLp://developer.yahoo.com/yql/guide/yql-‐execute-‐chapter.html

There are lot ofthings you can dowith YQL.

Play yourselfhttp://developer.yahoo.com/yql/

One more thing

RESOURCESAll Yahoo! APIs and Serviceshttp://developer.yahoo.com/everything.htmlYQL Documentationhttp://developer.yahoo.com/yqlYQL Consolehttp://developer.yahoo.com/yql/consoleYQL Github Account (Contribute Tables)http://github.com/yql/yql-tables

THANKS!http://www.slideshare.net/saurabhsahni Saurabh Sahni Twitter: @saurabhsahni Github: http://github.com/saurabhsahni Web: http://www.saurabhsahni.com

https://si0.twimg.com	8
http://www.unscatter.com	3
https://twimg0-a.akamaihd.net	1
http://tweetedtimes.com	1

YQL: Hacking on steroids - Yahoo! Open Hack Day 2012

by Saurabh Sahni on Aug 11, 2012

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 13

Statistics

YQL: Hacking on steroids - Yahoo! Open Hack Day 2012 Presentation Transcript