The trouble with data • You need to find data API • Get Access – Signup for key • Find data endpoint • Read docs to learn what parameters you have • Get data in obscure format • Use data after converting and filtering • More APIs you use, more is your annoyance
To make data access easy on the web,Yahoo! created YQL
YQL turns webservices and data on the web into databases.
Query Examples select * from yahoo.ﬁnance.quotes where symbol in ("^IXIC","^DJI","YHOO ","AAPL")
Query Examples select * from weather.byloca@on where loca@on in ("bangalore, in", ”new york, us")
• Query Examples Find hackday tweets: • SELECT * FROM twi]er.search where q=hackday’ • Search Yahoo! Answers for resolved ques@ons about cars: • select * from answers.search where query="cars" and type="resolved” • Find distance between Bangalore and Mumbai: • select * from geo.distance where place1="bangalore" and place2="mumbai” • Extract important terms from top stories on Yahoo! news: • select * from search.termextract where context in (select descrip@on from rss where url=h]p://rss.news.yahoo.com/rss/topstories)
Query Examples • Get Olympic medal list • select * from html where url=h]p://sports.yahoo.com/olympics/ medals.html and xpath=//*[@id=mediasportsoverallmedalcount]/ div/table/tbody/tr/td/a • Shorten a URL: • insert into yahoo.y.ahoo.it (url, keysize) values (h]p:// www.javarants.com, 5) • Search apartments in criagslist: • select * from craigslist.search where loca@on="bangalore" and type="apa" and query="indiranagar”
Query Examples • Scrape news from Yahoo! Finance: • select * from html where url="h]p://ﬁnance.yahoo.com/q? s=yhoo" and xpath=//div[@id="yﬁ_headlines"]/div/ul/li/ a’ • Select, ﬁlter data from google spreadsheets: • select * from csv where url="h]ps:// spreadsheets.google.com/pub?key=0ArYndzim-‐ lbrdF8wc3A5QWl1ZGRpdkxRZk80SU9zUXc&output=csv" and col5 like Bangalore%’ ;