Web as a data resource

Get me my Data ! Use the web as a data source

Hack U @ Georgia Tech - March 8 – 12 2011 2

Subramanyan Murali yahoo Mail Engineer Hacker, Photographer, Traveler @rmsguhan

Putting something together in 24 hours can be a real rush ! 5

Internet is a data exchange medium

Need innovative ways to make the data into information that is useful 7

Any resource on the internet is a source of data … 8

… that means that there is a lot of data  9

Loads and loads of data and data services  10

Data that is complex and in many formats  11

Navigating through all the complexity can be hard !#$%& 12

How to solve the data problem ? 13

http://pipes.yahoo.comsaves the day! 14

Graphical interface to play with data ! 15

Predefined presentations !RSS, JSON, Map, badge 17

Pipes is a visual tool, where is my command line ?? 18

Put your hands up for YQL ! 19

Common data access language across the webdeveloper.yahoo.com/yql 20

SQL like syntaxStandard and easy to use 21

select {what} from {service} where {condition} in ({sub-query} ) 22

Several services can be chained using in () 23

Lots of Integrations and predefined tablesanyone can be part of the interface 25

Fetching data from a resource has never been simpler ! 26

http://query.yahooapis.com/v1/ public/yql?q={query} &format={format} 27

Irrespective of the source, output can be standardized … 28

… thus giving you a solid & consistent data foundation 29

Input can be XML, HTML, CSV, micro-formats, Atom feeds, JSON, RSS 30

Mix and match YQL and Pipes ? 32

Pipes provides you with processing capabilities 33

Interlinking YQL and Pipes = fewer lines of code 35

select * from google.translate where q in (select text from twitter.search where q="georgia tech") and target='fr';

Fire off multiple queries at once

select * from query.multi where queries=‘{query1};{query2}’;

Create your own tables with Open tables www.datatables.org/

Recommended