Web as a data resource

Get me my Data !
Use the web as a data source

Hack U @ Georgia Tech - March 8 – 12 2011
2

Subramanyan Murali
yahoo
Mail Engineer
Hacker, Photographer, Traveler
@rmsguhan

Data and hacks
4

Putting something together in 24 hours can be a real rush !
5

Internet is a data exchange medium

Need innovative ways to make the data into information that is useful
7

Any resource on the internet is a source of data …
8

… that means that there is a lot of data 
9

Loads and loads of data and data services 
10

Data that is complex and in many formats 
11

Navigating through all the complexity can be hard !#$%&
12

How to solve the data problem ?
13

http://pipes.yahoo.comsaves the day!
14

Graphical interface to play with data !
15

WYSIWYG = pipes
16

Predefined presentations !RSS, JSON, Map, badge
17

Pipes is a visual tool, where is my command line ??
18

Put your hands up for YQL !
19

Common data access language across the webdeveloper.yahoo.com/yql
20

SQL like syntaxStandard and easy to use
21

select {what} from {service} where {condition} in ({sub-query} )
22

Several services can be chained using in ()
23

YQL Console
24

Lots of Integrations and predefined tablesanyone can be part of the interface
25

Fetching data from a resource has never been simpler !
26

http://query.yahooapis.com/v1/ public/yql?q={query} &format={format}
27

Irrespective of the source, output can be standardized …
28

… thus giving you a solid & consistent data foundation
29

Input can be XML, HTML, CSV, micro-formats, Atom feeds, JSON, RSS
30

Output can be JSON or XML
31

Mix and match YQL and Pipes ?
32

Pipes provides you with processing capabilities
33

YQL is another feed source
34

Interlinking YQL and Pipes = fewer lines of code
35

Not limited to Yahoo! sources

select * from google.translate where q in (select text from twitter.search where q="georgia tech") and target='fr';

Fire off multiple queries at once

select * from query.multi where queries=‘{query1};{query2}’;

Create your own tables with Open tables www.datatables.org/

http://www.yqlblog.net/

Go explore !developer.yahoo.com/yql/console/

Happy Hacking !!
Photos @ http://flickr.com/photos/rmsguhan

Web as a data resource

by Subramanyan Guhan

Accessibility

Upload Details

Usage Rights

1 Embed 1

Statistics

Web as a data resource Presentation Transcript