Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Data Aggregation
using pipes
by Pratibha Gannavarapu
Who am I?
Have been a Technical yahoo for 4 years
Now a Programme Manager
Have done MIT from Sydney University
Creating data mashups
What am I doing here?
There is abundant data on the web;
In different formats like RSS, CSV, XML, Atom, Json;
displayed as web pages
Web
Web
Get the data from multiple locations
What is a mash?
Process for a purpose
Purpose can be an application
What is a mash?
Display the result
What is a mash?
Tools to create a Mash
Pipes
Yahoo Query Language
Pipes – What is it?
Pipes is a free online service that can be used for remixing
feeds and output in RSS 2.0, RSS 1.0, JSO...
Pipes – How to?
Pipes has a visual editor that makes creating a new
pipe as easy as.. Umm.. falling off a log :p
Pipes - editor
canvas
modules
debugger
More explanation?
Lets get our hands dirty
Pipes - Example
My purpose is my application.
“Get all finance news headlines for Yahoo! Nasdaq
ticker YHOO”
Pipes - Purpose
Pipes – Example
Pipes - Example
Pipes - Example
To achieve my purpose
I need RSS feeds from both the sites and merge them
together
http://www.google.com/f...
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - Example
Pipes - anatomy
Each pipe consists of 2 or more modules and each
module has one or more terminals.
We wire the modules tog...
Pipes - dictionary
Canvas
Modules
Wiring
Input and Output Terminals
Pipes – other stuff
Private pipes vs public pipes
Private pipes can be shared using their URLs
Pipes once published are av...
Pipes – what can you do?
Combine 2 or more feeds or take user input and
filter, sort or concatenate them!
You can also exe...
Pipes – Engine
Currently pipes is using V1 engine. It will be
upgraded to V2 (Beta) which is also used for YQL.
So if you ...
Pipes – Cool stuff
It is easy to use with its “visual editor”.
Pipes is a life savor in enabling to jumble data
without tu...
The other way
YQL
YQL - Intro
YQL is more Programmatic, more robust.
SQL-like language that is easy to learn and
understand to run queries.
...
YQL – Example Query
select * from flickr.photos.search where
text=”Yahoo" limit 3
YQL - Console
rest query
query
results
tables
YQL
YQL is a more powerful engine
providing an ability to create open
data tables
Resources
http://developer.yahoo.com/pipes
http://developer.yahoo.com/yql
Blog – http://stills360.com/blog/
Twitter – @pratiba
Facebook - @pratiba
Workshop
Take user input and search for books & following tech
magazines
Google book search base url :
http://books.google...
Workshop
Take user input
Use it as param for book search
Get feeds from Magazine
Search for items with user input in title...
Upcoming SlideShare
Loading in …5
×

Data Aggregation using Yahoo! Pipes

3,347 views

Published on

This presentation gives an introduction to Yahoo Pipes and Yahoo Query Language and shows how to create Data Mashups and Data aggregations using pipes.

  • Be the first to comment

Data Aggregation using Yahoo! Pipes

  1. 1. Data Aggregation using pipes by Pratibha Gannavarapu
  2. 2. Who am I? Have been a Technical yahoo for 4 years Now a Programme Manager Have done MIT from Sydney University
  3. 3. Creating data mashups What am I doing here?
  4. 4. There is abundant data on the web; In different formats like RSS, CSV, XML, Atom, Json; displayed as web pages Web
  5. 5. Web
  6. 6. Get the data from multiple locations What is a mash?
  7. 7. Process for a purpose Purpose can be an application What is a mash?
  8. 8. Display the result What is a mash?
  9. 9. Tools to create a Mash Pipes Yahoo Query Language
  10. 10. Pipes – What is it? Pipes is a free online service that can be used for remixing feeds and output in RSS 2.0, RSS 1.0, JSON, Atom.
  11. 11. Pipes – How to? Pipes has a visual editor that makes creating a new pipe as easy as.. Umm.. falling off a log :p
  12. 12. Pipes - editor canvas modules debugger
  13. 13. More explanation? Lets get our hands dirty Pipes - Example
  14. 14. My purpose is my application. “Get all finance news headlines for Yahoo! Nasdaq ticker YHOO” Pipes - Purpose
  15. 15. Pipes – Example
  16. 16. Pipes - Example
  17. 17. Pipes - Example To achieve my purpose I need RSS feeds from both the sites and merge them together http://www.google.com/finance/company_news?q=N ASDAQ:YHOO&output=rss http://finance.yahoo.com/rss/headline?s=YHOO
  18. 18. Pipes - Example
  19. 19. Pipes - Example
  20. 20. Pipes - Example
  21. 21. Pipes - Example
  22. 22. Pipes - Example
  23. 23. Pipes - Example
  24. 24. Pipes - Example
  25. 25. Pipes - Example
  26. 26. Pipes - Example
  27. 27. Pipes - Example
  28. 28. Pipes - anatomy Each pipe consists of 2 or more modules and each module has one or more terminals. We wire the modules together.
  29. 29. Pipes - dictionary Canvas Modules Wiring Input and Output Terminals
  30. 30. Pipes – other stuff Private pipes vs public pipes Private pipes can be shared using their URLs Pipes once published are available for search engines to crawl Hosted pipes make life much more easier
  31. 31. Pipes – what can you do? Combine 2 or more feeds or take user input and filter, sort or concatenate them! You can also execute regular expressions, date formatting and much much more!
  32. 32. Pipes – Engine Currently pipes is using V1 engine. It will be upgraded to V2 (Beta) which is also used for YQL. So if you have a V1 pipe, migrate to V2
  33. 33. Pipes – Cool stuff It is easy to use with its “visual editor”. Pipes is a life savor in enabling to jumble data without tumbling into code. Hosting and SEO are taken care of!! Yay!
  34. 34. The other way YQL
  35. 35. YQL - Intro YQL is more Programmatic, more robust. SQL-like language that is easy to learn and understand to run queries. It returns either XML or JSON
  36. 36. YQL – Example Query select * from flickr.photos.search where text=”Yahoo" limit 3
  37. 37. YQL - Console rest query query results tables
  38. 38. YQL YQL is a more powerful engine providing an ability to create open data tables
  39. 39. Resources http://developer.yahoo.com/pipes http://developer.yahoo.com/yql
  40. 40. Blog – http://stills360.com/blog/ Twitter – @pratiba Facebook - @pratiba
  41. 41. Workshop Take user input and search for books & following tech magazines Google book search base url : http://books.google.com/books/feeds/volumes User input should be passed as param q. So the final query looks as: http://books.google.com/books/feeds/volumes?q=footbal l+-soccer&start-index=11&max-results=10 http://feeds.informationweek.com/infoweek/all_stories_b logs
  42. 42. Workshop Take user input Use it as param for book search Get feeds from Magazine Search for items with user input in title or description Merge the feeds from both sources Sort them to show latest ones first

×