0
Jonathan Trevor
jtrevor@yahoo-inc.com
Apt near Park


How do you find an apartment near a park?
Apt near Park

• Craigslist apartment listings
 • For each apartment:
   • Click on map link
   • Check distance to a park...
Apt near Park
• Data is available
 • Craigslist apartment RSS feed
 • Yahoo! Local API to find Parks
• Can do it in about 5...
Apt near Park
• Basically combine feeds + web services
• Yet another custom mashup
 • HousingMaps, ChicagoCrime, ...


• W...
Pipes
  grep -iv yahoo.com squid.log |
  sort | uniq -c | sort -n >
  top_sources.txt


• Unix Pipes for the Web
• Build u...
Pipes
• A free service that lets you remix and
  create data mashups using a visual editor
• No need to host, we do it for...
Pipes
• A free service that lets you remix and
  create data mashups using a visual editor
• No need to host, we do it for...
Pipes
• A free service that lets you remix and
  create data mashups using a visual editor
• No need to host, we do it for...
Demo




Apartment near Something
Piecing things together
      in the cloud
Any Input
Craigslist

 Yahoo!

 Google

  Ebay


Your data
  here!
Any Output
Any Output
Any Output
Any Output
Any Output
Any Output
Any Output
Any Output
Any Output
Any Output
Any Output

             RSS Readers

               Badges

               HTML

              Your app
                h...
Any Process
                    Yahoo! Local
Fetch




        Your Web Service
             Here!




             Sort
Openness
Craigslist
                                                RSS Readers
 Yahoo!      Fetch
                       ...
Examples
Hot Deals Search




•   searches across many different deal hunting sites on the internet
    looking for the best prices...
Fantasy Sports search




•   get the edge on your friends with a single RSS feed based on
    searching 70 sites for fant...
Geoannotated Reuters News




 •   takes an RSS feed from the Reuters news service, and quot;geocodesquot;
     each item ...
Who’s Viewed My
LinkedIn Profile
Kiva Loans by Location




•   gets a list of the micro-loans people have been making through
    the Kiva site, and shows...
Yahoo! Buzz Image
     Search
Craigslist house lookup
with static Yahoo map
Yahoo Finance Stock
Quote Watch List Feed
Contact's Favorite's
Yahoo Unanswered
         Questions




•   finds those questions in the Y! answers site that don't
    currently have an a...
Babbler by Max Case




• Translates IM messages in Second Life
LastTube




• uses content from Last.fm and YouTube.You can
  watch Youtube’s content based on your Recently
  Listened T...
Advantages to developers
     (why use an online service to do this?)

 • Leveraging large infrastructure
  • Faster acces...
Network services
• On Y! network services (fast)
 • Geocoding, Local, Search ...
 • Shortcuts, Term Extraction, Translatio...
Run / Get the data
• Each Pipe gets its own “hosted” page




• Use the REST-style URLs to get the data
Run / Get the data
• Each Pipe gets its own “hosted” page




• Use the REST-style URLs to get the data
Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alt...
Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alt...
Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alt...
Edit REST-style queries
http://pipes.yahoo.com/pipes/pipe.run?
_id=1mrlkB232xGjJDdwXqIxGw
&_render=json
&location=palo+alt...
Disadvantages
     (Why not to use an online service?)



• Away from data
 • personal (desktop)
 • organizational (intran...
Design and
Implementation
Target Users

• The top 10% of the web 2.0 pyramid
 • Coders, re-mixers, bloggers
• Assume prior knowledge
 • Concepts... ...
Architecture
Your client   Your application         Browser   Editor




                        Web Site/API



         ...
Editor

• Edits Pipe definitions
• Heavy lifting performed by Engine
• Rivals a desktop experience
 • Almost everything is ...
Editor Design

• Instant “ON”, no plugins
 • “download this” gets in the way
• Data flow applications are well suited to
  ...
Engine
• Executes Pipes
• Pipes are defined by a simple definition
  format
• Parallelizes as much of the execution as
  pos...
Web addressable data
• is very malformed
• can be slow
• needs considerate access
• can be untrustworthy
• can be inaccess...
Data in the Engine
• is “cleaned” (and repaired) into UTF-8
• is cached for
 • performance
 • playing well with others
 • ...
A year in the wild
(20+ releases, 250k+ Pipes later)
What’s popular
What’s popular
What’s popular
What’s popular
Typical Pipes/mashups

• Four types of mashup
 • Feed aggregation with filtering
 • Two-source mashups
 • Data transformati...
Reasons for adoption
• Lower barrier to use
 • Graphical editor made it quick to write
    Pipes, attracted non-developers...
Inaccessible data
• Lots of requests for more rich and personal
  data
 • Text documents, word documents, mail,
    Excel ...
Power...

• We started by only supporting RSS
 • high-level building blocks and operations
 • good for common tasks and no...
...vs Complexity
• Added sources for parsing JSON, XML,
  CSV, ICAL ...
• Added modules that could do more and
  be combin...
Unexpected breadth

• Experts who want to exploit the service
• Non-programers with much simpler needs
• In Pipes
 • Its e...
Making it easy to use
• Many Pipes provide data that’s useful while mobile
 • Geosensitive information
 • Time sensitive d...
Doing the mobile mash
Doing the mobile mash
Making it easy to use
                             Com
                               ing
                                ...
Badges
Badges
Badges
Badges
Conclusion 1/2
• Provides powerful data functions to any client
• Consumes data from many services
• Common data formats m...
Conclusion 2/2

• (Some) outstanding technical issues
 • Common authentication API
 • Inter-service common rate limiting
 ...
Q&A
Jonathan Trevor Yahoo Pipes
Upcoming SlideShare
Loading in...5
×

Jonathan Trevor Yahoo Pipes

2,686

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,686
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
79
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "Jonathan Trevor Yahoo Pipes"

  1. 1. Jonathan Trevor jtrevor@yahoo-inc.com
  2. 2. Apt near Park How do you find an apartment near a park?
  3. 3. Apt near Park • Craigslist apartment listings • For each apartment: • Click on map link • Check distance to a park on the map • Tedious
  4. 4. Apt near Park • Data is available • Craigslist apartment RSS feed • Yahoo! Local API to find Parks • Can do it in about 50 lines of Perl code #!/usr/bin/perl -w use strict; use LWP::Simple; use XML::Simple; ...
  5. 5. Apt near Park • Basically combine feeds + web services • Yet another custom mashup • HousingMaps, ChicagoCrime, ... • Would be nice if there was an easier way...
  6. 6. Pipes grep -iv yahoo.com squid.log | sort | uniq -c | sort -n > top_sources.txt • Unix Pipes for the Web • Build useful applications from simple primitives
  7. 7. Pipes • A free service that lets you remix and create data mashups using a visual editor • No need to host, we do it for you Craigslist Yahoo! Local
  8. 8. Pipes • A free service that lets you remix and create data mashups using a visual editor • No need to host, we do it for you Craigslist Yahoo! Local
  9. 9. Pipes • A free service that lets you remix and create data mashups using a visual editor • No need to host, we do it for you Craigslist Yahoo! Local
  10. 10. Demo Apartment near Something
  11. 11. Piecing things together in the cloud
  12. 12. Any Input Craigslist Yahoo! Google Ebay Your data here!
  13. 13. Any Output
  14. 14. Any Output
  15. 15. Any Output
  16. 16. Any Output
  17. 17. Any Output
  18. 18. Any Output
  19. 19. Any Output
  20. 20. Any Output
  21. 21. Any Output
  22. 22. Any Output
  23. 23. Any Output RSS Readers Badges HTML Your app here!
  24. 24. Any Process Yahoo! Local Fetch Your Web Service Here! Sort
  25. 25. Openness Craigslist RSS Readers Yahoo! Fetch Yahoo! Local Badges Google Your Web Service Here! HTML Ebay Sort Your app Your data here! here!
  26. 26. Examples
  27. 27. Hot Deals Search • searches across many different deal hunting sites on the internet looking for the best prices.You can search for particular items or just let the pipe find the best of what's available
  28. 28. Fantasy Sports search • get the edge on your friends with a single RSS feed based on searching 70 sites for fantasy sports blog articles
  29. 29. Geoannotated Reuters News • takes an RSS feed from the Reuters news service, and quot;geocodesquot; each item - making it possible to show where that news item is happening on a map of the world.
  30. 30. Who’s Viewed My LinkedIn Profile
  31. 31. Kiva Loans by Location • gets a list of the micro-loans people have been making through the Kiva site, and shows the amazing variety of people and places that these loan are helping out.
  32. 32. Yahoo! Buzz Image Search
  33. 33. Craigslist house lookup with static Yahoo map
  34. 34. Yahoo Finance Stock Quote Watch List Feed
  35. 35. Contact's Favorite's
  36. 36. Yahoo Unanswered Questions • finds those questions in the Y! answers site that don't currently have an answer - so you can show how smart you are and answer those tricky questions.
  37. 37. Babbler by Max Case • Translates IM messages in Second Life
  38. 38. LastTube • uses content from Last.fm and YouTube.You can watch Youtube’s content based on your Recently Listened Tracks scrobbled to Last.fm.
  39. 39. Advantages to developers (why use an online service to do this?) • Leveraging large infrastructure • Faster access to network resources • Faster access to network services • System-wide knowledge • Leverage inter-organizational agreements • Easy to “string” together with other services • Easy to use (REST-style URLs)
  40. 40. Network services • On Y! network services (fast) • Geocoding, Local, Search ... • Shortcuts, Term Extraction, Translation • developer.yahoo.com • Off Y! • Google, AWS, Dapper etc
  41. 41. Run / Get the data • Each Pipe gets its own “hosted” page • Use the REST-style URLs to get the data
  42. 42. Run / Get the data • Each Pipe gets its own “hosted” page • Use the REST-style URLs to get the data
  43. 43. Edit REST-style queries http://pipes.yahoo.com/pipes/pipe.run? _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction The ID of the Pipe
  44. 44. Edit REST-style queries http://pipes.yahoo.com/pipes/pipe.run? _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction The format of the output (rss, json, kml, ical, csv)
  45. 45. Edit REST-style queries http://pipes.yahoo.com/pipes/pipe.run? _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction The per Pipe user customizable parameters
  46. 46. Edit REST-style queries http://pipes.yahoo.com/pipes/pipe.run? _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction Optional JSON callback function
  47. 47. Disadvantages (Why not to use an online service?) • Away from data • personal (desktop) • organizational (intranet) • Security and trust
  48. 48. Design and Implementation
  49. 49. Target Users • The top 10% of the web 2.0 pyramid • Coders, re-mixers, bloggers • Assume prior knowledge • Concepts... loops, data types • End-users benefit indirectly
  50. 50. Architecture Your client Your application Browser Editor Web Site/API Engine Database Sources
  51. 51. Editor • Edits Pipe definitions • Heavy lifting performed by Engine • Rivals a desktop experience • Almost everything is now possible in a browser
  52. 52. Editor Design • Instant “ON”, no plugins • “download this” gets in the way • Data flow applications are well suited to visual programming • Learn and propagate by “View Source” and “Clone”
  53. 53. Engine • Executes Pipes • Pipes are defined by a simple definition format • Parallelizes as much of the execution as possible • Not limited to RSS, supports many common web addressable formats • but...
  54. 54. Web addressable data • is very malformed • can be slow • needs considerate access • can be untrustworthy • can be inaccessible from “here” (behind firewall etc)
  55. 55. Data in the Engine • is “cleaned” (and repaired) into UTF-8 • is cached for • performance • playing well with others • several HTTP proxy layers • serve stale and force caching • is “sanitized”
  56. 56. A year in the wild (20+ releases, 250k+ Pipes later)
  57. 57. What’s popular
  58. 58. What’s popular
  59. 59. What’s popular
  60. 60. What’s popular
  61. 61. Typical Pipes/mashups • Four types of mashup • Feed aggregation with filtering • Two-source mashups • Data transformation and geocoding • Complex mashups using REST APIs • Geocoding remains a “mashup” favorite
  62. 62. Reasons for adoption • Lower barrier to use • Graphical editor made it quick to write Pipes, attracted non-developers • “View Source” for learning/tweaking • Wide array of data input formats and data output formats enabled Pipes to become a useful “component” in a larger ecology • Web 2.0 responsiveness to community
  63. 63. Inaccessible data • Lots of requests for more rich and personal data • Text documents, word documents, mail, Excel spreadsheets • Workarounds (to some) emerged • Online spreadsheets, calendars (gcal) with private RSS feeds and so on
  64. 64. Power... • We started by only supporting RSS • high-level building blocks and operations • good for common tasks and novice users • We listened to our user’s desires
  65. 65. ...vs Complexity • Added sources for parsing JSON, XML, CSV, ICAL ... • Added modules that could do more and be combined in many ways • At the cost of simplicity • Harder to explain, use, compose • Stretching the capabilities of a visual development environment
  66. 66. Unexpected breadth • Experts who want to exploit the service • Non-programers with much simpler needs • In Pipes • Its easy to make useful data in the cloud • Its not easy enough to use it after • Users like things in one place
  67. 67. Making it easy to use • Many Pipes provide data that’s useful while mobile • Geosensitive information • Time sensitive decisions • Letting the cloud do the work allows rich thin clients (in addition to thick heavy ones) • enable novice...expert developers to create (simple) mobile applications that give the right information at the right time
  68. 68. Doing the mobile mash
  69. 69. Doing the mobile mash
  70. 70. Making it easy to use Com ing (ver • Badges are frequently requested y) soo n! • Despite other solutions • Initially three variants visualizing common types of data in Pipes • Geo - map badge • Flickr/Images - image badge • Aggregators, transformers - list badge
  71. 71. Badges
  72. 72. Badges
  73. 73. Badges
  74. 74. Badges
  75. 75. Conclusion 1/2 • Provides powerful data functions to any client • Consumes data from many services • Common data formats means any part of the cloud can become the input • Dapper, AWS, Google spreadsheets • ...or take the output • 1/3 Google mashups are powered by Pipes
  76. 76. Conclusion 2/2 • (Some) outstanding technical issues • Common authentication API • Inter-service common rate limiting scheme • Bridges to local data / dbs
  77. 77. Q&A
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×