Your SlideShare is downloading. ×
Jonathan Trevor Yahoo Pipes
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Jonathan Trevor Yahoo Pipes


Published on

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Jonathan Trevor
  • 2. Apt near Park How do you find an apartment near a park?
  • 3. Apt near Park • Craigslist apartment listings • For each apartment: • Click on map link • Check distance to a park on the map • Tedious
  • 4. Apt near Park • Data is available • Craigslist apartment RSS feed • Yahoo! Local API to find Parks • Can do it in about 50 lines of Perl code #!/usr/bin/perl -w use strict; use LWP::Simple; use XML::Simple; ...
  • 5. Apt near Park • Basically combine feeds + web services • Yet another custom mashup • HousingMaps, ChicagoCrime, ... • Would be nice if there was an easier way...
  • 6. Pipes grep -iv squid.log | sort | uniq -c | sort -n > top_sources.txt • Unix Pipes for the Web • Build useful applications from simple primitives
  • 7. Pipes • A free service that lets you remix and create data mashups using a visual editor • No need to host, we do it for you Craigslist Yahoo! Local
  • 8. Pipes • A free service that lets you remix and create data mashups using a visual editor • No need to host, we do it for you Craigslist Yahoo! Local
  • 9. Pipes • A free service that lets you remix and create data mashups using a visual editor • No need to host, we do it for you Craigslist Yahoo! Local
  • 10. Demo Apartment near Something
  • 11. Piecing things together in the cloud
  • 12. Any Input Craigslist Yahoo! Google Ebay Your data here!
  • 13. Any Output
  • 14. Any Output
  • 15. Any Output
  • 16. Any Output
  • 17. Any Output
  • 18. Any Output
  • 19. Any Output
  • 20. Any Output
  • 21. Any Output
  • 22. Any Output
  • 23. Any Output RSS Readers Badges HTML Your app here!
  • 24. Any Process Yahoo! Local Fetch Your Web Service Here! Sort
  • 25. Openness Craigslist RSS Readers Yahoo! Fetch Yahoo! Local Badges Google Your Web Service Here! HTML Ebay Sort Your app Your data here! here!
  • 26. Examples
  • 27. Hot Deals Search • searches across many different deal hunting sites on the internet looking for the best prices.You can search for particular items or just let the pipe find the best of what's available
  • 28. Fantasy Sports search • get the edge on your friends with a single RSS feed based on searching 70 sites for fantasy sports blog articles
  • 29. Geoannotated Reuters News • takes an RSS feed from the Reuters news service, and quot;geocodesquot; each item - making it possible to show where that news item is happening on a map of the world.
  • 30. Who’s Viewed My LinkedIn Profile
  • 31. Kiva Loans by Location • gets a list of the micro-loans people have been making through the Kiva site, and shows the amazing variety of people and places that these loan are helping out.
  • 32. Yahoo! Buzz Image Search
  • 33. Craigslist house lookup with static Yahoo map
  • 34. Yahoo Finance Stock Quote Watch List Feed
  • 35. Contact's Favorite's
  • 36. Yahoo Unanswered Questions • finds those questions in the Y! answers site that don't currently have an answer - so you can show how smart you are and answer those tricky questions.
  • 37. Babbler by Max Case • Translates IM messages in Second Life
  • 38. LastTube • uses content from and YouTube.You can watch Youtube’s content based on your Recently Listened Tracks scrobbled to
  • 39. Advantages to developers (why use an online service to do this?) • Leveraging large infrastructure • Faster access to network resources • Faster access to network services • System-wide knowledge • Leverage inter-organizational agreements • Easy to “string” together with other services • Easy to use (REST-style URLs)
  • 40. Network services • On Y! network services (fast) • Geocoding, Local, Search ... • Shortcuts, Term Extraction, Translation • • Off Y! • Google, AWS, Dapper etc
  • 41. Run / Get the data • Each Pipe gets its own “hosted” page • Use the REST-style URLs to get the data
  • 42. Run / Get the data • Each Pipe gets its own “hosted” page • Use the REST-style URLs to get the data
  • 43. Edit REST-style queries _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction The ID of the Pipe
  • 44. Edit REST-style queries _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction The format of the output (rss, json, kml, ical, csv)
  • 45. Edit REST-style queries _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction The per Pipe user customizable parameters
  • 46. Edit REST-style queries _id=1mrlkB232xGjJDdwXqIxGw &_render=json &location=palo+alto%2C+ca &mindist=2 &what=parks &_callback=foofunction Optional JSON callback function
  • 47. Disadvantages (Why not to use an online service?) • Away from data • personal (desktop) • organizational (intranet) • Security and trust
  • 48. Design and Implementation
  • 49. Target Users • The top 10% of the web 2.0 pyramid • Coders, re-mixers, bloggers • Assume prior knowledge • Concepts... loops, data types • End-users benefit indirectly
  • 50. Architecture Your client Your application Browser Editor Web Site/API Engine Database Sources
  • 51. Editor • Edits Pipe definitions • Heavy lifting performed by Engine • Rivals a desktop experience • Almost everything is now possible in a browser
  • 52. Editor Design • Instant “ON”, no plugins • “download this” gets in the way • Data flow applications are well suited to visual programming • Learn and propagate by “View Source” and “Clone”
  • 53. Engine • Executes Pipes • Pipes are defined by a simple definition format • Parallelizes as much of the execution as possible • Not limited to RSS, supports many common web addressable formats • but...
  • 54. Web addressable data • is very malformed • can be slow • needs considerate access • can be untrustworthy • can be inaccessible from “here” (behind firewall etc)
  • 55. Data in the Engine • is “cleaned” (and repaired) into UTF-8 • is cached for • performance • playing well with others • several HTTP proxy layers • serve stale and force caching • is “sanitized”
  • 56. A year in the wild (20+ releases, 250k+ Pipes later)
  • 57. What’s popular
  • 58. What’s popular
  • 59. What’s popular
  • 60. What’s popular
  • 61. Typical Pipes/mashups • Four types of mashup • Feed aggregation with filtering • Two-source mashups • Data transformation and geocoding • Complex mashups using REST APIs • Geocoding remains a “mashup” favorite
  • 62. Reasons for adoption • Lower barrier to use • Graphical editor made it quick to write Pipes, attracted non-developers • “View Source” for learning/tweaking • Wide array of data input formats and data output formats enabled Pipes to become a useful “component” in a larger ecology • Web 2.0 responsiveness to community
  • 63. Inaccessible data • Lots of requests for more rich and personal data • Text documents, word documents, mail, Excel spreadsheets • Workarounds (to some) emerged • Online spreadsheets, calendars (gcal) with private RSS feeds and so on
  • 64. Power... • We started by only supporting RSS • high-level building blocks and operations • good for common tasks and novice users • We listened to our user’s desires
  • 65. ...vs Complexity • Added sources for parsing JSON, XML, CSV, ICAL ... • Added modules that could do more and be combined in many ways • At the cost of simplicity • Harder to explain, use, compose • Stretching the capabilities of a visual development environment
  • 66. Unexpected breadth • Experts who want to exploit the service • Non-programers with much simpler needs • In Pipes • Its easy to make useful data in the cloud • Its not easy enough to use it after • Users like things in one place
  • 67. Making it easy to use • Many Pipes provide data that’s useful while mobile • Geosensitive information • Time sensitive decisions • Letting the cloud do the work allows rich thin clients (in addition to thick heavy ones) • enable developers to create (simple) mobile applications that give the right information at the right time
  • 68. Doing the mobile mash
  • 69. Doing the mobile mash
  • 70. Making it easy to use Com ing (ver • Badges are frequently requested y) soo n! • Despite other solutions • Initially three variants visualizing common types of data in Pipes • Geo - map badge • Flickr/Images - image badge • Aggregators, transformers - list badge
  • 71. Badges
  • 72. Badges
  • 73. Badges
  • 74. Badges
  • 75. Conclusion 1/2 • Provides powerful data functions to any client • Consumes data from many services • Common data formats means any part of the cloud can become the input • Dapper, AWS, Google spreadsheets • ...or take the output • 1/3 Google mashups are powered by Pipes
  • 76. Conclusion 2/2 • (Some) outstanding technical issues • Common authentication API • Inter-service common rate limiting scheme • Bridges to local data / dbs
  • 77. Q&A