plumbing for the next web

Slide 1: Pipelines: plumbing for the next web Ian Forrester backstage.bbc.co.uk cubicgarden.com

Slide 2: Overview In depth discussion of the abstract concept of flow * • Not discussing the merits of UNIX or XML pipelines • Assumes an understanding of a pipe and pipeline •

Slide 3: Imagine. . . Processing data from your local machine with data • on-line Chaining together services you use regularly • Automating the process so it ran at convenient • times Not requiring programming knowledge • A process that is open, extensible and sharable •

Slide 4: The Landscape

Slide 5: Hot on the Horizon REST APIs are dotted all over the landscape • Web applications are opening up • Useful web services with revenue models • RSS and ATOM feeds are becoming ubiquitous • Desktop search is standard in Vista and OSX • Widgets and Gadgets are becoming useful • New projects to organise desktop meta-data •

Slide 6: REST XML API Everywhere

Slide 7: REST XML API Everywhere

Slide 8: Opening Up the Silos

Slide 9: Opening Up the Silos

Slide 10: Web Services with Revenue Models

Slide 11: Ubiquitous Feeds

Slide 12: Widgets and Gadgets

Slide 13: Desktop Search

Slide 14: Semantic Desktop projects

Slide 15: Semantic Desktop projects

Slide 16: File System Meta-data

Slide 17: How People are Building Things Particls • Automator • Yahoo Pipes •

Slide 18: Touchstone / Particls Particls: An alerts/updates and attention • management platform Ingests many different input sources via input • adapters Works out importance of new information to the • user and displays a proportional alert Alerts take many forms based on urgency and relevancy • levels Alerts are controlled with output adapters. •

Slide 19: Particls

Slide 20: A Pipeline Application

Slide 21: XML Configurations

Slide 22: Outline Processing Markup Language (OPML)

Slide 23: Attention Profiling Markup Language (APML)

Slide 24: Particls Internals

Slide 25: Learning from Particls Particls Limitations: • Main purpose is an alert and attention engine • No standard underline definable language • Not sharable outside of the pipes application • Choice of inputs and outputs limited • No automation • Semi-proprietary • Particls Highlights • Breakthrough web application • GUI has all the right elements • Form level abstract is great • Cloning pipes is very smart • Pushing new attention markup standard •

Slide 26: Automator A beautiful graphical user interface for Applescript • “Accomplish all of your time-consuming, repetitive manual • tasks quickly, efficiently and effortlessly with Automator Workflows. It’s simple to create custom Workflows just by dragging items, pointing and clicking. Perform the task once using your Automator Workflow or many times — even share Workflows with friends.” Apple.com •

Slide 27: Apple Automator

Slide 28: Automator

Slide 29: Applescript

Slide 30: Automator Internals

Slide 31: Learning from Automator Automator Limitations • Proprietary • Requires Applescript • Underlying definable language non-XML based • Works better as automation software • Not natively web enabled • Automator Highlights • Break through desktop application • Powerful Gui • Form level abstract is great • Great automation • Sharing of scripts very simple • Choice of inputs and outputs extensible via code •

Slide 32: Yahoo Pipes New beta service that enables users to create • mash-ups with a graphical user interface Mash-ups can be saved so other people can edit • and re-use them Still geeky but easier than creating mash-ups by • coding “a milestone in the history of the Internet\" - Tim O'Reilly •

Slide 33: Y! Pipes

Slide 34: Y! Pipes s

Slide 35: Y! Pipes

Slide 36: Y! Pipes Internals

Slide 37: Learning from Y! Pipes Yahoo! Pipes Limitations • Only has access to data and content on-line • No XSL transformations • No standard underlying definable language • Not sharable outside of the Y! Pipes application • Choice of inputs and outputs limited • No automation • Proprietary • Yahoo! Pipes Highlights • Breakthrough web application • GUI has all the right elements • Form level abstract is great • Cloning pipes is very smart •

Slide 38: Fitting the Puzzle Together Combine: • Underlying XML definitions of Particls • Slimline and smooth look of Particls • Form control of Automator Power and control of Automator • Visual pipeline GUI of Yahoo! Pipes • Automation of Automator • Sharing of Automator and Yahoo! Pipes •

Slide 39: flow *

Slide 40: So what is flow*?

Slide 43: The flow * System

Slide 44: Adding Widgets to the Mix

Slide 45: Separate the Desktop API

Slide 46: But. . .there is no flow * flow * currently does not exist • flow * is not just an application • flow * is not just a service • flow * is a combination of good practices served up • for the power users

Slide 47: My Current flow * Setup

Slide 48: More Configuration Ideas

Slide 51: Flow* is. . . The glue that allows you to break out of the silos • on-line and off-line The ability to pipe services together using their • APIs and feeds as though programming code or interacting with the site directly XPROC documents with namespaces elements (User-generated Pipelines) Three levels of abstraction in one file •

Slide 52: Core User-generated Pipeline Principles Definable • Core pipeline defined as XML • Graphical • Levels of abstraction from the XML • Standard • Using the new W3C standard XPROC • Shareable • As easy to share as a OPML file • Open • Can be extended and plugged into widget engines • Non-proprietary • No lock-in, uses standard web and XML technologies •

Slide 53: Three Levels

Slide 54: Three Levels of Abstraction

Slide 55: Three Levels of Users, One Purpose

Slide 56: XPROC Level

Slide 57: Pipe GUI Level

Slide 58: Task Level

Slide 59: Pipelines Options

Slide 60: The Current Landscape

Slide 61: Particls: An Attention Platform

Slide 62: Automator: Strictly for OSX Only

Slide 63: Y! Pipes: No Access to the Local System

Slide 64: Y! Pipes: Could Access Local System

Slide 65: Yahoo widgets authenticator

Slide 66: Securing the Desktop and Cloud

Slide 67: Y! Pipes: All the Right Elements

Slide 68: Microsoft just around the corner?

Slide 69: Pipelines

Slide 70: User-generated Pipelines Pipelines generation is only limited by our own • creativity and the APIs available XPROC feeds Flow* like Javascript feeds • Greasemonkey User-generated Pipelines could be... • XPROC+SVG+XFORMS • XPROC+XUL • XPROC+RDF+XFORMS • XPROC+FlexML • XPROC+XHTML •

Slide 71: Pipeline Example

Slide 72: Pipeline Example

Slide 73: Complex Pipeline Example

Slide 76: The Ecosystem of the API

Slide 77: No fully functional API? Forget it now... your dead •

Slide 78: Described in XPROC (part 1) <?xml version=\"1.0\" encoding=\"UTF-8\"?> <p:pipeline name=\"encoder example\" xmlns:p=\"http://www.w3.org/2007/03/xproc\"> <p:input port=\"video\" href=\"file://///127.0.0.1/my videos/\"/> <p:choose name=\"free-or-pay\"> <p:when test=\"form:///encoder.dropdown='yes'\"> <p:output port=\"http://www.amazon.com/ws/ec2\"> <p:parameter name=\"amazon/ec2image\" value=\"video-encoding\"/> <p:parameter name=\"amazon/encodeto\" value=\"flv\"/> <p:parameter name=\"amazon/encodeto\" value=\"h.264\"/> <p:parameter name=\"amazon/encodeto\" value=\"mpeg4\"/> <p:parameter name=\"amazon/account\" value=\"form:///amazon.account\"/> <p:parameter name=\"amazon/password\" value=\"form:///amazon.password\"/> <p:parameter name=\"amazon/accesskey\" value=\"form:///amazon.accesskey\"/> <p:parameter name=\"amazon/piority\" value=\"1\"/> </p:output> </p:when> <p:when test=\"form:///encoder.dropdown='no'\"> <p:output port=\"http://www.heywatch.com/\"> <p:parameter name=\"encode\" value=\"flv\"/> <p:parameter name=\"encode\" value=\"h.264\"/> <p:parameter name=\"encode\" value=\"mpeg4\"/> <p:parameter name=\"login\" value=\"form:///account\"/> <p:parameter name=\"password\" value=\"form:///password\"/> </p:output> </p:when> <p:otherwise> <p:output port=\"local://start.loocal.application\" href=\"file://///127.0.0.1/C/program files/mediacleaner/mediaclean.exe\"/> </p:otherwise> </p:choose>

Slide 79: Described in XPROC (part 2) <p:choose name=\"stoarage-options\"> <p:when test=\"form:///storage.selector='amazon'\"> <p:output port=\"http://www.amazon.com/ws/s3\"> <p:parameter name=\"amazon/account\" value=\"form:///amazon.account\"/> <p:parameter name=\"amazon/password\" value=\"form:///amazon.password\"/> <p:parameter name=\"amazon/accesskey\" value=\"form:///amazon.accesskey\"/> <p:parameter name=\"amazon/bucket\" value=\"form:///amazon.bucket\"/> </p:output> </p:when> <p:when test=\"form:///stroage.selector='bliptv'\"> <p:output port=\"http://www.blip.tv\"> <p:parameter name=\"login\" value=\"form:///bliptv.account\"/> <p:parameter name=\"password\" value=\"form:///bliptv.password\"/> </p:output> </p:when> <p:when test=\"form:///stroage.selector='youtube'\"> <p:output port=\"http://www.youtube.com\"> <p:parameter name=\"login\" value=\"form:///youtube.account\"/> <p:parameter name=\"password\" value=\"form:///youtube.password\"/> </p:output> </p:when> <p:otherwise> <p:output port=\"result\" href=\"ftp://user:pass@ftp.cubicgarden.com/myvideo/\"/> </p:otherwise> </p:choose> </p:pipeline>

Slide 80: What happens next?

Slide 81: To do Build flow* using a Rich Internet Application • language like XUL or Apollo Write an Authentication layer for the APIs – Open • ID based broker system for APIs? Decide on one way to write User-generated • Pipelines which works for flow* Write schema for namespaced extensions into • user-generated pipelines (XPROC document) Investigate the semantic desktop projects • Explorer W3Cs draft Widget specification •

Slide 82: Summary User generated pipelines are viable for: • People wanting automation • Non-programmers • Power users who can't be bothered to load up their IDE • Flow* (although only in early development) is very • powerful All the elements are in place, it just needs a developer or • company with a some vision

Slide 83: Final thought Mashing-up, remixing, sharing is part of our culture •

Slide 84: Thanks for listening, questions? Ian Forrester - pipelines@cubicgarden.com • http://www.cubicgarden.com/blojsom/blog/pipelines/ • Flickr pictures : Creative Commons Attribution-NonCommercial-NoDerivs License • http://www.flickr.com/photos/andidfl/ • http://www.flickr.com/photos/_f1guy68_/ • http://www.flickr.com/photos/coolinbox/ • http://www.flickr.com/photos/belljar/ • http://www.flickr.com/photos/dgbalancesrocks/ • http://www.flickr.com/photos/whinger/ • http://www.flickr.com/photos/wasabicube/ • http://www.flickr.com/photos/jacy •

Slideshare.net (beta)