Functional Programming with Streams in node.js

Who Am I?  HP webOS Framework Team  @CrabDude (Twitter, Github, etc…)  Organizer BayNode & Dallas.node  Author:   trycatch – async try/catch in node.js   stepup – step node.js control flow + trycatch   tiki – in-browser package manager

The Problem VFS VFS VFShermesClient hermesClient hermesClient hermesBuild 3rd Party API hermes hermesDeploy Ares (browser)

The Goal  Don’t buffer any data  Don’t write to disk  Minimize latency  Allow reuse of functionality (e.g., readdir)  Functionality may be local or remote (e.g., getFolder)

Why node.js?“Nodes goal is to provide an easy way to build scalable network programs.” – nodejs.org

node.js === JavaScriptJavaScript is the future of the web, and code reuse is its unfair advantage.

Basics  EventEmitter  Stream  Traditional streams (aka non-node.js lazy streams)

EventEmitter: Example

EventEmitter: Extend

Streams“Streams are to time as arrays are to space.” – Jed Schmidt @ JSConf.eu/2010

Readable Streams  Events: data, end, error, close  Methods: pause, resume, end, destroy

Writable Streams  Events: drain, error, close, pause, resume  Methods: write, end, destroy

readable.pipe(writeable)Readable streams can be piped to writable streams (and vice-versa!)  on(‘data’) => write()  on(‘end’) => end()  on(‘drain’) => resume()  on(‘close’) => destroy()  on(‘error’) => on(‘error’)  on(‘pause’) => pause()

readable.pipe(writeable): Example

Lazy StreamsStreams in functional programming…

Lazy Streams: Higher Order FunctionsA higher-order function is a function that does at least one of the following:  Take one or more functions as input  output a function

Lazy Streams: Lists

Got All That?

Hermes: Revisited FTP Dropbox Box.net FilesystemhermesFtp hermesDropbox hermesBox hermesFilesystem hermesBuild 3rd Party API hermesDeploy Ares (browser)

Still Got All That?

We need…We need to recursively zip a directory…  Let’s start with abstract implementation non-specific functional versions of…   Filter paths from readdir   Filter directories from paths   Filter files from paths   Recursively return all file paths in a directory

We need…  More “abstract implementation non-specific functional versions” (errr…) of…   ls (aka readdir)   getFile (aka readFile)   getFolder (stream of gets)

We need…  Yet More AINSFV of...   Zip a stream of files   Allow node.js streams to be piped to lazy streams   pipe(): lazy stream => node.js stream   pump(): node.js stream => lazy stream

Almost there!

Now we can…Zip a folder up using an arbitrary…  getFile / readFile  ls / readdir… And stream the response! Yeeeeehaw!

Sweeeeeet.

Now What?Since all the implementation specific functions are passed in, we can…

Port this to the browser!

Say whaaat?node-browserify by @Substack does a lot of this already   Stream, Buffer, Path, etc…We need…   Complete buffer implementation (browser has TypedArrays & ArrayBuffer)   fs (browser has FileSystem API)   zlib…   inflate(), deflate(), gzip(), gunzip(), unzip()

Stay Tuned.I’m currently porting the fs and buffer modules to the browser, and eventually zlib.http://github.com/crabdude/fs-browserifyhttp://github.com/crabdude/buffer-browserifyhttp://github.com/crabdude/zlib-browserify

We’re HIRING! webOS Framework TeamIf you think this is pretty cool… Or you think I’M pretty cool…Stop by the Enyo booth or talk to me afterwards.

Thank you. adam.crabtree@palm.comhttps://github.com/Gozala/streamer/blob/master/readme.jshttp://en.wikipedia.org/wiki/Higher-order_functionhttps://github.com/mikeal/requesthttp://felixge.s3.amazonaws.com/11/nodejs-streams.pdfhttps://github.com/substack/node-browserifyFunctional Programming in JavaScript:http://igstan.ro/posts/2011-05-02-understanding-monads-with- javascript.htmlhttp://ndc2011.macsimum.no/mp4/Day2%20Thursday/ Track6%201500-1600.mp4http://drboolean.tumblr.com/http://osteele.com/sources/javascript/functional/

http://www.scoop.it	5914
http://us-w1.rockmelt.com	8
http://translate.googleusercontent.com	5
http://webcache.googleusercontent.com	3
http://pixel67.tumblr.com	1
http://a0.twimg.com	1
https://si0.twimg.com	1

Functional Programming with Streams in node.js

by Adam Crabtree

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 5,933

Statistics