My talk at HTML5DevConf && FluentConf covering how I combined some basic functional programming techniques with node.js streams to abstract a streaming VFS zipping layer that can also (theoretically / eventually) run in the browser.

1. Functional Programming with
Streams in node.js

2. Who Am I?
 HP webOS Framework Team
 @CrabDude (Twitter, Github, etc…)
 Organizer BayNode & Dallas.node
 Author:
 trycatch – async try/catch in node.js
 stepup – step node.js control flow + trycatch
 tiki – in-browser package manager

3. The Problem
VFS VFS VFS
hermesClient hermesClient hermesClient
hermesBuild 3rd Party API
hermes hermesDeploy
Ares
(browser)

4. The Goal
 Don’t buffer any data
 Don’t write to disk
 Minimize latency
 Allow reuse of functionality (e.g., readdir)
 Functionality may be local or remote (e.g., getFolder)

5. Why node.js?
“Node's goal is to provide an easy way to build
scalable network programs.”
– nodejs.org

6. node.js === JavaScript
JavaScript is the future of the web,
and code reuse is its unfair advantage.

8. Basics
 EventEmitter
 Stream
 Traditional streams (aka non-node.js lazy streams)

9. EventEmitter: Example

10. EventEmitter: Extend

11. Streams
“Streams are to time as arrays are to space.”
– Jed Schmidt @ JSConf.eu/2010

12. Readable Streams
 Events: data, end, error, close
 Methods: pause, resume, end, destroy

13. Writable Streams
 Events: drain, error, close, pause, resume
 Methods: write, end, destroy

14. readable.pipe(writeable)
Readable streams can be piped to writable streams
(and vice-versa!)
 on(‘data’) => write()
 on(‘end’) => end()
 on(‘drain’) => resume()
 on(‘close’) => destroy()
 on(‘error’) => on(‘error’)
 on(‘pause’) => pause()

15. readable.pipe(writeable): Example

16. Lazy Streams
Streams in functional programming…

17. Lazy Streams: Higher Order Functions
A higher-order function is a function that does at least
one of the following:
 Take one or more functions as input
 output a function

18. Lazy Streams: Lists

19. Got All That?

20. Hermes: Revisited
FTP Dropbox Box.net Filesystem
hermesFtp hermesDropbox hermesBox hermesFilesystem
hermesBuild 3rd Party API
hermesDeploy
Ares
(browser)

21. Still Got All That?

22. We need…
We need to recursively zip a directory…
 Let’s start with abstract implementation non-specific
functional versions of…
 Filter paths from readdir
 Filter directories from paths
 Filter files from paths
 Recursively return all file paths in a directory

24. We need…
 More “abstract implementation non-specific functional
versions” (errr…) of…
 ls (aka readdir)
 getFile (aka readFile)
 getFolder (stream of gets)

26. We need…
 Yet More AINSFV of...
 Zip a stream of files
 Allow node.js streams to be piped to lazy streams
 pipe(): lazy stream => node.js stream
 pump(): node.js stream => lazy stream

28. Almost there!

29. Now we can…
Zip a folder up using an arbitrary…
 getFile / readFile
 ls / readdir
… And stream the response! Yeeeeehaw!

31. Sweeeeeet.

32. Now What?
Since all the implementation specific functions
are passed in, we can…

33. Port this to the browser!

34. Say whaaat?
node-browserify by @Substack does a lot of this already
 Stream, Buffer, Path, etc…
We need…
 Complete buffer implementation (browser has
TypedArrays & ArrayBuffer)
 fs (browser has FileSystem API)
 zlib…
 inflate(), deflate(), gzip(), gunzip(), unzip()

35. Stay Tuned.
I’m currently porting the fs and buffer modules to
the browser, and eventually zlib.
http://github.com/crabdude/fs-browserify
http://github.com/crabdude/buffer-browserify
http://github.com/crabdude/zlib-browserify

36. We’re HIRING!
webOS Framework Team
If you think this is pretty cool…
Or you think I’M pretty cool…
Stop by the Enyo booth or talk to me afterwards.

37. Thank you.
adam.crabtree@palm.com
https://github.com/Gozala/streamer/blob/master/readme.js
http://en.wikipedia.org/wiki/Higher-order_function
https://github.com/mikeal/request
http://felixge.s3.amazonaws.com/11/nodejs-streams.pdf
https://github.com/substack/node-browserify
Functional Programming in JavaScript:
http://igstan.ro/posts/2011-05-02-understanding-monads-with-
javascript.html
http://ndc2011.macsimum.no/mp4/Day2%20Thursday/
Track6%201500-1600.mp4
http://drboolean.tumblr.com/
http://osteele.com/sources/javascript/functional/