Functional Programming with     Streams in node.js
Who Am I?  HP webOS Framework Team  @CrabDude (Twitter, Github, etc…)  Organizer BayNode & Dallas.node  Author:    tr...
The Problem     VFS         VFS           VFShermesClient hermesClient hermesClient              hermesBuild        3rd Pa...
The Goal  Don’t buffer any data  Don’t write to disk  Minimize latency  Allow reuse of functionality (e.g., readdir) ...
Why node.js?“Nodes goal is to provide an easy way to build  scalable network programs.”                                   ...
node.js === JavaScriptJavaScript is the future of the web,  and code reuse is its unfair advantage.
Basics  EventEmitter  Stream  Traditional streams (aka non-node.js lazy streams)
EventEmitter: Example
EventEmitter: Extend
Streams“Streams are to time as arrays are to space.”              – Jed Schmidt @ JSConf.eu/2010
Readable Streams  Events: data, end, error, close  Methods: pause, resume, end, destroy
Writable Streams  Events: drain, error, close, pause, resume  Methods: write, end, destroy
readable.pipe(writeable)Readable streams can be piped to writable streams  (and vice-versa!)  on(‘data’) => write()  on(...
readable.pipe(writeable): Example
Lazy StreamsStreams in functional programming…
Lazy Streams: Higher Order FunctionsA higher-order function is a function that does at least  one of the following:  Take...
Lazy Streams: Lists
Got All That?
Hermes: Revisited     FTP        Dropbox        Box.net   FilesystemhermesFtp hermesDropbox hermesBox hermesFilesystem    ...
Still Got All That?
We need…We need to recursively zip a directory…  Let’s start with abstract implementation non-specific   functional versi...
We need…  More “abstract implementation non-specific functional   versions” (errr…) of…    ls (aka readdir)    getFile ...
We need…  Yet More AINSFV of...    Zip a stream of files    Allow node.js streams to be piped to lazy streams       pi...
Almost there!
Now we can…Zip a folder up using an arbitrary…  getFile / readFile  ls / readdir… And stream the response! Yeeeeehaw!
Sweeeeeet.
Now What?Since all the implementation specific functions  are passed in, we can…
Port this to the browser!
Say whaaat?node-browserify by @Substack does a lot of this already    Stream, Buffer, Path, etc…We need…    Complete buf...
Stay Tuned.I’m currently porting the fs and buffer modules to  the browser, and eventually zlib.http://github.com/crabdude...
We’re HIRING!            webOS Framework TeamIf you think this is pretty cool…   Or you think I’M pretty cool…Stop by the ...
Thank you.        adam.crabtree@palm.comhttps://github.com/Gozala/streamer/blob/master/readme.jshttp://en.wikipedia.org/wi...
Functional Programming with Streams in node.js
Functional Programming with Streams in node.js
Functional Programming with Streams in node.js
Functional Programming with Streams in node.js
Functional Programming with Streams in node.js
Upcoming SlideShare
Loading in …5
×

Functional Programming with Streams in node.js

18,897
-1

Published on

My talk at HTML5DevConf && FluentConf covering how I combined some basic functional programming techniques with node.js streams to abstract a streaming VFS zipping layer that can also (theoretically / eventually) run in the browser.

Published in: Technology

Functional Programming with Streams in node.js

  1. 1. Functional Programming with Streams in node.js
  2. 2. Who Am I?  HP webOS Framework Team  @CrabDude (Twitter, Github, etc…)  Organizer BayNode & Dallas.node  Author:   trycatch – async try/catch in node.js   stepup – step node.js control flow + trycatch   tiki – in-browser package manager
  3. 3. The Problem VFS VFS VFShermesClient hermesClient hermesClient hermesBuild 3rd Party API hermes hermesDeploy Ares (browser)
  4. 4. The Goal  Don’t buffer any data  Don’t write to disk  Minimize latency  Allow reuse of functionality (e.g., readdir)  Functionality may be local or remote (e.g., getFolder)
  5. 5. Why node.js?“Nodes goal is to provide an easy way to build scalable network programs.” – nodejs.org
  6. 6. node.js === JavaScriptJavaScript is the future of the web, and code reuse is its unfair advantage.
  7. 7. Basics  EventEmitter  Stream  Traditional streams (aka non-node.js lazy streams)
  8. 8. EventEmitter: Example
  9. 9. EventEmitter: Extend
  10. 10. Streams“Streams are to time as arrays are to space.” – Jed Schmidt @ JSConf.eu/2010
  11. 11. Readable Streams  Events: data, end, error, close  Methods: pause, resume, end, destroy
  12. 12. Writable Streams  Events: drain, error, close, pause, resume  Methods: write, end, destroy
  13. 13. readable.pipe(writeable)Readable streams can be piped to writable streams (and vice-versa!)  on(‘data’) => write()  on(‘end’) => end()  on(‘drain’) => resume()  on(‘close’) => destroy()  on(‘error’) => on(‘error’)  on(‘pause’) => pause()
  14. 14. readable.pipe(writeable): Example
  15. 15. Lazy StreamsStreams in functional programming…
  16. 16. Lazy Streams: Higher Order FunctionsA higher-order function is a function that does at least one of the following:  Take one or more functions as input  output a function
  17. 17. Lazy Streams: Lists
  18. 18. Got All That?
  19. 19. Hermes: Revisited FTP Dropbox Box.net FilesystemhermesFtp hermesDropbox hermesBox hermesFilesystem hermesBuild 3rd Party API hermesDeploy Ares (browser)
  20. 20. Still Got All That?
  21. 21. We need…We need to recursively zip a directory…  Let’s start with abstract implementation non-specific functional versions of…   Filter paths from readdir   Filter directories from paths   Filter files from paths   Recursively return all file paths in a directory
  22. 22. We need…  More “abstract implementation non-specific functional versions” (errr…) of…   ls (aka readdir)   getFile (aka readFile)   getFolder (stream of gets)
  23. 23. We need…  Yet More AINSFV of...   Zip a stream of files   Allow node.js streams to be piped to lazy streams   pipe(): lazy stream => node.js stream   pump(): node.js stream => lazy stream
  24. 24. Almost there!
  25. 25. Now we can…Zip a folder up using an arbitrary…  getFile / readFile  ls / readdir… And stream the response! Yeeeeehaw!
  26. 26. Sweeeeeet.
  27. 27. Now What?Since all the implementation specific functions are passed in, we can…
  28. 28. Port this to the browser!
  29. 29. Say whaaat?node-browserify by @Substack does a lot of this already   Stream, Buffer, Path, etc…We need…   Complete buffer implementation (browser has TypedArrays & ArrayBuffer)   fs (browser has FileSystem API)   zlib…   inflate(), deflate(), gzip(), gunzip(), unzip()
  30. 30. Stay Tuned.I’m currently porting the fs and buffer modules to the browser, and eventually zlib.http://github.com/crabdude/fs-browserifyhttp://github.com/crabdude/buffer-browserifyhttp://github.com/crabdude/zlib-browserify
  31. 31. We’re HIRING! webOS Framework TeamIf you think this is pretty cool… Or you think I’M pretty cool…Stop by the Enyo booth or talk to me afterwards.
  32. 32. Thank you. adam.crabtree@palm.comhttps://github.com/Gozala/streamer/blob/master/readme.jshttp://en.wikipedia.org/wiki/Higher-order_functionhttps://github.com/mikeal/requesthttp://felixge.s3.amazonaws.com/11/nodejs-streams.pdfhttps://github.com/substack/node-browserifyFunctional Programming in JavaScript:http://igstan.ro/posts/2011-05-02-understanding-monads-with- javascript.htmlhttp://ndc2011.macsimum.no/mp4/Day2%20Thursday/ Track6%201500-1600.mp4http://drboolean.tumblr.com/http://osteele.com/sources/javascript/functional/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×