Streams
FunctionalJs - London - May 2014
@darachennis
A <3able Stream
// by @TooTallNate - https://gist.github.com/TooTallNate/3947591 

// npm install lame ; npm install speak...
A <3able Stream
// by @TooTallNate - https://gist.github.com/TooTallNate/3947591 

// npm install lame ; npm install speak...
Sound Soup
var fs = require('fs');
var lame = require('lame');
var Speaker = require('speaker');
!
// Cache names of mp3's...
A brief history in
Node.js streams
UNIXy
• UNIX $ a | b | c | d
• NODE.js> a.pipe(b).pipe(c).pipe(d)
• same as:
• a.pipe(b);

b.pipe(c);

c.pipe(d);

// but,...
Streams 0
• Util.pump(Readable,Writable)
Streams 1
• EventEmitter and a pipe() function.
• pipe() is an instance of the Builder Pattern
• EventEmitter is an instan...
Streams 1: Example
var Stream = require(‘stream').Stream
var util = require('util')
function MyStream () {
Stream.call(thi...
Streams 2
• EventEmitter and a pipe() function.
• Readable, Writable, Duplex, Transform,
Passthrough (and classic) ‘behavi...
Streams 2: Using Readable
// From: Stream Handbook, Readable with Push
var Readable = require('stream').Readable;
var rs =...
Streams 2: Using Readable
// From: Stream Handbook
var Readable = require('stream').Readable;
var rs = Readable();
var c =...
Streams 2: Pop Quiz!!!
$ node readable-popquiz.js
abcdefghijklmnopqrstuvwxyz%
$
!
Streams 2: Pop Quiz!!!
// From: Stream Handbook (tweaked)
var Readable = require('stream').Readable;
var rs = Readable();
...
Streams 2: Pop Quiz!!!
$ node readable-popquiz.js
aaaabbbbccccddddeeeeffffgggghhhhiiiijjjjkkkkl
lllmmmmnnnnooooppppqqqqrrr...
Streams 2: Using Writable
// From: Stream Handbook
var fs = require('fs');
var ws = fs.createWriteStream('message.txt');
w...
Streams 2: Using Writable
var Writable = require('stream').Writable;
var src = process.stdin;
var dst = new Writable;
dst....
Streams 2: Transform
var Transform = stream.Transform || require('readable-stream').Transform;
function Hash(algorithm, en...
Streams 2: Transform
var hash = require('./hash.js');
var fs = require('fs');
var rs = fs.createReadStream(‘file.js’);
var...
Streams 2: Duplex
* Independent Readable and Writable channels
* Implement _read and _write
* Example use



- network pro...
Know a classic stream
// From: Stream Handbook
process.stdin.on('data', function (buf) {
console.log(buf);
});
process.std...
Streams 3
• Streams2 tweaked
• API the same as Streams2
Useful modules
• @DominicTarr’s event-stream 

https://github.com/dominictarr/event-stream
• @RVagg’s through2

https://gi...
• https://github.com/substack/stream-handbook -
@SubStack’s Stream handbook
• http://www.nearform.com/nodecrunch/dont-use-...
Meta Programming with Streams
and Pipes
Great British Node Conference	

October 8th 2013	

London	

!
Darach Ennis
Streams 101
! Readable Streams	

! IO / throughput oriented	

! Events – data, end, error, close	

! Methods – pause, resu...
Streams … 4IO
! IO Oriented	

! Finite, not infinite	

! Asynchronous	

! Lazy	

! Assumes in transit data, not in memory ...
CSV Reader .@maxogden
Beams
Streams for Compute
Beams 101
! Sources	

! Where data pours in	

! Sinks	

! Where results pop out	

! Operators	

! Filters – Drop or pass o...
Beams… 4CPU
! Compute Oriented	

! Infinite, not finite	

! Synchronous	

! Eager	

! Assumes in memory form, not encoded ...
Beams - Branch
Beams - Combine
Ex – Fly NodeCopter with Streams!
Extend Games to Robots!
Extend?
Meta Programming
A minor problem
! Eager: a.pipe(operation).pipe(z).pipe(a);	

! Implies:



RangeError: Maximum call stack size exceeded
Goto + Modules = Magic
Extend Beam with Modules
Questions & Thanks
! npm install beam	

! Compute oriented streams	

! Branch, Combine, Filter, Transform data	

! Extend ...
Dojo - nodeschool.io
• Basics
• Stream Adventure
• Functional Javascript
FunctionalJS - May 2014 - Streams
Upcoming SlideShare
Loading in...5
×

FunctionalJS - May 2014 - Streams

455

Published on

Talk about Node.js streams and related

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
455
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

FunctionalJS - May 2014 - Streams

  1. 1. Streams FunctionalJs - London - May 2014 @darachennis
  2. 2. A <3able Stream // by @TooTallNate - https://gist.github.com/TooTallNate/3947591 
 // npm install lame ; npm install speaker ; node mp3player.js var fs = require('fs'); var lame = require('lame'); var Speaker = require('speaker'); fs.createReadStream(process.argv[2]) .pipe(new lame.Decoder()) .on('format', function (format) { this.pipe(new Speaker(format)); });
  3. 3. A <3able Stream // by @TooTallNate - https://gist.github.com/TooTallNate/3947591 
 // npm install lame ; npm install speaker ; node mp3player.js var fs = require('fs'); var lame = require('lame'); var Speaker = require('speaker'); fs.createReadStream(process.argv[2]) .pipe(new lame.Decoder()) .on('format', function (format) { this.pipe(new Speaker(format)); });
  4. 4. Sound Soup var fs = require('fs'); var lame = require('lame'); var Speaker = require('speaker'); ! // Cache names of mp3's in cwd function isMp3(e) { return e.indexOf("mp3") > 1 }; var soundz = fs.readdirSync('.').filter(isMp3); ! // Every half second, play a random sound setInterval(function(){ blend() },500); function rand(min, max) { var offset = min; var range = (max - min) + 1; return Math.floor( Math.random() * range) + offset; } function blend() { fs.createReadStream(soundz[rand(0,soundz.length-1)]) .pipe(new lame.Decoder()) .on('format', function (format) { this.pipe(new Speaker(format)); }); }
  5. 5. A brief history in Node.js streams
  6. 6. UNIXy • UNIX $ a | b | c | d • NODE.js> a.pipe(b).pipe(c).pipe(d) • same as: • a.pipe(b);
 b.pipe(c);
 c.pipe(d);
 // but, less typing for the win o/
  7. 7. Streams 0 • Util.pump(Readable,Writable)
  8. 8. Streams 1 • EventEmitter and a pipe() function. • pipe() is an instance of the Builder Pattern • EventEmitter is an instance of Subject/Observer Pattern
  9. 9. Streams 1: Example var Stream = require(‘stream').Stream var util = require('util') function MyStream () { Stream.call(this) } util.inherits(MyStream, Stream) // stream logic and state management and back-pressure // you’re on your own (with the rest of the pipeline) // this was error prone and painful …
  10. 10. Streams 2 • EventEmitter and a pipe() function. • Readable, Writable, Duplex, Transform, Passthrough (and classic) ‘behaviours' • New ‘behaviours’ encourage good ‘backpressure' handling disciplines
  11. 11. Streams 2: Using Readable // From: Stream Handbook, Readable with Push var Readable = require('stream').Readable; var rs = new Readable; rs.push(‘beep ‘); rs.push(‘boopn’); rs.push(null); rs.pipe(process.stdout);
  12. 12. Streams 2: Using Readable // From: Stream Handbook var Readable = require('stream').Readable; var rs = Readable(); var c = 97; rs._read = function () { rs.push(String.fromCharCode(c++)); if (c > 'z'.charCodeAt(0)) rs.push(null); }; rs.pipe(process.stdout); // _read called once, prints ‘a-z'
  13. 13. Streams 2: Pop Quiz!!! $ node readable-popquiz.js abcdefghijklmnopqrstuvwxyz% $ !
  14. 14. Streams 2: Pop Quiz!!! // From: Stream Handbook (tweaked) var Readable = require('stream').Readable; var rs = Readable(); var c = 97; rs._read = function () { rs.push(String.fromCharCode(c++)); if (c > 'z'.charCodeAt(0)) rs.push(null); }; rs.pipe(process.stdout); // _read called once rs.pipe(process.stdout); // _read called twice rs.pipe(process.stdout); // _read called three times rs.pipe(process.stdout); // _read called four times
  15. 15. Streams 2: Pop Quiz!!! $ node readable-popquiz.js aaaabbbbccccddddeeeeffffgggghhhhiiiijjjjkkkkl lllmmmmnnnnooooppppqqqqrrrrssssttttuuuuvvvvww wwxxxxyyyyzzzz% $ … WAT? (Think about it) !
  16. 16. Streams 2: Using Writable // From: Stream Handbook var fs = require('fs'); var ws = fs.createWriteStream('message.txt'); ws.write('beep ‘); // Write … setTimeout(function () { ws.end(‘boopn'); // We’re done writing }, 1000);
  17. 17. Streams 2: Using Writable var Writable = require('stream').Writable; var src = process.stdin; var dst = new Writable; dst._write = function (chunk, enc, next) { console.log(chunk.toString()); next(); }; src.push("hit any key!"); src.pipe(dst);
  18. 18. Streams 2: Transform var Transform = stream.Transform || require('readable-stream').Transform; function Hash(algorithm, encoding, options) { Transform.call(this, options); this.digester = crypto.createHash(algorithm); this.encoding = encoding; } util.inherits(Hash, Transform); Hash.prototype._transform = function (chunk, enc, cb) { var bf = (Buffer.isBuffer(chunk)) ? chunk : new Buffer(chunk, enc); this.digester.update(bf); cb(); }; Hash.prototype._flush = function (cb) { this.push(this.digester.digest(this.encoding)); cb(); };
  19. 19. Streams 2: Transform var hash = require('./hash.js'); var fs = require('fs'); var rs = fs.createReadStream(‘file.js’); var md5Hash = hash.Hash('md5', 'base64'); // var sha1Hash = hash.Hash('sha1', 'base64'); rs.pipe(md5Hash).pipe(process.stdout);
  20. 20. Streams 2: Duplex * Independent Readable and Writable channels * Implement _read and _write * Example use
 
 - network protocols
 
 - serial communications with hardware
 
 - …
  21. 21. Know a classic stream // From: Stream Handbook process.stdin.on('data', function (buf) { console.log(buf); }); process.stdin.on('end', function () { console.log('__END__'); });
  22. 22. Streams 3 • Streams2 tweaked • API the same as Streams2
  23. 23. Useful modules • @DominicTarr’s event-stream 
 https://github.com/dominictarr/event-stream • @RVagg’s through2
 https://github.com/rvagg/through2 • @Raynos’s duplexer
 https://github.com/Raynos/duplexer
  24. 24. • https://github.com/substack/stream-handbook - @SubStack’s Stream handbook • http://www.nearform.com/nodecrunch/dont-use- nodes-core-stream-module#.U4I71JRdWuk - Why @Rvagg doesn’t use node’s core stream module (& nor should we!) Good to know …
  25. 25. Meta Programming with Streams and Pipes Great British Node Conference October 8th 2013 London ! Darach Ennis
  26. 26. Streams 101 ! Readable Streams ! IO / throughput oriented ! Events – data, end, error, close ! Methods – pause, resume, end, destroy ! Writeable Streams ! IO / throughput oriented ! Events - drain, pause, resume, error, close ! Methods – write, end, destroy
  27. 27. Streams … 4IO ! IO Oriented ! Finite, not infinite ! Asynchronous ! Lazy ! Assumes in transit data, not in memory forms ! Great for IO. Not efficient for compute
  28. 28. CSV Reader .@maxogden
  29. 29. Beams Streams for Compute
  30. 30. Beams 101 ! Sources ! Where data pours in ! Sinks ! Where results pop out ! Operators ! Filters – Drop or pass on data based on a UDF ! Transforms – Mutate data based on a UDF ! Branch with Filters ! Combine with Sinks
  31. 31. Beams… 4CPU ! Compute Oriented ! Infinite, not finite ! Synchronous ! Eager ! Assumes in memory form, not encoded or wire data ! Convenient for compute. Not good for IO
  32. 32. Beams - Branch
  33. 33. Beams - Combine
  34. 34. Ex – Fly NodeCopter with Streams!
  35. 35. Extend Games to Robots!
  36. 36. Extend? Meta Programming
  37. 37. A minor problem ! Eager: a.pipe(operation).pipe(z).pipe(a); ! Implies:
 
 RangeError: Maximum call stack size exceeded
  38. 38. Goto + Modules = Magic
  39. 39. Extend Beam with Modules
  40. 40. Questions & Thanks ! npm install beam ! Compute oriented streams ! Branch, Combine, Filter, Transform data ! Extend with goto and modules. ! npm install eep ! Beam will be integrated into eep soon ! Eep provides aggregate windows ! Slide, Tumble, Monotonic, Periodic ! Stats Library ! Ordering now supported ! npm install nodesapiens ! Robosapiens V1 for node.js ! Code on github ! https://github.com/darach
  41. 41. Dojo - nodeschool.io • Basics • Stream Adventure • Functional Javascript
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×