"It’s about time to embrace Streams" Luciano Mammino

K Y I V 2 0 1 9
Luciano Mammino (@loige)
IT’S ABOUT TIME TOIT’S ABOUT TIME TO
EMBRACE STREAMSEMBRACE STREAMS

loige.link/streams-kyiv
May 18th
1

// buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
const [,, src, dest] = process.argv
// read entire file content
const content = readFileSync(src)
// write that content somewhere else
writeFileSync(dest, content)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
@loige2

// buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
const {
readFileSync,
writeFileSync
} = require('fs')
// buffer-copy.js1
2
3
4
5
6
7
const [,, src, dest] = process.argv8
9
// read entire file content10
const content = readFileSync(src)11
12
// write that content somewhere else13
writeFileSync(dest, content)14
@loige2

// buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
const {
readFileSync,
writeFileSync
} = require('fs')
// buffer-copy.js1
2
3
4
5
6
7
9
12
// buffer-copy.js1
2
const {3
readFileSync,4
writeFileSync5
} = require('fs')6
7
9
10
11
12
@loige2

// buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
const {
readFileSync,
writeFileSync
} = require('fs')
// buffer-copy.js1
2
3
4
5
6
7
9
12
// buffer-copy.js1
2
const {3
readFileSync,4
writeFileSync5
} = require('fs')6
7
9
10
11
12
// buffer-copy.js1
2
const {3
readFileSync,4
writeFileSync5
} = require('fs')6
7
9
12
13
14
@loige2

WE DO THIS ALL THE TIMEWE DO THIS ALL THE TIME
@loige3

AND IT'S OKAND IT'S OK
@loige3

AND IT'S OKAND IT'S OK
BUT SOMETIMES ...BUT SOMETIMES ...
@loige3

@loige
ERR_FS_FILE_TOO_LARGE! ERR_FS_FILE_TOO_LARGE!
File size is greater than possible Buffer
4

IF BYTES IF BYTES WEREWERE BLOCKS... BLOCKS...@loige
6

MARIO CAN LIFTMARIO CAN LIFT
FEW BLOCKSFEW BLOCKS
@loige
7

BUT NOT TOO MANY...BUT NOT TOO MANY...@loige
?!
8

WHAT CAN WE DO IF WE HAVE TOWHAT CAN WE DO IF WE HAVE TO
MOVE MANY BLOCKS?MOVE MANY BLOCKS?
@loige9

WE CAN MOVE THEM ONE BY ONE!WE CAN MOVE THEM ONE BY ONE!
@loige
we stream them...
10

HELLO, I AM LUCIANO! HELLO, I AM LUCIANO!
11

Cloud Architect
11

Cloud Architect
Blog:
Twitter:
GitHub:
loige.co
@loige
@lmammino
11

code: loige.link/streams-examples
loige.link/streams-kyiv
12

01. BUFFERS VS01. BUFFERS VS
STREAMS STREAMS
@loige13

BUFFERBUFFER: DATA STRUCTURE TO STORE AND: DATA STRUCTURE TO STORE AND
TRANSFER ARBITRARY BINARY DATATRANSFER ARBITRARY BINARY DATA
@loige
*Note that this is loading all the content of the ﬁle in memory
*
14

STREAMSTREAM: ABSTRACT INTERFACE FOR: ABSTRACT INTERFACE FOR
WORKING WITH STREAMING DATAWORKING WITH STREAMING DATA
@loige
*It does not load all the data straight away
*
15

FILE COPY: FILE COPY: THE BUFFER WAYTHE BUFFER WAY
@loige
// buffer-copy.js
const {
readFileSync,
writeFileSync
} = require('fs')
1
2
3
4
5
6
7
8
9
10
16

FILE COPY: FILE COPY: THE STREAM WAYTHE STREAM WAY
// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
const srcStream = createReadStream(src)
const destStream = createWriteStream(dest)
srcStream.on('data', (data) => destStream.write(data))
1
2
3
4
5
6
7
8
9
10
11
@loige
* Careful: this implementation is not optimal
*
17

// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
1
2
3
4
5
6
7
8
9
10
11
createReadStream,
createWriteStream
// stream-copy.js1
2
const {3
4
5
} = require('fs')6
7
const srcStream = createReadStream(src)9
const destStream = createWriteStream(dest)10
srcStream.on('data', (data) => destStream.write(data))11
@loige
*
17

// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
1
2
3
4
5
6
7
8
9
10
11
createReadStream,
createWriteStream
// stream-copy.js1
2
const {3
4
5
} = require('fs')6
7
// stream-copy.js1
2
const {3
createReadStream,4
createWriteStream5
} = require('fs')6
7
9
10
@loige
*
17

// stream-copy.js
const {
createReadStream,
createWriteStream
} = require('fs')
1
2
3
4
5
6
7
8
9
10
11
createReadStream,
createWriteStream
// stream-copy.js1
2
const {3
4
5
} = require('fs')6
7
// stream-copy.js1
2
const {3
createReadStream,4
createWriteStream5
} = require('fs')6
7
9
10
srcStream.on('data', (data) => destStream.write(data))11 srcStream.on('data', (data) => destStream.write(data))
// stream-copy.js1
2
const {3
createReadStream,4
createWriteStream5
} = require('fs')6
7
11
@loige
*
17

MEMORY COMPARISON (~600MB FILE)MEMORY COMPARISON (~600MB FILE)
node inspectbrk buffercopy.js assets/poster.psd ~/Downloads/poster.psd
@loige18

MEMORY COMPARISON (~600MB FILE)MEMORY COMPARISON (~600MB FILE)
node inspectbrk streamcopy.js assets/poster.psd ~/Downloads/poster.psd
@loige19

LET'S TRY WITH A BIG FILE (~10GB)LET'S TRY WITH A BIG FILE (~10GB)
@loige20

LET'S TRY WITH A BIG FILE (~10GB)LET'S TRY WITH A BIG FILE (~10GB)
node inspectbrk streamcopy.js assets/thematrixhd.mkv ~/Downloads/thematrixhd.mkv
@loige21

STREAMS VS BUFFERS STREAMS VS BUFFERS
Streams keep a low memory footprint
even with large amounts of data

Streams allows you to process data as
soon as it arrives
@loige22

03. STREAM TYPES03. STREAM TYPES
& APIS & APIS
@loige23

ALL STREAMS ARE ALL STREAMS ARE EVENT EMITTERSEVENT EMITTERS
A stream instance is an object that emits events when its internal
state changes, for instance:
s.on('readable', () => {}) // ready to be consumed
s.on('data', (chunk) => {}) // new data is available
s.on('error', (err) => {}) // some error happened
s.on('end', () => {}) // no more data available
The events available depend from the type of stream
@loige24

READABLEREADABLE STREAMS STREAMS
A readable stream represents a source from which data is consumed.
Examples:
fs readStream
process.stdin
HTTP response (client-side)
HTTP request (server-side)
AWS S3 GetObject (data field)
It supports two modes for data consumption: flowing and paused (or non-
flowing) mode.
@loige25

READABLE STREAMSREADABLE STREAMS
Data is read from source automatically and chunks are emitted as soon
as they are available.
@loige26

@loige
1
2
3
Source data
Readable stream in
ﬂowing mode
data listener
27

@loige
12
3
Source data
Readable stream in
ﬂowing mode
Read
data listener
28

@loige
12
3
Source data
Readable stream in
ﬂowing mode
data listener
data
29

@loige
2
3
Source data
Readable stream in
ﬂowing mode
data listener
Read
30

@loige
2
3
Source data
Readable stream in
ﬂowing mode
data listener
data
31

@loige
3
Source data
Readable stream in
ﬂowing mode
data listener
Read
32

@loige
3
Source data
Readable stream in
ﬂowing mode
data listener
data
33

@loige
Source data
Readable stream in
ﬂowing mode
Read
data listener
(end)
34

@loige
Source data
Readable stream in
ﬂowing mode
data listener
end
(end)
When no more data is available, end is emitted.
35

// count-emojis-flowing.js
const { createReadStream } = require('fs')
const { EMOJI_MAP } = require('emoji') // from npm
const emojis = Object.keys(EMOJI_MAP)
const file = createReadStream(process.argv[2])
let counter = 0
file.on('data', chunk => {
for (let char of chunk.toString('utf8')) {
if (emojis.includes(char)) {
counter++
}
}
})
file.on('end', () => console.log(`Found ${counter} emojis`))
file.on('error', err => console.error(`Error reading file: ${err}`))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
@loige36

let counter = 0
counter++
}
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// count-emojis-flowing.js1
2
const { createReadStream } = require('fs')3
4
5
const emojis = Object.keys(EMOJI_MAP)6
7
const file = createReadStream(process.argv[2])8
let counter = 09
10
file.on('data', chunk => {11
for (let char of chunk.toString('utf8')) {12
if (emojis.includes(char)) {13
counter++14
}15
}16
})17
file.on('end', () => console.log(`Found ${counter} emojis`))18
file.on('error', err => console.error(`Error reading file: ${err}`))19
@loige36

let counter = 0
counter++
}
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
4
5
7
let counter = 09
10
counter++14
}15
}16
})17
2
const { EMOJI_MAP } = require('emoji') // from npm4
5
7
8
let counter = 09
10
counter++14
}15
}16
})17
@loige36

let counter = 0
counter++
}
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
4
5
7
let counter = 09
10
counter++14
}15
}16
})17
2
5
7
8
let counter = 09
10
counter++14
}15
}16
})17
let counter = 0
2
5
7
9
10
counter++14
}15
}16
})17
@loige36

let counter = 0
counter++
}
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
4
5
7
let counter = 09
10
counter++14
}15
}16
})17
2
5
7
8
let counter = 09
10
counter++14
}15
}16
})17
let counter = 0
2
5
7
9
10
counter++14
}15
}16
})17
})
2
5
7
let counter = 09
10
11
counter++14
}15
}16
17
@loige36

let counter = 0
counter++
}
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
4
5
7
let counter = 09
10
counter++14
}15
}16
})17
2
5
7
8
let counter = 09
10
counter++14
}15
}16
})17
let counter = 0
2
5
7
9
10
counter++14
}15
}16
})17
})
2
5
7
let counter = 09
10
11
counter++14
}15
}16
17
}
2
5
7
let counter = 09
10
12
counter++14
}15
16
})17
@loige36

let counter = 0
counter++
}
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
4
5
7
let counter = 09
10
counter++14
}15
}16
})17
2
5
7
8
let counter = 09
10
counter++14
}15
}16
})17
let counter = 0
2
5
7
9
10
counter++14
}15
}16
})17
})
2
5
7
let counter = 09
10
11
counter++14
}15
}16
17
}
2
5
7
let counter = 09
10
12
counter++14
}15
16
})17
counter++
}
2
5
7
let counter = 09
10
13
14
15
}16
})17
@loige36

loige.link/up-emojiart
@loige37

READABLE STREAMS AREREADABLE STREAMS ARE
ALSO ALSO ASYNC ITERATORSASYNC ITERATORS
((NODE.JS 10+)NODE.JS 10+)
@loige38

// count-emojis-async-iterator.js
async function main () {
let counter = 0
for await (let chunk of file) {
counter++
}
}
}
console.log(`Found ${counter} emojis`)
}
main()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 @loige39

let counter = 0
counter++
}
}
}
}
main()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
}
// count-emojis-async-iterator.js1
4
async function main () {5
let counter = 08
9
10
counter++13
}14
}15
16
17
console.log(`Found ${counter} emojis`)18
}19
20
main()21 @loige39

let counter = 0
counter++
}
}
}
}
main()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
}
4
async function main () {5
let counter = 08
9
10
counter++13
}14
}15
16
17
}19
20
main()21
}
4
5
let counter = 08
9
for await (let chunk of file) {10
counter++13
}14
}15
}16
17
19
20
main()21 @loige39

WRITABLEWRITABLE STREAMS STREAMS
A writable stream is an abstraction that allows to write data over a destination

Examples:
fs writeStream
process.stdout, process.stderr
HTTP request (client-side)
HTTP response (server-side)
AWS S3 PutObject (body parameter)
@loige40

// writable-http-request.js
const http = require('http')
const req = http.request(
{
hostname: 'enx6b07hdu6cs.x.pipedream.net',
method: 'POST'
},
resp => {
console.log(`Server responded with "${resp.statusCode}"`)
}
)
req.on('finish', () => console.log('request sent'))
req.on('close', () => console.log('Connection closed'))
req.on('error', err => console.error(`Request failed: ${err}`))
req.write('writing some content...n')
req.end('last write & close the stream')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
@loige41

{
method: 'POST'
},
resp => {
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
)
// writable-http-request.js1
const http = require('http')2
3
4
{5
hostname: 'enx6b07hdu6cs.x.pipedream.net',6
method: 'POST'7
},8
resp => {9
console.log(`Server responded with "${resp.statusCode}"`)10
}11
12
13
req.on('finish', () => console.log('request sent'))14
req.on('close', () => console.log('Connection closed'))15
req.on('error', err => console.error(`Request failed: ${err}`))16
17
req.write('writing some content...n')18
req.end('last write & close the stream')19
@loige41

{
method: 'POST'
},
resp => {
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
)
3
4
{5
method: 'POST'7
},8
resp => {9
}11
12
13
17
3
const req = http.request(4
{5
method: 'POST'7
},8
resp => {9
}11
)12
13
14
15
16
17
@loige41

{
method: 'POST'
},
resp => {
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
)
3
4
{5
method: 'POST'7
},8
resp => {9
}11
12
13
17
3
{5
method: 'POST'7
},8
resp => {9
}11
)12
13
14
15
16
17
3
{5
method: 'POST'7
},8
resp => {9
}11
)12
13
17
18
19
@loige41

loige.link/writable-http-req
@loige43

BACKPRESSUREBACKPRESSURE
When writing large amounts of data you
should make sure you handle the stop write
signal and the drain event

loige.link/backpressure
@loige44

// stream-copy-safe.js
const { createReadStream, createWriteStream } = require('fs')
const [, , src, dest] = process.argv
srcStream.on('data', data => {
const canContinue = destStream.write(data)
if (!canContinue) {
// we are overflowing the destination, we should pause
srcStream.pause()
// we will resume when the destination stream is drained
destStream.once('drain', () => srcStream.resume())
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
@loige45

if (!canContinue) {
srcStream.pause()
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// stream-copy-safe.js1
2
const { createReadStream, createWriteStream } = require('fs')3
4
const [, , src, dest] = process.argv5
8
srcStream.on('data', data => {9
10
if (!canContinue) {11
// we are overflowing the destination, we should pause12
srcStream.pause()13
// we will resume when the destination stream is drained14
destStream.once('drain', () => srcStream.resume())15
}16
})17
@loige45

if (!canContinue) {
srcStream.pause()
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
4
8
10
srcStream.pause()13
}16
})17
if (!canContinue) {
}
2
4
8
const canContinue = destStream.write(data)10
11
srcStream.pause()13
16
})17
@loige45

if (!canContinue) {
srcStream.pause()
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
4
8
10
srcStream.pause()13
}16
})17
if (!canContinue) {
}
2
4
8
11
srcStream.pause()13
16
})17
srcStream.pause()
2
4
8
13
}16
})17
@loige45

if (!canContinue) {
srcStream.pause()
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
4
8
10
srcStream.pause()13
}16
})17
if (!canContinue) {
}
2
4
8
11
srcStream.pause()13
16
})17
srcStream.pause()
2
4
8
13
}16
})17
2
4
8
srcStream.pause()13
15
}16
})17
@loige45

OTHER TYPES OF STREAMOTHER TYPES OF STREAM
Duplex Stream
streams that are both Readable and Writable.
(net.Socket)

Transform Stream
Duplex streams that can modify or transform the data as it is written
and read.
(zlib.createGzip(), crypto.createCipheriv())
@loige46

ANATOMY OF A TRANSFORM STREAMANATOMY OF A TRANSFORM STREAM
transform stream
@loige47

1. write data
transform stream
(readable stream)
@loige47

1. write data
transform stream
2. transform the data
(readable stream)
@loige47

1. write data
transform stream
3. read transformed data2. transform the data
(readable stream) (writable stream)
@loige47

GZIP EXAMPLEGZIP EXAMPLE
1. write data
transform stream
@loige48

1. write data
transform stream
@loige
Uncompressed data
48

1. write data
transform stream
@loige
Uncompressed data
compress
zlib.createGzip()
48

1. write data
transform stream
@loige
Uncompressed data Compressed data
compress
zlib.createGzip()
48

HOW CAN WE USE TRANSFORM STREAMS?HOW CAN WE USE TRANSFORM STREAMS?
Readable Transform Writable
49@loige

⚡ data
49@loige

⚡ data write()
49@loige

⚡ data write() ⚡ data
49@loige

⚡ data write() ⚡ data write()
49@loige

49@loige
(Backpressure)

pause()
49@loige
(Backpressure)

pause()
⚡ drain
49@loige
(Backpressure)

pause()
⚡ drainresume()
49@loige
(Backpressure)

pause()
⚡ drainresume()
49@loige
(Backpressure) (Backpressure)

pause()
⚡ drainresume()
pause()
49@loige

pause()
⚡ drainresume()
pause()
⚡ drain
49@loige

pause()
⚡ drainresume()
pause()
⚡ drainresume()
49@loige

pause()
⚡ drainresume()
pause()
⚡ drainresume()
49@loige
You also have to handle end events and errors!

gzipStream.on('data', data => {
if (!canContinue) {
gzipStream.pause()
destStream.once('drain', () => {
gzipStream.resume()
})
}
})
gzipStream.on('end', () => {
destStream.end()
})
// ⚠ TODO: handle errors!
// stream-copy-gzip.js
const {
createReadStream,
createWriteStream
} = require('fs')
const { createGzip } = require('zlib')
const gzipStream = createGzip()
const canContinue = gzipStream.write(data)
if (!canContinue) {
srcStream.pause()
gzipStream.once('drain', () => {
srcStream.resume()
})
}
})
srcStream.on('end', () => {
// check if there's buffered data left
const remainingData = gzipStream.read()
if (remainingData !== null) {
destStream.write()
}
gzipStream.end()
})
@loige50

03. PIPE()03. PIPE()
@loige51

readable
.pipe(tranform1)
.pipe(transform2)
.pipe(transform3)
.pipe(writable)
readable.pipe(writableDest)
@loige
Connects a readable stream to a writable stream
A transform stream can be used as a destination as well
It returns the destination stream allowing for a chain of pipes
52

// stream-copy-gzip-pipe.js
const {
createReadStream,
createWriteStream
} = require('fs')
srcStream
.pipe(gzipStream)
.pipe(destStream)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@loige53

// stream-copy-gzip-pipe.js
const {
createReadStream,
createWriteStream
} = require('fs')
srcStream
.pipe(gzipStream)
.pipe(destStream)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
srcStream
.pipe(gzipStream)
.pipe(destStream)
// stream-copy-gzip-pipe.js1
2
const {3
createReadStream,4
createWriteStream5
} = require('fs')6
const { createGzip } = require('zlib')7
8
const gzipStream = createGzip()11
13
14
15
16
@loige53

readable
.pipe(decompress)
.pipe(decrypt)
.pipe(convert)
.pipe(encrypt)
.pipe(compress)
.pipe(writeToDisk)
Setup complex pipelines with pipe
@loige
This is the most common way to use streams
54

readable
.on('error', handleErr)
.pipe(decompress)
.pipe(decrypt)
.pipe(convert)
.pipe(encrypt)
.pipe(compress)
.pipe(writeToDisk)
Handling errors (correctly)
@loige55

readable
.pipe(decompress)
.pipe(decrypt)
.pipe(convert)
.pipe(encrypt)
.pipe(compress)
.pipe(writeToDisk)
Handling errors (correctly)
@loige

handleErr should end and destroy the streams
(it doesn't happen automatically)

55

04. STREAM UTILITIES04. STREAM UTILITIES
@loige56

// stream-copy-gzip-pipeline.js
const { pipeline } = require('stream')
pipeline(
createReadStream(src),
createGzip(),
createWriteStream(dest),
function onEnd (err) {
if (err) {
console.error(`Error: ${err}`)
process.exit(1)
}
console.log('Done!')
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
stream.pipeline(...streams, callback) - Node.js 10+
@loige57

pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline(
)
// stream-copy-gzip-pipeline.js1
2
const { pipeline } = require('stream')3
6
8
9
createReadStream(src),10
createGzip(),11
createWriteStream(dest),12
function onEnd (err) {13
if (err) {14
console.error(`Error: ${err}`)15
process.exit(1)16
}17
18
console.log('Done!')19
}20
21
@loige57

pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline(
)
2
6
8
9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
21
createGzip(),
2
6
8
pipeline(9
10
11
12
if (err) {14
process.exit(1)16
}17
18
}20
)21
@loige57

pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline(
)
2
6
8
9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
21
createGzip(),
2
6
8
pipeline(9
10
11
12
if (err) {14
process.exit(1)16
}17
18
}20
)21
}
2
6
8
pipeline(9
createGzip(),11
13
if (err) {14
process.exit(1)16
}17
18
20
)21
@loige57

pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline(
)
2
6
8
9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
21
createGzip(),
2
6
8
pipeline(9
10
11
12
if (err) {14
process.exit(1)16
}17
18
}20
)21
}
2
6
8
pipeline(9
createGzip(),11
13
if (err) {14
process.exit(1)16
}17
18
20
)21
if (err) {
process.exit(1)
}
2
6
8
pipeline(9
createGzip(),11
14
15
16
17
18
}20
)21
@loige57

pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline(
)
2
6
8
9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
21
createGzip(),
2
6
8
pipeline(9
10
11
12
if (err) {14
process.exit(1)16
}17
18
}20
)21
}
2
6
8
pipeline(9
createGzip(),11
13
if (err) {14
process.exit(1)16
}17
18
20
)21
if (err) {
process.exit(1)
}
2
6
8
pipeline(9
createGzip(),11
14
15
16
17
18
}20
)21
2
6
8
pipeline(9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
19
}20
)21
@loige57

pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pipeline(
)
2
6
8
9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
21
createGzip(),
2
6
8
pipeline(9
10
11
12
if (err) {14
process.exit(1)16
}17
18
}20
)21
}
2
6
8
pipeline(9
createGzip(),11
13
if (err) {14
process.exit(1)16
}17
18
20
)21
if (err) {
process.exit(1)
}
2
6
8
pipeline(9
createGzip(),11
14
15
16
17
18
}20
)21
2
6
8
pipeline(9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
19
}20
)21
pipeline(
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@loige
You can pass multiple streams (they will be piped)
The last argument is a callback. If invoked with an
error, it means the pipeline failed at some point.
All the streams are ended and destroyed correctly.
57

// stream-copy-gzip-pump.js
const pump = require('pump') // from npm
pump( // just swap pipeline with pump!
createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
For Node.js < 10: pump - npm.im/pump
@loige58

createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// stream-copy-gzip-pump.js1
2
3
6
8
pump( // just swap pipeline with pump!9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
)21
@loige58

createGzip(),
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
3
6
8
pump( // just swap pipeline with pump!9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
)21
)
2
const pump = require('pump') // from npm3
6
8
9
createGzip(),11
if (err) {14
process.exit(1)16
}17
18
}20
21
@loige58

pumpify(...streams) -
Create reusable pieces of pipeline
npm.im/pumpify
@loige
Let's create EncGz, an application that
helps us to read and write encrypted-
gzipped ﬁles
59

// encgz-stream.js - utility library
const {
createCipheriv,
createDecipheriv,
randomBytes,
createHash
} = require('crypto')
const { createGzip, createGunzip } = require('zlib')
const pumpify = require('pumpify') // from npm
// calculates md5 of the secret (trimmed)
function getChiperKey (secret) {}
function createEncgz (secret) {
const initVect = randomBytes(16)
const cipherKey = getChiperKey(secret)
const encryptStream = createCipheriv('aes256', cipherKey, initVect)
const stream = pumpify(encryptStream, gzipStream)
stream.initVect = initVect
return stream
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
@loige60

const {
createCipheriv,
createDecipheriv,
randomBytes,
createHash
return stream
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
}
// encgz-stream.js - utility library1
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
} = require('crypto')8
const { createGzip, createGunzip } = require('zlib')9
const pumpify = require('pumpify') // from npm10
11
// calculates md5 of the secret (trimmed)12
function getChiperKey (secret) {}13
14
15
const initVect = randomBytes(16)16
const cipherKey = getChiperKey(secret)17
const encryptStream = createCipheriv('aes256', cipherKey, initVect)18
20
const stream = pumpify(encryptStream, gzipStream)21
stream.initVect = initVect22
23
return stream24
25
@loige60

const {
createCipheriv,
createDecipheriv,
randomBytes,
createHash
return stream
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
}
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
15
20
23
return stream24
25
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
function createEncgz (secret) {15
18
19
20
23
return stream24
}25
@loige60

const {
createCipheriv,
createDecipheriv,
randomBytes,
createHash
return stream
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
}
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
15
20
23
return stream24
25
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
18
19
20
23
return stream24
}25
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
20
21
23
return stream24
}25
@loige60

const {
createCipheriv,
createDecipheriv,
randomBytes,
createHash
return stream
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
}
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
15
20
23
return stream24
25
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
18
19
20
23
return stream24
}25
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
20
21
23
return stream24
}25
return stream
2
const {3
createCipheriv,4
createDecipheriv,5
randomBytes,6
createHash7
11
14
20
23
24
}25
@loige60

// encgz-stream.js (...continue from previous slide)
function createDecgz (secret, initVect) {
const decryptStream = createDecipheriv('aes256', cipherKey, initVect)
const gunzipStream = createGunzip()
const stream = pumpify(gunzipStream, decryptStream)
return stream
}
module.exports = {
createEncgz,
createDecgz
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@loige61

return stream
}
module.exports = {
createEncgz,
createDecgz
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
}
// encgz-stream.js (...continue from previous slide)1
2
3
const decryptStream = createDecipheriv('aes256', cipherKey, initVect)5
const gunzipStream = createGunzip()6
7
const stream = pumpify(gunzipStream, decryptStream)8
return stream9
10
11
module.exports = {12
createEncgz,13
createDecgz14
}15
@loige61

return stream
}
module.exports = {
createEncgz,
createDecgz
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
}
2
3
7
return stream9
10
11
createEncgz,13
createDecgz14
}15
2
function createDecgz (secret, initVect) {3
5
6
7
return stream9
}10
11
createEncgz,13
createDecgz14
}15
@loige61

return stream
}
module.exports = {
createEncgz,
createDecgz
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
}
2
3
7
return stream9
10
11
createEncgz,13
createDecgz14
}15
2
5
6
7
return stream9
}10
11
createEncgz,13
createDecgz14
}15
return stream
2
7
8
9
}10
11
createEncgz,13
createDecgz14
}15
@loige61

return stream
}
module.exports = {
createEncgz,
createDecgz
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
}
2
3
7
return stream9
10
11
createEncgz,13
createDecgz14
}15
2
5
6
7
return stream9
}10
11
createEncgz,13
createDecgz14
}15
return stream
2
7
8
9
}10
11
createEncgz,13
createDecgz14
}15
module.exports = {
createEncgz,
createDecgz
}
2
7
return stream9
}10
11
12
13
14
15
@loige61

// encgz.js - CLI to encrypt and gzip (from stdin to stdout)
const { createEncgz } = require('./encgz-stream')
const [, , secret] = process.argv
const encgz = createEncgz(secret)
console.error(`init vector: ${encgz.initVect.toString('hex')}`)
pipeline(
process.stdin,
encgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 @loige62

pipeline(
process.stdin,
encgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// encgz.js - CLI to encrypt and gzip (from stdin to stdout)1
2
const { createEncgz } = require('./encgz-stream')4
5
6
7
const encgz = createEncgz(secret)8
console.error(`init vector: ${encgz.initVect.toString('hex')}`)9
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21 @loige62

pipeline(
process.stdin,
encgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
5
6
7
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21
2
5
const [, , secret] = process.argv6
7
8
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21 @loige62

pipeline(
process.stdin,
encgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
5
6
7
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21
2
5
7
8
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21
pipeline(
)
2
5
7
10
11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
21 @loige62

pipeline(
process.stdin,
encgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
5
6
7
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21
2
5
7
8
10
pipeline(11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
)21
pipeline(
)
2
5
7
10
11
process.stdin,12
encgz,13
process.stdout,14
if (err) {16
process.exit(1)18
}19
}20
21
process.stdin,
encgz,
process.stdout,
2
5
7
10
pipeline(11
12
13
14
if (err) {16
process.exit(1)18
}19
}20
)21 @loige62

// decgz.js - CLI to gunzip and decrypt (from stdin to stdout)
const { createDecgz } = require('./encgz-stream')
const [, , secret, initVect] = process.argv
const decgz = createDecgz(secret, Buffer.from(initVect, 'hex'))
pipeline(
process.stdin,
decgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@loige63

pipeline(
process.stdin,
decgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// decgz.js - CLI to gunzip and decrypt (from stdin to stdout)1
2
const { createDecgz } = require('./encgz-stream')4
5
6
7
const decgz = createDecgz(secret, Buffer.from(initVect, 'hex'))8
9
pipeline(10
process.stdin,11
decgz,12
process.stdout,13
if (err) {15
process.exit(1)17
}18
}19
)20
@loige63

pipeline(
process.stdin,
decgz,
process.stdout,
if (err) {
process.exit(1)
}
}
)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
5
6
7
const decgz = createDecgz(secret, Buffer.from(initVect, 'hex'))8
9
pipeline(10
process.stdin,11
decgz,12
process.stdout,13
if (err) {15
process.exit(1)17
}18
}19
)20
2
5
const [, , secret, initVect] = process.argv6
7
8
9
pipeline(10
process.stdin,11
decgz,12
process.stdout,13
if (err) {15
process.exit(1)17
}18
}19
)20
@loige63

"It’s about time to embrace Streams" Luciano Mammino

"It’s about time to embrace Streams" Luciano Mammino

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to "It’s about time to embrace Streams" Luciano Mammino

Similar to "It’s about time to embrace Streams" Luciano Mammino (20)

More from Julia Cherniak

More from Julia Cherniak (11)

Recently uploaded

Recently uploaded (20)

"It’s about time to embrace Streams" Luciano Mammino