From Node.js to Design Patterns - BuildPiper

$ ~ whoami
👋I'm Luciano ( 🍕🍝 )
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
📔Co-Author of Node.js Design Patterns 👉
Let's connect!
(blog)
(twitter)
(twitch)
(github)
loige.co
@loige
loige
lmammino 2

Always re-imagining
We are a pioneering technology consultancy focused
on AWS and serverless
| |
Accelerated Serverless AI as a Service Platform Modernisation
loige
✉Reach out to us at
😇We are always looking for talent:
hello@fourTheorem.com
fth.link/careers
3

We host a weekly podcast about AWS
awsbites.com
loige 4

Fact: Async JavaScript is tricky!
callbacks
promises
Async/Await
async
generators
streams
event emitters
util.promisify()
Promise.all()
Promise.allSettled()
😱
loige 5

Agenda
Async WUT?!
Callbacks
Promises
Async / Await
async Patterns
Mixed style async
A performance trick!
loige 6

What does async even mean?
In JavaScript and in Node.js, input/output operations are non-
blocking.
Classic examples: reading the content of a file, making an HTTP
request, loading data from a database, etc.
loige 7

Blocking style vs JavaScript
Blocking style JavaScript
1. Assign a variable
2. Read data from a file
3. Print to stdout
3. Print to stdout
loige 8

3. Print to stdout
3. Print to stdout
loige 9

3. Print to stdout
3. Print to stdout
loige 10

3. Print to stdout
3. Print to stdout
loige 11

3. Print to stdout
3. Print to stdout
(done)
loige 12

3. Print to stdout
3. Print to stdout
(done)
(done)
loige 13

Non-blocking I/O is convenient:
you can do work while waiting for I/O!
But, what if we need to do something when
the I/O operation completes?
loige 14

Once upon a time there were...
Callbacks
loige 15

Anatomy of callback-based non-blocking code
doSomethingAsync(arg1, arg2, cb)
This is a callback
loige 16

doSomethingAsync(arg1, arg2, (err, data) => {
// ... do something with data
})
You are defining what happens when the I/O operations
completes (or fails) with a function.
doSomethingAsync will call that function for you!
loige
17

doSomethingAsync(arg1, arg2, (err, data) => {
if (err) {
// ... handle error
return
}
})
Always handle errors first!
loige
18

An example
Fetch the latest booking for a given user
If it exists print it
loige 19

getLatestBooking(userId, (err, booking) => {
if (err) {
console.error(err)
return
}
if (booking) {
console.log(`Found booking for user ${userId}`, booking)
} else {
console.log(`No booking found for user ${userId}`)
}
})
1
2
3
4
5
6
7
8
9
10
11
12
An example
loige 20

A more realistic example
If it exists, cancel it
If it was already paid for, refund the user
loige 21

if (err) {
console.error(err)
return
}
if (booking) {
cancelBooking(booking.id, (err) => {
if (err) {
console.error(err)
return
}
if (booking.paid) {
console.log('Booking was paid, refunding the user')
refundUser(userId, booking.paidAmount, (err) => {
if (err) {
console.error(err)
return
}
console.log('User refunded')
})
}
})
} else {
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
if (err) {
console.error(err)
return
}
if (booking.paid) {
if (err) {
console.error(err)
return
}
})
}
})
1
if (err) {
2
console.error(err)
3
return
4
}
5
6
if (booking) {
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
} else {
27
28
}
29
})
30
if (booking.paid) {
if (err) {
console.error(err)
return
}
})
}
1
if (err) {
2
console.error(err)
3
return
4
}
5
6
if (booking) {
7
8
9
if (err) {
10
console.error(err)
11
return
12
}
13
14
15
16
17
18
19
20
21
22
23
24
25
})
26
} else {
27
28
}
29
})
30
if (err) {
console.error(err)
return
}
})
1
if (err) {
2
console.error(err)
3
return
4
}
5
6
if (booking) {
7
8
9
if (err) {
10
console.error(err)
11
return
12
}
13
14
if (booking.paid) {
15
16
17
18
19
20
21
22
23
24
}
25
})
26
} else {
27
28
}
29
})
30 loige 22

if (err) {
console.error(err)
return
}
if (booking) {
if (err) {
console.error(err)
return
}
if (booking.paid) {
if (err) {
console.error(err)
return
}
})
}
})
} else {
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30 loige 23

if (err) {
console.error(err)
return
}
if (booking) {
if (err) {
console.error(err)
return
}
if (booking.paid) {
if (err) {
console.error(err)
return
}
})
}
})
} else {
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30 loige
THE PIRAMID OF
DOOM
(or callback hell 🔥)
24

Some times, just
refactoring the code
can help...
loige 25

function cancelAndRefundBooking(booking, cb) {
if (err) { return cb(err) }
if (!booking.paid) {
return cb(null, {refundedAmount: 0})
}
refundUser(booking.userId, booking.paidAmount, (err) => {
if (err) { return cb(err) }
return cb(null, {refundedAmount: booking.paidAmount})
})
})
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
loige 26

if (err) {
console.error(err)
return
}
if (booking) {
cancelAndRefundBooking(booking, (err, result) => {
if (err) {
console.error(err)
return
}
console.log(`Booking cancelled (${result.refundedAmount} refunded)`)
})
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
loige 27

😟
Is this the best we can do?
loige 28

Let's talk about
Promise
loige 29

With callbacks we are not in
charge!
We need to trust that the async function will call our
callbacks when the async work is completed!
loige 30

Promise help us to be more in control!
const promiseObj = doSomethingAsync(arg1, arg2)
An object that represents the
status of the async operation
loige 31

A promise object is a tiny state machine with 2 possible states
pending (still performing the async operation)
settled (completed)
✅fullfilled (witha value)
🔥rejected (with an error)
loige
32

promiseObj.then((data) => {
})
loige
33

promiseObj.then((data) => {
})
promiseObj.catch((err) => {
// ... handle errors
}
loige
34

Promises can be chained ⛓
This solves the pyramid of doom problem!
doSomethingAsync(arg1, arg2)
.then((result) => doSomethingElseAsync(result))
.then((result) => doEvenMoreAsync(result)
.then((result) => keepDoingStuffAsync(result))
.catch((err) => { /* ... */ })
35
loige

Promises can be chained ⛓
This solves the pyramid of doom problem!
doSomethingAsync(arg1, arg2)
.then((result) => doSomethingElseAsync(result))
// ...
.catch((err) => { /* ... */ })
.finally(() => { /* ... */ })
loige 36

How to create a promise
new Promise ((resolve, reject) => {
// ...
})
loige 37

new Promise ((resolve, reject) => {
// ... do something async
// reject(someError)
// resolve(someValue)
})
loige 38

Promise.resolve('SomeValue')
Promise.reject(new Error('SomeError'))
loige 39

How to create a promise (example)
function queryDB(client, query) {
return new Promise((resolve, reject) => {
client.executeQuery(query, (err, data) => {
if (err) {
return reject(err)
}
resolve(data)
})
})
}
1
2
3
4
5
6
7
8
9
10
11
loige 40

How to create a promise (example)
queryDB(dbClient, 'SELECT * FROM bookings')
.then((data) => {
})
.catch((err) => {
console.error('Failed to run query', err)
})
.finally(() => {
dbClient.disconnect()
})
1
2
3
4
5
6
7
8
9
10
1
.then((data) => {
2
3
})
4
.catch((err) => {
5
6
})
7
.finally(() => {
8
9
})
10
.then((data) => {
})
1
2
3
4
.catch((err) => {
5
6
})
7
.finally(() => {
8
9
})
10
.catch((err) => {
})
1
.then((data) => {
2
3
})
4
5
6
7
.finally(() => {
8
9
})
10
.finally(() => {
})
1
.then((data) => {
2
3
})
4
.catch((err) => {
5
6
})
7
8
9
10
.then((data) => {
})
.catch((err) => {
})
.finally(() => {
})
1
2
3
4
5
6
7
8
9
10
loige 41

Let's re-write our example with Promise
loige 42

getLatestBooking(userId)
.then((booking) => {
if (booking) {
return cancelBooking(booking.id)
}
})
.then((cancelledBooking) => {
if (cancelledBooking && cancelledBooking.paid) {
return refundUser(userId, cancelledBooking.paidAmount)
}
})
.then((refund) => {
if (refund) {
}
})
.catch((err) => {
console.error(err)
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
1
2
if (booking) {
3
4
5
}
6
7
})
8
9
10
11
12
}
13
})
14
.then((refund) => {
15
if (refund) {
16
17
}
18
})
19
.catch((err) => {
20
console.error(err)
21
})
22
if (booking) {
}
})
1
2
3
4
5
6
7
8
9
10
11
12
}
13
})
14
.then((refund) => {
15
if (refund) {
16
17
}
18
})
19
.catch((err) => {
20
console.error(err)
21
})
22
}
})
1
2
if (booking) {
3
4
5
}
6
7
})
8
9
10
11
12
13
14
.then((refund) => {
15
if (refund) {
16
17
}
18
})
19
.catch((err) => {
20
console.error(err)
21
})
22
.then((refund) => {
if (refund) {
}
})
1
2
if (booking) {
3
4
5
}
6
7
})
8
9
10
11
12
}
13
})
14
15
16
17
18
19
.catch((err) => {
20
console.error(err)
21
})
22
.catch((err) => {
console.error(err)
})
1
2
if (booking) {
3
4
5
}
6
7
})
8
9
10
11
12
}
13
})
14
.then((refund) => {
15
if (refund) {
16
17
}
18
})
19
20
21
22
if (booking) {
}
})
}
})
.then((refund) => {
if (refund) {
}
})
.catch((err) => {
console.error(err)
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
loige 43

enters...
Async/Await
loige 44

Sometimes, we just want to wait for a promise to
resolve before executing the next line...
const data = await promiseObj
// ... process the data
await allows us to do exactly that
loige 45

const data = await doSomethingAsync(arg1, arg2)
We don't have to assign the promise to a variable to use await
Sometimes, we just want to wait for a promise to
resolve before executing the next line...
loige 46

try {
const data = await doSomethingAsync(arg1, arg2)
} catch (err) {
// ... handle error
}
Unified error handling
If we await a promise that eventually rejects we can capture the error with a regular try/catch block
loige 47

Async functions
async function doSomethingAsync(arg1, arg2) {
// ...
}
special keyword that marks a function as async
loige 48

Async functions
return 'SomeValue'
}
function doSomethingAsync(arg1, arg2) {
return Promise.resolve('SomeValue')
}
loige 49

Async functions
throw new Error('SomeError')
}
function doSomethingAsync(arg1, arg2) {
return Promise.reject(new Error('SomeError'))
}
loige 50

Async functions
const res1 = await doSomethingElseAsync()
const res2 = await doEvenMoreAsync(res1)
const res3 = await keepDoingStuffAsync(res2)
// ...
}
inside an async function you can use await to
suspend the execution until the awaited promise
resolves
loige 51

Async functions
const res = await doSomethingElseAsync()
if (res) {
for (const record of res1.records) {
await updateRecord(record)
}
}
}
Async functions make it very easy to write code
that manages asynchronous control flow
loige 52

Let's re-write our example with async/await
loige 53

async function cancelLatestBooking(userId) {
const booking = await getLatestBooking(userId)
if (!booking) {
return
}
await cancelBooking(booking.id)
if (booking.paid) {
await refundUser(userId, booking.paidAmount)
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
}
1
2
3
if (!booking) {
4
5
return
6
}
7
8
9
10
11
12
if (booking.paid) {
13
14
15
16
}
17
18
1
2
3
if (!booking) {
4
5
return
6
}
7
8
9
10
11
12
if (booking.paid) {
13
14
15
16
}
17
}
18
if (!booking) {
return
}
1
2
3
4
5
6
7
8
9
10
11
12
if (booking.paid) {
13
14
15
16
}
17
}
18
1
2
3
if (!booking) {
4
5
return
6
}
7
8
9
10
11
12
if (booking.paid) {
13
14
15
16
}
17
}
18
1
2
3
if (!booking) {
4
5
return
6
}
7
8
9
10
11
12
if (booking.paid) {
13
14
15
16
}
17
}
18
if (booking.paid) {
}
1
2
3
if (!booking) {
4
5
return
6
}
7
8
9
10
11
12
13
14
15
16
17
}
18
if (!booking) {
return
}
if (booking.paid) {
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
loige 54

Mini summary
Async/Await generally helps to keep the code simple &
readable
To use Async/Await you need to understand Promise
To use Promise you need to understand callbacks
callbacks → Promise → async/await
Don't skip any step of the async journey!
loige 55

Sequential execution
const users = ['Peach', 'Toad', 'Mario', 'Luigi']
for (const userId of users) {
await cancelLatestBooking(userId)
}
1
2
3
4
5
1
2
3
4
}
5
}
1
2
3
4
5
1
2
3
4
}
5
}
1
2
3
4
5
loige 57

Sequential execution (gotcha!)
users.forEach(async (userId) => {
})
1
2
3
4
5
loige
⚠Don't do this with Array.map() or Array.forEach()
Array.forEach() will run the provided function without
awaiting for the returned promise, so all the invocation will
actually happen concurrently!
58

Concurrent execution (Promise.all)
await Promise.all(
users.map(
userId => cancelLatestBooking(userId)
)
)
1
2
3
4
5
6
7
loige
Promise.all() receives a list of promises and it returns a
new Promise. This promise will resolve once all the original
promises resolve, but it will reject as soon as ONE promise
rejects
59

Concurrent execution (Promise.allSettled)
const results = await Promise.allSettled(
users.map(
userId => cancelLatestBooking(userId)
)
)
1
2
3
4
5
6
7
loige
[
{ status: 'fulfilled', value: true },
{ status: 'fulfilled', value: true },
{ status: 'rejected', reason: Error },
{ status: 'fulfilled', value: true }
]
60

You want to use async/await but...
you have a callback-based API! 😣
loige 62

Node.js offers promise-based alternative APIs
Callback-based Promise-based
setTimeout, setImmediate, setInterval import timers from 'timers/promises'
import fs from 'fs' import fs from 'fs/promises'
import stream from 'stream' import stream from 'stream/promises'
import dns from 'dns' import dns from 'dns/promises'
loige 63

util.promisify()
import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback)
import { promisify } from 'util'
const gzipPromise = promisify(gzip)
const compressed = await gzipPromise(Buffer.from('Hello from Node.js'))
console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00>
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
loige 64

Promisify by hand 🖐
function gzipPromise (buffer, options) {
gzip(buffer, options, (err, gzippedData) => {
if (err) {
return reject(err)
}
resolve(gzippedData)
})
})
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
}
1
2
3
4
5
if (err) {
6
return reject(err)
7
}
8
9
10
})
11
})
12
13
14
15
16
})
1
2
3
4
5
if (err) {
6
return reject(err)
7
}
8
9
10
})
11
12
}
13
14
15
16
})
1
2
3
4
5
if (err) {
6
return reject(err)
7
}
8
9
10
11
})
12
}
13
14
15
16
if (err) {
return reject(err)
}
1
2
3
4
5
6
7
8
9
10
})
11
})
12
}
13
14
15
16
1
2
3
4
5
if (err) {
6
return reject(err)
7
}
8
9
10
})
11
})
12
}
13
14
15
16
1
2
3
4
5
if (err) {
6
return reject(err)
7
}
8
9
10
})
11
})
12
}
13
14
15
16
if (err) {
return reject(err)
}
})
})
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
loige 65

What if we we want to do the opposite? 🤷
Convert a promise-based function to a callback-based one
loige 66

var env = nunjucks.configure('views')
env.addFilter('videoTitle', function(videoId, cb) {
// ... fetch the title through youtube APIs
// ... extract the video title
// ... and call the callback with the title
}, true)
1
2
3
4
5
6
7
OK, this is not a common use case, so let me give you a real
example!
Nunjucks async filters
{{ data | myCustomFilter }}
We are forced to pass a callback-based
function here!
Ex: {{ youtubeId | videoTitle }}
loige 67

util.callbackify()
import { callbackify } from 'util'
import Innertube from 'youtubei.js' // from npm
async function videoTitleFilter (videoId) {
const youtube = await new Innertube({ gl: 'US' })
const details = await youtube.getDetails(videoId)
return details.title
}
const videoTitleFilterCb = callbackify(videoTitleFilter)
videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => {
if (err) {
console.error(err)
return
}
console.log(videoTitle)
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
1
2
3
4
5
6
7
}
8
9
10
11
12
if (err) {
13
console.error(err)
14
return
15
}
16
17
18
})
19
1
2
3
4
5
6
7
}
8
9
10
11
12
if (err) {
13
console.error(err)
14
return
15
}
16
17
18
})
19
}
1
2
3
4
5
6
7
8
9
10
11
12
if (err) {
13
console.error(err)
14
return
15
}
16
17
18
})
19
1
2
3
4
5
6
7
}
8
9
10
11
12
if (err) {
13
console.error(err)
14
return
15
}
16
17
18
})
19
if (err) {
console.error(err)
return
}
})
1
2
3
4
5
6
7
}
8
9
10
11
12
13
14
15
16
17
18
19
}
if (err) {
console.error(err)
return
}
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 loige 68

Callbackify by hand ✋
// ...
}
function videoTitleFilterCb (videoId, cb) {
videoTitleFilter(videoId)
.then((videoTitle) => cb(null, videoTitle))
.catch(cb)
}
// ...
})
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function videoTitleFilterCb (videoId, cb) {
videoTitleFilter(videoId)
.then((videoTitle) => cb(null, videoTitle))
.catch(cb)
}
1
2
3
// ...
4
}
5
6
7
8
9
10
11
12
13
// ...
14
})
15
loige 69

OK, Cool!
But is this stuff worth it?
🧐
loige 70

Let me show you a cool performance
trick for Web Servers!
😎
loige 71

The request batching pattern
one user
/api/hotels/rome
Web server DB
loige 72

multiple users (no batching)
Web server DB
/api/hotels/rome
/api/hotels/rome
/api/hotels/rome
73
loige

multiple users (with batching!)
Web server DB
/api/hotels/rome
/api/hotels/rome
/api/hotels/rome
74
📘 Requests in-flight
/api/hotels/rome
✅
loige

The web server
import { createServer } from 'http'
const urlRegex = /^/api/hotels/([w-]+)$/
createServer(async (req, res) => {
const url = new URL(req.url, 'http://localhost')
const matches = urlRegex.exec(url.pathname)
if (!matches) {
res.writeHead(404, 'Not found')
return res.end()
}
const [_, city] = matches
const hotels = await getHotelsForCity(city)
res.writeHead(200)
res.end(JSON.stringify({ hotels }))
}).listen(8000)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
res.writeHead(200)
17
18
}).listen(8000)
19
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
res.writeHead(200)
17
18
}).listen(8000)
19
}).listen(8000)
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
res.writeHead(200)
17
18
19
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
res.writeHead(200)
17
18
}).listen(8000)
19
if (!matches) {
return res.end()
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
res.writeHead(200)
17
18
}).listen(8000)
19
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
res.writeHead(200)
17
18
}).listen(8000)
19
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
res.writeHead(200)
17
18
}).listen(8000)
19
res.writeHead(200)
1
2
3
4
5
6
7
8
if (!matches) {
9
10
return res.end()
11
}
12
13
14
15
16
17
18
}).listen(8000)
19
if (!matches) {
return res.end()
}
res.writeHead(200)
}).listen(8000)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
loige 75

The data fetching function (with batching)
let pendingRequests = new Map()
function getHotelsForCity (cityId) {
if (pendingRequests.has(cityId)) {
return pendingRequests.get(cityId)
}
const asyncOperation = db.query({
text: 'SELECT * FROM hotels WHERE cityid = $1',
values: [cityId],
})
pendingRequests.set(cityId, asyncOperation)
asyncOperation.finally(() => {
pendingRequests.delete(cityId)
})
return asyncOperation
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20 loige 76

Benchmarks
loige.link/req-batch-bench
Without request batching With request batching (+90% avg req/sec)*
* This is an artificial benchmark and results might vary significantly in real-life scenarios. Always run your own
benchmarks before deciding whether this optimization can have a positive effect for you.
loige 77

Closing Notes
JavaScript can be a very powerful and convenient language
when we have to deal with a lot of I/O (e.g. web servers)
The async story has evolved a lot in the last 10-15 years: new
patterns and language constructs have emerged
Async/Await is probably the best way to write async code today
To use Async/Await correctly you need to understand Promise
and callbacks
Take your time and invest in learning the fundamentals
loige 78

Cover Picture by on
Marc-Olivier Jodoin Unsplash
THANKS! 🙌❤
nodejsdp.link
loige
Grab these slides!
😍Grab the book
79

From Node.js to Design Patterns - BuildPiper

Recommended

Recommended

More Related Content

Similar to From Node.js to Design Patterns - BuildPiper

Similar to From Node.js to Design Patterns - BuildPiper (20)

More from Luciano Mammino

More from Luciano Mammino (20)

Recently uploaded

Recently uploaded (20)

From Node.js to Design Patterns - BuildPiper