You finally built that amazing start up idea you had in mind for years and you did it using Node.js! That's Great! You just launched it on Hacker News and you are very happy and proud... but now more and more people are using it and you start to have a nasty fear that Node.js won't scale, because you now... it's single threaded! Is your project doomed now? Do you have to invest your time on rewriting it in something like C++ or maybe Rust or even Go? You'd rather invest your time on adding valuable features for your users rather than learning a new language and rewriting everything from scratch, but what if nothing works anymore? And... by the way, what the heck ""single threaded"" really means?! Fear no more, dear fellow developer! In this talk we will discuss the architecture of Node.js going through its strengths and its weaknesses. We will then talk about scalability and I will share some valuable tips and tricks to make your Node.js app scale! Spoiler alert: you probably won't need Go or Rust :)

1. Node.js
Scalability Tips
Luciano Mammino ( )@loige
2020-07-09
loige.link/node-scale
1

2. 👋 Hello, I am Luciano!
Principal Software Engineer at FabFitFun
Blog:
Twitter:
GitHub:
loige.co
@loige
@lmammino
nodejsdp.link
2

3. @loige
Get the slides!
loige.link/node-scale
3

4. Node.js + Scalability? 🤔
@loige
4

5. @loige
5

6. @loige
6

7. @loige
6

8. @loige
6

9. @loige
6

10. @loige
7

11. @loige
7

12. @loige
7

13. @loige
7

14. 🤣
@loige
7

15. "Scalability is the property of a system to
handle a growing amount of work by
adding resources to the system"
— Wikipedia
@loige
8

16. "A service is said to be scalable if when
we increase the resources in a
system, it results in increased
performance in a manner
proportional to resources added"
— Werner Vogels
@loige
9

17. 🛫
Tip 1.
Establish a baseline
@loige
10

18. /?data=Hello%20Shift
https://repl.it/@lmammino/QRGen
@loige
11

19. const { createServer } = require('http')
const { URL } = require('url')
const QRCode = require('qrcode')
createServer(function handler (req, res) {
const url = new URL(req.url, 'http://localhost:8080')
const data = url.searchParams.get('data')
if (!data) {
res.writeHead(400) // bad request
return res.end()
}
res.writeHead(200, { 'Content-Type': 'image/png' })
QRCode.toFileStream(res, data, { width: 300 })
})
.listen(8080)
@loige
12

20. autocannon -c 200 --on-port / -- node server.js
wrk
node server.js&
wrk -t8 -c200 -d10s http://localhost:8080/
@loige
13

21. autocannon -c 200 --on-port /?data=Hello%20Shift -- node server.js
Running 10s test @ http://localhost:8080/?data=Hello%20Shift
200 connections
┌─────────┬─────────┬─────────┬─────────┬─────────┬────────────┬─────────┬────────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼─────────┼─────────┼─────────┼─────────┼────────────┼─────────┼────────────┤
│ Latency │ 1899 ms │ 1951 ms │ 2053 ms │ 2054 ms │ 1964.92 ms │ 99.9 ms │ 3364.03 ms │
└─────────┴─────────┴─────────┴─────────┴─────────┴────────────┴─────────┴────────────┘
┌───────────┬─────┬──────┬─────────┬────────┬────────┬────────┬─────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤
│ Req/Sec │ 0 │ 0 │ 30 │ 199 │ 99.5 │ 94.27 │ 30 │
├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤
│ Bytes/Sec │ 0 B │ 0 B │ 50.7 kB │ 336 kB │ 168 kB │ 159 kB │ 50.7 kB │
└───────────┴─────┴──────┴─────────┴────────┴────────┴────────┴─────────┘
Req/Bytes counts sampled once per second.
995 requests in 10.08s, 1.68 MB read
@loige
14

22. autocannon -c 200 --on-port /?data=Hello%20Shift -- node server.js
Running 10s test @ http://localhost:8080/?data=Hello%20Shift
200 connections
┌─────────┬─────────┬─────────┬─────────┬─────────┬────────────┬─────────┬────────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼─────────┼─────────┼─────────┼─────────┼────────────┼─────────┼────────────┤
│ Latency │ 1899 ms │ 1951 ms │ 2053 ms │ 2054 ms │ 1964.92 ms │ 99.9 ms │ 3364.03 ms │
└─────────┴─────────┴─────────┴─────────┴─────────┴────────────┴─────────┴────────────┘
┌───────────┬─────┬──────┬─────────┬────────┬────────┬────────┬─────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤
│ Req/Sec │ 0 │ 0 │ 30 │ 199 │ 99.5 │ 94.27 │ 30 │
├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤
│ Bytes/Sec │ 0 B │ 0 B │ 50.7 kB │ 336 kB │ 168 kB │ 159 kB │ 50.7 kB │
└───────────┴─────┴──────┴─────────┴────────┴────────┴────────┴─────────┘
Req/Bytes counts sampled once per second.
995 requests in 10.08s, 1.68 MB read
@loige
14

23. ⛅
Tip 1-bis
Also, find out your ceiling
@loige
15

24. const { createServer } = require('http')
createServer((req, res) => {
if (req.method === 'GET' && req.url === '/') {
res.writeHead(200, { 'Content-Type': 'text/plain' })
res.end('Hello Worldn')
} else {
res.statusCode = 404
res.end()
}
})
.listen(8080)
@loige
16

25. autocannon -c 200 --on-port / -- node server.js
Running 10s test @ http://localhost:8080/
200 connections
┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬──────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────────┤
│ Latency │ 3 ms │ 5 ms │ 11 ms │ 14 ms │ 5.51 ms │ 2.71 ms │ 80.63 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴──────────┘
┌───────────┬─────────┬─────────┬────────┬─────────┬─────────┬─────────┬─────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤
│ Req/Sec │ 21087 │ 21087 │ 34623 │ 35487 │ 33258.4 │ 4107.01 │ 21077 │
├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 3.29 MB │ 3.29 MB │ 5.4 MB │ 5.54 MB │ 5.19 MB │ 641 kB │ 3.29 MB │
└───────────┴─────────┴─────────┴────────┴─────────┴─────────┴─────────┴─────────┘
Req/Bytes counts sampled once per second.
333k requests in 10.1s, 51.9 MB read
@loige
17

26. autocannon -c 200 --on-port / -- node server.js
Running 10s test @ http://localhost:8080/
200 connections
┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬──────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────────┤
│ Latency │ 3 ms │ 5 ms │ 11 ms │ 14 ms │ 5.51 ms │ 2.71 ms │ 80.63 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴──────────┘
┌───────────┬─────────┬─────────┬────────┬─────────┬─────────┬─────────┬─────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤
│ Req/Sec │ 21087 │ 21087 │ 34623 │ 35487 │ 33258.4 │ 4107.01 │ 21077 │
├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 3.29 MB │ 3.29 MB │ 5.4 MB │ 5.54 MB │ 5.19 MB │ 641 kB │ 3.29 MB │
└───────────┴─────────┴─────────┴────────┴─────────┴─────────┴─────────┴─────────┘
Req/Bytes counts sampled once per second.
333k requests in 10.1s, 51.9 MB read
@loige
17

27. 🍾
Tip 2.
Find your bottleneck
@loige
18

28. Clinic.js
clinicjs.org
@loige
19

29. clinic doctor --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js
@loige
20

30. clinic flame --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js
@loige
21

31. clinic bubble --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js
@loige
22

32. 🎳
Tip 3.
Understand your goals
@loige
23

33. What do we optimize for?
Throughput?
Memory?
Latency?
@loige
24

34. 👁
Tip 4.
Always "observe"
@loige
25

35. I mean, in production!
Logs - Metrics - Traces
@loige
26

36. 🚀
Tip 5.
Scale your architecture
@loige
27

37. Performance != Scalability
@loige
28

38. How can we scale a system
by adding resources?
@loige
29

39. The " "Scale Cube
x-axis
cloning
z-axis
partitioning
y-axis
functional
decomposition
@loige
30

40. The " "Scale Cube
x-axis
cloning
z-axis
partitioning
y-axis
functional
decomposition
@loige
30

41. Cloning
Reverse proxy
31
Inside the same server
Load Balancer
Using multiple server
@loige

42. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

43. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

44. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

45. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

46. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

47. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

48. The modulecluster
Master process
Worker
process
Worker
process
Worker
process
32
@loige

49. const cluster = require('cluster')
const numCPUs = require('os').cpus().length
if (cluster.isMaster) {
// Fork workers
for (let i = 0; i < numCPUs; i++) {
cluster.fork()
}
} else {
// Worker code
require('./server.js')
}
@loige
33

50. const cluster = require('cluster')
const numCPUs = require('os').cpus().length
if (cluster.isMaster) {
// Fork workers
for (let i = 0; i < numCPUs; i++) {
cluster.fork()
}
} else {
// Worker code
require('./server.js')
} 3-4x req/sec
(8 core)
@loige
33

51. You could also use
Check out !
Worker Threads
piscina
@loige
34

52. Cloning is the easiest strategy to
scale a service...
... as long as your application is
Stateless
@loige
35

53. API Gateway
Functional decomposition
a.k.a. "Micro-services"
36
/products
/cart
cart DB
products
DB
@loige

54. API Gateway
Functional decomposition
a.k.a. "Micro-services"
37
/products
/cart
Functional decomposition can
also be combined with cloning!
cart DB
products
DB
@loige

55. Node.js is great for microservices
@loige
38

56. Microservices can also help with
scaling the organisation!
@loige
39

57. Microservices add complexity
Observability
Deployments
Versioning
Integration
@loige
40

58. Partitioning
Service and Data Partitioning along Customer Boundaries
Shard partitioning
/products/[A-L]/
/products/[M-Z]/
DB 2
41
DB 1
@loige

59. Partitioning is generally used
to scale databases
and
SaaS software geographically
@loige
42

60. Summary
@loige
43

61. Summary
🛫 Establish a baseline
@loige
43

62. Summary
🛫 Establish a baseline
🍾 Find your bottleneck
@loige
43

63. Summary
🛫 Establish a baseline
🍾 Find your bottleneck
🎳 Understand your goals
@loige
43

64. Summary
🛫 Establish a baseline
🍾 Find your bottleneck
🎳 Understand your goals
👁 Always "observe"
@loige
43

65. Summary
🛫 Establish a baseline
🍾 Find your bottleneck
🎳 Understand your goals
👁 Always "observe"
🚀 Scale your architecture
(cloning, decomposition & partitioning)
@loige
43

66. Thank you!
Special thanks to ,
, ,
, ,
, ,
, ,
, , ,
, , ,
, , ,
, , ,
Icons and SVGs by
@StefanoAbalsamo
@matteocollina @dagonzago
@NullishCoalesce @DublinSvelte
@KViglucci @gjohnson391
@lucamaraschi @laurekamalandua
@giltayar @mrm8488 @adrirai
@harafise @EugeneWare @Jauny
@tlivings @michaelcfine @leojino
@shahidontech @Lordoomer @zsadigov
@dottorblaster
freepik.com
loige.link/node-scale
@loige
44