This document discusses best practices for running Node.js applications in production environments. It recommends: 1) Designing applications to fail and recover at multiple levels from single nodes to entire datacenters; 2) Avoiding failures by relying on services like databases in the cloud whenever possible; and 3) Isolating applications and their dependencies to prevent failures in one part from affecting others. Regular monitoring, logging and analysis of applications is also advised to identify issues and guide scaling decisions.

1. FAIL... THE RIGHT
WAY
NODE.JS IN PRODUCTION
ssw2014.formidablelabs.com
 | 
@ryan_roemer formidablelabs.com

2. WELCOME TO PRODUCTION
Production can be a rough place for
your Node.js apps. Things can go very
wrong out in the wild.

3. FORMIDABLE LABS

4. 3:00 AM

5. OUR FOCUS
Whether on PAAS, IAAS, or bare metal.
Design for Failure: Keep your Node.js apps up
Avoidance: Get yourself out of the failover business
Isolate: One failure at a time
Analyze: Debug and diagnose problems quickly

6. 1. DESIGN FOR FAILURE
Fail and recover at multiple levels.
Let's look at failure from a system
perspective.

7. SINGLE NODE.JS WORKER
Never ignore errors.
Have a strong bias for killing the
worker.
Handle: uncaughtException,
Listen: foo.on("error")
Domains

8. MULTIPLE NODE.JS WORKERS
cluster recluster
Use or to
multiplex CPUs and isolate errors.
Workers: die early on errors
Master: monitor and kill workers

9. MULTIPLE NODE.JS WORKERS
var recluster = require("recluster");
var cluster = recluster("./server.js");
cluster.run();
// Hot reload: kill -s SIGUSR2 CLUSTER_PID
process.on("SIGUSR2", function() {
console.log("Got SIGUSR2, reloading
cluster...");
cluster.reload();
});

10. SERVER
monit
Use or alternatives
Restart the Node.js master

11. SERVICE
Load-balancers
Heartbeat / ping monitors
Availability zones, etc.

12. MAKE IT HOT
Everything up to this point should have
hot failover.

13. DATACENTER
Hot failover across
datacenters?
Typically very costly
But, the real deal if you're serious

14. DISASTER RECOVERY
"Business Continuity"
Don't let a technological problem end your business
Have a worst case, "lose some data" recovery plan

15. 2. AVOID FAILURES
Get out of the business of failover
when you don't have to do it yourself.

16. RESOURCES TO NOT SUPPORT
Don't rely on system / service
resources you don't need to.
Disk: NAS, disks, SSDs.
Datastores: DB, cloud services.
... Load Balancers, DNS, etc.

17. HOW TO AVOID
Use SAAS wherever possible! (DB, LBs, storage).
Or PAAS for some Node.js apps.
Design Stateless, fungible servers (no disk risks).

18. 3. ISOLATE FAILURES
Isolate failures you can't
avoid.

19. RESOURCES TO SUPPORT
Look to resources you must depend on:
CPU/Load: Run out of this and it's over.
HTTP: Each different host you hit.
Datastores: Connections? Different Hosts?
... also, memory, I/O, etc. and combinations thereof

20. SOME ANECDOTES
Node.js apps can be bad neighbors.
DB (auto-suggest) vs. HTTP (vendor translations)
DB (CRUD app) vs. CPU/Load (co-located PHP app)
Read vs. Write DB operations.

21. HOW TO ISOLATE
Create "micro-services" that stand on their own.
Monitor for cross-pressure and respond. (Next section!)

22. 4. ANALYZE EVERYTHING
Data drives problem discovery
and action.

23. LOG, MONITOR, MINE

24. DECISIONS, GOALS
Things to look for in Node.js apps...
Identify
Resource pressure: CPU, I/O,
memory, network
Performance: Throughput,
latency
Errors/Bugs: Quantitative,
qualitative
Decide
Scale up, scale down?
Separate services?

25. RECAP
Design for failure
Avoid
Isolate
Analyze

26. THANKS!
ssw2014.formidablelabs.com
 | 
@ryan_roemer formidablelabs.com