The document discusses various aspects of using Node.js in production environments at Joyent, focusing on tools and practices for observability, debugging, and logging. Key topics include the use of the node-vasync library for asynchronous control flow, methods for post-mortem debugging, and dynamic instrumentation with DTrace. Additionally, it highlights utilities like the Bunyan logging library, Restify for web service APIs, and techniques for profiling applications.

1. Node.js In Production At Joyent
__
.
.
_| |_
| .-. . . .-. :--. ||_
_|
;|
|| |(.-' | | |
|__|
`--' `-' `;-| `-' ' ' `-'
/ ;
`-'
Node.js at Joyent:
Engineering for Production
Joshua M. Clulow
jmc@joyent.com
Joshua M. Clulow
@jmclulow
⊕ Joyent

2. Node.js In Production At Joyent
node-vasync
Observable Asynchronous Control Flow
* Written by Dave Pacheco (Joyent)
davepacheco/node-vasync on github
* Similar in concept to caolan/async, but with a focus
on observability
* Returns status objects updated as flow progresses that
can be inspected in core files, or exported via web
services or REPLs
* Provides the barrier() primitive: specify some set of
named operations (you can extend the set mid-flight)
and get a drain event when they all complete
Joshua M. Clulow
⊕ Joyent

3. Node.js In Production At Joyent
Kang
Distributed System Observability
* Written by Dave Pacheco (Joyent)
davepacheco/kang on github
* A library for exposing self-describing state from
a set of processes via a web service
* A client library and command-line tool for collecting,
aggregating and inspecting that state
* The exposed state is a set of "objects" arranged by
"type", formatted as a JSON payload
Joshua M. Clulow
⊕ Joyent

4. Node.js In Production At Joyent
Post-mortem Debugging
* Traditionally done for C software on UNIX systems
* A program could call abort(), or some signals
(e.g. SIGSEGV) would end the program and write
a core file
* The program can be restarted immediately so that
service may be restored
* Engineers can inspect this core file later with
a debugger, similar to a black box flight recorder
* Operators may also generate a core file without
interrupting the program using gcore(1)
Joshua M. Clulow
⊕ Joyent

5. Node.js In Production At Joyent
Post-mortem Debugging in Node.js
* From node v0.10.8 onwards, we can abort() on
an uncaught exception by using:
node
--abort-on-uncaught-exception
app.js
* This preserves the stack that caused the exception
in the core file, where we can see it in the
debugger
* Use this flag everywhere!
Joshua M. Clulow
⊕ Joyent

6. Node.js In Production At Joyent
mdb(1)
The Modular Debugger
* The illumos (and SmartOS) debugger
* Supports node core files through mdb_v8.so
::jsstack
-- Javascript stack traces
::findjsobjects
-- inspect all Javascript heap objects
::jsprint
-- print properties of a single object
Joshua M. Clulow
⊕ Joyent

7. Node.js In Production At Joyent
DTrace
Production-safe Dynamic Instrumentation
* Zero disabled overhead; low enabled overhead
* In-kernel aggregation for high quantity events
* First class support for handling JSON-formatted
payloads with the json() subroutine
* Add semantically interesting probe points to
your applications with chrisa/node-dtrace-provider
* Easily measure latency between events and plot the
distribution
Joshua M. Clulow
⊕ Joyent

8. Node.js In Production At Joyent
Bunyan
JSON-formatted Logging Library
* Written by Trent Mick (Joyent)
trentm/node-bunyan on github
* Logs should be formatted for machines,
not for people
* Use a filter to make them human-readable
* Log auditing information at INFO to files
* Log verbosely (off by default) to DEBUG, and
consume those logs via DTrace probes (bunyan -p)
(uses node-dtrace-provider!)
Joshua M. Clulow
⊕ Joyent

9. Node.js In Production At Joyent
Restify
REST framework for Web Service APIs
* Written by Mark Cavage (Joyent)
mcavage/node-restify on github
* Route handler chains, like Express
* Built-in DTrace probes to measure per-route and
per-handler latency
* Easy integration with Bunyan for logging
Joshua M. Clulow
⊕ Joyent

10. Node.js In Production At Joyent
Flamegraphs
Profiling and Call Stack Visualisation
* A node implementation, davepacheco/node-stackvis,
based on Brendan Gregg's original Flamegraph work
* Use DTrace to profile the application, i.e. regularly
sample the current stack
* Collect the like components of each sample and draw them
together to give a picture of where the program
spends its on-CPU time
* Pipe your stacks through c++filt to demangle the C++
method names
Joshua M. Clulow
⊕ Joyent

11. Node.js In Production At Joyent
General Development Tips
* Compile C/C++ code with -fno-omit-frame-pointer,
so that stack traces work in DTrace and mdb(1)
* Prefix your property names, making them easier to
find when debugging core files, e.g.
var iface = { if_id:
0x3423,
if_name:
'interface 0',
if_ipaddr: '192.168.1.1' };
* Name your 'anonymous' functions, making them easier
to identify in stack traces
Joshua M. Clulow
⊕ Joyent

12. Node.js In Production At Joyent
Thanks For Listening!
Read more about our Node.js Engineering Practices:
http://www.joyent.com/developers/node
Joshua M. Clulow
⊕ Joyent