Node.js at Joyent: Engineering for Production


Published on

Joyent is one of the largest deployers of Node.js in production systems. In order to successfully deploy large-scale, distributed systems, we must understand the systems we build! For us, that means having first-class tools for debugging our software, and understanding and improving its performance.

Come on a whirlwind tour of the tools and techniques we use at Joyent as we build out large-scale distributed software with Node.js: from mdb for Post-Mortem Debugging, to Flame Graphs for performance analysis; from DTrace for dynamic, production-safe instrumentation and tracing, to JSON-formatted logging with Bunyan.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Node.js at Joyent: Engineering for Production

  1. 1. Node.js In Production At Joyent __ . . _| |_ | .-. . . .-. :--. ||_ _| ;| || |(.-' | | | |__| `--' `-' `;-| `-' ' ' `-' / ; `-' Node.js at Joyent: Engineering for Production Joshua M. Clulow Joshua M. Clulow @jmclulow ⊕ Joyent
  2. 2. Node.js In Production At Joyent node-vasync Observable Asynchronous Control Flow * Written by Dave Pacheco (Joyent) davepacheco/node-vasync on github * Similar in concept to caolan/async, but with a focus on observability * Returns status objects updated as flow progresses that can be inspected in core files, or exported via web services or REPLs * Provides the barrier() primitive: specify some set of named operations (you can extend the set mid-flight) and get a drain event when they all complete Joshua M. Clulow ⊕ Joyent
  3. 3. Node.js In Production At Joyent Kang Distributed System Observability * Written by Dave Pacheco (Joyent) davepacheco/kang on github * A library for exposing self-describing state from a set of processes via a web service * A client library and command-line tool for collecting, aggregating and inspecting that state * The exposed state is a set of "objects" arranged by "type", formatted as a JSON payload Joshua M. Clulow ⊕ Joyent
  4. 4. Node.js In Production At Joyent Post-mortem Debugging * Traditionally done for C software on UNIX systems * A program could call abort(), or some signals (e.g. SIGSEGV) would end the program and write a core file * The program can be restarted immediately so that service may be restored * Engineers can inspect this core file later with a debugger, similar to a black box flight recorder * Operators may also generate a core file without interrupting the program using gcore(1) Joshua M. Clulow ⊕ Joyent
  5. 5. Node.js In Production At Joyent Post-mortem Debugging in Node.js * From node v0.10.8 onwards, we can abort() on an uncaught exception by using: node --abort-on-uncaught-exception app.js * This preserves the stack that caused the exception in the core file, where we can see it in the debugger * Use this flag everywhere! Joshua M. Clulow ⊕ Joyent
  6. 6. Node.js In Production At Joyent mdb(1) The Modular Debugger * The illumos (and SmartOS) debugger * Supports node core files through ::jsstack -- Javascript stack traces ::findjsobjects -- inspect all Javascript heap objects ::jsprint -- print properties of a single object Joshua M. Clulow ⊕ Joyent
  7. 7. Node.js In Production At Joyent DTrace Production-safe Dynamic Instrumentation * Zero disabled overhead; low enabled overhead * In-kernel aggregation for high quantity events * First class support for handling JSON-formatted payloads with the json() subroutine * Add semantically interesting probe points to your applications with chrisa/node-dtrace-provider * Easily measure latency between events and plot the distribution Joshua M. Clulow ⊕ Joyent
  8. 8. Node.js In Production At Joyent Bunyan JSON-formatted Logging Library * Written by Trent Mick (Joyent) trentm/node-bunyan on github * Logs should be formatted for machines, not for people * Use a filter to make them human-readable * Log auditing information at INFO to files * Log verbosely (off by default) to DEBUG, and consume those logs via DTrace probes (bunyan -p) (uses node-dtrace-provider!) Joshua M. Clulow ⊕ Joyent
  9. 9. Node.js In Production At Joyent Restify REST framework for Web Service APIs * Written by Mark Cavage (Joyent) mcavage/node-restify on github * Route handler chains, like Express * Built-in DTrace probes to measure per-route and per-handler latency * Easy integration with Bunyan for logging Joshua M. Clulow ⊕ Joyent
  10. 10. Node.js In Production At Joyent Flamegraphs Profiling and Call Stack Visualisation * A node implementation, davepacheco/node-stackvis, based on Brendan Gregg's original Flamegraph work * Use DTrace to profile the application, i.e. regularly sample the current stack * Collect the like components of each sample and draw them together to give a picture of where the program spends its on-CPU time * Pipe your stacks through c++filt to demangle the C++ method names Joshua M. Clulow ⊕ Joyent
  11. 11. Node.js In Production At Joyent General Development Tips * Compile C/C++ code with -fno-omit-frame-pointer, so that stack traces work in DTrace and mdb(1) * Prefix your property names, making them easier to find when debugging core files, e.g. var iface = { if_id: 0x3423, if_name: 'interface 0', if_ipaddr: '' }; * Name your 'anonymous' functions, making them easier to identify in stack traces Joshua M. Clulow ⊕ Joyent
  12. 12. Node.js In Production At Joyent Thanks For Listening! Read more about our Node.js Engineering Practices: Joshua M. Clulow ⊕ Joyent