Node.js at Joyent: Engineering for Production
Upcoming SlideShare
Loading in...5
×
 

Node.js at Joyent: Engineering for Production

on

  • 1,329 views

Joyent is one of the largest deployers of Node.js in production systems. In order to successfully deploy large-scale, distributed systems, we must understand the systems we build! For us, that means ...

Joyent is one of the largest deployers of Node.js in production systems. In order to successfully deploy large-scale, distributed systems, we must understand the systems we build! For us, that means having first-class tools for debugging our software, and understanding and improving its performance.

Come on a whirlwind tour of the tools and techniques we use at Joyent as we build out large-scale distributed software with Node.js: from mdb for Post-Mortem Debugging, to Flame Graphs for performance analysis; from DTrace for dynamic, production-safe instrumentation and tracing, to JSON-formatted logging with Bunyan.

Statistics

Views

Total Views
1,329
Views on SlideShare
1,267
Embed Views
62

Actions

Likes
0
Downloads
8
Comments
0

1 Embed 62

https://twitter.com 62

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Node.js at Joyent: Engineering for Production Node.js at Joyent: Engineering for Production Presentation Transcript

  • Node.js In Production At Joyent __ . . _| |_ | .-. . . .-. :--. ||_ _| ;| || |(.-' | | | |__| `--' `-' `;-| `-' ' ' `-' / ; `-' Node.js at Joyent: Engineering for Production Joshua M. Clulow jmc@joyent.com Joshua M. Clulow @jmclulow ⊕ Joyent
  • Node.js In Production At Joyent node-vasync Observable Asynchronous Control Flow * Written by Dave Pacheco (Joyent) davepacheco/node-vasync on github * Similar in concept to caolan/async, but with a focus on observability * Returns status objects updated as flow progresses that can be inspected in core files, or exported via web services or REPLs * Provides the barrier() primitive: specify some set of named operations (you can extend the set mid-flight) and get a drain event when they all complete Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Kang Distributed System Observability * Written by Dave Pacheco (Joyent) davepacheco/kang on github * A library for exposing self-describing state from a set of processes via a web service * A client library and command-line tool for collecting, aggregating and inspecting that state * The exposed state is a set of "objects" arranged by "type", formatted as a JSON payload Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Post-mortem Debugging * Traditionally done for C software on UNIX systems * A program could call abort(), or some signals (e.g. SIGSEGV) would end the program and write a core file * The program can be restarted immediately so that service may be restored * Engineers can inspect this core file later with a debugger, similar to a black box flight recorder * Operators may also generate a core file without interrupting the program using gcore(1) Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Post-mortem Debugging in Node.js * From node v0.10.8 onwards, we can abort() on an uncaught exception by using: node --abort-on-uncaught-exception app.js * This preserves the stack that caused the exception in the core file, where we can see it in the debugger * Use this flag everywhere! Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent mdb(1) The Modular Debugger * The illumos (and SmartOS) debugger * Supports node core files through mdb_v8.so ::jsstack -- Javascript stack traces ::findjsobjects -- inspect all Javascript heap objects ::jsprint -- print properties of a single object Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent DTrace Production-safe Dynamic Instrumentation * Zero disabled overhead; low enabled overhead * In-kernel aggregation for high quantity events * First class support for handling JSON-formatted payloads with the json() subroutine * Add semantically interesting probe points to your applications with chrisa/node-dtrace-provider * Easily measure latency between events and plot the distribution Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Bunyan JSON-formatted Logging Library * Written by Trent Mick (Joyent) trentm/node-bunyan on github * Logs should be formatted for machines, not for people * Use a filter to make them human-readable * Log auditing information at INFO to files * Log verbosely (off by default) to DEBUG, and consume those logs via DTrace probes (bunyan -p) (uses node-dtrace-provider!) Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Restify REST framework for Web Service APIs * Written by Mark Cavage (Joyent) mcavage/node-restify on github * Route handler chains, like Express * Built-in DTrace probes to measure per-route and per-handler latency * Easy integration with Bunyan for logging Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Flamegraphs Profiling and Call Stack Visualisation * A node implementation, davepacheco/node-stackvis, based on Brendan Gregg's original Flamegraph work * Use DTrace to profile the application, i.e. regularly sample the current stack * Collect the like components of each sample and draw them together to give a picture of where the program spends its on-CPU time * Pipe your stacks through c++filt to demangle the C++ method names Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent General Development Tips * Compile C/C++ code with -fno-omit-frame-pointer, so that stack traces work in DTrace and mdb(1) * Prefix your property names, making them easier to find when debugging core files, e.g. var iface = { if_id: 0x3423, if_name: 'interface 0', if_ipaddr: '192.168.1.1' }; * Name your 'anonymous' functions, making them easier to identify in stack traces Joshua M. Clulow ⊕ Joyent
  • Node.js In Production At Joyent Thanks For Listening! Read more about our Node.js Engineering Practices: http://www.joyent.com/developers/node Joshua M. Clulow ⊕ Joyent