Download as PDF, PPTX
![Query Logs in Real
Time
$ cat http.log | grep audit| bunyan -0 --strict -
c 'this.res.statusCode === 200' | json -ga
‘req.timers["fetch-api"]' req.url | awk '{print $1
" " substr($2,0,7)}' | daggr -k 2 -v 1 quantize](https://image.slidesharecdn.com/observablenode-150728221112-lva1-app6892/85/Observable-Node-js-Applications-EnterpriseJS-24-320.jpg)
Uploaded byYunong Xiao
Observable Node.js Applications - EnterpriseJS
This document discusses making Node.js applications more observable at scale. It describes using tools like Bunyan for structured logging, Restify for building observable REST applications, and Unix tools for querying logs. Observability is important for production systems to detect errors, latency issues, and memory leaks. The presenter advocates building observability into applications from the start through techniques like scoped child loggers and request capture streams.
More Related Content
![[231] the simplicity of cluster apps with circuit](https://cdn.slidesharecdn.com/ss_thumbnails/213thesimplicityofclusterappswithcircuit-150915022150-lva1-app6892-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Observable Node.js Applications - EnterpriseJS
Observable Node.js Applications - EnterpriseJS
- 1. Observable Node.js Applications atScale Yunong Xiao, Senior Node.js Engineer, UI Platform, Netflix @yunongx, yunong@netflix.com July 2015EnterpriseJS
- 2.
- 9.
- 10. What’s Wrong? Increased ErrorsIncreased Latency Memory Leak
- 11.
- 12.
- 14. DATA “It is acapital mistake to theorize before one has DATA. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Sherlock Holmes -A Scandal in Bohemia
- 15.
- 17. Hard Harder Insane DevelopmentProduction Production w/ Direct Customer Impact
- 18.
- 20.
- 21.
- 22.
- 23. Unix Philosophy grep(1), cut(1),wc(1), awk(1), perl(1), jq(1), json(1), daggr(1)
- 24. Query Logs inReal Time $ cat http.log | grep audit| bunyan -0 --strict - c 'this.res.statusCode === 200' | json -ga ‘req.timers["fetch-api"]' req.url | awk '{print $1 " " substr($2,0,7)}' | daggr -k 2 -v 1 quantize
- 25.
- 26.
- 28. { "name": "demo", "hostname": "Yunongs-MacBook-Pro.local", "pid":35919, "level": 30, "authInfo": { "username": "yunong", "password": "xxxxxx" }, "msg": "got request", "time": "2015-07-07T21:46:01.317Z", "v": 0 }
- 31. Features • Lightweight API •Log levels: trace, debug, info, warn, error, fatal • Extensible Streams interface. • Custom object rendering with serializers.
- 32. HTTP Request • MakeAPI requests • Persist state to file system • Query database • Update caches
- 33. Async HTTP Request •Make API requests • Persist state to file system • Query database • Update caches
- 37. node-vasync • Observable Asyncworkflow • https://github.com/davepacheco/node-vasync
- 38. async f() state failedf() successful f() pending f() finished f() # of errors
- 39.
- 41.
- 42. How Do ISee this in Prod? • Logs • Core Dumps • REPL • HTTP API
- 43.
- 44. • Observability • Metrics •Bunyan integration
- 45.
- 46.
- 47. Audit Logsstatus coderequest ID (UUID) URL request headers response headers individual handler timers req latency
- 49.
- 50. Request Capture Stream •Captures all log statements at trace level in memory. • Dump all logs for a particular request only on error.
- 53.
- 58. Logging • Native Bunyanintegration • Request capture stream • Audit logs • Scoped child loggers
- 59. What about DataProcessing?
- 60. Examples Find all logsfor a specific request. $ grep $uuid restify.log
- 61. Examples Count the #of non-200 responses. $ grep restify-audit restify.log | bunyan -c ‘this.res.statusCode !== 200` -0 | wc -l
- 62. Examples Show all requeststhat took longer than 200ms. $ grep restify-audit restify.log | bunyan -c ‘this.latency > 200` -0 | wc -l
- 63. Advanced Example Request latencydistribution by URL
- 64. restify + Bunyan •Streaming JSON. • Processing using Unix tools is easy. • Helpful tools: • Unix: cut(1), wc(1), grep(1), awk(1), perl(1), json(1), jq(1) … • JSON: https://github.com/trentm/json (npm install -g json) • daggr: https://github.com/joyent/daggr (npm install -g daggr)
- 65. Increased Errors IncreasedLatency
- 66. restify Core Contributors atNetflix @stinkydofu, @mjr578, @yunongx
- 67. restify Interested? Contribute torestify! restify.com github.com/restify/node-restify
- 68. distributed processing • elasticsearch • spark •hive bunyanrestify audit logs req id req headers req latency handler latencies URL res codeErrors res headers Processing w/ Unix tools • grep(1) • awk(1) • cut(1) • wc(1) • sort(1) • json(1) • bunyan(1) • … req scoped logs application specific context req capture stream dumps all logs on error DTrace* req latency handler latencies * Where available Production Observability
- 69. Observable Toolkit • vasync:https://github.com/davepacheco/node-vasync • Observable async operations • bunyan: https://github.com/trentm/node-bunyan • Streaming JSON logs • restify: restify.com • Observable REST applications • Unix Tools • Easily process JSON logs
- 70.
- 71.
- 72. Thanks • Questions? We’rehiring! • yunong@netflix.com • @yunongx • http://restify.com July 2015EnterprisJS