Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Observable Node.js Applications - EnterpriseJS

1,401 views

Published on

Learn how we build Node.js applications at Netflix with observability in mind.

Published in: Technology
  • Be the first to comment

Observable Node.js Applications - EnterpriseJS

  1. 1. Observable Node.js Applications at Scale Yunong Xiao, Senior Node.js Engineer, UI Platform, Netflix @yunongx, yunong@netflix.com July 2015EnterpriseJS
  2. 2. Node.js @ Netflix
  3. 3. Production is War
  4. 4. What’s Wrong? Increased Errors Increased Latency Memory Leak
  5. 5. Software is Complex
  6. 6. Drunk Man Anti-Method
  7. 7. DATA “It is a capital mistake to theorize before one has DATA. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Sherlock Holmes -A Scandal in Bohemia
  8. 8. Observability From the Ground Up
  9. 9. Hard Harder Insane Development Production Production w/ Direct Customer Impact
  10. 10. Dev === Prod
  11. 11. node-bunyan https://github.com/trentm/node-bunyan
  12. 12. node-bunyan Streaming JSON logging format
  13. 13. Streaming JSON
  14. 14. Unix Philosophy grep(1), cut(1), wc(1), awk(1), perl(1), jq(1), json(1), daggr(1)
  15. 15. Query Logs in Real Time $ cat http.log | grep audit| bunyan -0 --strict - c 'this.res.statusCode === 200' | json -ga ‘req.timers["fetch-api"]' req.url | awk '{print $1 " " substr($2,0,7)}' | daggr -k 2 -v 1 quantize
  16. 16. Illegible!
  17. 17. bunyan(1) CLI
  18. 18. { "name": "demo", "hostname": "Yunongs-MacBook-Pro.local", "pid": 35919, "level": 30, "authInfo": { "username": "yunong", "password": "xxxxxx" }, "msg": "got request", "time": "2015-07-07T21:46:01.317Z", "v": 0 }
  19. 19. Features • Lightweight API • Log levels: trace, debug, info, warn, error, fatal • Extensible Streams interface. • Custom object rendering with serializers.
  20. 20. HTTP Request • Make API requests • Persist state to file system • Query database • Update caches
  21. 21. Async HTTP Request • Make API requests • Persist state to file system • Query database • Update caches
  22. 22. node-vasync • Observable Async workflow • https://github.com/davepacheco/node-vasync
  23. 23. async f() state failed f() successful f() pending f() finished f() # of errors
  24. 24. Example cb() not invoked
  25. 25. Results function three pending
  26. 26. How Do I See this in Prod? • Logs • Core Dumps • REPL • HTTP API
  27. 27. Observable Node.js HTTP
  28. 28. • Observability • Metrics • Bunyan integration
  29. 29. restify http://restify.com http://github.com/restify/node-restify
  30. 30. Audit Logs
  31. 31. Audit Logsstatus code request ID (UUID) URL request headers response headers individual handler timers req latency
  32. 32. Too many logs!
  33. 33. Request Capture Stream • Captures all log statements at trace level in memory. • Dump all logs for a particular request only on error.
  34. 34. Scoped Child Loggers
  35. 35. Logging • Native Bunyan integration • Request capture stream • Audit logs • Scoped child loggers
  36. 36. What about Data Processing?
  37. 37. Examples Find all logs for a specific request. $ grep $uuid restify.log
  38. 38. Examples Count the # of non-200 responses. $ grep restify-audit restify.log | bunyan -c ‘this.res.statusCode !== 200` -0 | wc -l
  39. 39. Examples Show all requests that took longer than 200ms. $ grep restify-audit restify.log | bunyan -c ‘this.latency > 200` -0 | wc -l
  40. 40. Advanced Example Request latency distribution by URL
  41. 41. restify + Bunyan • Streaming JSON. • Processing using Unix tools is easy. • Helpful tools: • Unix: cut(1), wc(1), grep(1), awk(1), perl(1), json(1), jq(1) … • JSON: https://github.com/trentm/json (npm install -g json) • daggr: https://github.com/joyent/daggr (npm install -g daggr)
  42. 42. Increased Errors Increased Latency
  43. 43. restify Core Contributors at Netflix @stinkydofu, @mjr578, @yunongx
  44. 44. restify Interested? Contribute to restify! restify.com github.com/restify/node-restify
  45. 45. distributed processing • elasticsearch • spark • hive bunyanrestify audit logs req id req headers req latency handler latencies URL res codeErrors res headers Processing w/ Unix tools • grep(1) • awk(1) • cut(1) • wc(1) • sort(1) • json(1) • bunyan(1) • … req scoped logs application specific context req capture stream dumps all logs on error DTrace* req latency handler latencies * Where available Production Observability
  46. 46. Observable Toolkit • vasync: https://github.com/davepacheco/node-vasync • Observable async operations • bunyan: https://github.com/trentm/node-bunyan • Streaming JSON logs • restify: restify.com • Observable REST applications • Unix Tools • Easily process JSON logs
  47. 47. One more thing…
  48. 48. DTrace
  49. 49. Thanks • Questions? We’re hiring! • yunong@netflix.com • @yunongx • http://restify.com July 2015EnterprisJS

×