Jesang Yoon
CTO / Kanizsa Lab Co., Ltd
• LinkedIn: https://kr.linkedin.com/in/jesangyoon
• Rocket Punch:https://www.rocketpunch.com/@JeSangYoon
• yoonjs2@kanizsalab.com
• Yoonjs2@naver.com
• Javascript를이용하여 (Web)서버 프로그래밍을
할 수 있도록 도와주는 Framework
• GoogleChrome V8 Engine 을 Runtime으로 사용
• POSIX AsyncThread를 이용하여
Non Blocking Asynchronous I/O를 지원
• Single Threaded, MessageQueue, Event Based
• Joyent라는 회사가 Maintain을 하고 있는 오픈소스 프로젝트https://nodejs.org
TJ Fontaine
3rd BDFL of node.js
Issac Schlueter
2nd BDFL of node.js
Founder or NPM
Ryan Dahl
Founder of node.js
2011
2012
2013
2014
2015
2009
Mikael Rogers
Core Maintainer of io.js
https://nodejs.org/en/download/releases/
분열된 커뮤니티 통합
node.js + io.js
Joyent 와 의견차이로 인해
node 핵심 개발자들이
io.js 로 분열
Tomcat
reference: http://strongloop.com
Multithreaded Model
Tomcat 등의 Java기반 Webserver
• Pro
• Simple to implement
• Many, Many references
• Con
• 1 Thread/Request = Memory Inefficient
• Concurrency Problems
• Blocking IO = Bottleneck
reference: http://strongloop.com
node.js
Event Loop Model
node.js, Vert.x 등의 최신 Webserver
• Pro
• Blazingly Fast = Non-blocking I/O Advantage
• Memory efficient (? = Message Queue Cost)
• Single threaded = Less Concurrency Problem
• Con
• Hard to Implement (Callbacks)
• Inefficient CPU Performance
• Hard to Scale-up
• Hard to build large application
(better to be Lambda or Microservices)
Frontend Server
AWS t2.mirco x 2 ~
3.3GHz Xeon, 1G Ram
Clustered Mode (3 process)
1 Year 2 Month
in production
Frontend Server
AWS t2.mirco x 2 ~
3.3GHz Xeon, 1G Ram
Clustered Mode
In preparation
Tool Pro Con Suggestion
Vim Directly edit in server
No auto-completion
No syntax checking
Emergency hot fix in server
Sublime Text
Light weight than IDE
Many plugins
No auto-completion
No syntax checking
Replacement of notepad
Visual Studio
Convenient Scheme
Many functions
Too heavy
Full support only for Windows (sucks)
If you are big fan of Windows
PyCharm
Many functions
Nice remote features
Less heavy
Must install node.js plugin
If you working with Python and
node.js same time
IntelliJ IDEA
Many functions
Built-in node.js support
Too heavy (Multi-purpose) If you are big fan of all-in-one
Because of dynamic/interpreter aspect of JS, IDE may nice for productivity,
but mix tools for purpose may required also.
There is no common structure for node.js applications
but you’d better to stick with CommonJS way (Although may not good for web compatibility)
Root
Module_A
application.js
index.js
module_1.js
var module_A = require(“./Module_A”)(a);
module_A.foo(x);
module.export = function(a) {
return {
foo: function(x) { console.log(x) }
};
}
module.export = function(a) {
require(“./module_1.js”)(a);
// require(“./module_2.js”);
}
main
Due to the difference language paradigm and conceptually incomplete to support OOP,
Using OOP style in node.js is possible but better not to strictly rely on it.
Better to apply only for specific objects (like Error/Exception Object)
Error
HttpError
function Error(message) { // In JS world, function is object too
this.message = “I am error”;
};
var util = require(”util”);
function HttpError(responseCode, message) {
Error.call(message); // call parent constructor
this.responseCode = responseCode;
this.name = ”HttpError”;
};
util.inherits(HttpError, Error); // explicitly use node.js inheritence
BadRequestError
If you need compatibility between Web JS and node.js, Do not use this way.
Instead, use inheritance using JS prototype
Its very prone to be encounter complex callbacks since asynchronous feature of node.js (or async JS)
It’s very bad practice of JS programming and should be avoided.
Trivia: Joyent core developer still sticks with callbacks since they want to keep low-level simplicity
but It’s very controversy between many node.js developers. (and they blame joyent about that)
To overcome drawbacks of callbacks, Promise like techniques are widely used.
The concept of Promise is using Object to communicate between callbacks.
Promise (Promise A+) had become standard feature of ES6 Javascript.
var file = reqire(“./file”);
file.open(function(err, fp) {
// How to handle errors?
file.read(fp, function(err, content) {
for (var item in content) {
file.replace(item, “new item”, function (err, fp) {
file.commit(fp, function(err) {
// How to close file when its done?
});
});
}
});
});
var file = reqire(“./file”);
file.open()
.then(function (fp) {
return file.read(fp);
})
.all(function (item) {
return file.replace(item, ‘new item’);
})
.then(function(fp) {
return file.commit(fp);
})
.then(function() {
return file.close();
})
.catch(function(err) {
console.log(‘Errors are handled:’ + err);
return file.revert();
});
https://promisesaplus.com/
Old Javascript callback hell
New programming concept born from resolving callback hell
Must see: http://blog.namangoel.com/dealing-with-callback-hell
Promise A+ Reactive
Programming
Generators
Library Purpose Pro Con Suggestion
Restify
RESTful Server
Framework
- Light-weight
- Very focused for RESTful API
Implementation
- Less middleware feature
- Poor documentation
If you fed up with Express JS and
only you need is RESTful API
Sequelize ORM
- Best ORM Library
- Well documented
- Lack of some GIS feature
- Auto sync may bad
Be safe to use sync()
Underscore
Common
Utilities
- 100% Web compatible
- Light and easy to use
- Lack of function for arguments
object
If you not on ES6, use it.
Bluebird Promise A+
- 100% Compatible with
ES 6 Promise
- Best Promise A+ Library
- Has some memory leaks
- Less documentation
If you not on ES6, use it.
Mocha TDD/BDD
- Best BDD/TDD Framework
- Best with should.js
- May be old If you have no preference, use it.
Passport JS
Oauth
/ Social Login
- Best Login Library
- Many Strategies
- Some strategies may need to be
fixed on your own.
If you don’t want to build your own
login feature, use it.
node.js and JS community is very large and rapidly growing,
So you have to checkout good libraries at NPM and Github to reduce time to search.
Many web frameworks exists for node.js but all of them has pro/cons = no superb one.
You have to choose right tool for your right purposes. (Avoid Tech Masturbation)
Name Express JS Hapi Restify Strongloop
Pro • All-in-one (Web/RESTful)
• Many 3rd party middlewares
• Service by Manifest
• Walmart Support
• Light weight
• Dtrace support
• Only for RESTful Service
• For Enterprise
• Nice dashboard, Profiler
• Swagger included
Con
• Very complex to use/fix
• Soon will be decayed
• Static content can be
served via nginx or apache
• Less middleware
• Express code cannot be used
• Poor documentation
• Have to build everything you
needed.
• Only for Enterprise
• Memory leaks
• Very heavy
• No OSS
Due to the structure limitations, node.js may very bad choice for certain applications
If you need intensive CPU jobs, consider use Python or Java. (http://imjuni.tistory.com/694)
Events are not processed in enough time.
Event queue is rapidly grow
Memory is growing fast
Heavy CPU process/wait in callbacks
Segmentation Fault may happen
V8 Engine GC has two way in memory management, each will executed respectively
https://strongloop.com/strongblog/node-js-performance-garbage-collection/
Full GC
(Mark and Sweep & Mark and Compact)
Short GC
(Scavengering)
Checkout RSS/Heap Size/Heap total to checkout memory leaks.
Watch your Heap growth pattern before and after GC using Continuous request for period
Full GC
(Mark and Sweep & Mark and Compact)
Short GC
(Scavengering)
Normal
Memory Leaking
Segment Fault
OS kills app
Heap & RSS are kepp growing
Running server with high pressure of request may not help to find out actual problem.
Using google chrome dev tools and few other libraries, you can profile your applications
Must see: https://www.joyent.com/blog/walmart-node-js-memory-leak
Golden rules
1. Narrow down causes
2. Wait, Collect, Compare
3. Check your throught-put
4. Most of all: Build with KISS
Wallmart Memory LeakCase (Running for 15 hours )
Due to the Javascripts less type safe and very dynamic features,
Unit Testing is very crucial to assure quality of Application (NO TDD, NO DEV)
Type What to Test Goal How to Test Together with
Module Test Module To assure quality
of module
UT/TDD with
randominputs
Feature Test API To assure quality
of each features
UT/BDD with
random inputs
Load Runner
Scenario Test Business Logic To assure quality
of scenario
UT/BDD with
random inputs
Load Runner
You have to run regression Test for all above 3 (If possible).
Managing bulk of codes in Unit Testing is not that easy
Kanizsa Lab's Software Testing Policy
Single Mode Clustered Mode
PID: 1
PID: 1
master
PID: 2
slave 1
PID: 3
slave 2
PID: 1
master
PID: 2
slave 1
PID: 3
slave 2
Clustered Mode
with Forever
Forever
To enable scalability in node.js, Clustered Mode is supported (Process Fork) but you have to manage
slaves by yourself. Using forever daemon to watch master process status is encouraged.
Recommended size of slaves = number of CPUs (But you’d better check out enough memory exists)
Logging is very important but have to keep a “golden mean”
1) Do not give burden to process 2) Enough to be detailed 3) Machine Understandable 4) Server wide standard format
PID: 1
node
bunyan
CRON
JOB
Logrotate
File
AWS
S3upload
JSON
node bunyan is the best library to logging with JSON
bunyan is also main logging library of Joyent Cloud
Not recommended:
Upload to NoSQL directly via app internal library
Seems too much in early stages:
Using transport system like Flume, Kafka, Fluentd
these technology required by enough large clusters
Logrotate
node.js is still rapidly growing and expanding its supporting platforms from server to embedded
devices. the best advantage of node.js: Fast I/O with less memory and easy to start makes it a very
fascinating framework to learn for developers. So keep watch it!
IoT.js (Forked version of node.js) by Samsung node.js running on Raspberry-Pi as application framework
node.js 실무 - node js in practice by Jesang Yoon

node.js 실무 - node js in practice by Jesang Yoon

  • 2.
    Jesang Yoon CTO /Kanizsa Lab Co., Ltd • LinkedIn: https://kr.linkedin.com/in/jesangyoon • Rocket Punch:https://www.rocketpunch.com/@JeSangYoon • yoonjs2@kanizsalab.com • Yoonjs2@naver.com
  • 3.
    • Javascript를이용하여 (Web)서버프로그래밍을 할 수 있도록 도와주는 Framework • GoogleChrome V8 Engine 을 Runtime으로 사용 • POSIX AsyncThread를 이용하여 Non Blocking Asynchronous I/O를 지원 • Single Threaded, MessageQueue, Event Based • Joyent라는 회사가 Maintain을 하고 있는 오픈소스 프로젝트https://nodejs.org
  • 4.
    TJ Fontaine 3rd BDFLof node.js Issac Schlueter 2nd BDFL of node.js Founder or NPM Ryan Dahl Founder of node.js 2011 2012 2013 2014 2015 2009 Mikael Rogers Core Maintainer of io.js https://nodejs.org/en/download/releases/ 분열된 커뮤니티 통합 node.js + io.js Joyent 와 의견차이로 인해 node 핵심 개발자들이 io.js 로 분열
  • 5.
    Tomcat reference: http://strongloop.com Multithreaded Model Tomcat등의 Java기반 Webserver • Pro • Simple to implement • Many, Many references • Con • 1 Thread/Request = Memory Inefficient • Concurrency Problems • Blocking IO = Bottleneck
  • 6.
    reference: http://strongloop.com node.js Event LoopModel node.js, Vert.x 등의 최신 Webserver • Pro • Blazingly Fast = Non-blocking I/O Advantage • Memory efficient (? = Message Queue Cost) • Single threaded = Less Concurrency Problem • Con • Hard to Implement (Callbacks) • Inefficient CPU Performance • Hard to Scale-up • Hard to build large application (better to be Lambda or Microservices)
  • 7.
    Frontend Server AWS t2.mircox 2 ~ 3.3GHz Xeon, 1G Ram Clustered Mode (3 process) 1 Year 2 Month in production Frontend Server AWS t2.mirco x 2 ~ 3.3GHz Xeon, 1G Ram Clustered Mode In preparation
  • 9.
    Tool Pro ConSuggestion Vim Directly edit in server No auto-completion No syntax checking Emergency hot fix in server Sublime Text Light weight than IDE Many plugins No auto-completion No syntax checking Replacement of notepad Visual Studio Convenient Scheme Many functions Too heavy Full support only for Windows (sucks) If you are big fan of Windows PyCharm Many functions Nice remote features Less heavy Must install node.js plugin If you working with Python and node.js same time IntelliJ IDEA Many functions Built-in node.js support Too heavy (Multi-purpose) If you are big fan of all-in-one Because of dynamic/interpreter aspect of JS, IDE may nice for productivity, but mix tools for purpose may required also.
  • 10.
    There is nocommon structure for node.js applications but you’d better to stick with CommonJS way (Although may not good for web compatibility) Root Module_A application.js index.js module_1.js var module_A = require(“./Module_A”)(a); module_A.foo(x); module.export = function(a) { return { foo: function(x) { console.log(x) } }; } module.export = function(a) { require(“./module_1.js”)(a); // require(“./module_2.js”); } main
  • 11.
    Due to thedifference language paradigm and conceptually incomplete to support OOP, Using OOP style in node.js is possible but better not to strictly rely on it. Better to apply only for specific objects (like Error/Exception Object) Error HttpError function Error(message) { // In JS world, function is object too this.message = “I am error”; }; var util = require(”util”); function HttpError(responseCode, message) { Error.call(message); // call parent constructor this.responseCode = responseCode; this.name = ”HttpError”; }; util.inherits(HttpError, Error); // explicitly use node.js inheritence BadRequestError If you need compatibility between Web JS and node.js, Do not use this way. Instead, use inheritance using JS prototype
  • 12.
    Its very proneto be encounter complex callbacks since asynchronous feature of node.js (or async JS) It’s very bad practice of JS programming and should be avoided. Trivia: Joyent core developer still sticks with callbacks since they want to keep low-level simplicity but It’s very controversy between many node.js developers. (and they blame joyent about that)
  • 13.
    To overcome drawbacksof callbacks, Promise like techniques are widely used. The concept of Promise is using Object to communicate between callbacks. Promise (Promise A+) had become standard feature of ES6 Javascript. var file = reqire(“./file”); file.open(function(err, fp) { // How to handle errors? file.read(fp, function(err, content) { for (var item in content) { file.replace(item, “new item”, function (err, fp) { file.commit(fp, function(err) { // How to close file when its done? }); }); } }); }); var file = reqire(“./file”); file.open() .then(function (fp) { return file.read(fp); }) .all(function (item) { return file.replace(item, ‘new item’); }) .then(function(fp) { return file.commit(fp); }) .then(function() { return file.close(); }) .catch(function(err) { console.log(‘Errors are handled:’ + err); return file.revert(); }); https://promisesaplus.com/ Old Javascript callback hell
  • 14.
    New programming conceptborn from resolving callback hell Must see: http://blog.namangoel.com/dealing-with-callback-hell Promise A+ Reactive Programming Generators
  • 15.
    Library Purpose ProCon Suggestion Restify RESTful Server Framework - Light-weight - Very focused for RESTful API Implementation - Less middleware feature - Poor documentation If you fed up with Express JS and only you need is RESTful API Sequelize ORM - Best ORM Library - Well documented - Lack of some GIS feature - Auto sync may bad Be safe to use sync() Underscore Common Utilities - 100% Web compatible - Light and easy to use - Lack of function for arguments object If you not on ES6, use it. Bluebird Promise A+ - 100% Compatible with ES 6 Promise - Best Promise A+ Library - Has some memory leaks - Less documentation If you not on ES6, use it. Mocha TDD/BDD - Best BDD/TDD Framework - Best with should.js - May be old If you have no preference, use it. Passport JS Oauth / Social Login - Best Login Library - Many Strategies - Some strategies may need to be fixed on your own. If you don’t want to build your own login feature, use it. node.js and JS community is very large and rapidly growing, So you have to checkout good libraries at NPM and Github to reduce time to search.
  • 16.
    Many web frameworksexists for node.js but all of them has pro/cons = no superb one. You have to choose right tool for your right purposes. (Avoid Tech Masturbation) Name Express JS Hapi Restify Strongloop Pro • All-in-one (Web/RESTful) • Many 3rd party middlewares • Service by Manifest • Walmart Support • Light weight • Dtrace support • Only for RESTful Service • For Enterprise • Nice dashboard, Profiler • Swagger included Con • Very complex to use/fix • Soon will be decayed • Static content can be served via nginx or apache • Less middleware • Express code cannot be used • Poor documentation • Have to build everything you needed. • Only for Enterprise • Memory leaks • Very heavy • No OSS
  • 17.
    Due to thestructure limitations, node.js may very bad choice for certain applications If you need intensive CPU jobs, consider use Python or Java. (http://imjuni.tistory.com/694) Events are not processed in enough time. Event queue is rapidly grow Memory is growing fast Heavy CPU process/wait in callbacks Segmentation Fault may happen
  • 18.
    V8 Engine GChas two way in memory management, each will executed respectively https://strongloop.com/strongblog/node-js-performance-garbage-collection/ Full GC (Mark and Sweep & Mark and Compact) Short GC (Scavengering)
  • 19.
    Checkout RSS/Heap Size/Heaptotal to checkout memory leaks. Watch your Heap growth pattern before and after GC using Continuous request for period Full GC (Mark and Sweep & Mark and Compact) Short GC (Scavengering) Normal Memory Leaking Segment Fault OS kills app Heap & RSS are kepp growing
  • 20.
    Running server withhigh pressure of request may not help to find out actual problem. Using google chrome dev tools and few other libraries, you can profile your applications Must see: https://www.joyent.com/blog/walmart-node-js-memory-leak Golden rules 1. Narrow down causes 2. Wait, Collect, Compare 3. Check your throught-put 4. Most of all: Build with KISS Wallmart Memory LeakCase (Running for 15 hours )
  • 21.
    Due to theJavascripts less type safe and very dynamic features, Unit Testing is very crucial to assure quality of Application (NO TDD, NO DEV) Type What to Test Goal How to Test Together with Module Test Module To assure quality of module UT/TDD with randominputs Feature Test API To assure quality of each features UT/BDD with random inputs Load Runner Scenario Test Business Logic To assure quality of scenario UT/BDD with random inputs Load Runner You have to run regression Test for all above 3 (If possible). Managing bulk of codes in Unit Testing is not that easy Kanizsa Lab's Software Testing Policy
  • 22.
    Single Mode ClusteredMode PID: 1 PID: 1 master PID: 2 slave 1 PID: 3 slave 2 PID: 1 master PID: 2 slave 1 PID: 3 slave 2 Clustered Mode with Forever Forever To enable scalability in node.js, Clustered Mode is supported (Process Fork) but you have to manage slaves by yourself. Using forever daemon to watch master process status is encouraged. Recommended size of slaves = number of CPUs (But you’d better check out enough memory exists)
  • 23.
    Logging is veryimportant but have to keep a “golden mean” 1) Do not give burden to process 2) Enough to be detailed 3) Machine Understandable 4) Server wide standard format PID: 1 node bunyan CRON JOB Logrotate File AWS S3upload JSON node bunyan is the best library to logging with JSON bunyan is also main logging library of Joyent Cloud Not recommended: Upload to NoSQL directly via app internal library Seems too much in early stages: Using transport system like Flume, Kafka, Fluentd these technology required by enough large clusters Logrotate
  • 24.
    node.js is stillrapidly growing and expanding its supporting platforms from server to embedded devices. the best advantage of node.js: Fast I/O with less memory and easy to start makes it a very fascinating framework to learn for developers. So keep watch it! IoT.js (Forked version of node.js) by Samsung node.js running on Raspberry-Pi as application framework