Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Real World Lessons on the Pain Points of
Node.js Applications
@Ben_Hall
Ben@BenHall.me.uk
OcelotUproar.com / Katacoda.com
@Ben_Hall / Blog.BenHall.me.uk
Tech Support > Tester > Developer >
Founder
Software Development Studio
WHOAMI?
Agenda
• Creating strong foundation
– Node v5, NPM, Security
• Error Handling
• Async/Promises
• Deploying / Scaling
• Per...
Provide overview of the main
point points we’ve
experienced and how we
resolved them
Strong Foundations
Upgrading from Node v0.10.38
to v5.3.0
https://github.com/nodejs/node/wiki/
API-changes-between-v0.10-and-v4
• The domain module has been scheduled for
deprecati...
https://nodejs.org/en/blog/release/v5
.0.0/
• Breaking changes.
• When parsing HTTP, don't add duplicates of the
following...
Docker to test deployment
• Didn’t need to install anything on host
> docker run -it -v $(pwd):/src -p 3000 node:5
root@co...
Default to Underscore.js ?
Fixing NPM
Lock down NPM dependencies
because no-one respects SemVer
AngularJs 1.2 => 1.3
"dependencies": {
"angular": "^1.2.16”
}
Angular 1.2 => 1.3
> angular.element(document)
[#document]
> angular.element(document)
TypeError: undefined is not a funct...
Lock Down Dependencies
Randomly breaking builds and
deployments will occur otherwise
$ npm shrinkwrap
Lock down dependencies to what’s
running locally
Hard code versions in
package.json
"dependencies": {
"angular": “1.2.23”
}
$ npm outdated
.npmrc
• https://docs.npmjs.com/misc/config
> cat .npmrc
save=true
save-exact=true
npm install -g
Replaced Glup, Grunt with Make
• Bugs, Bugs everywhere!
templates:
handlebars views/templates/*.hbs -f public/js/templates.js
> make templates
Security
Cover Your Bases
• Barry Dorrans, Troy Hunt, Niall Merrigan
• Troy Hunt and Barry Dorrans sessions
• A security testers to...
Child Process Exec
child_process.exec(req.query.url, function (err, data) {
console.log(data);
});
https://localhost:49155...
Cross-Site Request Forgery
var csrf = require('csurf');
var csrfProtection = csrf({ cookie: true });
var parseForm = bodyP...
Rate Limiting
var ratelimit = require('koa-ratelimit');
var ipBasedRatelimit = ratelimit({
db: redis.createClient(),
durat...
Security Audit NPM Packages
> npm install nsp
> nsp check
(+) 18 vulnerabilities found
https://nodesecurity.io/
• Root Path Disclosure (2x)
• Regular Expression Denial of Service (10x)
• Incorrect Handling of ...
NPM Credentials Leaks
• https://github.com/ChALkeR/notes/blob/mast
er/Do-not-underestimate-credentials-
leaks.md
Create Strong Foundations
Error Handling
Wasn’t great from the start
Try {} Catch {}
Try {} Catch {}
Domains haven’t really worked
https://raw.githubusercontent.com/strongloop/zone/master/showcase/curl/curl-zone.js
Zones? No. Not really
Returning String as Error
Strongly typed errors
Generators + Error Handling
Async Flow Control
Promises
Promises… Promises… Never break
your promises.
Personally, I never make promises.
http://domenic.me/2012/10/14/youre-missing-the-point-of-promises/
Not part of the Node
Makes integration more difficult.
Makes swapping code in / out more
painful.
Callbacks
So good it’s got it’s own website
callbackhell.com
“The goal isn’t about removing
levels of indentation but rather
writing modular code that is easy to
reason about”
Strongl...
Loops + Async Callbacks
Loops + Async Callbacks
“You can't get into callback hell if
you don't go there.”
Isaac Schlueter
Generators are coming!
See Node >=0.11.2
http://blog.alexmaccaw.com/how-yield-will-transform-node
http://blog.alexmaccaw.com/how-yield-will-transform-node
DEPLOY!
Single Threaded
CPU Intensive?
Array Filtering / Sorting / Processing
Have more threads
Solved Problem?
var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;
if (cluster.isMas...
> NODE_DEBUG=cluster node server.js
23521,Master Worker 23524 online
23521,Master Worker 23526 online
23521,Master Worker ...
Deploying with Docker
Container
https://www.docker.com/whatisdocker/
Container
Deploying Node App via Docker
> cat Dockerfile
FROM node:5-onbuild
EXPOSE 3000
> docker build –t my-node-app .
> docker ru...
59,040
environments/day/server
112,320
environments/day/server
> docker run -d 
-p 80:80 
-v /var/run/docker.sock:/tmp/docker.sock:ro 
jwilder/nginx-proxy
> docker run --name web 
-e VI...
Load Balancer
Nginx Proxy
Node Node
Node Node
Node Node
Nginx Proxy
Node Node
Node Node
Node Node
Nginx Proxy
Node Node
No...
Health Endpoints
router.get('/_specialfunctions/_check', function(req, res) {
async.parallel([
check_docker,
check_starter...
var check_docker = function(cb) {
docker.ping(function(err) { handle_error('docker', err, cb);});
};
var check_redis = fun...
Careful!
socket.io and global state
Sticky Sessions
Compiled Nginx + OSS Modules
Global State as a Service
Microservices FTW!!
Code Performance Still Matters
Performance
var Ocelite = require('ocelite');
var db = new Ocelite();
db.init('data.db', ['user'], function() {
db.save('user', {name:...
> npm install v8-profiler
const profiler = require('v8-profiler')
const fs = require('fs')
var profilerRunning = false
fun...
JetBrains WebStorm
Debugging
require(‘debug’);
Webstorm
VS Code
Summary
• Update to Node.js v5
• Start using ES6
• Security
• Manage your errors
• Forgot making promises
• Scale using Do...
Thank you!
@Ben_Hall
Ben@BenHall.me.uk
Blog.BenHall.me.uk
www.Katacoda.com
www.OcelotUproar.com
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Real World Lessons on the Pain Points of Node.js Applications
Upcoming SlideShare
Loading in …5
×

Real World Lessons on the Pain Points of Node.js Applications

793 views

Published on

Presented at NDC London on 13th January 2016

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Real World Lessons on the Pain Points of Node.js Applications

  1. 1. Real World Lessons on the Pain Points of Node.js Applications @Ben_Hall Ben@BenHall.me.uk OcelotUproar.com / Katacoda.com
  2. 2. @Ben_Hall / Blog.BenHall.me.uk Tech Support > Tester > Developer > Founder Software Development Studio WHOAMI?
  3. 3. Agenda • Creating strong foundation – Node v5, NPM, Security • Error Handling • Async/Promises • Deploying / Scaling • Performance • Debugging
  4. 4. Provide overview of the main point points we’ve experienced and how we resolved them
  5. 5. Strong Foundations
  6. 6. Upgrading from Node v0.10.38 to v5.3.0
  7. 7. https://github.com/nodejs/node/wiki/ API-changes-between-v0.10-and-v4 • The domain module has been scheduled for deprecation, awaiting an alternative for those who absolutely need domains. • fs.exists() is now deprecated. It is suggested to use either fs.access() or fs.stat(). Please read the documentation carefully. • Updated setImmediate() to process the full queue each turn of the event loop, instead of one per queue.
  8. 8. https://nodejs.org/en/blog/release/v5 .0.0/ • Breaking changes. • When parsing HTTP, don't add duplicates of the following headers • HTTP methods and header names must now conform to the RFC 2616 "token" rule • zlib: Decompression now throws on truncated input • buffer: Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time.
  9. 9. Docker to test deployment • Didn’t need to install anything on host > docker run -it -v $(pwd):/src -p 3000 node:5 root@container:> npm install root@container:> npm start
  10. 10. Default to Underscore.js ?
  11. 11. Fixing NPM
  12. 12. Lock down NPM dependencies because no-one respects SemVer
  13. 13. AngularJs 1.2 => 1.3 "dependencies": { "angular": "^1.2.16” }
  14. 14. Angular 1.2 => 1.3 > angular.element(document) [#document] > angular.element(document) TypeError: undefined is not a function
  15. 15. Lock Down Dependencies Randomly breaking builds and deployments will occur otherwise
  16. 16. $ npm shrinkwrap Lock down dependencies to what’s running locally
  17. 17. Hard code versions in package.json "dependencies": { "angular": “1.2.23” }
  18. 18. $ npm outdated
  19. 19. .npmrc • https://docs.npmjs.com/misc/config > cat .npmrc save=true save-exact=true
  20. 20. npm install -g
  21. 21. Replaced Glup, Grunt with Make • Bugs, Bugs everywhere!
  22. 22. templates: handlebars views/templates/*.hbs -f public/js/templates.js > make templates
  23. 23. Security
  24. 24. Cover Your Bases • Barry Dorrans, Troy Hunt, Niall Merrigan • Troy Hunt and Barry Dorrans sessions • A security testers toolkit - Niall Merrigan • https://vimeo.com/131641274 • Going beyond OWASP - Barry Dorrans • https://vimeo.com/131642364
  25. 25. Child Process Exec child_process.exec(req.query.url, function (err, data) { console.log(data); }); https://localhost:49155/api/openUrlInDefaultBrowser?url=c:/windows/sy stem32/calc.exe Thanks TrendMicro Antivirus on Windows! https://code.google.com/p/google-security-research/issues/detail?id=693
  26. 26. Cross-Site Request Forgery var csrf = require('csurf'); var csrfProtection = csrf({ cookie: true }); var parseForm = bodyParser.urlencoded({ extended: false }); app.get('/form', csrfProtection, function(req, res) { res.render('send', { csrfToken: req.csrfToken() }); }); app.post('/process', parseForm, csrfProtection, function(req, res) { res.send('data is being processed'); }); https://blog.risingstack.com/node-js-security-checklist/
  27. 27. Rate Limiting var ratelimit = require('koa-ratelimit'); var ipBasedRatelimit = ratelimit({ db: redis.createClient(), duration: 60000, max: 10, id: function (context) { return context.ip; } }); app.post('/login', ipBasedRatelimit, handleLogin); https://blog.risingstack.com/node-js-security-checklist/
  28. 28. Security Audit NPM Packages > npm install nsp > nsp check (+) 18 vulnerabilities found
  29. 29. https://nodesecurity.io/ • Root Path Disclosure (2x) • Regular Expression Denial of Service (10x) • Incorrect Handling of Non-Boolean Comparisons During Minification • Denial-of-Service Extended Event Loop Blocking • Denial-of-Service Memory Exhaustion • Symlink Arbitrary File Overwrite • Remote Memory Disclosure (2x)
  30. 30. NPM Credentials Leaks • https://github.com/ChALkeR/notes/blob/mast er/Do-not-underestimate-credentials- leaks.md
  31. 31. Create Strong Foundations
  32. 32. Error Handling
  33. 33. Wasn’t great from the start
  34. 34. Try {} Catch {}
  35. 35. Try {} Catch {}
  36. 36. Domains haven’t really worked
  37. 37. https://raw.githubusercontent.com/strongloop/zone/master/showcase/curl/curl-zone.js Zones? No. Not really
  38. 38. Returning String as Error
  39. 39. Strongly typed errors
  40. 40. Generators + Error Handling
  41. 41. Async Flow Control
  42. 42. Promises Promises… Promises… Never break your promises. Personally, I never make promises.
  43. 43. http://domenic.me/2012/10/14/youre-missing-the-point-of-promises/
  44. 44. Not part of the Node Makes integration more difficult. Makes swapping code in / out more painful.
  45. 45. Callbacks So good it’s got it’s own website callbackhell.com
  46. 46. “The goal isn’t about removing levels of indentation but rather writing modular code that is easy to reason about” Strongloop Blog http://strongloop.com/strongblog/node-js-callback-hell-promises-generators/
  47. 47. Loops + Async Callbacks
  48. 48. Loops + Async Callbacks
  49. 49. “You can't get into callback hell if you don't go there.” Isaac Schlueter
  50. 50. Generators are coming! See Node >=0.11.2
  51. 51. http://blog.alexmaccaw.com/how-yield-will-transform-node
  52. 52. http://blog.alexmaccaw.com/how-yield-will-transform-node
  53. 53. DEPLOY!
  54. 54. Single Threaded
  55. 55. CPU Intensive? Array Filtering / Sorting / Processing
  56. 56. Have more threads
  57. 57. Solved Problem?
  58. 58. var cluster = require('cluster'); var http = require('http'); var numCPUs = require('os').cpus().length; if (cluster.isMaster) { // Fork workers. for (var i = 0; i < numCPUs; i++) { cluster.fork(); } cluster.on('exit', function(worker, code, signal) { console.log('worker ' + worker.process.pid + ' died'); }); } else { // Workers can share any TCP connection // In this case it is an HTTP server http.createServer(function(req, res) { res.writeHead(200); res.end("hello worldn"); }).listen(8000); }
  59. 59. > NODE_DEBUG=cluster node server.js 23521,Master Worker 23524 online 23521,Master Worker 23526 online 23521,Master Worker 23523 online 23521,Master Worker 23528 online
  60. 60. Deploying with Docker
  61. 61. Container https://www.docker.com/whatisdocker/ Container
  62. 62. Deploying Node App via Docker > cat Dockerfile FROM node:5-onbuild EXPOSE 3000 > docker build –t my-node-app . > docker run –p 3000:3000 my-node-app
  63. 63. 59,040 environments/day/server
  64. 64. 112,320 environments/day/server
  65. 65. > docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy > docker run --name web -e VIRTUAL_HOST=www.katacoda.com my-node-app
  66. 66. Load Balancer Nginx Proxy Node Node Node Node Node Node Nginx Proxy Node Node Node Node Node Node Nginx Proxy Node Node Node Node Node Node
  67. 67. Health Endpoints router.get('/_specialfunctions/_check', function(req, res) { async.parallel([ check_docker, check_starter, check_redis, check_pg ], function(err, results) { if(err) { console.log("Health check failed", err); res.status(500); return res.json({healthy: false, details: err}); } res.json({healthy: true}); }) });
  68. 68. var check_docker = function(cb) { docker.ping(function(err) { handle_error('docker', err, cb);}); }; var check_redis = function(cb) { redis.status(function(err, connected) { if(err === null && connected === "ready") { cb(); } else { handle_error('redis', {msg: 'Not Connected', err: err}, cb); } }) }; var check_pg = function(cb) { pg.status(function(err) { handle_error('postgres', err, cb);}); };
  69. 69. Careful! socket.io and global state
  70. 70. Sticky Sessions Compiled Nginx + OSS Modules
  71. 71. Global State as a Service Microservices FTW!!
  72. 72. Code Performance Still Matters
  73. 73. Performance
  74. 74. var Ocelite = require('ocelite'); var db = new Ocelite(); db.init('data.db', ['user'], function() { db.save('user', {name: 'Barbara Fusinska', twitter: 'basiafusinska'}, ['twitter'], function() { db.get('user', 'twitter', 'basiafusinska', function(err, obj) { console.log(obj); }); }); });
  75. 75. > npm install v8-profiler const profiler = require('v8-profiler') const fs = require('fs') var profilerRunning = false function toggleProfiling () { if (profilerRunning) { const profile = profiler.stopProfiling() console.log('stopped profiling') profile.export() .pipe(fs.createWriteStream('./myapp-'+Date.now()+'.cpuprofile')) .once('error', profiler.deleteAllProfiles) .once('finish', profiler.deleteAllProfiles) profilerRunning = false return } profiler.startProfiling() profilerRunning = true console.log('started profiling') } process.on('SIGUSR2', toggleProfiling) > kill -SIGUSR2 <pid>
  76. 76. JetBrains WebStorm
  77. 77. Debugging
  78. 78. require(‘debug’);
  79. 79. Webstorm
  80. 80. VS Code
  81. 81. Summary • Update to Node.js v5 • Start using ES6 • Security • Manage your errors • Forgot making promises • Scale using Docker
  82. 82. Thank you! @Ben_Hall Ben@BenHall.me.uk Blog.BenHall.me.uk www.Katacoda.com www.OcelotUproar.com

×