Node.js Patterns for Discerning Developers
Upcoming SlideShare
Loading in...5
×
 

Node.js Patterns for Discerning Developers

on

  • 11,971 views

Slides from my talk "Node.js Patterns for Discerning Developers" given at Pittsburgh TechFest 2013. This talk detailed common design pattern for Node.js, as well as common anti-patterns to avoid.

Slides from my talk "Node.js Patterns for Discerning Developers" given at Pittsburgh TechFest 2013. This talk detailed common design pattern for Node.js, as well as common anti-patterns to avoid.

Statistics

Views

Total Views
11,971
Views on SlideShare
11,872
Embed Views
99

Actions

Likes
29
Downloads
57
Comments
3

4 Embeds 99

https://twitter.com 65
http://blog.hengkiardo.com 31
http://www.linkedin.com 2
http://www.google.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • @Adam Good points! Actually, when I give the talk I specifically mention both of those things (setTimeout being async and to actually handle errors), but you are right, it doesn't come across with just the slides. I'm currently updating the talk for next month, so I appreciate the feedback!
    Are you sure you want to
    Your message goes here
    Processing…
  • Dude, this presentation is chock-full of incorrect information--I don't even know where to start. Things like 'setTimeout bringing your script to a grinding halt' is not only categorically not true, it's probably hurting new developers who find this. Whilst it is true that the app is single-threaded and that the app will execute the method in the timeout, it will continue to execute code until said timeout occurs.

    if (err) { return console.log(err); } is an anti-pattern, because it's gibberish code that does literally nothing but abort your handler's execution. console.log() doesn't even return anything, making your example that much worse. It's also categorically bad practice to not rethrow your errors if you're not going to handle them in the callback. If you're going to teach noobs some patterns, perhaps you should teach them the right patterns.
    Are you sure you want to
    Your message goes here
    Processing…
  • good stuff, thx.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Images from http://wallpapersus.com/
  • Note: enumerable is not set to true by default for inherited attributes. You can change this, but its often not worth it
  • New in ecmascript 5. <br /> <br /> Problem: you can’t pass in constructor attributes within Object.create.
  • What happens when we add error handling?
  • Callback contains an object filled with results from each function
  • Each function gets input from the last function
  • Map returns an array of RESULTS, one for each input object/value
  • Map returns a sub-array of objects/values form the input array, filtered by a boolean test function <br />

Node.js Patterns for Discerning Developers Node.js Patterns for Discerning Developers Presentation Transcript

  • Node.js Patterns For the Discerning Developer C. Aaron Cois, Ph.D. :: Carnegie Mellon University, SEI
  • Me @aaroncois www.codehenge.net github.com/cacois Disclaimer: Though I am an employee of the Software Engineering Institute at Carnegie Mellon University, this work was not funded by the SEI and does not reflect the work or opinions of the SEI or its customers.
  • Let’s talk about
  • Node.js Basics • JavaScript • Asynchronous • Non-blocking I/O • Event-driven
  • So, JavaScript?
  • The Basics
  • Prototype-based Programming • JavaScript has no classes • Instead, functions define objects function Person() {} var p = new Person(); Image: http://tech2.in.com/features/gaming/five-wacky-gaming-hardware-to-look-forward-to/315742 Prototype
  • Classless Programming What do classes do for us? • Define local scope / namespace • Allow private attributes / methods • Encapsulate code • Organize applications in an object-oriented way
  • Prototype-based Programming function Person(firstname, lastname){ this.firstname = firstname; this.lastname = lastname; } var p = new Person(“Philip”, “Fry”); What else can do that?
  • Prototype Inheritance function Person(firstname, lastname){ this.firstname = firstname; this.lastname = lastname; } // Create new class Employee = Person;//Inherit from superclass Employee.prototype = { marital_status: 'single', salute: function() { return 'My name is ' + this.firstname; } } var p = new Employee (“Philip”, “Fry”);
  • Watch out! function Person(firstname, lastname){ this.firstname = firstname; this.lastname = lastname; } // Create new class Employee = Person;//Inherit from superclass Employee.prototype = { marital_status: 'single', salute: function() { return 'My name is ' + this.firstname; } } var p = new Employee (“Philip”, “Fry”); The ‘new’ is very important! If you forget, your new object will have global scope internally
  • Another option function Person(firstname, lastname){ this.firstname = firstname; this.lastname = lastname; } Employee = Person;//Inherit from superclass Employee.prototype = { marital_status: 'single', salute: function() { return 'My name is ' + this.firstname; } } var p = Object.create(Employee); p.firstname = 'Philip'; p.lastname = 'Fry'; Works, but you can’t initialize attributes in constructor
  • Anti-Pattern: JavaScript Imports • Spread code around files • Link libraries • No way to maintain private local scope/state/namespace • Leads to: – Name collisions – Unnecessary access
  • Pattern: Modules • An elegant way of encapsulating and reusing code • Adapted from YUI, a few years before Node.js • Takes advantage of the anonymous closure features of JavaScript Image: http://wallpapersus.com/
  • Modules in the Wild var http = require('http'), io = require('socket.io'), _ = require('underscore'); If you’ve programmed in Node, this looks familiar
  • Anatomy of a module var privateVal = 'I am Private!'; module.exports = { answer: 42, add: function(x, y) { return x + y; } } mymodule.js
  • Usage mod = require('./mymodule'); console.log('The answer: '+ mod.answer); var sum = mod.add(4,5); console.log('Sum: ' + sum);
  • Modules are used everywhere // User model var mongoose = require('mongoose') , Schema = mongoose.Schema; var userSchema = new Schema({ name: {type: String, required: true}, email: {type: String, required: true}, githubid: String, twitterid: String, dateCreated: {type: Date, default: Date.now} }); userSchema.methods.validPassword = function validPass(pass) { // validate password… } module.exports = mongoose.model('User', userSchema);
  • My config files? Modules. var config = require('config.js'); console.log('Configured user is: ' + config.user); module.exports = { user: 'maurice.moss' } config.js app.js
  • Asynchronous
  • Asynchronous Programming • Node is entirely asynchronous • You have to think a bit differently • Failure to understand the event loop and I/O model can lead to anti-patterns
  • Event Loop Node.js Event Loop Node app
  • Event Loop Node.js Event Loop Node apps pass async tasks to the event loop, along with a callback (function, callback) Node app
  • Event Loop Node.js Event Loop The event loop efficiently manages a thread pool and executes tasks efficiently… Thread 1 Thread 2 Thread n … Task 1 Task 2 Task 3 Task 4 Return 1 Callback1() …and executes each callback as tasks complete Node app
  • Async I/O The following tasks should be done asynchronously, using the event loop: • I/O operations • Heavy computation • Anything requiring blocking
  • Your Node app is single-threaded
  • Anti-pattern: Synchronous Code for (var i = 0; i < 100000; i++){ // Do anything } Your app only has one thread, so: …will bring your app to a grinding halt
  • Anti-pattern: Synchronous Code But why would you do that? Good question. But in other languages (Python), you may do this: for file in files: f = open(file, ‘r’) print f.readline()
  • Anti-pattern: Synchronous Code The Node.js equivalent is: Based on examples from: https://github.com/nodebits/distilled-patterns/ var fs = require('fs'); for (var i = 0; i < files.length; i++){ data = fs.readFileSync(files[i]); console.log(data); } …and it will cause severe performance problems
  • Pattern: Async I/O fs = require('fs'); fs.readFile('f1.txt','utf8',function(err,data){ if (err) { // handle error } console.log(data); });
  • Async I/O fs = require('fs'); fs.readFile('f1.txt','utf8',function(err,data){ if (err) { // handle error } console.log(data); }); Anonymous, inline callback
  • Async I/O fs = require('fs'); fs.readFile('f1.txt','utf8', function(err,data){ if (err) { // handle error } console.log(data); } ); Equivalent syntax
  • Callback Hell When working with callbacks, nesting can get quite out of hand…
  • Callback Hell var db = require('somedatabaseprovider'); //get recent posts http.get('/recentposts', function(req, res) { // open database connection db.openConnection('host', creds,function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],function(err,users){ conn.close(); res.send(users[0]); }); } }); });
  • Callback Hell var db = require('somedatabaseprovider'); //get recent posts http.get('/recentposts', function(req, res) { // open database connection db.openConnection('host', creds,function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],function(err,users){ conn.close(); res.send(users[0]); }); } }); }); Get recent posts from web service API
  • Callback Hell var db = require('somedatabaseprovider'); //get recent posts http.get('/recentposts', function(req, res) { // open database connection db.openConnection('host', creds,function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],function(err,users){ conn.close(); res.send(users[0]); }); } }); }); Open connection to DB
  • Callback Hell var db = require('somedatabaseprovider'); //get recent posts http.get('/recentposts', function(req, res) { // open database connection db.openConnection('host', creds,function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],function(err,users){ conn.close(); res.send(users[0]); }); } }); }); Get user from DB for each post
  • Callback Hell var db = require('somedatabaseprovider'); //get recent posts http.get('/recentposts', function(req, res) { // open database connection db.openConnection('host', creds,function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],function(err,users){ conn.close(); res.send(users[0]); }); } }); }); Return users
  • Callback Hell var db = require('somedatabaseprovider'); //get recent posts http.get('/recentposts', function(req, res) { // open database connection db.openConnection('host', creds,function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],function(err,users){ conn.close(); res.send(users[0]); }); } }); });
  • Anti-Pattern: Callback Hell fs.readdir(source, function(err, files) { if (err) { console.log('Error finding files: ' + err) } else { files.forEach(function(filename, fileIndex) { console.log(filename) gm(source + filename).size(function(err, values) { if (err) { console.log('Error identifying file size: ' + err) } else { console.log(filename + ' : ' + values) aspect = (values.width / values.height) widths.forEach(function(width, widthIndex) { height = Math.round(width / aspect) console.log('resizing ' + filename + 'to ' + height + 'x' + height) this.resize(width, height).write(destination+'w’+width+'_’+filename, function(err){ if (err) console.log('Error writing file: ' + err) }) }.bind(this)) } }) }) } }) http://callbackhell.com/
  • Solutions • Separate anonymous callback functions (cosmetic) • Async.js • Promises • Generators
  • Pattern: Separate Callbacks fs = require('fs'); callback = function(err,data){ if (err) { // handle error } console.log(data); } fs.readFile('f1.txt','utf8',callback);
  • Can Turn This var db = require('somedatabaseprovider'); http.get('/recentposts', function(req, res){ db.openConnection('host', creds, function(err, conn){ res.param['posts'].forEach(post) { conn.query('select * from users where id=' + post['user'],function(err,results){ conn.close(); res.send(results[0]); }); } }); });
  • Into This var db = require('somedatabaseprovider'); http.get('/recentposts', afterRecentPosts); function afterRecentPosts(req, res) { db.openConnection('host', creds, function(err, conn) { afterDBConnected(res, conn); }); } function afterDBConnected(err, conn) { res.param['posts'].forEach(post) { conn.query('select * from users where id='+post['user'],afterQuery); } } function afterQuery(err, results) { conn.close(); res.send(results[0]); }
  • This is really a Control Flow issue
  • Pattern: Async.js Async.js provides common patterns for async code control flow https://github.com/caolan/async Also provides some common functional programming paradigms
  • Serial/Parallel Functions • Sometimes you have linear serial/parallel computations to run, without branching callback growth Function 1 Function 2 Function 3 Function 4 Function 1 Function 2 Function 3 Function 4
  • Serial/Parallel Functions async.parallel([ function(){ ... }, function(){ ... } ], callback); async.series([ function(){ ... }, function(){ ... } ]);
  • Serial/Parallel Functions async.parallel([ function(){ ... }, function(){ ... } ], callback); async.series([ function(){ ... }, function(){ ... } ], callback); Single Callback
  • Waterfall Async.waterfall([ function(callback){ ... }, function(input,callback){ ... }, function(input,callback){ ... }, ], callback);
  • Map var arr = ['file1','file2','file3']; async.map(arr, fs.stat, function(err, results){ // results is an array of stats for each file console.log('File stats: ' + JSON.stringify(results)); });
  • Filter var arr = ['file1','file2','file3']; async.filter(arr, fs.exists, function(results){ // results is a list of the existing files console.log('Existing files: ' + results); });
  • With great power…
  • Carefree var fs = require('fs'); for (var i = 0; i < 10000; i++) { fs.readFileSync(filename); } With synchronous code, you can loop as much as you want: The file is opened once each iteration. This works, but is slow and defeats the point of Node.
  • Synchronous Doesn’t Scale What if we want to scale to 10,000+ concurrent users? File I/O becomes the bottleneck Users get in a long line
  • Async to the Rescue var fs = require('fs'); function onRead(err, file) { if (err) throw err; } for (var i = 0; i < 10000; i++) { fs.readFile(filename, onRead); } What happens if I do this asyncronously?
  • Ruh Roh The event loop is fast This will open the file 10,000 times at once This is unnecessary…and on most systems, you will run out of file descriptors!
  • Pattern: The Request Batch • One solution is to batch requests • Piggyback on existing requests for the same file • Each file then only has one open request at a time, regardless of requesting clients
  • // Batching wrapper for fs.readFile() var requestBatches = {}; function batchedReadFile(filename, callback) { // Is there already a batch for this file? if (filename in requestBatches) { // if so, push callback into batch requestBatches[filename].push(callback); return; } // If not, start a new request var callbacks = requestBatches[filename] = [callback]; fs.readFile(filename, onRead); // Flush out the batch on complete function onRead(err, file) { delete requestBatches[filename]; for(var i = 0;i < callbacks.length; i++) { // execute callback, passing arguments along callbacks[i](err, file); } } } Based on examples from: https://github.com/nodebits/distilled-patterns/
  • // Batching wrapper for fs.readFile() var requestBatches = {}; function batchedReadFile(filename, callback) { // Is there already a batch for this file? if (filename in requestBatches) { // if so, push callback into batch requestBatches[filename].push(callback); return; } // If not, start a new request var callbacks = requestBatches[filename] = [callback]; fs.readFile(filename, onRead); // Flush out the batch on complete function onRead(err, file) { delete requestBatches[filename]; for(var i = 0;i < callbacks.length; i++) { // execute callback, passing arguments along callbacks[i](err, file); } } } Based on examples from: https://github.com/nodebits/distilled-patterns/ Is this file already being read?
  • // Batching wrapper for fs.readFile() var requestBatches = {}; function batchedReadFile(filename, callback) { // Is there already a batch for this file? if (filename in requestBatches) { // if so, push callback into batch requestBatches[filename].push(callback); return; } // If not, start a new request var callbacks = requestBatches[filename] = [callback]; fs.readFile(filename, onRead); // Flush out the batch on complete function onRead(err, file) { delete requestBatches[filename]; for(var i = 0;i < callbacks.length; i++) { // execute callback, passing arguments along callbacks[i](err, file); } } } Based on examples from: https://github.com/nodebits/distilled-patterns/ If not, start a new file read operation
  • // Batching wrapper for fs.readFile() var requestBatches = {}; function batchedReadFile(filename, callback) { // Is there already a batch for this file? if (filename in requestBatches) { // if so, push callback into batch requestBatches[filename].push(callback); return; } // If not, start a new request var callbacks = requestBatches[filename] = [callback]; fs.readFile(filename, onRead); // Flush out the batch on complete function onRead(err, file) { delete requestBatches[filename]; for(var i = 0;i < callbacks.length; i++) { // execute callback, passing arguments along callbacks[i](err, file); } } } Based on examples from: https://github.com/nodebits/distilled-patterns/ When read finished, return to all requests
  • Usage //Request the resource 10,000 times at once for (var i = 0; i < 10000; i++) { batchedReadFile(file, onComplete); } function onComplete(err, file) { if (err) throw err; else console.log('File contents: ' + file); } Based on examples from: https://github.com/nodebits/distilled-patterns/
  • Pattern: The Request Batch This pattern is effective on many read-type operations, not just file reads Example: also good for web service API calls
  • Shortcomings Batching requests is great for high request spikes Often, you are more likely to see steady requests for the same resource This begs for a caching solution
  • Pattern: Request Cache Let’s try a simple cache Persist the result forever and check for new requests for same resource
  • // Caching wrapper around fs.readFile() var requestCache = {}; function cachingReadFile(filename, callback) { //Do we have resource in cache? if (filename in requestCache) { var value = requestCache[filename]; // Async behavior: delay result till next tick process.nextTick(function () { callback(null, value); }); return; } // If not, start a new request fs.readFile(filename, onRead); // Cache the result if there is no error function onRead(err, contents) { if (!err) requestCache[filename] = contents; callback(err, contents); } } Based on examples from: https://github.com/nodebits/distilled-patterns/
  • Usage // Request the file 10,000 times in series // Note: for serial requests we need to iterate // with callbacks, rather than within a loop var its = 10000; cachingReadFile(file, next); function next(err, contents) { console.log('File contents: ' + contents); if (!(its--)) return; cachingReadFile(file, next); } Based on examples from: https://github.com/nodebits/distilled-patterns/
  • Almost There! You’ll notice two issues with the Request Cache as presented: • Concurrent requests are an issue again • Cache invalidation not handled Let’s combine cache and batch strategies:
  • // Wrapper for both caching and batching of requests var requestBatches = {}, requestCache = {}; function readFile(filename, callback) { if (filename in requestCache) { // Do we have resource in cache? var value = requestCache[filename]; // Delay result till next tick to act async process.nextTick(function () { callback(null, value); }); return; } if (filename in requestBatches) {// Else, does file have a batch? requestBatches[filename].push(callback); return; } // If neither, create new batch and request var callbacks = requestBatches[filename] = [callback]; fs.readFile(filename, onRead); // Cache the result and flush batch function onRead(err, file) { if (!err) requestCache[filename] = file; delete requestBatches[filename]; for (var i=0;i<callbacks.length;i++) { callbacks[i](err, file); } } } Based on examples from: https://github.com/nodebits/distilled-patterns/
  • scale-fs I wrote a module for scalable File I/O https://www.npmjs.org/package/scale-fs Usage: var fs = require(’scale-fs'); for (var i = 0; i < 10000; i++) { fs.readFile(filename); }
  • Final Thoughts Most anti-patterns in Node.js come from: • Sketchy JavaScript heritage • Inexperience with Asynchronous Thinking Remember, let the Event Loop do the heavy lifting!
  • Thanks Code samples from this talk at: https://github.com/cacois/node- patterns-discerning
  • Disclaimer Though I am an employee of the Software Engineering Institute at Carnegie Mellon University, this wok was not funded by the SEI and does not reflect the work or opinions of the SEI or its customers.
  • Let’s chat @aaroncois www.codehenge.net github.com/cacois
  • Node.js Event Loop The event loop efficiently manages a thread pool and executes tasks efficiently… Thread 1 Thread 2 Thread n … Task 1 Task 2 Task 3 Task 4 Return 1 Callback1() …and executes each callback as tasks complete Node.js app Node apps pass async tasks to the event loop, along with a callback (function, callback) 1 2 3