Your SlideShare is downloading. ×
Practical Use of MongoDB for Node.js
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Practical Use of MongoDB for Node.js

20,138

Published on

Presentation I gave at MongoDC in June 2012 on practical use of MongoDB in node.js

Presentation I gave at MongoDC in June 2012 on practical use of MongoDB in node.js

Published in: Technology
0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
20,138
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
242
Comments
0
Likes
19
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Transcript

    • 1. Practical Use of MongoDB for Node.js Or: Going Faster with Node.js and MongoDB MongoDC June, 2012 Jonathan Altman @async_io
    • 2. Going Faster with Node.js and MongoDB• Getting started with Node.js and MongoDB development fast• Building a web application with Node.js and MongoDB fast• Making your Node.js/MongoDB-powered web application faster, better, stronger
    • 3. What is Node.js?
    • 4. To a Front-End Developer:• node: • jQuery:var util = require(util), $(.station_map).live("pagecreate", function() {" path = require(path),     if(navigator.geolocation) {" fs = require(fs);       navigator.geolocation.getCurrentPosition(functi on(position){var fileName = process.argv[2];         initializeMap(position.coords.latitude,positipath.exists(fileName, function(exists) { on.coords.longitude);" if (!exists) {       });" " util.puts(fileName + not found);     }" " return;   });" } // snippet taken from https://github.com/mikeymckay/" Capital-Bikeshare/blob/master/capitalbikeshare.js" util.puts(w00t! + fileName + exists!);});// https://gist.github.com/2988106 Doesn’t the code look just like the event-driven javascript DHTML- based web apps have been written in for years?
    • 5. Front-End Similarity is Cool• Node, like front-end web development, is built from the ground-up to be asynchronous javascript coding• Specifically, a lot of node.js’ power is its ability to deal with lots of activity where most of it is just waiting on I/O: user input, network, database, filesystem• This capability is based on the very assumptions Ryan Dahl, node.js’ creator, made in building the system• But the asynchronous nature imposes some costs in terms of coding style and library design: you must buy in to the asynchronous
    • 6. To A Network Admin:var http = require(http);http.createServer(function(request, response) { var proxy = http.createClient(80, request.headers[host]) var proxy_request = proxy.request(request.method, request.url, request.headers); proxy_request.addListener(response, function (proxy_response) { proxy_response.addListener(data, function(chunk) { response.write(chunk, binary); }); proxy_response.addListener(end, function() { response.end(); }); response.writeHead(proxy_response.statusCode, proxy_response.headers); }); request.addListener(data, function(chunk) { proxy_request.write(chunk, binary); }); request.addListener(end, function() { proxy_request.end(); });}).listen(8080);// from http://www.catonmat.net/http-proxy-in-nodejs/
    • 7. Why Node Works Well for Networked Services• Async nature and native support for sockets/http makes it easy to write performant proxies• The sample on the previous slide is a fully-functioning http proxy, 20 lines of code
    • 8. To A Site With Many Simultaneous Clients // server.jsvar app = require(http).createServer(handler), io = require(socket.io).listen(app), fs = require(fs)// Start an HTTP server on port 8080app.listen(8080);function handler(req, res) { // Hardcode *all* HTTP requests to this server to serve up index.html fs.readFile( __dirname + /index.html, function (err, data) { if (err) { res.writeHead(500); return res.end(Error loading index.html); } res.writeHead(200); res.end(data); } );}// After any socket connects, SEND it a custom news eventio.sockets.on( connection, function (socket) { socket.emit(news, {hello: world}); }); // from http://www.zachstronaut.com/posts/2011/08/17/node-socket-io-curious.html
    • 9. How Node.js Works Well for Simultaneous Clients• Implements websockets, which is a standard for handling more interactive web connectivity• Frequently used by game and chat environments• Asynchronous nature again makes it easy to handle lots of connections because they can be multiplexed easily since they just become one more resource like disk, backend network, and database to wait on• Like many other things, support for websockets was easy because node enables support out of the box for lots of libs/modules
    • 10. What Does Node.js Give Me?• It is very good/fast when your application spends time waiting on expensive resources: networks, databases, disks• Javascript is nice, programming model familiar to some• Good libraries for many things• Headaches and/or slowdowns when your bottlenecks are not already handled by node.js• You must think and code in asynchronous javascript/callbacks
    • 11. Example of a Headache• If you have something that doesn’t have a good async module to do what you need, and it’s resource intensive, then you have to build the library to do it...asynchronously• HeatNode: https://github.com/jonathana/heatNode is web middleware I wrote. Part of it does some intensive image processing• Image processing is slow, and this code abuses node.js’ processing• Also, if you do have code that needs ordering or sequencing, you will need to incorporate a library like promise or async to handle that for you
    • 12. What is MongoDB?• Really? You’re at a MongoDB conference
    • 13. What is MongoDB?• Really? You’re at a MongoDB conference• Seriously: “MongoDB is an open source, document-oriented database designed with both scalability and developer agility in mind. Instead of storing your data in tables and rows as you would with a relational database, in MongoDB you store JSON-like documents with dynamic schemas.” http://www.10gen.com/what-is-mongodb
    • 14. What is MongoDB to node.js? It is a high-fidelity datastore for node.js
    • 15. What Do I Mean By High Fidelity?• High fidelity on several axes:• Emphasis on speed• Similarity of processing model: node.js async meshes well with mongodb fire-and-forget insert/update• Plasticity: node.js is built using scripting languages, MongoDB is document- oriented and therefore schema-less• JSON as the lingua franca data format in both node.js and MongoDB (ignoring the BSON internally). It’s JSON all the way down—until you hit the turtles
    • 16. Getting Started Faster with Node.jsWe are going to cover getting your environment up and running
    • 17. Use nave for virtual environmentsjonathan@ogmore ~ $ nave use stable######################################################################## 100.0%fetched from http://nodejs.org/dist/v0.8.0/node-v0.8.0.tar.gz variables: { host_arch: x64, node_install_npm: true, node_use_openssl: true, strict_aliasing: true, target_arch: x64, v8_use_snapshot: true}}# Several hundreds of lines of build output deleted...cp -rf out/Release/node /home/jonathan/.nave/installed/0.8.0/bin/node# some more stuff deletedln -sf ../lib/node_modules/npm/bin/npm-cli.js /home/jonathan/.nave/installed/0.8.0/bin/npmshebang #!/home/jonathan/.nave/installed/0.8.0/bin/node /home/jonathan/.nave/installed/0.8.0/lib/node_modules/npm/bin/npm-cli.jsusing 0.8.0jonathan@ogmore ~ $ exitexitjonathan@ogmore ~ $ nave use 0.6.18Already installed: 0.6.18using 0.6.18jonathan@ogmore ~ $
    • 18. Why nave?• Allows you to manage versioning of your node install• Similar to python’s virtualenv or ruby’s rbenv• You can build multiple predictable environments for your code• Example shows installing 0.8.0, but 0.6.19 was stable up until the day I ran that command• My example code needed 0.6.18. Nave is good, QED• There is also nvm, but I like the fidelity to virtualenv and rbenv
    • 19. npm Manages Packages• Similar to ruby’s rubygems or python’s pip/easy_install• 2 main modes: global and local• You will want some packages globally, but most locally• Rule of thumb: global for packages usable from command line or generally applicable, local for everything else• npm install -g node-inspector # enables graphical debugging• npm install -g express # this is a web framework, more in a second
    • 20. Getting Started Faster with aNode.js + MongoDB Web Stack
    • 21. You Have Some Choices• Connect/Express: these are the granddaddy• Geddy: looks cool to me• Mojito: From Yahoo, tight integration with YUI client AND server• Derby: newer, emerging, interesting• Meteor: newer, emerging, interesting• RailwayJS or TowerJS: Add more full-stack web like Rails or Django on top of express and connect, respectively
    • 22. But...everyauth
    • 23. Ecosystem, Ecosystem, Ecosystem• Everyauth is a canonical example (previous slide hid lots of authentication methods it supports, btw)• There are npm packages for everything for connect/express. I might be exaggerating about everything• RailwayJS is built on top of express• TowerJS is built on top of connect• We will go with express: it does not provide a full MVC framework, but rolling your own on top is fairly easy
    • 24. Adding MongoDB to Our Stack• connect-mongodb: web framework middleware for a MongoDB- backed session• node-mongodb-native: native node driver for MongoDB• mongolia: “non-magic” layer on the mongodb native driver• mongoose: Javascript<->MongoDB object mapper• mongoose-auth: everyauth integration with mongoose. It supports a subset of everyauth’s authentication systems• jugglingdb: multi-dbms support, including MongoDB. However, this adds another layer of idirection on top of Mongoose
    • 25. My Example Stack Choice• express• mongoose + connect-mongodb: not a slam dunk choice, but it is fast to get started• mongoose-auth+everyauth: everyauth integration with mongoose• ejs: embedded javascript page templating <% // yay! %>. Syntax may not be everybody’s cup of tea, but it does make it easy to adapt existing static HTML• bootstrap: twitter’s ui toolkit. Works very well with single-page apps, local caching on full page loads should help, and again we get to start up fast
    • 26. And Away We Go...jonathan@ogmore /mnt/hgfs/src $ express create expspreleton create : expspreleton create : expspreleton/package.json create : expspreleton/app.js [...] create : expspreleton/public/images dont forget to install dependencies: $ cd expspreleton && npm installjonathan@ogmore /mnt/hgfs/src $ ls expsreletonapp.js package.json public routes views
    • 27. Now Add Extra Packages• package.json is where you specify what packages you’ll depend on• It is like Rails’ Gemfile• mongoose and mongoose-auth give us access to MongoDB• I snuck ejs in here too
    • 28. Adding MongoDB (+ejs){ "name": "application-name" , "version": "0.0.1" , "private": true , "dependencies": { "express": "2.5.10" , "ejs": ">= 0.0.1" , "mongoose-auth": ">= 0.0.12" , "mongoose": ">=2.4.8" , "everyauth": ">=0.2.28" }}
    • 29. Install the Packages Locallyjonathan@ogmore /mnt/hgfs/src $ cd expspreleton && npm installnpm installnpm http GET https://registry.npmjs.org/mongoose-authnpm http GET https://registry.npmjs.org/express/2.5.10[...]ejs@0.7.2 ./node_modules/ejsexpress@2.5.10 ./node_modules/express├── qs@0.4.2[...]└── connect@1.9.1 (formidable@1.0.11)everyauth@0.2.32 ./node_modules/everyauth├── node-wsfederation@0.1.1[...]└── connect@1.9.1 (mime@1.2.5, formidable@1.0.11, qs@0.5.0)mongoose-auth@0.0.12 ./node_modules/mongoose-auth├── mongoose-types@1.0.3└── bcrypt@0.5.0mongoose@2.7.0 ./node_modules/mongoose├── hooks@0.2.1└── mongodb@1.0.2 (bson@0.0.6)jonathan@ogmore /mnt/hgfs/src/expspreleton $ls node_modulesejs everyauth express mongoose mongoose-auth
    • 30. Build It!• Set up bootstrap static resources• Set up a bootstrap layout and index page template• Add conf.js• Wire conf.js into app.js• Add mongoose-auth to app.js• Build models
    • 31. Set Up Static Bootstrap Resources• Download bootstrap from http://twitter.github.com/bootstrap/• Unpack it to your public directory• Personal preference: I put it inside node.js’ public directory at public/tbs: jonathan@ogmore /mnt/hgfs/src/expspreleton $ ls public/tbs css img js
    • 32. Setup Bootstrap Layout and Index Page• Bootstrap’s examples page is a good place to go to get nice starting templates: http://twitter.github.com/bootstrap/ examples.html. Click on the examples to view, and save the view source• The bulk of the page becomes your layout.ejs. Then you find the part that changes and it becomes your index.ejs--and eventually the other pages: jonathan@ogmore /mnt/hgfs/src/expspreleton $ ls views index.ejs layout.ejs
    • 33. Add a conf.js File• Good place to store application-specific configuration• Frequently not a good file to put into code repository due to environment (production versus development) differences where you do not want the production values visible• Example: jonathan@ogmore /mnt/hgfs/src/expspreleton $ cat conf.js module.exports = { sessionSecret: YouHadBetterNotStoreYourSecretThisWay , google: { clientId: _your_client_id_goes_here_ , clientSecret: _your_client_secret_goes_here_ } };
    • 34. Wire in conf.js• app.js is the “main” node.js file in an express app• You can also see some other setup stuff /** * Module dependencies. */ var express = require(express) , routes = require(./routes); var conf = require(./conf);
    • 35. Add Authentication with mongoose-auth• See the app.js in the example application in mongoose-auth source code: https://github.com/bnoguchi/ mongoose-auth/blob/master/example/server.js• The magic here is mongoose-auth wiring itself into the UserSchema• Here is the crux part, but what happens below this snippet is your configuration part: var everyauth = require(everyauth) , Promise = everyauth.Promise; everyauth.debug = true; var mongoose = require(mongoose) , Schema = mongoose.Schema , ObjectId = mongoose.SchemaTypes.ObjectId; var UserSchema = new Schema({}) , User; var mongooseAuth = require(mongoose-auth); UserSchema.plugin(mongooseAuth, {
    • 36. Enable and Use Authentication• The everyauth code here is what is doing the authentication checking• Here is some example code that implements a login button: <div class="btn-group pull-right"> <%if (!everyauth.loggedIn) { %> <a class="btn" href="/auth/google"> " <i class="icon-cog"></i>Log in <% } else { %> <a class="btn" href="/logout"> <i class="icon-user"></i> <%= everyauth.google.user.name %> <% } %> </a> </div>
    • 37. Make Your Other Schemas• For example, create and add into your app a Link schema. This is the Link.js module: var mongoose = require(mongoose) , Schema = mongoose.Schema , ObjectId = mongoose.SchemaTypes.ObjectId; var LinkSchema = new Schema({ id " : ObjectId , title : String , url : String , description : String , created : Date , tags" :[String] , owner" : ObjectId }); module.exports = mongoose.model(Link, LinkSchema);
    • 38. Run Your Appjonathan@ogmore /mnt/hgfs/src/expspreleton $ nave use 0.6.18Already installed: 0.6.18using 0.6.18jonathan@ogmore /mnt/hgfs/src/expspreleton $ node app.jsExpress server listening on port 3000 in development mode
    • 39. An App!
    • 40. Logging In Via Google
    • 41. We Are In
    • 42. Go Faster Speeding Up and Fixing Your Site• node-debugger is your friend• Understand the performance implications of: • Layers of abstraction • The node event loop• As your application scales up, speed means performance not development velocity: replace easy with faster but more complex
    • 43. Node Debugger• Here’s a stack trace? What now? SyntaxError: Unexpected identifier at Object.Function (unknown source) at Object.compile (/mnt/hgfs/src/expspreleton/node_modules/ejs/lib/ejs.js:209:12) at Function.compile (/mnt/hgfs/src/expspreleton/node_modules/express/lib/view.js:68:33) at ServerResponse._render (/mnt/hgfs/src/expspreleton/node_modules/express/lib/view.js: 417:18) at ServerResponse.render (/mnt/hgfs/src/expspreleton/node_modules/express/lib/view.js:318:17) at /mnt/hgfs/src/expspreleton/routes/index.js:9:7 at callbacks (/mnt/hgfs/src/expspreleton/node_modules/express/lib/router/index.js:272:11) at param (/mnt/hgfs/src/expspreleton/node_modules/express/lib/router/index.js:246:11) at pass (/mnt/hgfs/src/expspreleton/node_modules/express/lib/router/index.js:253:5) at Router._dispatch (/mnt/hgfs/src/expspreleton/node_modules/express/lib/router/index.js: 280:4)
    • 44. node --debug app.js!
    • 45. Real Life
    • 46. Performance: Levels of Indirection• Learn the underlying implementation of the libraries you use. They may impose a performance overhead for things you do not need• For example, Mongoose was great for getting started but involves extra indirection to map json keys to object members• Mongoose does not optimize its MongoDB access patterns for performance by default: you may want to specify subsets/index ordering• You may find as performance needs grow that you will want to replace Mongoose with your own custom classes using the native driver
    • 47. Node Asynchronous Is Not Free• The first rule of node.js is do not block the event loop• The second rule of node.js is do not block the event loop• The first time you write something in node.js, you will block the event loop * Apologies to Chuck Pahlaniuk
    • 48. Be Aware of Event Loop Blocking• Basically: if you do something that is processor-intensive you will block the loop and your performance will plummet• If you do something with slow resources that does not use a well-designed asynchronous library and callbacks, you will block and performance will plummet• Learn how the libraries you use are implemented. If you have to implement async libraries, understand how well-behaved node.js modules use the node api• If you need to impose ordering on a large number of asynchronous operations or have deeply nested callbacks upon callbacks, consider a library like async.js or futures to implement more orderly sequencing
    • 49. Resources• Me: http://async.io/• The github repo for the sample app: https://github.com/ jonathana/expspreleton
    • 50. Links to the Components Discussed• node.js: http://nodejs.org/• MongoDB: http://www.mongodb.org/ and http://www.10gen.com/• nave: https://github.com/isaacs/nave/• npm: installs with node now, but info is at http://npmjs.org/
    • 51. Node.js Web Toolkit Packages• Connect/Express: http://www.senchalabs.org/connect/ and http://expressjs.com/• Geddy: http://geddyjs.org/index.html• Mojito: http://developer.yahoo.com/cocktails/mojito/• Derby: http://derbyjs.com/• Meteor: http://www.meteor.com/• RailwayJS: http://railwayjs.com/• TowerJS: http://towerjs.org/
    • 52. General Node.js Modules• node-inspector: https://github.com/dannycoates/node-inspector/• ejs: https://github.com/visionmedia/ejs• everyauth: http://everyauth.com/• async.js: https://github.com/caolan/async• futures: https://github.com/coolaj86/futures
    • 53. Node.js MongoDB Modules• connect-mongodb: https://github.com/masylum/connect- mongodb• node-mongodb-native: https://github.com/mongodb/node- mongodb-native/• mongolia: https://github.com/masylum/mongolia• mongoose: http://mongoosejs.com/• mongoose-auth: https://github.com/bnoguchi/mongoose-auth/
    • 54. Thank You!

    ×