• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Lessons from 4 years of driver develoment
 

Lessons from 4 years of driver develoment

on

  • 485 views

Lessons from 4 years of developing the mongodb driver for node.js

Lessons from 4 years of developing the mongodb driver for node.js

Statistics

Views

Total Views
485
Views on SlideShare
428
Embed Views
57

Actions

Likes
1
Downloads
4
Comments
0

3 Embeds 57

http://christiankvalheim.com 49
http://localhost 7
http://feedly.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Lessons from 4 years of driver develoment Lessons from 4 years of driver develoment Presentation Transcript

    • 4 years of Node.js and Mongodb Christian Amor Kvalheim Now at 10gen Friday, August 16, 13
    • How did I get into Node.js Friday, August 16, 13
    • Thu Jan 7 13:53:14 2010 First Commit Friday, August 16, 13
    • Growth Friday, August 16, 13
    • IT was crap • It had a “Hash” and “Ordered Hash” class • Complete mess with horrible structure • Used Hacked JSpec for testing at the start Friday, August 16, 13
    • Node was liquid • Constant changes in Node.js • Constant changes in driver to work with Node.js • OH yeah no NPM yet Friday, August 16, 13
    • Challenges Friday, August 16, 13
    • Types Oh Types • No int64 support in Javascript • Oh yeah Numbers is a Double (53 bit int max) • Decided to use Google Long closure library class Friday, August 16, 13
    • Features • Started adding features like GridFS in late jan 2010 • Replicaset and Mongos support in 2010/2011 • Stream support for Cursor and GridFS • High Availability Friday, August 16, 13
    • C++ • Profiled and found BSON parser to be main bottleneck • Wrote horrible C++ parser • Later cleaned up by a real C++ developer from Lucasarts (RIP) Friday, August 16, 13
    • C++ • Initial version had own implementation of all BSON types • C++ Long, Binary etc. • Hard as hell to maintain • Bought me nothing in the end asV8 changed Friday, August 16, 13
    • Making JS BSON faster Buffers • Another major refactoring to support properly • Calculate size of Document • Single allocation instead of multiple small ones • Using Buffers (optimized by v8) • Indexed entry a[0] direct memory access Friday, August 16, 13
    • Pool handling • No traditional Connection Pool • No checking OUT and IN Connections • Connection pinning • The “Spice” must flow Friday, August 16, 13
    • Write concerns • Mongodb wire protocol • Insert/Update/Remove + GetLastError • Operations must be pipelined • Write the op + GetLastError in one operation Friday, August 16, 13
    • Reconnecting • No Blocking == difficult reconnect • Buffer operations until reconnect • Still a possible problem waiting for a solution Friday, August 16, 13
    • Custom Serialization off Object to BSON • Added toBSON support • Specify on your objects and return the true representation of the object • Modeled after toJSON Friday, August 16, 13
    • Replicaset • Problem • Servers come and leave the cluster • Need to update status of servers • Need to pick the right server for the readPreference Friday, August 16, 13
    • Replicaset • Single thread SUCKS • Have to use setTimeout/setInterval • Has tons of subtle state changes • Never going to get Webworkers in Node.js • Processes are to granular Friday, August 16, 13
    • Multi Platform Friday, August 16, 13
    • Windows • Hello Windows Users • Node.js goes Windows • What to do About C++ ? • Build DLL’s for win32 and win64 in Windows 7 and package with driver Friday, August 16, 13
    • Windows Build Hell • Oh yeah Node extensions only build with Visual Studio 2010 + Windows 7 64 bit SDK • Maybe better now but I doubt it • Node-gyp will fail to build extensions on Win causes confusion Friday, August 16, 13
    • Windows Build Hell • Confusion about build process on Windows • More “Cannot” install module questions on Win than you can swing a cat at • Mostly PBKC • Who the Hell still uses XP (SERIOUSLY) Friday, August 16, 13
    • ARM • Build C++ extension on ARM • Learned about memory alignment difference between ARM and X86 • Runs on Raspberry PI (HUZZA) Friday, August 16, 13
    • Regret Friday, August 16, 13
    • Regrets var A = function() { this.something = "a" } A.prototype.some = function() { return this.something } Friday, August 16, 13
    • Should have var B = function() { var something = "a" this.some = function() { return something } } Friday, August 16, 13
    • Lesson • If it’s possible to access internal state they will access internal state • Only expose state and functions you wish end user should be able to use • Examples of My BAD • Collection class • Cursor class Friday, August 16, 13
    • Testing It Friday, August 16, 13
    • Testing it ALL • In the beginning • Simple integration testing with JSpec • Later moved to JUnit Friday, August 16, 13
    • 3D Reality hits Mongo 2.0 Mongo 2.2 Mongo 2.4 Node 0.8 Node 0.10 Node 0.X with auth replicaset sharding + 2.2 features + 2.4 features etc etc etc etc etc etc Friday, August 16, 13
    • A New Beginning • New test framework • Introduce configurations (replset/etc) • Introduce runners (in auth mode, etc) • Run tests against configurations • Test does not manage configurations • Still not there • Need to work on the concept more Friday, August 16, 13
    • Documenting It Friday, August 16, 13
    • Documentation • Had to provide tons of usage examples • Realized I had them in code already • Built some tools to help me generate most of the docs • Spaghetti JS + Python Spinx Friday, August 16, 13
    • Documentation /** * Example of inserting a document containing functions * * @_class collection * @_function insert * @ignore */ exports.test = function(configuration, test) { var db = configuration.db(); // DOC_LINE var db = new Db('test', new Server('locahost', 27017)); // DOC_START // Fetch a collection to insert document into var collection = db.collection("simple_document_insert_with_function_safe"); // Insert a single document collection.insert({hello:'world' , func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) { test.equal(null, err); // Fetch the document collection.findOne({hello:'world'}, function(err, item) { test.equal(null, err); test.ok("function() {}", item.code); test.done(); }) }); // DOC_END } Friday, August 16, 13
    • Result Friday, August 16, 13
    • Open Source Friday, August 16, 13
    • On Backwards Compatibility • Drivers should be boring • Change is bad • Yeah some of the API sucks but to many depends on it • This week CoffeeScript next week LiveScript (REALLY?) Friday, August 16, 13
    • On Managing The Project • Challenge people to help • Be specific about what you need • Give credit for pull requests • Ignore flames or criticism • Just ignore bad faith issues • Your code is always going to suck, roll with it Friday, August 16, 13
    • Wrapping Up Friday, August 16, 13
    • Take aways • No Environment is Static • Testing Drivers isVery Hard • Don’t take criticism personally • Challenge people to help you • Most people are nice and will help • Credit people who help you Friday, August 16, 13
    • In Conclusion • It’s been A crazy 3 1/2 years of serving the Node community • I’ve made tons of mistakes a long the way • I’m sure to make a ton of mistakes in the future • I would do it all again in a second Friday, August 16, 13
    • Feel free to Ask Questions Friday, August 16, 13