Lessons from 4 years of driver develoment

4 years of Node.js and
Mongodb
Christian Amor Kvalheim
Now at 10gen
Friday, August 16, 13

How did I get into
Node.js

Thu Jan 7 13:53:14
2010
First Commit

IT was crap
• It had a “Hash” and “Ordered Hash” class
• Complete mess with horrible structure
• Used Hacked JSpec for testing at the start

Node was liquid
• Constant changes in Node.js
• Constant changes in driver to work with
Node.js
• OH yeah no NPM yet

Challenges

Types Oh Types
• No int64 support in Javascript
• Oh yeah Numbers is a Double (53 bit int
max)
• Decided to use Google Long closure
library class

Features
• Started adding features like GridFS in late
jan 2010
• Replicaset and Mongos support in
2010/2011
• Stream support for Cursor and GridFS
• High Availability

C++
• Proﬁled and found BSON parser to be
main bottleneck
• Wrote horrible C++ parser
• Later cleaned up by a real C++ developer
from Lucasarts (RIP)

C++
• Initial version had own implementation of
all BSON types
• C++ Long, Binary etc.
• Hard as hell to maintain
• Bought me nothing in the end asV8
changed

Making JS BSON faster
Buffers
• Another major refactoring to support
properly
• Calculate size of Document
• Single allocation instead of multiple small
ones
• Using Buffers (optimized by v8)
• Indexed entry a[0] direct memory access

Pool handling
• No traditional Connection Pool
• No checking OUT and IN Connections
• Connection pinning
• The “Spice” must ﬂow

Write concerns
• Mongodb wire protocol
• Insert/Update/Remove + GetLastError
• Operations must be pipelined
• Write the op + GetLastError in one
operation

Reconnecting
• No Blocking == difﬁcult reconnect
• Buffer operations until reconnect
• Still a possible problem waiting for a
solution

Custom Serialization
off Object to BSON
• Added toBSON support
• Specify on your objects and return the
true representation of the object
• Modeled after toJSON

Replicaset
• Problem
• Servers come and leave the cluster
• Need to update status of servers
• Need to pick the right server for the
readPreference

Replicaset
• Single thread SUCKS
• Have to use setTimeout/setInterval
• Has tons of subtle state changes
• Never going to get Webworkers in Node.js
• Processes are to granular

Multi Platform

Windows
• Hello Windows Users
• Node.js goes Windows
• What to do About C++ ?
• Build DLL’s for win32 and win64 in
Windows 7 and package with driver

Windows Build Hell
• Oh yeah Node extensions only build with
Visual Studio 2010 + Windows 7 64 bit
SDK
• Maybe better now but I doubt it
• Node-gyp will fail to build extensions on
Win causes confusion

Windows Build Hell
• Confusion about build process on
Windows
• More “Cannot” install module questions
on Win than you can swing a cat at
• Mostly PBKC
• Who the Hell still uses XP (SERIOUSLY)

ARM
• Build C++ extension on ARM
• Learned about memory alignment
difference between ARM and X86
• Runs on Raspberry PI (HUZZA)

Regrets
var A = function() {
this.something = "a"
}
A.prototype.some = function() {
return this.something
}

Should have
var B = function() {
var something = "a"
this.some = function() {
return something
}
}

Lesson
• If it’s possible to access internal state they
will access internal state
• Only expose state and functions you wish
end user should be able to use
• Examples of My BAD
• Collection class
• Cursor class

Testing It

Testing it ALL
• In the beginning
• Simple integration testing with JSpec
• Later moved to JUnit

3D Reality hits
Mongo 2.0 Mongo 2.2 Mongo 2.4
Node 0.8
Node 0.10
Node 0.X
with auth
replicaset
sharding
+
2.2 features
+
2.4 features
etc etc etc
etc etc etc

A New Beginning
• New test framework
• Introduce configurations (replset/etc)
• Introduce runners (in auth mode, etc)
• Run tests against configurations
• Test does not manage configurations
• Still not there
• Need to work on the concept more

Documenting It

Documentation
• Had to provide tons of usage examples
• Realized I had them in code already
• Built some tools to help me generate
most of the docs
• Spaghetti JS + Python Spinx

Documentation
/**
* Example of inserting a document containing functions
*
* @_class collection
* @_function insert
* @ignore
*/
exports.test = function(configuration, test) {
var db = configuration.db();
// DOC_LINE var db = new Db('test', new Server('locahost', 27017));
// DOC_START
// Fetch a collection to insert document into
var collection = db.collection("simple_document_insert_with_function_safe");
// Insert a single document
collection.insert({hello:'world'
, func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) {
test.equal(null, err);
// Fetch the document
collection.findOne({hello:'world'}, function(err, item) {
test.equal(null, err);
test.ok("function() {}", item.code);
test.done();
})
});
// DOC_END
}

Open Source

On Backwards
Compatibility
• Drivers should be boring
• Change is bad
• Yeah some of the API sucks but to many
depends on it
• This week CoffeeScript next week
LiveScript (REALLY?)

On Managing The
Project
• Challenge people to help
• Be speciﬁc about what you need
• Give credit for pull requests
• Ignore ﬂames or criticism
• Just ignore bad faith issues
• Your code is always going to suck, roll with
it

Wrapping Up

Take aways
• No Environment is Static
• Testing Drivers isVery Hard
• Don’t take criticism personally
• Challenge people to help you
• Most people are nice and will help
• Credit people who help you

In Conclusion
• It’s been A crazy 3 1/2 years of serving
the Node community
• I’ve made tons of mistakes a long the
way
• I’m sure to make a ton of mistakes in
the future
• I would do it all again in a second

Feel free to Ask
Questions

Lessons from 4 years of driver develoment

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lessons from 4 years of driver develoment

Similar to Lessons from 4 years of driver develoment (20)

More from christkv

More from christkv (9)

Recently uploaded

Recently uploaded (20)

Lessons from 4 years of driver develoment