4 years of Node.js and
Mongodb
Christian Amor Kvalheim
Now at 10gen
Friday, August 16, 13
How did I get into
Node.js
Friday, August 16, 13
Thu Jan 7 13:53:14
2010
First Commit
Friday, August 16, 13
Growth
Friday, August 16, 13
IT was crap
• It had a “Hash” and “Ordered Hash” class
• Complete mess with horrible structure
• Used Hacked JSpec for tes...
Node was liquid
• Constant changes in Node.js
• Constant changes in driver to work with
Node.js
• OH yeah no NPM yet
Frida...
Challenges
Friday, August 16, 13
Types Oh Types
• No int64 support in Javascript
• Oh yeah Numbers is a Double (53 bit int
max)
• Decided to use Google Lon...
Features
• Started adding features like GridFS in late
jan 2010
• Replicaset and Mongos support in
2010/2011
• Stream supp...
C++
• Profiled and found BSON parser to be
main bottleneck
• Wrote horrible C++ parser
• Later cleaned up by a real C++ dev...
C++
• Initial version had own implementation of
all BSON types
• C++ Long, Binary etc.
• Hard as hell to maintain
• Bought...
Making JS BSON faster
Buffers
• Another major refactoring to support
properly
• Calculate size of Document
• Single alloca...
Pool handling
• No traditional Connection Pool
• No checking OUT and IN Connections
• Connection pinning
• The “Spice” mus...
Write concerns
• Mongodb wire protocol
• Insert/Update/Remove + GetLastError
• Operations must be pipelined
• Write the op...
Reconnecting
• No Blocking == difficult reconnect
• Buffer operations until reconnect
• Still a possible problem waiting fo...
Custom Serialization
off Object to BSON
• Added toBSON support
• Specify on your objects and return the
true representatio...
Replicaset
• Problem
• Servers come and leave the cluster
• Need to update status of servers
• Need to pick the right serv...
Replicaset
• Single thread SUCKS
• Have to use setTimeout/setInterval
• Has tons of subtle state changes
• Never going to ...
Multi Platform
Friday, August 16, 13
Windows
• Hello Windows Users
• Node.js goes Windows
• What to do About C++ ?
• Build DLL’s for win32 and win64 in
Windows...
Windows Build Hell
• Oh yeah Node extensions only build with
Visual Studio 2010 + Windows 7 64 bit
SDK
• Maybe better now ...
Windows Build Hell
• Confusion about build process on
Windows
• More “Cannot” install module questions
on Win than you can...
ARM
• Build C++ extension on ARM
• Learned about memory alignment
difference between ARM and X86
• Runs on Raspberry PI (H...
Regret
Friday, August 16, 13
Regrets
var A = function() {
this.something = "a"
}
A.prototype.some = function() {
return this.something
}
Friday, August...
Should have
var B = function() {
var something = "a"
this.some = function() {
return something
}
}
Friday, August 16, 13
Lesson
• If it’s possible to access internal state they
will access internal state
• Only expose state and functions you w...
Testing It
Friday, August 16, 13
Testing it ALL
• In the beginning
• Simple integration testing with JSpec
• Later moved to JUnit
Friday, August 16, 13
3D Reality hits
Mongo 2.0 Mongo 2.2 Mongo 2.4
Node 0.8
Node 0.10
Node 0.X
with auth
replicaset
sharding
+
2.2 features
+
2...
A New Beginning
• New test framework
• Introduce configurations (replset/etc)
• Introduce runners (in auth mode, etc)
• Run...
Documenting It
Friday, August 16, 13
Documentation
• Had to provide tons of usage examples
• Realized I had them in code already
• Built some tools to help me ...
Documentation
/**
* Example of inserting a document containing functions
*
* @_class collection
* @_function insert
* @ign...
Result
Friday, August 16, 13
Open Source
Friday, August 16, 13
On Backwards
Compatibility
• Drivers should be boring
• Change is bad
• Yeah some of the API sucks but to many
depends on ...
On Managing The
Project
• Challenge people to help
• Be specific about what you need
• Give credit for pull requests
• Igno...
Wrapping Up
Friday, August 16, 13
Take aways
• No Environment is Static
• Testing Drivers isVery Hard
• Don’t take criticism personally
• Challenge people t...
In Conclusion
• It’s been A crazy 3 1/2 years of serving
the Node community
• I’ve made tons of mistakes a long the
way
• ...
Feel free to Ask
Questions
Friday, August 16, 13
Upcoming SlideShare
Loading in …5
×

Lessons from 4 years of driver develoment

1,218 views

Published on

Lessons from 4 years of developing the mongodb driver for node.js

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,218
On SlideShare
0
From Embeds
0
Number of Embeds
412
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Lessons from 4 years of driver develoment

  1. 1. 4 years of Node.js and Mongodb Christian Amor Kvalheim Now at 10gen Friday, August 16, 13
  2. 2. How did I get into Node.js Friday, August 16, 13
  3. 3. Thu Jan 7 13:53:14 2010 First Commit Friday, August 16, 13
  4. 4. Growth Friday, August 16, 13
  5. 5. IT was crap • It had a “Hash” and “Ordered Hash” class • Complete mess with horrible structure • Used Hacked JSpec for testing at the start Friday, August 16, 13
  6. 6. Node was liquid • Constant changes in Node.js • Constant changes in driver to work with Node.js • OH yeah no NPM yet Friday, August 16, 13
  7. 7. Challenges Friday, August 16, 13
  8. 8. Types Oh Types • No int64 support in Javascript • Oh yeah Numbers is a Double (53 bit int max) • Decided to use Google Long closure library class Friday, August 16, 13
  9. 9. Features • Started adding features like GridFS in late jan 2010 • Replicaset and Mongos support in 2010/2011 • Stream support for Cursor and GridFS • High Availability Friday, August 16, 13
  10. 10. C++ • Profiled and found BSON parser to be main bottleneck • Wrote horrible C++ parser • Later cleaned up by a real C++ developer from Lucasarts (RIP) Friday, August 16, 13
  11. 11. C++ • Initial version had own implementation of all BSON types • C++ Long, Binary etc. • Hard as hell to maintain • Bought me nothing in the end asV8 changed Friday, August 16, 13
  12. 12. Making JS BSON faster Buffers • Another major refactoring to support properly • Calculate size of Document • Single allocation instead of multiple small ones • Using Buffers (optimized by v8) • Indexed entry a[0] direct memory access Friday, August 16, 13
  13. 13. Pool handling • No traditional Connection Pool • No checking OUT and IN Connections • Connection pinning • The “Spice” must flow Friday, August 16, 13
  14. 14. Write concerns • Mongodb wire protocol • Insert/Update/Remove + GetLastError • Operations must be pipelined • Write the op + GetLastError in one operation Friday, August 16, 13
  15. 15. Reconnecting • No Blocking == difficult reconnect • Buffer operations until reconnect • Still a possible problem waiting for a solution Friday, August 16, 13
  16. 16. Custom Serialization off Object to BSON • Added toBSON support • Specify on your objects and return the true representation of the object • Modeled after toJSON Friday, August 16, 13
  17. 17. Replicaset • Problem • Servers come and leave the cluster • Need to update status of servers • Need to pick the right server for the readPreference Friday, August 16, 13
  18. 18. Replicaset • Single thread SUCKS • Have to use setTimeout/setInterval • Has tons of subtle state changes • Never going to get Webworkers in Node.js • Processes are to granular Friday, August 16, 13
  19. 19. Multi Platform Friday, August 16, 13
  20. 20. Windows • Hello Windows Users • Node.js goes Windows • What to do About C++ ? • Build DLL’s for win32 and win64 in Windows 7 and package with driver Friday, August 16, 13
  21. 21. Windows Build Hell • Oh yeah Node extensions only build with Visual Studio 2010 + Windows 7 64 bit SDK • Maybe better now but I doubt it • Node-gyp will fail to build extensions on Win causes confusion Friday, August 16, 13
  22. 22. Windows Build Hell • Confusion about build process on Windows • More “Cannot” install module questions on Win than you can swing a cat at • Mostly PBKC • Who the Hell still uses XP (SERIOUSLY) Friday, August 16, 13
  23. 23. ARM • Build C++ extension on ARM • Learned about memory alignment difference between ARM and X86 • Runs on Raspberry PI (HUZZA) Friday, August 16, 13
  24. 24. Regret Friday, August 16, 13
  25. 25. Regrets var A = function() { this.something = "a" } A.prototype.some = function() { return this.something } Friday, August 16, 13
  26. 26. Should have var B = function() { var something = "a" this.some = function() { return something } } Friday, August 16, 13
  27. 27. Lesson • If it’s possible to access internal state they will access internal state • Only expose state and functions you wish end user should be able to use • Examples of My BAD • Collection class • Cursor class Friday, August 16, 13
  28. 28. Testing It Friday, August 16, 13
  29. 29. Testing it ALL • In the beginning • Simple integration testing with JSpec • Later moved to JUnit Friday, August 16, 13
  30. 30. 3D Reality hits Mongo 2.0 Mongo 2.2 Mongo 2.4 Node 0.8 Node 0.10 Node 0.X with auth replicaset sharding + 2.2 features + 2.4 features etc etc etc etc etc etc Friday, August 16, 13
  31. 31. A New Beginning • New test framework • Introduce configurations (replset/etc) • Introduce runners (in auth mode, etc) • Run tests against configurations • Test does not manage configurations • Still not there • Need to work on the concept more Friday, August 16, 13
  32. 32. Documenting It Friday, August 16, 13
  33. 33. Documentation • Had to provide tons of usage examples • Realized I had them in code already • Built some tools to help me generate most of the docs • Spaghetti JS + Python Spinx Friday, August 16, 13
  34. 34. Documentation /** * Example of inserting a document containing functions * * @_class collection * @_function insert * @ignore */ exports.test = function(configuration, test) { var db = configuration.db(); // DOC_LINE var db = new Db('test', new Server('locahost', 27017)); // DOC_START // Fetch a collection to insert document into var collection = db.collection("simple_document_insert_with_function_safe"); // Insert a single document collection.insert({hello:'world' , func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) { test.equal(null, err); // Fetch the document collection.findOne({hello:'world'}, function(err, item) { test.equal(null, err); test.ok("function() {}", item.code); test.done(); }) }); // DOC_END } Friday, August 16, 13
  35. 35. Result Friday, August 16, 13
  36. 36. Open Source Friday, August 16, 13
  37. 37. On Backwards Compatibility • Drivers should be boring • Change is bad • Yeah some of the API sucks but to many depends on it • This week CoffeeScript next week LiveScript (REALLY?) Friday, August 16, 13
  38. 38. On Managing The Project • Challenge people to help • Be specific about what you need • Give credit for pull requests • Ignore flames or criticism • Just ignore bad faith issues • Your code is always going to suck, roll with it Friday, August 16, 13
  39. 39. Wrapping Up Friday, August 16, 13
  40. 40. Take aways • No Environment is Static • Testing Drivers isVery Hard • Don’t take criticism personally • Challenge people to help you • Most people are nice and will help • Credit people who help you Friday, August 16, 13
  41. 41. In Conclusion • It’s been A crazy 3 1/2 years of serving the Node community • I’ve made tons of mistakes a long the way • I’m sure to make a ton of mistakes in the future • I would do it all again in a second Friday, August 16, 13
  42. 42. Feel free to Ask Questions Friday, August 16, 13

×