Successfully reported this slideshow.

Lessons from 4 years of driver develoment

1

Share

Upcoming SlideShare
Travel with your mock server
Travel with your mock server
Loading in …3
×
1 of 42
1 of 42

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Lessons from 4 years of driver develoment

  1. 1. 4 years of Node.js and Mongodb Christian Amor Kvalheim Now at 10gen Friday, August 16, 13
  2. 2. How did I get into Node.js Friday, August 16, 13
  3. 3. Thu Jan 7 13:53:14 2010 First Commit Friday, August 16, 13
  4. 4. Growth Friday, August 16, 13
  5. 5. IT was crap • It had a “Hash” and “Ordered Hash” class • Complete mess with horrible structure • Used Hacked JSpec for testing at the start Friday, August 16, 13
  6. 6. Node was liquid • Constant changes in Node.js • Constant changes in driver to work with Node.js • OH yeah no NPM yet Friday, August 16, 13
  7. 7. Challenges Friday, August 16, 13
  8. 8. Types Oh Types • No int64 support in Javascript • Oh yeah Numbers is a Double (53 bit int max) • Decided to use Google Long closure library class Friday, August 16, 13
  9. 9. Features • Started adding features like GridFS in late jan 2010 • Replicaset and Mongos support in 2010/2011 • Stream support for Cursor and GridFS • High Availability Friday, August 16, 13
  10. 10. C++ • Profiled and found BSON parser to be main bottleneck • Wrote horrible C++ parser • Later cleaned up by a real C++ developer from Lucasarts (RIP) Friday, August 16, 13
  11. 11. C++ • Initial version had own implementation of all BSON types • C++ Long, Binary etc. • Hard as hell to maintain • Bought me nothing in the end asV8 changed Friday, August 16, 13
  12. 12. Making JS BSON faster Buffers • Another major refactoring to support properly • Calculate size of Document • Single allocation instead of multiple small ones • Using Buffers (optimized by v8) • Indexed entry a[0] direct memory access Friday, August 16, 13
  13. 13. Pool handling • No traditional Connection Pool • No checking OUT and IN Connections • Connection pinning • The “Spice” must flow Friday, August 16, 13
  14. 14. Write concerns • Mongodb wire protocol • Insert/Update/Remove + GetLastError • Operations must be pipelined • Write the op + GetLastError in one operation Friday, August 16, 13
  15. 15. Reconnecting • No Blocking == difficult reconnect • Buffer operations until reconnect • Still a possible problem waiting for a solution Friday, August 16, 13
  16. 16. Custom Serialization off Object to BSON • Added toBSON support • Specify on your objects and return the true representation of the object • Modeled after toJSON Friday, August 16, 13
  17. 17. Replicaset • Problem • Servers come and leave the cluster • Need to update status of servers • Need to pick the right server for the readPreference Friday, August 16, 13
  18. 18. Replicaset • Single thread SUCKS • Have to use setTimeout/setInterval • Has tons of subtle state changes • Never going to get Webworkers in Node.js • Processes are to granular Friday, August 16, 13
  19. 19. Multi Platform Friday, August 16, 13
  20. 20. Windows • Hello Windows Users • Node.js goes Windows • What to do About C++ ? • Build DLL’s for win32 and win64 in Windows 7 and package with driver Friday, August 16, 13
  21. 21. Windows Build Hell • Oh yeah Node extensions only build with Visual Studio 2010 + Windows 7 64 bit SDK • Maybe better now but I doubt it • Node-gyp will fail to build extensions on Win causes confusion Friday, August 16, 13
  22. 22. Windows Build Hell • Confusion about build process on Windows • More “Cannot” install module questions on Win than you can swing a cat at • Mostly PBKC • Who the Hell still uses XP (SERIOUSLY) Friday, August 16, 13
  23. 23. ARM • Build C++ extension on ARM • Learned about memory alignment difference between ARM and X86 • Runs on Raspberry PI (HUZZA) Friday, August 16, 13
  24. 24. Regret Friday, August 16, 13
  25. 25. Regrets var A = function() { this.something = "a" } A.prototype.some = function() { return this.something } Friday, August 16, 13
  26. 26. Should have var B = function() { var something = "a" this.some = function() { return something } } Friday, August 16, 13
  27. 27. Lesson • If it’s possible to access internal state they will access internal state • Only expose state and functions you wish end user should be able to use • Examples of My BAD • Collection class • Cursor class Friday, August 16, 13
  28. 28. Testing It Friday, August 16, 13
  29. 29. Testing it ALL • In the beginning • Simple integration testing with JSpec • Later moved to JUnit Friday, August 16, 13
  30. 30. 3D Reality hits Mongo 2.0 Mongo 2.2 Mongo 2.4 Node 0.8 Node 0.10 Node 0.X with auth replicaset sharding + 2.2 features + 2.4 features etc etc etc etc etc etc Friday, August 16, 13
  31. 31. A New Beginning • New test framework • Introduce configurations (replset/etc) • Introduce runners (in auth mode, etc) • Run tests against configurations • Test does not manage configurations • Still not there • Need to work on the concept more Friday, August 16, 13
  32. 32. Documenting It Friday, August 16, 13
  33. 33. Documentation • Had to provide tons of usage examples • Realized I had them in code already • Built some tools to help me generate most of the docs • Spaghetti JS + Python Spinx Friday, August 16, 13
  34. 34. Documentation /** * Example of inserting a document containing functions * * @_class collection * @_function insert * @ignore */ exports.test = function(configuration, test) { var db = configuration.db(); // DOC_LINE var db = new Db('test', new Server('locahost', 27017)); // DOC_START // Fetch a collection to insert document into var collection = db.collection("simple_document_insert_with_function_safe"); // Insert a single document collection.insert({hello:'world' , func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) { test.equal(null, err); // Fetch the document collection.findOne({hello:'world'}, function(err, item) { test.equal(null, err); test.ok("function() {}", item.code); test.done(); }) }); // DOC_END } Friday, August 16, 13
  35. 35. Result Friday, August 16, 13
  36. 36. Open Source Friday, August 16, 13
  37. 37. On Backwards Compatibility • Drivers should be boring • Change is bad • Yeah some of the API sucks but to many depends on it • This week CoffeeScript next week LiveScript (REALLY?) Friday, August 16, 13
  38. 38. On Managing The Project • Challenge people to help • Be specific about what you need • Give credit for pull requests • Ignore flames or criticism • Just ignore bad faith issues • Your code is always going to suck, roll with it Friday, August 16, 13
  39. 39. Wrapping Up Friday, August 16, 13
  40. 40. Take aways • No Environment is Static • Testing Drivers isVery Hard • Don’t take criticism personally • Challenge people to help you • Most people are nice and will help • Credit people who help you Friday, August 16, 13
  41. 41. In Conclusion • It’s been A crazy 3 1/2 years of serving the Node community • I’ve made tons of mistakes a long the way • I’m sure to make a ton of mistakes in the future • I would do it all again in a second Friday, August 16, 13
  42. 42. Feel free to Ask Questions Friday, August 16, 13

×