SlideShare a Scribd company logo
4 years of Node.js and
Mongodb
Christian Amor Kvalheim
Now at 10gen
Friday, August 16, 13
How did I get into
Node.js
Friday, August 16, 13
Thu Jan 7 13:53:14
2010
First Commit
Friday, August 16, 13
Growth
Friday, August 16, 13
IT was crap
• It had a “Hash” and “Ordered Hash” class
• Complete mess with horrible structure
• Used Hacked JSpec for testing at the start
Friday, August 16, 13
Node was liquid
• Constant changes in Node.js
• Constant changes in driver to work with
Node.js
• OH yeah no NPM yet
Friday, August 16, 13
Challenges
Friday, August 16, 13
Types Oh Types
• No int64 support in Javascript
• Oh yeah Numbers is a Double (53 bit int
max)
• Decided to use Google Long closure
library class
Friday, August 16, 13
Features
• Started adding features like GridFS in late
jan 2010
• Replicaset and Mongos support in
2010/2011
• Stream support for Cursor and GridFS
• High Availability
Friday, August 16, 13
C++
• Profiled and found BSON parser to be
main bottleneck
• Wrote horrible C++ parser
• Later cleaned up by a real C++ developer
from Lucasarts (RIP)
Friday, August 16, 13
C++
• Initial version had own implementation of
all BSON types
• C++ Long, Binary etc.
• Hard as hell to maintain
• Bought me nothing in the end asV8
changed
Friday, August 16, 13
Making JS BSON faster
Buffers
• Another major refactoring to support
properly
• Calculate size of Document
• Single allocation instead of multiple small
ones
• Using Buffers (optimized by v8)
• Indexed entry a[0] direct memory access
Friday, August 16, 13
Pool handling
• No traditional Connection Pool
• No checking OUT and IN Connections
• Connection pinning
• The “Spice” must flow
Friday, August 16, 13
Write concerns
• Mongodb wire protocol
• Insert/Update/Remove + GetLastError
• Operations must be pipelined
• Write the op + GetLastError in one
operation
Friday, August 16, 13
Reconnecting
• No Blocking == difficult reconnect
• Buffer operations until reconnect
• Still a possible problem waiting for a
solution
Friday, August 16, 13
Custom Serialization
off Object to BSON
• Added toBSON support
• Specify on your objects and return the
true representation of the object
• Modeled after toJSON
Friday, August 16, 13
Replicaset
• Problem
• Servers come and leave the cluster
• Need to update status of servers
• Need to pick the right server for the
readPreference
Friday, August 16, 13
Replicaset
• Single thread SUCKS
• Have to use setTimeout/setInterval
• Has tons of subtle state changes
• Never going to get Webworkers in Node.js
• Processes are to granular
Friday, August 16, 13
Multi Platform
Friday, August 16, 13
Windows
• Hello Windows Users
• Node.js goes Windows
• What to do About C++ ?
• Build DLL’s for win32 and win64 in
Windows 7 and package with driver
Friday, August 16, 13
Windows Build Hell
• Oh yeah Node extensions only build with
Visual Studio 2010 + Windows 7 64 bit
SDK
• Maybe better now but I doubt it
• Node-gyp will fail to build extensions on
Win causes confusion
Friday, August 16, 13
Windows Build Hell
• Confusion about build process on
Windows
• More “Cannot” install module questions
on Win than you can swing a cat at
• Mostly PBKC
• Who the Hell still uses XP (SERIOUSLY)
Friday, August 16, 13
ARM
• Build C++ extension on ARM
• Learned about memory alignment
difference between ARM and X86
• Runs on Raspberry PI (HUZZA)
Friday, August 16, 13
Regret
Friday, August 16, 13
Regrets
var A = function() {
this.something = "a"
}
A.prototype.some = function() {
return this.something
}
Friday, August 16, 13
Should have
var B = function() {
var something = "a"
this.some = function() {
return something
}
}
Friday, August 16, 13
Lesson
• If it’s possible to access internal state they
will access internal state
• Only expose state and functions you wish
end user should be able to use
• Examples of My BAD
• Collection class
• Cursor class
Friday, August 16, 13
Testing It
Friday, August 16, 13
Testing it ALL
• In the beginning
• Simple integration testing with JSpec
• Later moved to JUnit
Friday, August 16, 13
3D Reality hits
Mongo 2.0 Mongo 2.2 Mongo 2.4
Node 0.8
Node 0.10
Node 0.X
with auth
replicaset
sharding
+
2.2 features
+
2.4 features
etc etc etc
etc etc etc
Friday, August 16, 13
A New Beginning
• New test framework
• Introduce configurations (replset/etc)
• Introduce runners (in auth mode, etc)
• Run tests against configurations
• Test does not manage configurations
• Still not there
• Need to work on the concept more
Friday, August 16, 13
Documenting It
Friday, August 16, 13
Documentation
• Had to provide tons of usage examples
• Realized I had them in code already
• Built some tools to help me generate
most of the docs
• Spaghetti JS + Python Spinx
Friday, August 16, 13
Documentation
/**
* Example of inserting a document containing functions
*
* @_class collection
* @_function insert
* @ignore
*/
exports.test = function(configuration, test) {
var db = configuration.db();
// DOC_LINE var db = new Db('test', new Server('locahost', 27017));
// DOC_START
// Fetch a collection to insert document into
var collection = db.collection("simple_document_insert_with_function_safe");
// Insert a single document
collection.insert({hello:'world'
, func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) {
test.equal(null, err);
// Fetch the document
collection.findOne({hello:'world'}, function(err, item) {
test.equal(null, err);
test.ok("function() {}", item.code);
test.done();
})
});
// DOC_END
}
Friday, August 16, 13
Result
Friday, August 16, 13
Open Source
Friday, August 16, 13
On Backwards
Compatibility
• Drivers should be boring
• Change is bad
• Yeah some of the API sucks but to many
depends on it
• This week CoffeeScript next week
LiveScript (REALLY?)
Friday, August 16, 13
On Managing The
Project
• Challenge people to help
• Be specific about what you need
• Give credit for pull requests
• Ignore flames or criticism
• Just ignore bad faith issues
• Your code is always going to suck, roll with
it
Friday, August 16, 13
Wrapping Up
Friday, August 16, 13
Take aways
• No Environment is Static
• Testing Drivers isVery Hard
• Don’t take criticism personally
• Challenge people to help you
• Most people are nice and will help
• Credit people who help you
Friday, August 16, 13
In Conclusion
• It’s been A crazy 3 1/2 years of serving
the Node community
• I’ve made tons of mistakes a long the
way
• I’m sure to make a ton of mistakes in
the future
• I would do it all again in a second
Friday, August 16, 13
Feel free to Ask
Questions
Friday, August 16, 13

More Related Content

What's hot

JRuby: Pushing the Java Platform Further
JRuby: Pushing the Java Platform FurtherJRuby: Pushing the Java Platform Further
JRuby: Pushing the Java Platform Further
Charles Nutter
 
Bringing Concurrency to Ruby - RubyConf India 2014
Bringing Concurrency to Ruby - RubyConf India 2014Bringing Concurrency to Ruby - RubyConf India 2014
Bringing Concurrency to Ruby - RubyConf India 2014
Charles Nutter
 
Deep dive - Concourse CI/CD and Pipelines
Deep dive  - Concourse CI/CD and PipelinesDeep dive  - Concourse CI/CD and Pipelines
Deep dive - Concourse CI/CD and Pipelines
Syed Imam
 
Rex - Lightning Talk yapc.eu 2013
Rex - Lightning Talk yapc.eu 2013Rex - Lightning Talk yapc.eu 2013
Rex - Lightning Talk yapc.eu 2013
Jan Gehring
 
A Recovering Java Developer Learns to Go
A Recovering Java Developer Learns to GoA Recovering Java Developer Learns to Go
A Recovering Java Developer Learns to Go
Matt Stine
 
Docker intro
Docker introDocker intro
Docker intro
Lenz Gschwendtner
 
Using Nagios to monitor your WO systems
Using Nagios to monitor your WO systemsUsing Nagios to monitor your WO systems
Using Nagios to monitor your WO systems
WO Community
 
Subversion @ JUG Milano 11 dic 2009
Subversion @ JUG Milano 11 dic 2009Subversion @ JUG Milano 11 dic 2009
Subversion @ JUG Milano 11 dic 2009
Andrea Francia
 
From Test to Live with Rex
From Test to Live with RexFrom Test to Live with Rex
From Test to Live with Rex
Jan Gehring
 
Don’t turn your logs into cuneiform
Don’t turn your logs into cuneiformDon’t turn your logs into cuneiform
Don’t turn your logs into cuneiform
Andrey Rebrov
 
Node.js and Ruby
Node.js and RubyNode.js and Ruby
Node.js and Ruby
Michael Bleigh
 
Coding in the context era
Coding in the context eraCoding in the context era
Coding in the context era
lestrrat
 
Briefly Rust - Daniele Esposti - Codemotion Rome 2017
Briefly Rust - Daniele Esposti - Codemotion Rome 2017Briefly Rust - Daniele Esposti - Codemotion Rome 2017
Briefly Rust - Daniele Esposti - Codemotion Rome 2017
Codemotion
 
Vagrant, Ansible and Docker - How they fit together for productive flexible d...
Vagrant, Ansible and Docker - How they fit together for productive flexible d...Vagrant, Ansible and Docker - How they fit together for productive flexible d...
Vagrant, Ansible and Docker - How they fit together for productive flexible d...
Samuel Lampa
 
[Js hcm] Java script- Testing the awesome
[Js hcm] Java script- Testing the awesome[Js hcm] Java script- Testing the awesome
[Js hcm] Java script- Testing the awesome
JavaScript Meetup HCMC
 
CRaSH the shell for the JVM
CRaSH the shell for the JVMCRaSH the shell for the JVM
CRaSH the shell for the JVM
jviet
 
Mocha, chai and sinon
Mocha, chai and sinonMocha, chai and sinon
Mocha, chai and sinon
Andrew Dixon
 
LuaNode: Asynchronous I/O for Lua
LuaNode: Asynchronous I/O for LuaLuaNode: Asynchronous I/O for Lua
LuaNode: Asynchronous I/O for Lua
Ignacio Burgueño
 
Introduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" EditionIntroduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" Edition
Jérôme Petazzoni
 
BBC's GraphDB (formerly Owlim) AWS Cloud Migration
BBC's GraphDB (formerly Owlim) AWS Cloud MigrationBBC's GraphDB (formerly Owlim) AWS Cloud Migration
BBC's GraphDB (formerly Owlim) AWS Cloud Migration
logomachy
 

What's hot (20)

JRuby: Pushing the Java Platform Further
JRuby: Pushing the Java Platform FurtherJRuby: Pushing the Java Platform Further
JRuby: Pushing the Java Platform Further
 
Bringing Concurrency to Ruby - RubyConf India 2014
Bringing Concurrency to Ruby - RubyConf India 2014Bringing Concurrency to Ruby - RubyConf India 2014
Bringing Concurrency to Ruby - RubyConf India 2014
 
Deep dive - Concourse CI/CD and Pipelines
Deep dive  - Concourse CI/CD and PipelinesDeep dive  - Concourse CI/CD and Pipelines
Deep dive - Concourse CI/CD and Pipelines
 
Rex - Lightning Talk yapc.eu 2013
Rex - Lightning Talk yapc.eu 2013Rex - Lightning Talk yapc.eu 2013
Rex - Lightning Talk yapc.eu 2013
 
A Recovering Java Developer Learns to Go
A Recovering Java Developer Learns to GoA Recovering Java Developer Learns to Go
A Recovering Java Developer Learns to Go
 
Docker intro
Docker introDocker intro
Docker intro
 
Using Nagios to monitor your WO systems
Using Nagios to monitor your WO systemsUsing Nagios to monitor your WO systems
Using Nagios to monitor your WO systems
 
Subversion @ JUG Milano 11 dic 2009
Subversion @ JUG Milano 11 dic 2009Subversion @ JUG Milano 11 dic 2009
Subversion @ JUG Milano 11 dic 2009
 
From Test to Live with Rex
From Test to Live with RexFrom Test to Live with Rex
From Test to Live with Rex
 
Don’t turn your logs into cuneiform
Don’t turn your logs into cuneiformDon’t turn your logs into cuneiform
Don’t turn your logs into cuneiform
 
Node.js and Ruby
Node.js and RubyNode.js and Ruby
Node.js and Ruby
 
Coding in the context era
Coding in the context eraCoding in the context era
Coding in the context era
 
Briefly Rust - Daniele Esposti - Codemotion Rome 2017
Briefly Rust - Daniele Esposti - Codemotion Rome 2017Briefly Rust - Daniele Esposti - Codemotion Rome 2017
Briefly Rust - Daniele Esposti - Codemotion Rome 2017
 
Vagrant, Ansible and Docker - How they fit together for productive flexible d...
Vagrant, Ansible and Docker - How they fit together for productive flexible d...Vagrant, Ansible and Docker - How they fit together for productive flexible d...
Vagrant, Ansible and Docker - How they fit together for productive flexible d...
 
[Js hcm] Java script- Testing the awesome
[Js hcm] Java script- Testing the awesome[Js hcm] Java script- Testing the awesome
[Js hcm] Java script- Testing the awesome
 
CRaSH the shell for the JVM
CRaSH the shell for the JVMCRaSH the shell for the JVM
CRaSH the shell for the JVM
 
Mocha, chai and sinon
Mocha, chai and sinonMocha, chai and sinon
Mocha, chai and sinon
 
LuaNode: Asynchronous I/O for Lua
LuaNode: Asynchronous I/O for LuaLuaNode: Asynchronous I/O for Lua
LuaNode: Asynchronous I/O for Lua
 
Introduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" EditionIntroduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" Edition
 
BBC's GraphDB (formerly Owlim) AWS Cloud Migration
BBC's GraphDB (formerly Owlim) AWS Cloud MigrationBBC's GraphDB (formerly Owlim) AWS Cloud Migration
BBC's GraphDB (formerly Owlim) AWS Cloud Migration
 

Similar to Lessons from 4 years of driver develoment

Intro to JavaScript Testing
Intro to JavaScript TestingIntro to JavaScript Testing
Intro to JavaScript Testing
Ran Mizrahi
 
Introduction to node.js by Ran Mizrahi @ Reversim Summit
Introduction to node.js by Ran Mizrahi @ Reversim SummitIntroduction to node.js by Ran Mizrahi @ Reversim Summit
Introduction to node.js by Ran Mizrahi @ Reversim Summit
Ran Mizrahi
 
Cook Up a Runtime with The New OSGi Resolver - Neil Bartlett
Cook Up a Runtime with The New OSGi Resolver - Neil BartlettCook Up a Runtime with The New OSGi Resolver - Neil Bartlett
Cook Up a Runtime with The New OSGi Resolver - Neil Bartlett
mfrancis
 
Intro to PHP Testing
Intro to PHP TestingIntro to PHP Testing
Intro to PHP Testing
Ran Mizrahi
 
Continuous Integration for IOS Apps
Continuous Integration for IOS AppsContinuous Integration for IOS Apps
Continuous Integration for IOS Apps
Allan Davis
 
Advanced I/O in browser
Advanced I/O in browserAdvanced I/O in browser
Advanced I/O in browser
Eugene Lazutkin
 
Ab(Using) the MetaCPAN API for Fun and Profit v2013
Ab(Using) the MetaCPAN API for Fun and Profit v2013Ab(Using) the MetaCPAN API for Fun and Profit v2013
Ab(Using) the MetaCPAN API for Fun and Profit v2013
Olaf Alders
 
Introduction to Express and Grunt
Introduction to Express and GruntIntroduction to Express and Grunt
Introduction to Express and Grunt
Peter deHaan
 
Best Of Jdk 7
Best Of Jdk 7Best Of Jdk 7
Best Of Jdk 7
Kaniska Mandal
 
Scalable JavaScript
Scalable JavaScriptScalable JavaScript
Scalable JavaScript
Ynon Perek
 
Testing con spock
Testing con spockTesting con spock
Testing con spock
Fátima Casaú Pérez
 
How we're building Wercker
How we're building WerckerHow we're building Wercker
How we're building Wercker
Micha Hernandez van Leuffen
 
DanNotes 2013: OpenNTF Domino API
DanNotes 2013: OpenNTF Domino APIDanNotes 2013: OpenNTF Domino API
DanNotes 2013: OpenNTF Domino API
Paul Withers
 
Connect2016 AD1387 Integrate with XPages and Java
Connect2016 AD1387 Integrate with XPages and JavaConnect2016 AD1387 Integrate with XPages and Java
Connect2016 AD1387 Integrate with XPages and Java
Julian Robichaux
 
AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...
AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...
AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...
panagenda
 
eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...
eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...
eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...
PatrickCrompton
 
Oscon Java Testing on the Fast Lane
Oscon Java Testing on the Fast LaneOscon Java Testing on the Fast Lane
Oscon Java Testing on the Fast Lane
Andres Almiray
 
What to expect from Java 9
What to expect from Java 9What to expect from Java 9
What to expect from Java 9
Ivan Krylov
 
Ember and containers
Ember and containersEmber and containers
Ember and containers
Matthew Beale
 
How to build a PostgreSQL-backed website quickly
How to build a PostgreSQL-backed website quicklyHow to build a PostgreSQL-backed website quickly
How to build a PostgreSQL-backed website quickly
John Ashmead
 

Similar to Lessons from 4 years of driver develoment (20)

Intro to JavaScript Testing
Intro to JavaScript TestingIntro to JavaScript Testing
Intro to JavaScript Testing
 
Introduction to node.js by Ran Mizrahi @ Reversim Summit
Introduction to node.js by Ran Mizrahi @ Reversim SummitIntroduction to node.js by Ran Mizrahi @ Reversim Summit
Introduction to node.js by Ran Mizrahi @ Reversim Summit
 
Cook Up a Runtime with The New OSGi Resolver - Neil Bartlett
Cook Up a Runtime with The New OSGi Resolver - Neil BartlettCook Up a Runtime with The New OSGi Resolver - Neil Bartlett
Cook Up a Runtime with The New OSGi Resolver - Neil Bartlett
 
Intro to PHP Testing
Intro to PHP TestingIntro to PHP Testing
Intro to PHP Testing
 
Continuous Integration for IOS Apps
Continuous Integration for IOS AppsContinuous Integration for IOS Apps
Continuous Integration for IOS Apps
 
Advanced I/O in browser
Advanced I/O in browserAdvanced I/O in browser
Advanced I/O in browser
 
Ab(Using) the MetaCPAN API for Fun and Profit v2013
Ab(Using) the MetaCPAN API for Fun and Profit v2013Ab(Using) the MetaCPAN API for Fun and Profit v2013
Ab(Using) the MetaCPAN API for Fun and Profit v2013
 
Introduction to Express and Grunt
Introduction to Express and GruntIntroduction to Express and Grunt
Introduction to Express and Grunt
 
Best Of Jdk 7
Best Of Jdk 7Best Of Jdk 7
Best Of Jdk 7
 
Scalable JavaScript
Scalable JavaScriptScalable JavaScript
Scalable JavaScript
 
Testing con spock
Testing con spockTesting con spock
Testing con spock
 
How we're building Wercker
How we're building WerckerHow we're building Wercker
How we're building Wercker
 
DanNotes 2013: OpenNTF Domino API
DanNotes 2013: OpenNTF Domino APIDanNotes 2013: OpenNTF Domino API
DanNotes 2013: OpenNTF Domino API
 
Connect2016 AD1387 Integrate with XPages and Java
Connect2016 AD1387 Integrate with XPages and JavaConnect2016 AD1387 Integrate with XPages and Java
Connect2016 AD1387 Integrate with XPages and Java
 
AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...
AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...
AD1387: Outside The Box: Integrating with Non-Domino Apps using XPages and Ja...
 
eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...
eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...
eSynergy Andy Hawkins - Enabling DevOps through next generation configuration...
 
Oscon Java Testing on the Fast Lane
Oscon Java Testing on the Fast LaneOscon Java Testing on the Fast Lane
Oscon Java Testing on the Fast Lane
 
What to expect from Java 9
What to expect from Java 9What to expect from Java 9
What to expect from Java 9
 
Ember and containers
Ember and containersEmber and containers
Ember and containers
 
How to build a PostgreSQL-backed website quickly
How to build a PostgreSQL-backed website quicklyHow to build a PostgreSQL-backed website quickly
How to build a PostgreSQL-backed website quickly
 

More from christkv

From SQL to MongoDB
From SQL to MongoDBFrom SQL to MongoDB
From SQL to MongoDB
christkv
 
New in MongoDB 2.6
New in MongoDB 2.6New in MongoDB 2.6
New in MongoDB 2.6
christkv
 
Storage talk
Storage talkStorage talk
Storage talk
christkv
 
Mongo db ecommerce
Mongo db ecommerceMongo db ecommerce
Mongo db ecommerce
christkv
 
Cdr stats-vo ip-analytics_solution_mongodb_meetup
Cdr stats-vo ip-analytics_solution_mongodb_meetupCdr stats-vo ip-analytics_solution_mongodb_meetup
Cdr stats-vo ip-analytics_solution_mongodb_meetup
christkv
 
Mongodb intro
Mongodb introMongodb intro
Mongodb intro
christkv
 
Schema design
Schema designSchema design
Schema design
christkv
 
Node js mongodriver
Node js mongodriverNode js mongodriver
Node js mongodriver
christkv
 
Node.js and ruby
Node.js and rubyNode.js and ruby
Node.js and ruby
christkv
 

More from christkv (9)

From SQL to MongoDB
From SQL to MongoDBFrom SQL to MongoDB
From SQL to MongoDB
 
New in MongoDB 2.6
New in MongoDB 2.6New in MongoDB 2.6
New in MongoDB 2.6
 
Storage talk
Storage talkStorage talk
Storage talk
 
Mongo db ecommerce
Mongo db ecommerceMongo db ecommerce
Mongo db ecommerce
 
Cdr stats-vo ip-analytics_solution_mongodb_meetup
Cdr stats-vo ip-analytics_solution_mongodb_meetupCdr stats-vo ip-analytics_solution_mongodb_meetup
Cdr stats-vo ip-analytics_solution_mongodb_meetup
 
Mongodb intro
Mongodb introMongodb intro
Mongodb intro
 
Schema design
Schema designSchema design
Schema design
 
Node js mongodriver
Node js mongodriverNode js mongodriver
Node js mongodriver
 
Node.js and ruby
Node.js and rubyNode.js and ruby
Node.js and ruby
 

Recently uploaded

Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
janagijoythi
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
Priyanka Aash
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
Ivanti
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
Priyanka Aash
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
Priyanka Aash
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 

Recently uploaded (20)

Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
 
Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
(CISOPlatform Summit & SACON 2024) Regulation & Response In Banks.pdf
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
(CISOPlatform Summit & SACON 2024) Orientation by CISO Platform_ Using CISO P...
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 

Lessons from 4 years of driver develoment

  • 1. 4 years of Node.js and Mongodb Christian Amor Kvalheim Now at 10gen Friday, August 16, 13
  • 2. How did I get into Node.js Friday, August 16, 13
  • 3. Thu Jan 7 13:53:14 2010 First Commit Friday, August 16, 13
  • 5. IT was crap • It had a “Hash” and “Ordered Hash” class • Complete mess with horrible structure • Used Hacked JSpec for testing at the start Friday, August 16, 13
  • 6. Node was liquid • Constant changes in Node.js • Constant changes in driver to work with Node.js • OH yeah no NPM yet Friday, August 16, 13
  • 8. Types Oh Types • No int64 support in Javascript • Oh yeah Numbers is a Double (53 bit int max) • Decided to use Google Long closure library class Friday, August 16, 13
  • 9. Features • Started adding features like GridFS in late jan 2010 • Replicaset and Mongos support in 2010/2011 • Stream support for Cursor and GridFS • High Availability Friday, August 16, 13
  • 10. C++ • Profiled and found BSON parser to be main bottleneck • Wrote horrible C++ parser • Later cleaned up by a real C++ developer from Lucasarts (RIP) Friday, August 16, 13
  • 11. C++ • Initial version had own implementation of all BSON types • C++ Long, Binary etc. • Hard as hell to maintain • Bought me nothing in the end asV8 changed Friday, August 16, 13
  • 12. Making JS BSON faster Buffers • Another major refactoring to support properly • Calculate size of Document • Single allocation instead of multiple small ones • Using Buffers (optimized by v8) • Indexed entry a[0] direct memory access Friday, August 16, 13
  • 13. Pool handling • No traditional Connection Pool • No checking OUT and IN Connections • Connection pinning • The “Spice” must flow Friday, August 16, 13
  • 14. Write concerns • Mongodb wire protocol • Insert/Update/Remove + GetLastError • Operations must be pipelined • Write the op + GetLastError in one operation Friday, August 16, 13
  • 15. Reconnecting • No Blocking == difficult reconnect • Buffer operations until reconnect • Still a possible problem waiting for a solution Friday, August 16, 13
  • 16. Custom Serialization off Object to BSON • Added toBSON support • Specify on your objects and return the true representation of the object • Modeled after toJSON Friday, August 16, 13
  • 17. Replicaset • Problem • Servers come and leave the cluster • Need to update status of servers • Need to pick the right server for the readPreference Friday, August 16, 13
  • 18. Replicaset • Single thread SUCKS • Have to use setTimeout/setInterval • Has tons of subtle state changes • Never going to get Webworkers in Node.js • Processes are to granular Friday, August 16, 13
  • 20. Windows • Hello Windows Users • Node.js goes Windows • What to do About C++ ? • Build DLL’s for win32 and win64 in Windows 7 and package with driver Friday, August 16, 13
  • 21. Windows Build Hell • Oh yeah Node extensions only build with Visual Studio 2010 + Windows 7 64 bit SDK • Maybe better now but I doubt it • Node-gyp will fail to build extensions on Win causes confusion Friday, August 16, 13
  • 22. Windows Build Hell • Confusion about build process on Windows • More “Cannot” install module questions on Win than you can swing a cat at • Mostly PBKC • Who the Hell still uses XP (SERIOUSLY) Friday, August 16, 13
  • 23. ARM • Build C++ extension on ARM • Learned about memory alignment difference between ARM and X86 • Runs on Raspberry PI (HUZZA) Friday, August 16, 13
  • 25. Regrets var A = function() { this.something = "a" } A.prototype.some = function() { return this.something } Friday, August 16, 13
  • 26. Should have var B = function() { var something = "a" this.some = function() { return something } } Friday, August 16, 13
  • 27. Lesson • If it’s possible to access internal state they will access internal state • Only expose state and functions you wish end user should be able to use • Examples of My BAD • Collection class • Cursor class Friday, August 16, 13
  • 29. Testing it ALL • In the beginning • Simple integration testing with JSpec • Later moved to JUnit Friday, August 16, 13
  • 30. 3D Reality hits Mongo 2.0 Mongo 2.2 Mongo 2.4 Node 0.8 Node 0.10 Node 0.X with auth replicaset sharding + 2.2 features + 2.4 features etc etc etc etc etc etc Friday, August 16, 13
  • 31. A New Beginning • New test framework • Introduce configurations (replset/etc) • Introduce runners (in auth mode, etc) • Run tests against configurations • Test does not manage configurations • Still not there • Need to work on the concept more Friday, August 16, 13
  • 33. Documentation • Had to provide tons of usage examples • Realized I had them in code already • Built some tools to help me generate most of the docs • Spaghetti JS + Python Spinx Friday, August 16, 13
  • 34. Documentation /** * Example of inserting a document containing functions * * @_class collection * @_function insert * @ignore */ exports.test = function(configuration, test) { var db = configuration.db(); // DOC_LINE var db = new Db('test', new Server('locahost', 27017)); // DOC_START // Fetch a collection to insert document into var collection = db.collection("simple_document_insert_with_function_safe"); // Insert a single document collection.insert({hello:'world' , func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) { test.equal(null, err); // Fetch the document collection.findOne({hello:'world'}, function(err, item) { test.equal(null, err); test.ok("function() {}", item.code); test.done(); }) }); // DOC_END } Friday, August 16, 13
  • 37. On Backwards Compatibility • Drivers should be boring • Change is bad • Yeah some of the API sucks but to many depends on it • This week CoffeeScript next week LiveScript (REALLY?) Friday, August 16, 13
  • 38. On Managing The Project • Challenge people to help • Be specific about what you need • Give credit for pull requests • Ignore flames or criticism • Just ignore bad faith issues • Your code is always going to suck, roll with it Friday, August 16, 13
  • 40. Take aways • No Environment is Static • Testing Drivers isVery Hard • Don’t take criticism personally • Challenge people to help you • Most people are nice and will help • Credit people who help you Friday, August 16, 13
  • 41. In Conclusion • It’s been A crazy 3 1/2 years of serving the Node community • I’ve made tons of mistakes a long the way • I’m sure to make a ton of mistakes in the future • I would do it all again in a second Friday, August 16, 13
  • 42. Feel free to Ask Questions Friday, August 16, 13