GlobalsDB: Its significance for Node.js Developers

4,108 views
3,911 views

Published on

Presentation given at the London Node Users Group, 2011

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
4,108
On SlideShare
0
From Embeds
0
Number of Embeds
46
Actions
Shares
0
Downloads
28
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

GlobalsDB: Its significance for Node.js Developers

  1. 1. The Globals Database Its significance for Node.js developers Rob Tweed M/Gateway Developments Ltd http://www.mgateway.com Twitter: @rtweed
  2. 2. Background <ul><li>Director, M/Gateway Developments Ltd </li></ul><ul><li>Web/Ajax/Mobile web technologies </li></ul><ul><li>Healthcare, Financial Services </li></ul><ul><li>Business/enterprise applications </li></ul><ul><ul><li>Browser-based </li></ul></ul><ul><ul><li>Interactive </li></ul></ul><ul><ul><li>Secure </li></ul></ul><ul><ul><li>Internet & Intranet </li></ul></ul><ul><ul><li>Database intensive </li></ul></ul>
  3. 3. What databases? <ul><li>“ Global storage-based” </li></ul><ul><ul><li>Cach é (“native storage”) </li></ul></ul><ul><ul><li>GT.M </li></ul></ul><ul><li>Very high performance & reliability </li></ul><ul><li>Very low maintenance </li></ul><ul><li>NoSQL database that pre-dates the NoSQL era </li></ul><ul><li>Tried and tested for business-critical use </li></ul><ul><li>Dominate healthcare and financial services </li></ul><ul><li>Much wider applicability, but generally little known </li></ul><ul><ul><li>Key limitation: accessed via an outdated language </li></ul></ul>
  4. 4. What’s my interest in Node.js? <ul><li>Using Javascript since late 1990’s </li></ul><ul><li>Douglas Crockford </li></ul><ul><ul><li>“ Loopage” </li></ul></ul><ul><li>Javascript on the server </li></ul><ul><ul><li>Direct access to global storage via Javascript? </li></ul></ul>
  5. 5. Globals Database http://globalsdb.org
  6. 6. What is Globals? <ul><li>Core database engine from Cach é </li></ul><ul><li>No native language </li></ul><ul><li>None of Cach é’s object/relational functionality </li></ul><ul><li>Just the core global-storage engine </li></ul><ul><li>Free </li></ul><ul><ul><li>But not Open Source </li></ul></ul><ul><ul><li>Otherwise extremely liberal licence </li></ul></ul><ul><li>Two APIs: </li></ul><ul><ul><li>Java </li></ul></ul><ul><ul><li>Node.js </li></ul></ul>
  7. 7. Global storage <ul><li>Global Persistent Variables </li></ul><ul><ul><li>aka “Globals” </li></ul></ul><ul><li>Globals = unit of persistent storage </li></ul><ul><ul><li>Schema-free </li></ul></ul><ul><ul><li>Hierarchically structured </li></ul></ul><ul><ul><li>Sparse </li></ul></ul><ul><ul><li>Dynamic </li></ul></ul><ul><ul><li>“ persistent associative array” </li></ul></ul><ul><ul><li>Each array element = “a global node” </li></ul></ul>
  8. 8. Anatomy of a Global Node <ul><li>A Global node has: </li></ul><ul><ul><li>A name </li></ul></ul><ul><ul><li>0, 1 or more subscripts </li></ul></ul><ul><ul><li>String value </li></ul></ul><ul><ul><li>globalName[subscript1,subscript2,..subscript n ] == value </li></ul></ul>
  9. 9. Simple key/value storage telephone “617-555-1414” “Tweed, Rob” “211-555-9012” “James, George” telephone[“617-555-1414”] == “Tweed, Rob”
  10. 10. Multi-dimensional structures “firstNode” 5 “lastNode” 2 “node” 4 list “myList” “nodeCounter” 5 “nextNode” 2 “previousNode” 5 “value” “George” 5 “nextNode” 4 “value” “Rob” 2 “value” “John” “previousNode” 4
  11. 11. Globals = Universal NoSQL Engine <ul><li>Document storage </li></ul><ul><ul><li>Automatic mapping to/from JSON </li></ul></ul><ul><li>Columnar </li></ul><ul><li>Graph </li></ul><ul><li>Native XML database </li></ul><ul><li>Relational </li></ul><ul><li>Object </li></ul>
  12. 12. Global storage as a db technology <ul><li>Low-level storage engine </li></ul><ul><li>No built-in indexing </li></ul><ul><ul><li>You create/define the indices you need </li></ul></ul><ul><li>Schema-free </li></ul><ul><ul><li>Database behaviour defined in your application logic </li></ul></ul><ul><li>No built-in query language </li></ul><ul><ul><li>APIs for node hierarchy traversal </li></ul></ul><ul><li>All the basic building blocks for creating your own tailored NoSQL database </li></ul>
  13. 13. Why bother? <ul><li>Lots of Open Source NoSQL databases for Node.js: </li></ul><ul><ul><li>Redis </li></ul></ul><ul><ul><li>CouchDB </li></ul></ul><ul><ul><li>MongoDB </li></ul></ul><ul><ul><li>Riak </li></ul></ul><ul><ul><li>etc </li></ul></ul>
  14. 14. Globals Node.js APIs <ul><li>In-process </li></ul><ul><ul><li>No network layer </li></ul></ul><ul><li>API style: </li></ul><ul><ul><li>Asynchronous </li></ul></ul><ul><ul><li>Synchronous </li></ul></ul>
  15. 15. Why Globals? My view: <ul><li>Ridiculously high performance </li></ul><ul><li>One single multi-purpose database </li></ul><ul><li>Synchronous coding </li></ul>
  16. 16. Objection, m’lud!
  17. 17. In-process database <ul><li>That sounds like too intimate a relationship </li></ul><ul><ul><li>A problem in the database will bring down the Node.js process </li></ul></ul><ul><li>On the other hand… </li></ul><ul><ul><li>No network bottleneck </li></ul></ul><ul><ul><li>Potential for very high performance </li></ul></ul>
  18. 18. Synchronous APIs <ul><li>Gospel according to St. Async of the Node </li></ul>
  19. 19. Introducing Q-Oper8 https://github.com/robtweed/Q-Oper8
  20. 20. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects
  21. 21. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Incoming request
  22. 22. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects If necessary, Put it on the queue
  23. 23. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects If child process is free, send first request Queue processing triggered
  24. 24. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Child process unavailable
  25. 25. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Process request
  26. 26. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Finished: Send back result object
  27. 27. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Process result and send response
  28. 28. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Child process available
  29. 29. Q-Oper8 architecture Main Node.js server process Pre-spawned Child processes Queue of request objects Queue processing triggered
  30. 30. Q-Oper8 benefits <ul><li>Child processes can use blocking I/O </li></ul><ul><ul><li>Only handle one request at a time </li></ul></ul><ul><ul><li>Nobody else to block </li></ul></ul><ul><li>Main server process uses kosher async logic </li></ul><ul><li>Main process is isolated from activity in child processes: </li></ul><ul><ul><li>Blocking I/O </li></ul></ul><ul><ul><li>Synchronous coding </li></ul></ul><ul><ul><li>Heavy computation </li></ul></ul><ul><ul><li>in-process database </li></ul></ul><ul><li>Allows Node.js to use multiple CPU cores </li></ul>
  31. 31. Q-Oper8 + Globals Main Node.js server process Pre-spawned Child processes Queue of request objects Pure async activity here Globals db APIs Globals db APIs Globals db APIs
  32. 32. Q-Oper8 + Globals Performance <ul><li>My test machine: </li></ul><ul><ul><li>HP Proliant ML115G Server </li></ul></ul><ul><ul><ul><li>4-core AMD Opteron 2.1GHz CPU </li></ul></ul></ul><ul><ul><ul><li>8 Gb memory </li></ul></ul></ul><ul><ul><ul><li>640 Gb 7200 RPM SATA hard drive </li></ul></ul></ul><ul><ul><ul><ul><li>(Western Digital Caviar Blue) </li></ul></ul></ul></ul><ul><ul><li>Ubuntu Linux 10.10 Server (64-bit) </li></ul></ul><ul><ul><li>Node.js 0.4.0 </li></ul></ul><ul><ul><li>Globals DB </li></ul></ul><ul><ul><li>Under £200-worth of machine </li></ul></ul>
  33. 33. Q-Oper8 No-op test <ul><li>Passing simple message each way </li></ul><ul><ul><li>No other processing </li></ul></ul><ul><li>Determine “steady state”: </li></ul><ul><ul><li>Add requests to queue as fast as child processes can process them </li></ul></ul><ul><li>Optimum performance with 3 child processes </li></ul><ul><li>18,350 per second </li></ul>
  34. 34. Q-Oper8 + Globals <ul><li>Increasing numbers of Set commands per request </li></ul><ul><ul><li>Each Set creates one new global node </li></ul></ul><ul><li>Different APIs: </li></ul><ul><ul><li>Async parallel </li></ul></ul><ul><ul><li>Async nested </li></ul></ul><ul><ul><li>Sync </li></ul></ul><ul><li>Measured steady-state maximum rate </li></ul>
  35. 35. Q-Oper8 + Globals
  36. 36. Q-Oper8 + Globals Sync Async parallel Async nested Requests/ sec No of Global Sets per request 1 2 5
  37. 37. Q-Oper8 + Globals Requests/ Sec (Sync) No of Global Sets per request 1000 500 200 100
  38. 38. Q-Oper8 + Globals Global sets/ Sec (Sync) No of Global Sets per request 1000 500 200 100
  39. 39. Conclusions <ul><li>Globals’ synchronous APIs are more than twice as fast as its asynchronous ones </li></ul><ul><li>The more Globals activity you do per request, the better the db performance </li></ul><ul><li>On my small server, performance maxed at 190,000 global node sets/sec </li></ul><ul><ul><li>Redis-benchmark on same server: 88,000 Sets/sec </li></ul></ul>
  40. 40. Globals in Use <ul><li>ewdDOM module </li></ul><ul><ul><li>Persistent XML DOM implementation </li></ul></ul><ul><ul><li>Uses Globals for storage </li></ul></ul><ul><ul><li>DOMs created and stored in globals </li></ul></ul><ul><ul><li>Access to DOMs is via synchronous APIs </li></ul></ul><ul><ul><li>APIs hit the data in situ, not an in-memory copy </li></ul></ul><ul><ul><li>https://github.com/robtweed/ewdDOM </li></ul></ul>
  41. 41. ewdDOM: Example var document = ewdDOM.getDocument('rob'); var node1 = document.createElement(&quot;testElement&quot;); document.getElementById('myNode').appendChild(node1); document.getElementById('myNewNode').text = ‘Some new text'; document.output(); <ul><li>Synchronous logic allows full use of OO syntax </li></ul><ul><li>Each level in the “dot syntax” represents a number of global node </li></ul><ul><li>manipulations, with result passed to its child method </li></ul><ul><li>Not feasible using async logic with call-backs </li></ul>
  42. 42. Globals in Use <ul><li>M/DB-g </li></ul><ul><ul><li>Emulation of Amazon SimpleDB </li></ul></ul><ul><ul><li>100% API-compatible </li></ul></ul><ul><ul><li>Development history: </li></ul></ul><ul><ul><ul><li>Pure global database native-language implementation (Cach é and GT.M) </li></ul></ul></ul><ul><ul><ul><li>Node.js + async access to GT.M </li></ul></ul></ul><ul><ul><ul><ul><li>Painful! Incomplete </li></ul></ul></ul></ul><ul><ul><ul><li>Node.js + sync access to Globals </li></ul></ul></ul><ul><ul><ul><ul><li>Easy! Line by line copy of original implementation </li></ul></ul></ul></ul>
  43. 43. Sync v Async <ul><li>Async perfect for event-handling and other I/O </li></ul><ul><li>Globals makes sync coding possible for database access/manipulation </li></ul><ul><ul><li>Higher performance </li></ul></ul><ul><ul><li>Easier and quicker to write </li></ul></ul><ul><ul><li>Much simpler to maintain </li></ul></ul><ul><ul><li>Allows creation of higher-level OO APIs </li></ul></ul><ul><ul><li>Node.js for business/enterprise applications becomes feasible to consider </li></ul></ul><ul><ul><ul><li>Developer and maintenance time = the primary IT costs </li></ul></ul></ul>
  44. 44. Globals: Conclusion <ul><li>Significance: </li></ul><ul><ul><li>Extremely high performance database for use with Node.js </li></ul></ul><ul><ul><li>Very adaptable database engine (universal NoSQL engine) </li></ul></ul><ul><ul><ul><li>One database for all needs </li></ul></ul></ul><ul><ul><ul><li>“ virtual storage” for Node.js? </li></ul></ul></ul><ul><ul><li>Possibility of synchronous database coding </li></ul></ul>
  45. 45. Appendix
  46. 46. Macbook Air benchmark <ul><li>OS X Lion </li></ul><ul><li>Ubuntu 10.10 VM + Fusion 3 </li></ul><ul><li>1 CPU + 512Mb memory: </li></ul><ul><ul><li>1 Q-Oper8 child process </li></ul></ul><ul><ul><li>Max global sets: ~ 38,000 / sec </li></ul></ul><ul><li>2 CPU + 512Mb memory: </li></ul><ul><ul><li>2 Q-Oper8 child process </li></ul></ul><ul><ul><li>Max global sets: ~ 77,000 / sec </li></ul></ul>
  47. 47. ewdDOM Benchmark <ul><li>Create small DOM programmatically </li></ul><ul><ul><li>177 X Write: </li></ul></ul><ul><ul><ul><li>164 X set </li></ul></ul></ul><ul><ul><ul><li>13 X kill </li></ul></ul></ul><ul><ul><li>322 X Read: </li></ul></ul><ul><ul><ul><li>110 X get </li></ul></ul></ul><ul><ul><ul><li>212 X data </li></ul></ul></ul><ul><li>1000 requests put on Q-Oper8 queue </li></ul><ul><li>3.5 seconds to process </li></ul><ul><ul><li>92,000 reads/sec + 50,570 writes/sec </li></ul></ul>

×