Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Persistence as a Language Feature

986 views

Published on

Presentation to the InterSystems Developer Community, UK 2017, describing my work on making JavaScript a first-class scripting language for Cache

Published in: Software
  • Be the first to comment

Data Persistence as a Language Feature

  1. 1. Copyright © 2017 M/Gateway Developments Ltd ^ redux: Data Persistence as a Language Feature Rob Tweed M/Gateway Developments Ltd Twitter: @rtweed http://qewdjs.com
  2. 2. Copyright © 2017 M/Gateway Developments Ltd A bit of background • Royal Marsden Hospital: 1980s • Touche Ross Management Consultants (now Deloitte): early 1990s – NHS-wide Networking Project • Independent consultant & software developer since 1993 – Specialising in Web and associated technologies, particularly in healthcare
  3. 3. Copyright © 2017 M/Gateway Developments Ltd Products • WebLink: – Co-creator of the original web connectivity for Caché • WebLink Developer: – Web Application Development framework • pre-cursor to CSP • eXtc: – XML DOM framework • EWD: – 2nd -generation web application development framework – XML based custom pages – Compiled to CSP or WebLink (or even PHP)
  4. 4. Copyright © 2017 M/Gateway Developments Ltd Example Usage
  5. 5. Copyright © 2017 M/Gateway Developments Ltd A bit of background • Node.js: – Since August 2010 (v0.2 had just come out!) – Pioneered the integration of Caché and Node.js – Was instrumental in getting the cache.node interface developed • Originally for GlobalsDB
  6. 6. Copyright © 2017 M/Gateway Developments Ltd Language or Database? • Caché ObjectScript: – programming language with integrated database? • Caché Database: – Database with integrated scripting language for its manipulation (Caché ObjectScript)?
  7. 7. Copyright © 2017 M/Gateway Developments Ltd Language • Caché ObjectScript shows its age – Rightly or wrongly, opinion from IT Mainstream developers is not favourable – Tainted, in particular, by this: • https://thedailywtf.com/articles/comments/A_Case_of_the_MUMPS/8 – As a result of the association with Mumps by, eg: – https://en.wikipedia.org/wiki/Cach%C3%A9_ObjectScript
  8. 8. Copyright © 2017 M/Gateway Developments Ltd Language • Caché ObjectScript typical objections: – Limited – Old-Fashioned – To a modern developer, it's: • A step back in time • Lacking in many of the language capabilities they expect • Lacks an extensive ecosystem of pre-built functions and modules • A career-limiting move
  9. 9. Copyright © 2017 M/Gateway Developments Ltd Database • Uniquely powerful • The original NoSQL database – Pre-dated the NoSQL movement by several decades • "Universal NoSQL" – AKA "Multi-Model database"
  10. 10. Copyright © 2017 M/Gateway Developments Ltd Universal NoSQL Published in 2010
  11. 11. Copyright © 2017 M/Gateway Developments Ltd Database • But in order to manipulate and use that database to its fullest capabilities, you need to learn its scripting language – Caché ObjectScript • A key reason why, as a database, it's almost unknown in the mainstream of IT – I've yet to meet a JavaScript developer who has even heard of Caché
  12. 12. Copyright © 2017 M/Gateway Developments Ltd Solution? • Recurring suggestions within the developer community that Caché ObjectScript needs modernising – New language capabilities – Bring its syntax into line with modern languages – Create an ecosystem of functions & modules
  13. 13. Copyright © 2017 M/Gateway Developments Ltd My Opinion • This is the wrong approach • Nobody in the mainstream of IT wants to learn a new language just to access a database • They'd expect to be able to use a modern language
  14. 14. Copyright © 2017 M/Gateway Developments Ltd Language or Database? • Is Caché first and foremost a database or a language? – Misses a unique feature of Caché which provides its key power and which: • We take for granted • The mainstream of IT don't "get"
  15. 15. Copyright © 2017 M/Gateway Developments Ltd What is a database? • It's that thing "over there" that persists our data
  16. 16. Copyright © 2017 M/Gateway Developments Ltd What is a database? • It's expected to be "over there" • It's a separate server / sub-system • It's accessed: – over a network connection – via some protocol and/or syntax • REST • SQL
  17. 17. Copyright © 2017 M/Gateway Developments Ltd So what? • Creates an explicit distinction between stuff you do in your programming language and persisted data • Creates separate disciplines – Programming experts – Database experts – Many people span both, but, to many programmers, databases are a mystery
  18. 18. Copyright © 2017 M/Gateway Developments Ltd The Unique Feature • The underlying "Global Storage" of Caché • Why? – It blurs the distinction between what's permanently persisted and what's in memory – In memory: sparse arrays – On disk: identical sparse arrays – Only difference in syntax terms is a preceeding ^
  19. 19. Copyright © 2017 M/Gateway Developments Ltd Revise the Definition • Yes it's a database and a language, but it warrants a new description: • It's a language with data persistence: – as a seamless language feature
  20. 20. Copyright © 2017 M/Gateway Developments Ltd The Problem • It's a language with data persistence: – as a seamless language feature • But the language is obsolete and a barrier to new entry and mainstream adoption
  21. 21. Copyright © 2017 M/Gateway Developments Ltd The Solution • It's a language with data persistence: – as a seamless language feature • Add a modern language as a first-class alternative to Caché ObjectScript – With data persistence as a similarly seamless feature – An equivalent, in that language, to ^
  22. 22. Copyright © 2017 M/Gateway Developments Ltd The Question • What Language would fit the bill?
  23. 23. Copyright © 2017 M/Gateway Developments Ltd From the Mainstream Perspective • One of the currently "big" languages – C / C++ too low level – Java – Ruby – Go – Python – JavaScript
  24. 24. Copyright © 2017 M/Gateway Developments Ltd From our Perspective • Must be as fast as Native Caché ObjectScript – For general-purpose coding – For database access • > 1 million Global Nodes per second
  25. 25. Copyright © 2017 M/Gateway Developments Ltd From both Perspectives • Should be a language suitable for all the kinds of application that Caché ObjectScript is currently used for: – Desktop – Browser-based – Mobile – Batch processing / server-side heavy-lifting
  26. 26. Copyright © 2017 M/Gateway Developments Ltd From both Perspectives • Should map seamlessly between Global Storage and a commonly-accepted native in-memory representation with that language – No "impedance mismatch" • That mapping should make sense within the context of that language – ie laguage-centric
  27. 27. Copyright © 2017 M/Gateway Developments Ltd Global Storage Key Features • Schema-Free • Hierarchically-structured • Dynamic creation / destruction • Ideally suited for "late-bound" access
  28. 28. Copyright © 2017 M/Gateway Developments Ltd Suitability: Java Feature Adoption Widespread General-purpose Code Speed Excellent Application suitability Not used in browser, but otherwise widespread acceptance and adoption Data persistence impedance mismatch Designed around pre-defined, compiled classes / schema, ie early- bound access
  29. 29. Copyright © 2017 M/Gateway Developments Ltd Suitability: Ruby Feature Adoption Fairly limited by comparison to other languages Well-liked by its users General-purpose Code Speed Poor Application suitability Not used in browser. Poor choice for mobile development Data persistence impedance mismatch Good
  30. 30. Copyright © 2017 M/Gateway Developments Ltd Suitability: Go Feature Adoption Fairly limited by comparison to other languages. Well-liked by its users General-purpose Code Speed Excellent Application suitability Not used in browser. Otherwise very good Data persistence impedance mismatch Not so good – designed for pre- defined, pre-compiled mappings Miscellaneous Can we depend on Google supporting it in the long term?
  31. 31. Copyright © 2017 M/Gateway Developments Ltd Suitability: Python Feature Adoption Pretty high and growing. Well-liked by its users General-purpose Code Speed Poor Application suitability Not used in browser. Not suitable for mobile development Data persistence impedance mismatch Good – dictionaries / JSON map well
  32. 32. Copyright © 2017 M/Gateway Developments Ltd Suitability: JavaScript Feature Adoption High and still growing. Node.js now massively popular. Language has some quirks! General-purpose Code Speed Node.js performance is excellent Application suitability Used in browser. Ideal for mobile development. Usual cited deficiency is CPU-heavy processing, but that can be addressed Data persistence impedance mismatch JavaScript objects are dynamic and map almost perfectly onto Global Storage Miscellaneous Only 1 language skill for entire development stack, front and back (Node.js at back-end)
  33. 33. Copyright © 2017 M/Gateway Developments Ltd JavaScript as a Native Language for Caché • It's a very good fit, for all sorts of reasons: – Interpreted – Loosely-typed in a similar way – JavaScript Objects (JSON) • Schema-free • Dynamic • hierarchical
  34. 34. Copyright © 2017 M/Gateway Developments Ltd Global Storage myGlobal("a")=123 myGlobal("b","c1")="foo" myGlobal("b","c2")="foo2" myGlobal("d","e1","f1")="bar1" myGlobal("d","e1","f2")="bar2" myGlobal("d","e2","f1")="bar1" myGlobal("d","e2","f2")="bar2" myGlobal("d","e2","f3")="bar3" myGlobal "a" 123 "b" "c2" "foo2" "d" "c1" "foo" "e2" "e1" "f2" "bar2" "f1" "bar1" "f2" "bar2" "f1" "bar1" "f3" "bar3" Hierarchical diagram representing a Global
  35. 35. Copyright © 2017 M/Gateway Developments Ltd myGlobal = { a: 123, b: { c1: 'foo', c2: 'foo2' } d: { e1: { f1: 'bar1', f2: 'bar2' }, e2: { f1: 'bar1', f2: 'bar2', f3: 'bar3' } } } myGlobal("a")=123 myGlobal("b","c1")="foo" myGlobal("b","c2")="foo2" myGlobal("d","e1","f1")="bar1" myGlobal("d","e1","f2")="bar2" myGlobal("d","e2","f1")="bar1" myGlobal("d","e2","f2")="bar2" myGlobal("d","e2","f3")="bar3" One to one correspondence between any tree of Global nodes and a Javascript object
  36. 36. Copyright © 2017 M/Gateway Developments Ltd myGlobal = { a: 123, b: { c1: 'foo', c2: 'foo2' } d: { e1: { f1: 'bar1', f2: 'bar2' }, e2: { f1: 'bar1', f2: 'bar2', f3: 'bar3' } } } myGlobal("a")=123 myGlobal("b","c1")="foo" myGlobal("b","c2")="foo2" myGlobal("d","e1","f1")="bar1" myGlobal("d","e1","f2")="bar2" myGlobal("d","e2","f1")="bar1" myGlobal("d","e2","f2")="bar2" myGlobal("d","e2","f3")="bar3" Subscript <=> Property
  37. 37. Copyright © 2017 M/Gateway Developments Ltd myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3 Global Node JavaScript Object
  38. 38. Copyright © 2017 M/Gateway Developments Ltd myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3 Global Node JavaScript Object But… JavaScript Objects are in memory Global Nodes are persistent, on disk
  39. 39. Copyright © 2017 M/Gateway Developments Ltd myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3 Global Node JavaScript Object Could we abstract Global Nodes as Persistent JavaScript Objects?
  40. 40. Copyright © 2017 M/Gateway Developments Ltd ewd-document-store • JavaScript abstraction of Global Storage – https://github.com/robtweed/ewd-document-store
  41. 41. Copyright © 2017 M/Gateway Developments Ltd ewd-document-store • Abstracts Global Storage as: – A document database – Persistent JavaScript Objects
  42. 42. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode(name, subscripts); Represents a node within a Global (aka Document) May or may not physically exist at this point ie this does NOT create a node DocumentNode Object
  43. 43. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode(name, subscripts); Represents a node within a document's hierarchy May or may not physically exist at this point ie this does NOT create a node Creates an instance of an object that represents that node, with a set of methods and properties to manipulate and access that node DocumentNode Object
  44. 44. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") DocumentNode Object
  45. 45. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") create: dnode.value = 'foo2'; DocumentNode Object
  46. 46. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") create: dnode.value = 'foo2'; get: var x = dnode.value; DocumentNode Object
  47. 47. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") create: dnode.value = 'foo2'; get: var x = dnode.value; value is a read/write property DocumentNode Object
  48. 48. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") create: dnode.value = 'foo2'; get: var x = dnode.value; delete: dnode.delete(); DocumentNode Object
  49. 49. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") create: dnode.value = 'foo2'; get: var x = dnode.value; delete: dnode.delete(); Note: delete will also delete any lower-level DocumentNodes in the underlying Global Storage tree DocumentNode Object
  50. 50. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']); represents myDoc("d","e2") create: dnode.value = 'foo2'; get: var x = dnode.value; delete: dnode.delete(); exists?: var exists = dnode.exists; // true | false DocumentNode Object
  51. 51. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d']); dnode.hasChildren // true dnode.hasValue // false dnode.exists // true myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1")="bar1" myDoc("d","e1","f2")="bar2" myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3"
  52. 52. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2', 'f3']); dnode.hasChildren // false dnode.hasValue // true dnode.exists // true myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1")="bar1" myDoc("d","e1","f2")="bar2" myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3"
  53. 53. Copyright © 2017 M/Gateway Developments Ltd var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2', 'f4']); dnode.hasChildren // false dnode.hasValue // false dnode.exists // false Even though the node doesn’t physically exist, you can define a DocumentNode Object for it myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1")="bar1" myDoc("d","e1","f2")="bar2" myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3"
  54. 54. Copyright © 2017 M/Gateway Developments Ltd myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3" var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']); Traversing Documents
  55. 55. Copyright © 2017 M/Gateway Developments Ltd myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3" var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']); e2Node.forEachChild(function(nodeName) { // do something with the node name / subscript for this iteration }); forEachChild() Method
  56. 56. Copyright © 2017 M/Gateway Developments Ltd myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3" var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']); e2Node.forEachChild(function(nodeName) { // 1st iteration nodeName === 'f1’ }); forEachChild() Method
  57. 57. Copyright © 2017 M/Gateway Developments Ltd myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3" var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']); e2Node.forEachChild(function(nodeName) { // 2nd iteration nodeName === 'f2’ }); forEachChild() Method
  58. 58. Copyright © 2017 M/Gateway Developments Ltd myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3" var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']); e2Node.forEachChild(function(nodeName) { // 3rd iteration nodeName === 'f3’ }); forEachChild() Method
  59. 59. Copyright © 2017 M/Gateway Developments Ltd myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")="bar3" var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']); e2Node.forEachChild(function(nodeName) { console.log(nodeName); }); forEachChild() Method f1 f2 f3
  60. 60. Copyright © 2017 M/Gateway Developments Ltd getDocument() var doc = new this.documentStore.DocumentNode('myDoc'); var myObj = doc.getDocument(); myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1a")="bar1a" myDoc("d","e1","f2a")="bar2a" myDoc("d","e2","f1b")="bar1b" myDoc("d","e2","f2b")="bar2b" myDoc("d","e2","f3b")="bar3b" myObj = { a: 123, b: { c1: 'foo', c2: 'foo2' }, d: { e1: { f1a: 'bar1a', f2a: 'bar2a' }, e2: { f1b: 'bar1b', f2b: 'bar2b', f3b: 'bar3b' } } } creates
  61. 61. Copyright © 2017 M/Gateway Developments Ltd getDocument() var doc = new this.documentStore.DocumentNode('myDoc'); var myObj = doc.getDocument(); myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1a")="bar1a" myDoc("d","e1","f2a")="bar2a" myDoc("d","e2","f1b")="bar1b" myDoc("d","e2","f2b")="bar2b" myDoc("d","e2","f3b")="bar3b" myObj = { a: 123, b: { c1: 'foo', c2: 'foo2' }, d: { e1: { f1a: 'bar1a', f2a: 'bar2a' }, e2: { f1b: 'bar1b', f2b: 'bar2b', f3b: 'bar3b' } } } myObj is a standard in-memory JavaScript object containing a copy of the data that was held within the DocumentNode's sub-tree of nodes creates
  62. 62. Copyright © 2017 M/Gateway Developments Ltd setDocument() var myObj = { // create the object shown to the right }; myObj = { a: 123, b: { c1: 'foo', c2: 'foo2' }, d: { e1: { f1a: 'bar1a', f2a: 'bar2a' }, e2: { f1b: 'bar1b', f2b: 'bar2b', f3b: 'bar3b' } } }
  63. 63. Copyright © 2017 M/Gateway Developments Ltd setDocument() var myObj = { // create the object shown to the right }; var doc = new this.documentStore.DocumentNode('myDoc'); // doesn't yet exist on disk myObj = { a: 123, b: { c1: 'foo', c2: 'foo2' }, d: { e1: { f1a: 'bar1a', f2a: 'bar2a' }, e2: { f1b: 'bar1b', f2b: 'bar2b', f3b: 'bar3b' } } }
  64. 64. Copyright © 2017 M/Gateway Developments Ltd setDocument() var myObj = { // create the object shown to the right }; var doc = new this.documentStore.DocumentNode('myDoc'); doc.setDocument(myObj); myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1a")="bar1a" myDoc("d","e1","f2a")="bar2a" myDoc("d","e2","f1b")="bar1b" myDoc("d","e2","f2b")="bar2b" myDoc("d","e2","f3b")="bar3b" myObj = { a: 123, b: { c1: 'foo', c2: 'foo2' }, d: { e1: { f1a: 'bar1a', f2a: 'bar2a' }, e2: { f1b: 'bar1b', f2b: 'bar2b', f3b: 'bar3b' } } } creates
  65. 65. Copyright © 2017 M/Gateway Developments Ltd setDocument() var myObj = { // create the object shown to the right }; var doc = new this.documentStore.DocumentNode('myDoc'); doc.setDocument(myObj); myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1a")="bar1a" myDoc("d","e1","f2a")="bar2a" myDoc("d","e2","f1b")="bar1b" myDoc("d","e2","f2b")="bar2b" myDoc("d","e2","f3b")="bar3b" myObj = { a: 123, b: { c1: 'foo', c2: 'foo2' }, d: { e1: { f1a: 'bar1a', f2a: 'bar2a' }, e2: { f1b: 'bar1b', f2b: 'bar2b', f3b: 'bar3b' } } } myDoc is created on disk, containing a copy of the data that was held within the myObj object, now mapped to Global Storage nodes creates
  66. 66. Copyright © 2017 M/Gateway Developments Ltd Making This Possible • The cache.node interface • Included with Caché (and IRIS) • Runs in-process with Caché • Connects to Caché's C++ Callin Interface – Directly accessing the Caché database engine – Bypassing the language interface
  67. 67. Copyright © 2017 M/Gateway Developments Ltd cache.node interface Node.js Process Caché Process cache.node module C++call-ininterface JavaScript Globals Functions Objects
  68. 68. Copyright © 2017 M/Gateway Developments Ltd cache.node APIs • Low-level • Basic Global primitives • "Global-centric" • Not JavaScript-centric
  69. 69. Copyright © 2017 M/Gateway Developments Ltd ewd-document-store • Builds on top of the cache.node APIs • Creates a JavaScript-centric abstraction of Global Storage – Mapping to/from JavaScript Objects • The JavaScript equivalent of ^ – Local JavaScript objects (in-memory) – Persistent JavaScript objects (on-disk)
  70. 70. Copyright © 2017 M/Gateway Developments Ltd Node.js Architecture • Everything executes in a single process – You might have 1000's of users doing stuff concurrently • They're all sharing the same process!
  71. 71. Copyright © 2017 M/Gateway Developments Ltd Brainchild of Ryan Dahl
  72. 72. Copyright © 2017 M/Gateway Developments Ltd Node.js = server-side JavaScript • Ryan Dahl wasn't actually a big JavaScript fan – Node.js wasn't designed to be server-side JavaScript • But he realised that it was a convenient language for such an environment, as it's designed for exactly the same kind of way of working in the browser • Google's V8 JavaScript engine was Open Sourced, so he used it to provide the language for Node.js – (which is actually mostly written in C++)
  73. 73. Copyright © 2017 M/Gateway Developments Ltd Node.js Architecture • Non-blocking I/O – No user activity must block the process, or everyone grinds to a halt • Event-driven, asynchronous logic – Fire off the request, but don't wait for the results • Carry on with the next task – When results come back, handle them at the next opportunity
  74. 74. Copyright © 2017 M/Gateway Developments Ltd ewd-document-store requires synchronous APIs var myDoc = new this.documentStore.DocumentNode('myDoc'); var f3Node = myDoc.$('d').$('e2').$('f3'); myDoc.$d.$e2.$f3.value = 'New value'; myDoc("a")=123 myDoc("b","c1")="foo" myDoc("b","c2")="foo2" myDoc("d","e1","f1")="bar1" myDoc("d","e1","f2")="bar2" myDoc("d","e2","f1")="bar1" myDoc("d","e2","f2")="bar2" myDoc("d","e2","f3")=”New value" Requires properly chainable functions
  75. 75. Copyright © 2017 M/Gateway Developments Ltd The Problem is Concurrency • All concurrent users in Node.js share the same process
  76. 76. Copyright © 2017 M/Gateway Developments Ltd Is there a way to avoid this? • Would it be possible to create a locally- available environment where my Node.js code runs in an isolated container where concurrency isn't an issue?
  77. 77. Copyright © 2017 M/Gateway Developments Ltd
  78. 78. Copyright © 2017 M/Gateway Developments Ltd What Is QEWD? • Essentially it's a multi-purpose Node.js- based run-time Platform
  79. 79. Copyright © 2017 M/Gateway Developments Ltd What Is QEWD? • Essentially it's a multi-purpose Node.js- based run-time Platform • Creates an isolated run-time container for your message/request handler functions, allowing: – CPU-intensive work – Database abstractions using synchronous logic
  80. 80. Copyright © 2017 M/Gateway Developments Ltd QEWD's Architecture • Master Process • Pool of Worker Processes
  81. 81. Copyright © 2017 M/Gateway Developments Ltd QEWD's Architecture • Master Process – Handles and queues all incoming requests from client • HTTP/REST requests via Express or Koa.js • WebSocket requests via socket.io – Returns responses to client
  82. 82. Copyright © 2017 M/Gateway Developments Ltd QEWD's Architecture • Pool of Persistent Worker Processes – Where all the processing occurs – A single queued request is dispatched to an available worker process – Each Worker process handles a single request at a time
  83. 83. Copyright © 2017 M/Gateway Developments Ltd QEWD's Architecture • The queue / dispatcher / worker-process pool management part of QEWD is handled by a module named ewd-qoper8
  84. 84. Copyright © 2017 M/Gateway Developments Ltd Master Node.js Process Queue Queue processor/ dispatcher Incoming Requests QEWD Architecture Every incoming request is passed from Express and placed in a queue No further processing of requests occurs in the master process Express or Koa.js socket.io HTTP REST WebSocket
  85. 85. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Queue dispatcher is invoked whenever a request is added to the queue
  86. 86. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Worker process started if none available
  87. 87. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Redis/Cache QEWD & application-specific Modules loaded Custom Worker Module Custom Worker Module
  88. 88. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Node.js Worker Process Custom Worker Module Request passed to worker Redis/Cache
  89. 89. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Worker flagged as Unavailable Node.js Worker Process Custom Worker Module Begin processing message Redis/Cache
  90. 90. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing Another incoming request Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache
  91. 91. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module If worker pool size not exceeded, another worker is started and request passed to it Redis/Cache Redis/Cache
  92. 92. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache If entire Worker Pool is busy: Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache
  93. 93. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher If entire Worker Pool is busy: New requests remain in queue Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache
  94. 94. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache As soon as a worker is available again, a queued message can be passed to it Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache Available Node.js Worker Process Custom Worker Module Redis/Cache
  95. 95. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Finished Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Redis/Cache A user's handler function signals completion using the function: finished(responseObject); This returns the response object to the master process
  96. 96. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Finished Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module And the response is returned to the client that sent the original request (via Express/Koa/socket.io) Redis/Cache
  97. 97. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Available Node.js Worker Process Custom Worker Module The finished() function also automatically returns the worker process back to the available pool So it can now handle the next queued request Redis/Cache
  98. 98. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Available Node.js Worker Process Custom Worker Module Worker processes, once started, are persistent No start-up / tear-down cost Redis/Cache
  99. 99. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Node.js concurrency is handled by the master process. 100% asynchronous logic The master process does almost nothing The perfect Node.js application: no CPU-intensive or long- running tasks, so very high-performance Multiple Concurrent Incoming requests
  100. 100. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Node.js concurrency is handled by the master process. 100% asynchronous logic The master process does almost nothing The perfect Node.js application: no CPU-intensive or long- running tasks, so very high-performance All the actual work happens in the isolated worker processes Multiple Concurrent Incoming requests
  101. 101. Copyright © 2017 M/Gateway Developments Ltd QEWD Architecture Master Node.js Process Queue Queue processor/ dispatcher Available Node.js Worker Process Custom Worker Module Worker processes only handle a single request at a time Completely isolated run-time environment for handler functions No need for concerns about Node.js concurrency, so synchronous APIs can be used Redis/Cache
  102. 102. Copyright © 2017 M/Gateway Developments Ltd QEWD + ewd-document-store + cache.node • JavaScript as a first-class language for Caché • Provides the JavaScript equivalent of ^ – Local JavaScript objects (in-memory) – Persistent JavaScript objects (on-disk) • JavaScript with data persistence as a language feature
  103. 103. Copyright © 2017 M/Gateway Developments Ltd Just one remaining issue • Native Caché ObjectScript performance: – Set: >1 million Global Nodes / sec • cache.node – Around 8 – 10% of that performance
  104. 104. Copyright © 2017 M/Gateway Developments Ltd The reason? • The Google V8 Engine's API that is used by cache.node – Has a bottleneck • Usually thought to be insignificant • Becomes significant for the kinds of things done by ewd-document-store
  105. 105. Copyright © 2017 M/Gateway Developments Ltd If it can be fixed • We'd have the fastest Node.js database on the planet – By a significant margin • We'd have data persistence as a feature of JavaScript – Until it's fully demonstrable, the IT mainstream aren't going to "get it"
  106. 106. Copyright © 2017 M/Gateway Developments Ltd Can It Be Fixed? • Probably requires Google to sort it out • Logged on their bug tracker – But no sign of them doing anything to fix it • Watch this space • If it can be fixed, this is potentially huge – And we all stand to benefit
  107. 107. Copyright © 2017 M/Gateway Developments Ltd Additional Reading • Universal NoSQL Paper: – http://mgateway.com/docs/universalNoSQL.pdf • QEWD.js: – https://qewdjs.com • JavaScript Abstraction of Global Storage: – http://docs.qewdjs.com/qewd_training.html • In particular, parts 17 – 27 • QEWD.js from a Node.js perspective: – http://bit.ly/2gnnyh5 – http://bit.ly/2ikY1sS
  108. 108. Copyright © 2017 M/Gateway Developments Ltd ^ redux: Data Persistence as a Language Feature Rob Tweed M/Gateway Developments Ltd Twitter: @rtweed http://qewdjs.com

×