Copyright © 2017 M/Gateway Developments Ltd
^ redux:
Data Persistence as a
Language Feature
Rob Tweed
M/Gateway Developments Ltd
Twitter: @rtweed
http://qewdjs.com

Copyright © 2017 M/Gateway Developments Ltd
A bit of background
• Royal Marsden Hospital: 1980s
• Touche Ross Management Consultants
(now Deloitte): early 1990s
– NHS-wide Networking Project
• Independent consultant & software
developer since 1993
– Specialising in Web and associated
technologies, particularly in healthcare

Copyright © 2017 M/Gateway Developments Ltd
Products
• WebLink:
– Co-creator of the original web connectivity for Caché
• WebLink Developer:
– Web Application Development framework
• pre-cursor to CSP
• eXtc:
– XML DOM framework
• EWD:
– 2nd
-generation web application development framework
– XML based custom pages
– Compiled to CSP or WebLink (or even PHP)

Copyright © 2017 M/Gateway Developments Ltd
Example Usage

Copyright © 2017 M/Gateway Developments Ltd
A bit of background
• Node.js:
– Since August 2010 (v0.2 had just come out!)
– Pioneered the integration of Caché and
Node.js
– Was instrumental in getting the cache.node
interface developed
• Originally for GlobalsDB

Copyright © 2017 M/Gateway Developments Ltd
Language or Database?
• Caché ObjectScript:
– programming language with integrated
database?
• Caché Database:
– Database with integrated scripting language
for its manipulation (Caché ObjectScript)?

Copyright © 2017 M/Gateway Developments Ltd
Language
• Caché ObjectScript shows its age
– Rightly or wrongly, opinion from IT
Mainstream developers is not favourable
– Tainted, in particular, by this:
• https://thedailywtf.com/articles/comments/A_Case_of_the_MUMPS/8
– As a result of the association with Mumps by,
eg:
– https://en.wikipedia.org/wiki/Cach%C3%A9_ObjectScript

Copyright © 2017 M/Gateway Developments Ltd
Language
• Caché ObjectScript typical objections:
– Limited
– Old-Fashioned
– To a modern developer, it's:
• A step back in time
• Lacking in many of the language capabilities they
expect
• Lacks an extensive ecosystem of pre-built
functions and modules
• A career-limiting move

Copyright © 2017 M/Gateway Developments Ltd
Database
• Uniquely powerful
• The original NoSQL database
– Pre-dated the NoSQL movement by several
decades
• "Universal NoSQL"
– AKA "Multi-Model database"

Copyright © 2017 M/Gateway Developments Ltd
Universal NoSQL
Published in 2010

Copyright © 2017 M/Gateway Developments Ltd
Database
• But in order to manipulate and use that
database to its fullest capabilities, you
need to learn its scripting language
– Caché ObjectScript
• A key reason why, as a database, it's
almost unknown in the mainstream of IT
– I've yet to meet a JavaScript developer who has even
heard of Caché

Copyright © 2017 M/Gateway Developments Ltd
Solution?
• Recurring suggestions within the
developer community that Caché
ObjectScript needs modernising
– New language capabilities
– Bring its syntax into line with modern
languages
– Create an ecosystem of functions & modules

Copyright © 2017 M/Gateway Developments Ltd
My Opinion
• This is the wrong approach
• Nobody in the mainstream of IT wants to
learn a new language just to access a
database
• They'd expect to be able to use a modern
language

Copyright © 2017 M/Gateway Developments Ltd
Language or Database?
• Is Caché first and foremost a database or
a language?
– Misses a unique feature of Caché which
provides its key power and which:
• We take for granted
• The mainstream of IT don't "get"

Copyright © 2017 M/Gateway Developments Ltd
What is a database?
• It's that thing "over there" that persists our
data

Copyright © 2017 M/Gateway Developments Ltd
What is a database?
• It's expected to be "over there"
• It's a separate server / sub-system
• It's accessed:
– over a network connection
– via some protocol and/or syntax
• REST
• SQL

Copyright © 2017 M/Gateway Developments Ltd
So what?
• Creates an explicit distinction between
stuff you do in your programming
language and persisted data
• Creates separate disciplines
– Programming experts
– Database experts
– Many people span both, but, to many
programmers, databases are a mystery

Copyright © 2017 M/Gateway Developments Ltd
The Unique Feature
• The underlying "Global Storage" of Caché
• Why?
– It blurs the distinction between what's permanently
persisted and what's in memory
– In memory: sparse arrays
– On disk: identical sparse arrays
– Only difference in syntax terms is a preceeding ^

Copyright © 2017 M/Gateway Developments Ltd
Revise the Definition
• Yes it's a database and a language, but it
warrants a new description:
• It's a language with data persistence:
– as a seamless language feature

Copyright © 2017 M/Gateway Developments Ltd
The Problem
• It's a language with data persistence:
– as a seamless language feature
• But the language is obsolete and a barrier
to new entry and mainstream adoption

Copyright © 2017 M/Gateway Developments Ltd
The Solution
• It's a language with data persistence:
– as a seamless language feature
• Add a modern language as a first-class
alternative to Caché ObjectScript
– With data persistence as a similarly seamless
feature
– An equivalent, in that language, to ^

Copyright © 2017 M/Gateway Developments Ltd
The Question
• What Language would fit the bill?

Copyright © 2017 M/Gateway Developments Ltd
From the Mainstream Perspective
• One of the currently "big" languages
– C / C++ too low level
– Java
– Ruby
– Go
– Python
– JavaScript

Copyright © 2017 M/Gateway Developments Ltd
From our Perspective
• Must be as fast as Native Caché
ObjectScript
– For general-purpose coding
– For database access
• > 1 million Global Nodes per second

Copyright © 2017 M/Gateway Developments Ltd
From both Perspectives
• Should be a language suitable for all the
kinds of application that Caché
ObjectScript is currently used for:
– Desktop
– Browser-based
– Mobile
– Batch processing / server-side heavy-lifting

Copyright © 2017 M/Gateway Developments Ltd
From both Perspectives
• Should map seamlessly between Global
Storage and a commonly-accepted native
in-memory representation with that
language
– No "impedance mismatch"
• That mapping should make sense within
the context of that language
– ie laguage-centric

Copyright © 2017 M/Gateway Developments Ltd
Global Storage Key Features
• Schema-Free
• Hierarchically-structured
• Dynamic creation / destruction
• Ideally suited for "late-bound" access

Copyright © 2017 M/Gateway Developments Ltd
Suitability: Java
Feature
Adoption Widespread
General-purpose Code Speed Excellent
Application suitability Not used in browser, but otherwise
widespread acceptance and adoption
Data persistence impedance
mismatch
Designed around pre-defined,
compiled classes / schema, ie early-
bound access

Copyright © 2017 M/Gateway Developments Ltd
Suitability: Ruby
Feature
Adoption Fairly limited by comparison to other
languages
Well-liked by its users
General-purpose Code Speed Poor
Application suitability Not used in browser. Poor choice for
mobile development
Data persistence impedance
mismatch
Good

Copyright © 2017 M/Gateway Developments Ltd
Suitability: Go
Feature
Adoption Fairly limited by comparison to other
languages. Well-liked by its users
General-purpose Code Speed Excellent
Application suitability Not used in browser. Otherwise very
good
Data persistence impedance
mismatch
Not so good – designed for pre-
defined, pre-compiled mappings
Miscellaneous Can we depend on Google
supporting it in the long term?

Copyright © 2017 M/Gateway Developments Ltd
Suitability: Python
Feature
Adoption Pretty high and growing. Well-liked
by its users
General-purpose Code Speed Poor
Application suitability Not used in browser. Not suitable for
mobile development
Data persistence impedance
mismatch
Good – dictionaries / JSON map well

Copyright © 2017 M/Gateway Developments Ltd
Suitability: JavaScript
Feature
Adoption High and still growing. Node.js now
massively popular. Language has
some quirks!
General-purpose Code Speed Node.js performance is excellent
Application suitability Used in browser. Ideal for mobile
development. Usual cited deficiency
is CPU-heavy processing, but that
can be addressed
Data persistence impedance
mismatch
JavaScript objects are dynamic and
map almost perfectly onto Global
Storage
Miscellaneous Only 1 language skill for entire
development stack, front and back
(Node.js at back-end)

Copyright © 2017 M/Gateway Developments Ltd
JavaScript as a Native
Language for Caché
• It's a very good fit, for all sorts of reasons:
– Interpreted
– Loosely-typed in a similar way
– JavaScript Objects (JSON)
• Schema-free
• Dynamic
• hierarchical

Copyright © 2017 M/Gateway Developments Ltd
Global Storage
myGlobal("a")=123
myGlobal("b","c1")="foo"
myGlobal("b","c2")="foo2"
myGlobal("d","e1","f1")="bar1"
myGlobal("d","e1","f2")="bar2"
myGlobal("d","e2","f1")="bar1"
myGlobal("d","e2","f2")="bar2"
myGlobal("d","e2","f3")="bar3"
myGlobal
"a" 123
"b"
"c2" "foo2"
"d"
"c1" "foo"
"e2"
"e1"
"f2" "bar2"
"f1" "bar1"
"f2" "bar2"
"f1" "bar1"
"f3" "bar3"
Hierarchical diagram representing a Global

Copyright © 2017 M/Gateway Developments Ltd
myGlobal = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
}
d: {
e1: {
f1: 'bar1',
f2: 'bar2'
},
e2: {
f1: 'bar1',
f2: 'bar2',
f3: 'bar3'
}
}
}
myGlobal("a")=123
myGlobal("b","c1")="foo"
myGlobal("b","c2")="foo2"
myGlobal("d","e1","f1")="bar1"
myGlobal("d","e1","f2")="bar2"
myGlobal("d","e2","f1")="bar1"
myGlobal("d","e2","f2")="bar2"
myGlobal("d","e2","f3")="bar3"
One to one correspondence between any tree of Global nodes
and a Javascript object

Copyright © 2017 M/Gateway Developments Ltd
myGlobal = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
}
d: {
e1: {
f1: 'bar1',
f2: 'bar2'
},
e2: {
f1: 'bar1',
f2: 'bar2',
f3: 'bar3'
}
}
}
myGlobal("a")=123
myGlobal("b","c1")="foo"
myGlobal("b","c2")="foo2"
myGlobal("d","e1","f1")="bar1"
myGlobal("d","e1","f2")="bar2"
myGlobal("d","e2","f1")="bar1"
myGlobal("d","e2","f2")="bar2"
myGlobal("d","e2","f3")="bar3"
Subscript <=> Property

Copyright © 2017 M/Gateway Developments Ltd
myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3
Global Node JavaScript Object

Copyright © 2017 M/Gateway Developments Ltd
myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3
Global Node JavaScript Object
But…
JavaScript Objects are in memory
Global Nodes are persistent, on disk

Copyright © 2017 M/Gateway Developments Ltd
myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3
Global Node JavaScript Object
Could we abstract Global Nodes as
Persistent JavaScript Objects?

Copyright © 2017 M/Gateway Developments Ltd
ewd-document-store
• JavaScript abstraction of Global Storage
– https://github.com/robtweed/ewd-document-store

Copyright © 2017 M/Gateway Developments Ltd
ewd-document-store
• Abstracts Global Storage as:
– A document database
– Persistent JavaScript Objects

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode(name, subscripts);
Represents a node within a Global (aka Document)
May or may not physically exist at this point
ie this does NOT create a node
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode(name, subscripts);
Represents a node within a document's hierarchy
May or may not physically exist at this point
ie this does NOT create a node
Creates an instance of an object that represents that node, with
a set of methods and properties to manipulate and access that node
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
create: dnode.value = 'foo2';
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
create: dnode.value = 'foo2';
get: var x = dnode.value;
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
create: dnode.value = 'foo2';
get: var x = dnode.value;
value is a read/write property
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
create: dnode.value = 'foo2';
get: var x = dnode.value;
delete: dnode.delete();
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
create: dnode.value = 'foo2';
get: var x = dnode.value;
delete: dnode.delete();
Note: delete will also delete any lower-level DocumentNodes
in the underlying Global Storage tree
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
create: dnode.value = 'foo2';
get: var x = dnode.value;
delete: dnode.delete();
exists?: var exists = dnode.exists; // true | false
DocumentNode Object

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d']);
dnode.hasChildren // true
dnode.hasValue // false
dnode.exists // true
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1")="bar1"
myDoc("d","e1","f2")="bar2"
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2', 'f3']);
dnode.hasChildren // false
dnode.hasValue // true
dnode.exists // true
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1")="bar1"
myDoc("d","e1","f2")="bar2"
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"

Copyright © 2017 M/Gateway Developments Ltd
var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2', 'f4']);
dnode.hasChildren // false
dnode.hasValue // false
dnode.exists // false
Even though the node doesn’t physically exist, you can define a
DocumentNode Object for it
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1")="bar1"
myDoc("d","e1","f2")="bar2"
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"

Copyright © 2017 M/Gateway Developments Ltd
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"
var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
Traversing Documents

Copyright © 2017 M/Gateway Developments Ltd
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"
var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
e2Node.forEachChild(function(nodeName) {
// do something with the node name / subscript for this iteration
});
forEachChild() Method

Copyright © 2017 M/Gateway Developments Ltd
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"
var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
e2Node.forEachChild(function(nodeName) {
// 1st
iteration nodeName === 'f1’
});
forEachChild() Method

Copyright © 2017 M/Gateway Developments Ltd
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"
var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
e2Node.forEachChild(function(nodeName) {
// 2nd
iteration nodeName === 'f2’
});
forEachChild() Method

Copyright © 2017 M/Gateway Developments Ltd
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"
var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
e2Node.forEachChild(function(nodeName) {
// 3rd
iteration nodeName === 'f3’
});
forEachChild() Method

Copyright © 2017 M/Gateway Developments Ltd
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")="bar3"
var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
e2Node.forEachChild(function(nodeName) {
console.log(nodeName);
});
forEachChild() Method
f1
f2
f3

Copyright © 2017 M/Gateway Developments Ltd
getDocument()
var doc = new this.documentStore.DocumentNode('myDoc');
var myObj = doc.getDocument();
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1a")="bar1a"
myDoc("d","e1","f2a")="bar2a"
myDoc("d","e2","f1b")="bar1b"
myDoc("d","e2","f2b")="bar2b"
myDoc("d","e2","f3b")="bar3b"
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
creates

Copyright © 2017 M/Gateway Developments Ltd
getDocument()
var doc = new this.documentStore.DocumentNode('myDoc');
var myObj = doc.getDocument();
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1a")="bar1a"
myDoc("d","e1","f2a")="bar2a"
myDoc("d","e2","f1b")="bar1b"
myDoc("d","e2","f2b")="bar2b"
myDoc("d","e2","f3b")="bar3b"
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
myObj is a standard in-memory JavaScript object
containing a copy of the data that was held within
the DocumentNode's sub-tree of nodes
creates

Copyright © 2017 M/Gateway Developments Ltd
setDocument()
var myObj = { // create the object shown to the right };
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}

Copyright © 2017 M/Gateway Developments Ltd
setDocument()
var myObj = { // create the object shown to the right };
var doc = new this.documentStore.DocumentNode('myDoc');
// doesn't yet exist on disk myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}

Copyright © 2017 M/Gateway Developments Ltd
setDocument()
var myObj = { // create the object shown to the right };
var doc = new this.documentStore.DocumentNode('myDoc');
doc.setDocument(myObj);
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1a")="bar1a"
myDoc("d","e1","f2a")="bar2a"
myDoc("d","e2","f1b")="bar1b"
myDoc("d","e2","f2b")="bar2b"
myDoc("d","e2","f3b")="bar3b"
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
creates

Copyright © 2017 M/Gateway Developments Ltd
setDocument()
var myObj = { // create the object shown to the right };
var doc = new this.documentStore.DocumentNode('myDoc');
doc.setDocument(myObj);
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1a")="bar1a"
myDoc("d","e1","f2a")="bar2a"
myDoc("d","e2","f1b")="bar1b"
myDoc("d","e2","f2b")="bar2b"
myDoc("d","e2","f3b")="bar3b"
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
myDoc is created on disk, containing a copy of
the data that was held within the myObj
object, now mapped to Global Storage nodes
creates

Copyright © 2017 M/Gateway Developments Ltd
Making This Possible
• The cache.node interface
• Included with Caché (and IRIS)
• Runs in-process with Caché
• Connects to Caché's C++ Callin Interface
– Directly accessing the Caché database engine
– Bypassing the language interface

Copyright © 2017 M/Gateway Developments Ltd
cache.node interface
Node.js
Process
Caché
Process
cache.node
module
C++call-ininterface
JavaScript
Globals Functions Objects

Copyright © 2017 M/Gateway Developments Ltd
cache.node APIs
• Low-level
• Basic Global primitives
• "Global-centric"
• Not JavaScript-centric

Copyright © 2017 M/Gateway Developments Ltd
ewd-document-store
• Builds on top of the cache.node APIs
• Creates a JavaScript-centric abstraction of
Global Storage
– Mapping to/from JavaScript Objects
• The JavaScript equivalent of ^
– Local JavaScript objects (in-memory)
– Persistent JavaScript objects (on-disk)

Copyright © 2017 M/Gateway Developments Ltd
Node.js Architecture
• Everything executes in a single process
– You might have 1000's of users doing stuff
concurrently
• They're all sharing the same process!

Copyright © 2017 M/Gateway Developments Ltd
Brainchild of Ryan Dahl

Copyright © 2017 M/Gateway Developments Ltd
Node.js = server-side JavaScript
• Ryan Dahl wasn't actually a big JavaScript fan
– Node.js wasn't designed to be server-side JavaScript
• But he realised that it was a convenient
language for such an environment, as it's
designed for exactly the same kind of way of
working in the browser
• Google's V8 JavaScript engine was Open
Sourced, so he used it to provide the language
for Node.js
– (which is actually mostly written in C++)

Copyright © 2017 M/Gateway Developments Ltd
Node.js Architecture
• Non-blocking I/O
– No user activity must block the process, or everyone
grinds to a halt
• Event-driven, asynchronous logic
– Fire off the request, but don't wait for the results
• Carry on with the next task
– When results come back, handle them at the next
opportunity

Copyright © 2017 M/Gateway Developments Ltd
ewd-document-store requires
synchronous APIs
var myDoc = new this.documentStore.DocumentNode('myDoc');
var f3Node = myDoc.$('d').$('e2').$('f3');
myDoc.$d.$e2.$f3.value = 'New value';
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1")="bar1"
myDoc("d","e1","f2")="bar2"
myDoc("d","e2","f1")="bar1"
myDoc("d","e2","f2")="bar2"
myDoc("d","e2","f3")=”New value"
Requires properly chainable functions

Copyright © 2017 M/Gateway Developments Ltd
The Problem is Concurrency
• All concurrent users in Node.js share the
same process

Copyright © 2017 M/Gateway Developments Ltd
Is there a way to avoid this?
• Would it be possible to create a locally-
available environment where my Node.js
code runs in an isolated container where
concurrency isn't an issue?

Copyright © 2017 M/Gateway Developments Ltd

Copyright © 2017 M/Gateway Developments Ltd
What Is QEWD?
• Essentially it's a multi-purpose Node.js-
based run-time Platform

Copyright © 2017 M/Gateway Developments Ltd
What Is QEWD?
• Essentially it's a multi-purpose Node.js-
based run-time Platform
• Creates an isolated run-time container for
your message/request handler functions,
allowing:
– CPU-intensive work
– Database abstractions using synchronous
logic

Copyright © 2017 M/Gateway Developments Ltd
QEWD's Architecture
• Master Process
• Pool of Worker Processes

Copyright © 2017 M/Gateway Developments Ltd
QEWD's Architecture
• Master Process
– Handles and queues all incoming requests
from client
• HTTP/REST requests via Express or Koa.js
• WebSocket requests via socket.io
– Returns responses to client

Copyright © 2017 M/Gateway Developments Ltd
QEWD's Architecture
• Pool of Persistent Worker Processes
– Where all the processing occurs
– A single queued request is dispatched to an
available worker process
– Each Worker process handles a single
request at a time

Copyright © 2017 M/Gateway Developments Ltd
QEWD's Architecture
• The queue / dispatcher / worker-process
pool management part of QEWD is
handled by a module named ewd-qoper8

Copyright © 2017 M/Gateway Developments Ltd
Master Node.js Process
Queue
Queue
processor/
dispatcher
Incoming
Requests
QEWD Architecture
Every incoming request
is passed from Express
and placed in a queue
No further processing
of requests occurs in
the master process
Express
or
Koa.js
socket.io
HTTP
REST
WebSocket

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Queue dispatcher is
invoked whenever a
request is added to
the queue

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor/
dispatcher
Worker process
started if none
available

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor/
dispatcher
Redis/Cache
QEWD &
application-specific
Modules loaded
Custom
Worker
Module
Custom
Worker
Module

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Request passed
to worker
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Worker flagged as Unavailable
Node.js Worker Process
Custom
Worker
Module
Begin processing message
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailable / processing
Another incoming
request Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
If worker pool size not exceeded,
another worker is started
and request passed to it
Redis/Cache
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
If entire Worker Pool is busy:
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
If entire Worker Pool is busy:
New
requests
remain
in queue
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
As soon as a worker is available again,
a queued message can be passed to it
Unavailable / processing
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
Available
Node.js Worker Process
Custom
Worker
Module
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Finished
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Redis/Cache
A user's handler function signals
completion using the function:
finished(responseObject);
This returns the response
object to the master
process

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Finished
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
Node.js Worker Process
Custom
Worker
Module
And the response is
returned to the client
that sent the original request
(via Express/Koa/socket.io)
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Available
Node.js Worker Process
Custom
Worker
Module
The finished() function
also automatically returns
the worker process back
to the available pool
So it can now handle
the next queued request
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Available
Node.js Worker Process
Custom
Worker
Module
Worker processes, once started,
are persistent
No start-up / tear-down cost
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Node.js concurrency is handled
by the master process.
100% asynchronous logic
The master process does
almost nothing
The perfect Node.js application:
no CPU-intensive or long-
running tasks, so very
high-performance
Multiple
Concurrent
Incoming
requests

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Node.js concurrency is handled
by the master process.
100% asynchronous logic
The master process does
almost nothing
The perfect Node.js application:
no CPU-intensive or long-
running tasks, so very
high-performance
All the actual work happens in the
isolated worker processes
Multiple
Concurrent
Incoming
requests

Copyright © 2017 M/Gateway Developments Ltd
QEWD Architecture
Master Node.js Process
Queue
Queue
processor/
dispatcher
Available
Node.js Worker Process
Custom
Worker
Module
Worker processes only handle
a single request at a time
Completely isolated run-time
environment for handler functions
No need for concerns about
Node.js concurrency, so
synchronous APIs can be used
Redis/Cache

Copyright © 2017 M/Gateway Developments Ltd
QEWD + ewd-document-store +
cache.node
• JavaScript as a first-class language for
Caché
• Provides the JavaScript equivalent of ^
– Local JavaScript objects (in-memory)
– Persistent JavaScript objects (on-disk)
• JavaScript with data persistence as a
language feature

Copyright © 2017 M/Gateway Developments Ltd
Just one remaining issue
• Native Caché ObjectScript performance:
– Set: >1 million Global Nodes / sec
• cache.node
– Around 8 – 10% of that performance

Copyright © 2017 M/Gateway Developments Ltd
The reason?
• The Google V8 Engine's API that is used
by cache.node
– Has a bottleneck
• Usually thought to be insignificant
• Becomes significant for the kinds of things done by
ewd-document-store

Copyright © 2017 M/Gateway Developments Ltd
If it can be fixed
• We'd have the fastest Node.js database
on the planet
– By a significant margin
• We'd have data persistence as a feature
of JavaScript
– Until it's fully demonstrable, the IT mainstream
aren't going to "get it"

Copyright © 2017 M/Gateway Developments Ltd
Can It Be Fixed?
• Probably requires Google to sort it out
• Logged on their bug tracker
– But no sign of them doing anything to fix it
• Watch this space
• If it can be fixed, this is potentially huge
– And we all stand to benefit

Copyright © 2017 M/Gateway Developments Ltd
Additional Reading
• Universal NoSQL Paper:
– http://mgateway.com/docs/universalNoSQL.pdf
• QEWD.js:
– https://qewdjs.com
• JavaScript Abstraction of Global Storage:
– http://docs.qewdjs.com/qewd_training.html
• In particular, parts 17 – 27
• QEWD.js from a Node.js perspective:
– http://bit.ly/2gnnyh5
– http://bit.ly/2ikY1sS

Copyright © 2017 M/Gateway Developments Ltd
^ redux:
Data Persistence as a
Language Feature
Rob Tweed
M/Gateway Developments Ltd
Twitter: @rtweed
http://qewdjs.com