Data Persistence as a Language Feature

Copyright © 2017 M/Gateway Developments Ltd
^ redux:
Data Persistence as a
Language Feature
Rob Tweed
M/Gateway Developments Ltd
Twitter: @rtweed
http://qewdjs.com

A bit of background
• Royal Marsden Hospital: 1980s
• Touche Ross Management Consultants
(now Deloitte): early 1990s
– NHS-wide Networking Project
• Independent consultant & software
developer since 1993
– Specialising in Web and associated
technologies, particularly in healthcare

Products
• WebLink:
– Co-creator of the original web connectivity for Caché
• WebLink Developer:
– Web Application Development framework
• pre-cursor to CSP
• eXtc:
– XML DOM framework
• EWD:
– 2nd
-generation web application development framework
– XML based custom pages
– Compiled to CSP or WebLink (or even PHP)

Example Usage

A bit of background
• Node.js:
– Since August 2010 (v0.2 had just come out!)
– Pioneered the integration of Caché and
Node.js
– Was instrumental in getting the cache.node
interface developed
• Originally for GlobalsDB

Language or Database?
• Caché ObjectScript:
– programming language with integrated
database?
• Caché Database:
– Database with integrated scripting language
for its manipulation (Caché ObjectScript)?

Language
• Caché ObjectScript shows its age
– Rightly or wrongly, opinion from IT
Mainstream developers is not favourable
– Tainted, in particular, by this:
• https://thedailywtf.com/articles/comments/A_Case_of_the_MUMPS/8
– As a result of the association with Mumps by,
eg:
– https://en.wikipedia.org/wiki/Cach%C3%A9_ObjectScript

Language
• Caché ObjectScript typical objections:
– Limited
– Old-Fashioned
– To a modern developer, it's:
• A step back in time
• Lacking in many of the language capabilities they
expect
• Lacks an extensive ecosystem of pre-built
functions and modules
• A career-limiting move

Database
• Uniquely powerful
• The original NoSQL database
– Pre-dated the NoSQL movement by several
decades
• "Universal NoSQL"
– AKA "Multi-Model database"

Universal NoSQL
Published in 2010

Database
• But in order to manipulate and use that
database to its fullest capabilities, you
need to learn its scripting language
– Caché ObjectScript
• A key reason why, as a database, it's
almost unknown in the mainstream of IT
– I've yet to meet a JavaScript developer who has even
heard of Caché

Solution?
• Recurring suggestions within the
developer community that Caché
ObjectScript needs modernising
– New language capabilities
– Bring its syntax into line with modern
languages
– Create an ecosystem of functions & modules

My Opinion
• This is the wrong approach
• Nobody in the mainstream of IT wants to
learn a new language just to access a
database
• They'd expect to be able to use a modern
language

Language or Database?
• Is Caché first and foremost a database or
a language?
– Misses a unique feature of Caché which
provides its key power and which:
• We take for granted
• The mainstream of IT don't "get"

What is a database?
• It's that thing "over there" that persists our
data

What is a database?
• It's expected to be "over there"
• It's a separate server / sub-system
• It's accessed:
– over a network connection
– via some protocol and/or syntax
• REST
• SQL

So what?
• Creates an explicit distinction between
stuff you do in your programming
language and persisted data
• Creates separate disciplines
– Programming experts
– Database experts
– Many people span both, but, to many
programmers, databases are a mystery

The Unique Feature
• The underlying "Global Storage" of Caché
• Why?
– It blurs the distinction between what's permanently
persisted and what's in memory
– In memory: sparse arrays
– On disk: identical sparse arrays
– Only difference in syntax terms is a preceeding ^

Revise the Definition
• Yes it's a database and a language, but it
warrants a new description:
• It's a language with data persistence:
– as a seamless language feature

The Problem
• But the language is obsolete and a barrier
to new entry and mainstream adoption

The Solution
• Add a modern language as a first-class
alternative to Caché ObjectScript
– With data persistence as a similarly seamless
feature
– An equivalent, in that language, to ^

The Question
• What Language would fit the bill?

From the Mainstream Perspective
• One of the currently "big" languages
– C / C++ too low level
– Java
– Ruby
– Go
– Python
– JavaScript

From our Perspective
• Must be as fast as Native Caché
ObjectScript
– For general-purpose coding
– For database access
• > 1 million Global Nodes per second

From both Perspectives
• Should be a language suitable for all the
kinds of application that Caché
ObjectScript is currently used for:
– Desktop
– Browser-based
– Mobile
– Batch processing / server-side heavy-lifting

From both Perspectives
• Should map seamlessly between Global
Storage and a commonly-accepted native
in-memory representation with that
language
– No "impedance mismatch"
• That mapping should make sense within
the context of that language
– ie laguage-centric

Global Storage Key Features
• Schema-Free
• Hierarchically-structured
• Dynamic creation / destruction
• Ideally suited for "late-bound" access

Suitability: Java
Feature
Adoption Widespread
General-purpose Code Speed Excellent
Application suitability Not used in browser, but otherwise
widespread acceptance and adoption
Data persistence impedance
mismatch
Designed around pre-defined,
compiled classes / schema, ie early-
bound access

Suitability: Ruby
Feature
Adoption Fairly limited by comparison to other
languages
Well-liked by its users
General-purpose Code Speed Poor
Application suitability Not used in browser. Poor choice for
mobile development
mismatch
Good

Suitability: Go
Feature
Adoption Fairly limited by comparison to other
languages. Well-liked by its users
General-purpose Code Speed Excellent
Application suitability Not used in browser. Otherwise very
good
mismatch
Not so good – designed for pre-
defined, pre-compiled mappings
Miscellaneous Can we depend on Google
supporting it in the long term?

Suitability: Python
Feature
Adoption Pretty high and growing. Well-liked
by its users
General-purpose Code Speed Poor
Application suitability Not used in browser. Not suitable for
mobile development
mismatch
Good – dictionaries / JSON map well

Suitability: JavaScript
Feature
Adoption High and still growing. Node.js now
massively popular. Language has
some quirks!
General-purpose Code Speed Node.js performance is excellent
Application suitability Used in browser. Ideal for mobile
development. Usual cited deficiency
is CPU-heavy processing, but that
can be addressed
mismatch
JavaScript objects are dynamic and
map almost perfectly onto Global
Storage
Miscellaneous Only 1 language skill for entire
development stack, front and back
(Node.js at back-end)

JavaScript as a Native
Language for Caché
• It's a very good fit, for all sorts of reasons:
– Interpreted
– Loosely-typed in a similar way
– JavaScript Objects (JSON)
• Schema-free
• Dynamic
• hierarchical

Global Storage
myGlobal("a")=123
myGlobal("b","c1")="foo"
myGlobal("b","c2")="foo2"
myGlobal("d","e1","f1")="bar1"
myGlobal
"a" 123
"b"
"c2" "foo2"
"d"
"c1" "foo"
"e2"
"e1"
"f2" "bar2"
"f1" "bar1"
"f2" "bar2"
"f1" "bar1"
"f3" "bar3"
Hierarchical diagram representing a Global

myGlobal = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
}
d: {
e1: {
f1: 'bar1',
f2: 'bar2'
},
e2: {
f1: 'bar1',
f2: 'bar2',
f3: 'bar3'
}
}
}
myGlobal("a")=123
One to one correspondence between any tree of Global nodes
and a Javascript object

myGlobal = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
}
d: {
e1: {
f1: 'bar1',
f2: 'bar2'
},
e2: {
f1: 'bar1',
f2: 'bar2',
f3: 'bar3'
}
}
}
myGlobal("a")=123
Subscript <=> Property

myGlobal("d","e2","f3") <=> myGlobal.d.e2.f3
Global Node JavaScript Object

But…
JavaScript Objects are in memory
Global Nodes are persistent, on disk

Could we abstract Global Nodes as
Persistent JavaScript Objects?

ewd-document-store
• JavaScript abstraction of Global Storage
– https://github.com/robtweed/ewd-document-store

ewd-document-store
• Abstracts Global Storage as:
– A document database
– Persistent JavaScript Objects

var dnode = new this.documentStore.DocumentNode(name, subscripts);
Represents a node within a Global (aka Document)
May or may not physically exist at this point
ie this does NOT create a node
DocumentNode Object

var dnode = new this.documentStore.DocumentNode(name, subscripts);
Represents a node within a document's hierarchy
May or may not physically exist at this point
ie this does NOT create a node
Creates an instance of an object that represents that node, with
a set of methods and properties to manipulate and access that node
DocumentNode Object

var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2']);
represents myDoc("d","e2")
DocumentNode Object

create: dnode.value = 'foo2';
DocumentNode Object

get: var x = dnode.value;
DocumentNode Object

value is a read/write property
DocumentNode Object

delete: dnode.delete();
DocumentNode Object

Note: delete will also delete any lower-level DocumentNodes
in the underlying Global Storage tree
DocumentNode Object

exists?: var exists = dnode.exists; // true | false
DocumentNode Object

var dnode = new this.documentStore.DocumentNode('myDoc', ['d']);
dnode.hasChildren // true
dnode.hasValue // false
dnode.exists // true
myDoc("a")=123
myDoc("b","c1")="foo"
myDoc("b","c2")="foo2"
myDoc("d","e1","f1")="bar1"

var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2', 'f3']);
dnode.hasChildren // false
dnode.hasValue // true
dnode.exists // true
myDoc("a")=123

var dnode = new this.documentStore.DocumentNode('myDoc', ['d', 'e2', 'f4']);
dnode.hasChildren // false
dnode.hasValue // false
dnode.exists // false
Even though the node doesn’t physically exist, you can define a
DocumentNode Object for it
myDoc("a")=123

var e2Node = new this.documentStore.DocumentNode('myDoc', ['d', e2']);
Traversing Documents

e2Node.forEachChild(function(nodeName) {
// do something with the node name / subscript for this iteration
});
forEachChild() Method

// 1st
iteration nodeName === 'f1’
});

// 2nd
});

// 3rd
});

console.log(nodeName);
});
f1
f2
f3

getDocument()
var doc = new this.documentStore.DocumentNode('myDoc');
var myObj = doc.getDocument();
myDoc("a")=123
myDoc("d","e1","f1a")="bar1a"
myDoc("d","e2","f1b")="bar1b"
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
creates

getDocument()
var myObj = doc.getDocument();
myDoc("a")=123
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
myObj is a standard in-memory JavaScript object
containing a copy of the data that was held within
the DocumentNode's sub-tree of nodes
creates

setDocument()
var myObj = { // create the object shown to the right };
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}

setDocument()
// doesn't yet exist on disk myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}

setDocument()
doc.setDocument(myObj);
myDoc("a")=123
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
creates

setDocument()
doc.setDocument(myObj);
myDoc("a")=123
myObj = {
a: 123,
b: {
c1: 'foo',
c2: 'foo2'
},
d: {
e1: {
f1a: 'bar1a',
f2a: 'bar2a'
},
e2: {
f1b: 'bar1b',
f2b: 'bar2b',
f3b: 'bar3b'
}
}
}
myDoc is created on disk, containing a copy of
the data that was held within the myObj
object, now mapped to Global Storage nodes
creates

Making This Possible
• The cache.node interface
• Included with Caché (and IRIS)
• Runs in-process with Caché
• Connects to Caché's C++ Callin Interface
– Directly accessing the Caché database engine
– Bypassing the language interface

cache.node interface
Node.js
Process
Caché
Process
cache.node
module
C++call-ininterface
JavaScript
Globals Functions Objects

cache.node APIs
• Low-level
• Basic Global primitives
• "Global-centric"
• Not JavaScript-centric

ewd-document-store
• Builds on top of the cache.node APIs
• Creates a JavaScript-centric abstraction of
Global Storage
– Mapping to/from JavaScript Objects
• The JavaScript equivalent of ^
– Local JavaScript objects (in-memory)
– Persistent JavaScript objects (on-disk)

Node.js Architecture
• Everything executes in a single process
– You might have 1000's of users doing stuff
concurrently
• They're all sharing the same process!

Brainchild of Ryan Dahl

Node.js = server-side JavaScript
• Ryan Dahl wasn't actually a big JavaScript fan
– Node.js wasn't designed to be server-side JavaScript
• But he realised that it was a convenient
language for such an environment, as it's
designed for exactly the same kind of way of
working in the browser
• Google's V8 JavaScript engine was Open
Sourced, so he used it to provide the language
for Node.js
– (which is actually mostly written in C++)

Node.js Architecture
• Non-blocking I/O
– No user activity must block the process, or everyone
grinds to a halt
• Event-driven, asynchronous logic
– Fire off the request, but don't wait for the results
• Carry on with the next task
– When results come back, handle them at the next
opportunity

ewd-document-store requires
synchronous APIs
var myDoc = new this.documentStore.DocumentNode('myDoc');
var f3Node = myDoc.$('d').$('e2').$('f3');
myDoc.$d.$e2.$f3.value = 'New value';
myDoc("a")=123
myDoc("d","e2","f3")=”New value"
Requires properly chainable functions

The Problem is Concurrency
• All concurrent users in Node.js share the
same process

Is there a way to avoid this?
• Would it be possible to create a locally-
available environment where my Node.js
code runs in an isolated container where
concurrency isn't an issue?

What Is QEWD?
• Essentially it's a multi-purpose Node.js-
based run-time Platform

What Is QEWD?
• Essentially it's a multi-purpose Node.js-
based run-time Platform
• Creates an isolated run-time container for
your message/request handler functions,
allowing:
– CPU-intensive work
– Database abstractions using synchronous
logic

QEWD's Architecture
• Master Process
• Pool of Worker Processes

QEWD's Architecture
• Master Process
– Handles and queues all incoming requests
from client
• HTTP/REST requests via Express or Koa.js
• WebSocket requests via socket.io
– Returns responses to client

QEWD's Architecture
• Pool of Persistent Worker Processes
– Where all the processing occurs
– A single queued request is dispatched to an
available worker process
– Each Worker process handles a single
request at a time

QEWD's Architecture
• The queue / dispatcher / worker-process
pool management part of QEWD is
handled by a module named ewd-qoper8

Master Node.js Process
Queue
Queue
processor/
dispatcher
Incoming
Requests
QEWD Architecture
Every incoming request
is passed from Express
and placed in a queue
No further processing
of requests occurs in
the master process
Express
or
Koa.js
socket.io
HTTP
REST
WebSocket

QEWD Architecture
Queue
Queue
processor/
dispatcher
Queue dispatcher is
invoked whenever a
request is added to
the queue

QEWD Architecture
Node.js Worker Process
Queue
Queue
processor/
dispatcher
Worker process
started if none
available

QEWD Architecture
Queue
Queue
processor/
dispatcher
Redis/Cache
QEWD &
application-specific
Modules loaded
Custom
Worker
Module
Custom
Worker
Module

QEWD Architecture
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Custom
Worker
Module
Request passed
to worker
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Worker flagged as Unavailable
Custom
Worker
Module
Begin processing message
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Unavailable / processing
Another incoming
request Node.js Worker Process
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Custom
Worker
Module
Custom
Worker
Module
Custom
Worker
Module
If worker pool size not exceeded,
another worker is started
and request passed to it
Redis/Cache
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
If entire Worker Pool is busy:
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
If entire Worker Pool is busy:
New
requests
remain
in queue
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
As soon as a worker is available again,
a queued message can be passed to it
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
Available
Custom
Worker
Module
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Finished
Custom
Worker
Module
Custom
Worker
Module
Custom
Worker
Module
Redis/Cache
A user's handler function signals
completion using the function:
finished(responseObject);
This returns the response
object to the master
process

QEWD Architecture
Queue
Queue
processor/
dispatcher
Finished
Custom
Worker
Module
Custom
Worker
Module
Custom
Worker
Module
And the response is
returned to the client
that sent the original request
(via Express/Koa/socket.io)
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Available
Custom
Worker
Module
The finished() function
also automatically returns
the worker process back
to the available pool
So it can now handle
the next queued request
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Available
Custom
Worker
Module
Worker processes, once started,
are persistent
No start-up / tear-down cost
Redis/Cache

QEWD Architecture
Queue
Queue
processor/
dispatcher
Node.js concurrency is handled
by the master process.
100% asynchronous logic
The master process does
almost nothing
The perfect Node.js application:
no CPU-intensive or long-
running tasks, so very
high-performance
Multiple
Concurrent
Incoming
requests

QEWD Architecture
Queue
Queue
processor/
dispatcher
Node.js concurrency is handled
by the master process.
100% asynchronous logic
The master process does
almost nothing
The perfect Node.js application:
no CPU-intensive or long-
running tasks, so very
high-performance
All the actual work happens in the
isolated worker processes
Multiple
Concurrent
Incoming
requests

QEWD Architecture
Queue
Queue
processor/
dispatcher
Available
Custom
Worker
Module
Worker processes only handle
a single request at a time
Completely isolated run-time
environment for handler functions
No need for concerns about
Node.js concurrency, so
synchronous APIs can be used
Redis/Cache

QEWD + ewd-document-store +
cache.node
• JavaScript as a first-class language for
Caché
• Provides the JavaScript equivalent of ^
– Local JavaScript objects (in-memory)
– Persistent JavaScript objects (on-disk)
• JavaScript with data persistence as a
language feature

Just one remaining issue
• Native Caché ObjectScript performance:
– Set: >1 million Global Nodes / sec
• cache.node
– Around 8 – 10% of that performance

The reason?
• The Google V8 Engine's API that is used
by cache.node
– Has a bottleneck
• Usually thought to be insignificant
• Becomes significant for the kinds of things done by
ewd-document-store

If it can be fixed
• We'd have the fastest Node.js database
on the planet
– By a significant margin
• We'd have data persistence as a feature
of JavaScript
– Until it's fully demonstrable, the IT mainstream
aren't going to "get it"

Can It Be Fixed?
• Probably requires Google to sort it out
• Logged on their bug tracker
– But no sign of them doing anything to fix it
• Watch this space
• If it can be fixed, this is potentially huge
– And we all stand to benefit

Additional Reading
• Universal NoSQL Paper:
– http://mgateway.com/docs/universalNoSQL.pdf
• QEWD.js:
– https://qewdjs.com
• JavaScript Abstraction of Global Storage:
– http://docs.qewdjs.com/qewd_training.html
• In particular, parts 17 – 27
• QEWD.js from a Node.js perspective:
– http://bit.ly/2gnnyh5
– http://bit.ly/2ikY1sS

Data Persistence as a Language Feature

More Related Content

What's hot

Similar to Data Persistence as a Language Feature

More from Rob Tweed

Recently uploaded

Data Persistence as a Language Feature