Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright © 2016 M/Gateway Developments Ltd
EWD 3 Training Course
Part 2
Overview of EWD 3
Rob Tweed
Director, M/Gateway D...
Copyright © 2016 M/Gateway Developments Ltd
EWD 3 Design Aims
• Modularising previous-generation (EWD.js) into
independent...
Copyright © 2016 M/Gateway Developments Ltd
Core EWD 3 Component
• ewd-qoper8
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
• Very high-performance Node.js Message
Queue
• Master Node.js proc...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
• Creates an isolated run-time environment
for each handler functio...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
• Worker processes are persistent
– Started on demand
– When a work...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
• Master process API
– this.addToQueue(messageObject)
• Adds a mess...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
• What the worker does to process a
message is up to you
• You defi...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
addToQueue...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailabl...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailabl...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailabl...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Unavailabl...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Finished
N...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Finished
t...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Available
...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8
Master Node.js Process
Queue
Queue
processor/
dispatcher
Available
...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 is a building block
• Not really useful on its own
• Completely gen...
Copyright © 2016 M/Gateway Developments Ltd
Express + ewd-qoper8
• Express is the standard Node.js web server
– REST, Web ...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + Express
Queue
Queue
processor/
dispatcher
Express
HTTP(S)
Interfa...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + Express
+ ewd-qoper8-express
Queue
Queue
processor/
dispatcher
Ex...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + Express
+ ewd-qoper8-express
Queue
Queue
processor/
dispatcher
Ex...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 Database Dependency?
• ewd-qoper8 itself does not have a
dependency...
Copyright © 2016 M/Gateway Developments Ltd
Using ewd-qoper8 with Caché
• Use a separate module:
– ewd-qoper8-cache
• Norm...
Copyright © 2016 M/Gateway Developments Ltd
Using ewd-qoper8 with GT.M
• Use a separate module:
– ewd-qoper8-gtm
• Normall...
Copyright © 2016 M/Gateway Developments Ltd
Using ewd-qoper8 with Redis
• Use a separate module:
– ewd-qoper8-redis
• Norm...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
Node.js Worker Process
Master Node.js Process
Qu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-gtm
Node.js Worker Process
Master Node.js Process
Queu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-redis
Node.js Worker Process
Master Node.js Process
Qu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
Node.js Worker Process
Master Node.js Process
Qu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-gtm
Node.js Worker Process
Master Node.js Process
Queu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
Node.js Worker Process
Master Node.js Process
Qu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-gtm
Node.js Worker Process
Master Node.js Process
Queu...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-redis
Node.js Worker Process
Master Node.js Process
Qu...
Copyright © 2016 M/Gateway Developments Ltd
JavaScript-centric database access
• ewd-qoper8-cache, ewd-qoper8-gtm and
ewd-...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
+ ewd-document-store
Node.js Worker Process
Mast...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
+ ewd-document-store
Node.js Worker Process
Mast...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-gtm
+ ewd-document-store
Node.js Worker Process
Master...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-redis
+ ewd-document-store
Node.js Worker Process
Mast...
Copyright © 2016 M/Gateway Developments Ltd
ewd-document-store
myDocument
“a” 123
“b”
“c2” “foo2”
“d”
“c1” “foo”
“e2”
“e1”...
Copyright © 2016 M/Gateway Developments Ltd
ewd-document-store
myDocument
“a” 123
“b”
“c2” “foo2”
“d”
“c1” “foo”
“e2”
“e1”...
Copyright © 2016 M/Gateway Developments Ltd
ewd-document-store
myDocument
“a” 123
“b”
“c2” “foo2”
“d”
“c1” “foo”
“e2”
“e1”...
Copyright © 2016 M/Gateway Developments Ltd
ewd-document-store
myDocument
“a” 123
“b”
“c2” “foo2”
“d”
“c1” “foo”
“e2”
“e1”...
Copyright © 2016 M/Gateway Developments Ltd
ewd-document-store
myDocument
“a” 123
“b”
“c2” “foo2”
“d”
“c1” “foo”
“e2”
“e1”...
Copyright © 2016 M/Gateway Developments Ltd
ewd-document-store
myDocument
“a” 123
“b”
“c2” “foo2”
“d”
“c1” “foo”
“e2”
“e1”...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
+ ewd-document-store
Node.js Worker Process
Mast...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-gtm
+ ewd-document-store
Node.js Worker Process
Master...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-redis
+ ewd-document-store
Node.js Worker Process
Mast...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 for synchronous
database access
• A worker process only handles one...
Copyright © 2016 M/Gateway Developments Ltd
Sessions
• EWD 3 uses a stateless model for web access
• Each request from a c...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
+ ewd-document-store + ewd-session
Node.js Worke...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-gtm
+ ewd-document-store + ewd-session
Node.js Worker ...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-redis
+ ewd-document-store + ewd-session
Node.js Worke...
Copyright © 2016 M/Gateway Developments Ltd
ewd-session APIs
• sessions.create(applicationName)
– Creates a new session, a...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache*+
ewd-document-store + ewd-session
Node.js Worke...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache*
+ ewd-document-store + ewd-session
Node.js Work...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache*
+ ewd-document-store + ewd-session
Node.js Work...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8 + ewd-qoper8-cache
+ ewd-document-store + ewd-session
+ Express
Nod...
Copyright © 2016 M/Gateway Developments Ltd
Pre-packaged super-modules
• QEWD (formally known as ewd-xpress)
– Quick and E...
Copyright © 2016 M/Gateway Developments Ltd
QEWD (using Caché)
Node.js Worker Process
Master Node.js Process
Queue
Queue
p...
Copyright © 2016 M/Gateway Developments Ltd
QEWD (using GT.M)
Node.js Worker Process
Master Node.js Process
Queue
Queue
pr...
Copyright © 2016 M/Gateway Developments Ltd
QEWD (using Redis)
Node.js Worker Process
Master Node.js Process
Queue
Queue
p...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8-vistarpc (using Caché)
Node.js Worker Process
Master Node.js Proces...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8-vistarpc (using GT.M)
Node.js Worker Process
Master Node.js Process...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8-vistarpc
• Login: (POST)
– Access & verify codes
– XUS SIGNON SETUP...
Copyright © 2016 M/Gateway Developments Ltd
ewd-qoper8-vistarpc
• If logged in - runRPC:
– Use Session token in Authorizat...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Node.js Worker Process
Master Node.js Process
Queue
Queue
processor...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
Worker Process ewd-...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
Worker Process ewd-...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
REST
Server
REST
Se...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
REST
Server
REST
Se...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
REST
Server
REST
Se...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
REST
Server
REST
Se...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8
Master Process
Express
REST/
Web
Service
Client
Worker Process ewd-...
Copyright © 2016 M/Gateway Developments Ltd
Aggregate response
{
Server_1_name: {responseObject},
Server_2_name: {errorObj...
Copyright © 2016 M/Gateway Developments Ltd
Interception / Customisation
• ewd-feder8 emits events:
– When it receives a r...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Copyright © 2016 M/Gateway Developments Ltd
ewd-feder8 "dance"
Master Process
Express
REST/
Web
Service
Client
REST
Server...
Upcoming SlideShare
Loading in …5
×

EWD 3 Training Course Part 2: EWD 3 Overview

1,304 views

Published on

This presentation is part 2 of the EWD 3 Training Course. It provides a quick, high-level overview of the suite of modules that constitute EWD 3 and shows how they interoperate, allowing you to build a sophisticated Node.js platform

Published in: Software

EWD 3 Training Course Part 2: EWD 3 Overview

  1. 1. Copyright © 2016 M/Gateway Developments Ltd EWD 3 Training Course Part 2 Overview of EWD 3 Rob Tweed Director, M/Gateway Developments Ltd Twitter: @rtweed
  2. 2. Copyright © 2016 M/Gateway Developments Ltd EWD 3 Design Aims • Modularising previous-generation (EWD.js) into independent components • Mix and match "buffet" of components • Interoperable with other Node.js modules, eg Express • No core dependence on particular databases • Redis, Caché and GT.M can be used as embedded multi- model databases • But any other Node.js-interfaced database can be used too
  3. 3. Copyright © 2016 M/Gateway Developments Ltd Core EWD 3 Component • ewd-qoper8
  4. 4. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 • Very high-performance Node.js Message Queue • Master Node.js process: • Queue & dispatch mechanism • Pool of persistent Worker Node.js processes • These process the requests that are put on the queue
  5. 5. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 • Creates an isolated run-time environment for each handler function • Each worker process only handles a single request at a time • Comparable to AWS Lambda's run-time container • Concurrency is handled by the master process only • Its sole purpose is to queue and dispatch incoming requests to an available worker process, and handle the response/results from the worker
  6. 6. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 • Worker processes are persistent – Started on demand – When a worker process finishes processing a request, it is returned to the available pool, ready to process another request – No startup / tear-down cost
  7. 7. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 • Master process API – this.addToQueue(messageObject) • Adds a message to the queue – this.on('response', function(responseObject, workerPid) {..} • Handles the response from worker – Usually simply returning the response to the client that sent the original request message to the master process
  8. 8. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 • What the worker does to process a message is up to you • You define a custom worker module • Worker API: – this.on('message', function(messageObject, send, finished) {..} • finished(responseObject): – Mandatory function – Sends final response to master process – Releases Worker process back to available pool • send(responseObject): – Optional function – Sends a response to master process – Worker process remains unavailable
  9. 9. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Node.js Worker Process Node.js Worker Process
  10. 10. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher addToQueue(obj)
  11. 11. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher
  12. 12. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher
  13. 13. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module ewd-qoper8
  14. 14. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8
  15. 15. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Unavailable Node.js Worker Process Custom Worker Module ewd-qoper8 Begin processing message
  16. 16. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Unavailable worker.on('message', function(messageObj) { // process message }); Node.js Worker Process Custom Worker Module ewd-qoper8 Begin processing message
  17. 17. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing addToQueue(obj) Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8
  18. 18. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 Node.js Worker Process Custom Worker Module ewd-qoper8 Worker pool size not exceeded:
  19. 19. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 Worker pool all busy: Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 New request remains in queue
  20. 20. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 As soon as a worker is available again: Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 Available Node.js Worker Process Custom Worker Module ewd-qoper8
  21. 21. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 Worker pool all busy: Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 Unavailable / processing Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8
  22. 22. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Finished Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8 finished(responseObject);
  23. 23. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Finished this.on('response', function(responseObj) { // do something with the response }); Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module Node.js Worker Process Custom Worker Module ewd-qoper8
  24. 24. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Available Node.js Worker Process Custom Worker Module ewd-qoper8 Worker automatically returned to the available pool NB: Workers are NOT destroyed after use Removes overhead of continually starting worker processes
  25. 25. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Master Node.js Process Queue Queue processor/ dispatcher Available Node.js Worker Process Custom Worker Module ewd-qoper8 Worker processes only handle a single request at a time Completely isolated run-time environment for your handler functions Worker process becomes immediately available for the next queued request when processing completed
  26. 26. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 is a building block • Not really useful on its own • Completely generic, general-purpose message queue • Core piece of infrastructure on which the rest of EWD 3 is built – ewd-qoper8 can also be used independently of the other EWD 3 modules and integrated with any other Node.js modules, databases etc
  27. 27. Copyright © 2016 M/Gateway Developments Ltd Express + ewd-qoper8 • Express is the standard Node.js web server – REST, Web Services, interactive applications – Express is a Node.js module – Customised by defining middleware – ewd-qoper8 can be configured as Express middleware • Requests sent to Express by client • Routed to ewd-qoper8 • ewd-qoper8 queues and processes them • ewd-qoper8 response returned via Express to client
  28. 28. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + Express Queue Queue processor/ dispatcher Express HTTP(S) Interface WebSocket socket.io Interface addToQueue(req) Node.js Worker Process Custom Worker Module ewd-qoper8
  29. 29. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + Express + ewd-qoper8-express Queue Queue processor/ dispatcher Express HTTP(S) Interface Automates and simplifies the use of ewd-qoper8 with Express for HTTP requests Adds special Express router function app.use('/qoper8', qx.router()); ewd-qoper8-express ewd-qoper8- express Node.js Worker Process Custom Worker Module ewd-qoper8
  30. 30. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + Express + ewd-qoper8-express Queue Queue processor/ dispatcher Express HTTP(S) Interface Message constructed from Express req object – all information needed for back-end processing by Worker: -path -method -headers -query -body -etc ewd-qoper8-express ewd-qoper8- express Node.js Worker Process Custom Worker Module ewd-qoper8
  31. 31. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 Database Dependency? • ewd-qoper8 itself does not have a dependency on a database – In-memory queue – Master and Workers are just Node.js processes * see later parts of this course for details on how to add queue backup / resilience
  32. 32. Copyright © 2016 M/Gateway Developments Ltd Using ewd-qoper8 with Caché • Use a separate module: – ewd-qoper8-cache • Normally used by worker processes • Makes use of cache.node interface module – cache.node is a proprietary InterSystems module that is included with Caché – Allows access from JavaScript/Node.js to: » Caché extrinsic functions » Global storage » Caché Objects & methods – JavaScript API defined by InterSystems » Low-level
  33. 33. Copyright © 2016 M/Gateway Developments Ltd Using ewd-qoper8 with GT.M • Use a separate module: – ewd-qoper8-gtm • Normally used by worker processes • Makes use of NodeM interface module – NodeM is an Open Source module for use with GT.M, which implements the cache.node APIs – Allows access from JavaScript/Node.js to: » GT.M extrinsic functions » Global storage – JavaScript API originally defined by InterSystems » Low-level
  34. 34. Copyright © 2016 M/Gateway Developments Ltd Using ewd-qoper8 with Redis • Use a separate module: – ewd-qoper8-redis • Normally used by worker processes • Makes use of ewd-redis-globals module – ewd-redis-globals is an Open Source module for use with Redis, which: » Implements Global Storage functionality » implements the cache.node APIs – Allows access from JavaScript/Node.js to: » Global storage » Uses the tcp-netx module to connect to Redis – JavaScript API originally defined by InterSystems » Low-level
  35. 35. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module cache.node ewd-qoper8-cache Caché ewd-qoper8
  36. 36. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-gtm Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module nodem ewd-qoper8-gtm GT.M ewd-qoper8 Equivalent module for use with GT.M Uses David Wicksell’s NodeM module which emulates cache.node for GT.M
  37. 37. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-redis Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module tcp-netx ewd-qoper8-redis Redis ewd-qoper8 Equivalent module for use with Redis Redis behaves as a Global Storage Database via the same cache.node APIs ewd-redis-globals
  38. 38. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché function Custom Worker Module ewd-qoper8 For legacy Mumps or Caché ObjectScript Function integration
  39. 39. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-gtm Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher nodem ewd-qoper8-gtm GT.M function Custom Worker Module ewd-qoper8 For legacy Mumps Function integration
  40. 40. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché Globals Custom Worker Module ewd-qoper8 Basic low-level Mumps-centric API
  41. 41. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-gtm Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher nodem ewd-qoper8-gtm GT.M Globals Custom Worker Module ewd-qoper8 Basic low-level Mumps-centric API
  42. 42. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-redis Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module tcp-netx ewd-qoper8-redis Redis ewd-qoper8 ewd-redis-globals Globals Basic low-level API
  43. 43. Copyright © 2016 M/Gateway Developments Ltd JavaScript-centric database access • ewd-qoper8-cache, ewd-qoper8-gtm and ewd-qoper8-redis auto-install ewd- document-store • ewd-document-store provides a JavaScript-oriented abstraction of Global Storage database: – "persistent JavaScript Objects" – Document database • Fine-grained storage of JSON
  44. 44. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Custom Worker Module ewd-qoper8
  45. 45. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals Persistent JavaScript Objects & Fine-grained Document Database Custom Worker Module ewd-qoper8
  46. 46. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-gtm + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher nodem ewd-qoper8-gtm GT.M ewd-document-store Globals Persistent JavaScript Objects & Fine-grained Document Database Custom Worker Module ewd-qoper8
  47. 47. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-redis + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module tcp-netx Redis ewd-qoper8 ewd-redis-globals Globals ewd-qoper8-gtm ewd-document-store Persistent JavaScript Objects & Fine-grained Document Database
  48. 48. Copyright © 2016 M/Gateway Developments Ltd ewd-document-store myDocument “a” 123 “b” “c2” “foo2” “d” “c1” “foo” “e2” “e1” “f2” “bar2” “f1” “bar1” “f2” “bar2” “f1” “bar1” “f3” “bar3” myDocument = { a: 123, b: { c1: 'foo', c2: 'foo2' } d: { e1: { f1: 'bar1', f2: 'bar2' }, e2: { f1: 'bar1', f2: 'bar2', f3: 'bar3' } } }
  49. 49. Copyright © 2016 M/Gateway Developments Ltd ewd-document-store myDocument “a” 123 “b” “c2” “foo2” “d” “c1” “foo” “e2” “e1” “f2” “bar2” “f1” “bar1” “f2” “bar2” “f1” “bar1” “f3” “bar3” var f3node = new this.documentStore.DocumentNode('myDocument', ['d', 'e2', 'f3']);
  50. 50. Copyright © 2016 M/Gateway Developments Ltd ewd-document-store myDocument “a” 123 “b” “c2” “foo2” “d” “c1” “foo” “e2” “e1” “f2” “bar2” “f1” “bar1” “f2” “bar2” “f1” “bar1” “f3” “bar3” var value = f3Node.value; // 'bar3'
  51. 51. Copyright © 2016 M/Gateway Developments Ltd ewd-document-store myDocument “a” 123 “b” “c2” “foo2” “d” “c1” “foo” “e2” “e1” “f2” “bar2” “f1” “bar1” “f2” “bar2” “f1” “bar1” “f3” “bar3” var value = f3Node.parent.$('f2').value; // 'bar2'
  52. 52. Copyright © 2016 M/Gateway Developments Ltd ewd-document-store myDocument “a” 123 “b” “c2” “foo2” “d” “c1” “foo” “e2” “e1” “f2” “bar2” “f1” “bar1” “f2” “bar2” “f1” “bar1” “f3” “bar3” myDocument = { a: 123, b: { c1: 'foo', c2: 'foo2' } d: { e1: { f1: 'bar1', f2: 'bar2' }, e2: { f1: 'bar1', f2: 'bar2', f3: 'bar3' } } } var docOnDisk = new this.documentStore.DocumentNode('myDocument'); var localDoc = docOnDisk.getDocument();
  53. 53. Copyright © 2016 M/Gateway Developments Ltd ewd-document-store myDocument “a” 123 “b” “c2” “foo2” “d” “c1” “foo” “e2” “e1” “f2” “bar2” “f1” “bar1” “f2” “bar2” “f1” “bar1” “f3” “bar3” myDocument = { a: 123, b: { c1: 'foo', c2: 'foo2' } d: { e1: { f1: 'bar1', f2: 'bar2' }, e2: { f1: 'bar1', f2: 'bar2', f3: 'bar3' } } } var docOnDisk = new this.documentStore.DocumentNode('myDocument'); docOnDisk.setDocument(localDoc);
  54. 54. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals This powerful & intuitive abstraction requires synchronous access ewd-qoper8 provides the isolated run-time container to make this possible Custom Worker Module ewd-qoper8
  55. 55. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-gtm + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher nodem ewd-qoper8-gtm GT.M ewd-document-store Globals Custom Worker Module ewd-qoper8 This powerful & intuitive abstraction requires synchronous access ewd-qoper8 provides the isolated run-time container to make this possible
  56. 56. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-redis + ewd-document-store Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module tcp-netx Redis ewd-qoper8 ewd-redis-globals Globals ewd-qoper8-gtm ewd-document-store This powerful & intuitive abstraction requires synchronous access ewd-qoper8 provides the isolated run-time container to make this possible
  57. 57. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 for synchronous database access • A worker process only handles one request object at a time • Any blocking database (or other) I/O has no impact on the processing of other requests – Each request independently handled, isolated in its own worker
  58. 58. Copyright © 2016 M/Gateway Developments Ltd Sessions • EWD 3 uses a stateless model for web access • Each request from a client may be processed by a different ewd-qoper8 worker process • Usually you need to maintain state information across a sequence of client requests • EWD Sessions provide this capability • The ewd-session module allows you to create and maintain Sessions – Sessions are automatically mapped to ewd-document DocumentNode objects
  59. 59. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache + ewd-document-store + ewd-session Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session Session storage projected as DocumentNode objects Custom Worker Module ewd-qoper8
  60. 60. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-gtm + ewd-document-store + ewd-session Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher nodem ewd-qoper8-gtm GT.M ewd-document-store Globals ewd-session Session storage projected as DocumentNode objects Custom Worker Module ewd-qoper8
  61. 61. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-redis + ewd-document-store + ewd-session Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher Custom Worker Module tcp-netx Redis ewd-qoper8 ewd-redis-globals Globals ewd-qoper8-gtm ewd-document-store ewd-session Session storage projected as DocumentNode objects
  62. 62. Copyright © 2016 M/Gateway Developments Ltd ewd-session APIs • sessions.create(applicationName) – Creates a new session, associated with the specified application – Returns a session token • Randomly-generated uuid string • sessions.authenticate(token) – Checks that token is valid and has not yet expired – returns the session object for the token, or an error
  63. 63. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache*+ ewd-document-store + ewd-session Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session var session = sessions.create('vista'); Send back session.token Custom Worker Module ewd-qoper8 * Applies also to ewd-qoper8-gtm and ewd-qoper8-redis
  64. 64. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache* + ewd-document-store + ewd-session Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session var result = sessions.authenticate(token); Returns result.session if successful / result.error if not Custom Worker Module ewd-qoper8 * Applies also to ewd-qoper8-gtm and ewd-qoper8-redis
  65. 65. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache* + ewd-document-store + ewd-session Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session Session object extends GlobalNode object Adds session-specific properties & methods Custom Worker Module ewd-qoper8 * Applies also to ewd-qoper8-gtm and ewd-qoper8-redis
  66. 66. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8 + ewd-qoper8-cache + ewd-document-store + ewd-session + Express Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session Express HTTP(S) Interface WebSocket socket.io Interface Custom Worker Module ewd-qoper8 Functions
  67. 67. Copyright © 2016 M/Gateway Developments Ltd Pre-packaged super-modules • QEWD (formally known as ewd-xpress) – Quick and Easy Web Development platform for: • Interactive browser-based applications – Web-sockets or Ajax • Web and REST services • Embedded persistent JSON store / session cache using: – Caché, GT.M or Redis • Compatible with all modern JavaScript front-end frameworks • ewd-qoper8-vistarpc – REST interface to VistA RPCs • ewd-feder8 – Federation / integration platform • Extension of QEWD – Lightweight enterprise service bus functionality
  68. 68. Copyright © 2016 M/Gateway Developments Ltd QEWD (using Caché) Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher ewd-xpress cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session Express HTTP(S) Interface WebSocket socket.io Interface browserewd-client Custom Worker Module ewd-qoper8 Functions
  69. 69. Copyright © 2016 M/Gateway Developments Ltd QEWD (using GT.M) Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher ewd-xpress nodem ewd-qoper8-gtm GT.M ewd-document-store Globals ewd-session Express HTTP(S) Interface WebSocket socket.io Interface browserewd-client Custom Worker Module ewd-qoper8 Functions
  70. 70. Copyright © 2016 M/Gateway Developments Ltd QEWD (using Redis) Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher ewd-xpress ewd-redis-globals ewd-qoper8-redis Redis ewd-document-store Globals ewd-session Express HTTP(S) Interface WebSocket socket.io Interface browserewd-client Custom Worker Module ewd-qoper8 tcp-netx
  71. 71. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8-vistarpc (using Caché) Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher ewd-qoper8- vistarpc cache.node ewd-qoper8-cache ewd-document-store Globals ewd-session Express HTTP(S) Interface REST RPCs Interface function Caché & VistA Symbol Table Custom Worker Module ewd-qoper8 Symbol Table Function
  72. 72. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8-vistarpc (using GT.M) Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher ewd-qoper8- vistarpc nodem ewd-qoper8-gtm ewd-document-store Globals ewd-session Express HTTP(S) Interface REST RPCs Interface function GT.M & VistA Symbol Table Custom Worker Module ewd-qoper8 Symbol Table Function
  73. 73. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8-vistarpc • Login: (POST) – Access & verify codes – XUS SIGNON SETUP – XUS AV CODE – Returns error or greeting object – If successful: • Creates new Session • Symbol table saved to Session – Contains DUZ etc – No direct access by REST client – Only Indirect access via Authorization token • Return Session Token & VistA greeting 20 – 30ms round trip
  74. 74. Copyright © 2016 M/Gateway Developments Ltd ewd-qoper8-vistarpc • If logged in - runRPC: – Use Session token in Authorization header – POST: • RPC name • RPC arguments object – Symbol table restored from Session • Contains DUZ etc • Determines whether user can run RPC – Standard Mumps-side "business as usual" logic – Return RPC result – Save Symbol table to Session
  75. 75. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Node.js Worker Process Master Node.js Process Queue Queue processor/ dispatcher ewd-feder8 cache.node ewd-qoper8-cache Caché ewd-document-store Globals ewd-session Express HTTP(S) Interface Custom Worker Module ewd-qoper8 REST/ Web Service Client ewd-xpress Also works with GT.M and Redis
  76. 76. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client Worker Process ewd-feder8 ewd-qoper8 REST Server REST Server REST Server Web Service Server Web Service Server cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session
  77. 77. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client Worker Process ewd-feder8 ewd-qoper8 REST Server REST Server REST Server Web Service Server Web Service Server cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Named Group
  78. 78. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session
  79. 79. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Copy of request sent to all servers in group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session
  80. 80. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Responses Received asynchronously Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session
  81. 81. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Last response Received Aggregate response object created Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session
  82. 82. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 Master Process Express REST/ Web Service Client Worker Process ewd-feder8 Custom Worker Module ewd-qoper8 REST Server REST Server REST Server Web Service Server Web Service Server cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Named Group Aggregate response Returned to client via Master process
  83. 83. Copyright © 2016 M/Gateway Developments Ltd Aggregate response { Server_1_name: {responseObject}, Server_2_name: {errorObject}, Server_3_name: {responseObject} }
  84. 84. Copyright © 2016 M/Gateway Developments Ltd Interception / Customisation • ewd-feder8 emits events: – When it receives a request from a client – When it receives a response from an end- point server – When an aggregate response is ready for return to a client • These can be used to: – Intercept the flow through ewd-feder8; and – Customise the behaviour of ewd-feder8
  85. 85. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session
  86. 86. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Intercept response Possibly modify it And/or determine action based on response
  87. 87. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Forward new request to group of servers
  88. 88. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Intercept aggregate Response Decide what to do
  89. 89. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Send new request
  90. 90. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Intercept response Create final response
  91. 91. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Return final response to client
  92. 92. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session Cache can be Used to store/ build up final response during "dance"
  93. 93. Copyright © 2016 M/Gateway Developments Ltd ewd-feder8 "dance" Master Process Express REST/ Web Service Client REST Server REST Server REST Server Web Service Server Web Service Server Named Group Worker Process ewd-feder8 ewd-qoper8 cache.node ewd-qoper8-cache Caché ewd-document-store ewd-session EWD Sessions can also be used – potentially easier to use and manage

×