Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EWD 3 Training Course Part 34: QEWD Resilient Mode

412 views

Published on

This is part 34 of the EWD 3 Training Course. This presentation explains how you can add resilience to the message queue used by QEWD by making it save all incoming requests to your embedded Global Storage database. In Resilient Mode, QEWD also saves a copy of all responses sent to clients, so the stored activity information can be used as an audit trail.

Published in: Software

EWD 3 Training Course Part 34: QEWD Resilient Mode

  1. 1. Copyright © 2016 M/Gateway Developments Ltd EWD 3 Training Course Part 34 QEWD Resilient Mode Rob Tweed Director, M/Gateway Developments Ltd Twitter: @rtweed
  2. 2. Copyright © 2016 M/Gateway Developments Ltd QEWD's Request Queue • QEWD uses the ewd-qoper8 module to provide the master process queue and worker dispatch mechanism • ewd-qoper8's queue is memory based – A simple JavaScript array • If the master process stops unexpectedly or whilst the system is very busy, you might lose some request messages
  3. 3. Copyright © 2016 M/Gateway Developments Ltd QEWD Audit Information • By default, QEWD does not keep any history or audit information, eg: – A history of incoming request messages – A history of corresponding responses
  4. 4. Copyright © 2016 M/Gateway Developments Ltd QEWD Resilient Mode • QEWD's built in, optional Resilient Mode addresses these concerns, at least for WebSocket/Ajax messages – Backs up incoming requests to your embedded database • Redis or Cache • Not yet available for GT.M – Backs up response objects to your embedded database – Flags completed messages in the database • When your message handler's finished() function is invoked
  5. 5. Copyright © 2016 M/Gateway Developments Ltd QEWD Recovery • If the QEWD master process stops and is restarted – Any clients/browsers that were previously running an interactive application (WebSockets or Ajax) will re- register automatically • Re-establishes the client's QEWD Session – On re-registration, QEWD checks to see if that Session had any uncompleted requests in the Resilient Mode backup database • If so, they are re-queued automatically
  6. 6. Copyright © 2016 M/Gateway Developments Ltd QEWD Resilient Mode Storage • The QEWD master process saves any incoming messages asynchronously to your embedded Global Storage database, using the Document Store abstraction – By default, saved in a document named ewdQueue – You can opt to change this name – This is done in parallel to ewd-qoper8's in-memory queue • ewdQueue is only used as a backup • Resilient mode shouldn't significantly affect ewd-qoper8's performance
  7. 7. Copyright © 2016 M/Gateway Developments Ltd QEWD Audit Trail • The ewdQueue Document can be used as an audit trail. It contains a record of – Every incoming request object – Every outgoing response object • You can determine the retention period – Default is 1 hour – Any records older than the retention period are automatically purged by QEWD
  8. 8. Copyright © 2016 M/Gateway Developments Ltd What about Web/REST Services? • Web/REST Service HTTP requests are not currently saved into the ewdQueue Document • Reason: – If the master process stops, the connection context between the client and Express is lost – So it would not be possible to return a response for a re-queued HTTP request • This may change in future – Particularly where you've linked the client to a QEWD Session: this would allow re-establishment of client connection context
  9. 9. Copyright © 2016 M/Gateway Developments Ltd Enabling Resilient Mode • Simple change to your QEWD startup file • Add the resilientMode property to the config object • Setting it to true applies the defaults: – ewdQueue as the Document Name – 1 hour retention period • Can over-ride these via an object instead
  10. 10. Copyright © 2016 M/Gateway Developments Ltd Edit your QEWD Startup File var config = { managementPassword: 'keepThisSecret!', serverName: 'My QEWD Server', port: 8080, poolSize: 2, database: { type: 'cache', params: { path: 'c:InterSystemsCache2015-2mgr’ }, resilientMode: true } }; var qewd = require('qewd').master; qewd.start(config); Default mode enabled
  11. 11. Copyright © 2016 M/Gateway Developments Ltd Edit your QEWD Startup File var config = { managementPassword: 'keepThisSecret!', serverName: 'My QEWD Server', port: 8080, poolSize: 2, database: { type: 'cache', params: { path: 'c:InterSystemsCache2015-2mgr’ }, resilientMode: { keepPeriod: 3600 * 24 // keep for 24 hours } } }; var qewd = require('qewd').master; qewd.start(config); Specify the required retention period in seconds
  12. 12. Copyright © 2016 M/Gateway Developments Ltd Edit your QEWD Startup File var config = { managementPassword: 'keepThisSecret!', serverName: 'My QEWD Server', port: 8080, poolSize: 2, database: { type: 'cache', params: { path: 'c:InterSystemsCache2015-2mgr’ }, resilientMode: { queueDocumentName: 'myAuditTrail' } } }; var qewd = require('qewd').master; qewd.start(config); Save the audit trail information into a Document named myAuditTrail instead
  13. 13. Copyright © 2016 M/Gateway Developments Ltd Edit your QEWD Startup File var config = { managementPassword: 'keepThisSecret!', serverName: 'My QEWD Server', port: 8080, poolSize: 2, database: { type: 'cache', params: { path: 'c:InterSystemsCache2015-2mgr’ }, resilientMode: { queueDocumentName: 'myAuditTrail', keepPeriod: 3600*24*30 // 30 day retention } } }; var qewd = require('qewd').master; qewd.start(config);
  14. 14. Copyright © 2016 M/Gateway Developments Ltd Restart QEWD • Resilient Mode is now enabled • Logging / backup of queued requests and responses will now commence
  15. 15. Copyright © 2016 M/Gateway Developments Ltd Queue Document Structure { 'message': { {{timeStamp}}: { 'token': //EWD Session token, 'content': //queued request message payload } }, 'pending': { sessionToken: { {{timeStamp}}: ''} } } Where {{timeStamp}} is a unique index/property created from the system clock time
  16. 16. Copyright © 2016 M/Gateway Developments Ltd Time-stamp format Logic used is: var time = process.hrtime(); var ix = time[0] * 1e9 + time[1]; // time in nanoseconds var timeStamp = new Date().getTime() + '-' + ix; So the date/time for each audit record can be obtained from: timeStamp.split('-')[0]
  17. 17. Copyright © 2016 M/Gateway Developments Ltd Queue Document Structure During message processing by worker process, workerStatus information added: { 'message': { {{timeStamp}}: { 'token': //EWD Session token, 'content': //queued request message payload, workerStatus: 'started' || 'error' } }, 'pending': { sessionToken: { {{timeStamp}}: ''} } }
  18. 18. Copyright © 2016 M/Gateway Developments Ltd Queue Document Structure When processing by a worker process is completed, workerStatus will be finished and pending index record is deleted { 'message': { {{timeStamp}}: { 'token': //EWD Session token, 'content': //queued request message payload, workerStatus: 'finished' } } }
  19. 19. Copyright © 2016 M/Gateway Developments Ltd Queue Document Structure Response objects are logged in the response array { 'message': { {{timeStamp}}: { 'token': //EWD Session token, 'content': //queued request message payload, workerStatus: 'finished', response: [ // one or more response objects ] } } }
  20. 20. Copyright © 2016 M/Gateway Developments Ltd QEWD Audit Trail • The ewdQueue Document is a standard ewd-document-store Document, so it can be accessed and analysed using the standard DocumentStore / DocumentNode APIs. • You can also inspect it using the qewd- monitor application • It can be deleted or manipulated without affecting QEWD's operation

×