Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MICRE: Microservices In MediCal Research Environments

23 views

Published on

MICRE: Microservices In MediCal Research Environments

Published in: Technology
  • Be the first to comment

  • Be the first to like this

MICRE: Microservices In MediCal Research Environments

  1. 1. MICRE: Microservices In MediCal Research Environments Martin Chapman October 5th KDL, King’s College London
  2. 2. Overview How we can integrate the microservice design pattern into the software development practices of medical researchers, in order to improve software quality. 1
  3. 3. Microservices for medical research: the benefits • Resilience and scaling. • Health is a domain in which research is increasingly involving end-users (e.g., ’artificial intelligence’ logic, and its ability to support the autonomous diagnosis of patients). • Software supports this research (e.g. a decision-support system realises this AI logic). • This software must therefore adequately support interactions with these end-users, by being scalable and resilient. 2
  4. 4. Microservices for medical research: the benefits • Resilience and scaling. • Health is a domain in which research is increasingly involving end-users (e.g., ’artificial intelligence’ logic, and its ability to support the autonomous diagnosis of patients). • Software supports this research (e.g. a decision-support system realises this AI logic). • This software must therefore adequately support interactions with these end-users, by being scalable and resilient. • Technology heterogeneity and Composability • Important in research, particularly health research which is often cross-discipline, where researchers have different ways of developing software and different traditions. • Important for reusing existing (research software). 2
  5. 5. Microservices for medical research: the benefits • Ease of deployment • Important in helping to determine scientific reproducibility, e.g. easily spinning up an experiment architecture produced by another researcher, to ascertain if the same results are obtained. • Important for software sustainability, e.g. easily migrating the same system as the underlying hardware changes. 2
  6. 6. Microservices for medical research: the benefits • Ease of deployment • Important in helping to determine scientific reproducibility, e.g. easily spinning up an experiment architecture produced by another researcher, to ascertain if the same results are obtained. • Important for software sustainability, e.g. easily migrating the same system as the underlying hardware changes. • Disadvantages? Complexity and cost. 2
  7. 7. Microservices for medical research: what is needed? 1. A model for the development of software as microservices in health, dictating, for example, the technologies that should be used. 2. Tooling to enable researchers to use the paradigm in the short-term, e.g. an IDE through which researchers are able to graphically segmented their software into different services. 3. Training in the paradigm, in order to ensure its long-term visibility as an option when developing software. 3
  8. 8. Microservices for medical research: what is needed? 1. A model for the development of software as microservices in health, dictating, for example, the technologies that should be used. 2. Tooling to enable researchers to use the paradigm in the short-term, e.g. an IDE through which researchers are able to graphically segmented their software into different services. 3. Training in the paradigm, in order to ensure its long-term visibility as an option when developing software. 3
  9. 9. An initial model: development workflow 1. Develop (web) applications as a set of individual server components, written in Javascript, that communicate via HTTP. 2. Containerise those components. 3. Separate those components across different virtual machines where necessary. 4
  10. 10. Workflow: 1. Programming Logic Construct service logic (including relevant versioning and test development for CI): 1 const archiver = require(’archiver ’); 2 const output = fs.createWriteStream(__dirname + ’/example.zip’); 3 const archive = archiver(’zip’); 4 archive.pipe(output); zip.js 5
  11. 11. Workflow: 2. Server encapsulation Encapsulate logic in programmatic server: 1 router.get("/generate /:zipId/", async function(req , res , next) { 2 const archiver = require(’archiver ’); 3 const output = fs.createWriteStream(__dirname + ’/example.zip’); 4 const archive = archiver(’zip’); 5 archive.pipe(output); 6 }); routes/zip.js 6
  12. 12. Workflow: 3. Container encapsulation Encapsulate programmatic server in image, and reference associated ’out-of-the-box’ services: 1 FROM node :14 2 ... 3 RUN npm install --only=production 4 ... 5 EXPOSE 3003 6 CMD ["npm", "start"] Dockerfile 7
  13. 13. Workflow: 3. Container encapsulation Encapsulate programmatic server in image, and reference associated ’out-of-the-box’ services: 1 webapp: 2 build: web 3 ports: 4 - 3003:3003 5 6 mariadb: 7 build: web/db 8 9 proxy: 10 build: web/proxy docker-compose.yml 7
  14. 14. Workflow: 4. Remote server orchestration and deployment Create remote machine, and deploy containers: 1 docker -machine create --driver openstack 2 eval $(docker -machine env [machine -name]) 3 docker -compose build 4 docker -compose up -d 8
  15. 15. Architecture 1: Phenoflow i Allowing users to author and generate software (workflows) that can be used to automatically identify diseases in patient health records: Web Portal Generator Visualiser Implementation Units Author(s) User customise author, expand data workflow visualisation workflowworkflow, visualisation, implementation units 9
  16. 16. Architecture 1: Phenoflow ii Deployed services: 2e4557df2a35: phenoflow webapp 1 511fb466f64c: phenoflow generator 1 9ea42fbf0a2e: phenoflow mariadb 1 25a87c51195c: visualiser spring 1 fa067dd94772: visualiser mongo 1 c6bf5584a599: visualiser git-server 1 - hack to enable the visualiser to work. 5e06520d9e56: visualiser sparql 1 - Receives SPARQL queries to execute against triplestore. • Phenoflow: A Microservice Architecture for Portable Workflow-based Phenotype Definitions. https://doi.org/10.1101/2020.07.01.20144196 10
  17. 17. Architecture 2: Clinical guidelines i Allowing users to store, and identify interactions between, clinical guidelines: Interaction Interaction API (JS Server) Reasoner Reasoner API (Prolog Server) Prolog Reasoner Store Store API API Guidelines Apache Jena Fuseki 11
  18. 18. Architecture 2: Clinical guidelines ii API developed to structure interactions between services, and with end-users. Microservice Endpoint Endpoint Description Interaction /guideline/create Create a guideline set. /guideline/(add|delete) Add or delete infor- mation from a guide- line set. /guideline/[knowledge]/ (add|delete) Add or delete sup- porting guideline knowledge. /guideline/[data]+/get Get data from a guideline. /guidelines/[action] Perform a processing action on a guideline set. Reasoner /guidelines/.+ All paths with the plural word guide- lines are handled by the reasoner, as these relate to the process- ing of guidelines. Store /guideline/.+ All paths with the singular word guide- line are handled by the store, as these relate to the storage and retrieval of guide- line information. 12
  19. 19. Architecture 2: Clinical guidelines iii Deployed services: fbfd91d79c34: tmrweb webapp 1 96771158cbaa: tmrweb backend 1 4139fa64f0b8: tmrweb fuseki 1 • https://github.com/kclconsult/tmrweb 13
  20. 20. Architecture 3: CONSULT i A modular decision-support system (DSS) intended to help patients suffering from chronic conditions self-manage their treatments: Blood pressure (Nokia) Pulse and Activity (Garmin) Heart Rate / ECG (VitalPatch) EHR (EMIS) Device Integration (Nokia) Device Integration (Garmin) Device Integration (Vitalpatch) Sensor-FHIR converter EHR Integration (EMIS) EHR-FHIR converter HAPI FHIR Message Passer Dialogue Manager Authentication Server Provenance Data Miner Argumentation Engine Tablet Browser Chat Server UI backend PC Browser Sensor data Sensor data Sensor data EHR data FHIR resources FHIR resources FHIR resources Processed patient data Patient data Fragment Credentials Processed patient data Processed patient data, goal Results Dialogue responses Data summaries, tips device-web-facing ehr-web-facing internal-secure internal-comp internal-lightweight user-web-facing 14
  21. 21. Architecture 3: CONSULT ii Example deployed services: f10ac72af667: device-integration nokia proxy 1 99c9d9c6f528: device-integration nokia webapp 1 c3bca50f82d6: device-integration nokia mariadb 1 eaff2a38ea77: sensor-fhir-converter rabbit 1 98efbddefd66: sensor-fhir-converter webapp-queue 1 f48fc4aca88e: fhir-server proxy 1 84b3dcce41dd: fhir-server hapi 1 cfb0f4a79ab0: fhir-server mariadb 1 ... • https://kclhi.org/consult/demo/?a=UGU2YmFxRUQ6dWtlN2JQRXk= • https://github.com/kclconsult 15
  22. 22. Other examples • JupyterHub architecture tailored for COVID-19 analysis: https://github.com/kclhi/jupyter-docker • ’Nexus’ automated programming assessment architecture. https://api.semanticscholar.org/CorpusID:3815787 16
  23. 23. Developer experiences • The process works, and can significantly change the performance of software, its intelligibility and ease of deployment. This was especially true of the guidelines software. • There was a lack of familiarity with, and thus some resistance to, the use of microservices for development. • The notion of manually setting up a server, and manually installing things, is still very much the norm. • The fact that microservices allow for technological heterogeneity allowed people to still use the technologies they were familiar with, which was a big help in the adoption of the paradigm. 17
  24. 24. From workflow to model Explore the use of: • Container management software (Kubernetes, Swarm), capitalising on the fact that services are currently ‘scale-ready’. • Proprietary cloud services, like ’serverless’ AWS Lambda, rather than simply running everything in containers. • Different technologies, e.g. communication mechanisms such as RPC, and other languages (such as Typescript over JS). • Additional practices, such as CD for updating deployed containers. 18

×