© 2017 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Rule-based dispatching of events
to a serverless services armada
Daniel Buchholz
Dev Manager / Cloud Architect - Düsseldorf
@danbdo
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
GHX is all about supplier chain processes in healthcare
Hospitals
Groups
Suppliers
Mfr‘s
Distributors
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
What‘s was the problem, anyway?
Public Domain, https://commons.wikimedia.org/w/index.php?curid=868126
Individual catalog systems in all
(current) target markets
Each with different schema
All supposed to talk to each other
All doing a reasonably similar job
Operational efficiencies
Mappings / Conversions
Expandability
Catalog management
© 2017 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Legacy systems
this
that
More
stuff
Squeezed
in...
WHOA!
add
modify
delete
Legacy catalog processing
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
The Path Forward
consume
process
report
CSV Source type A
Source type BGS1
XML
JSON
Source type C
Suppliers
Groups
Hospitals
InboundConsumers/Processing
...
...
...
...
BI BI
...
...
...
...
...
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
The Path Forward
consume
process
report
CSV Source type A
Source type BGS1
XML
JSON
Source type C
Suppliers
Groups
Hospitals
InboundConsumers/Processing
...
...
...
...
BI BI
...
...
...
...
...
ETL
Queueing / Mapping / Routing
Single Source Of Truth
events
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Consumers/Processing
Deep dive
consume
process
report
CSV Source type A
Source type BGS1
XML
JSON
Source type C
Suppliers
Groups
Hospitals
Inbound
...
...
...
...
BI BI
...
...
...
...
...
ETL
Queueing / Mapping / Routing
Single Source Of Truth
events
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
„Each with different schema“
Catalog A
Att A: abc
Att B: def
Att C: ghi
Att D: jkl
Catalog B
Att P: 4711
Att B: blah
Att D: stuff
Catalog C
Att W: key1
Att X: key2
Att Y: foo
Att Z: bar
>> Schema-less data store
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
„Each with different schema“
Catalog A
Att A: abc
Att B: def
Att C: ghi
Att D: jkl
Pkey: hash(A)
Catalog B
Att P: 4711
Att B: blah
Att D: stuff
Pkey: hash(P)
Catalog C
Att W: key1
Att X: key2
Att Y: foo
Att Z: bar
Pkey:hash(W/X)
>> Infer internal Pkey values
>> DynamoDB
>> Schema-less data store
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
ETL
files de-dup delta
Item hash
Item
events
map
Item key
Schema
very specific very generic
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Consumers/Processing
Deep dive
consume
process
report
CSV Source type A
Source type BGS1
XML
JSON
Source type C
Suppliers
Groups
Hospitals
Inbound
...
...
...
...
BI BI
...
...
...
...
...
ETL
Queueing / Mapping / Routing
Single Source Of Truth
events
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Dispatching item events
catalog data
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Don‘t want to deploy full stack for each change
Dispatching item events
catalog data
Dispatcher
Function
„All doing a reasonably similar job“
>> Migrate tasks from legacy
>> Keep it open to changes
>> Static workflows wouldn‘t work
>> need rules engine
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Dispatcher Config
catalog data
Dispatcher
Function Dispatcher
Config Store
2 - Description
1 - Name
3 - Conditions
4 – Action(s)
success / failure
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Dispatcher Services
catalog data
Dispatcher
Function Dispatcher
Config Store
...Svc A Svc DSvc B Svc C
Search..
Engines
A Dispatcher – not a Broker >> fire & forgetHighly de-coupled
>> limit blast radius
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Dispatcher Endpoints
catalog data
Dispatcher
Function Dispatcher
Config Store
...Svc A Svc DSvc B Svc C
Search..
Engines
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Audits / Traces
v4 v3/v4 v2/v3 v1/v2 v0/v1
OPS teams need visibility >> full audit / full trace
DEV probably too
live audit
Causes & Effects
Svc A trace
Svc B trace
Consumer X
Consumer Y
Svc B trace
Svc C trace
Consumer Y
Svc A trace
BI A
BI B
Consumer X
Svc A trace
Svc B trace
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Core Services
catalog data
Dispatcher
Function
...Svc A Svc DSvc B Svc C
Search..
Engines
Audit log
Trace log
Core Services
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
Versioning/Deployment options
80/20
Svc A Svc D Svc E
Svc E2Svc A2
In your face
Svc D2
Traffic shifting Parallel / Audit
© 2018 Global Healthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute.
So...
This is our architecture
Happy to hear feedback

Rule-based dispatching of events to a serverless services armada

  • 1.
    © 2017 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Rule-based dispatching of events to a serverless services armada Daniel Buchholz Dev Manager / Cloud Architect - Düsseldorf @danbdo
  • 2.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. GHX is all about supplier chain processes in healthcare Hospitals Groups Suppliers Mfr‘s Distributors
  • 3.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. What‘s was the problem, anyway? Public Domain, https://commons.wikimedia.org/w/index.php?curid=868126 Individual catalog systems in all (current) target markets Each with different schema All supposed to talk to each other All doing a reasonably similar job Operational efficiencies Mappings / Conversions Expandability Catalog management
  • 4.
    © 2017 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Legacy systems this that More stuff Squeezed in... WHOA! add modify delete Legacy catalog processing
  • 5.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. The Path Forward consume process report CSV Source type A Source type BGS1 XML JSON Source type C Suppliers Groups Hospitals InboundConsumers/Processing ... ... ... ... BI BI ... ... ... ... ...
  • 6.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. The Path Forward consume process report CSV Source type A Source type BGS1 XML JSON Source type C Suppliers Groups Hospitals InboundConsumers/Processing ... ... ... ... BI BI ... ... ... ... ... ETL Queueing / Mapping / Routing Single Source Of Truth events
  • 7.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Consumers/Processing Deep dive consume process report CSV Source type A Source type BGS1 XML JSON Source type C Suppliers Groups Hospitals Inbound ... ... ... ... BI BI ... ... ... ... ... ETL Queueing / Mapping / Routing Single Source Of Truth events
  • 8.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. „Each with different schema“ Catalog A Att A: abc Att B: def Att C: ghi Att D: jkl Catalog B Att P: 4711 Att B: blah Att D: stuff Catalog C Att W: key1 Att X: key2 Att Y: foo Att Z: bar >> Schema-less data store
  • 9.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. „Each with different schema“ Catalog A Att A: abc Att B: def Att C: ghi Att D: jkl Pkey: hash(A) Catalog B Att P: 4711 Att B: blah Att D: stuff Pkey: hash(P) Catalog C Att W: key1 Att X: key2 Att Y: foo Att Z: bar Pkey:hash(W/X) >> Infer internal Pkey values >> DynamoDB >> Schema-less data store
  • 10.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. ETL files de-dup delta Item hash Item events map Item key Schema very specific very generic
  • 11.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Consumers/Processing Deep dive consume process report CSV Source type A Source type BGS1 XML JSON Source type C Suppliers Groups Hospitals Inbound ... ... ... ... BI BI ... ... ... ... ... ETL Queueing / Mapping / Routing Single Source Of Truth events
  • 12.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Dispatching item events catalog data
  • 13.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Don‘t want to deploy full stack for each change Dispatching item events catalog data Dispatcher Function „All doing a reasonably similar job“ >> Migrate tasks from legacy >> Keep it open to changes >> Static workflows wouldn‘t work >> need rules engine
  • 14.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Dispatcher Config catalog data Dispatcher Function Dispatcher Config Store 2 - Description 1 - Name 3 - Conditions 4 – Action(s) success / failure
  • 15.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Dispatcher Services catalog data Dispatcher Function Dispatcher Config Store ...Svc A Svc DSvc B Svc C Search.. Engines A Dispatcher – not a Broker >> fire & forgetHighly de-coupled >> limit blast radius
  • 16.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Dispatcher Endpoints catalog data Dispatcher Function Dispatcher Config Store ...Svc A Svc DSvc B Svc C Search.. Engines
  • 17.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Audits / Traces v4 v3/v4 v2/v3 v1/v2 v0/v1 OPS teams need visibility >> full audit / full trace DEV probably too live audit Causes & Effects Svc A trace Svc B trace Consumer X Consumer Y Svc B trace Svc C trace Consumer Y Svc A trace BI A BI B Consumer X Svc A trace Svc B trace
  • 18.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Core Services catalog data Dispatcher Function ...Svc A Svc DSvc B Svc C Search.. Engines Audit log Trace log Core Services
  • 19.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. Versioning/Deployment options 80/20 Svc A Svc D Svc E Svc E2Svc A2 In your face Svc D2 Traffic shifting Parallel / Audit
  • 20.
    © 2018 GlobalHealthcare Exchange, LLC. All rights reserved. GHX proprietary information. Do not copy or distribute. So... This is our architecture Happy to hear feedback

Editor's Notes

  • #2 „This is our architecture“
  • #3 Just Context
  • #4 + theoretically have to support new(ish) GLOBAL systems that use catalog data from multiple regional CMSs
  • #6 No ZigZag of files!
  • #11 Individual services
  • #13 Thought about calling him JEFF
  • #18 CorellationIDs