Building and Evolving a Dependency-Graph Based Microservice Archit...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Building and Evolving a Dependency-Graph
Based Microservice Architecture
Lars Francke – Partner and Co-Founder @ OpenCore
...

© 2019 OpenCore GmbH & Co. KG 2
About Me – Lars Francke
• Partner & Co-Founder at OpenCore
• We do Hadoop/Big Data/insert ...

The Problem
© 2019 OpenCore GmbH & Co. KG 3

© 2019 OpenCore GmbH & Co. KG 4
The problem
No one here knows
the dependencies
between all our
Microservices
anymore!
We d...

© 2019 OpenCore GmbH & Co. KG 5
The problem
Didn't the London
team already build a
service to check zip
codes?
Why has thi...

Microservice Architectures
© 2019 OpenCore GmbH & Co. KG 6

Choreography

© 2019 OpenCore GmbH & Co. KG 8
Choreography
Also (often) called
Event-Driven

© 2019 OpenCore GmbH & Co. KG 9
Choreography
• Services coordinate amongst themselves
• No central service
• "Smart endpoi...

© 2019 OpenCore GmbH & Co. KG 10
Choreography
• Hard to keep track of everything & get an overview
• Harder to verify at b...

© 2019 OpenCore GmbH & Co. KG 11
Choreography
When I said no central service
What I meant was that we obviously
still do h...

Orchestration
Source:
https://commons.wikimedia.org/wiki/File:Peter_Oundjian_-
_Conductor_of_Toronto_Symphony_Orchestra_20...

© 2019 OpenCore GmbH & Co. KG 13
Orchestration
• One central "coordinator" that tells everyone what to do
• Like a conduct...

© 2019 OpenCore GmbH & Co. KG 14
Orchestration
• Less flexible
• Adding a new service requires coordination and potentiall...

© 2019 OpenCore GmbH & Co. KG 15
Orchestography
Natural question to ask:
Which is better?

© 2019 OpenCore GmbH & Co. KG 16
Orchestography

© 2019 OpenCore GmbH & Co. KG 17
Orchestography
Both have their uses!

© 2019 OpenCore GmbH & Co. KG 18
Microservices
• Microservices are often used to split up a single monolithic app into
mul...

© 2019 OpenCore GmbH & Co. KG 19
Orchestography
• Orchestration lends itself more to "workflow" oriented tasks which are s...

© 2019 OpenCore GmbH & Co. KG 20
Orchestography
Application 1
(Orchestrated)
Kafka
(or similar,
for Choreography)
Applicat...

© 2019 OpenCore GmbH & Co. KG 21
Example

Naming things is hard
© 2019 OpenCore GmbH & Co. KG 22

© 2019 OpenCore GmbH & Co. KG 23
Cattle vs. Pets
Show of hands
Who here has (had) servers with names like:
Sources: https:...

© 2019 OpenCore GmbH & Co. KG 24
Pets
Names like that are a good sign that these servers might be your Pets
They often hav...

© 2019 OpenCore GmbH & Co. KG 25
Cattle
The industry has moved on (or is in the process) to treating Servers (and
services...

© 2019 OpenCore GmbH & Co. KG 26
Cattle
• The Cloud was a big "enabler" for this movement
• Servers have more or less rand...

© 2019 OpenCore GmbH & Co. KG 27
Cattle
If we agree that this is a good thing…
…why do you have a topic called
customerCre...

Technology to the rescue
Lars, tell us what to do!
© 2019 OpenCore GmbH & Co. KG 28

© 2019 OpenCore GmbH & Co. KG 29
L
We are not the first to struggle with this
Surprising, I know

© 2019 OpenCore GmbH & Co. KG 30
Good Compan(y|ies)
Source: https://commons.wikimedia.org/wiki/File:ING_Group_N.V._Logo.svg

© 2019 OpenCore GmbH & Co. KG 31
Netflix Conductor
• Netflix OSS Project: https://github.com/Netflix/conductor
• "Conducto...

© 2019 OpenCore GmbH & Co. KG 32
Netflix Conductor
• Workflows consist of Tasks
• Conductor itself can store some Payload ...

$© 2019 OpenCore GmbH & Co. KG 33 Netflix Conductor – Tasks [ { "name": "verify_if_idents_are_added", "retryCount": 3, "ret...$

$© 2019 OpenCore GmbH & Co. KG 34 Netflix Conductor – Workflow Pt. 1 { "name": "add_netflix_identation", "description": "Ad...$

© 2019 OpenCore GmbH & Co. KG 35
Netflix Conductor – Workflow Pt. 2
"type": "DECISION",
"caseValueParam": "case_value_para...

© 2019 OpenCore GmbH & Co. KG 36
Uber Cadence
• Uber Project: https://github.com/uber/cadence
• "Cadence is a distributed,...

© 2019 OpenCore GmbH & Co. KG 37
Uber Cadence
• Cadence handles Task state & Queues for us
• Your Workflow is implemented ...

© 2019 OpenCore GmbH & Co. KG 38
Uber Cadence – Example
@Override public void execute(String customerId) {
activities.send...

© 2019 OpenCore GmbH & Co. KG 39
Expedia Stream Registry
• Expedia project (originally HomeAway):
https://github.com/Exped...

© 2019 OpenCore GmbH & Co. KG 40
Expedia Stream Registry
• It manages Clusters as well as "Streams" of data
• Including sc...

© 2019 OpenCore GmbH & Co. KG 41
Others
There are others:
• ING Baker
• ING Project: https://github.com/ing-bank/baker
• "...

Where does this leave us?
© 2019 OpenCore GmbH & Co. KG 42

© 2019 OpenCore GmbH & Co. KG 43
The Current State
• Most existing tools require you to explicitly model your dependency g...

© 2019 OpenCore GmbH & Co. KG 44
The Current State
• Unfortunately, the perfect solution doesn't yet exist
• The Orchestra...

© 2019 OpenCore GmbH & Co. KG 45
Does this seem familiar?

© 2019 OpenCore GmbH & Co. KG 46
Wishlist
• We need better support for Event-Driven (Choreography) style
architectures
• W...

© 2019 OpenCore GmbH & Co. KG 47
Wishlist
• We'd like a tool that
• allows us to register logical streams of data,
• Used ...

© 2019 OpenCore GmbH & Co. KG 48
Wishlist
• This tool could use this information to
• automatically build an optimal DAG,
...

© 2019 OpenCore GmbH & Co. KG 49
Wishlist
• For those who use Apache Spark:
• In Spark you define all your actions and tra...

© 2019 OpenCore GmbH & Co. KG 50
Example

© 2019 OpenCore GmbH & Co. KG 51
Example
Service A Service BTopic "xqdrnc"

© 2019 OpenCore GmbH & Co. KG 52
Example
Service A Service BTopic "xqdrnc"
Infra
Service
if (booking.travel_agency == "Tho...

$© 2019 OpenCore GmbH & Co. KG 53 Example Service A Service BTopic "xqdrnc" if (booking.travel_agency == "Thomas Cook") { f...$

$© 2019 OpenCore GmbH & Co. KG 54 Example if (booking.travel_agency == "Thomas Cook") { fail() } Service A Service BTopic "...$

© 2019 OpenCore GmbH & Co. KG 55
Wishlist
• This tool could also (optionally) automatically run or re-run the services
usi...

© 2019 OpenCore GmbH & Co. KG 56
Wishlist
• A side effect would be an automatically up-to-date Governance/Data
Catalog
• T...

© 2019 OpenCore GmbH & Co. KG 57
Orchestography
Service 1
(Orchestrated)
Kafka
(or similar,
for Choreography)
Service 3
(O...

Questions
What are your questions?
lars.francke@opencore.com
@lars_francke
© 2019 OpenCore GmbH & Co. KG 58