SlideShare a Scribd company logo
What 5 years of implementation
microservices has taught me
Jeppe Cramon - @jeppec
Cloud Create ApS
What 15 year of implementation
distributed systems has taught
me
Jeppe Cramon - @jeppec
Cloud Create ApS
@jeppec
@jeppec
Thanks
Goodbye 
@jeppec
What patterns have
stuck with me?
@jeppec
Let’s go back to 2005
When Internet Explorer 6 ruled the world
@jeppec
And portals were
@jeppec
https://our.cool.bank-portal
Customer 🧩
Name: Peter Hansen
Address: Some Road 1
Some Zip Some City
…
Customer relations 🧩
Spouse: Mette Hansen
Children: Jacob Hansen
Ditte Hansen
Accounts 🧩
Account Number Account Name Balance
9128-21892982981 Salary Account 214,59
9128-89812981189 Budget Account -345,75
9128-12217698156 Savings Account 17.230,15
Account details: 9128-21892982981 🧩
Account Name:
Interest rate: 3% 🔧
Salary Account
@jeppec
Learnings
The good
• Smaller dedicated UI pieces allow many developers to work independently
on the same portal page
• Small views only used by a single portlet was easy to change as it has fewer
couplings
• Reactive UI was cool. Click a part in a portlet and through a UI event one or
more portlets would react and change their content. Decoupling
The bad
• Portlet technology was very heavy and very inflexible
@jeppec
SOA for For The Win
Build tiny services that could be combined to larger services
@jeppec
Data
Storage
Data Storage
Data
Storage
Data
Service
Portal 1 Portal 2
Data
Storage
Application
Process Service layer
Activity/Task Service layer
Data/Entity Service layer
Layered SOA
@jeppec
SOAP queries anyone?
@jeppec
Resemble much?
@jeppec
Ensuring
consistency
Was even harder
@jeppec
Sales system
Sale
Delivery
system Deliveries
Customer/CR
M system Customer
SAP Bookkeeping
Complete
Purchase
Transaction
Coordinator
Transactional
Resource
Prepare
Phase
Commit
Phase
2 Phase Commit
@jeppec
What’s wrong with distributed transactions?
• Transactions lock resources while active
• Services are autonomous
• Can’t be expected to finish within a certain time interval
• Locking keeps other transactions from completing their job
• Locking doesn’t scale
• X Phase Commit is fragile by design
@jeppec
Essential complexity of 2 way integration
Component CComponent BComponent A
UI
Service Service
B:Service()
call C:Service()
call A:Service()
commit()
Service
Local transaction between
Component A, B and C
@jeppec
What’s the challenge with using
RPC/REST/SOAP/…
between
distributed components?
@jeppec
Synchronous calls lower our tolerance for faults
• When you get an IO error
• When servers crash or restarts
• When databases are down
• When deadlocks occurs in our databases
• Do you retry?
With synchronous style Service interaction we can loose business data if there’s no automatic retry
Or we risk creating data more than once if the operation isn’t idempotent*
Client Server
Duplicated Response
Duplicated Request
Processing
Response
Request Processing
The same message can be
processed more than once
*Idempotence describes the quality of an operation
in which result and state does not change if the
operation is performed more than 1 time
@jeppec
B:Service()
call C:Service()
call A:Service()
if (A:Call-Failed:Too-Busy?)
Wait-A-While()
call A:Service()
if (A:Call-Failed:Too-Busy?)
Wait-A-Little-While-Longer()
call A:Service()
if (A:Call-Failed:IO-Error?)
Save-We-Need-Check-If-Call-A-Succeded-After-All
AND We-Need-To-Retry call C:Service and call B:Service
AND Tell-Customer-That-This-Operation-Perhaps-Went-Well
if (A:Call-Went-Well?)
commit()
Accidental complexity from distributed service integration
Component CComponent BSystem A
UI
Service Service Service
Local transaction between
Component B and C
@jeppec
Service autonomy
Component
B
Component
C
Component
A
System X
Service A
Component
B
Component
C
System X
Slow/unreliable network
Different SLA
Slow system
@jeppec
Learnings
The bad
• Task/Activity services needs to perform UPDATES across multiple data/entity
services. Requires distributed transactions
• The more synchronous request-response remote calls you have to make the
more it hurts Performance.
• Robustness is lower. If one data/entity services is down it can take down
many other services.
• Coupling is higher. Multiple task/activity services manage the same CRUD
data/entity services
• Cohesion is likely lower as multiple task/activity services need to replicate
entity related logic
@jeppec
Conclusion
Friends don’t let friends use layered SOA
@jeppec
Skip forward a little later
To when Enterprise Service Busses were king
@jeppec
Services != Decoupling
@jeppec
Availability goes down
(without additional instances of each service)
Combined availability: 97%
@jeppec
Learnings
Good
• Very early in the project a full end to end correlation logging framework and
application was built, so you knew exactly which services were being called
by whom, where the time was spent
Bad
• When it takes > 10 min to perform the primary use case you have a serious
performance overhead
• When you have circular Service calls you have a big boundary issue
• When a single service changes its interface and it takes days to adjust other
services before everything works again, then you’re far from decoupled
@jeppec
ESB’s the the Rescue?
SOA Web of Mess hidden by ESB
Inventory
Purchase
ShippingPortal
ESB
@jeppec
ESB Reality
SOA Web of Mess hidden by ESB
Inventory
Purchase
ShippingPortal
ESB
@jeppec
ESB Learnings
• Very often the ESB’s only role is to decouple teams
• You’re effectively introducing a middle management layer with a thin promise
of increased security, monitoring and decoupling
• Often most services exposed are 1-1 with the original interface. Any
decoupling is imaginary and the ESB is a bottleneck.
• Or even worse they follow the layered SOA approach
• Monitoring and security is an end-to-end concern
• Putting some magic technology in the middle doesn’t solve the real problem
@jeppec
Forward to 2009 and our first
microservices
A customer facing self service portal
@jeppec
https://our.micro-cool.bank-portal
Customer
Name: Peter Hansen
Address: Some Road 1
Some Zip Some City
Customer relations
Spouse: Mette Hansen
Children: Jacob Hansen
Ditte Hansen
Account Number Account Name Balance
9128-21892982981 Salary Account 214,59
9128-89812981189 Budget Account -345,75
9128-12217698156 Savings Account 17.230,15
Account details: 9128-21892982981
Account Name:
Interest rate: 3% 🔧
Salary Account
@jeppec
https://our.micro-cool.bank-portal
Customer
Name: Peter Hansen
Address: Some Road 1
Some Zip Some City
Customer relations
Spouse: Mette Hansen
Children: Jacob Hansen
Ditte Hansen
Overview Page
(owned by the
portal
application)
<<UI Component>>
Customer Details Component
<<UI Component>>
Customer Relations
Component
UI layer
Backend layer
Customer Details
Microservice
Customer Relations
Microservice
HTTPS HTTPS
@jeppec
An UI is composed of
widgets from
different services
34
@jeppec
Widget Widget
Page
Widget
Widget
Widget Widget
• Overall structure of the page
is “owned” by the application.
• Each widget is independent
Composite page layout
Widget Widget Widget
@jeppec
Learnings
The good:
• Each section on the page is componentized.
• The page knows NOTHING about how information is fetched
• Very low horizontal coupling
• Each component is responsible for ensuring availability and stability.
You build it, you run it, you fix it 
• Individual component/service failure doesn’t bring down the entire
page
The bad:
• The microservices tended to be overly small and overlapped ➡️ code
duplication and shared databases
@jeppec
Master Data Management
example using Events
2011-2012
@jeppec
System A
System
B
System
C
System
D
4 systems with shared data and no clear master for data
No Unique
Customer Profile
@jeppec
Login
Contact
Information
Memberships
CustomerMaster
System A
System
B
System
C
System
D
New solution
@jeppec
One way Master Data duplication
System A Master
How to integrate?
@jeppec
One way Master Data duplication
System A Master
Query
Query result
1. Read Ids of all Entities that match criteria
2. For each Id load the Entity
3. Result: 1+n problem
@jeppec
@jeppec
One way Data duplication using Events
System A Master
Events
@jeppec
System A
Update Customer
Contact Info
Update
Customer
ContactInfo
Process
”UpdateCustomerContactInfo”
command
Event Publisher
”CustomerContactInfoUpdated” Event
Customer View
Customer
local/cached replica
System D
BI/And other parties
System B
Customer Master
@jeppec
Back-end Data
System A System D
OSB
Private Customer
Service Proxy
ESB Event
Receiver
CustomerCore
Private Customer
Service
Event
Queue
Event Topic
Command
Internal Event
External Event
External
PrivateCustomer
EventReceiver
Internal
PrivateCustomer
EventReceiverInternal Services
System A System D
Event Topic
With SSNWithout SSN ?
Contains:
Social Security Number (SSN), Who’s
allowed to see SSN and who the Event
is relevant for
External Services
Security control
@jeppec
Learnings
The Good:
• We were able to allow all the external source systems to continue with their existing domain
models
• We gained a centralized overview of all customers and their memberships
• The external systems rarely needed to contact the Customer Service
• Breakdowns in communication paths or systems didn’t bring down all other systems
• Use of Optimistic Concurrency combined with eventsourcing made for easy coordination of (rare)
out of sync updates
• CQRS proved to be a good fit for an event sourced solution
• Resilience and stability
The Bad:
• Use of ESB again proved to be a big time waster
• Full resilience, e.g. during Queue/Topic data loss or coding errors in external systems, required
additional Reconciliation service operations (to fetch missing events)
@jeppec
The final piece of the puzzle
The Domain
@jeppec
"I consider 'getting the boundaries right'
the single design decision with the most
significant impact over the entire life of a
software project."
@ziobrando
@jeppec
Monoliths can be well designed
Unfortunately many monoliths experience high degrees of coupling, resulting in them
earning a bad name
@jeppec
The best coupling?
The one you avoid
@jeppec 51
@jeppec
Focus on identifying
52
Business Capability and
Bounded Contexts are
in many ways similar
• What we want to achieve is
• problem domain and
• solution domain
• ALIGNMENT
@jeppec
Business Capability alignment
“The advantage of business capabilities is their remarkable level of
stability. If we take a typical insurance organisation, it will likely
have sales, marketing, policy administration, claims management,
risk assessment, billing, payments, customer service, human
resource management, rate management, document
management, channel management, commissions management,
compliance, IT support and human task management capabilities.
In fact, any insurance organisation will very likely have many of
these capabilities.”
See http://bill-poole.blogspot.dk/2008/07/business-capabilities.html
@jeppec
Business – IT alignment
• We want the Business and IT to speak the same Ubiquitous language
• Want want our architecture to be aligned with the business capabilities
• Because these capabilities are stable
@jeppec
Many perspectives on data
Online Retail System
Product
Unit Price
Promotional Price
Promotion End Date
Stock Keeping Unit (SKU)
Quantity On Hand (QOH)
Location Code
Price
Quantity Ordered
Name
The lifecycle of the data is VERY important!
Customer
Pricing
Inventory
Sales
Management Reporting
@jeppec
Smaller models & clear data ownership
Retail System
Pricing
Product
ProductID
Unit Price
Promotional Price
…
Pricing
Inventory
Product
ProductID
SKU
QOH
Location Code
…
Inventory
Sales
Product
ProductID
Name
Description
Quantity Ordered
…
Sales
Shared Entity identity
DDD:
Bounded
Context
Business
Capability
@jeppec
Bounded Contexts and Aggregates
Sales
Product
Customer
customerId
…
Order
orderId
customerId
…
OrderLine
orderId
productId
quantity
timestamp
…
ProductCategory
productCategoryId
…
Pricing
Product
productId
productCategoryId
name
tag
...
Product-Price
productId
normalPrice
discountPeriods
…
@jeppec
Using Business Events to drive Business Processes
Sales Service
Shipping
Billing
Sales
Customers
MessageChannel
Online Ordering System
Web Shop
(Composite UI)
Billing Service
Shipping Service
<<External>>
Order Accepted
AcceptOrder
The sales
fulfillment
processing can
now begin…
Cmd Handler
Order Accepted
Apply
@jeppec
Choreographed Event Driven Processes
Sales Service
Order
Accepted
Billing Service
Order Fulfilment
(Saga/
Process-Manager)
Shipping Service
Online Ordering System
MessageChannel(e.g.aTopic)
Order
Accepted
Order
Accepted
Customer
Billed
Customer
Billed
Order
Approved
Order
Approved
Works as a Finite
State Machine
(WorkFlow)
handling the life
cycle of Shipping and
thereby forms a very
central new
Aggregate in the
System
@jeppec
If we align the problem domain with the solution domain
Bounded Context 1 Bounded Context 3Bounded Context 2
UI
BL
DAO
UI
BL
DAO
UI
BL
DAO
Vertical coupling
is
unavoidable
We want to avoid
horizontal coupling
@jeppec
Internal vs External Events
{
EventType: "OrderWasAccepted",
CustomerId: "50D1F244-ABBC-4EC7-BDCA-E4934C124A89",
OrderId: "C199322A-01F1-4E56-918E-7A63529F8FA3",
Items: [
{
ProductId: "4CD22C4B-600C-4477-B5BF-48ABDEE4DA61",
Amount: "100",
AmountUnit: "Pieces"
},
{
ProductId: "56E6BD19-660C-464A-9120-100DAF579855",
Amount: "10",
AmountUnit: "Litres"
}
]
}
Internal Event:
{
EventType: "OrderWasAccepted",
OrderId: "C199322A-01F1-4E56-918E-7A63529F8FA3"
}
External Event:
@jeppec 63
@jeppec
A Service is
• The technical authority for a given bounded context/business-capability
• It is the owner of all the data and business rules that support this
bounded context – everywhere
• It forms a single source of truth for that bounded context
http://udidahan.com/2010/11/15/the-known-unknowns-of-soa/
@jeppec 65
@jeppec
Microservices promise a solution
to our problem
Monolith
Microservice Microservice
Microservice MicroserviceMicroservice
Microservice
Microservice
@jeppec
There’s never time to do it right
But there’s always time to do it over
67
@jeppec 68
@jeppec
Service and deployment
• A Service represents a logical responsibility boundary
• Logical responsibility and physical deployment of a Service
DOES NOT have to be 1-to-1
• It’s too constraining
• We need more degrees of freedom
• Philippe Krutchen 4+1 views of architecture: Logical and Physical
designs should be independent of each other
A service needs to be deployed everywhere its data is needed
@jeppec
@jeppec
Service Microservices
1..*
Is implemented by
A Service is the technical authority of a specific
Bounded-Context/Business Capability
e.g. Sales, Shipping, Billing
Service vs Microservices
@jeppec
Service Microservices
1..*
Is implemented by
Service vs Microservices
Microservices are a division of Services along Transactional boundaries (a transaction stays within the
boundary of a Microservice)
Microservices are the individual deployable units of a Service with their own Endpoints. Could e.g. be the
split between Read and Write models (CQRS) - each would be their own Microservices
@jeppec
A Service represents a logical boundary
73
Service
Microservice
Microservice Microservice
Microservice
@jeppec
Services are the corner stone
• We talk in terms of Services capabilities and the processes/use-cases
they support
• Microservices are an implementation detail
• They are much less stable (which is a good thing – it means they’re easier to
replace)
@jeppec
There’s cost in deploying 1000’s of microservices
(or 100.000’s serverless functions)
75
@jeppec
Autonomous Components
Not everything needs to be individually deployable
76
@jeppec
This means they CAN, but they don’t HAVE to be deployed
individually.
Design for Distribution
But take advantage of locality
@jeppec
Let’s be even more pragmatic
In which case we allow other services
to call them using local calls
78
@jeppec
AC in code
public class PricingEngineAc extends AutonomousComponent {
public static AutonomousComponentId SERVICE_AC_ID = SALES_SERVICE_ID.ac(”pricing_engine_ac");
…
public PricingEngineAc(CurrencyConverter currencyConverter) {
this.currencyConverter = currencyConverter;
}
@Override
public void onInitialize(IConfigureACEnvironment acSetup) {
acSetup.withAutonomousComponentId(SERVICE_AC_ID)
.usingServiceDataSource()
.withBusConfiguration(cfg -> {
bus.registerAxonReplayableTopicPublisher(InternalPricingEvents.TOPIC_NAME,
replayFromAggregate(Pricing.class)
.dispatchAggregateEventsOfType(InternalPricingEvents.class));
bus.subscribeTopic(SERVICE_AC_ID.topicSubscriber(”InventoryEvents"),
ExternalInventoryEvents.TOPIC_NAME,
new InventoryTopicSubscription(bus));
})
.runOnBusStartup((bus, axonContext) -> {
});
}
}
@jeppec
Service deployment
• Many services can be deployed to the same physical server
• Many services can be deployed in the same application
• Application boundary is a Process boundary which is a physical boundary
• A Service is a logical boundary
• Service deployment is not restricted to tiers either
• Part of service A and B can be deployed to the Web tier
• Another part of Service A and B can be deployed to the backend/app-service tier of the same
application
• The same service can be deployed to multiple tiers / multiple applications
• ie. applications and services are not the same and does not share the same boundaries
• Multiple services can be “deployed” to the same UI page (service mashup)
@jeppec
An Application is the plate where Components are
co-deployed
81
Sales service components
Inventory service components
…
@jeppec
iOS Home Banking
Customer information
Legal and contract information
Accounts
@jeppec
iOS Home banking
Customer information
Legal and contract information
Accounts
Credit card
Mortgage loans
Web banking portal
Bank Back-office application
@jeppec
Service A Service B Service C
IT-OPS
Web shop
Warehouse
Back office
App logic
(Layered, CQRS,…)
Warehouse
Storage
UI Components
A Services owns its UI Components
@jeppec 85
@jeppec 86
@jeppec
Composite page example
Page Context:
{id: ISBN-10 0-321-83457-7 }
Images
Books
Reviews
Pricing
Inventory
OthersAlsoBought
Pricing
Reviews
Books
Images
Books
@jeppec
A Service represents a logical boundary
88
Service
Autonomous
Component
Autonomous
Component
Adapter
UI component
@jeppec
Autonomous Component
• Can be deployed alone or co-located, together with one or more adapters from the same service
• Works transparently in a clustered environment
Core logic
(use case controllers/aggregates)
Primary/Driving Adapter
GUI / API / …
Primary/Driving Port
Secondary/Driven Adapter
Database/Notifications/…
Secondary/Driven Port
@jeppec
Autonomous Components can be co-deployed together
with Application backends
web_shop (Spring Boot fat–jar)
pricing_engine_ac
discounts_ac
webshop_basket_adapters
api_products
product_details_ac
product_details_adapters
frontend
api_pricing
app
libs
products
pricing
branding
@jeppec
Application in code
@Configuration
@ComponentScan(basePackages = { "com.mycoolshop.webshop",
"com.mycoolshop.adapters",
"com.mycoolshop.itops.spring" })
public class Application extends IAmASpringBootApplication {
@Override
protected ApplicationIdentifier getApplicationIdentifier() {
return ApplicationIdentifier.from(”WebShop");
}
@Override
protected Collection<AutonomousComponent> getAutonomousComponentsHostedInThisApplication() {
CurrencyExchangeRateAc currencyExchangeRateAc = new CurrencyExchangeRateAc();
return list(
new PricingEngineAc(currencyExchangeRateAc.getCurrencyConverter()),
new DiscountsAc(currencyExchangeRateAc.getCurrencyConverter()),
new InventoryAc(),
new CustomersAc(),
currencyExchangeRateAc
);
}
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
@jeppec 92
@jeppec
Autonomous Component
AC - 1
AC - 1AC - 1
AC - 1
AC - 2
AC - 2
AC - 2
AC - 2
AC - 3
AC - 3
AC - 3
AC - 3
AC - 4
AC - 4
AC - 4
AC - 4
Federated
Bus
@jeppec
pricing_engine_ac
(deployed on 10.25.26.102)
pricing_engine_ac
(deployed on 10.25.26.101)
Bus Bus
Bus Bus
inventory_ac
(deployed on 10.25.26.104)
inventory_ac
(deployed on 10.25.26.103)
Federated Bus
@jeppec
Topics
Bus features
AC - 4
AC - 1
AC - 3
bus.registerAxonReplayableTopicPublisher(InternalPricingEvents.TOPIC_NAME,
replayFromAggregate(Pricing.class)
.dispatchAggregateEventsOfType(
InternalPricingEvents.class
)
);
bus.subscribeTopic(SERVICE_AC_ID.topicSubscriber(”Pricing"),
InternalPricingEvents.TOPIC_NAME,
new PricingTopicSubscription(bus));
@jeppec
Topics
public class PricingTopicSubscription extends BusMessageHandlerDelegator {
@BusMessagePayloadHandler
private void on(PrincingChanged e, BusMessage<PricingChanged> msg) {
…
}
…
}
@jeppec
Topics
97
CreateOrder
Sales_Service:Orders_Ac
Webshop Application
Customer_Service:Customer_Agreements_Ac
OrderCreated
OrderCreatedEvent
Publish
CreateCustomerAgreement
Sales_Service:OrderEvents
<<Topic>>
OrderCreatedEvent
@jeppec
Client handled subscriptions
• Highly resilient pattern for an Event Driven Architecture that’s backed by
Event-Sourced AC’s
• In this model the publisher of the Events is responsible for the durability of
all its Events, typically to an EventStore/EventLog.
• Each client (subscriber) maintains durable information of the last event it has
received from each publisher.
• When ever the client starts up it makes a subscription to the publisher
where it states from which point in time it wants events published/streamed
to it.
• This effectively means that publisher can remain simple and the client
(subscriber) can remain simple and we don’t need additional sophisticated
broker infrastructure such as Kafka+ZooKeeper.
@jeppec
Client handled subscriptions
Publisher
Subscriber A
Local storage
EventStore
Subscriber B
Local storage
Topic
Subscription
Topic
Subscription
TopicSubscriptionHandler
TopicSubscriptionHandler
EventEvent
Event Event
EventBus
Event
Event
Distributed Event Bus,
which ensures that
live events published
on an AC node in the
cluster can be seen
by all AC’s of the
same type
Singe Instance
Subscriber, which
ensures that only
one instance of
Subscriber B has
an active
subscription(s).
Other instances of
the same
subscriber are
hot-standby
<<Topic Subscriber>>
Customer_Service:Customer_Agreements_Ac:OrderEvents
<<Topic Publisher>>
Sales_Service:OrderEvents
@jeppec
Topics
Bus features
Features:
• The Bus provides automatic and durable handling of Redelivery in case of message handling failure
through a Redelivery Policy
• Exponential Backoff
• Max retries
• Dead letter/Error Queue
• Support for resubscription at any point in the timeline of an Event Stream
• Automatically tracking of resubscription points - aka. resubscribe at last tracked point
AC - 4
AC - 1
AC - 3
@jeppec
Bus features
Notifications:
• Durable notifications with failover support
• bus.notify(notificationQueue, notificationMessage)
Notifications
Distributed
Broadcast
Broadcast:
• Broadcast a Message to all AC’s in the cluster
• Broadcast a Message to a UI client (all, per user, per privilege)
@jeppec
Bus features
Features:
• Support for sending a message to a single consumer
• Default pattern is Competing Consumers
• The bus provides durability for all messages send on a Queue
• The Bus provider automatic and durable handling of Redelivery in case of message handling failure
through a Redelivery Policy
• Exponential Backoff
• Max retries
• Dead letter/Error Queue
Durable Queues
@jeppec
At least once delivery
In a distributed systems message delivery can and will fail!
Therefore, everything that can handle messages are built with
idempotency in mind:
• We always check for the existence of an Aggregate before creating it
• All business methods in an aggregate check if the operation/side effect
has already taken place
• View Repositories support automatic reordering of messages that
come out of order due to redelivery(if this is required by the View)
@jeppec
Bus features
Single Instance Task:
• Ensures that only one Active instance of a Task is active in the cluster at one time
• Other tasks of the same type are in hot standby
• Used to e.g. group multiple subscribers, to ensure that all subscribers are either all
active or standby.
• Used by our ViewRepositories
Distributed
SingleInstanceTask
bus.createClusterSingleInstanceTask(”MyTask",
new MyTask ()); // Where MyTask implements Lifecycle
@jeppec
Bus features
Process Manager:
• Defines durable business processes
as a flow of Events
Process Manager
Sales Service
Order
Accepted
Billing Service
Order Fulfilment
(Saga/
Process-Manager)
Shipping Service
MessageChannel(e.g.aTopic)
Order
Accepted
Order
Accepted
Customer
Billed
Customer
Billed
Order
Approved
Order
Approved
@jeppec
Application features
Workflow Manager:
• Defines the Tasks that required human intervention, such as:
• Approvals (e.g. Contract approval)
• Assistance/Help with a business problem
• Incident handling (e.g. a technical problem identified by a developer)
• Authorization (e.g. request manager approval)
• Reminders
• Common tasks supported: Claiming Tasks, Escalating Tasks, Completing Tasks
Workflow Manager
@jeppec
Application features
Identity Management:
• Common handling of Users, Privileges, Roles
• Automatic enforcement of privilege requirements
• Support for Multitenancy (Contractual level, Service Agreement Level)
Identity Management
@jeppec
Correlation logging
Our infrastructure automatically captures information about an API call,
a Message delivery in the form of a CallContext:
• Message Id - the Id of the message/call being handled.
• Correlation Id – an Id that binds API calls and message handlings across
multiple services/AC’s together
• When – time of the “call”
• Who – which user performed the “call”
• Meta Data: Which Event or Command caused this “call”
@jeppec
Correlation logging
Source: https://docs.particular.net/serviceinsight/#flow-diagram
Thanks :)
Blog: https://cramonblog.wordpress.com/
Homepage: http://cloudcreate.dk/
Twitter: @jeppec

More Related Content

What's hot

SOA Pattern Event Driven Messaging
SOA Pattern Event Driven MessagingSOA Pattern Event Driven Messaging
SOA Pattern Event Driven Messaging
WSO2
 
EventDrivenArchitecture
EventDrivenArchitectureEventDrivenArchitecture
EventDrivenArchitecture
Hiroshi Ono
 

What's hot (13)

REST != WebAPI
REST != WebAPIREST != WebAPI
REST != WebAPI
 
SOA Pattern Event Driven Messaging
SOA Pattern Event Driven MessagingSOA Pattern Event Driven Messaging
SOA Pattern Event Driven Messaging
 
Designing DDD Aggregates
Designing DDD AggregatesDesigning DDD Aggregates
Designing DDD Aggregates
 
Implementing DDD with C#
Implementing DDD with C#Implementing DDD with C#
Implementing DDD with C#
 
Event-Driven Service-oriented Architecture (EDSOA)
Event-Driven Service-oriented Architecture (EDSOA)Event-Driven Service-oriented Architecture (EDSOA)
Event-Driven Service-oriented Architecture (EDSOA)
 
Mulesoft torronto meetup_16
Mulesoft torronto meetup_16Mulesoft torronto meetup_16
Mulesoft torronto meetup_16
 
DDD Basics - Context mapping
DDD Basics - Context mappingDDD Basics - Context mapping
DDD Basics - Context mapping
 
Event sourcing
Event sourcingEvent sourcing
Event sourcing
 
Architecting for Change: An Agile Approach
Architecting for Change: An Agile ApproachArchitecting for Change: An Agile Approach
Architecting for Change: An Agile Approach
 
Cqrs api v2
Cqrs api v2Cqrs api v2
Cqrs api v2
 
EventDrivenArchitecture
EventDrivenArchitectureEventDrivenArchitecture
EventDrivenArchitecture
 
Application Architecture
Application ArchitectureApplication Architecture
Application Architecture
 
CQRS recipes or how to cook your architecture
CQRS recipes or how to cook your architectureCQRS recipes or how to cook your architecture
CQRS recipes or how to cook your architecture
 

Similar to What 5 years of implementation microservices has taught me

AppComputing_BudgetEngine_vs_Excel_Spreadsheet
AppComputing_BudgetEngine_vs_Excel_SpreadsheetAppComputing_BudgetEngine_vs_Excel_Spreadsheet
AppComputing_BudgetEngine_vs_Excel_Spreadsheet
jackyu8888
 
Enablement Session - IBP Transformation Final_C.pptx
Enablement Session - IBP Transformation Final_C.pptxEnablement Session - IBP Transformation Final_C.pptx
Enablement Session - IBP Transformation Final_C.pptx
Kaustubh M
 
ECM Renovation Roadshow - Welcome
ECM Renovation Roadshow - WelcomeECM Renovation Roadshow - Welcome
ECM Renovation Roadshow - Welcome
Zia Consulting
 

Similar to What 5 years of implementation microservices has taught me (20)

Updated: Should you be using an Event Driven Architecture
Updated: Should you be using an Event Driven ArchitectureUpdated: Should you be using an Event Driven Architecture
Updated: Should you be using an Event Driven Architecture
 
AppComputing_BudgetEngine_vs_Excel_Spreadsheet
AppComputing_BudgetEngine_vs_Excel_SpreadsheetAppComputing_BudgetEngine_vs_Excel_Spreadsheet
AppComputing_BudgetEngine_vs_Excel_Spreadsheet
 
NYLI & The Cloud: Leveraging SaaS for Disaster Recovery, Cost Savings, and An...
NYLI & The Cloud: Leveraging SaaS for Disaster Recovery, Cost Savings, and An...NYLI & The Cloud: Leveraging SaaS for Disaster Recovery, Cost Savings, and An...
NYLI & The Cloud: Leveraging SaaS for Disaster Recovery, Cost Savings, and An...
 
MSP Development Theme
MSP Development ThemeMSP Development Theme
MSP Development Theme
 
Denver Salesforce Developer User Group dec 2016 lightning components
Denver Salesforce Developer User Group dec 2016 lightning componentsDenver Salesforce Developer User Group dec 2016 lightning components
Denver Salesforce Developer User Group dec 2016 lightning components
 
Get the Message Across: Seamlessly Transport Data to Apps, Anywhere
Get the Message Across: Seamlessly Transport Data to Apps, AnywhereGet the Message Across: Seamlessly Transport Data to Apps, Anywhere
Get the Message Across: Seamlessly Transport Data to Apps, Anywhere
 
Data Microservices with Spring Cloud
Data Microservices with Spring CloudData Microservices with Spring Cloud
Data Microservices with Spring Cloud
 
Relational data modeling trends for transactional applications
Relational data modeling trends for transactional applicationsRelational data modeling trends for transactional applications
Relational data modeling trends for transactional applications
 
RahulParwekar_Resume
RahulParwekar_ResumeRahulParwekar_Resume
RahulParwekar_Resume
 
Sencha and Spring (Spring 2GX 2013)
Sencha and Spring (Spring 2GX 2013) Sencha and Spring (Spring 2GX 2013)
Sencha and Spring (Spring 2GX 2013)
 
Sunitha_Resume
Sunitha_ResumeSunitha_Resume
Sunitha_Resume
 
Enablement Session - IBP Transformation Final_C.pptx
Enablement Session - IBP Transformation Final_C.pptxEnablement Session - IBP Transformation Final_C.pptx
Enablement Session - IBP Transformation Final_C.pptx
 
Rexx Shih
Rexx ShihRexx Shih
Rexx Shih
 
Hostel management
Hostel managementHostel management
Hostel management
 
Issue Tracking System - My B.E. Project
Issue Tracking System - My B.E. ProjectIssue Tracking System - My B.E. Project
Issue Tracking System - My B.E. Project
 
The Evolution of a Scrappy Startup to a Successful Web Service
The Evolution of a Scrappy Startup to a Successful Web ServiceThe Evolution of a Scrappy Startup to a Successful Web Service
The Evolution of a Scrappy Startup to a Successful Web Service
 
Srikanth_hadoop
Srikanth_hadoopSrikanth_hadoop
Srikanth_hadoop
 
Why is the application running so slowly?
Why is the application running so slowly?Why is the application running so slowly?
Why is the application running so slowly?
 
ECM Renovation Roadshow - Welcome
ECM Renovation Roadshow - WelcomeECM Renovation Roadshow - Welcome
ECM Renovation Roadshow - Welcome
 
PraveenResume_TIBCO
PraveenResume_TIBCOPraveenResume_TIBCO
PraveenResume_TIBCO
 

Recently uploaded

Recently uploaded (20)

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 

What 5 years of implementation microservices has taught me

  • 1. What 5 years of implementation microservices has taught me Jeppe Cramon - @jeppec Cloud Create ApS
  • 2. What 15 year of implementation distributed systems has taught me Jeppe Cramon - @jeppec Cloud Create ApS
  • 6. @jeppec Let’s go back to 2005 When Internet Explorer 6 ruled the world
  • 8. @jeppec https://our.cool.bank-portal Customer 🧩 Name: Peter Hansen Address: Some Road 1 Some Zip Some City … Customer relations 🧩 Spouse: Mette Hansen Children: Jacob Hansen Ditte Hansen Accounts 🧩 Account Number Account Name Balance 9128-21892982981 Salary Account 214,59 9128-89812981189 Budget Account -345,75 9128-12217698156 Savings Account 17.230,15 Account details: 9128-21892982981 🧩 Account Name: Interest rate: 3% 🔧 Salary Account
  • 9. @jeppec Learnings The good • Smaller dedicated UI pieces allow many developers to work independently on the same portal page • Small views only used by a single portlet was easy to change as it has fewer couplings • Reactive UI was cool. Click a part in a portlet and through a UI event one or more portlets would react and change their content. Decoupling The bad • Portlet technology was very heavy and very inflexible
  • 10. @jeppec SOA for For The Win Build tiny services that could be combined to larger services
  • 11. @jeppec Data Storage Data Storage Data Storage Data Service Portal 1 Portal 2 Data Storage Application Process Service layer Activity/Task Service layer Data/Entity Service layer Layered SOA
  • 15. @jeppec Sales system Sale Delivery system Deliveries Customer/CR M system Customer SAP Bookkeeping Complete Purchase Transaction Coordinator Transactional Resource Prepare Phase Commit Phase 2 Phase Commit
  • 16. @jeppec What’s wrong with distributed transactions? • Transactions lock resources while active • Services are autonomous • Can’t be expected to finish within a certain time interval • Locking keeps other transactions from completing their job • Locking doesn’t scale • X Phase Commit is fragile by design
  • 17. @jeppec Essential complexity of 2 way integration Component CComponent BComponent A UI Service Service B:Service() call C:Service() call A:Service() commit() Service Local transaction between Component A, B and C
  • 18. @jeppec What’s the challenge with using RPC/REST/SOAP/… between distributed components?
  • 19. @jeppec Synchronous calls lower our tolerance for faults • When you get an IO error • When servers crash or restarts • When databases are down • When deadlocks occurs in our databases • Do you retry? With synchronous style Service interaction we can loose business data if there’s no automatic retry Or we risk creating data more than once if the operation isn’t idempotent* Client Server Duplicated Response Duplicated Request Processing Response Request Processing The same message can be processed more than once *Idempotence describes the quality of an operation in which result and state does not change if the operation is performed more than 1 time
  • 20. @jeppec B:Service() call C:Service() call A:Service() if (A:Call-Failed:Too-Busy?) Wait-A-While() call A:Service() if (A:Call-Failed:Too-Busy?) Wait-A-Little-While-Longer() call A:Service() if (A:Call-Failed:IO-Error?) Save-We-Need-Check-If-Call-A-Succeded-After-All AND We-Need-To-Retry call C:Service and call B:Service AND Tell-Customer-That-This-Operation-Perhaps-Went-Well if (A:Call-Went-Well?) commit() Accidental complexity from distributed service integration Component CComponent BSystem A UI Service Service Service Local transaction between Component B and C
  • 21. @jeppec Service autonomy Component B Component C Component A System X Service A Component B Component C System X Slow/unreliable network Different SLA Slow system
  • 22. @jeppec Learnings The bad • Task/Activity services needs to perform UPDATES across multiple data/entity services. Requires distributed transactions • The more synchronous request-response remote calls you have to make the more it hurts Performance. • Robustness is lower. If one data/entity services is down it can take down many other services. • Coupling is higher. Multiple task/activity services manage the same CRUD data/entity services • Cohesion is likely lower as multiple task/activity services need to replicate entity related logic
  • 23. @jeppec Conclusion Friends don’t let friends use layered SOA
  • 24. @jeppec Skip forward a little later To when Enterprise Service Busses were king
  • 26. @jeppec Availability goes down (without additional instances of each service) Combined availability: 97%
  • 27. @jeppec Learnings Good • Very early in the project a full end to end correlation logging framework and application was built, so you knew exactly which services were being called by whom, where the time was spent Bad • When it takes > 10 min to perform the primary use case you have a serious performance overhead • When you have circular Service calls you have a big boundary issue • When a single service changes its interface and it takes days to adjust other services before everything works again, then you’re far from decoupled
  • 28. @jeppec ESB’s the the Rescue? SOA Web of Mess hidden by ESB Inventory Purchase ShippingPortal ESB
  • 29. @jeppec ESB Reality SOA Web of Mess hidden by ESB Inventory Purchase ShippingPortal ESB
  • 30. @jeppec ESB Learnings • Very often the ESB’s only role is to decouple teams • You’re effectively introducing a middle management layer with a thin promise of increased security, monitoring and decoupling • Often most services exposed are 1-1 with the original interface. Any decoupling is imaginary and the ESB is a bottleneck. • Or even worse they follow the layered SOA approach • Monitoring and security is an end-to-end concern • Putting some magic technology in the middle doesn’t solve the real problem
  • 31. @jeppec Forward to 2009 and our first microservices A customer facing self service portal
  • 32. @jeppec https://our.micro-cool.bank-portal Customer Name: Peter Hansen Address: Some Road 1 Some Zip Some City Customer relations Spouse: Mette Hansen Children: Jacob Hansen Ditte Hansen Account Number Account Name Balance 9128-21892982981 Salary Account 214,59 9128-89812981189 Budget Account -345,75 9128-12217698156 Savings Account 17.230,15 Account details: 9128-21892982981 Account Name: Interest rate: 3% 🔧 Salary Account
  • 33. @jeppec https://our.micro-cool.bank-portal Customer Name: Peter Hansen Address: Some Road 1 Some Zip Some City Customer relations Spouse: Mette Hansen Children: Jacob Hansen Ditte Hansen Overview Page (owned by the portal application) <<UI Component>> Customer Details Component <<UI Component>> Customer Relations Component UI layer Backend layer Customer Details Microservice Customer Relations Microservice HTTPS HTTPS
  • 34. @jeppec An UI is composed of widgets from different services 34
  • 35. @jeppec Widget Widget Page Widget Widget Widget Widget • Overall structure of the page is “owned” by the application. • Each widget is independent Composite page layout Widget Widget Widget
  • 36. @jeppec Learnings The good: • Each section on the page is componentized. • The page knows NOTHING about how information is fetched • Very low horizontal coupling • Each component is responsible for ensuring availability and stability. You build it, you run it, you fix it  • Individual component/service failure doesn’t bring down the entire page The bad: • The microservices tended to be overly small and overlapped ➡️ code duplication and shared databases
  • 37. @jeppec Master Data Management example using Events 2011-2012
  • 38. @jeppec System A System B System C System D 4 systems with shared data and no clear master for data No Unique Customer Profile
  • 40. @jeppec One way Master Data duplication System A Master How to integrate?
  • 41. @jeppec One way Master Data duplication System A Master Query Query result 1. Read Ids of all Entities that match criteria 2. For each Id load the Entity 3. Result: 1+n problem
  • 43. @jeppec One way Data duplication using Events System A Master Events
  • 44. @jeppec System A Update Customer Contact Info Update Customer ContactInfo Process ”UpdateCustomerContactInfo” command Event Publisher ”CustomerContactInfoUpdated” Event Customer View Customer local/cached replica System D BI/And other parties System B Customer Master
  • 45. @jeppec Back-end Data System A System D OSB Private Customer Service Proxy ESB Event Receiver CustomerCore Private Customer Service Event Queue Event Topic Command Internal Event External Event External PrivateCustomer EventReceiver Internal PrivateCustomer EventReceiverInternal Services System A System D Event Topic With SSNWithout SSN ? Contains: Social Security Number (SSN), Who’s allowed to see SSN and who the Event is relevant for External Services Security control
  • 46. @jeppec Learnings The Good: • We were able to allow all the external source systems to continue with their existing domain models • We gained a centralized overview of all customers and their memberships • The external systems rarely needed to contact the Customer Service • Breakdowns in communication paths or systems didn’t bring down all other systems • Use of Optimistic Concurrency combined with eventsourcing made for easy coordination of (rare) out of sync updates • CQRS proved to be a good fit for an event sourced solution • Resilience and stability The Bad: • Use of ESB again proved to be a big time waster • Full resilience, e.g. during Queue/Topic data loss or coding errors in external systems, required additional Reconciliation service operations (to fetch missing events)
  • 47. @jeppec The final piece of the puzzle The Domain
  • 48. @jeppec "I consider 'getting the boundaries right' the single design decision with the most significant impact over the entire life of a software project." @ziobrando
  • 49. @jeppec Monoliths can be well designed Unfortunately many monoliths experience high degrees of coupling, resulting in them earning a bad name
  • 53. Business Capability and Bounded Contexts are in many ways similar • What we want to achieve is • problem domain and • solution domain • ALIGNMENT
  • 54. @jeppec Business Capability alignment “The advantage of business capabilities is their remarkable level of stability. If we take a typical insurance organisation, it will likely have sales, marketing, policy administration, claims management, risk assessment, billing, payments, customer service, human resource management, rate management, document management, channel management, commissions management, compliance, IT support and human task management capabilities. In fact, any insurance organisation will very likely have many of these capabilities.” See http://bill-poole.blogspot.dk/2008/07/business-capabilities.html
  • 55. @jeppec Business – IT alignment • We want the Business and IT to speak the same Ubiquitous language • Want want our architecture to be aligned with the business capabilities • Because these capabilities are stable
  • 56. @jeppec Many perspectives on data Online Retail System Product Unit Price Promotional Price Promotion End Date Stock Keeping Unit (SKU) Quantity On Hand (QOH) Location Code Price Quantity Ordered Name The lifecycle of the data is VERY important! Customer Pricing Inventory Sales Management Reporting
  • 57. @jeppec Smaller models & clear data ownership Retail System Pricing Product ProductID Unit Price Promotional Price … Pricing Inventory Product ProductID SKU QOH Location Code … Inventory Sales Product ProductID Name Description Quantity Ordered … Sales Shared Entity identity DDD: Bounded Context Business Capability
  • 58. @jeppec Bounded Contexts and Aggregates Sales Product Customer customerId … Order orderId customerId … OrderLine orderId productId quantity timestamp … ProductCategory productCategoryId … Pricing Product productId productCategoryId name tag ... Product-Price productId normalPrice discountPeriods …
  • 59. @jeppec Using Business Events to drive Business Processes Sales Service Shipping Billing Sales Customers MessageChannel Online Ordering System Web Shop (Composite UI) Billing Service Shipping Service <<External>> Order Accepted AcceptOrder The sales fulfillment processing can now begin… Cmd Handler Order Accepted Apply
  • 60. @jeppec Choreographed Event Driven Processes Sales Service Order Accepted Billing Service Order Fulfilment (Saga/ Process-Manager) Shipping Service Online Ordering System MessageChannel(e.g.aTopic) Order Accepted Order Accepted Customer Billed Customer Billed Order Approved Order Approved Works as a Finite State Machine (WorkFlow) handling the life cycle of Shipping and thereby forms a very central new Aggregate in the System
  • 61. @jeppec If we align the problem domain with the solution domain Bounded Context 1 Bounded Context 3Bounded Context 2 UI BL DAO UI BL DAO UI BL DAO Vertical coupling is unavoidable We want to avoid horizontal coupling
  • 62. @jeppec Internal vs External Events { EventType: "OrderWasAccepted", CustomerId: "50D1F244-ABBC-4EC7-BDCA-E4934C124A89", OrderId: "C199322A-01F1-4E56-918E-7A63529F8FA3", Items: [ { ProductId: "4CD22C4B-600C-4477-B5BF-48ABDEE4DA61", Amount: "100", AmountUnit: "Pieces" }, { ProductId: "56E6BD19-660C-464A-9120-100DAF579855", Amount: "10", AmountUnit: "Litres" } ] } Internal Event: { EventType: "OrderWasAccepted", OrderId: "C199322A-01F1-4E56-918E-7A63529F8FA3" } External Event:
  • 64. @jeppec A Service is • The technical authority for a given bounded context/business-capability • It is the owner of all the data and business rules that support this bounded context – everywhere • It forms a single source of truth for that bounded context http://udidahan.com/2010/11/15/the-known-unknowns-of-soa/
  • 66. @jeppec Microservices promise a solution to our problem Monolith Microservice Microservice Microservice MicroserviceMicroservice Microservice Microservice
  • 67. @jeppec There’s never time to do it right But there’s always time to do it over 67
  • 69. @jeppec Service and deployment • A Service represents a logical responsibility boundary • Logical responsibility and physical deployment of a Service DOES NOT have to be 1-to-1 • It’s too constraining • We need more degrees of freedom • Philippe Krutchen 4+1 views of architecture: Logical and Physical designs should be independent of each other A service needs to be deployed everywhere its data is needed
  • 71. @jeppec Service Microservices 1..* Is implemented by A Service is the technical authority of a specific Bounded-Context/Business Capability e.g. Sales, Shipping, Billing Service vs Microservices
  • 72. @jeppec Service Microservices 1..* Is implemented by Service vs Microservices Microservices are a division of Services along Transactional boundaries (a transaction stays within the boundary of a Microservice) Microservices are the individual deployable units of a Service with their own Endpoints. Could e.g. be the split between Read and Write models (CQRS) - each would be their own Microservices
  • 73. @jeppec A Service represents a logical boundary 73 Service Microservice Microservice Microservice Microservice
  • 74. @jeppec Services are the corner stone • We talk in terms of Services capabilities and the processes/use-cases they support • Microservices are an implementation detail • They are much less stable (which is a good thing – it means they’re easier to replace)
  • 75. @jeppec There’s cost in deploying 1000’s of microservices (or 100.000’s serverless functions) 75
  • 76. @jeppec Autonomous Components Not everything needs to be individually deployable 76
  • 77. @jeppec This means they CAN, but they don’t HAVE to be deployed individually. Design for Distribution But take advantage of locality
  • 78. @jeppec Let’s be even more pragmatic In which case we allow other services to call them using local calls 78
  • 79. @jeppec AC in code public class PricingEngineAc extends AutonomousComponent { public static AutonomousComponentId SERVICE_AC_ID = SALES_SERVICE_ID.ac(”pricing_engine_ac"); … public PricingEngineAc(CurrencyConverter currencyConverter) { this.currencyConverter = currencyConverter; } @Override public void onInitialize(IConfigureACEnvironment acSetup) { acSetup.withAutonomousComponentId(SERVICE_AC_ID) .usingServiceDataSource() .withBusConfiguration(cfg -> { bus.registerAxonReplayableTopicPublisher(InternalPricingEvents.TOPIC_NAME, replayFromAggregate(Pricing.class) .dispatchAggregateEventsOfType(InternalPricingEvents.class)); bus.subscribeTopic(SERVICE_AC_ID.topicSubscriber(”InventoryEvents"), ExternalInventoryEvents.TOPIC_NAME, new InventoryTopicSubscription(bus)); }) .runOnBusStartup((bus, axonContext) -> { }); } }
  • 80. @jeppec Service deployment • Many services can be deployed to the same physical server • Many services can be deployed in the same application • Application boundary is a Process boundary which is a physical boundary • A Service is a logical boundary • Service deployment is not restricted to tiers either • Part of service A and B can be deployed to the Web tier • Another part of Service A and B can be deployed to the backend/app-service tier of the same application • The same service can be deployed to multiple tiers / multiple applications • ie. applications and services are not the same and does not share the same boundaries • Multiple services can be “deployed” to the same UI page (service mashup)
  • 81. @jeppec An Application is the plate where Components are co-deployed 81 Sales service components Inventory service components …
  • 82. @jeppec iOS Home Banking Customer information Legal and contract information Accounts
  • 83. @jeppec iOS Home banking Customer information Legal and contract information Accounts Credit card Mortgage loans Web banking portal Bank Back-office application
  • 84. @jeppec Service A Service B Service C IT-OPS Web shop Warehouse Back office App logic (Layered, CQRS,…) Warehouse Storage UI Components A Services owns its UI Components
  • 87. @jeppec Composite page example Page Context: {id: ISBN-10 0-321-83457-7 } Images Books Reviews Pricing Inventory OthersAlsoBought Pricing Reviews Books Images Books
  • 88. @jeppec A Service represents a logical boundary 88 Service Autonomous Component Autonomous Component Adapter UI component
  • 89. @jeppec Autonomous Component • Can be deployed alone or co-located, together with one or more adapters from the same service • Works transparently in a clustered environment Core logic (use case controllers/aggregates) Primary/Driving Adapter GUI / API / … Primary/Driving Port Secondary/Driven Adapter Database/Notifications/… Secondary/Driven Port
  • 90. @jeppec Autonomous Components can be co-deployed together with Application backends web_shop (Spring Boot fat–jar) pricing_engine_ac discounts_ac webshop_basket_adapters api_products product_details_ac product_details_adapters frontend api_pricing app libs products pricing branding
  • 91. @jeppec Application in code @Configuration @ComponentScan(basePackages = { "com.mycoolshop.webshop", "com.mycoolshop.adapters", "com.mycoolshop.itops.spring" }) public class Application extends IAmASpringBootApplication { @Override protected ApplicationIdentifier getApplicationIdentifier() { return ApplicationIdentifier.from(”WebShop"); } @Override protected Collection<AutonomousComponent> getAutonomousComponentsHostedInThisApplication() { CurrencyExchangeRateAc currencyExchangeRateAc = new CurrencyExchangeRateAc(); return list( new PricingEngineAc(currencyExchangeRateAc.getCurrencyConverter()), new DiscountsAc(currencyExchangeRateAc.getCurrencyConverter()), new InventoryAc(), new CustomersAc(), currencyExchangeRateAc ); } public static void main(String[] args) { SpringApplication.run(Application.class, args); } }
  • 93. @jeppec Autonomous Component AC - 1 AC - 1AC - 1 AC - 1 AC - 2 AC - 2 AC - 2 AC - 2 AC - 3 AC - 3 AC - 3 AC - 3 AC - 4 AC - 4 AC - 4 AC - 4 Federated Bus
  • 94. @jeppec pricing_engine_ac (deployed on 10.25.26.102) pricing_engine_ac (deployed on 10.25.26.101) Bus Bus Bus Bus inventory_ac (deployed on 10.25.26.104) inventory_ac (deployed on 10.25.26.103) Federated Bus
  • 95. @jeppec Topics Bus features AC - 4 AC - 1 AC - 3 bus.registerAxonReplayableTopicPublisher(InternalPricingEvents.TOPIC_NAME, replayFromAggregate(Pricing.class) .dispatchAggregateEventsOfType( InternalPricingEvents.class ) ); bus.subscribeTopic(SERVICE_AC_ID.topicSubscriber(”Pricing"), InternalPricingEvents.TOPIC_NAME, new PricingTopicSubscription(bus));
  • 96. @jeppec Topics public class PricingTopicSubscription extends BusMessageHandlerDelegator { @BusMessagePayloadHandler private void on(PrincingChanged e, BusMessage<PricingChanged> msg) { … } … }
  • 98. @jeppec Client handled subscriptions • Highly resilient pattern for an Event Driven Architecture that’s backed by Event-Sourced AC’s • In this model the publisher of the Events is responsible for the durability of all its Events, typically to an EventStore/EventLog. • Each client (subscriber) maintains durable information of the last event it has received from each publisher. • When ever the client starts up it makes a subscription to the publisher where it states from which point in time it wants events published/streamed to it. • This effectively means that publisher can remain simple and the client (subscriber) can remain simple and we don’t need additional sophisticated broker infrastructure such as Kafka+ZooKeeper.
  • 99. @jeppec Client handled subscriptions Publisher Subscriber A Local storage EventStore Subscriber B Local storage Topic Subscription Topic Subscription TopicSubscriptionHandler TopicSubscriptionHandler EventEvent Event Event EventBus Event Event Distributed Event Bus, which ensures that live events published on an AC node in the cluster can be seen by all AC’s of the same type Singe Instance Subscriber, which ensures that only one instance of Subscriber B has an active subscription(s). Other instances of the same subscriber are hot-standby <<Topic Subscriber>> Customer_Service:Customer_Agreements_Ac:OrderEvents <<Topic Publisher>> Sales_Service:OrderEvents
  • 100. @jeppec Topics Bus features Features: • The Bus provides automatic and durable handling of Redelivery in case of message handling failure through a Redelivery Policy • Exponential Backoff • Max retries • Dead letter/Error Queue • Support for resubscription at any point in the timeline of an Event Stream • Automatically tracking of resubscription points - aka. resubscribe at last tracked point AC - 4 AC - 1 AC - 3
  • 101. @jeppec Bus features Notifications: • Durable notifications with failover support • bus.notify(notificationQueue, notificationMessage) Notifications Distributed Broadcast Broadcast: • Broadcast a Message to all AC’s in the cluster • Broadcast a Message to a UI client (all, per user, per privilege)
  • 102. @jeppec Bus features Features: • Support for sending a message to a single consumer • Default pattern is Competing Consumers • The bus provides durability for all messages send on a Queue • The Bus provider automatic and durable handling of Redelivery in case of message handling failure through a Redelivery Policy • Exponential Backoff • Max retries • Dead letter/Error Queue Durable Queues
  • 103. @jeppec At least once delivery In a distributed systems message delivery can and will fail! Therefore, everything that can handle messages are built with idempotency in mind: • We always check for the existence of an Aggregate before creating it • All business methods in an aggregate check if the operation/side effect has already taken place • View Repositories support automatic reordering of messages that come out of order due to redelivery(if this is required by the View)
  • 104. @jeppec Bus features Single Instance Task: • Ensures that only one Active instance of a Task is active in the cluster at one time • Other tasks of the same type are in hot standby • Used to e.g. group multiple subscribers, to ensure that all subscribers are either all active or standby. • Used by our ViewRepositories Distributed SingleInstanceTask bus.createClusterSingleInstanceTask(”MyTask", new MyTask ()); // Where MyTask implements Lifecycle
  • 105. @jeppec Bus features Process Manager: • Defines durable business processes as a flow of Events Process Manager Sales Service Order Accepted Billing Service Order Fulfilment (Saga/ Process-Manager) Shipping Service MessageChannel(e.g.aTopic) Order Accepted Order Accepted Customer Billed Customer Billed Order Approved Order Approved
  • 106. @jeppec Application features Workflow Manager: • Defines the Tasks that required human intervention, such as: • Approvals (e.g. Contract approval) • Assistance/Help with a business problem • Incident handling (e.g. a technical problem identified by a developer) • Authorization (e.g. request manager approval) • Reminders • Common tasks supported: Claiming Tasks, Escalating Tasks, Completing Tasks Workflow Manager
  • 107. @jeppec Application features Identity Management: • Common handling of Users, Privileges, Roles • Automatic enforcement of privilege requirements • Support for Multitenancy (Contractual level, Service Agreement Level) Identity Management
  • 108. @jeppec Correlation logging Our infrastructure automatically captures information about an API call, a Message delivery in the form of a CallContext: • Message Id - the Id of the message/call being handled. • Correlation Id – an Id that binds API calls and message handlings across multiple services/AC’s together • When – time of the “call” • Who – which user performed the “call” • Meta Data: Which Event or Command caused this “call”
  • 110. Thanks :) Blog: https://cramonblog.wordpress.com/ Homepage: http://cloudcreate.dk/ Twitter: @jeppec