Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Autonomous microservices for a Financial System


Published on

There are many articles about microservices, but few cover how to apply Microservice principles to non-trivial domains. In this presentation, Jeppe will show how INPAY design highly autonomous microservices and composite applications. We will get into the underlying principles, such as how to discover service boundaries, how to communicate between services, what role does the UI play. Finally we will look at how services are implemented and supported by the infrastructure.

Keywords: DDD, Microservices, CQRS, Event Sourcing, Event Driven Architecture, Federated Bus, Composite UI

Published in: Technology

Autonomous microservices for a Financial System

  1. 1. AUTONOMOUS MICROSERVICES FOR A FINANCIAL SYSTEM Jeppe Cramon - @jeppec Chief Architect - INPAY
  2. 2. “Inpay’s vision is to transform global trade by making cross-border payments as fast, easy, safe, and cost-efficient as local bank transfers” Jacob Tackmann Thomsen Founder & CEO 2
  3. 3. Our development teams main goal: Build a flexible and scalable solution that can grow with the company 3
  5. 5. WE ALL WANT TO MOVE FAST But many organization experience that they cannot move fast
  7. 7. High coupling
  8. 8. It usually starts simple “The” DB UI Logic Data Access
  9. 9. And we end up with…
  10. 10. Reality rears its ugly head
  11. 11. It’s hard to see the sky
  12. 12. How to ensure ending up there? Be in a hurry! 12
  13. 13. Being in a hurry results in writing bad code Bad code slows us down every time we want to change something 13
  14. 14. Bad code is rigid Changing one thing requires you to modify a lot of other code to make the codebase consistent again 14
  15. 15. Bad code is fragile 15 When you change a thing you end up breaking something else!
  16. 16. Coupling • Coupling is a measure of dependencies: • Efferent coupling – who do you depend on • Afferent coupling – who depends on you • Different aspects on coupling: • Temporal coupling (processing time of Y affects X – if Y is down can X continue?) • Spatial coupling (where on the network are the Y instances and how does X know?) • Behavioral coupling (how much about Y’s behavior does X understand) X Y
  17. 17. End result “Nobody is allowed to modify that module!” or “Let’s rewrite!” 17
  18. 18. If we want to go fast, then we need to do a good job This means: • Take your time • Study the problem • Work iteratively to refine until you end up at a simple solution 18
  19. 19. Microservices promise a solution to our problem Monolith Microservice Microservice Microservice MicroserviceMicroservice Microservice Microservice
  20. 20. There’s never time to do it right But there’s always time to do it over 20
  21. 21. THERE IS VALUE IN MAKING THINGS SMALLER For one thing it is easier to reason about them in isolation It’s easier to replace them – since coupling is explicit
  22. 22. Microservices as an architectural style 22
  23. 23. ARCHITECTURE IS ESSENTIAL Because it influences how fast we can respond to changes and what the cost of those changes are
  24. 24. What is the right level of coupling? This highly depends on • How likely are things to change • And what parts that change together 24
  25. 25. MICROSERVICES TELL US TO ADHERE TO THE SRP Single Responsibility Principle (SRP) – “A Microservice should have one reason – and only one reason – to change”
  26. 26. Does your domain model adhere to the SRP?
  27. 27. How to create a big entangled domain model? 27
  28. 28. Result will be fragmented domain logic Focus on Nouns (Entities) and retrofit Verbs later 28
  29. 29. The Verbs capture the intent/task we wish to accomplish – i.e. the use cases 29 A Use case typically involve more than one Entity
  30. 30. If we primarily model around nouns/entities we can easily violate the SRP Changes to requirements is likely to require changes to multiple entity classes 30
  31. 31. If a change to requirements forces us to “refactor” the structure of your domain model Then it wasn’t a model of the domain! 31
  32. 32. A change to requirements should be a “change to how we do business” Not a change to what business we’re in! 32
  33. 33. "We don't sell domains, WE SELL USE CASES!" Jim Coplien 33
  34. 34. So instead of focusing on Nouns We should focus on Verbs! 34
  35. 35. Second reason for big entangled domain models Trying to create a single model to serve everyone 35
  36. 36. Every system has Multiple Users with multiple roles Multiple Stakeholders 36
  37. 37. It’s very hard to come up with one canonical model that’s good for everybody 37
  38. 38. Different perspectives on data With in a given Domain, e.g. Retail, there will exist multiple bounded contexts/sub-domains/business capabilities such as: • Product management • Purchase • Sales • Pricing • Inventory • Shipping • Support • Accounting • Management Each of these lines of business have very specific and unique needs which are relevant for them alone in order to conduct their business. They might use the same name for the entities they’re interested in or they might use different names for the same logical entity.
  39. 39. Many perspectives on data Online Retail System Product Unit Price Promotional Price Promotion End Date Stock Keeping Unit (SKU) Quantity On Hand (QOH) Location Code Price Quantity Ordered Name The lifecycle of the data is VERY important! Customer Pricing Inventory Sales Management Reporting
  40. 40. Smaller models & clear data ownership Retail System Pricing Product ProductID Unit Price Promotional Price … Pricing Inventory Product ProductID SKU QOH Location Code … Inventory Sales Product ProductID Name Description Quantity Ordered … Sales Shared Entity identity
  41. 41. How can we discover bounded contexts? 41
  42. 42. BOUNDED CONTEXT The setting in which a word or a statement appears that determines its meaning 42
  44. 44. Sequential data gathering? Time
  45. 45. Or design by committee?
  46. 46. REQUIREMENTS GATHERING TENDS TO FOLLOW THE STEPS OF CLASSICAL OO DESIGN We write use cases and afterwards look for nouns and verbs
  47. 47. SOFTWARE DEVELOPMENT IS JOURNEY Involving collective learning
  49. 49. Event Storming • Event Storming is a fun way of bringing developers and business experts together and drive your analysis from the outside and quickly explore complex business domains in hours instead of days/weeks.
  50. 50. Event Storming • Invented by Alberto Brandolini • See
  51. 51. Event Storming • Invite the right people • people with questions and people with answers. They provide the perfect mix of curiosity and wisdom. • Provide unlimited modeling space
  52. 52. Event Storming • Explore the domain starting from Domain/Business Events Event
  53. 53. Why Events • Simple semantic and notation • Represent the state transitions • Gives us precision • Easy to grasp by everybody in the room 53
  54. 54. Events An Event is non-prescriptive of what should happen in other parts of the system. It leaves this open to the recipients, so that they themselves determine what to do based on occurrence of the event. Events always carry a name in its past-tense form: OrderWasAccepted, OrderHasShipped, CustomerWasReimbursed Other qualities • Immutable, i.e. content cannot be changed • Always carries the ID of the Business Object it relates to • An event can and will typically will be published to multiple consumers. • The publisher of the event does not know who the recipients are • And it doesn’t know what the recipients intend to do with the event “An Event describes something that HAS happened”
  55. 55. Why Events • Simple semantic and notation • Represent the state transitions • Gives us precision • Easy to grasp by everybody in the room • Place them along a timeline to visualize the process • An event might be the predecessor or the follower of another one. 55
  56. 56. Event Storming • What causes Domain Events? • User Action model it as a Command • External system • Time • Other Domain Events • Documents External Time Command Role Event Event Document
  57. 57. Events are often the side effect of Commands A Command is prescriptive of what should happen. This is a stronger form of coupling than Events or Documents. A Command’s primary goal is to capture USER INTENT A Command supports a single usecase and targets a single Business Object Commands always carry a name in its imperative form: CreateOrder, ShipOrder, CancelOrder, ReimburseCustomer, etc. Other qualities • Immutable, i.e. content cannot be changed • Always carries the ID of the Business Object that it relates to • A command can only be handled by ONE service • The Operations that consume Commands don’t return values – but can throw exceptions “A command describes a Task that you want someone else to carry out for you and the recipient can reject the Command”
  58. 58. Commands & Events Commands mutate Business Object state which results in one or more Events being published Command Event(s) AcceptOrder OrderAccepted ShipOrder OrderShipped AddComment CommentAdded QuarantineReview ReviewQuarantined UnquarantineReview ReviewUnquarantined
  59. 59. Also, UI’s give you valuable input Create Task based UI mockups 59
  60. 60. 60
  61. 61. Domain modeling 61
  62. 62. When discussing with Business Experts Focus on data fields and NOT entities 62
  63. 63. Focus on which data fields clump together (or are talked about together) and what fields are separate Place the fields in piles, that later will become your bounded contexts 63
  64. 64. Don’t name the piles before you know what they are This avoids cognitive bias 64
  65. 65. Give the piles gibberish or color names 65
  66. 66. UI BL DAO UI BL Entity UI BL Entity UI BL Entity UI BL Entity What we see, is that we move from
  67. 67. Towards Bounded Context that are fairly small (but not too small) and rarely need to call each other Sales Pricing InventoryShipping ✕ ✕ ✕ UI BL DAO UI BL DAO UI BL DAO UI BL DAO
  68. 68. So where does (micro) services fit into this? 68
  69. 69. A Service is • The technical authority for a given bounded context • It is the owner of all the data and business rules that support this bounded context – everywhere (including the UI) • It forms a single source of truth for that bounded context
  70. 70. SOA PRINCIPLE SERVICES ARE AUTONOMOUS Autonomy means that our service is independent and self- contained and as far as possible doesn’t directly depend on other services to be functional.
  71. 71. Service autonomy Component B Component C Component A System X Service A Component B Component C System X Slow/unreliable network Different SLA Slow system
  72. 72. SERVICES ARE AUTONOMOUS For a service to be autonomous is must NOT share state
  73. 73. SERVICES ARE AUTONOMOUS Autonomy is essential for Scalability (scale out clustering) Reliability (fail over clustering)
  74. 74. SERVICES ARE AUTONOMOUS Autonomy is essential for Reusability Adaptability
  75. 75. How do we communicate between our services? 75
  76. 76. Synchronous calls are the crystal meth of programming At first you make good progress but then the sheer horror becomes evident when you realise the scalability limitations and how the brittleness holds back both performance and development flexibility. By then it is too late to save. We need the reactive properties and then apply protocols for the message interactions. Without considering the protocols of interaction this world of micro-services will become a coordination nightmare. Martin Thompson
  77. 77. But all the cool kids tell us to use REST between (micro)services? 77
  78. 78. Service Dependencies Source:
  80. 80. Synchronous calls lower our tolerance for faults • When you get an IO error • When servers crash or restarts • When databases are down • When deadlocks occurs in our databases • Do you retry? With synchronous style Service interaction we can loose business data, there’s no automatic retry or we risk creating data more than once because idempotence* often is an after though 80 Client Server Duplicated Response Duplicated Request Processing Response Request Processing The same message can be processed more than once *Idempotence describes the quality of an operation in which result and state does not change if the operation is performed more than 1 time
  81. 81. Also remember: REST isn’t magic!
  83. 83. The 8 Fallacies of Distributed Computing These fallacies are assumptions architects, designers and developers of distributed systems are likely to make. The fallacies will be proven wrong in the long run - resulting in all sorts of troubles and pains for the solution and architects who made the assumptions. 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. See for a walkthrough of the fallacies and why they’re fallacies
  85. 85. Essential complexity of 2 way integration Component C Component B Component A UI Service Service B:Service() call C:Service() call A:Service() commit() Service Local transaction between System A, B and C
  86. 86. B:Service() call C:Service() call A:Service() if (A:Call-Failed:Too-Busy?) Wait-A-While() call A:Service() if (A:Call-Failed:Too-Busy?) Wait-A-Little-While-Longer() call A:Service() if (A:Call-Failed:IO-Error?) Save-We-Need-Check-If-Call-A-Succeded-After-All AND We-Need-To-Retry call C:Service and call B:Service AND Tell-Customer-That-This-Operation-Perhaps-Went-Well if (A:Call-Went-Well?) commit() Accidental complexity from distributed service integration Component C Component B System A UI Service Service Service Local transaction between System B and C
  87. 87. What’s wrong with distributed transactions? • Transactions lock resources while active • Services are autonomous • Can’t be expected to finish within a certain time interval • Locking keeps other transactions from completing their job • Locking doesn’t scale • And what about timeouts? • X Phase Commit is fragile by design
  88. 88. WE NEED TO CHANGE FOCUS FROM SHORT TECHNICAL TRANSACTIONS To long running business transactions supporting business processes
  89. 89. Using Business Events to drive Business Processes Sales Service Shipping Billing Sales Customers MessageChannel Online Ordering System Web Shop (Composite UI) Billing Service Shipping Service Order Accepted Event AcceptOrder Command The sales fulfillment processing can now begin…
  90. 90. Choreographed Event Driven Processes Sales Service Order Accepted Invoicing Service Order Fulfilment (Saga/ Process-Manager) Shipping Service Online Ordering System MessageChannel(e.g.aTopic) Order Accepted Order Accepted Customer Billed Customer Billed Order Approved Order Approved Works as a Finite State Machine (WorkFlow) handling the life cycle of Shipping and thereby forms a very central new Aggregate in the System
  92. 92. Services/Business-capabilities in INPAY Currency Service Banking Service Identity Management Service Sales Service PSP Service 3rd party Providers Service Virtual Banking Service IT Operations Compliance Service
  93. 93. Service and deployment • A Service represents a logical responsibility boundary • Logical responsibility and physical deployment of a Service DOES NOT have to be 1-to-1 • It’s too constraining • We need more degrees of freedom • Philippe Krutchen 4+1 views of architecture: Logical and Physical designs should be independent of each other A service needs to be deployed everywhere its data is needed
  94. 94. We need more fine grained building blocks Autonomous Components
  95. 95. A Service represents a logical boundary 95 Service Autonomous Component Autonomous Component UI component UI component
  96. 96. An Application is the plate where Components are co-deployed 96 Sales service components PSP service components …
  97. 97. Service deployment • Many services can be deployed to the same physical server • Many services can be deployed in the same application • Application boundary is a Process boundary which is a physical boundary • A Service is a logical boundary • Service deployment is not restricted to tiers either • Part of service A and B can be deployed to the Web tier • Another part of Service A and B can be deployed to the backend/app-service tier of the same application • The same service can be deployed to multiple tiers / multiple applications • ie. applications and services are not the same and does not share the same boundaries • Multiple services can be “deployed” to the same UI page (service mashup) • Multiple services can cooperate to fulfill a larger use-case (e.g. a workflow or a business process)
  98. 98. Service Autonomous Component 1..* Is implemented by A Service is the technical authority of a specific Bounded-Context/Business Capability e.g. Sales, Shipping, Billing Services support business processes. Business processes naturally span multiple services, but there will always be a single service that is the actual authority on the business process. Service vs Autonomous Components
  99. 99. Also known as Microservices Service Autonomous Component 1..* Is implemented by Service vs Autonomous Components Autonomous-Components/Microservices are a division of Services along Transactional boundaries (a transaction stays within the boundary of a Microservice) Microservices are the individually logical deployable units of a Service with their own Endpoints. Could e.g. be the split between Read and Write models (CQRS) - each would be their own Microservice
  100. 100. Autonomous Component • Can be deployed alone or co-located, together with one or more adapters from the same service • Works transparently in a clustered environment Core logic (use case controllers/aggregates) Primary/Driving Adapter GUI / API / … Primary/Driving Port Secondary/Driven Adapter Database/Notifications/… Secondary/Driven Port
  101. 101. Services are the corner stone • We talk in terms of Services/business capabilities and the processes/use- cases they support • Autonomous-Components/Microservices are an implementation detail • They are much less stable (which is a good thing – it means they’re easier to replace) • With regards to other Services • subscribe to events from • send commands to (less common) • call operations (in rare occasions)
  102. 102. Services/Bounded Contexts and Aggregates Sales Service PSP Service Virtual Banking Service Customer customerId … Contract contractId customerId … VBFeeSchedule contractId … PSPFeeSchedule contractId … BillingTemplate contractId …
  103. 103. CQRS A single model cannot be appropriate for reporting, searching and transactional behavior Greg Young, 2008
  104. 104. Commands, Events and Query Models Read model Read model Events UI Domain modelQuery/Read model ”AcceptOrder” command ”OrderAccepted” event ”Find all Accepted Orders” Query Commands are Imperative: DoStuff Events are Past tense: StuffDone
  105. 105. Event Sourcing Aggregates track their own Domain Events and derive state from them Time 07:39 Time 07:40 Time 07:41 Time 07:45 Time 07:46 Time 07:50
  106. 106. Infrastructure patterns 106
  107. 107. Create Contract Sales Service : Contracts_Ac Contract Manager Application PSP Service : Agreement_Ac ContractCreated ContractCreated Publish PayInTemplateCreated ContractEvents <<Topic>>
  108. 108. Client handled subscriptions • Highly resilient pattern for an Event Driven Architecture that’s backed by Event- Sourced AC’s • In this model the publisher of the Events is responsible for the durability of all its Events, typically to an EventStore/EventLog. • Each client (subscriber) maintains durable information of the last event it has received from each publisher. • When ever the client starts up it makes a subscription to the publisher where it states from which point in time it wants events published. • This effectively means that publisher can remain simple and the client (subscriber) can remain simple and we don’t need additional sophisticated broker infrastructure such as Kafka+ZooKeeper.
  109. 109. Client handled subscriptions Publisher Subscriber A Local storage EventStore Subscriber B Local storage Topic Subscription Topic Subscription TopicSubscriptionHandler TopicSubscriptionHandler EventEvent Event Event EventBus Event Event
  110. 110. psp_fees_ac (deployed on psp_fees_ac (deployed on Bus Bus Bus Bus sales_contract_ac (deployed on sales_contract_ac (deployed on Federated Bus
  111. 111. Distributed Bus Topic Queue Topic Publisher side Topic Subscriber side Queue Sender side Queue Receiver side Local Topic Publisher Local Topic Subscriber Local Queue Sender Local Queue Receiver Distributed per Service EventBus Local Topic Subscriber Local Topic Subscriber Local Queue Sender Distributed Notifications Distributed Broadcast Distributed SingleInstance Task
  112. 112. What about applications? 112
  113. 113. Applications in INPAY Identity Management Application Contract Manager Application Treasury Application Compliance Application Operations Application CRM PSP Gateway PSP Merchant Application ERP
  114. 114. Who owns the UI? • For a service to be fully autonomous is must be self contained – which means it must: • Own its UI • Own its Business Logic • Own its Data model User interface Business Logic Data model & storage Service A similar approach is available under the name Self Contained Systems
  115. 115. Applications and Services iOS Homebanking Call center support portal Bank Backoffice application Customer information service Legal and contract information service Accounts service Credit card service Mortgage loans service
  116. 116. Screen view != View model Think in composites 116
  117. 117. 117
  118. 118. 118
  119. 119. Application UI’s • An applications is a composition of different services • The composition can be: • Resource Oriented Client Architecture (ROCA) style service integration ( • Mashup of Service UI components in a dedicated Application – aka. Composite UI Both solutions involve integration via Services web interfaces to minimize coupling to other services. Image from
  120. 120. Composite UI - example Page Context: {id: ISBN-10 0-321-83457-7 } ImageService BookService ReviewService PriceService InventoryService OthersAlsoBoughtService PriceService ReviewService BookService ImageService BookService
  121. 121. Widget Widget Page Widget Service A Service B Service C Widget Widget Widget Service A Service B Service C Widget Service C • Overall structure of the page is “owned” by the application. • Each widget is owned and delivered by the underlying Service. Page layout
  122. 122. AUTONOMOUS-COMPONENTS/ MICROSERVICES ARE LOGICAL DEPLOYABLE UNITS That doesn’t mean they HAVE to be deployed individually. Design for Distribution But take advantage of locality
  123. 123. Autonomous Components can be co-deployed together with Application backends contract_manager (Spring Boot fat–jar) sales_contract_ac sales_customer_ac sales_contract_manager_adapters api_psp psp_fees_ac psp_contract_manager_adapters frontend api_sales app libs contract customer fees
  124. 124. Be pragmatic Some services are more stable In which case we allow other services to call them using local calls 124
  125. 125. Application in code @Configuration @ComponentScan(basePackages = { "com.inpay.contractmanager", "com.inpay.adapters", "com.inpay.itops.spring" }) public class Application extends InpaySpringBootApplication { public Application() { super(); } @Override protected ApplicationIdentifier getApplicationIdentifier() { return ApplicationIdentifier.from("ContractManager"); } @Override protected Collection<AutonomousComponent> getAutonomousComponentsHostedInThisApplication() { CurrencyExchangeRateAc currencyExchangeRateAc = new CurrencyExchangeRateAc(); return list( new PSPFeeScheduleAc(currencyExchangeRateAc.getCurrencyConverter()), new VBFeeScheduleAc(currencyExchangeRateAc.getCurrencyConverter()), new ContractAc(), new CustomersAc(), currencyExchangeRateAc ); } public static void main(String[] args) {, args); } }
  126. 126. AC in code public class PSPAgreementAc extends HzBackedAutonomousComponent { public static AutonomousComponentId SERVICE_AC_ID ="psp_agreement_ac"); … public PSPAgreementAc(CurrencyConverter currencyConverter) { this.currencyConverter = currencyConverter; } @Override public void onInitialize(IConfigureACEnvironment acSetup) { acSetup.withAutonomousComponentId(SERVICE_AC_ID).usingServiceDataSource() .withBusConfiguration(cfg -> { cfg.getAxonContext() .subscribeAnnotatedCommandHandler(new TemplateCmdHandler( cfg.getAxonContext().eventSourcedRepository(PSPTemplate.class), currencyConverter)); …. manageLifecycleFor(templateViewRepository = new TemplateViewRepository(cfg, currencyConverter)); }) .runOnBusStartup((bus, axonContext) -> { bus.registerAxonReplayableTopicPublisher(InternalTemplateEvents.TOPIC_NAME, replayFromAggregate(PSPTemplate.class) .dispatchAggregateEventsOfType(InternalTemplateEvents.class)); bus.subscribeTopic(SERVICE_AC_ID.topicSubscriber("ContractEvents"), ExternalContractEvents.TOPIC_NAME, new SalesTopicSubscription(bus)); }); } public TemplateViewRepository getTemplateViewRepository() { return templateViewRepository; } }
  127. 127. AC, autonomy and “shared” service data Service DB DB Autonomous Component Autonomous Component Autonomous Component Autonomous Component DB
  128. 128. 50 shades of inter service AC Autonomy* Endpoint Process Database Storage Shared Shared Shared Shared Own Shared Shared Shared Own Own Shared Shared Own Shared Own Shared Own Own Own Shared Own Own Own Own Lower Autonomy Higher Autonomy * No RPC in use!
  129. 129. 5 Microservices Do’s • Identify Business Capabilities (or Bounded Contexts) and split your services according to them. A service is owned by one team that builds and runs the service. This gives you proper business and IT alignment and allows pin point accuracy with regards to spending money to solve problems. • Spend time to understand the business processes and the domain. At first you must go slow to go fast. Building microservices properly takes time and is not trivial. Identify how likely things are to change and what things change together. • Focus on the business side effects - also know as the Events and make them a first class principle. Avoid RPC/REST/Request-Response between Services - events are the API. • Consider building composite application and Backend’s For Frontend's (BFF’s) to decouple services further. An application is owned by a dedicated team, but may borrow developers from service teams. • Learn from history. Don’t repeat the mistakes that gave (misapplied) SOA a bad name. Also, microservice might not be as small as you think - we need low coupling as well as high cohesion :)
  130. 130. 5 Microservices Don’ts • Do not introduce a network boundary as an excuse to write better code - many have troubles with poorly modularized monoliths and believe that introducing network between modules magically solves the problem. If you don’t change your thinking and design, you will end up with a distributed monolith, which has all the disadvantages of a monolith, the disadvantages of a distributed system and none of the advantages of microservices • Don't split the atom! Distributed Transactions are never easy. Learn about the CAP theorem and avoid Request/Response API’s between Services. • Don't fall into the trap of “Not my problem”. When working on isolated code bases teams can loose sight of the big picture. • Identify the bottlenecks and possible solutions before deciding to split a problem into one or more microservices. There’s nothing the guarantees that your microservice scales better than your monolith. • Don't do big bang rewrites. Move towards microservices gradually while focusing on functional areas that can replace or support the old monolith. Don’t rewrite core business while being new to microservices.
  131. 131. Thanks :)