Building Event Driven Services with Stateful Streams
•
4 likes•5,780 views
Event Driven Services come in many shapes and sizes from tiny event driven functions that dip into an event stream, right through to heavy, stateful services which can facilitate request response. This practical talk makes the case for building this style of system using Stream Processing tools. We also walk through a number of patterns for how we actually put these things together.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building Event Driven Services with Stateful Streams
Similar to Building Event Driven Services with Stateful Streams (20)
More from Ben Stopford
More from Ben Stopford (17)
Recently uploaded
Recently uploaded (20)
Building Event Driven Services with Stateful Streams
- 1. 1 Building Event Driven Services with Stateful Streams Ben Stopford @benstopford
- 2. 2 Microservice are an evolution
- 4. 4 Apply the Single Responsibility Principal
- 9. 9 Service can have different release schedules Services Released Together Services Released Independently Stock Orders Payment Stock Orders Payment
- 10. 10 Allows us to scale
- 11. 11 Scale in people terms
- 12. 12 Microservices • Single Responsibility • Fine grained • Modularized • Independently deployable • Run by different teams
- 13. 13 When it comes to data, microservices promote independence.
- 14. 14 Stateful services have their own DB Orders Service Validation Service Payment Service Fulfillment Service Stock Service
- 16. 16Database Data on inside Data on outside Interface amplifies data Databases provide a rich point of coupling
- 17. 17 Service A Service B Service C So microservices avoid shared databases
- 18. 18 But most business services share the same core stream of facts. Catalog Authorisation
- 19. 19 Many databases leads to many data islands Orders Service Validation Service Payment Service Fulfillment Service Stock Service
- 20. 20 Difficult to compose a joined up view
- 21. 21 Services Evolution 1.0 - SOA 2.0 - Microservices 3.0 - What’s next???
- 22. 22 How do we get to 3.0? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
- 23. 23 How do we get to 3.0? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
- 24. 24 The synchronous world of request response protocols leads to tight, point-to-point couplings.
- 25. 25 Coupling A measure of the assumptions two services make about one another when they exchange information. Frank Laymann
- 26. 26 Change to one service => many dependencies to consider CHANGE
- 27. 27 Leverage Receiver Driven Flow Control
- 28. 28 UI Orders Service Stock Service Payment Provider Maybe order more stock Request-Response Model
- 29. 29 UI Orders Service Stock Service Repricing Service Payment Provider Maybe order more stock New services must be called explicitly
- 30. 30 UI Orders Service Stock Service Payment Provider Maybe order more stock Event Driven Model with Broker
- 31. 31 UI Orders Service Stock Service Payment Provider Maybe order more stock Flow Control is Receiver Driven: Pluggable Repricing Service
- 32. 32 Events + Brokers Decouple Architectures
- 33. 33 How do we get to 3.0? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
- 34. 34 Useful Grid Orders Service Payments Service Customers Service We need to join different datasets
- 35. 35 Useful Grid Orders Service Payments Service Customers Service Need a better way to handle data the data services expose
- 36. 36 How do we get to 3.0? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
- 37. 37 Useful Grid Orders Payments Customers Why Poll when we can Push
- 38. 38 Useful Grid Orders Service Payments Service Customers Service Streaming leads to being both event driven and data enabled Event Driven Data Enabled
- 39. 39 How do we get to 3.0? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
- 41. 41 Ledger Service Loan Validation Shared Narrative Fraud Service History Service Account Service Account History Login Service Loan Service A Journal of Service Interactions
- 42. 42 How do we get to 3.0? • Loose Coupling • Data Enabled • Event Driven • Operational Transparency
- 43. 43 Tenet 1: Events and Asynchronous processes better model the way businesses work
- 44. 44 Tenet 2: Services need to design for data-on-the- outside
- 45. 45 Tenet 3: The measure of a ‘good’ architecture is it’s ability to evolve
- 46. 46 Steps to Streaming Services
- 47. 47 1. Take Responsibility for the past and evolve
- 48. 48 Stay Simple. Take Responsibility for the past Browser Webserver
- 50. 50 2. Raise events. Don’t talk to services.
- 51. 51 Events => Decouple Events == Facts
- 52. 52 Raise events. Don’t talk to services Browser Webserver Orders Service
- 53. 53 Order Requested Order Received Browser Webserver Orders Service Raise events. Don’t talk to services
- 54. 54 Order Requested Order Validated Order Received Browser Webserver Orders Service Raise events. Don’t talk to services
- 55. 55 3. Use Kafka as a Service Backbone
- 56. 56 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Kafka as a Backbone for Events
- 57. 57 KAFKA - Highly available - Linearly scalable - Load balance consumers - HA consumers
- 58. 58 4. Use CDC to evolve away from legacy
- 60. 60KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use the Database as a ‘Seam’ Connect Products
- 61. 61 5. Make use of Schemas
- 62. 62KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Schemas to validate Backwards Compatibility Connect Products Schema Registry
- 63. 63 6. Use the Single Writer Principal
- 64. 64KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Apply the single writer principal Connect Products Schema Registry Order Completed
- 65. 65 Single Writer Principal - Creates local consistency points in the absence of Global Consistency - Makes schema upgrades easier to manage.
- 67. 67KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Apply Bandwidth Limits to services Connect Products Schema Registry Order Completed 100Mb/s
- 68. 68 8. Store Datasets in the Log
- 69. 69 Messaging that Remembers Orders Customers Payments Stock
- 70. 70 Rewind and Replay Rewind Replay Orders Service
- 71. 71 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Utilize Whole Datasets Connect Products Schema Registry Order Completed Repricing
- 72. 72 Compacted Topic: Delete superceded messages that share the same key K1 K1 K1 K2 K2 K2 K1 V1 V1 V2 V3 V2 V4 V3 Compacted Topics
- 73. 73 Orders Customers Payments Stock Single, Shared Source of Truth Data-on-the-Outside
- 74. 74 But how do you query a log?
- 75. 75 9. Use Materialized Views to Query the Log
- 76. 76 Materialized View Create via a query (select * from Orders Where Region == ‘USA’) Base Table Materialized Views in a DB
- 77. 77 Insert data Materialized View Create via a query (select * from Orders Where Region == ‘USA’) Query is rerun on data as it arrives Base Table Materialized Views in a DB
- 78. 78 In practice often more complex select Order.region, count(order.quantity) from Orders, Product, Customer where Product.group = ‘Houshold’ and Customer.type = external group by Order.region Orders Product Customer View Read Optimized
- 79. 79 Same problem but with services select Order.region, count(order.quantity) from Orders, Product, Customer where Product.group = ‘Houshold’ and Customer.type = external group by Order.region My Service View Read Optimized Orders Service Product Service Customer Service
- 80. 80 Use Materialized Views, but turned inside out!
- 81. 81 Or Embed Views with the Streams API Insert data Kafka Streams Business Logic KAFKA Orders Service Query in DSL
- 82. 82 On startup: Rewind and Replay Rewind Replay Orders Service
- 83. 83 Data Services: Distributed Materialized View Kafka Queryable State API Service KStreams View
- 84. 84 Create Views with Connect DB of choice Kafka Connect APIOrders Service Context Boundary Kafka
- 85. 85 10. Move Data to Code
- 86. 86 High Throughput Data Movement Service Service Service Service instance 1 Service instance 2 Service instance 3 Service instance 4 Kafka Cluster (many machines)
- 87. 87
- 88. 88 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Materialize Stock ‘View’ Inside Service KAFKA
- 89. 89 If messaging Remembers: Views don’t have to!
- 90. 90 11. Take only the data you need today
- 91. 91 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Take only the data you need! KAFKA
- 92. 92 Take only the data you need {“Stock Inventory”: { “Id”: “Foo1234”, “Vendor”: “Foo Industries”, “Description”: “This is a …”, “Delivery Category”: “ND”, “Stock Status”: [ “Items in Stock”: 53, “Items on Order”: 0 ]…etc…}
- 93. 93 Data Movement Be realistic: • Network is no longer the bottleneck • Indexing is: • In memory indexes help • Keep datasets focussed
- 94. 94 12. Use the log instead as a ‘database’ (for data-on-the-inside)
- 95. 95 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Save Internal State to the Log (Event Sourcing) KAFKA
- 96. 96 Append state as events Service Append KAFKA
- 97. 97 Rewind and Replay on startup Rewind Replay Service KAFKA
- 98. 98 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Order Service Loads Reserved Stocks on Startup KAFKA
- 99. 99 14. Use Transactions to tie Communication & State together
- 100. 100 OrderRequested (IPad) 2a. Order Validated 2c. Offset Commit 2b. IPad Reserved Internal State: Stock = 17 Reservations = 2 Tie Events & State with Transactions
- 101. 101 Connect TRANSACTION Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Stay Simple. Take Responsibility for the past KAFKA
- 102. 102 15. Compose Services as a Streaming Pipeline
- 103. 103 (1) Stateless Processor e.g. - Filter orders by region - Convert to local domain model - Simple rule
- 104. 104 (2) Stateless, Data Enabled Processor Similar to star schema • Facts are stream • Dimensions are GKTables e.g. Enrich Orders with Customer Info & Account details Stream GKTable GKTable Stream-Table Join
- 105. 105 (3) Gates e.g. rules engines or event correlation When Order & Payment then … Stream 1 Window Window Stream-Stream Join Stream 2
- 106. 106 (4) Stateful Processor e.g. - Average order amount by region - Posting Engine (reverse replace of previous position) State store backed up to Kafka
- 107. 107 (5) Stream-Aside Pattern public static void main(…){ … } JVM
- 109. 109 Combine them: 1. Stateless 2. Data enriched 3. Gates 4. Stateful processors 5. Stream-aside 6. Sidecar
- 110. 110 Group services into bounded contexts
- 112. 112 Richer Example
- 113. 113 All order logic, stateless service Orders-by-Customer view (KStreams + Queryable State API) UI Service Stream Maintenance & Monitoring Highly available, stateful service UI uses Orders-by- Customer View directly via Queryable State History Service pushes data to a local Elastic Search Instance Orders Service Derived View UI joins data Tables & Streams Fulfillment Service Analytics OrdersProduct Custo- mers KTables KTables KTables Schemas
- 114. 114 So…
- 115. 115 Good architectures have little to do with this:
- 116. 116 It’s about how systems evolves over time
- 117. 117 Request driven isn’t enough • High coupling • Hard to handle async flows • Hard to move and join datasets.
- 118. 118 Services need to be data enabled
- 119. 119 The decoupling effects of the broker make the architecture pluggable and data enabled Orders Customers Payments Stock
- 120. 120 ...with a single source of truth Orders Customers Payments Stock
- 121. 121 ...and many service specific views Orders Customers Payments Stock
- 122. 122 … which react to events as they occur Orders Customers Payments Stock
- 123. 123 Microservices enabled by a streaming platform • Loose coupling • Data Enabled • Event Driven • Operational transparency
- 124. 124 How do we get to 3.0? • Loose coupling • Embrace data • Event Driven • Operational transparency
- 126. 126 Tips and Tricks • Take only the data you need today • Limit the scope of request response interfaces • Build small services. Build big ones too. • Don’t fear the network. Fear the index. • The Sync-Async divide • Pure event driven with event streams and views. • Map request response to event driven (Leverage CQRS) • Too fine-grained services. • Too much data • KStreams rebalancing • Need for automation