SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
Building Event Driven Services with Stateful Streams
Event Driven Services come in many shapes and sizes from tiny event driven functions that dip into an event stream, right through to heavy, stateful services which can facilitate request response. This practical talk makes the case for building this style of system using Stream Processing tools. We also walk through a number of patterns for how we actually put these things together.
Event Driven Services come in many shapes and sizes from tiny event driven functions that dip into an event stream, right through to heavy, stateful services which can facilitate request response. This practical talk makes the case for building this style of system using Stream Processing tools. We also walk through a number of patterns for how we actually put these things together.
33.
33
How do we get to 3.0?
• Loose Coupling
• Data Enabled
• Event Driven
• Operational Transparency
34.
34
Useful Grid
Orders
Service
Payments
Service
Customers
Service
We need to join different datasets
35.
35
Useful Grid
Orders
Service
Payments
Service
Customers
Service
Need a better way to handle data the
data services expose
36.
36
How do we get to 3.0?
• Loose Coupling
• Data Enabled
• Event Driven
• Operational Transparency
37.
37
Useful Grid
Orders Payments
Customers
Why Poll when we can Push
38.
38
Useful Grid
Orders
Service
Payments
Service
Customers
Service
Streaming leads to being both event
driven and data enabled
Event Driven
Data Enabled
39.
39
How do we get to 3.0?
• Loose Coupling
• Data Enabled
• Event Driven
• Operational Transparency
41.
41
Ledger
Service
Loan
Validation
Shared Narrative
Fraud
Service
History
Service
Account
Service Account
History
Login
Service
Loan
Service
A Journal of Service Interactions
42.
42
How do we get to 3.0?
• Loose Coupling
• Data Enabled
• Event Driven
• Operational Transparency
43.
43
Tenet 1:
Events and Asynchronous processes
better model the way businesses work
44.
44
Tenet 2:
Services need to design for data-on-the-
outside
45.
45
Tenet 3:
The measure of a ‘good’ architecture is
it’s ability to evolve
62.
62KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Use Schemas to validate Backwards Compatibility
Connect
Products
Schema Registry
64.
64KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Apply the single writer principal
Connect
Products
Schema Registry
Order
Completed
65.
65
Single Writer Principal
- Creates local consistency points in
the absence of Global Consistency
- Makes schema upgrades easier to
manage.
67.
67KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Apply Bandwidth Limits to services
Connect
Products
Schema Registry
Order
Completed
100Mb/s
69.
69
Messaging that Remembers
Orders Customers
Payments
Stock
70.
70
Rewind and Replay
Rewind
Replay Orders
Service
71.
71
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Utilize Whole Datasets
Connect
Products
Schema Registry
Order
Completed Repricing
72.
72
Compacted Topic: Delete superceded messages that share the same key
K1
K1
K1
K2
K2
K2
K1
V1
V1
V2
V3
V2
V4
V3
Compacted Topics
73.
73
Orders Customers
Payments
Stock
Single, Shared Source of Truth
Data-on-the-Outside
76.
76
Materialized View
Create via a query
(select * from Orders
Where Region == ‘USA’)
Base
Table
Materialized Views in a DB
77.
77
Insert data Materialized View
Create via a query
(select * from Orders
Where Region == ‘USA’)
Query is rerun on data as it arrives
Base
Table
Materialized Views in a DB
78.
78
In practice often more complex
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = ‘Houshold’
and Customer.type = external
group by Order.region
Orders Product Customer
View
Read
Optimized
79.
79
Same problem but with services
select Order.region,
count(order.quantity)
from Orders, Product, Customer
where Product.group = ‘Houshold’
and Customer.type = external
group by Order.region
My Service View
Read
Optimized
Orders
Service
Product
Service
Customer
Service
80.
80
Use Materialized Views,
but turned inside out!
81.
81
Or Embed Views with the Streams API
Insert data
Kafka Streams
Business
Logic
KAFKA
Orders Service
Query
in DSL
82.
82
On startup: Rewind and Replay
Rewind
Replay
Orders Service
83.
83
Data Services: Distributed Materialized View
Kafka
Queryable
State API
Service
KStreams
View
84.
84
Create Views with Connect
DB of choice
Kafka
Connect
APIOrders
Service
Context
Boundary
Kafka
86.
86
High Throughput Data Movement
Service
Service
Service
Service instance 1
Service instance 2
Service instance 3
Service instance 4
Kafka Cluster
(many machines)
88.
88
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service Stock
Stock
Materialize Stock ‘View’ Inside Service
KAFKA
89.
89
If messaging Remembers:
Views don’t have to!
91.
91
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service Stock
Stock
Take only the data you need!
KAFKA
92.
92
Take only the data you need
{“Stock Inventory”: {
“Id”: “Foo1234”,
“Vendor”: “Foo Industries”,
“Description”: “This is a …”,
“Delivery Category”: “ND”,
“Stock Status”: [
“Items in Stock”: 53,
“Items on Order”: 0
]…etc…}
93.
93
Data Movement
Be realistic:
• Network is no longer the bottleneck
• Indexing is:
• In memory indexes help
• Keep datasets focussed
94.
94
12. Use the log instead as a ‘database’
(for data-on-the-inside)
95.
95
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Save Internal State to the Log (Event Sourcing)
KAFKA
96.
96
Append state as events
Service
Append
KAFKA
97.
97
Rewind and Replay on startup
Rewind
Replay Service
KAFKA
98.
98
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Order Service Loads Reserved Stocks on Startup
KAFKA
99.
99
14. Use Transactions to tie
Communication & State together
100.
100
OrderRequested
(IPad)
2a. Order Validated
2c. Offset Commit
2b. IPad Reserved
Internal State:
Stock = 17
Reservations = 2
Tie Events & State with Transactions
101.
101
Connect
TRANSACTION
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Stay Simple. Take Responsibility for the past
KAFKA
102.
102
15. Compose Services as a Streaming
Pipeline
103.
103
(1) Stateless Processor
e.g.
- Filter orders by region
- Convert to local domain model
- Simple rule
104.
104
(2) Stateless, Data Enabled Processor
Similar to star schema
• Facts are stream
• Dimensions are GKTables
e.g. Enrich Orders with Customer Info & Account details
Stream
GKTable
GKTable
Stream-Table
Join
105.
105
(3) Gates
e.g. rules engines or event correlation
When Order & Payment then …
Stream 1
Window
Window Stream-Stream
Join
Stream 2
106.
106
(4) Stateful Processor
e.g.
- Average order amount by region
- Posting Engine (reverse replace of previous position)
State store
backed up to Kafka
113.
113
All order logic,
stateless service
Orders-by-Customer view
(KStreams + Queryable State API)
UI Service
Stream
Maintenance &
Monitoring
Highly available, stateful service
UI uses Orders-by-
Customer View directly
via Queryable State
History Service pushes data to
a local Elastic Search Instance
Orders
Service
Derived
View
UI joins data
Tables & Streams
Fulfillment Service
Analytics
OrdersProduct Custo-
mers KTables
KTables KTables
Schemas
119.
119
The decoupling effects of the broker make
the architecture pluggable and data enabled
Orders Customers
Payments
Stock
120.
120
...with a single source of truth
Orders Customers
Payments
Stock
121.
121
...and many service specific views
Orders Customers
Payments
Stock
122.
122
… which react to events as they occur
Orders Customers
Payments
Stock
123.
123
Microservices enabled by a
streaming platform
• Loose coupling
• Data Enabled
• Event Driven
• Operational transparency
124.
124
How do we get to 3.0?
• Loose coupling
• Embrace data
• Event Driven
• Operational transparency
125.
125
Thank You
@benstopford
https://www.confluent.io/blog/tag/microservices/
126.
126
Tips and Tricks
• Take only the data you need today
• Limit the scope of request response interfaces
• Build small services. Build big ones too.
• Don’t fear the network. Fear the index.
• The Sync-Async divide
• Pure event driven with event streams and views.
• Map request response to event driven (Leverage CQRS)
• Too fine-grained services.
• Too much data
• KStreams rebalancing
• Need for automation