SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
The Kafka Summit version of this talk is more practical and includes code examples which walk though how to build a streaming application with Kafka Streams.
The Kafka Summit version of this talk is more practical and includes code examples which walk though how to build a streaming application with Kafka Streams.
16.
16
Most services share the same core facts.
Catalog
Most services live
in here
17.
17
A Global API
A Global Source of Truth
Events
18.
18
Events have two hats
Notification Data
replication
19.
19
Spectrum of use cases
Finer Grained,
Collaborative,
Connected
Courser Grained,
Non-collaborative,
Disconnected
Notification Data Replication
20.
20
Buying an iPad (with REST/RPC)
Submit
Order
shipOrder() getCustomer()
Orders
Service
Shipping
Service
Customer
Service
Webserver
21.
21
Events for Notification Only
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Notification
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
22.
22
Pluggability
Submit
Order
Order
Created
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
Repricing
getCustomer()
REST
Notification
23.
23
Events for Data Locality
Customer
Updated
Submit
Order
Order
Created
Data is
replicated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
24.
24
Stateless / Stateful Stream Processing
Relates to these hats
25.
25
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Orders
Service
Customer
Service
Webserver
KAFKA
KStreams
Shipping Service
Stateless Stream Processing
Notification
26.
26
Message Broker (Kafka)
Submit
Order
Order
Created
Orders
Service
Customer
Service
Webserver
KAFKA
KStreams
Shipping Service
KTable
Customer
Updated
Stateful Stream Processing
Data
replication
27.
27
Events have two hats
Notification
Data
Replication
Stateless Stream
Processing
Stateful Stream
Processing
32.
32
KAFKA
Whole customer
dataset cached in
Email Service
Stateful Streaming
(Stream-Stream-Table join)
33.
33
KAFKA
Whole customer
dataset cached in
Email Service
Customer information in 3 places
34.
34
POST
GET
Load
Balancer
ORDERSORDERS
OVTOPIC
Order
Validations
KAFKA
INVENTORY
Orders
Inventory
Fraud
Service
Order
Details
Service
Inventory
Service
Order
Created
Order
Validated
Orders
Service
CQRS
Materialized View
35.
35
POST
GET
KAFKA
Validate Order
Validate
Inventory
Validate FraudView
Proxy
CQRS
Validate Order
37.
37
POST
GET
KAFKA
Validate Order
Validate
Inventory
Validate FraudView
Proxy
Validate Warehouse Inventory
38.
38
Ensure there is enough Stock
Order Event
Get # IPads in
Stock
- Validate Quantity
- Reserve Item
Atomic Compare and Set
39.
39
Table
Stream
State Store
“Reserved Items” Topic
Orders
Topic
Warehouse Stock Topic
Orders Topic
1. Join
40.
40
Table
Stream
State Store
“Reserved Items” Topic
Orders
Topic
Order Validations Topic
2. Reserve Stock
State store behaves like a
little database
Warehouse Stock Topic
41.
41
KStreams Code for Inventory Service
//Create a State Store
kStreams.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STOCK_STORE_NAME));
//Process
kStreams.stream(“Orders”)
.join(inventory, KeyValue::new)
.transform(InventoryValidator::new, STOCK_STORE_NAME)
.to(“OrderValidations”);
42.
42
KStreams Code for Inventory Service
//Create a State Store
kStreams.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STOCK_STORE_NAME));
//Process
kStreams.stream(“Orders”)
.join(inventory, KeyValue::new)
.transform(InventoryValidator::new, STOCK_STORE_NAME)
.to(“OrderValidations”);
43.
43
KStreams Code for Inventory Service
//Create a State Store
kStreams.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STOCK_STORE_NAME));
//Process
kStreams.stream(“Orders”)
.join(inventory, KeyValue::new)
.transform(InventoryValidator::new, STOCK_STORE_NAME)
.to(“OrderValidations”);
44.
44
KStreams Code for Inventory Service
//Create a State Store
kStreams.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STOCK_STORE_NAME));
//Process
kStreams.stream(“Orders”)
.join(inventory, KeyValue::new)
.transform(InventoryValidator::new, STOCK_STORE_NAME)
.to(“OrderValidations”);
45.
45
Validate and update State Store
public KeyValue transform(…) {…
Long reserved = reservedStore.get(order.getProduct());
if (inWarehouse - reserved - order.quantity() >= 0) {
int amount = reserved + order.getQuantity();
reservedStore.put(order.product(), amount);
pass();
} else {
fail()
}
…
}
46.
46
Validate and update State Store
public KeyValue transform(…) {…
Long reserved = reservedStore.get(order.getProduct());
if (inWarehouse - reserved - order.quantity() >= 0) {
int amount = reserved + order.getQuantity();
reservedStore.put(order.product(), amount);
pass();
} else {
fail()
}
…
}
47.
47
Validate and update State Store
public KeyValue transform(…) {…
Long reserved = reservedStore.get(order.getProduct());
if (inWarehouse - reserved - order.quantity() >= 0) {
int amount = reserved + order.getQuantity();
reservedStore.put(order.product(), amount);
pass();
} else {
fail()
}
…
}
48.
48
Validate and update State Store
public KeyValue transform(…) {…
Long reserved = reservedStore.get(order.getProduct());
if (inWarehouse - reserved - order.quantity() >= 0) {
int amount = reserved + order.getQuantity();
reservedStore.put(order.product(), amount);
pass();
} else {
fail()
}
…
}
49.
49
Table
Stream
State Store
“Reserved Items” Topic
Orders
Topic
Inventory topic
Order Validations Topic
Transactional
Wrapper
State store behaves like a
little database
52.
52
Rekey does two things in this case
- Collocate data that must join
- Serially execute critical section
against single data.
53.
53
Instance 1 Instance 2
Partitioned by ProductId
Everything “iPad” Everything “iWatch”
Rekey: Collocates data that must join
JOIN JOIN
54.
54
Instance 1 Instance 2
Partitioned by ProductId
Everything “iPad” Everything “iWatch”
- Rekey: Serially execute critical section against
single data.
CaS CaS
55.
55
POST
GET KAFKA
Validate Order
Validate
Inventory
Validate FraudView
Proxy
CQRS
Validate Fraud
56.
56
Fraud Service
“Fail” an order if the total value of orders
for that user eclipses a threshold.
57.
57
Fraud Service
CREATE STREAM OrderValidations AS
SELECT userId, sum(amount)
FROM Orders
WINDOW TUMBLING(SIZE 60 MINUTE)
GROUP BY userId
HAVING sum(amount) > 10000;
58.
58
POST
GET
KAFKA
Validate Order
Validate
Inventory
Validate FraudView
Proxy
Mapping Synchronous and
Asynchronous
60.
60
POST
GET
Load
Balancer
ORDERSORDERS
OVTOPIC
Order
Validations
KAFKA
INVENTORY
Orders
Inventory
Fraud
Service
Order
Details
Service
Inventory
Service
Order
Created
Order
Validated
Orders
Service
CQRS
Materialized View
Fine grained Ecosystem
61.
61
Gotchas/Guidelines
- Use Schemas (Schema registry)
- Global KTables & KTables are subtly different on replay.
- Stream-table joins only join one way
- The defaults aren’t enough. See resiliency settings in the
documentation.
- Releasing
- Change application id trick:
- loses committed offsets
- Doesn’t delete topics, leaves them hanging
- Name state store changelog topics
- Start simple (Stateless)