Just for you: FREE 60-day trial to the world’s largest digital library.
The SlideShare family just got bigger. Enjoy access to millions of ebooks, audiobooks, magazines, and more from Scribd.
Cancel anytime.More Related Content
Related Books
Free with a 14 day trial from Scribd
Building Event Driven Services with Kafka Streams
- 1. 1 Building Event Driven Services with Kafka Streams Ben Stopford Technologist, Office of the CTO @benstopford
- 2. 2 There is a book! http://bit.ly/designing-event-driven-systems
- 3. 3 Storage Messaging Connect KSQL Messaging Database
- 4. 4 Build Features Build for the Future
- 5. 5 Evolution!
- 6. 6 EcosystemsApp Increasingly we build ecosystems
- 7. 7 But these aren’t static, they grow 7
- 8. 8 Span data centers US East 1 US East 2
- 9. 9 Span regions or clouds
- 10. 10 Handle Disconnectedness
- 11. 11 Span teams and departments
- 12. 12 How do you do this?
- 13. 13 Microservices HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP
- 14. 14 Returns Service Email Service T2’’ Tax Service Order Service Or we can use event streams
- 15. 15 One Problem is DATA
- 16. 16 Most services share the same core facts. Catalog Most services live in here
- 17. 17 A Global API A Global Source of Truth Events
- 18. 18 Events have two hats Notification Data replication
- 19. 19 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
- 20. 20 Buying an iPad (with REST/RPC) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
- 21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
- 22. 22 Pluggability Submit Order Order Created Orders Service Shipping Service Customer Service Webserver KAFKA Repricing getCustomer() REST Notification
- 23. 23 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
- 24. 24 Stateless / Stateful Stream Processing Relates to these hats
- 25. 25 Message Broker (Kafka) Submit Order Order Created getCustomer() REST Orders Service Customer Service Webserver KAFKA KStreams Shipping Service Stateless Stream Processing Notification
- 26. 26 Message Broker (Kafka) Submit Order Order Created Orders Service Customer Service Webserver KAFKA KStreams Shipping Service KTable Customer Updated Stateful Stream Processing Data replication
- 27. 27 Events have two hats Notification Data Replication Stateless Stream Processing Stateful Stream Processing
- 28. 28 Kafka Streams is an API library 2
- 29. 29 Features • Buffering • Filters • Join (Stream-Stream, Stream-Table, Table- Table) • Aggregations (Sum, Count...) • Any arbitrary code • HA (dynamic scaling, standby replicas, checkpoints, etc.)
- 30. 30 Building a simple KStreams Email Service
- 31. 31 KAFKA Buffer 5 mins Send Email Stateless Streaming (stream-stream join)
- 32. 32 KAFKA Whole customer dataset cached in Email Service Stateful Streaming (Stream-Stream-Table join)
- 33. 33 KAFKA Whole customer dataset cached in Email Service Customer information in 3 places
- 34. 34 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service Order Created Order Validated Orders Service CQRS Materialized View
- 35. 35 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Order
- 36. 36 Validate Order kStreams.stream(“Orders”) .mapValues((key, order) -> isValid(order) ? pass() : fail()) .to(“OrderValidations”);
- 37. 37 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Validate Warehouse Inventory
- 38. 38 Ensure there is enough Stock Order Event Get # IPads in Stock - Validate Quantity - Reserve Item Atomic Compare and Set
- 39. 39 Table Stream State Store “Reserved Items” Topic Orders Topic Warehouse Stock Topic Orders Topic 1. Join
- 40. 40 Table Stream State Store “Reserved Items” Topic Orders Topic Order Validations Topic 2. Reserve Stock State store behaves like a little database Warehouse Stock Topic
- 41. 41 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 42. 42 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 43. 43 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 44. 44 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 45. 45 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 46. 46 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 47. 47 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 48. 48 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
- 49. 49 Table Stream State Store “Reserved Items” Topic Orders Topic Inventory topic Order Validations Topic Transactional Wrapper State store behaves like a little database
- 50. 50 We need one more thing!
- 51. 51 Rekey by Product builder.stream(“Orders”) .selectKey((id, order) -> order.getProduct()) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
- 52. 52 Rekey does two things in this case - Collocate data that must join - Serially execute critical section against single data.
- 53. 53 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” Rekey: Collocates data that must join JOIN JOIN
- 54. 54 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” - Rekey: Serially execute critical section against single data. CaS CaS
- 55. 55 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Fraud
- 56. 56 Fraud Service “Fail” an order if the total value of orders for that user eclipses a threshold.
- 57. 57 Fraud Service CREATE STREAM OrderValidations AS SELECT userId, sum(amount) FROM Orders WINDOW TUMBLING(SIZE 60 MINUTE) GROUP BY userId HAVING sum(amount) > 10000;
- 58. 58 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Mapping Synchronous and Asynchronous
- 59. 59 Create a materialized view kStreams.stream(“Orders”) .groupByKey() .reduce((agg, v) -> v, STORE_NAME));
- 60. 60 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service Order Created Order Validated Orders Service CQRS Materialized View Fine grained Ecosystem
- 61. 61 Gotchas/Guidelines - Use Schemas (Schema registry) - Global KTables & KTables are subtly different on replay. - Stream-table joins only join one way - The defaults aren’t enough. See resiliency settings in the documentation. - Releasing - Change application id trick: - loses committed offsets - Doesn’t delete topics, leaves them hanging - Name state store changelog topics - Start simple (Stateless)
- 62. 62 So…
- 63. 63 Good architectures have little to do with this:
- 64. 64 It’s about how systems evolves over time
- 65. 65 Events provide the key to evolutionary architectures Notification Data replication
- 66. 66 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
- 67. 67 Streaming is the toolset for dealing with events as they move
- 68. 68 Thank You @benstopford Book: http://bit.ly/designing-event-driven-systems Twitter: @benstopford Slides: http://www.benstopford.com/