Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Event Driven Services with Kafka Streams

2,389 views

Published on

The Kafka Summit version of this talk is more practical and includes code examples which walk though how to build a streaming application with Kafka Streams.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building Event Driven Services with Kafka Streams

  1. 1. 1 Building Event Driven Services with Kafka Streams Ben Stopford Technologist, Office of the CTO @benstopford
  2. 2. 2 There is a book! http://bit.ly/designing-event-driven-systems
  3. 3. 3 Storage Messaging Connect KSQL Messaging Database
  4. 4. 4 Build Features Build for the Future
  5. 5. 5 Evolution!
  6. 6. 6 EcosystemsApp Increasingly we build ecosystems
  7. 7. 7 But these aren’t static, they grow 7
  8. 8. 8 Span data centers US East 1 US East 2
  9. 9. 9 Span regions or clouds
  10. 10. 10 Handle Disconnectedness
  11. 11. 11 Span teams and departments
  12. 12. 12 How do you do this?
  13. 13. 13 Microservices HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP
  14. 14. 14 Returns Service Email Service T2’’ Tax Service Order Service Or we can use event streams
  15. 15. 15 One Problem is DATA
  16. 16. 16 Most services share the same core facts. Catalog Most services live in here
  17. 17. 17 A Global API A Global Source of Truth Events
  18. 18. 18 Events have two hats Notification Data replication
  19. 19. 19 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
  20. 20. 20 Buying an iPad (with REST/RPC) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
  21. 21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
  22. 22. 22 Pluggability Submit Order Order Created Orders Service Shipping Service Customer Service Webserver KAFKA Repricing getCustomer() REST Notification
  23. 23. 23 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
  24. 24. 24 Stateless / Stateful Stream Processing Relates to these hats
  25. 25. 25 Message Broker (Kafka) Submit Order Order Created getCustomer() REST Orders Service Customer Service Webserver KAFKA KStreams Shipping Service Stateless Stream Processing Notification
  26. 26. 26 Message Broker (Kafka) Submit Order Order Created Orders Service Customer Service Webserver KAFKA KStreams Shipping Service KTable Customer Updated Stateful Stream Processing Data replication
  27. 27. 27 Events have two hats Notification Data Replication Stateless Stream Processing Stateful Stream Processing
  28. 28. 28 Kafka Streams is an API library 2
  29. 29. 29 Features • Buffering • Filters • Join (Stream-Stream, Stream-Table, Table- Table) • Aggregations (Sum, Count...) • Any arbitrary code • HA (dynamic scaling, standby replicas, checkpoints, etc.)
  30. 30. 30 Building a simple KStreams Email Service
  31. 31. 31 KAFKA Buffer 5 mins Send Email Stateless Streaming (stream-stream join)
  32. 32. 32 KAFKA Whole customer dataset cached in Email Service Stateful Streaming (Stream-Stream-Table join)
  33. 33. 33 KAFKA Whole customer dataset cached in Email Service Customer information in 3 places
  34. 34. 34 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service Order Created Order Validated Orders Service CQRS Materialized View
  35. 35. 35 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Order
  36. 36. 36 Validate Order kStreams.stream(“Orders”) .mapValues((key, order) -> isValid(order) ? pass() : fail()) .to(“OrderValidations”);
  37. 37. 37 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Validate Warehouse Inventory
  38. 38. 38 Ensure there is enough Stock Order Event Get # IPads in Stock - Validate Quantity - Reserve Item Atomic Compare and Set
  39. 39. 39 Table Stream State Store “Reserved Items” Topic Orders Topic Warehouse Stock Topic Orders Topic 1. Join
  40. 40. 40 Table Stream State Store “Reserved Items” Topic Orders Topic Order Validations Topic 2. Reserve Stock State store behaves like a little database Warehouse Stock Topic
  41. 41. 41 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
  42. 42. 42 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
  43. 43. 43 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
  44. 44. 44 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
  45. 45. 45 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
  46. 46. 46 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
  47. 47. 47 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
  48. 48. 48 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }
  49. 49. 49 Table Stream State Store “Reserved Items” Topic Orders Topic Inventory topic Order Validations Topic Transactional Wrapper State store behaves like a little database
  50. 50. 50 We need one more thing!
  51. 51. 51 Rekey by Product builder.stream(“Orders”) .selectKey((id, order) -> order.getProduct()) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);
  52. 52. 52 Rekey does two things in this case - Collocate data that must join - Serially execute critical section against single data.
  53. 53. 53 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” Rekey: Collocates data that must join JOIN JOIN
  54. 54. 54 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” - Rekey: Serially execute critical section against single data. CaS CaS
  55. 55. 55 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Fraud
  56. 56. 56 Fraud Service “Fail” an order if the total value of orders for that user eclipses a threshold.
  57. 57. 57 Fraud Service CREATE STREAM OrderValidations AS SELECT userId, sum(amount) FROM Orders WINDOW TUMBLING(SIZE 60 MINUTE) GROUP BY userId HAVING sum(amount) > 10000;
  58. 58. 58 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Mapping Synchronous and Asynchronous
  59. 59. 59 Create a materialized view kStreams.stream(“Orders”) .groupByKey() .reduce((agg, v) -> v, STORE_NAME));
  60. 60. 60 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service Order Created Order Validated Orders Service CQRS Materialized View Fine grained Ecosystem
  61. 61. 61 Gotchas/Guidelines - Use Schemas (Schema registry) - Global KTables & KTables are subtly different on replay. - Stream-table joins only join one way - The defaults aren’t enough. See resiliency settings in the documentation. - Releasing - Change application id trick: - loses committed offsets - Doesn’t delete topics, leaves them hanging - Name state store changelog topics - Start simple (Stateless)
  62. 62. 62 So…
  63. 63. 63 Good architectures have little to do with this:
  64. 64. 64 It’s about how systems evolves over time
  65. 65. 65 Events provide the key to evolutionary architectures Notification Data replication
  66. 66. 66 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
  67. 67. 67 Streaming is the toolset for dealing with events as they move
  68. 68. 68 Thank You @benstopford Book: http://bit.ly/designing-event-driven-systems Twitter: @benstopford Slides: http://www.benstopford.com/

×