Building Event Driven Services with Kafka Streams

1. 1 Building Event Driven Services with Kafka Streams Ben Stopford Technologist, Office of the CTO @benstopford

2. 2 There is a book! http://bit.ly/designing-event-driven-systems

3. 3 Storage Messaging Connect KSQL Messaging Database

4. 4 Build Features Build for the Future

5. 5 Evolution!

6. 6 EcosystemsApp Increasingly we build ecosystems

7. 7 But these aren’t static, they grow 7

8. 8 Span data centers US East 1 US East 2

9. 9 Span regions or clouds

10. 10 Handle Disconnectedness

11. 11 Span teams and departments

12. 12 How do you do this?

13. 13 Microservices HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP HTTP

14. 14 Returns Service Email Service T2’’ Tax Service Order Service Or we can use event streams

15. 15 One Problem is DATA

16. 16 Most services share the same core facts. Catalog Most services live in here

17. 17 A Global API A Global Source of Truth Events

18. 18 Events have two hats Notification Data replication

19. 19 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication

20. 20 Buying an iPad (with REST/RPC) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver

21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA

22. 22 Pluggability Submit Order Order Created Orders Service Shipping Service Customer Service Webserver KAFKA Repricing getCustomer() REST Notification

23. 23 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA

24. 24 Stateless / Stateful Stream Processing Relates to these hats

25. 25 Message Broker (Kafka) Submit Order Order Created getCustomer() REST Orders Service Customer Service Webserver KAFKA KStreams Shipping Service Stateless Stream Processing Notification

26. 26 Message Broker (Kafka) Submit Order Order Created Orders Service Customer Service Webserver KAFKA KStreams Shipping Service KTable Customer Updated Stateful Stream Processing Data replication

27. 27 Events have two hats Notification Data Replication Stateless Stream Processing Stateful Stream Processing

28. 28 Kafka Streams is an API library 2

29. 29 Features • Buffering • Filters • Join (Stream-Stream, Stream-Table, Table- Table) • Aggregations (Sum, Count...) • Any arbitrary code • HA (dynamic scaling, standby replicas, checkpoints, etc.)

30. 30 Building a simple KStreams Email Service

31. 31 KAFKA Buffer 5 mins Send Email Stateless Streaming (stream-stream join)

32. 32 KAFKA Whole customer dataset cached in Email Service Stateful Streaming (Stream-Stream-Table join)

33. 33 KAFKA Whole customer dataset cached in Email Service Customer information in 3 places

34. 34 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service Order Created Order Validated Orders Service CQRS Materialized View

35. 35 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Order

36. 36 Validate Order kStreams.stream(“Orders”) .mapValues((key, order) -> isValid(order) ? pass() : fail()) .to(“OrderValidations”);

37. 37 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Validate Warehouse Inventory

38. 38 Ensure there is enough Stock Order Event Get # IPads in Stock - Validate Quantity - Reserve Item Atomic Compare and Set

39. 39 Table Stream State Store “Reserved Items” Topic Orders Topic Warehouse Stock Topic Orders Topic 1. Join

40. 40 Table Stream State Store “Reserved Items” Topic Orders Topic Order Validations Topic 2. Reserve Stock State store behaves like a little database Warehouse Stock Topic

41. 41 KStreams Code for Inventory Service //Create a State Store kStreams.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STOCK_STORE_NAME)); //Process kStreams.stream(“Orders”) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);

45. 45 Validate and update State Store public KeyValue transform(…) {… Long reserved = reservedStore.get(order.getProduct()); if (inWarehouse - reserved - order.quantity() >= 0) { int amount = reserved + order.getQuantity(); reservedStore.put(order.product(), amount); pass(); } else { fail() } … }

49. 49 Table Stream State Store “Reserved Items” Topic Orders Topic Inventory topic Order Validations Topic Transactional Wrapper State store behaves like a little database

50. 50 We need one more thing!

51. 51 Rekey by Product builder.stream(“Orders”) .selectKey((id, order) -> order.getProduct()) .join(inventory, KeyValue::new) .transform(InventoryValidator::new, STOCK_STORE_NAME) .to(“OrderValidations”);

52. 52 Rekey does two things in this case - Collocate data that must join - Serially execute critical section against single data.

53. 53 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” Rekey: Collocates data that must join JOIN JOIN

54. 54 Instance 1 Instance 2 Partitioned by ProductId Everything “iPad” Everything “iWatch” - Rekey: Serially execute critical section against single data. CaS CaS

55. 55 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy CQRS Validate Fraud

56. 56 Fraud Service “Fail” an order if the total value of orders for that user eclipses a threshold.

57. 57 Fraud Service CREATE STREAM OrderValidations AS SELECT userId, sum(amount) FROM Orders WINDOW TUMBLING(SIZE 60 MINUTE) GROUP BY userId HAVING sum(amount) > 10000;

58. 58 POST GET KAFKA Validate Order Validate Inventory Validate FraudView Proxy Mapping Synchronous and Asynchronous

59. 59 Create a materialized view kStreams.stream(“Orders”) .groupByKey() .reduce((agg, v) -> v, STORE_NAME));

60. 60 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service Order Created Order Validated Orders Service CQRS Materialized View Fine grained Ecosystem

61. 61 Gotchas/Guidelines - Use Schemas (Schema registry) - Global KTables & KTables are subtly different on replay. - Stream-table joins only join one way - The defaults aren’t enough. See resiliency settings in the documentation. - Releasing - Change application id trick: - loses committed offsets - Doesn’t delete topics, leaves them hanging - Name state store changelog topics - Start simple (Stateless)

62. 62 So…

63. 63 Good architectures have little to do with this:

64. 64 It’s about how systems evolves over time

65. 65 Events provide the key to evolutionary architectures Notification Data replication

66. 66 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication

67. 67 Streaming is the toolset for dealing with events as they move

68. 68 Thank You @benstopford Book: http://bit.ly/designing-event-driven-systems Twitter: @benstopford Slides: http://www.benstopford.com/