Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

700 views

Published on

Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

Published in: Technology

Building Event Driven Services with Apache Kafka and Kafka Streams - Devoxx Belgium

  1. 1. 1 Building Event Driven Services with Apache Kafka and Kafka Streams Ben Stopford @benstopford
  2. 2. 2 Build Features Build for the Future
  3. 3. 3 Evolution!
  4. 4. 4 KAFKA Serving Layer (Cassandra etc.) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
  5. 5. 5 authorization_attempts possible_fraud Streaming Example
  6. 6. 6 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  7. 7. 7 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  8. 8. 8 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  9. 9. 9 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  10. 10. 10 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  11. 11. 11 CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
  12. 12. 12 Streaming == Manipulating Data in Flight
  13. 13. 13 Business Applications
  14. 14. 14 EcosystemsApp Increasingly we build ecosystems
  15. 15. 15 SOA / Microservices / EDA Customer Service Shipping Service
  16. 16. 16 The Problem is DATA
  17. 17. 17 Most services share the same core facts. Catalog Most services live in here
  18. 18. 18 So how do we share data between services? Orders Service Shipping Service Customer Service Webserver
  19. 19. 19 Buying an iPad (with REST) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
  20. 20. 20 Buying an iPad with Events Message Broker (Kafka) Notification Data is replicated (incrementally) Submit Order Order Created Customer Updated Orders Service Shipping Service Customer Service Webserver KAFKA
  21. 21. 21 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
  22. 22. 22 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
  23. 23. 23 Events have two hats Notification Data replication
  24. 24. 24 Events are the key to scalable service ecosystems
  25. 25. 25 Streaming is the toolset for dealing with events as they move!
  26. 26. 26 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  27. 27. 27 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  28. 28. 28 What is a Distributed Log?
  29. 29. 29 Shard on the way in Producing Services Kafka Consuming Services
  30. 30. 30 Each shard is a queue Producing Services Kafka Consuming Services
  31. 31. 31 Consumers share load Producing Services Kafka Consuming Services
  32. 32. 32 Reduces to a globally ordered queue
  33. 33. 33 Load Balanced Services
  34. 34. 34 Fault Tolerant Services
  35. 35. 35 Build ‘Always On’ Services Rely on Fault Tolerant Broker
  36. 36. 36 Services can “Rewind & Replay” the log Rewind & Replay
  37. 37. 37 Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
  38. 38. 38 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  39. 39. 39 Kafka Connect Kafka Connect Kafka Connect Kafka
  40. 40. 40 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  41. 41. 41 A database engine for data-in-flight
  42. 42. 42 SELECT card_number, count(*) FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; Continuously Running Queries
  43. 43. 43 Features: similar to database query engine JoinFilter Aggr- egate View Window
  44. 44. 44 KAFKA Buffer 5 mins Windows
  45. 45. 45 Lookup tables KAFKA
  46. 46. 46 Stateful Stream Processing stream Compacted stream Join Stream Table KafkaKafka Streams
  47. 47. 47 Scales Out
  48. 48. 48 Embeddable Orders Service Kafka Orders Service Orders Service
  49. 49. 49 Streaming is about 1. Processing data incrementally 2. Moving data to where it needs to be processed (quickly and efficiently) On Notification Data Replication
  50. 50. 50 Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
  51. 51. 51 Steps to Streaming Services
  52. 52. 52 1. Take Responsibility for the past and evolve
  53. 53. 53 Stay Simple. Take Responsibility for the past Browser Webserver
  54. 54. 54 Evolve Forwards Browser Webserver Orders Service
  55. 55. 55 2. Raise events. Don’t talk to services.
  56. 56. 56 Events have two hats Notification Data replication
  57. 57. 57 Raise events. Don’t talk to services Browser Webserver Orders Service
  58. 58. 58 KAFKA Order Requested Order Received Browser Webserver Orders Service Raise events. Don’t talk to services
  59. 59. 59 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Raise events. Don’t talk to services
  60. 60. 60 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Kafka as a Backbone for Events
  61. 61. 61 3. Use Connect (& CDC) to evolve away from legacy
  62. 62. 62 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Evolve away from Legacy
  63. 63. 63KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use the Database as a ‘Seam’ Connect Products
  64. 64. 64 4. Make use of Schemas
  65. 65. 65KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Use Schemas to validate Backwards Compatibility Connect Products Schema Registry
  66. 66. 66 5. Use the Single Writer Principal
  67. 67. 67KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Apply the single writer principal Connect Products Schema Registry Order Completed
  68. 68. 68 Orders Service Email Service T1 T2 T3 T4 REST Service T5 Single Writer Principal
  69. 69. 69 Single Writer Principal - Creates local consistency points in the absence of Global Consistency - Makes schema upgrades easier to manage.
  70. 70. 70 6. Store Datasets in the Log
  71. 71. 71 Messaging that Remembers Orders Customers Payments Stock
  72. 72. 72 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service New Service, No Problem! Connect Products Schema Registry Order Completed Repricing
  73. 73. 73 Orders Customers Payments Stock Single, Shared Source of Truth
  74. 74. 74 But how do you query a log?
  75. 75. 75 7. Move Data to Code
  76. 76. 76
  77. 77. 77 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Materialize Stock ‘View’ Inside Service KAFKA
  78. 78. 78 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Take only the data we need KAFKA
  79. 79. 79 Data Movement Be realistic: • Network is no longer the bottleneck • Indexing is: • In memory indexes help • Keep datasets focused
  80. 80. 80 8. Use the log instead as a ‘database’
  81. 81. 81 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Apply Event Sourcing KAFKA Table
  82. 82. 82 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Order Service Loads Reserved Stocks on Startup KAFKA
  83. 83. 83 Kafka has several features for reducing the need to move data on startup - Standby Replicas - Disk Checkpoints - Compacted topics
  84. 84. 84 9. Use Transactions to tie All Interactions Together
  85. 85. 85 OrderRequested (IPad) 2a. Order Validated 2c. Offset Commit 2b. IPad Reserved Internal State: Stock = 17 Reservations = 2 Tie Events & State with Transactions
  86. 86. 86 Connect TRANSACTION Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Transactions KAFKA
  87. 87. 87 10. Bridge the Sync/Async Divide with a Streaming Ecosystem
  88. 88. 88 POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service (see previous figure) Order Created Order Validated Orders View Q in CQRS Orders Service C is CQRS Services in the Micro: Orders Service Find the code online!
  89. 89. 89 Orders Customers Payments Stock Query Engine (Kstreams/KSQL) Larger Ecosystems HISTORICAL EVENT STREAMS
  90. 90. 90 Kafka KAFKA New York Tokyo London Global / Disconnected Ecosystems
  91. 91. 91 So…
  92. 92. 92 Good architectures have little to do with this:
  93. 93. 93 It’s about how systems evolves over time
  94. 94. 94 Request driven isn’t enough • High coupling • Hard to handle async flows • Hard to move and join datasets.
  95. 95. 95 Leverage the Duality of Events Notification Data replication
  96. 96. 96 With a toolset built for data in flight
  97. 97. 97 • Broadcast events • Retain them in the log • Evolve the event-stream with streaming functions • Recasting the event stream into views when you need to query. Event Driven Services
  98. 98. 98 Confluent Streaming Platform
  99. 99. 99 Thank You @benstopford Blog Series: https://www.confluent.io/blog/tag/microservices/

×