Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Designing an Asynchronous Microservices Runtime with Kafka

52 views

Published on

David Peterson, Systems Engineer APAC, Confluent covers the origins of how Apache Kafka came to be along with how it now underpins some of the worlds largest microservice ecosystems. Kafka and the microservice movement gathered momentum at the same time, Kafka for streams or data-in-motion and microservices as a new, fine-grained architectural principle.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Designing an Asynchronous Microservices Runtime with Kafka

  1. 1. David Peterson Systems Engineer, APAC, Confluent
  2. 2. Designing an Asynchronous Microservices Runtime with Kafka: State, Scale and Performance David Peterson – Systems Engineering Manager, APAC @davidseth
  3. 3. Overview ! Kafka ! Streams API ! Kappa architecture ! Microservice Scaling and State
  4. 4. 33 “We believe that the major contributor to this complexity in many systems is the handling of state and the burden that this adds when trying to analyse and reason about the system.”
 
 Out of the tar pit, 2006
  5. 5. Kafka?
  6. 6. It’s a massively scalable distributed, fault tolerant, publish & subscribe key/value datastore with infinite data retention computing unbounded, streaming data in real time.
  7. 7. Store Process Publish & Subscribe
  8. 8. Produce Consume Store
  9. 9. Orders Kafka Streams API
  10. 10. What’s underneath?
  11. 11. Producer & Consumer APIConnect API Streams API Open-source client libraries for numerous languages. Direct integration with your systems. Reliable and scalable integration of Kafka with other systems – no coding required. Low-level and DSL, create applications & Microservices 
 to process your data in real-time
  12. 12. Streams
  13. 13. 21 3 StreamTopic Table
  14. 14. 1. Topic
  15. 15. {“actor”:”bear”, “x”:410, “y”:20} {“actor”:”racoon”, “x”:380, “y”:20}
  16. 16. {“actor”:”bear”, “x”:380, “y”:22} {“actor”:”racoon”, “x”:350, “y”:20}
  17. 17. {“actor”:”bear”, “x”:350, “y”:25} {“actor”:”racoon”, “x”:330, “y”:19}
  18. 18. {“actor”:”bear”, “x”:310, “y”:32} {“actor”:”racoon”, “x”:280, “y”:16}
  19. 19. 2. Stream
  20. 20. Exposure Sheet 3. Table
  21. 21. Changelog stream — immutable events
  22. 22. Rebuild original table
  23. 23. All of your data
 is a stream of events
  24. 24. stop...where is my database?
 (you said scaling data was hard)
  25. 25. Streams are your persistence model
 They are also your local database
  26. 26. Kappa
  27. 27. Big Data was
 The More the Better ValueofData Volume of Data Stream Data is
 The Faster the Better ValueofData Age of Data
  28. 28. Lambda Big OR Fast Speed Table Batch Table DB Streams Hadoop
  29. 29. Lambda Big OR Fast Speed Table Batch Table DB Streams Hadoop Kappa 
 Big AND Fast KSQL Stream Kafka HDFSCassandra Elastic Topic A Micro- service
  30. 30. Everything is a Stream “You can’t step into the same rivers twice as the waters are forever flowing” - Heraclitus
  31. 31. “Turning the Database Inside Out”
  32. 32. Kappa
  33. 33. Kappa ! Log Data Store — An append-only immutable log store is the canonical Source of Truth ! Stream Processor— Data is consumed from the log store into a streaming computation system for real time processing ! Materialised Views — Provide optimised responses to queries. These views aren’t used as canonical stores: at any point, you can wipe them and regenerate them from the canonical data store
  34. 34. Kappa Big- Query Elastic ConsumersMicroservice } MongoCDC Events IoT Producers } Kafka Connect Kafka Connect Kafka Streams
  35. 35. Kappa — Materialised Views
  36. 36. Kafka is your escape hatch
  37. 37. Old world Event processor New world Stream processor
  38. 38. Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver - Orders Service calls Shipping Service to tell it to ship item. - Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢 ECommerce Microservices (with RPC)
  39. 39. Refactoring Orders and Shipping Submit Order Order Created getCustomer() RPC Orders Service Shipping Service Customer Service Webserver - Orders Service no longer knows about the Shipping Service (or any other service). - Events are fire and forget. Orders Topic
  40. 40. Refactoring Customers Customer Updated Submit Order Order Created Orders Service Shipping Service Customer Service Webserver - Call to Customer service is gone. - Instead data in replicated, as events, into the shipping service, where it is queried locally. Orders Topic Shipping Topic Customer Topic (via CDC)
  41. 41. Orders Table Customer Kafka Streams API Stream Processor — KStreams
  42. 42. Stream Processor — KSQL CREATE TABLE possible_fraud AS
 SELECT card_number, COUNT(*)
 FROM authorisation_attempts 
 WINDOW TUMBLING (SIZE 5 SECONDS)
 GROUP BY card_number
 HAVING COUNT(*) > 3; … per 5 sec windows Aggregate data
  43. 43. Stream Processor — KSQL CREATE TABLE possible_fraud AS
 SELECT card_number, COUNT(*)
 FROM authorisation_attempts 
 WINDOW TUMBLING (SIZE 5 SECONDS)
 GROUP BY card_number
 HAVING COUNT(*) > 3; … per 5 sec windows Aggregate data
  44. 44. Scaling and state handling
  45. 45. • Domain Data - Change Data Capture (database txn log) • Time series - IoT, sensors, metrics • Domain Events - Microservices, user actions Different types of event models
  46. 46. Ratings Service operating on the event stream ratings KStreams API used to count and sum ratings as they arrive Ratings Service raw-ratings ratings ratings-counts ratings-sums Browser Webserver
  47. 47. Consumer Group { user: 100 type: rating movie-id: 389 rating: 3 comment: ‘yuck’ } key# Partitions Partition assignment Partitions give you horizontal scale {...} Ratings Service
  48. 48. Consumer Group { user: 100 type: rating movie-id: 389 rating: 3 comment: ‘yuck’ } key# Partitions Partition assignment Partitions give you horizontal scale {...} {...} {...} Ratings Service
  49. 49. Access legacy relation data ratingsRatings Service Connect Sharing state, CDC captures events from your canonical system into the event stream raw-ratings ratings ratings-counts ratings-sums movies (CDC) Browser Webserver CDC
  50. 50. KTables - Materialised State inside the app ratingsRatings Service Lookup table created inside the Rating ServiceMovies Connect raw-ratings ratings ratings-counts ratings-sums movies (CDC) Browser Webserver CDC
  51. 51. Consumer Group Topic: movies (from CDC) partition 1 Interactive query partition 3 Scaling state & querying Ratings Service partition 2 One Microservice running Compacted Topic: movies-ratings-service
  52. 52. Consumer Group Topic: movies (from CDC) partition 1 partition 2 partition 3 Scaling state & querying Scale up: Three Microservices running Ratings Service Compacted Topic: movies-ratings-service
  53. 53. Consumer Group Topic: movies (from CDC) partition 1 partition 2 partition 3 Scaling state & querying Scale up: Three Microservices running Ratings Service Compacted Topic: movies-ratings-service
  54. 54. Create a new table, persist it to the log Browser Writable table created for rated movies Webserver ratingsRatings Service Movies Connect raw-ratings ratings ratings-counts ratings-sums movies (CDC) rated-movies rated-movies CDC
  55. 55. 1. Provide simplicity (and time travel) 2. Handle state (via Kafka Streams) 3. Provide a new paradigm: contextual data enrichment & logic processing Stream processor Key takeaways
  56. 56. Thank You @davidseth

×