Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Microservices in the Apache Kafka Ecosystem

14,951 views

Published on

In this presentation, Ben Stopford, Engineer, Confluent discusses the world of Microservices in the streaming world.

Published in: Software
  • You might also like this slide 'Apache Kafka vs MapR-ES: Fit for purpose/Decision tree': https://www.slideshare.net/sbaltagi/apache-kafka-vs-mapres-fit-for-purposedecision-tree
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Microservices in the Apache Kafka Ecosystem

  1. 1. Ben Stopford, Engineer, Confluent
  2. 2. • Event Driven Microservices • The toolset: Kafka, KStreams, Connect • 10 Principals for Streaming Services What we’ll cover
  3. 3. Microservices in the Kafka Ecosystem
  4. 4. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Microservices
  5. 5. So why Services?
  6. 6. The Monolith Can we do reuse, encapsulation?
  7. 7. The Monolith What happens when we grow?
  8. 8. Companies are inevitably a collection of applications They must work together to some degree
  9. 9. Inverse Conway Maneuver The 'Inverse Conway Maneuver' recommends evolving your team and organizational structure to promote your desired architecture. Org Structure Software Architecture
  10. 10. Eschew shared, mutable state
  11. 11. Service based approaches separate state
  12. 12. But state must inevitably be shared between services
  13. 13. Use a toolkit that embraces decentralisation
  14. 14. Some simple patterns of distributed systems
  15. 15. Request / Response
  16. 16. When do we need Request Response? Looking things up What is in my shopping basket?
  17. 17. Event Driven Async / Fire and Forget / Brokered
  18. 18. Businesses are often modeled as a sequence events. Add to cart Buy Make Payment Update Stock Email Conf Package Item Dispatch Fraud
  19. 19. When do we need Event Driven?
  20. 20. SOA / Microservices Message Broker Event Based Request/Response
  21. 21. Hybrids Event- Based Request/ Response
  22. 22. Request Response Buy Items Order Service Stock Service Ful- filment Service Fraud Det- ection Mmm… New IPad REST etc UI Service
  23. 23. Event Driven Buy Items Order Service Stock Service Ful- filment Service Fraud Det- ection Message Broker Mmm… New IPad UI Service
  24. 24. Hybrid Buy Items Order Service Stock Service Ful- filment Service Fraud Det- ection Message Broker Mmm… New IPad REST UI Service
  25. 25. As software engineers we are inevitably affected by the tools we surround ourselves with. Languages, frameworks, even processes all act to shape the software we build.
  26. 26. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service The tools we choose have a big effect on our architecture GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Event Driven Request Response
  27. 27. Kafka is well suited to Event Driven Architectures It will lead you down that path
  28. 28. The Tool Set
  29. 29. a Distributed Log
  30. 30. What is a Distributed Log?
  31. 31. Shard on the way in Producing Services Kafka Consuming Services
  32. 32. Each shard is a queue Producing Services Kafka Consuming Services
  33. 33. Consumers share load Producing Services Kafka Consuming Services
  34. 34. Reduces to a globally ordered queue
  35. 35. Load Balanced Services
  36. 36. Fault Tolerant Services
  37. 37. Build ‘Always On’ Services Rely on Fault Tolerant Broker
  38. 38. Services can “Rewind & Replay” the log Rewind & Replay
  39. 39. Compacted Log (retains only latest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
  40. 40. A database engine for data-in-flight
  41. 41. Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second Continuously Running Queries
  42. 42. What is stream processing engine? Data Index Query Engine Query Engine vs Database Finite source Stream Processor Infinite source
  43. 43. Windowing For unordered or unpredictable streams Sliding Fixed (tumbling)
  44. 44. Features: similar to database query engine JoinFilter Aggr- egate View Window
  45. 45. Stateful Stream Processing stream Compacted stream Join Stream Data Stream-Tabular Data Windowed Stream Locally Cached Table (disk resident) KafkaKafka Streams
  46. 46. Useful for Enrichment stream Compacted stream Join Orders Customers KafkaKafka Streams
  47. 47. Scales Out
  48. 48. Embeddable Orders Service Kafka Orders Service Orders Service
  49. 49. Kafka Connect View Replication
  50. 50. Sometimes you need to physically move data
  51. 51. Replicate it, so both copies are identical
  52. 52. Iterate via regeneration
  53. 53. Kafka Connect Kafka Connect Kafka Connect Kafka
  54. 54. So…
  55. 55. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful
  56. 56. Embeddable tool for data manipulation Adapt data streams (transformation, stream maintenance, analytic function)
  57. 57. Replicate Data Sources Exactly Create Regenerable, Streaming Views
  58. 58. How do we actually do this? 10 (opinionated) principals for Streaming Services
  59. 59. 1. Don’t use Kafka for shopping carts! (OK, you can, but use sparingly) Shopping Cart UI Service Broker/durability/broadcast add little to request response
  60. 60. Do use Kafka for your core business processing. Think “business events”. An order was created, a payment was received, a trade was booked etc. Pub/Sub. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
  61. 61. 2. Pick Topics with Business Significance Orders Payments Returns Invoices
  62. 62. Give your messages meaningful IDs and version them OrdersService1-Order-1234-v2 Should relate to the real world Should be Versioned (if mutable) Include the service name Note the key used for sharding in Kafka may not be this key
  63. 63. 3. Decouple publishers from subscribers GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
  64. 64. Add Request/Response only where needed GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service REST
  65. 65. 4. Use the log to regenerate state Avoid journaling incoming events
  66. 66. Event Source side effects • Use offsets to tie these back to the stream • Store in: • Kafka • Kstreams state store • Other DB
  67. 67. 5. Apply the Single Writer Principal • Change at source (by calling that service) • Let the change propagate back • Keep local copies read only. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service (1) Change Orders at source (2) Let the change propagate through
  68. 68. 6. Leverage keeping datasets inside the broker customer country supplier ex- rates
  69. 69. Leverage keeping only the latest version (table view) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
  70. 70. Join & Process on the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
  71. 71. 7. Prefer stream processing over maintaining historic views UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Orders Historic “copies” diverge
  72. 72. Join & Process on the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
  73. 73. 8. Sometimes you need historic views. => Replicate & Keep Read Only
  74. 74. Replicate Kafka Connect Kafka Connect Kafka Keep me Read Only
  75. 75. Iterate
  76. 76. Polyglotic Persistence
  77. 77. 9. Use Schemas (especially if data is retained)
  78. 78. Schemaless data doesn’t age well
  79. 79. Messages
  80. 80. Confluent Schema Registry can help Orders Service Schema Registry Email Service Avro
  81. 81. 10. Consider “Stream Management” Services Stream Management Kafka • Retaining data => Admin tasks • Similar to the role of a DBA • Data Migration • Repartitioning • Latest/versioned • Environment Management • CQRS
  82. 82. KStreams is a good toolset for this
  83. 83. Stream Management KStreams Latest Stream Versioned Stream
  84. 84. So… 1. Don’t use Kafka for shopping carts! 2. Pick Topics with Business Significance 3. Decouple publishers from subscribers 4. Use the log to regenerate state 5. Apply the Single Writer Principal 6. Leverage keeping datasets inside the broker 7. Prefer stream processing over maintaining historic views 8. Sometimes you need historic views. => Replicate Read Only 9. Use Schemas 10. Consider “Stream Management” Services
  85. 85. Microservices push us away from shared, mutable state
  86. 86. But state needs to be communicated
  87. 87. In an increasingly data-heavy world we need tools to do this efficiently
  88. 88. …and in real time.
  89. 89. We need a data-centric toolset to do this All Your Data Immutable Streams Event Driven Services Stream Management Services Legacy CDC View Replication Polyglotic Persistence tables streams Streaming Services
  90. 90. Keep it simple, Keep it moving
  91. 91. @benstopford Thanks!

×