Downloaded 12 times
Uploaded byBen Stopford
10 Principals for Effective Event Driven Microservices
The document discusses the principles and best practices for building event-driven microservices using Kafka and related tools. It emphasizes the need for a decentralized approach to architecture, efficient state management, and the importance of using Kafka for core business processing instead of traditional shopping cart systems. Ten key principles for streaming services are outlined to guide developers in implementing robust, scalable solutions within the Kafka ecosystem.
More Related Content
Similar to 10 Principals for Effective Event Driven Microservices
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
10 Principals for Effective Event Driven Microservices
- 1. Ben Stopford, Engineer,Confluent
- 2. • Event DrivenMicroservices • The toolset: Kafka, KStreams, Connect • 10 Principals for Streaming Services What we’ll cover
- 3.
- 4.
- 5.
- 6. The Monolith Can wedo reuse, encapsulation?
- 7. The Monolith What happenswhen we grow?
- 8. Companies are inevitably acollection of applicationsThey must work together to some degree
- 9. Inverse Conway Maneuver The'Inverse Conway Maneuver' recommends evolving your team and organizational structure to promote your desired architecture. Org Structure Software Architecture
- 10.
- 11.
- 12. But state mustinevitably be shared between services
- 13. Use a toolkitthat embraces decentralisation
- 15. Some simple patternsof distributed systems
- 16.
- 17. When do weneed Request Response? Looking things up What is in my shopping basket?
- 18. Event Driven Async /Fire and Forget / Brokered
- 19. Businesses are often modeledas a sequence events. Add to cart Buy Make Payment Update Stock Email Conf Package Item Dispatch Fraud
- 20. When do weneed Event Driven?
- 21. SOA / Microservices Message Broker EventBased Request/Response
- 22.
- 23.
- 24.
- 25.
- 26. As software engineers weare inevitably affected by the tools we surround ourselves with. Languages, frameworks, even processes all act to shape the software we build.
- 27. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service The tools wechoose have a big effect on our architecture GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Event Driven Request Response
- 28. Kafka is wellsuited to Event Driven Architectures It will lead you down that path
- 29.
- 30.
- 31. What is aDistributed Log?
- 32. Shard on theway in Producing Services Kafka Consuming Services
- 33. Each shard isa queue Producing Services Kafka Consuming Services
- 34.
- 35. Reduces to aglobally ordered queue
- 36.
- 37.
- 38. Build ‘Always On’Services Rely on Fault Tolerant Broker
- 39. Services can “Rewind& Replay” the log Rewind & Replay
- 40. Compacted Log (retains onlylatest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 41. A database enginefor data-in-flight
- 42. Max(price) From orders where ccy=‘GBP’ over1 day window emitting every second Continuously Running Queries
- 43. What is streamprocessing engine? Data Index Query Engine Query Engine vs Database Finite source Stream Processor Infinite source
- 44. Windowing For unordered orunpredictable streams Sliding Fixed (tumbling)
- 45. Features: similar to databasequery engine JoinFilter Aggr- egate View Window
- 46. Stateful Stream Processing stream Compacted stream Join Stream Data Stream-Tabular Data Windowed Stream LocallyCached Table (disk resident) KafkaKafka Streams
- 47.
- 48.
- 49.
- 50.
- 51. Sometimes you need tophysically move data
- 52. Replicate it, soboth copies are identical
- 53.
- 54.
- 55.
- 56. Service Backbone Scalable, FaultTolerant, Concurrent, Strongly Ordered, Stateful
- 57. Embeddable tool fordata manipulation Adapt data streams (transformation, stream maintenance, analytic function)
- 58. Replicate Data SourcesExactly Create Regenerable, Streaming Views
- 59. How do weactually do this? 10 (opinionated) principals for Streaming Services
- 60. 1. Don’t useKafka for shopping carts! (OK, you can, but use sparingly) Shopping Cart UI Service Broker/durability/broadcast add little to request response
- 61. Do use Kafkafor your core business processing. Think “business events”. An order was created, a payment was received, a trade was booked etc. Pub/Sub. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
- 62. 2. Pick Topicswith Business Significance Orders Payments Returns Invoices
- 63. Give your messagesmeaningful IDs and version them rdersService1-Order-1234- Should relate to the real world Should be Versioned (if mutable) Include the service name Note the key used for sharding in Kafka may not be this key
- 64. 3. Decouple publishersfrom subscribers GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
- 65. Add Request/Response only whereneeded GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service REST
- 66. 4. Use thelog to regenerate state Avoid journaling incoming events
- 67. Event Source sideeffects • Use offsets to tie these back to the stream • Store in: • Kafka • Kstreams state store • Other DB
- 68. 5. Apply theSingle Writer Principal • Change at source (by calling that service) • Let the change propagate back • Keep local copies read only. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service (1) Change Orders at source (2) Let the change propagate through
- 69. 6. Leverage keeping datasetsinside the broker customer country supplier ex- rates
- 70. Leverage keeping only thelatest version (table view) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 71. Join & Processon the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
- 72. 7. Prefer streamprocessing over maintaining historic views UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Orders Historic “copies” diverge
- 73. Join & Processon the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
- 74. 8. Sometimes you needhistoric views. => Replicate & Keep Read Only
- 75.
- 76.
- 77.
- 78. 9. Use Schemas (especiallyif data is retained)
- 79.
- 80. Confluent Schema Registry canhelp Orders Service Schema Registry Email Service Avro
- 81. 10. Consider “Stream Management”Services Stream Management Kafka • Retaining data => Admin tasks • Similar to the role of a DBA • Data Migration • Repartitioning • Latest/versioned • Environment Management • CQRS
- 82. KStreams is agood toolset for this
- 83.
- 84. So…1. Don’t useKafka for shopping carts! 2. Pick Topics with Business Significance 3. Decouple publishers from subscribers 4. Use the log to regenerate state 5. Apply the Single Writer Principal 6. Leverage keeping datasets inside the broker 7. Prefer stream processing over maintaining historic views 8. Sometimes you need historic views. => Replicate Read Only 9. Use Schemas 10. Consider “Stream Management” Services
- 85. Microservices push usaway from shared, mutable state
- 86. But state needsto be communicated
- 87. In an increasinglydata-heavy world we need tools to do this efficiently
- 88. …and in realtime.
- 89. We need adata-centric toolset to do this All Your Data Immutable Streams Event Driven Services Stream Management Services Legacy CDC View Replication Polyglotic Persistence tables streams Streaming Services
- 90. Keep it simple, Keepit moving
- 91.
Editor's Notes
- #4 Today we’re going to talk about two fields which sit somewhat apart. Stream Processing & Business Systems Services encourage us to share responsibilities. To rely on others. To collaborate and evolve a business’s view of the world over time. A world that is inherently decentralised. Inherently spread across many interconnected systems. Spreading many disparate subsets of our business’s state. Yet our approach rarely reflects this. We tend to think in centralised ways, we accumulate data. We protect it. We control it. We hoard it. But it does not need to be this way.
- #6 Today we’re going to talk about two fields which sit somewhat apart. Stream Processing & Business Systems Services encourage us to share responsibilities. To rely on others. To collaborate and evolve a business’s view of the world over time. A world that is inherently decentralised. Inherently spread across many interconnected systems. Spreading many disparate subsets of our business’s state. Yet our approach rarely reflects this. We tend to think in centralised ways, we accumulate data. We protect it. We control it. We hoard it. But it does not need to be this way.
- #15 My name is Ben Stopford. I work as an engineer on Apache Kafka. In a previous life I did the most centralised thing you could possibly do. I built a single, central database for a large financial institution. This talk is about exploring the exact opposite approach to this problem. It’s about thinking of the world in terms of the evolution and provisioning of state across many systems. It’s about embracing decentralisation. It’s about thinking in streams.