Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Microservices with Kafka Ecosystem

1,025 views

Published on

Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dig into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to
hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Published in: Data & Analytics

Microservices with Kafka Ecosystem

  1. 1. Microservices with Kafka Ecosystem Guido Schmutz @gschmutz doag2017
  2. 2. Guido Schmutz Working at Trivadis for more than 20 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Microservices with Kafka Ecosystem2
  3. 3. Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: Trivadis Services takes over the interacting operation of your IT systems. O P E R A T I O N Microservices with Kafka Ecosystem3
  4. 4. COPENHAGEN MUNICH LAUSANNE BERN ZURICH BRUGG GENEVA HAMBURG DÜSSELDORF FRANKFURT STUTTGART FREIBURG BASEL VIENNA With over 600 specialists and IT experts in your region. 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 million Financially self-supporting and sustainably profitable Experience from more than 1,900 projects per year at over 800 customers Microservices with Kafka Ecosystem4
  5. 5. Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. Why Kafka for Event-Driven Microservices? 5. Asynchronous Microservices in Enterprise Architecture 6. References Microservices with Kafka Ecosystem5
  6. 6. Microservices with Kafka Ecosystem6 Where do we come from?
  7. 7. Shop Rich UI Shop Backend Application Traditional Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database Microservices with Kafka Ecosystem7 sync request/response
  8. 8. Shop UI App Business Activity Service SOA Approach Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic Data GUI Service Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP Microservices with Kafka Ecosystem8
  9. 9. Shop UI App Business Activity Service Virtualized SOA Approach Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP Microservices with Kafka Ecosystem9
  10. 10. Microservices with Kafka Ecosystem10 What are Microservices?
  11. 11. Key Principles of Microservices Tightly Scoped Tightly Encapsulated behind concrete interfaces Responsible for managing their own data Highly cohesive Highly decoupled Independently deployable, self-contained and autonomous Microservices with Kafka Ecosystem11
  12. 12. Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Microservices with Kafka Ecosystem12
  13. 13. Microservice Approach with API Gateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Microservices with Kafka Ecosystem13
  14. 14. Synchronous World of Request-Response leads to tight, point-to-point couplings Microservices with Kafka Ecosystem14 problem in lower end of chain have a ripple effect on the other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State
  15. 15. Microservices with Kafka Ecosystem15 Why not Event-Driven?
  16. 16. 3 mechanisms through which services interact Request-Driven Event Driven Service Logic State Event Consume “IPad OrderedEvent” Command ”Order IPad” order(iPad) Event Publish “OrderValidatedEvent” Query ”Retrieve my Orders) getAllOrders(myself) Event Broker Microservices with Kafka Ecosystem16
  17. 17. 3 mechanisms through which services interact Commands • represents an action • request for some operation to be performed in another service • Something that will change the state of the system • Commands expect a response. Events • represent both a Fact and a Trigger • Something that has happened, expressed as a notification Queries • represent a request to look something up • Importantly, queries are side effect free; they leave the state of the system unchanged Microservices with Kafka Ecosystem17
  18. 18. Request-Driven Communication – Highest Coupling Shop UI Application UI Logic Order Microservice Logic State { } API Stock Microservice State { } API Logic Call PlaceOrderAPI Call UpdateStockAPI Maybe Call Reorder API Microservices with Kafka Ecosystem18
  19. 19. Decoupling through Events – Lowest Coupling Event Broker Microservices with Kafka Ecosystem19 Order Microservice Logic State { } API Stock Microservice State { } API Logic3) Raise OrderConfirmed Event 4) Listen to OrderConfirmed Event 5) Maybe Raise Reorder Event 2) Listen to OrderRequested Event 1) Raise OrderRequested Event sync request/response async request/response async, event pub/sub
  20. 20. Event-Driven (Async) Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Microservices with Kafka Ecosystem20 Event Broker sync request/response async request/response async, event pub/sub
  21. 21. Shop UI App Business Activity Service Process & Virtualized SOA Approach – Sync & Async Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus Orchestration Microservices with Kafka Ecosystem21
  22. 22. Command Query Responsibility Segregation (CQRS) Optimize different nonfunctional requirements for read and write behavior interface to the service is split between • commands that trigger changes in state • queries that provide read access to the state of resources used to support services with higher performance and capacity requirements for reading data than for writing data number of instances supporting queries can be scaled up independently of instance supporting commands Data Storage Write Data Store Read Data Store (Materialized Views) Service Command Service Query Service App UI Projection Service UI Logic Microservices with Kafka Ecosystem22
  23. 23. Event Sourcing persists the state of a business entity as a sequence of state-changing events Whenever state of business entity changes, a new event is appended to the list of events Since saving an event is a single operation, it is inherently atomic The application reconstructs an entity’s current state by replaying the events Data Storage Event Store Service Event Service Publisher App UI UI Logic Replayer Other App Microservices with Kafka Ecosystem23
  24. 24. Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern materializing views from the stored events Optionally Commands can be stored in event store and transformed into events by the command handler Data Storage Event Store Service Command Service App UI UI Logic Query Service Read Data Store (Materialized Views) Projection Service Command Handler Microservices with Kafka Ecosystem24
  25. 25. Using Event Sourcing with Microservices Microservices with Kafka Ecosystem25 “Event sourcing enables building a forward-compatible application architecture — the ability to add more applications in the future that need to process the same event but create a different materialized view.” Neha Narkhede, Confluent Blog Microservice State Command Handler { } API REST Event Store Projection Handler(s) Query Logic Event Handler(s) Event Subscribe
  26. 26. How many Event Stores do we need ? Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST OR Microservices with Kafka Ecosystem26
  27. 27. Have one source of truth Avoid double write! • Would need distributed transactions Write Event first then consume it from same micro service • “eat your own dog food” Microservice { } API StateLogic REST Event Store Event Microservice { } API StateConsumer REST Event Store Event Publisher Microservices with Kafka Ecosystem27
  28. 28. Data Store Publish Event through Database Customer Event Store Event Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Microservices with Kafka Ecosystem28
  29. 29. Why Kafka for Event-Driven Microservices? Microservices with Kafka Ecosystem29
  30. 30. Apache Kafka - Unix Analogy $ cat < in.txt | grep "kafka" | tr a-z A-Z > out.txt Kafka Connect API Kafka Connect APIKafka Streams API Kafka Core (Cluster) Adapted from: Confluent KSQL Microservices with Kafka Ecosystem30
  31. 31. Kafka High Level Architecture The who is who • Producers write data to brokers. • Consumers read data from brokers. • All this is distributed. The data • Data is stored in topics. • Topics are split into partitions, which are replicated. Kafka Cluster Consumer Consumer Consumer Producer Producer Producer Broker 1 Broker 2 Broker 3 Zookeeper Ensemble Microservices with Kafka Ecosystem31
  32. 32. Leverage the Log at the heart of Apache Kafka sits a distributed log collection of messages, appended sequentially to a file service ‘seeks’ to the position of the last message it read, then scans sequentially, reading messages in order log-structured character makes Kafka well suited to performing the role of an Event Store in Event Sourcing Event Hub 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 Reads are a single seek & scan Writes are append only Microservices with Kafka Ecosystem32
  33. 33. Scale-Out Architecture Microservices with Kafka Ecosystem33 topic consists of many partitions producer load load-balanced over all partitions consumer can consume with as many threads as there are partitions Producer 1 Consumer 1 Broker 1 Producer 2 Producer 3 Broker 2 Broker 3 Consumer 2 Consumer 3 Consumer 4 Consumer Group 1 Consumer Group 2 Kafka Cluster
  34. 34. Strong Ordering Guarantees most business systems need strong ordering guarantees messages that require relative ordering need to be sent to the same partition supply same key for all messages that require a relative order To maintain global ordering use a single partition topic Producer 1 Consumer 1 Broker 1 Broker 2 Broker 3 Consumer 2 Consumer 3 Key-1 Key-2 Key-3 Key-4 Key-5 Key-6 Key-3 Key-1 Microservices with Kafka Ecosystem34
  35. 35. Durable and Highly Available Messaging Producer 1 Broker 1 Broker 2 Broker 3 Producer 1 Broker 1 Broker 2 Broker 3 Consumer 1 Consumer 1 Consumer 2Consumer 2 Microservices with Kafka Ecosystem35
  36. 36. Durable and Highly Available Messaging (II) Producer 1 Broker 1 Broker 2 Broker 3 Producer 1 Broker 1 Broker 2 Broker 3 Consumer 1 Consumer 1 Consumer 2 Consumer 2 Microservices with Kafka Ecosystem36
  37. 37. Replay-ability – Logs never forget by keeping events in a log, we have a version control system for our data if you were to deploy a faulty program, the system might become corrupted, but it would always be recoverable sequence of events provides an audit point, so that you can examine exactly what happened rewind and reply events, once service is back and bug is fixed Event Hub 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 Replay Rewind Service Logic State Microservices with Kafka Ecosystem37
  38. 38. Hold Data for Long-Term – Data Retention Producer 1 Broker 1 Broker 2 Broker 3 1. Never 2. Time based (TTL) log.retention.{ms | minutes | hours} 3. Size based log.retention.bytes 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 --create --topic customers --replication-factor 1 --partitions 1 --config cleanup.policy=compact Microservices with Kafka Ecosystem38
  39. 39. Keep Topics in Compacted Form 0 1 2 3 4 5 6 7 8 9 10 11 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value 3 4 6 8 9 10 K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 Offset Key Value Compaction Microservices with Kafka Ecosystem39 V1 V2 V3 V4 V5 V6 V7 V8 V9V10 V11 K1 K3 K4 K5K2 K6
  40. 40. Topic Viewed as Event Stream or State Stream (Change Log) Event Stream State Stream (Change Log Stream) 2017-10-02T20:18:46 11,Normal,41.87,-87.67 2017-10-02T20:18:55 11,Normal,40.38,-89.17 2017-10-02T20:18:59 21,Normal,42.23,-91.78 2017-10-02T20:19:01 21,Normal,41.71,-91.32 2017-10-02T20:19:02 11,Normal,38.65,-90.2 2017-10-02T20:19:23 21,Normal41.71,-91.32 11 2017-10-02T20:18:46,11,Normal,41.87,-87.67 11 2017-10-02T20:18:55,11,Normal,40.38,-89.17 21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78 21 2017-10-02T20:19:01,21,Normal,41.71,-91.32 11 2017-10-02T20:19:02,11,Normal,38.65,-90.2 21 2017-10-02T20:19:23,21,Normal41.71,-91.32 Microservices with Kafka Ecosystem40
  41. 41. Keep Topics both in Original and Compacted Form Microservices with Kafka Ecosystem41 Kafka Broker A B C B A E K A E A C K B E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K Producer Consume and Produce OR TX
  42. 42. Legacy Microservice Change Data Capture (CDC) Microservices with Kafka Ecosystem42 RDBMS cdc-source trucking_ driver Driver Topic elasticsearch -sink NoSQL
  43. 43. Enrich Stream with Static Data with Kafka Streams Microservices with Kafka Ecosystem43 Movement Topic Consume and Produce Driver Table Driver Topic Driver Handler Join Driver Local State Movement & Driver Topic
  44. 44. Building Embedded, Materialized View with Kafka Streams Microservices with Kafka Ecosystem44 Movement & Driver Topic Consume and Produce Join Store Engine Metric Topic Join Join State Window Aggregation Result Store Result State
  45. 45. Building Embedded, Queryable Materialized View with Kafka Streams Microservices with Kafka Ecosystem45 Movement & Driver Topic Consume and Produce Join Store Engine Metric Topic Join Join State Window Aggregation Result Store Result State API Application
  46. 46. Asynchronous Microservices in Enterprise Architecture Microservices with Kafka Ecosystem46
  47. 47. Hadoop Clusterd Hadoop Cluster Big Data Cluster Asynchronous Microservices in Enterprise Architecture Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps SQL Search BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data Event Hub Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Analytics Cluster Stream Processor State { } API Event Stream Event Stream Service Microservices with Kafka Ecosystem47
  48. 48. References Microservices with Kafka Ecosystem48
  49. 49. References Microservices with Kafka Ecosystem49 Microservices Blog Series, Ben Stopford, Confluent: • https://www.confluent.io/blog/tag/microservices Apache Kafka for Microservices: A Confluent Online Talk Series: • https://www.confluent.io/landing-page/microservices-online-talk-series/ Turning the database inside-out with Apache Samza, Martin Kleppmann, Con • https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Immutability Changes Everything, Pat Helland, Salesforce: • http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: • https://www.youtube.com/watch?v=B1-gS0oEtYc
  50. 50. Technology on its own won't help you. You need to know how to use it properly. Microservices with Kafka Ecosystem50
  51. 51. Trivadis @ DOAG 2017 #opencompany Booth: 3rd Floor – next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit Microservices with Kafka Ecosystem51

×