Microservice composition or integration is probably the hardest thing in microservices architecture. Unlike conventional centralized ESB based integration, we need to leverage the smart-endpoints and dumb pipes terminology when it comes to integrating microservices. There two main microservices integration patterns; service orchestration (active integrations) and service choreography (reactive integration). In this talk, we will explore on, Microservice Orchestration, Microservice Choreography, Event Sourcing, CQRS and how Kafka can be leveraged to implement microservices composition

1. Microservices Integration Patterns with Kafka
Kasun Indrasiri
Director - Integration Architecture,
WSO2
Bay Area Apache Kafka Meetup
@ Lyft HQ, San Francisco

2. ● Microservices Integration/composition?
● Microservices Composition Patterns
○ Active/Synchronous Composition
○ Reactive Composition
● Kafka for building Reactive Composition.
● Hybrid of active and reactive composition.
Overview

3. Rise of Cloud Native
Applications

4. Integrating
Microservices and
Cloud Native Apps
Integration is essential to build
any real world use case with
cloud native apps.

5. ● APIs:
○ Commands - Actions, synchronous
○ Queries - Request for information, no state change, synchronous
● Events: Facts and notifications, asynchronous
● Streams: A sequence of events/data elements made available
over time
Microservices Integration/Composition
How services can interact over the network?

6. Active Composition/Orchestration Pattern
Synchronous/request-response communication
Consumer 1
API Service P API Service Q API Service R API Service S
Consumers
Consumer 2 Consumer 3
μ Service A
Proprietary
& Legacy
Systems
Web API /
SaaS
API Services/
Edge Services
Composite Service/
Integration Services
Core Services/
Atomic Services μ Service I
μ Service H
μ Service J
μ Service E
μ Service C
μ Service D
μ Service G
μ Service B
μ Service F
API
Management

7. ● RESTful Services
● gRPC/HTTP2
● GraphQL
● WebSockets
● Thrift
Active Composition/Orchestration Pattern
Synchronous/request-response communication

8. Reactive Composition/Choreography Pattern
μ Service A μ Service B μ Service C
Event Bus
Application
Asynchronous event-driven communication
Produce Consume
Produce Consume
Produce Consume
Produce Consume

9. ● Kafka
● AMQP
● JMS
Reactive Composition/Choreography Pattern
Asynchronous event-driven communication

10. Kafka in a nutshell
Source: “Microservices for Enterprise” https://goo.gl/p3rWF3

11. ● Consumers from different
consumer groups subscribe
to the same topic.
● Consumers of a given CG
can only consume from one
partition.
Reactive Composition with Kafka
Publisher-subscriber
Source: “Microservices for Enterprise” https://goo.gl/p3rWF3

12. ● Single consumer group and a
consumer per each partition.
● Polling and sequential
processing of consumer
records using offset.
Reactive Composition with Kafka
Single Consumer/Queuing
Source: “Microservices for Enterprise” https://goo.gl/p3rWF3
topic

13. Delivery Semantics: Exactly-Once/Atomic Broadcast
Reactive Composition with Kafka
● You can publish messages and they will be delivered one time
exactly by one or more receiving application.
● Producer side
○ Producer writes to the log but fails to get the ack. over network.
● Consumer side
○ Consumer consumes message from the log but fails before
updating its offset value.
Source: https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f

14. Delivery Semantics: Exactly-Once/Atomic Broadcast
Reactive Composition with Kafka
● Producer side
○ Idempotence : Producers can retry until it is successful
○ Kafka will detect the duplicates and remove them
● Consumer side
○ Store the offsets in the same DB as the derived state and update
both in a transaction.
○ Write both state updates and offsets together in a way that is
idempotent.
Source: https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f

15. Event Sourcing
● Persist each state changing events of an entity as a sequence
of immutable events.
● All such events are stored in an event bus and consumers can
derive the state by processing a sequence of events.
Shipping

16. ● Stream Processing in Microservices
○ Services that execute a business logic against a sequence of
events/data elements made available over time.
● Kafka Streams : a client library for building stream processing
on top of Kafka.
Reactive Composition with Kafka
Kafka Streams

17. ● Command Query Responsibility Segregation.
● Split the common data model into query and command
models.
● Kafka Stream - Interactive queries.
CQRS
Command
Source:
https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Query

18. ● Kafka does not look at your data but you need consistent
across your usage.
○ Avro, ProtoBuf, JSON, XML, ASN.1
○ Avro preferred schema for Kafka, Kafka schema registry
● Schema in practice
○ Associate an Avro schema with a topic.
○ Declare requirements for data that is produced into the topic
○ Provides the instructions on how to interpret data read from the topic
Using Schemas - Avro and Schema Registry

19. ● A tool for scalably and reliably streaming data between
Apache Kafka and other data systems.
Reactive Composition with Kafka
Kafka Connect
Source: https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines/

20. Kafka, Kafka Connect and Kafka Streams
Source: https://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams/

21. Active
or
Reactive?
● Active Composition
○ Ideal for interactive services
○ Service are NOT autonomous
● Reactive Composition
○ Ideal for asynchronous
interactions
○ Composition logic is opaque

22. Hybrid of Active and Reactive Composition
Consumer 1
API Service P API Service Q API Service R API Service S
Consumers
Consumer 2 Consumer 3
μ Service A
Event Bus
Proprietary
& Legacy
Systems
Web API /
SaaS
API Services/
Edge Services
Composite Service/
Integration Services
Core Services/
Atomic Services μ Service I
μ Service H
μ Service J
μ Service E
μ Service C
μ Service D
μ Service G
μ Service B
API
Management

23. ● Create a queryable materialized view using Kafka Streams or
a database with Kafka Connect.
Hybrid composition
Bridging request-response and event driven messaging
Source: https://github.com/confluentinc/kafka-streams-examples/tree/5.0.0-post/src/main/java/io/confluent/examples/streams/microservices

24. ● A queryable materialized view using Kafka Streams’
interactive queries.
Hybrid composition
Bridging request-response and event driven messaging

25. Implementation Technologies
● Java
○ All of the Kafka, Streams and Connect APIs are supported.
○ Confluent microservices example.
● Go, Python, .NET
● Ballerina (ballerina.io)
○ Cloud-native programming languages that makes it easy to write
microservices that integrate APIs.
https://ballerina.io/learn/by-guide/messaging-with-kafka/
● Integration Frameworks: Apache Camel, Spring Boot, WSO2
Integrator etc.

26. THANK YOU
Kasun Indrasiri
kasun@apache.org