Event-driven microservices communicate asynchronously using events published to an event backbone like Apache Kafka. This approach offers benefits like loose coupling, responsiveness, and independent scalability. Various patterns can be used to manage complexity, including having each service store its own data, using Kafka log compaction to replicate data across services, orchestrating multi-step processes through sagas, and separating read and write concerns with CQRS. Event storming helps identify the events, actors, and data in an application to structure it in an event-driven way.

1. Event-driven microservices
using Apache Kafka
Andrew Schofield
Chief Architect, Event Streams
IBM Hursley Park
© 2019 IBM Corporation

2. What and why microservices?
Microservices is a technique for structuring an
application as a collection of services:
• Self-contained with clear interfaces and a
distinct purpose
• Loosely coupled – communicate over a
network
• Independently deployable, scalable,
maintainable and testable
© 2019 IBM Corporation
Microservices ApplicationMonolithic Application

3. Being event-driven is different
© 2019 IBM Corporation
Source: Gartner - From data-centric to event-centric IT priority

4. Why event-driven?
• Proper loose coupling means asynchronous communication
• Enables responsive applications
• In the real world, things often take time to complete
© 2019 IBM Corporation

5. Event-driven microservices
• Microservices communicate primarily
using events, with APIs where required
• Microservices can produce and consume
events using the publish/subscribe
pattern
• Events are handled by an event
backbone
• Data is eventually consistent
© 2019 IBM Corporation
Microservice
Microservice
Microservice
Microservice
Microservice
Event
backbone
Publish
Service
discoveryMicroservice
Microservice
Microservice
Subscribe
Subscribe
API
API
Publish
JSON /
HTTP
JSON / HTTP
Microservices
application

6. Apache Kafka is an Open-Source Streaming Platform
© 2019 IBM Corporation
PUBLISH & SUBSCRIBE
Read and write streams of events, like a
traditional messaging system.
PROCESS
Support scalable stream processing applications
that react to events in real-time.
STORE
Store streams of data safely in a distributed,
replicated, fault-tolerant cluster.
https://kafka.apache.org/
Kafka
Cluster
Producer ProducerProducer
Consumer ConsumerConsumer
Stream
Processor
Stream
Processor
Source
Connector
Sink Connector

7. Event-driven terminology
Event A message usually representing a fact at a particular time
Event stream A continuous, ordered, unbounded sequence of events
Stream processing Computation optimized for continuous processing, often with state
Stream history The historical sequence of the events on an event stream
© 2019 IBM Corporation

8. Apache Kafka as the event backbone
© 2019 IBM Corporation

9. Apache Kafka as the event backbone
© 2019 IBM Corporation
Microservice
Microservice

10. Apache Kafka as the event backbone
© 2019 IBM Corporation
1
Building Blocks
1 :: Event sources
Microservice
Microservice

11. Apache Kafka as the event backbone
© 2019 IBM Corporation
1
2
Building Blocks
1 :: Event sources
2 :: Stream processing
Microservice
Microservice

12. Apache Kafka as the event backbone
© 2019 IBM Corporation
1
2 App
3
Building Blocks
1 :: Event sources
2 :: Stream processing
3 :: Event archive
Microservice
Microservice

13. Apache Kafka as the event backbone
© 2019 IBM Corporation
App
1
2
3
4
Building Blocks
1 :: Event sources
2 :: Stream processing
3 :: Event archive
4 :: Notifications
Microservice
Microservice

14. How to get started
• You could run an event storming workshop
https://www.ibm.com/cloud/garage/architectures/eventDrivenArchitecture/event-storming-methodology
• Identify:
• Events
• Actors
• Data
• Commands
© 2019 IBM Corporation

15. A simple stream processing example
© 2019 IBM Corporation
Event backbone
Transactions

16. A simple stream processing example
© 2019 IBM Corporation
Event backbone
Transactions
Alerts

17. A simple stream processing example
© 2019 IBM Corporation
Event backbone
Transactions
Alerts
SMS alerts
Email alerts
Push alerts

18. A simple stream processing example
© 2019 IBM Corporation
Event backbone
Send SMS
Transactions
Alerts
Send emails
Send push
notifications
SMS alerts
Email alerts
Push alerts

19. © 2019 IBM Corporation
A more complex event-driven example
https://ibm-cloud-architecture.github.io/refarch-kc/design/readme/

20. A more complex example – component view
© 2019 IBM Corporation
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
APIAPI

21. A more complex example – component view
© 2019 IBM Corporation
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
APIAPI

22. A more complex example – component view
© 2019 IBM Corporation
Event backbone
ships containers orders voyages
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
APIAPI

23. A more complex example – component view
© 2019 IBM Corporation
Event backbone
ships containers problems orders voyages
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
Streaming
analytics
APIAPI

24. Patterns for event-driven microservices
• You want:
• Loose coupling
• Data consistency
• Efficient queries
© 2019 IBM Corporation

25. Patterns for event-driven microservices
• You want:
• Loose coupling
• Data consistency
• Efficient queries
• You don’t want:
• Chaos
© 2019 IBM Corporation

26. Patterns for event-driven microservices
• You want:
• Loose coupling
• Data consistency
• Efficient queries
• You don’t want:
• Chaos
• You need PATTERNS
© 2019 IBM Corporation

27. Pattern – Database per service
• Each microservice persists its own data
+ Protects independence of the microservice against external change
- Introduces complexity for data consistency across microservices
Event backbone
ships containers problems orders voyages
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
© 2019 IBM Corporation

28. Kafka log compaction for data replication
© 2019 IBM Corporation
0
key:a
val:A1
1
key:b
val:B1
2
key:a
val:A2
3
key:c
val:C1
Partition 0
4
key:c
val:C2
5
key:b
val:
Producer
writes
Consumer
takes latest
values and
builds own
data store
reads
Key Value
a A2
b <deleted>
c C2
Old New
Source change log Read

29. Kafka log compaction for data replication
© 2019 IBM Corporation
0
key:a
val:A1
1
key:b
val:B1
2
key:a
val:A2
3
key:c
val:C1
Partition 0
4
key:c
val:C2
5
key:b
val:
2
key:a
val:A2
4
key:c
val:C2
5
key:b
val:
Periodic compaction eliminates duplicate keys
to minimize storage
Partition 0
(rewritten)
Key Value
a A2
b <deleted>
c C2

30. Pattern – Event sourcing
• Every state change is captured as an event stored in sequence
+ Publishing events is atomic
- Event store is hard to query efficiently
© 2019 IBM Corporation
OrderCreated
OrderUpdated
OrderUpdated
OrderConfirmed
OrderShipped
OrderDelivered
OrderInTransit
Orders
microservice
Orders
DB
Command
handler
USER
Time
OrderPending
OrderBooked
OrderAssigned

31. Pattern – Sagas
• Orchestration of multi-step operations across microservices
+ Data consistency across microservices without distributed transactions
- Programming can be complex, particularly for failure compensation
© 2019 IBM Corporation
Event backbone
Fleet
microservice
Containers
microservice
Voyages
microservice
Orders
microservice
1 2 3 4 5

32. Pattern – CQRS
• Command Query Responsibility Segregation
+ Atomic writes and efficient reads at the same time
- Complex to program correctly
© 2019 IBM Corporation
Event backbone
1 2 3 4 5
Fleet MS
WRITE MODEL
Fleet MS READ
MODEL
Write-optimized
store
Read-optimized
store
Write API Read API

33. Summary
• Event-driven microservices offer an effective way to build loosely-
coupled applications
• Techniques such as event storming can be used to derive the objects,
commands and events in an application
• Patterns such as event sourcing and sagas make the complexity
manageable
© 2019 IBM Corporation