This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservcies Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Microservices with Kafka
Ecosystem
Guido Schmutz
Frankfurt, 12.12.2017
@gschmutz guidoschmutz.wordpress.com

2. Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Microservices with Kafka Ecosystem2

3. COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers
Microservices with Kafka Ecosystem4

4. Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not Event Driven?
4. Why Kafka for Event-Driven Microservices?
5. Asynchronous Microservices in Enterprise Architecture
6. Summary
7. References
Microservices with Kafka Ecosystem5

5. Microservices with Kafka Ecosystem6
Where do we come from?

6. Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
Microservices with Kafka Ecosystem7
sync request/response

7. Shop UI App
Business
Activity Service
SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
Data
GUI
Service
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Microservices with Kafka Ecosystem8

8. Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
9

9. Microservices with Kafka Ecosystem10
What are Microservices?

10. Key Principles of Microservices
Tightly Scoped
Tightly Encapsulated behind concrete interfaces
Responsible for managing their own data
Highly cohesive
Highly decoupled
Independently deployable, self-contained and autonomous
Microservices with Kafka Ecosystem11

11. Microservice Approach Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Microservices with Kafka Ecosystem12

12. Microservice Approach
with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Microservices with Kafka Ecosystem13

13. Synchronous World of Request-Response leads to tight,
point-to-point couplings
Microservices with Kafka Ecosystem14
problem in lower end of chain have a
ripple effect on the other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State

14. Microservices with Kafka Ecosystem15
Why not Event-Driven?

15. 3 mechanisms through which services interact
Request-Driven Event Driven
Service
Logic
State
Event
Consume
“IPad OrderedEvent”
Command
”Order IPad”
order(iPad) : boolean
Event
Publish
“OrderValidatedEvent”
Query
”Retrieve my Orders)
getAllOrders(myself)
Event Broker
Microservices with Kafka Ecosystem16

16. Request-Driven Communication – Highest Coupling
Shop UI Application
UI Logic
Order Microservice
Logic State
{ }
API
Stock Microservice
State
{ }
API Logic
Call PlaceOrderAPI
Call UpdateStockAPI
Maybe Call Reorder API
Microservices with Kafka Ecosystem18

17. Decoupling through Events – Lowest Coupling
Event
Broker
Microservices with Kafka Ecosystem19
Order Microservice
Logic State
{ }
API
Stock Microservice
State
{ }
API Logic3) Raise
OrderConfirmed
Event
4) Listen to
OrderConfirmed
Event
5) Maybe Raise
Reorder Event
2) Listen to
OrderRequested
Event
1) Raise
OrderRequested
Event
sync request/response
async request/response
async, event pub/sub

18. Event-Driven (Async)
Microservice Approach
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Microservices with Kafka Ecosystem20
Event
Broker
sync request/response
async request/response
async, event pub/sub

19. Shop UI App
Business
Activity Service
Process & Virtualized SOA Approach – Sync & Async
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
Orchestration
Microservices with Kafka Ecosystem21

20. Command Query Responsibility Segregation (CQRS)
Optimize different nonfunctional
requirements for read and write behavior
interface to the service is split between
• commands that trigger changes in state
• queries that provide read access to the state
of resources
support services with higher performance
and capacity requirements for reading
data than for writing data
scare reading capacity independently of
writing
Data Storage
Write Data Store
Read Data Store
(Materialized Views)
Service
Command
Service
Query
Service
App
UI
Projection
Service
UI Logic
Microservices with Kafka Ecosystem22

21. Event Sourcing
persists the state of a business entity as a
sequence of state-changing events
Whenever state of business entity changes,
a new event is appended to the list of
events
Saving an event is a single operation and is
inherently atomic
The application reconstructs an entity’s
current state by replaying the events
Data
Storage
Event Store
Service
Event Service
Publisher
App
UI
UI Logic Replayer
Other
App
Microservices with Kafka Ecosystem23

22. Event Sourcing & CQRS
Event sourcing is commonly combined
with the CQRS pattern
materializing views from the stored
events
Optionally Commands can be stored
in event store and transformed into
events by the command handler
Data Storage
Event Store
Service
Command Service
App
UI
UI Logic
Query Service Read Data Store
(Materialized Views)
Projection
Service
Command Handler
Microservices with Kafka Ecosystem24

23. Using Event Sourcing with Microservices
Microservices with Kafka Ecosystem25
“Event sourcing enables building
a forward-compatible application
architecture — the ability to add
more applications in the future
that need to process the same
event but create a different
materialized view.”
Neha Narkhede, Confluent Blog
Microservice
State
Command
Handler
{ }
API
REST
Event Store
Projection
Handler(s)
Query Logic
Event Handler(s)
Event Subscribe

24. How many Event Stores do we need ?
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
OR
Microservices with Kafka Ecosystem26

25. Have only one „source of truth“
Avoid double write!
• Would need distributed transactions
Write Event first then consume it from
same micro service
• “eat your own dog food”
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API
StateConsumer
REST
Event Store
Event
Publisher
Microservices with Kafka Ecosystem27

26. Data Store
Publish Events through Database
Customer
Event Store
Event
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Microservices with Kafka Ecosystem28

27. Why Kafka for Event-Driven
Microservices?
Microservices with Kafka Ecosystem29

28. What is apache Apache Kafka?
The who is who
• Producers write data to brokers.
• Consumers read data from
brokers.
• All this is distributed.
The data
• Data is stored in topics.
• Topics are split into partitions,
which are replicated.
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
Microservices with Kafka Ecosystem31

29. Apache Kafka – scalable message processing and
more!
Microservices with Kafka Ecosystem32
Source
Connector
trucking_
driver
Kafka Broker
Sink
Connector
Stream
Processing

30. Leverage the Log
at the heart of Apache Kafka sits a
distributed log
collection of messages, appended
sequentially to a file
service ‘seeks’ to the position of the last
message it read, then scans sequentially,
reading messages in order
log-structured character makes Kafka
well suited to performing the role of an
Event Store in Event Sourcing
Event Hub
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22
Reads are a single
seek & scan
Writes are
append only
Microservices with Kafka Ecosystem33

31. Scale-Out Architecture
Microservices with Kafka Ecosystem34
topic consists of
many partitions
producer load
load-balanced
over all partitions
consumer can
consume with as
many threads as
there are
partitions
Producer 1
Consumer 1
Broker 1
Producer 2
Producer 3
Broker 2
Broker 3
Consumer 2
Consumer 3
Consumer 4
Consumer Group 1
Consumer Group 2
Kafka Cluster

32. Strong Ordering Guarantees
most business systems need strong
ordering guarantees
messages that require relative
ordering need to be sent to the same
partition
supply same key for
all messages that
require a relative order
To maintain global ordering use a
single partition topic
Producer 1
Consumer 1
Broker 1
Broker 2
Broker 3
Consumer 2
Consumer 3
Key-1
Key-2
Key-3
Key-4
Key-5
Key-6
Key-3
Key-1
Microservices with Kafka Ecosystem35

33. Replay-ability – Logs never forget
by keeping events in a log, we have a
version control system for our data
if you were to deploy a faulty program,
the system might become corrupted, but
it would always be recoverable
sequence of events provides an audit
point, so that you can examine exactly
what happened
rewind and reply events, once service is
back and bug is fixed
Event Hub
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22
Replay
Rewind
Service
Logic State
Microservices with Kafka Ecosystem38

34. Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact
Microservices with Kafka Ecosystem39

35. Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
Offset
Key
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
Microservices with Kafka Ecosystem40
V1 V2
V3
V4 V5
V6
V7
V8
V9V10 V11
K1 K3 K4 K5K2 K6

36. Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2017-10-02T20:18:46 11,Normal,41.87,-87.67
2017-10-02T20:18:55 11,Normal,40.38,-89.17
2017-10-02T20:18:59 21,Normal,42.23,-91.78
2017-10-02T20:19:01 21,Normal,41.71,-91.32
2017-10-02T20:19:02 11,Normal,38.65,-90.2
2017-10-02T20:19:23 21,Normal41.71,-91.32
11 2017-10-02T20:18:46,11,Normal,41.87,-87.67
11 2017-10-02T20:18:55,11,Normal,40.38,-89.17
21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78
21 2017-10-02T20:19:01,21,Normal,41.71,-91.32
11 2017-10-02T20:19:02,11,Normal,38.65,-90.2
21 2017-10-02T20:19:23,21,Normal41.71,-91.32
Microservices with Kafka Ecosystem41

37. Keep Topics both in Original and Compacted Form
Microservices with Kafka Ecosystem42
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Consume and Produce
OR
TX

38. Legacy Microservice
Change Data Capture (CDC)
Microservices with Kafka Ecosystem43
RDBMS cdc-source trucking_
driver
Driver Topic
elasticsearch
-sink NoSQL

39. Enrich Stream with Static Data with Kafka Streams
Microservices with Kafka Ecosystem44
Movement Topic Consume and Produce
Driver
Table
Driver Topic
Driver
Handler
Join
Driver Local State
Movement & Driver Topic

40. Building Embedded, Materialized View with Kafka
Streams
Microservices with Kafka Ecosystem45
Movement & Driver Topic
Consume and Produce
Join
Store
Engine Metric Topic
Join
Join State
Window
Aggregation
Result
Store
Result State

41. Building Embedded, Queryable Materialized View with
Kafka Streams
Microservices with Kafka Ecosystem46
Movement & Driver Topic
Consume and Produce
Join
Store
Engine Metric Topic
Join
Join State
Window
Aggregation
Result
Store
Result State
API Application

42. Asynchronous Microservices in
Enterprise Architecture
Microservices with Kafka Ecosystem47

43. Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Asynchronous Microservices in Enterprise Architecture
Location
Social
Click
stream
Sensor
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Call
Center
Mobile
Apps
SQL
Search
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Weather
Data
Event
Hub
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Analytics Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Service
48

44. Summary
Microservices with Kafka Ecosystem49

45. Summary
Microservices with Kafka Ecosystem50
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”

46. References
Microservices with Kafka Ecosystem51

47. References
Microservices with Kafka Ecosystem52
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc

48. Technology on its own won't help you.
You need to know how to use it properly.
Microservices with Kafka Ecosystem53