Building event-driven Microservices with Kafka Ecosystem

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Building event-driven Microservices
with Kafka Ecosystem
Guido Schmutz
Zurich, 8.3.2018
@gschmutz guidoschmutz.wordpress.com

Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Building event-driven Microservices with Kafka Ecosystem

Our company.
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and
technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:
Trivadis Services takes over the interacting operation of your IT systems.
O P E R A T I O N

COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers

Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not Event Driven?
4. What about streaming sources?
5. What about integrating legacy applications?
6. CQRS and Event Sourcing
7. What about (historical) data analytics?
8. Why Kafka for Event-Driven Microservices?
9. Summary

Where do we come from?

Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

Shop UI App
Business
Activity Service
SOA Approach
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

What are Microservices?

What are Microservices?
Tightly Scoped behind clear interfaces
Responsible for managing their own data (not necessarily the infrastructure)
Should be highly decoupled
Independently deployable, self-contained and autonomous
SOA done right ?!

Microservice Approach
Tightly Scoped
Responsible for managing their own data
Highly decoupled
Independently deployable, self-contained
and autonomous
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway

Synchronous World of Request-Response leads to tight,
point-to-point couplings
problem in lower end of chain have a
ripple effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State

Why not Event-Driven?

3 mechanisms through which services interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Event
Consume
OrderEvent{iPad}
Command
”Order IPad”
boolean order(IPad)
Event
Publish
OrderValidatedEvent{iPad}
Query
”Retrieve my Orders
List<Orders> getAllOrders(for)

Event-Driven (Async)
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Event
Store
sync request/response
async request/response
async, event pub/sub

What about streaming sources?

How to work with
streaming data sources
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Store
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream

Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Analytics Architecture
BI Tools
SQL
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service

What about integrating legacy
applications?

Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer

Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQL
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture

CQRS and Event Sourcing

Command Query Responsibility Segregation (CQRS)
Optimize different nonfunctional
requirements for read and write behavior
split between
• commands that trigger changes in state
• queries that provide read access to the state
of resources
support services with higher performance
and capacity requirements for reading
data than for writing data
Data Storage
Write Data Store
Read Data Store
(Materialized Views)
Service
Command
Service
Query
Service
App
UI
Projection
Service
UI Logic

Event Sourcing
persists the state of a business entity as a
sequence of state-changing events
Whenever state of business entity changes,
a new event is appended to the list of
events
Saving an event is a single operation and is
inherently atomic
The application reconstructs an entity’s
current state by replaying the events
Data
Storage
Event Store
Service
Event Service
Publisher
App
UI
UI Logic Replayer
Other
App

Event Sourcing & CQRS
Event sourcing is commonly combined
with the CQRS pattern
materializing views from the stored
events
Optionally Commands can be stored
in event store and transformed into
events by the command handler
Data Storage
Event Store
Service
Command Service
App
UI
UI Logic
Query Service Read Data Store
(Materialized Views)
Projection
Service
Command Handler

Have only one „source of truth“
Avoid double write!
• Would need distributed transactions
Write Event first then consume it from
same micro service
• “eat your own dog food”
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateConsumer
REST
Event Store
Event
Publisher

Using Event Sourcing with Microservices
“Event sourcing enables building
a forward-compatible application
architecture — the ability to add
more applications in the future
that need to process the same
event but create a different
materialized view.”
Neha Narkhede, Confluent Blog
Microservice
State
Command
Handler
{ }
API
REST
Event Store
Projection
Handler(s)
Query Logic
Event Handler(s)
Event Subscribe

How many Event Stores do we need ?
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API StateLogic
REST
Microservice
{ }
API StateLogic
REST
Event Store
Event
Microservice
{ }
API
State
Logic
REST
OR

What about (historical) data
analytics?

Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Data
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Store
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import

Why Kafka for Event-Driven
Microservices?

Apache Kafka – A Streaming Platform
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget

Apache Kafka – scalable message processing and
more!
Source
Connector
trucking_
driver
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka
RESTful RESTful

Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact

Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2015-10-02 11,Peter,Muster,3010,Berne
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen

Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
Offset
Key
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
V1 V2
V3
V4 V5
V6
V7
V8
V9V10 V11
K1 K3 K4 K5K2 K6

Keep Topics both in Original and Compacted Form
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Consume and Produce
OR
TX

Legacy Microservice
Change Data Capture (CDC)
RDBMS cdc-source trucking_
driver
Driver Topic
elasticsearc
h-sink NoSQL

Enrich Stream with Static Data with Kafka Streams
Bus Position Topic Consume and Produce
Driver
Table
Driver Topic
Driver
Handler
Join
Driver Local State
Bus Position & Driver Topic

Building Embedded, Materialized View with Kafka
Streams
Consume and Produce
Join
Store
Engine Metric Topic
Join
Join State
Window
Aggregation
Result
Store
Result State

Building Embedded, Queryable Materialized View with
Kafka Streams
Consume and Produce
Join
Store
Engine Metric Topic
Join
Join State
Window
Aggregation
Result
Store
Result State
API Application

Summary

Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
Data
Flow
Data
Flow
Change
Data
Capture
Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices

Summary
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”

References
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc

Technology on its own won't help you.
You need to know how to use it properly.

Building event-driven Microservices with Kafka Ecosystem

More Related Content

What's hot

Similar to Building event-driven Microservices with Kafka Ecosystem

More from Guido Schmutz

Recently uploaded

Building event-driven Microservices with Kafka Ecosystem