Building event-driven (Micro)Services with Apache Kafka Ecosystem

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Building event-driven Microservices
with Kafka Ecosystem
Guido Schmutz
nlOUG Tech Experience - 7.6.2018
@gschmutz guidoschmutz.wordpress.com

Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Building event-driven Microservices with Kafka Ecosystem

Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not more Event Driven?
4. Why Kafka for Event-Driven Microservices?
5. What about streaming sources?
6. What about integrating legacy applications?
7. What about (historical) data analytics?
8. Summary

Where do we come from?

Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

Shop UI App
Business
Activity Service
SOA Approach
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

What are Microservices?

Microservice Approach
Tightly Scoped behind clear
interfaces
Responsible for managing their
own data (not necessarily the
infrastructure)
Should be highly decoupled
Independently deployable, self-
contained and autonomous
SOA done right ?!
Tightly Scoped
Responsible for managing their own data
Highly decoupled
Independently deployable, self-contained
and autonomous
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

with API Gateway
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway

Synchronous World of Request-Response leads to tight,
point-to-point couplings
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State

Why not more Event-Driven?

3 mechanisms through which services interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Event
Consume
OrderEvent{iPad}
Command
”Order IPad”
boolean order(IPad)
Event
Publish
OrderValidatedEvent{iPad}
Query
”Retrieve my Orders
List<Orders> getAllOrders(for)

Event-Driven (Async)
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Event
Store
sync request/response
async request/response
async, event pub/sub

Why Kafka for Event-Driven
Microservices?

Apache Kafka – A Streaming Platform
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget

Apache Kafka – Schema Registry
Producer
Kafka Broker
Consumer
Schema
Registry
Kafka Kafka
avro
schema
avroavro
avro
schema

Demo
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API
Order
Order Logic
REST
REST
Event
Hub
Customer

Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact

Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2015-10-02 11,Peter,Muster,3010,Berne
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen

Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
Offset
Key
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
V1
V2
V3 V4
V5
V6
V7
V8 V9
V10
V11
K1
K3
K4
K5
K2
K6

Keep Topics both in Original and Compacted Form
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Kafka Broker
A B C B A E K A E A C K
B E A C K
Producer
Consume and Produce
OR
TX

Apache Kafka – scalable message processing and
more!
Source
Connector
trucking_
driver
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka

Enrich Stream with Static Data with Kafka Streams
Order Topic Consume and Produce
Customer
Table
Customer Topic
Customer
Handler
Join
Customer Local State
Order & Customer Topic

Building Embedded, Materialized View with Kafka
Streams
Order Topic
Consume and Produce
Aggregation
Store
Click Stream Topic
Window
Aggregation
Aggregation State
Join
Join
Store
Result State
Window
Aggregation

Building Embedded, Queryable Materialized View with
Kafka Streams
Order Topic
Consume and Produce
Aggregation
Store
Click Stream Topic
Window
Aggregation
Aggregation State
Join
Join
Store
Result State
Window
Aggregation
API Application

What about streaming sources?

How to work with
streaming data sources
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Store
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream

Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Analytics Architecture
BI Tools
SQL
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service

What about integrating legacy
applications?

Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer

Legacy Microservice
Change Data Capture (CDC) with Kafka
RDBMS cdc-source trucking_
driver
Customer Topic
elasticsearch-
sink NoSQL

Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQL
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture

What about (historical) data
analytics?

Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Data
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Store
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import

Summary

Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
Data
Flow
Data
Flow
Change
Data
Capture
Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices

Summary
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”

References
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc

Technology on its own won't help you.
You need to know how to use it properly.

Building event-driven (Micro)Services with Apache Kafka Ecosystem

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building event-driven (Micro)Services with Apache Kafka Ecosystem

Similar to Building event-driven (Micro)Services with Apache Kafka Ecosystem (20)

More from Guido Schmutz

More from Guido Schmutz (20)

Recently uploaded

Recently uploaded (20)

Building event-driven (Micro)Services with Apache Kafka Ecosystem