Building Event-Driven (Micro) Services with Apache Kafka

gschmutz
Building event-driven (Micro)Services
with Apache Kafka
SEACON 2019 – 24.5.2019
Guido Schmutz (guido.schmutz@trivadis.com)
gschmutz http://guidoschmutz.wordpress.com

gschmutz
Agenda
1. Where do we come from?
2. What about Microservices?
3. Can we do better?
4. How does Apache Kafka help?
5. What about Streaming Data Sources?
6. What about integrating Legacy Applications?
7. What about (historical) data analytics?
8. Summary

gschmutz
Guido Schmutz
Working at Trivadis for more than 22 years
Oracle Groundbreaker Ambassador & Oracle ACE Director
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
158th edition

gschmutz
Where do we come from?

gschmutz
Shop Rich UI
Shop Backend Application
“Layered Architecture” Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
sync request/response

gschmutz
Shop UI App
Business
Activity Service
SOA Approach
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP

gschmutz
Business
Activity Service
Virtualized SOA Approach
The raise of the
Enterprise Service
Bus (ESB)
Search BAS
Customer DAO
Order DAO
Order BAS
Business Entity
Service
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Shop UI App
Shop UI UI Logic
GUI
Shop Web App
Shop UI UI Logic
GUI
X

gschmutz
Shop UI App
Business
Activity Service
Orchestrated & Virtualized SOA Approach – Sync / Async
The raise of
orchestration engines
(BPEL & BPMN)
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
X
Orchestration

gschmutz
What about Microservices?

gschmutz
Customer Microservice
Microservice Approach
Tightly Scoped behind interfaces
Highly decoupled
Independently deployable
Bounded Context/Aggregate (DDD)
Responsible for their data (does not mean
they need their own DB!)
Smart Endpoints and Dump Pipes
just SOA done right ? What about
capabilities the “smart pipes” provided?
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST

gschmutz
Synchronous Request-Response lead to tight, point-to-
point couplings
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
RESTRESTRESTREST
REST REST REST

gschmutz
Microservice Approach
with API Gateway
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
X

gschmutz
Microservice Approach with Side Car (i.e. K8s & Istio)
Side-car can provide:
• retry
• load-balancing
• circuit breaker, throttling
• security
• …
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Side
Car
Side
Car
Side
Car
REST
RESTRESTREST

gschmutz
Microservice Approach with Side Car (i.e. K8s & Istio)
• Side-car and it’s advanced routing
capabilities can be used to switch to
new service version
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3 – v1
{ }
API Logic
State
Side
Car
Side
Car
Service 3 – v2
{ }
API Logic
State
RESTREST REST
REST

gschmutz
Side Car provides lot of value …. but
• we still have to change the “data
owner” service ….
if a new service requires the
same information
Service 1
{ }
API Logic
State
Service 2
{ }
API Logic
State
Service 4
Logic
State
Service 3
{ }
API Logic
State
{ }
API
New Service
Logic
State
{ }
API
REST
REST REST REST REST

gschmutz
Can we do even(t) better?

gschmutz
Events
Distribute to all handlers
strong ordering req’s
No results
Queries
Route with load balancing
Sometimes scatter-gather
Provide result
3 mechanisms through which services can interact
Commands
Route to single handler
Use consistent hashing
Provide Result
Adapted from Axon IQ

gschmutz
Stock Microservice
Event-Driven (Async) Microservice Approach
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
REST
REST
REST
REST
API
Gateway
Event
Hub
sync request/response
async request/response
async, event pub/sub
Customer
Mat View
Event Hub
• Pub / Sub messaging
• Topics schema-less
• Message coupling between services
• Domain Events from DDD
• This is not event sourcing!

gschmutz
How does Apache Kafka help?

gschmutz
Apache Kafka – highly scalable message broker
Kafka Cluster
Consumer Consumer Consumer
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer Producer Producer
kafkacat
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
Event Hub

gschmutz
Log Compaction – specially useful for Master Data
0 1 2 3 4 5 6 7 8 9 10 11
11 12 11 14 13 12 11 15 15 12 16 12
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12
Offset
Key
Value
3 4 6 8 9 10
14 13 11 15 12 16
V4 V5 V7 V9 V10 V11
Offset
Key
Value
Compaction
V1
V2
V3 V7
V5
V6
V4
V8 V9
V10
V11
11
13
14
15
12
16
Id: 11
Name: Peter Muster
Street: Münstergasse 5
City: 3013 Bern
Category: B
Id: 11
Name: Peter Muster
Street: Chaumonweg 108
City: 3098 Spiegel
Category: B
Id: 11
Name: Peter Muster
City: 3098 Spiegel
Category: A
Id: 11
Name: Peter Muster
City: 3098 Spiegel
Category: A
11
12
V12

gschmutz
Example
{ }
Customer API CustomerCustomer Logic
Order Microservice
{ }
Order API OrderOrder Logic
REST REST
Event
Hub
Customer
Mat View
Schema
Registry
Schema
Customer
(compacted)
Implementation: https://github.com/gschmutz/event-driven-microservices-demo

gschmutz
Example
@Configuration
public class KafkaConfig {
private String bootstrapServers;
private String schemaRegistryURL;
@Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryURL);
return props;
}
@Bean
public ProducerFactory<String, Customer> producerFactory() { .. }
@Bean
public KafkaTemplate<String, Customer> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
@Component
public class CustomerEventProducer {
@Autowired
private KafkaTemplate<String, Person> kafkaTemplate;
@Value("${kafka.topic.customer}")
String kafkaTopic;
public void produce(Customer customer) {
kafkaTemplate.send(kafkaTopic, customer.getId().toString(), customer);
}
}

gschmutz
Adding a new service by
bootstrapping from Event Hub
Customer Search
Microservice
{ }
REST
Schema
Registry
Schema
Consume
from Offset 0
{ }
Order Microservice
{ }
Order API OrderOrder Logic
REST REST
Event
Hub
Customer
Mat View
Customer
(compacted)
Implementation: https://github.com/gschmutz/event-driven-microservices-demo

gschmutz
What about Streaming Data
Sources?

gschmutz
How to work with
streaming data sources
{ }
Customer API
Customer
Customer Logic
Order Microservice
{ }
Order API
Order
Order Logic
{ }
Product API
Product
Product Logic
Stock Microservice
{ }
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Hub
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream

gschmutz
Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Processing & Microservices Architecture
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Hub
Service

gschmutz
Apache Kafka – scalable message processing
and more!
Source
Connector
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka

gschmutz
Kafka Connect
Source
Connector
Sink
Connector

gschmutz
Kafka Streams
• Programmatic API, “just” a Java library
• Native streaming
• fault-tolerant local state
• Fixed, Sliding and Session Windowing
• Stream-Stream / Stream-Table Joins
• At-least-once and exactly-once
KTable<Integer, Customer> customers = builder.stream(”customer");
KStream<Integer, Order> orders = builder.stream(”order");
KStream<Integer, String> joined = orders.leftJoin(customers, …);
joined.to(”orderEnriched");
trucking_
driver
Kafka Broker
Java Application
Kafka Streams

gschmutz
KSQL
• Stream Processing with zero coding using
SQL-like language
• part of Confluent Platform (community
edition)
• built on top of Kafka Streams
• interactive (CLI) and headless (command file)
CREATE STREAM customer_s WITH (kafka_topic='customer', value_format='AVRO');
SELECT * FROM customer_s WHERE address->country = 'Switzerland';
...
trucking_
driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands

gschmutz
What about integrating Legacy
Applications?

gschmutz
Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
SQ
L
Search / Explore
Online & Mobile
Apps
Search
Service
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Hub
Service
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture

gschmutz
Data Store
Integrate existing systems through CDC
Capture changes directly on database
Change Data Capture (CDC) => think
like a database-wide trigger
Make existing legacy systems an event
producer
Customer
Event Hub
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard

gschmutz
Legacy Microservice
Change Data Capture (CDC) with
Kafka Broker and Kafka Connect
RDBMS cdc-source trucking_
driver
Kafka Broker
elasticsearch-
sink
NoSQL
Customer Topic

gschmutz
Microservice
Change Data Capture (CDC) with
Kafka Broker and Kafka Connect
cdc-source
elasticsearch-
sink
NoSQL
Enhance
Kafka Broker
Customer Topic
CustomerEnhanced Topic
State
Legacy
RDBMS

gschmutz
What about (historical) data
analytics?

gschmutz
Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
D
ata
Flow
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
SQL
Export
SearchEvent
Hub
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Change Data
Capture
File Import / SQL Import

gschmutz
Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
D
ata
Flow
D
ata
Flow
Change
Data
Capture
Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices

gschmutz
Summary
• not all communication need to be synchronous => distinguish into
commands, events, queries
• Events should use a schema with support for backward and forward
compatibility
• Kafka handles event streaming very well
• brings many more interesting features beyond just “message passing”, i.e. Log
compaction
• Kafka broker is not a full-fledged Event Store
• For Event Sourcing additional capabilities are needed (Kafka Streams, Axon, …)
• See also: Kafka as an Event Store: is it good enough? (slides, video)

gschmutz
Further Information
• Kafka as an Event Store: is it good enough?, Guido Schmutz, Trivadis: https://www.slideshare.net/gschmutz/kafka-
as-an-event-store-is-it-good-enough
• Microservices Blog Series, Ben Stopford, Confluent: https://www.confluent.io/blog/tag/microservices
• Apache Kafka for Microservices: A Confluent Online Talk Series: https://www.confluent.io/landing-
page/microservices-online-talk-series/
• Schemas, Contracts, and Compatibility, Gwen Shapira, Confluent: https://www.confluent.io/blog/schemas-
contracts-compatibility
• Should You Put Several Event Types in the Same Kafka Topic?, Martin Kleppmann:
https://www.confluent.io/blog/put-several-event-types-kafka-topic/
• Turning the database inside-out with Apache Samza, Martin Kleppmann: https://www.confluent.io/blog/turning-the-
database-inside-out-with-apache-samza/
• Immutability Changes Everything, Pat Helland, Salesforce: http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

gschmutz
Technology on its own won't help
you.
You need to know how to use it
properly.
gschmutz

gschmutz
Event Sourcing
persists the state of an aggregate as a
sequence of state-changing events
Each event describes a state change
that occurred to the aggregate in the
past
new event is appended to the list of
events
an aggregate’s current state is
reconstructed by replaying the events
=> a.k.a ”rehydration”
Rehydration also needed for queries
Event Store
ServiceApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
Service
Subscribe
publish
publish
apply (append)
REST
Data Storage
trigger replycommand
command
1 2
3
4
5
5

gschmutz
Event Sourcing & CQRS
Event sourcing is commonly combined
with the CQRS pattern
Combines best of Event Sourcing and
CQRS
Project events published by Event
Store into Read Model (Materialized
Views)
Write Model and Read Model might
only support eventual consistency
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
1
Event Store
publish
apply (append)
trigger reply
2
3
4
5
5
6

gschmutz
Event Store Capabilities
1. Append Events efficiently
2. Read aggregate’s events in order
3. Full Sequential Read (over all
aggregates)
4. Consistent writes
5. Event versioning
6. Subscribable event stream
7. Correction events (O)
8. Ingestion & event time, bi-temporal (O)
9. Adhoc-Query on event store (O)
10. Snapshot Optimization (O)
11. High-Availability and Reliability (O)

gschmutz
Kafka as an Event Store
# Capability Kafka Broker
1 Append events efficiently yes
2 Read aggregate’s events in order not efficiently
3 Full sequential Read yes
4 Consistent Writes no
5 Event Versioning yes (if Avro is used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header
9 Snapshot Optimization (O) no
10 Ad-Hoc Query on Events (O) no
11 High-Availability and Reliability (O) yes

Building Event-Driven (Micro) Services with Apache Kafka

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Event-Driven (Micro) Services with Apache Kafka

Similar to Building Event-Driven (Micro) Services with Apache Kafka (20)

More from Guido Schmutz

More from Guido Schmutz (20)

Recently uploaded

Recently uploaded (20)

Building Event-Driven (Micro) Services with Apache Kafka