This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
Building event-driven (Micro)Services with Apache Kafka
1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Ereignis-getriebene Microservices
mit Apache Kafka
Guido Schmutz
JUG Saxony Day - 28.9.2018
@gschmutz guidoschmutz.wordpress.com
2. Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Building event-driven Microservices with Kafka Ecosystem
3. COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers
Building event-driven Microservices with Kafka Ecosystem
4. Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not more Event Driven?
4. Why Kafka for Event-Driven Microservices?
5. What about streaming sources?
6. What about integrating legacy applications?
7. What about (historical) data analytics?
8. Summary
Building event-driven Microservices with Kafka Ecosystem
6. Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UI
DataGUI
Data
Storage
Shared
Database
Building event-driven Microservices with Kafka Ecosystem
sync request/response
7. Shop UI App
Business
Activity Service
SOA Approach
Building event-driven Microservices with Kafka Ecosystem
Contract-first
Web Services
Technical layers
offer their own
interfaces
Reuse on each
level
Lower layer
often wraps
legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
GUI
Business Entity
ServiceShop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
8. Shop UI App
Business
Activity Service
Virtualized SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI UI Logic
GUI
Business Entity
Service
Shop Web App
Shop UI UI Logic
GUI
Data
Storage
Customer
Database
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order and
Product DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Building event-driven Microservices with Kafka Ecosystem
10. Microservice Approach
Building event-driven Microservices with Kafka Ecosystem
Tightly Scoped behind clear
interfaces
Responsible for managing their
own data (not necessarily the
infrastructure)
Should be highly decoupled
Independently deployable, self-
contained and autonomous
SOA done right ?!
Tightly Scoped
Responsible for managing their own
data
Highly decoupled
Independently deployable, self-
contained and autonomous
{
}
Customer API
Customer
Customer Logic
Order Microservice
{
}
Order API
Order
Order Logic
Product Microservice
{
}
Product API
Product
Product Logic
Stock Microservice
{
}
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
11. Microservice Approach
with API Gateway
Customer Microservice
{
}
Customer API
Customer
Customer Logic
Order Microservice
{
}
Order API
Order
Order Logic
Product Microservice
{
}
Product API
Product
Product Logic
Stock Microservice
{
}
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Building event-driven Microservices with Kafka Ecosystem
12. Synchronous World of Request-Response leads to tight,
point-to-point couplings
Building event-driven Microservices with Kafka Ecosystem
problem in lower end of chain have a ripple
effect on other service
• crash of service
• overloaded service / slow response time
• change of interface
Service 2Service 1
{ }
API
Logic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
14. Three mechanisms through which services can interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Consume Event
“Event Ordered”
OrderEvent
Command
”Order IPad”
boolean order(IPad)
Publish Event
”Event Validated”
OrderValidatedEvent
Query
”Retrieve my Orders
List<Orders> getAllOrders(for)
Building event-driven Microservices with Kafka Ecosystem
Event Broker
Service
Logic
State
15. Event-Driven (Async)
Microservice Approach
Customer Microservice
{
}
Customer API
Customer
Customer Logic
Order Microservice
{
}
Order API
Order
Order Logic
Product Microservice
{
}
Product API
Product
Product Logic
Stock Microservice
{
}
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
API
Gateway
Building event-driven Microservices with Kafka Ecosystem
Event
Store
sync request/response
async request/response
async, event pub/sub
16. Why Kafka for Event-Driven
Microservices?
Building event-driven Microservices with Kafka Ecosystem
17. Apache Kafka – A Streaming Platform
Building event-driven Microservices with Kafka Ecosystem
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget
18. Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL)
log.retention.{ms | minutes | hours}
3. Size based
log.retention.bytes
4. Log compaction based
(entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181
--create --topic customers
--replication-factor 1
--partitions 1
--config cleanup.policy=compact
Building event-driven Microservices with Kafka Ecosystem
20. Topic Viewed as Event Stream or State Stream (Change
Log)
Event Stream State Stream (Change Log Stream)
2015-10-
02
11,Peter,Muster,3010,Berne
2016-10-
04
12,Paul,Steffen,8001,Zurich
2016-12-
02
21,Lisa,Meier,3043,Ittigen
2017-05-
03
11,Peter,Muster,3015,Berne
2017-05-
03
21,Lisa,Steffen,8001,Zurich
2017-07-
03
11,Peter,Muster,3052,Zollikofe
n
Building event-driven Microservices with Kafka Ecosystem
2015-10-
02
11,Peter,Muster,3010,Berne
2016-10-
04
12,Paul,Steffen,8001,Zurich
2016-12-
02
21,Lisa,Meier,3043,Ittigen
2017-05-
03
11,Peter,Muster,3015,Berne
2017-05-
03
21,Lisa,Steffen,8001,Zurich
2017-07-
03
11,Peter,Muster,3052,Zollikof
en
21. Keep Topics both in Original and Compacted Form
Building event-driven Microservices with Kafka Ecosystem
Kafka Broker
A B C B A E K A E A
C K
B E A C
K
Producer
Kafka Broker
A B C B A E K A E A
C K
B E A C
K
Producer
Consume and Produce
OR
TX
22. What about streaming sources?
Building event-driven Microservices with Kafka Ecosystem
23. How to work with
streaming data sources
Customer Microservice
{
}
Customer API
Customer
Customer Logic
Order Microservice
{
}
Order API
Order
Order Logic
Product Microservice
{
}
Product API
Product
Product Logic
Stock Microservice
{
}
Stock API
Stock
Stock Logic
Shop Web App
Shop UI UI Logic
GUI
REST
REST
REST
REST
Event
Store
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Event Stream
Building event-driven Microservices with Kafka Ecosystem
24. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Streaming Processing & Microservices Architecture
BI Tools
Search / Explore
Online & Mobile
Apps
Search
Event
Stream
Results
Stream Analytics
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Building event-driven Microservices with Kafka Ecosystem
25. Stream Processing Capability - Windowing
Introduction to Stream Processing
Computations over events done using
windows of data
Due to size and never-ending nature of it,
it’s not feasible to keep entire stream of
data in memory
A window of data represents a certain
amount of data where we can perform
computations on
Windows give the power to keep a
working memory and look back at recent
data efficiently
Time
Stream of Data Window of Data
26. Sliding Window (aka
Hopping Window) - uses
eviction and trigger policies
that are based on time: window
length and sliding interval
length
Fixed Window (aka Tumbling
Window) - eviction policy always
based on the window being full
and trigger policy based on
either the count of items in the
window or time
Session Window – composed
of sequences of temporarily
related events terminated by a
gap of inactivity greater than
some timeout
Stream Processing Capability - Windowing
Introduction to Stream Processing
Time TimeTime
27. Stream Processing Capability - Joining
Introduction to Stream Processing
Challenges of joining streams
1. Data streams need to be aligned as they
come because they have different timestamps
2. since streams are never-ending, the joins
must be limited; otherwise join will never end
3. join needs to produce results continuously as
there is no end to the data
Stream to Static (Table) Join
Stream to Stream Join (one window join)
Stream to Stream Join (two window join)
Stream-
to-Static
Join
Stream-
to-Stream
Join
Stream-
to-Stream
Join
Time
Time
Time
28. Stream Processing Capability - State Management
Introduction to Stream Processing
Necessary if stream processing use case
is dependent on previously seen data or
external data
Windowing, Joining and Pattern
Detection use State Management behind
the scenes
State Management services can be made
available for custom state handling logic
State needs to be managed as close to
the stream processor as possible
Options for State Management
How does it handle failures? If a machine
crashes and the/some state is lost?
In-Memory
Replicated,
Distributed
Store
Local,
Embedded
Store
Operational Complexity and Features
Low high
29. Apache Kafka – scalable message processing and
more!
Building event-driven Microservices with Kafka Ecosystem
Source
Connector
trucking_
driver
Kafka Broker
Sink
Connector
Stream
Processing
Schema
Registry
Kafka Kafka
30. Kafka Connect - Overview
Source
Connecto
r
Sink
Connecto
r
Introduction to Stream Processing
31. Kafka Streams - Overview
Introduction to Stream Processing
Designed as a simple and lightweight
library in Apache Kafka
no other dependencies than Kafka
Supports fault-tolerant local state
Supports Windowing (Fixed, Sliding and
Session) and Stream-Stream / Stream-
Table Joins
Millisecond processing latency, no
micro-batching
At-least-once and exactly-once
processing guarantees
KTable<Integer, Customer> customers =
builder.stream(”customer");
KStream<Integer, Order> orders =
builder.stream(”order");
KStream<Integer, String> enriched =
orders.leftJoin(customers, …);
joined.to(”orderEnriched");
trucking_
driver
Kafka Broker
Java Application
Kafka Streams
32. KSQL - Overview
Introduction to Stream Processing
STREAM and TABLE as first-class
citizens
• STREAM = data in motion
• TABLE = collected state of a stream
Stream Processing with zero coding
using SQL-like language
Built on top of Kafka Streams
Interactive (CLI) and headless (command
file)
ksql> CREATE STREAM customer_s
WITH (kafka_topic=‘customer',
value_format=‘AVRO');
Message
----------------
Stream created
ksql> SELECT * FROM customer_s
WHERE address->country = ‘Switzerland’;
...
trucking_
driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands
33. What about integrating legacy
applications?
Building event-driven Microservices with Kafka Ecosystem
34. Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
StateLogic
CDC
CDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
Results
Stream Processor
Reference /
Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a
global database trigger
Transform existing systems to event
producer
Building event-driven Microservices with Kafka Ecosystem
35. Legacy Microservice
Change Data Capture (CDC) with Kafka
Building event-driven Microservices with Kafka Ecosystem
RDBMS
cdc-
source
trucking_
driver
Customer Topic
elasticsearch
-sink NoSQL
36. Hadoop Clusterd
Hadoop Cluster
Stream Processing
Cluster
Integrate existing systems through CDC
BI Tools
Search / Explore
Online & Mobile
Apps
Search
Event
Stream
Results
Stream Processor
Reference /
Models
Dashboard
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Microservice Cluster
Microservice State
{ }
API
Event
Stream
Event
Stream
Event
Store
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
Building event-driven Microservices with Kafka Ecosystem
37. What about (historical) data
analytics?
Building event-driven Microservices with Kafka Ecosystem
38. Streaming & (Big) Data Analytics Architecture
Event
Stream
Hadoop Clusterd
Hadoop Cluster
Big Data Cluster
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
SQL
Search
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
Event
Store
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
File Import / SQL Import
Building event-driven Microservices with Kafka Ecosystem
40. Hadoop Clusterd
Hadoop Cluster
Big Data
Summary
Billing &
Ordering
CRM /
Profile
Marketing
Campaigns
SQL
Search
BI Tools
Enterprise Data
Warehouse
Search / Explore
Online & Mobile
Apps
File Import / SQL Import
Event
Hub
Parallel
Processing
Storage
Storage
RawRefined
Results
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Service
Location
Social
Click
stream
Sensor
Data
Mobile
Apps
Weather
Data
Stream Processing
Microservices
41. Summary
Building event-driven Microservices with Kafka Ecosystem
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into
• commands
• events
• queries
• Kafka is well suited as an event broker / event store
• brings many more interesting features beyond just “message passing”
42. References
Building event-driven Microservices with Kafka Ecosystem
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc
43. Technology on its own won't help you.
You need to know how to use it properly.
Building event-driven Microservices with Kafka Ecosystem