Guido Schmutz, profile picture
Uploaded byGuido Schmutz
10,277 views

Kafka as an Event Store - is it Good Enough?

The document discusses the viability of using Kafka as an event store, outlining traditional application building methods and contrasting them with event sourcing and CQRS patterns. It elaborates on the concepts of event sourcing, the benefits of persisting state-changing events, and compares Kafka with other event store implementations. Additionally, it provides insights on Kafka's capabilities and implementation strategies for event sourcing and integration with frameworks like Axon.

Data & Analytics◦
Related topics:
Kafka Summit•

Embed presentation

Downloaded 252 times
gschmutz Kafka as an Event Store – is it Good Enough? Guido Schmutz Kafka Summit New York – 2.4.2019 gschmutz http://guidoschmutz.wordpress.com
gschmutz Agenda 1. How do we build applications traditionally? 2. CQRS & Event Sourcing 3. What exactly is an Event Store? 4. Implementing Event Store 5. Summary Kafka as an Event Store – is it Good Enough?
gschmutz Guido Schmutz Working at Trivadis for more than 22 years Oracle Groundbreaker Ambassador & Oracle ACE Director Consultant, Trainer, Solution Architect for Java, Oracle, Microservices and Big Data / Fast Data Head of Trivadis Architecture Board More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz 151st edition Kafka as an Event Store – is it Good Enough?
gschmutz How do we build applications traditionally? Kafka as an Event Store – is it Good Enough?
gschmutz Data Access Layer Monolithic Applications – using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Kafka as an Event Store – is it Good Enough?
gschmutz Data Access Layer Monolithic Applications – using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Object/Relational Impedance Mismatch • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity Kafka as an Event Store – is it Good Enough?
gschmutz • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity Data Access Layer Monolithic Applications – using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Object/Relational Mapping SELECT acc.id, acc.account_number, acc.account_type.id, acct.name, cus.first_name, cus.last_name, addr.street, addr.city FROM account_t acc , account_type_t acct , customer_t cus , cust_adr_t cusa , address_t addr WHERE acc.account_type_id = acct.account_type_id AND adc.customer_id = cus.customer_id AND cusa.customer_id = cus.customer_id AND cusa.address_id = addr.address_id AND cusa.type = 'MAIN' Kafka as an Event Store – is it Good Enough?
gschmutz Microservices - using Layered Architecture Microservices … • are responsible for their data • might use NoSQL instead of RDBMS • often still use traditional approach to persistence • “Data silos” do no longer support database join • keep synchronous communication to a minimum Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Account API Account Account Logic Product Microservice { } Product API Product Product Logic Finance App Finance UI UI Logic GUI REST REST REST Kafka as an Event Store – is it Good Enough? sync request/response async, event pub/sub
gschmutz Domain Driven Design (DDD) – Concepts Kafka as an Event Store – is it Good Enough? Domain Objects – hold the state of the application Entity – Domain Objects with an identity Value Object – an immutable type that is distinguishable only by the state of its properties and has no identity Aggregate - A cluster of domain objects that can be treated as a single unit Aggregate Root – one object of aggregate is root object. Any reference from outside goes through aggregate root Aggregate Root Account Aggregate Customer Aggregate Aggregate Root
gschmutz Events Distribute to all handlers strong ordering req’s No results Queries Route with load balancing Sometimes scatter-gather Provide result Three mechanisms through which services can interact Kafka as an Event Store – is it Good Enough? Commands Route to single handler Use consistent hashing Provide Result
gschmutz Microservices with Event-driven communication Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic REST REST REST Pub / Sub Customer Mat View Kafka as an Event Store – is it Good Enough? sync request/response async, event pub/sub Finance App Finance UI UI Logic GUI This is Event Streaming and not really Event Sourcing
gschmutz CQRS & Event Sourcing Kafka as an Event Store – is it Good Enough?
gschmutz Command Query Responsibility Segregation (CQRS) Optimize for write and read differently API is split between • commands - trigger changes in state • queries - provide read access to the state Still using CRUD pattern, but separates ”R” from CRUD Might involve eventual consistency between write and read model Data Storage Write Model Read Model (read-only) Service Command API Query API App UI Projection Handler UI Logic Kafka as an Event Store – is it Good Enough? publish command query project read insert update delete 1 2 3 4
gschmutz Event Sourcing – Persist state-changing events and not state Kafka as an Event Store – is it Good Enough? # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000
gschmutz Event Sourcing persists the state of an aggregate as a sequence of state-changing events Each event describes a state change that occurred to the aggregate in the past new event is appended to the list of events an aggregate’s current state is reconstructed by replaying the events => a.k.a ”rehydration” Rehydration also needed for queries Kafka as an Event Store – is it Good Enough? Event Store ServiceApp UI UI Logic Command API & Handler Event Handler(s) Service Subscribe publish publish apply (append) REST Data Storage trigger replycommand command 1 2 3 4 5 5
gschmutz Event Sourcing - ”Rehydrate” State Kafka as an Event Store – is it Good Enough? 1. Create an empty Aggregate object 2. Read all events stored for that Aggregate from event store 3. Apply each event to the Aggregate object in the correct order AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 Account <empty> Account id: 123 accountType: Savings balance: 0 Account id: 123 accountType: Savings balance: 3000 transactions: [+1000, +2000] Account id: 123 accountType: Savings balance: 2500 transactions:[+1000, +2000, -500] Account id: 123 accountType: Savings balance: 1000 transactions: [+1000] applyTo applyTo applyTo applyTo
gschmutz Event Sourcing – Write Path CreateAccount command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) AccountCreated id: 123 accountType: Savings 10 # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} Account Aggregate <empty>
gschmutz Event Sourcing – Write Path DepositMoney command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 10 20 Account Aggregate id: 123 accountType: Savings balance: 0
gschmutz Event Sourcing – Write Path DepositMoney command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 10 20 100 Account Aggregate id: 123 accountType: Savings balance: 1000
gschmutz Event Sourcing – Write Path WithdrawMoney command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000 Account Aggregate id: 123 accountType: Savings balance: 3000
gschmutz Event Sourcing - Potential Benefits Kafka as an Event Store – is it Good Enough? 1. Subscribe to changes from other Aggregates 2. Examine a historical record of every change that has ever been applied on the model 3. Use the event store data for trend, forcast and other business analytics 4. Consider “what if” questions by replaying events to Aggregates which have experimental enhancements 5. Patch errors by adding ”correction” events (if it is legally allowed) 6. Perform “undo” and “redo” operations by replying varying sets of Events
gschmutz Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern Combines best of Event Sourcing and CQRS Project events published by Event Store into Read Model (Materialized Views) Write Model and Read Model might only support eventual consistency Kafka as an Event Store – is it Good Enough? AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project 1 Event Store publish apply (append) trigger reply 2 3 4 5 5 6
gschmutz What is an Event Store? Kafka as an Event Store – is it Good Enough?
gschmutz Event Store Capabilities Kafka as an Event Store – is it Good Enough? 1. Append Events efficiently 2. Read aggregate’s events in order 3. Full Sequential Read (over all aggregates) 4. Consistent writes 5. Event versioning 6. Subscribable event stream 7. Correction events (O) 8. Ingestion & event time, bi-temporal (O) 9. Adhoc-Query on event store (O) 10. Snapshot Optimization (O) 11. High-Availability and Reliability (O)
gschmutz Implementing an Event Store Kafka as an Event Store – is it Good Enough?
gschmutz Event Store Implementations Kafka as an Event Store – is it Good Enough? • Event Store (https://eventstore.org/) – by Greg Young • Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ • Axon DB (https://axoniq.io/) - by Axon IQ • Eventuate (https://eventuate.io/) – by Eventuate.io • Serialized (https://serialized.io/) – by Serialized.io • Build your own …. • Apache Kafka ???
gschmutz Implementing an Event Store: using Kafka Broker Kafka as an Event Store – is it Good Enough?
gschmutz Kafka as an Event Store Kafka as an Event Store – is it Good Enough? 1. One, single-partitioned Kafka topic per Aggregate 2. One, partitioned Kafka topic per Aggregate Type 3. One single, highly partitioned Kafka topic for all Aggregate Types Should you put several Event Types in the same Kafka topic?: https://www.confluent.io/blog/put-several-event-types-kafka-topic/
gschmutz 1) One, single-partitioned Kafka topic per Aggregate Instance Kafka as an Event Store – is it Good Enough? This will guarantee that the events are stored in order Reading state of an aggregate is as simple as reading a topic from offset 0 Not really feasible as there will be just too many topics needed Kafka Customer Aggregate Account Aggregate
gschmutz 2) One, partitioned Kafka topic per Aggregate Type Kafka as an Event Store – is it Good Enough? Required number of partitions is dependent on number of aggregate instances Events are produced with aggregate-id as the key guarantees that events are stored in order For reading state of an aggregate, all data of all aggregate instances have to be scanned => slow Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
gschmutz 3) One single, highly partitioned Kafka topic for all Aggregate Types Kafka as an Event Store – is it Good Enough? Required number of partitions is dependent on number of aggregate types * instances Events are produced with aggregate-id as the key guarantees that events are stored in order For reading state of an aggregate, all data of all aggregate types & instances have to be scanned => really slow Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
gschmutz Kafka as an Event Store Kafka as an Event Store – is it Good Enough? # Capability Kafka Broker 1 Append events efficiently yes 2 Read aggregate’s events in order not efficiently 3 Full sequential Read yes 4 Consistent Writes no 5 Event Versioning yes (if Avro is used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header 9 Snapshot Optimization (O) no 10 Ad-Hoc Query on Events (O) no 11 High-Availability and Reliability (O) yes
gschmutz Event Store Kafka is not a Database … a Database is not Kafka We can use Kafka to run part of our own Event Store implementation add a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Kafka as an Event Store – is it Good Enough? Other App Consumer
gschmutz Event Store Kafka is not a Database … a Database is not Kafka We can use Kafka to run our own Event Store implementation adding a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Kafka as an Event Store – is it Good Enough? Other App Consumer
gschmutz Event StoreEvent Store Two solutions for avoiding «dual write» Write Event first then consume it to write it to database Write through database (CDC, outbox design pattern) Application { } API Database Biz Logic REST Kafka as an Event Store – is it Good Enough? Event Hub Other App Biz Logic Application { } API Database REST Biz Logic CDC Event Hub CDC Connector Other App Biz Logic Publish
gschmutz Implementing an Event Store: using Axon Framework Kafka as an Event Store – is it Good Enough?
gschmutz Axon Kafka as an Event Store – is it Good Enough? • Spring Boot with Axon Framework for Application • MongoDB for Event Store • Kafka Broker for Event Bus • Kafka Streams or KSQL for Projection Handler • Kafka Connect / Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
gschmutz Event Sourcing with Axon Kafka as an Event Store – is it Good Enough? Account Events Account Command Account Aggregate Account Command Response Account App Event Store Account Customer Projection Command Handler Event Handler Account Query Projection Handler Query Handler Account Query Account Query Response Customer Event https://github.com/gschmutz/various-demos/tree/master/event-sourcing
gschmutz Event Sourcing with Axon - Aggregate Kafka as an Event Store – is it Good Enough? @Aggregate public class AccountAggregate{ @AggregateIdentifier private String id; private BigDecimal balance; private String forCustomerId; private String accountType; @CommandHandler ... @EventSourcingHandler ...
gschmutz Event Sourcing with Axon - Command Handler Kafka as an Event Store – is it Good Enough? @CommandHandler public AccountAggregate(AccountCreateCommand command) { Assert.hasLength(command.getForCustomerId(), "CustomerId must have a value"); Assert.hasLength(command.getAccountType(), "AccountType must have a value"); ... apply(new AccountCreatedEvent(command.getId(), command.getForCustomerId(), command.getAccountType(), new BigDecimal("0"))); }
gschmutz Event Sourcing with Axon – Command Handler Kafka as an Event Store – is it Good Enough? @CommandHandler public void on(WithdrawMoneyCommand command) { Assert.isTrue(command.getAmount() > 0, "Amount should be a positive number"); if(command.getAmount().compareTo(this.balance) > 0 ) { throw new InsufficientBalanceException( "Insufficient balance. Trying to withdraw:" + command.getAmount() + ", but current balance is: " + this.balance); } apply(new MoneyWithdrawnEvent(command.getId(), command.getAmount())); }
gschmutz Event Sourcing with Axon – Event Handler Kafka as an Event Store – is it Good Enough? @EventSourcingHandler public void handle(AccountCreatedEvent event) { id = event.getId(); forCustomerId = event.getForCustomerId(); accountType = event.getAccountType(); balance = event.getBalance(); } @EventSourcingHandler public void handle(MoneyWithdrawnEvent event) { balance = balance.subtract(event.getAmount()); }
gschmutz Event Sourcing with Axon – Projection Handler Kafka as an Event Store – is it Good Enough? public class AccountQueryController { @Autowired private AccountRepository accRepo; @EventHandler public void on(AccountCreatedEvent event,@Timestamp Instant instant) { Account account = new Account(event.getId(),event.getBalance(), event.getAccHolder(),event.getAccHolderName(), instant.toString()); accRepo.insert(account); } @EventHandler public void on(MoneyDepositedEvent event,@Timestamp Instant instant) { Account account = accRepo.findByAccountNo(event.getId()); account.setBalance(account.getBalance().add(event.getAmount())); account.setLastUpdated(instant.toString()); accRepo.save(account); }
gschmutz Axon Fwk with Axon DB Kafka as an Event Store – is it Good Enough? • Spring Boot with Axon Framework for Application • Axon DB for Event Store and Event Bus • Spring Boot for Projection Handler • Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
gschmutz Axon as an Event Store Kafka as an Event Store – is it Good Enough? # Capability Axon Framework Axon Framework & Axon DB 1 Append events efficiently yes yes 2 Read aggregate’s events in order yes yes 3 Full sequential Read yes yes 4 Consistent Writes yes yes 5 Event Versioning yes yes 6 Subscribeable Event Stream yes yes 7 Correction events (O) no no 8 Event time & ingestion time, aka. Bi-temporal (O) no no 9 Snapshot Optimization (O) yes yes 10 Ad-Hoc Query on Events (O) yes yes 11 High-Availability and Reliability (O) possible yes
gschmutz Implementing an Event Store: using Kafka and Kafka Streams Kafka as an Event Store – is it Good Enough?
gschmutz Kafka & Kafka Streams Kafka as an Event Store – is it Good Enough? Kafka Streams with State for Event Store Kafka Broker for Event Bus Kafka Streams or KSQL for Projection Handler No reply of events, current snapshot is held in state store AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
gschmutz Account Event Handler Event Sourcing with Kafka Streams Account Created Money Deposited Money Withdrawn Command Account Command Handler Command Response Account API Account Snapshot Account Snapshot Customer Snapshot Account Customer API Account Customer Projection Account Customer Projector Account Customer Rec persist https://github.com/gschmutz/various-demos/tree/master/event-sourcing
gschmutz Kafka & Kafka Streams as an Event Store Kafka as an Event Store – is it Good Enough? # Capability Kafka & Kafka Streams 1 Append events efficiently yes 2 Read aggregate’s events in order no (snapshot state only holds current snapshot) 3 Full sequential Read no 4 Consistent Writes yes (only one event per aggregate in flight) 5 Event Versioning yes (if Avro used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no 9 Snapshot Optimization (O) yes (snapshot state only) 10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …) 11 High-Availability and Reliability (O) yes
gschmutz Summary Kafka as an Event Store – is it Good Enough?
gschmutz Summary • Event Sourcing and CQRS might be more natural to business people than IT => we are used to work with “CRUD based persistence” • Event Sourcing provides history and logging for free • Kafka Broker alone is really “just” Event Streaming, not Event Sourcing • Axon Framework supports the implementation of Event Sourcing applications with Pluggable Event Store and Event Bus implementations • Axon DB implements an Event Store and an Event Bus • Kafka and Kafka Streams with State Store supports event sourcing in a ”streaming fashion” Kafka as an Event Store – is it Good Enough?
gschmutz Technology on its own won't help you. You need to know how to use it properly. Kafka as an Event Store – is it Good Enough?

More Related Content

PPTX
Introduction to Apache Kafka
byAIMDek Technologies
 
ODP
Stream processing using Kafka
byKnoldus Inc.
 
PDF
Angular - Chapter 4 - Data and Event Handling
byWebStackAcademy
 
PDF
Kafka Streams: What it is, and how to use it?
byconfluent
 
PDF
MongoDB Aggregation Framework
byCaserta
 
PDF
Service Worker Presentation
byKyle Dorman
 
PDF
Event Streaming with Kafka Streams and Spring Cloud Stream | Soby Chacko, VMware
byHostedbyConfluent
 
PPTX
Autoscaling Flink with Reactive Mode
byFlink Forward
 
Introduction to Apache Kafka
byAIMDek Technologies
 
Stream processing using Kafka
byKnoldus Inc.
 
Angular - Chapter 4 - Data and Event Handling
byWebStackAcademy
 
Kafka Streams: What it is, and how to use it?
byconfluent
 
MongoDB Aggregation Framework
byCaserta
 
Service Worker Presentation
byKyle Dorman
 
Event Streaming with Kafka Streams and Spring Cloud Stream | Soby Chacko, VMware
byHostedbyConfluent
 
Autoscaling Flink with Reactive Mode
byFlink Forward
 

What's hot

PDF
Introduction to Kafka Streams
byGuozhang Wang
 
PDF
Angular and The Case for RxJS
bySandi Barr
 
PPTX
The RabbitMQ Message Broker
byMartin Toshev
 
PDF
Event Sourcing with Kotlin, who needs frameworks!
byNico Krijnen
 
PPTX
Introduction to RxJS
byAbul Hasan
 
PPTX
Kafka 101
byClement Demonchy
 
PPTX
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
bySlim Baltagi
 
PDF
Cours design pattern m youssfi partie 6 proxy
byENSET, Université Hassan II Casablanca
 
PPTX
Monoliths and Microservices
byBozhidar Bozhanov
 
PDF
Angular - Chapter 5 - Directives
byWebStackAcademy
 
PPTX
ARCHITECTURE MICROSERVICE : TOUR D’HORIZON DU CONCEPT ET BONNES PRATIQUES
bySOAT
 
PDF
Using Kafka: Anatomy of the Flowable event registry
byFlowable
 
PPT
Hibernate architecture
byAnurag
 
PPTX
Intro to WebSockets
byGaurav Oberoi
 
PPTX
02 api gateway
byJanani Velmurugan
 
PDF
Introduction to AMQP Messaging with RabbitMQ
byDmitriy Samovskiy
 
ODP
Introduction to Python Celery
byMahendra M
 
PDF
angular fundamentals.pdf
byNuttavutThongjor1
 
PDF
Implementing Domain Events with Kafka
byAndrei Rugina
 
PPTX
Write microservice in golang
byBo-Yi Wu
 
Introduction to Kafka Streams
byGuozhang Wang
 
Angular and The Case for RxJS
bySandi Barr
 
The RabbitMQ Message Broker
byMartin Toshev
 
Event Sourcing with Kotlin, who needs frameworks!
byNico Krijnen
 
Introduction to RxJS
byAbul Hasan
 
Kafka 101
byClement Demonchy
 
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree
bySlim Baltagi
 
Cours design pattern m youssfi partie 6 proxy
byENSET, Université Hassan II Casablanca
 
Monoliths and Microservices
byBozhidar Bozhanov
 
Angular - Chapter 5 - Directives
byWebStackAcademy
 
ARCHITECTURE MICROSERVICE : TOUR D’HORIZON DU CONCEPT ET BONNES PRATIQUES
bySOAT
 
Using Kafka: Anatomy of the Flowable event registry
byFlowable
 
Hibernate architecture
byAnurag
 
Intro to WebSockets
byGaurav Oberoi
 
02 api gateway
byJanani Velmurugan
 
Introduction to AMQP Messaging with RabbitMQ
byDmitriy Samovskiy
 
Introduction to Python Celery
byMahendra M
 
angular fundamentals.pdf
byNuttavutThongjor1
 
Implementing Domain Events with Kafka
byAndrei Rugina
 
Write microservice in golang
byBo-Yi Wu
 

Similar to Kafka as an Event Store - is it Good Enough?

PDF
Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019
byconfluent
 
PDF
Kafka as an event store - is it good enough?
byGuido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
PDF
Building event-driven Microservices with Kafka Ecosystem
byGuido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Building Event-Driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
PPTX
Jun Rao, Confluent | Kafka Summit SF 2019 Keynote ft. Chris Kasten, Walmart Labs
byconfluent
 
PDF
Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
byconfluent
 
PPTX
Beyond Microservices: Streams, State and Scalability
byconfluent
 
PDF
Developing event-driven microservices with event sourcing and CQRS (phillyete)
byChris Richardson
 
PDF
Event Driven Services Part 2: Building Event-Driven Services with Apache Kafka
byBen Stopford
 
PDF
Building Event-Driven (Micro) Services with Apache Kafka
byGuido Schmutz
 
PDF
Microservices with Kafka Ecosystem
byGuido Schmutz
 
PDF
Eventsourcing you-are-doing-it-wrong-vxdparis
byDavid Schmitz
 
PDF
Building Event-Driven Services with Apache Kafka
byconfluent
 
PDF
#JaxLondon: Building microservices with Scala, functional domain models and S...
byChris Richardson
 
PDF
Building Microservices with Scala, functional domain models and Spring Boot -...
byJAXLondon2014
 
PDF
Event driven microservices
byAnthony Martin
 
PPTX
10 Principals for Effective Event Driven Microservices
byBen Stopford
 
Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019
byconfluent
 
Kafka as an event store - is it good enough?
byGuido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Building event-driven Microservices with Kafka Ecosystem
byGuido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Building Event-Driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Jun Rao, Confluent | Kafka Summit SF 2019 Keynote ft. Chris Kasten, Walmart Labs
byconfluent
 
Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...
byconfluent
 
Beyond Microservices: Streams, State and Scalability
byconfluent
 
Developing event-driven microservices with event sourcing and CQRS (phillyete)
byChris Richardson
 
Event Driven Services Part 2: Building Event-Driven Services with Apache Kafka
byBen Stopford
 
Building Event-Driven (Micro) Services with Apache Kafka
byGuido Schmutz
 
Microservices with Kafka Ecosystem
byGuido Schmutz
 
Eventsourcing you-are-doing-it-wrong-vxdparis
byDavid Schmitz
 
Building Event-Driven Services with Apache Kafka
byconfluent
 
#JaxLondon: Building microservices with Scala, functional domain models and S...
byChris Richardson
 
Building Microservices with Scala, functional domain models and Spring Boot -...
byJAXLondon2014
 
Event driven microservices
byAnthony Martin
 
10 Principals for Effective Event Driven Microservices
byBen Stopford
 

More from Guido Schmutz

PDF
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
PDF
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
PDF
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
PDF
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
PDF
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
PDF
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Streaming Visualisation
byGuido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
PDF
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
Streaming Visualisation
byGuido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 

Recently uploaded

PPTX
[DSC Europe 25] Dusan Jovicic - AI Story: From on-prem to cloud and back agai...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...
byDataScienceConferenc1
 
PDF
Kate Scrivens-OECD-Community well-being and climate risks.pdf
byStatsCommunications
 
PPTX
Exploring Quantum Computing Simulation Tools and Fundamentals Algorithms.pptx
byfarhanafayk
 
PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PPTX
Orientation-on-ARAL-Check-in-Assessment.pptx
byJenTeruel1
 
PPTX
software define network authentication .pptx
byAhmedGouda93
 
PPTX
First introduction to pipeline pilot a datascience software
byJamesImmanuel4
 
PPTX
Time serise forcasting of non-durble consumer goods industry
byvkumaranand89
 
PPTX
GRADE 8 MATH - Q3 - LESSON 1 -LINEAR EQUATION IN ONE VARIABLE.pptx
byEdlynMalangPanti
 
PPTX
essential in computer science structure and other
byhncseven7
 
PPTX
Python Functions and Modules Detailed Notes.pptx
byJayantAbhinav
 
PPTX
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
PDF
Natal SM 2025 - Latihan Liturgi Kelas 0 - 3 SD.pdf
byWenny63
 
PPT
Perceptron and Activation Function in deep learning
byShrideviS7
 
PPTX
[DSC Europe 25] Boris Perkovic - Lost in performance.pptx
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...
byDataScienceConferenc1
 
PDF
[DSC Europe 25] Marija Vlajkovic & Andrea Radonjanin - Integration of AI tool...
byDataScienceConferenc1
 
[DSC Europe 25] Dusan Jovicic - AI Story: From on-prem to cloud and back agai...
byDataScienceConferenc1
 
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...
byDataScienceConferenc1
 
Kate Scrivens-OECD-Community well-being and climate risks.pdf
byStatsCommunications
 
Exploring Quantum Computing Simulation Tools and Fundamentals Algorithms.pptx
byfarhanafayk
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
Orientation-on-ARAL-Check-in-Assessment.pptx
byJenTeruel1
 
software define network authentication .pptx
byAhmedGouda93
 
First introduction to pipeline pilot a datascience software
byJamesImmanuel4
 
Time serise forcasting of non-durble consumer goods industry
byvkumaranand89
 
GRADE 8 MATH - Q3 - LESSON 1 -LINEAR EQUATION IN ONE VARIABLE.pptx
byEdlynMalangPanti
 
essential in computer science structure and other
byhncseven7
 
Python Functions and Modules Detailed Notes.pptx
byJayantAbhinav
 
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
Natal SM 2025 - Latihan Liturgi Kelas 0 - 3 SD.pdf
byWenny63
 
Perceptron and Activation Function in deep learning
byShrideviS7
 
[DSC Europe 25] Boris Perkovic - Lost in performance.pptx
byDataScienceConferenc1
 
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...
byDataScienceConferenc1
 
[DSC Europe 25] Marija Vlajkovic & Andrea Radonjanin - Integration of AI tool...
byDataScienceConferenc1
 

Kafka as an Event Store - is it Good Enough?

  • 1.
    gschmutz Kafka as anEvent Store – is it Good Enough? Guido Schmutz Kafka Summit New York – 2.4.2019 gschmutz http://guidoschmutz.wordpress.com
  • 2.
    gschmutz Agenda 1. How dowe build applications traditionally? 2. CQRS & Event Sourcing 3. What exactly is an Event Store? 4. Implementing Event Store 5. Summary Kafka as an Event Store – is it Good Enough?
  • 3.
    gschmutz Guido Schmutz Working atTrivadis for more than 22 years Oracle Groundbreaker Ambassador & Oracle ACE Director Consultant, Trainer, Solution Architect for Java, Oracle, Microservices and Big Data / Fast Data Head of Trivadis Architecture Board More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz 151st edition Kafka as an Event Store – is it Good Enough?
  • 4.
    gschmutz How do webuild applications traditionally? Kafka as an Event Store – is it Good Enough?
  • 5.
    gschmutz Data Access Layer MonolithicApplications – using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Kafka as an Event Store – is it Good Enough?
  • 6.
    gschmutz Data Access Layer MonolithicApplications – using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Object/Relational Impedance Mismatch • Traditional approach to persistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity Kafka as an Event Store – is it Good Enough?
  • 7.
    gschmutz • Traditional approach topersistence • Store current state • CRUD operations • Coupling between read & write • Increased Complexity Data Access Layer Monolithic Applications – using Layered Architecture User Interface Account UI Service Layer { } Account API Database Customer API REST Customer UI Domain Model Account DAO { }REST Customer DAO DB Model Object/Relational Mapping SELECT acc.id, acc.account_number, acc.account_type.id, acct.name, cus.first_name, cus.last_name, addr.street, addr.city FROM account_t acc , account_type_t acct , customer_t cus , cust_adr_t cusa , address_t addr WHERE acc.account_type_id = acct.account_type_id AND adc.customer_id = cus.customer_id AND cusa.customer_id = cus.customer_id AND cusa.address_id = addr.address_id AND cusa.type = 'MAIN' Kafka as an Event Store – is it Good Enough?
  • 8.
    gschmutz Microservices - usingLayered Architecture Microservices … • are responsible for their data • might use NoSQL instead of RDBMS • often still use traditional approach to persistence • “Data silos” do no longer support database join • keep synchronous communication to a minimum Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Account API Account Account Logic Product Microservice { } Product API Product Product Logic Finance App Finance UI UI Logic GUI REST REST REST Kafka as an Event Store – is it Good Enough? sync request/response async, event pub/sub
  • 9.
    gschmutz Domain Driven Design(DDD) – Concepts Kafka as an Event Store – is it Good Enough? Domain Objects – hold the state of the application Entity – Domain Objects with an identity Value Object – an immutable type that is distinguishable only by the state of its properties and has no identity Aggregate - A cluster of domain objects that can be treated as a single unit Aggregate Root – one object of aggregate is root object. Any reference from outside goes through aggregate root Aggregate Root Account Aggregate Customer Aggregate Aggregate Root
  • 10.
    gschmutz Events Distribute to allhandlers strong ordering req’s No results Queries Route with load balancing Sometimes scatter-gather Provide result Three mechanisms through which services can interact Kafka as an Event Store – is it Good Enough? Commands Route to single handler Use consistent hashing Provide Result
  • 11.
    gschmutz Microservices with Event-drivencommunication Customer Microservice { } Customer API Customer Customer Logic Account Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic REST REST REST Pub / Sub Customer Mat View Kafka as an Event Store – is it Good Enough? sync request/response async, event pub/sub Finance App Finance UI UI Logic GUI This is Event Streaming and not really Event Sourcing
  • 12.
    gschmutz CQRS & EventSourcing Kafka as an Event Store – is it Good Enough?
  • 13.
    gschmutz Command Query ResponsibilitySegregation (CQRS) Optimize for write and read differently API is split between • commands - trigger changes in state • queries - provide read access to the state Still using CRUD pattern, but separates ”R” from CRUD Might involve eventual consistency between write and read model Data Storage Write Model Read Model (read-only) Service Command API Query API App UI Projection Handler UI Logic Kafka as an Event Store – is it Good Enough? publish command query project read insert update delete 1 2 3 4
  • 14.
    gschmutz Event Sourcing –Persist state-changing events and not state Kafka as an Event Store – is it Good Enough? # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000
  • 15.
    gschmutz Event Sourcing persists thestate of an aggregate as a sequence of state-changing events Each event describes a state change that occurred to the aggregate in the past new event is appended to the list of events an aggregate’s current state is reconstructed by replaying the events => a.k.a ”rehydration” Rehydration also needed for queries Kafka as an Event Store – is it Good Enough? Event Store ServiceApp UI UI Logic Command API & Handler Event Handler(s) Service Subscribe publish publish apply (append) REST Data Storage trigger replycommand command 1 2 3 4 5 5
  • 16.
    gschmutz Event Sourcing -”Rehydrate” State Kafka as an Event Store – is it Good Enough? 1. Create an empty Aggregate object 2. Read all events stored for that Aggregate from event store 3. Apply each event to the Aggregate object in the correct order AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 Account <empty> Account id: 123 accountType: Savings balance: 0 Account id: 123 accountType: Savings balance: 3000 transactions: [+1000, +2000] Account id: 123 accountType: Savings balance: 2500 transactions:[+1000, +2000, -500] Account id: 123 accountType: Savings balance: 1000 transactions: [+1000] applyTo applyTo applyTo applyTo
  • 17.
    gschmutz Event Sourcing –Write Path CreateAccount command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) AccountCreated id: 123 accountType: Savings 10 # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} Account Aggregate <empty>
  • 18.
    gschmutz Event Sourcing –Write Path DepositMoney command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 10 20 Account Aggregate id: 123 accountType: Savings balance: 0
  • 19.
    gschmutz Event Sourcing –Write Path DepositMoney command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 10 20 100 Account Aggregate id: 123 accountType: Savings balance: 1000
  • 20.
    gschmutz Event Sourcing –Write Path WithdrawMoney command Kafka as an Event Store – is it Good Enough? Create an event for every state change of Aggregate Persist the stream to event store (preserving event order) # Timestamp Aggregate ID Event Event Payload 1 10 A32B3DE AccountCreated { id: 123, accountType: Savings} 2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000} 3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000} 4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500} AccountCreated id: 123 accountType: Savings MoneyDeposited id: 123 amount: 1000 MoneyDeposited id: 123 amount: 2000 MoneyWithdrawn id: 123 amount: 500 10 20 100 2000 Account Aggregate id: 123 accountType: Savings balance: 3000
  • 21.
    gschmutz Event Sourcing -Potential Benefits Kafka as an Event Store – is it Good Enough? 1. Subscribe to changes from other Aggregates 2. Examine a historical record of every change that has ever been applied on the model 3. Use the event store data for trend, forcast and other business analytics 4. Consider “what if” questions by replaying events to Aggregates which have experimental enhancements 5. Patch errors by adding ”correction” events (if it is legally allowed) 6. Perform “undo” and “redo” operations by replying varying sets of Events
  • 22.
    gschmutz Event Sourcing &CQRS Event sourcing is commonly combined with the CQRS pattern Combines best of Event Sourcing and CQRS Project events published by Event Store into Read Model (Materialized Views) Write Model and Read Model might only support eventual consistency Kafka as an Event Store – is it Good Enough? AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project 1 Event Store publish apply (append) trigger reply 2 3 4 5 5 6
  • 23.
    gschmutz What is anEvent Store? Kafka as an Event Store – is it Good Enough?
  • 24.
    gschmutz Event Store Capabilities Kafkaas an Event Store – is it Good Enough? 1. Append Events efficiently 2. Read aggregate’s events in order 3. Full Sequential Read (over all aggregates) 4. Consistent writes 5. Event versioning 6. Subscribable event stream 7. Correction events (O) 8. Ingestion & event time, bi-temporal (O) 9. Adhoc-Query on event store (O) 10. Snapshot Optimization (O) 11. High-Availability and Reliability (O)
  • 25.
    gschmutz Implementing an EventStore Kafka as an Event Store – is it Good Enough?
  • 26.
    gschmutz Event Store Implementations Kafkaas an Event Store – is it Good Enough? • Event Store (https://eventstore.org/) – by Greg Young • Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ • Axon DB (https://axoniq.io/) - by Axon IQ • Eventuate (https://eventuate.io/) – by Eventuate.io • Serialized (https://serialized.io/) – by Serialized.io • Build your own …. • Apache Kafka ???
  • 27.
    gschmutz Implementing an EventStore: using Kafka Broker Kafka as an Event Store – is it Good Enough?
  • 28.
    gschmutz Kafka as anEvent Store Kafka as an Event Store – is it Good Enough? 1. One, single-partitioned Kafka topic per Aggregate 2. One, partitioned Kafka topic per Aggregate Type 3. One single, highly partitioned Kafka topic for all Aggregate Types Should you put several Event Types in the same Kafka topic?: https://www.confluent.io/blog/put-several-event-types-kafka-topic/
  • 29.
    gschmutz 1) One, single-partitionedKafka topic per Aggregate Instance Kafka as an Event Store – is it Good Enough? This will guarantee that the events are stored in order Reading state of an aggregate is as simple as reading a topic from offset 0 Not really feasible as there will be just too many topics needed Kafka Customer Aggregate Account Aggregate
  • 30.
    gschmutz 2) One, partitionedKafka topic per Aggregate Type Kafka as an Event Store – is it Good Enough? Required number of partitions is dependent on number of aggregate instances Events are produced with aggregate-id as the key guarantees that events are stored in order For reading state of an aggregate, all data of all aggregate instances have to be scanned => slow Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
  • 31.
    gschmutz 3) One single,highly partitioned Kafka topic for all Aggregate Types Kafka as an Event Store – is it Good Enough? Required number of partitions is dependent on number of aggregate types * instances Events are produced with aggregate-id as the key guarantees that events are stored in order For reading state of an aggregate, all data of all aggregate types & instances have to be scanned => really slow Possible optimization: only read the partition where aggregate instance is stored Kafka Customer Aggregate Account Aggregate
  • 32.
    gschmutz Kafka as anEvent Store Kafka as an Event Store – is it Good Enough? # Capability Kafka Broker 1 Append events efficiently yes 2 Read aggregate’s events in order not efficiently 3 Full sequential Read yes 4 Consistent Writes no 5 Event Versioning yes (if Avro is used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header 9 Snapshot Optimization (O) no 10 Ad-Hoc Query on Events (O) no 11 High-Availability and Reliability (O) yes
  • 33.
    gschmutz Event Store Kafka isnot a Database … a Database is not Kafka We can use Kafka to run part of our own Event Store implementation add a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Kafka as an Event Store – is it Good Enough? Other App Consumer
  • 34.
    gschmutz Event Store Kafka isnot a Database … a Database is not Kafka We can use Kafka to run our own Event Store implementation adding a database to get missing capabilities But be careful with Dual Write! • Would need distributed transactions • Otherwise no guarantee for both writes to happen Application { } API DatabaseBiz Logic REST Event Hub Kafka as an Event Store – is it Good Enough? Other App Consumer
  • 35.
    gschmutz Event StoreEvent Store Twosolutions for avoiding «dual write» Write Event first then consume it to write it to database Write through database (CDC, outbox design pattern) Application { } API Database Biz Logic REST Kafka as an Event Store – is it Good Enough? Event Hub Other App Biz Logic Application { } API Database REST Biz Logic CDC Event Hub CDC Connector Other App Biz Logic Publish
  • 36.
    gschmutz Implementing an EventStore: using Axon Framework Kafka as an Event Store – is it Good Enough?
  • 37.
    gschmutz Axon Kafka as anEvent Store – is it Good Enough? • Spring Boot with Axon Framework for Application • MongoDB for Event Store • Kafka Broker for Event Bus • Kafka Streams or KSQL for Projection Handler • Kafka Connect / Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
  • 38.
    gschmutz Event Sourcing withAxon Kafka as an Event Store – is it Good Enough? Account Events Account Command Account Aggregate Account Command Response Account App Event Store Account Customer Projection Command Handler Event Handler Account Query Projection Handler Query Handler Account Query Account Query Response Customer Event https://github.com/gschmutz/various-demos/tree/master/event-sourcing
  • 39.
    gschmutz Event Sourcing withAxon - Aggregate Kafka as an Event Store – is it Good Enough? @Aggregate public class AccountAggregate{ @AggregateIdentifier private String id; private BigDecimal balance; private String forCustomerId; private String accountType; @CommandHandler ... @EventSourcingHandler ...
  • 40.
    gschmutz Event Sourcing withAxon - Command Handler Kafka as an Event Store – is it Good Enough? @CommandHandler public AccountAggregate(AccountCreateCommand command) { Assert.hasLength(command.getForCustomerId(), "CustomerId must have a value"); Assert.hasLength(command.getAccountType(), "AccountType must have a value"); ... apply(new AccountCreatedEvent(command.getId(), command.getForCustomerId(), command.getAccountType(), new BigDecimal("0"))); }
  • 41.
    gschmutz Event Sourcing withAxon – Command Handler Kafka as an Event Store – is it Good Enough? @CommandHandler public void on(WithdrawMoneyCommand command) { Assert.isTrue(command.getAmount() > 0, "Amount should be a positive number"); if(command.getAmount().compareTo(this.balance) > 0 ) { throw new InsufficientBalanceException( "Insufficient balance. Trying to withdraw:" + command.getAmount() + ", but current balance is: " + this.balance); } apply(new MoneyWithdrawnEvent(command.getId(), command.getAmount())); }
  • 42.
    gschmutz Event Sourcing withAxon – Event Handler Kafka as an Event Store – is it Good Enough? @EventSourcingHandler public void handle(AccountCreatedEvent event) { id = event.getId(); forCustomerId = event.getForCustomerId(); accountType = event.getAccountType(); balance = event.getBalance(); } @EventSourcingHandler public void handle(MoneyWithdrawnEvent event) { balance = balance.subtract(event.getAmount()); }
  • 43.
    gschmutz Event Sourcing withAxon – Projection Handler Kafka as an Event Store – is it Good Enough? public class AccountQueryController { @Autowired private AccountRepository accRepo; @EventHandler public void on(AccountCreatedEvent event,@Timestamp Instant instant) { Account account = new Account(event.getId(),event.getBalance(), event.getAccHolder(),event.getAccHolderName(), instant.toString()); accRepo.insert(account); } @EventHandler public void on(MoneyDepositedEvent event,@Timestamp Instant instant) { Account account = accRepo.findByAccountNo(event.getId()); account.setBalance(account.getBalance().add(event.getAmount())); account.setLastUpdated(instant.toString()); accRepo.save(account); }
  • 44.
    gschmutz Axon Fwk withAxon DB Kafka as an Event Store – is it Good Enough? • Spring Boot with Axon Framework for Application • Axon DB for Event Store and Event Bus • Spring Boot for Projection Handler • Spring Boot to persist in read model • NoSQL and/or RDBMS for read model AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
  • 45.
    gschmutz Axon as anEvent Store Kafka as an Event Store – is it Good Enough? # Capability Axon Framework Axon Framework & Axon DB 1 Append events efficiently yes yes 2 Read aggregate’s events in order yes yes 3 Full sequential Read yes yes 4 Consistent Writes yes yes 5 Event Versioning yes yes 6 Subscribeable Event Stream yes yes 7 Correction events (O) no no 8 Event time & ingestion time, aka. Bi-temporal (O) no no 9 Snapshot Optimization (O) yes yes 10 Ad-Hoc Query on Events (O) yes yes 11 High-Availability and Reliability (O) possible yes
  • 46.
    gschmutz Implementing an EventStore: using Kafka and Kafka Streams Kafka as an Event Store – is it Good Enough?
  • 47.
    gschmutz Kafka & KafkaStreams Kafka as an Event Store – is it Good Enough? Kafka Streams with State for Event Store Kafka Broker for Event Bus Kafka Streams or KSQL for Projection Handler No reply of events, current snapshot is held in state store AggregateApp UI UI Logic Command API & Handler Event Handler(s) REST Data Storage Query API Read Model (read-only) { } REST Projection Handler publish command query read project Event Store publish apply (append) trigger reply
  • 48.
    gschmutz Account Event Handler Event Sourcingwith Kafka Streams Account Created Money Deposited Money Withdrawn Command Account Command Handler Command Response Account API Account Snapshot Account Snapshot Customer Snapshot Account Customer API Account Customer Projection Account Customer Projector Account Customer Rec persist https://github.com/gschmutz/various-demos/tree/master/event-sourcing
  • 49.
    gschmutz Kafka & KafkaStreams as an Event Store Kafka as an Event Store – is it Good Enough? # Capability Kafka & Kafka Streams 1 Append events efficiently yes 2 Read aggregate’s events in order no (snapshot state only holds current snapshot) 3 Full sequential Read no 4 Consistent Writes yes (only one event per aggregate in flight) 5 Event Versioning yes (if Avro used) 6 Subscribeable Event Stream yes 7 Correction events (O) no 8 Event time & ingestion time, aka. Bi-temporal (O) no 9 Snapshot Optimization (O) yes (snapshot state only) 10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …) 11 High-Availability and Reliability (O) yes
  • 50.
    gschmutz Summary Kafka as anEvent Store – is it Good Enough?
  • 51.
    gschmutz Summary • Event Sourcingand CQRS might be more natural to business people than IT => we are used to work with “CRUD based persistence” • Event Sourcing provides history and logging for free • Kafka Broker alone is really “just” Event Streaming, not Event Sourcing • Axon Framework supports the implementation of Event Sourcing applications with Pluggable Event Store and Event Bus implementations • Axon DB implements an Event Store and an Event Bus • Kafka and Kafka Streams with State Store supports event sourcing in a ”streaming fashion” Kafka as an Event Store – is it Good Enough?
  • 52.
    gschmutz Technology on itsown won't help you. You need to know how to use it properly. Kafka as an Event Store – is it Good Enough?