Andrzej Ludwikowski, profile picture
Uploaded byAndrzej Ludwikowski
PDF, PPTX152 views

Event Sourcing - what could go wrong - Devoxx BE

The document discusses event sourcing and command query responsibility segregation (CQRS) patterns at different levels of implementation. It covers the benefits of event sourcing including having a complete log of state changes, improved debugging, performance, scalability, and microservices integration. It also discusses challenges with increasing complexity, eventual consistency, and different event store and serialization options.

Embed presentation

Download as PDF, PPTX
1 / 122
2 / 122
3 / 122
4 / 122
5 / 122
6 / 122
7 / 122
8 / 122
9 / 122
10 / 122
11 / 122
12 / 122
13 / 122
14 / 122
15 / 122
16 / 122
17 / 122
18 / 122
19 / 122
20 / 122
21 / 122
22 / 122
23 / 122
24 / 122
25 / 122
26 / 122
27 / 122
28 / 122
29 / 122
30 / 122
31 / 122
32 / 122
33 / 122
34 / 122
35 / 122
36 / 122
37 / 122
38 / 122
39 / 122
40 / 122
41 / 122
42 / 122
43 / 122
44 / 122
45 / 122
46 / 122
47 / 122
48 / 122
49 / 122
50 / 122
51 / 122
52 / 122
53 / 122
54 / 122
55 / 122
56 / 122
57 / 122
58 / 122
59 / 122
60 / 122
61 / 122
62 / 122
63 / 122
64 / 122
65 / 122
66 / 122
67 / 122
68 / 122
69 / 122
70 / 122
71 / 122
72 / 122
73 / 122
74 / 122
75 / 122
76 / 122
77 / 122
78 / 122
79 / 122
80 / 122
81 / 122
82 / 122
83 / 122
84 / 122
85 / 122
86 / 122
87 / 122
88 / 122
89 / 122
90 / 122
91 / 122
92 / 122
93 / 122
94 / 122
95 / 122
96 / 122
97 / 122
98 / 122
99 / 122
100 / 122
101 / 122
102 / 122
103 / 122
104 / 122
105 / 122
106 / 122
107 / 122
108 / 122
109 / 122
110 / 122
111 / 122
112 / 122
113 / 122
114 / 122
115 / 122
116 / 122
117 / 122
118 / 122
119 / 122
120 / 122
121 / 122
122 / 122
Event Sourcing what could possibly go wrong? Andrzej Ludwikowski
About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
What is Event Sourcing? DB ShoppingCart { items=[itemA, itemB] }
What is Event Sourcing? DB DB ShoppingCart { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
What is Event Sourcing? DB DB ShoppingCart { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability ● microservices integration pattern
ES and CQRS Client Query Service Data access Queries Read model Read model Read models Command Service Domain Event store Commands
ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Transaction Command Service Domain Event store Commands
ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● Implementing event sourcing using a relational database
ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction Command Service Domain Event store Commands
ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased event store load ? - lags
ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction event bus Command Service Domain Event store Commands
ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Event store Commands
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Event store Commands
ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction At-least-once delivery Command Service Domain Event store Commands
ES and CQRS level 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
ES and CQRS level 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction ?
ES and CQRS level 3.2 Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation - locking vs Single Writer
ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of event store (ES based on Kafka)?
Not covered but worth to check ● Command Sourcing ● Event Collaboration
ES implementation? ● custom ● library ● framework
ES from domain perspective ● commands, events, state ● 2 methods ○ process(command: Command): List[Event] ○ apply(event: Event): State
ES from application perspective ● snapshotting ● fail-over/recover ● persistence ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
ES packaging ● Keep your domain clean with ES
import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} case class User (userId: UserId, name: String, email: Email, funds: Funds) { def process(command: UserCommand): Either[List[UserEvent]] = command match { case c: UpdateEmail => updateEmail(c) case c: DepositFunds => deposit(c) case c: WithdrawFunds => withdraw(c) ... } def apply(event: UserEvent): User = ??? //pattern matching }
ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
ES packaging ● domain logic ● domain validation ● 0 ES framework imports
library vs. framework ● Akka Persistence vs. Lagom ● Akka Persistence Typed
kalix.io
Kalix Fundamentals Self-Managed On-Prem Application Business Logic Frameworks Database Transport Security Infrastructure Kubernetes Operating System Virtualization Servers, Storage, Networking In the Cloud Application Business Logic Frameworks Database Transport Security Kubernetes Operating System Virtualization Servers, Storage, Networking Infrastructure Major challenges remain: • Architectural Complexity • Skill Availability • Time to Market • Cost Issues Self-Managed Managed by Cloud / Service Provider 42
Kalix Fundamentals Self-Managed On-Prem Application Business Logic Frameworks Database Transport Security Infrastructure Kubernetes Operating System Virtualization Servers, Storage, Networking In the Cloud Application Business Logic Frameworks Database Transport Security Kubernetes Operating System Virtualization Servers, Storage, Networking Infrastructure Self-Managed Managed by Cloud / Service Provider 43 Kalix Application Business Logic Frameworks Database Transport Security Kubernetes Operating System Virtualization Servers, Storage, Networking Infrastructure
Event store ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
Event store for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
Plain text Binary human-readable deserialization required
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
Speed https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
Size https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
● backward - V2 can read V1 Full compatibility Application Event store V1 V2
● forward - V2 can read V3 Full compatibility Application Event store V1, V2 V2 Application Application V2 V3
● forward - V2 can read V3 Full compatibility Event store Read model Read model Read models Projector V2 V3
Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
Protobuf schema management + clean domain - a lot of boilerplate code
Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
Avro deserialization Bytes Deserialization Object Reader Schema Writer Schema
Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
Avro + less boilerplate code +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
Avro vs. Protocol Buffers ● The best serialization strategy for Event Sourcing
Event payload ● delta event ● rich event (event enrichment) ● + metadata ○ seq_num ○ created_at ○ event_id ○ command_id ○ correlation_id
State replay time ● snapshotting ● write-through cache
Memory consumption
Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
Fixing state ● healing command
Updating all aggregates User(id) Command(user_id) Event(user_id) Event(user_id) Event(user_id)
Handling duplicates Event store Read model Read model Read models Projector event bus At-least-once delivery
https://www.seriouspuzzles.com/unicorns-in-fairy-land-500pc-jigsaw-puzzle-by-eurographics/
Handling duplicates Events Read model Read model Read models Projector event bus At-least-once delivery Event store
Handling duplicates Events Read model Read model Read models Projector event bus idempotent updates Event store
Event + seq_no Event + seq_no Handling duplicates Events Read model Read model Read models Projector event bus Event + seq_no read model update + seq_no Event store
Broken read model Events ad model ead model Read models Projector event bus Event store
Broken read model Events ad model ead model Read models Projector event bus read model update + offset (manual offset management) Event store
Multi aggregate transactional update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
Pessimistic compensation action User account Cinema show
Pessimistic compensation action User account Cinema show charged
Pessimistic compensation action User account Cinema show charged booked
Pessimistic compensation action User account Cinema show charged booked sold out
Pessimistic compensation action User account Cinema show charged booked booked sold out
Pessimistic compensation action User account Cinema show charged booked booked sold out
Pessimistic compensation action User account Cinema show charged booked booked sold out refund
Optimistic compensation action User account Cinema show charged booked sold out
Optimistic compensation action User account Cinema show booked booked sold out overbooked
Optimistic compensation action User account Cinema show charged booked booked sold out overbooked ?
Saga Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Updater event bus Transaction
Saga - choreography Command Service Domain Event store Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
Saga - orchestration Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Event store Commands
Saga ● should be persistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
Saga ● Sagas with ES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern ● Microservice Patterns
ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
ES and CQRS level 3.2 Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
Cluster = split brain 1 5 4 3 Load balancer 2
Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
Cluster = split brain 1 5 4 3 Load balancer 2
Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
Cluster best practises ● remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
kalix.io
Summary ● carefully choose ES lib/framework ● there is no perfect database for event sourcing ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
Rate me please :)
About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski

More Related Content

PDF
Event Sourcing - what could go wrong - Jfokus 2022
byAndrzej Ludwikowski
 
PDF
Event sourcing - what could possibly go wrong ? Devoxx PL 2021
byAndrzej Ludwikowski
 
PDF
Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemo...
byCodemotion
 
PDF
Event Sourcing - what could possibly go wrong?
byAndrzej Ludwikowski
 
PDF
Andrzej Ludwikowski - Event Sourcing - co może pójść nie tak?
bySegFaultConf
 
PPTX
Spring I_O 2024 - Flexible Spring with Event Sourcing.pptx
bySteve Pember
 
PDF
Building Microservices with Event Sourcing and CQRS
byMichael Plöd
 
PDF
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing...
byconfluent
 
Event Sourcing - what could go wrong - Jfokus 2022
byAndrzej Ludwikowski
 
Event sourcing - what could possibly go wrong ? Devoxx PL 2021
byAndrzej Ludwikowski
 
Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemo...
byCodemotion
 
Event Sourcing - what could possibly go wrong?
byAndrzej Ludwikowski
 
Andrzej Ludwikowski - Event Sourcing - co może pójść nie tak?
bySegFaultConf
 
Spring I_O 2024 - Flexible Spring with Event Sourcing.pptx
bySteve Pember
 
Building Microservices with Event Sourcing and CQRS
byMichael Plöd
 
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing...
byconfluent
 

Similar to Event Sourcing - what could go wrong - Devoxx BE

PPTX
Event sourcing
byTouraj Ebrahimi
 
PPTX
Event Sourcing & CQRS: Down the rabbit hole
byJulian May
 
PDF
#hacksummit 2016 - event-driven microservices – Events on the outside, on the...
byChris Richardson
 
PDF
Event sourcing and CQRS: Lessons from the trenches
byDavid Jiménez Martínez
 
PPTX
Lessons learned from applying Event sourcing and CQRS
bySavvas Kleanthous
 
PDF
Events on the outside, on the inside and at the core - Chris Richardson
byJAXLondon_Conference
 
PDF
Events on the outside, on the inside and at the core (jaxlondon)
byChris Richardson
 
PPTX
Event Driven Architectures
byAvinash Ramineni
 
PDF
A year with event sourcing and CQRS
bySteve Pember
 
PPTX
Intro to event sourcing and CQRS
bySavvas Kleanthous
 
PPTX
Event Driven Architectures - Phoenix Java Users Group 2013
byclairvoyantllc
 
PDF
Building and Deploying Microservices with Event Sourcing, CQRS and Docker
byC4Media
 
PDF
Event Sourcing: Introduction & Challenges
byMichael Plöd
 
PPTX
Cqrs and Event Sourcing Intro For Developers
bywojtek_s
 
PDF
Developing functional domain models with event sourcing (sbtb, sbtb2015)
byChris Richardson
 
PDF
CQRS + Event Sourcing
byMike Bild
 
PPTX
CQRS / ES & DDD Demystified
byVic Metcalfe
 
PPTX
Viadeo - The Kasper way
byLoïc Dias Da Silva
 
PDF
Event storage in a distributed system
bySteve Pember
 
PDF
5 things you may want to know before an Event Sourcing Project
byMichele Mauro
 
Event sourcing
byTouraj Ebrahimi
 
Event Sourcing & CQRS: Down the rabbit hole
byJulian May
 
#hacksummit 2016 - event-driven microservices – Events on the outside, on the...
byChris Richardson
 
Event sourcing and CQRS: Lessons from the trenches
byDavid Jiménez Martínez
 
Lessons learned from applying Event sourcing and CQRS
bySavvas Kleanthous
 
Events on the outside, on the inside and at the core - Chris Richardson
byJAXLondon_Conference
 
Events on the outside, on the inside and at the core (jaxlondon)
byChris Richardson
 
Event Driven Architectures
byAvinash Ramineni
 
A year with event sourcing and CQRS
bySteve Pember
 
Intro to event sourcing and CQRS
bySavvas Kleanthous
 
Event Driven Architectures - Phoenix Java Users Group 2013
byclairvoyantllc
 
Building and Deploying Microservices with Event Sourcing, CQRS and Docker
byC4Media
 
Event Sourcing: Introduction & Challenges
byMichael Plöd
 
Cqrs and Event Sourcing Intro For Developers
bywojtek_s
 
Developing functional domain models with event sourcing (sbtb, sbtb2015)
byChris Richardson
 
CQRS + Event Sourcing
byMike Bild
 
CQRS / ES & DDD Demystified
byVic Metcalfe
 
Viadeo - The Kasper way
byLoïc Dias Da Silva
 
Event storage in a distributed system
bySteve Pember
 
5 things you may want to know before an Event Sourcing Project
byMichele Mauro
 

More from Andrzej Ludwikowski

PDF
Event-driven systems without pulling your hair out
byAndrzej Ludwikowski
 
PDF
Performance tests - it's a trap
byAndrzej Ludwikowski
 
PDF
Cassandra lesson learned - extended
byAndrzej Ludwikowski
 
PDF
Performance tests with Gatling (extended)
byAndrzej Ludwikowski
 
PPTX
Stress test your backend with Gatling
byAndrzej Ludwikowski
 
PPTX
Performance tests with Gatling
byAndrzej Ludwikowski
 
PDF
Cassandra - lesson learned
byAndrzej Ludwikowski
 
PPTX
Annotation processing tool
byAndrzej Ludwikowski
 
Event-driven systems without pulling your hair out
byAndrzej Ludwikowski
 
Performance tests - it's a trap
byAndrzej Ludwikowski
 
Cassandra lesson learned - extended
byAndrzej Ludwikowski
 
Performance tests with Gatling (extended)
byAndrzej Ludwikowski
 
Stress test your backend with Gatling
byAndrzej Ludwikowski
 
Performance tests with Gatling
byAndrzej Ludwikowski
 
Cassandra - lesson learned
byAndrzej Ludwikowski
 
Annotation processing tool
byAndrzej Ludwikowski
 

Recently uploaded

PPTX
Aviation Fleet Management Software for Airlines & Operators
bySISGAIN
 
PDF
Cloud Engineering Services | Infysion Technologies
bysolutioninginfysion
 
PDF
From Sound Workflow Nets to LTLf Declarative Specifications
byLucaBarbaro3
 
PPTX
How to Extract Google Maps Business Leads Efficiently
bywakasoftteam
 
PDF
DSD-INT 2025 Multi-scale modeling to simulate Land Subsidence in the Central ...
byDeltares
 
PPTX
Best Application Development Consulting Company in Houston
byDesss Applying Technologies
 
PDF
2026 Social Media Trends_ Must-Know Strategies for Dubai Business Owners.pdf
byMAQ MAQ
 
PPTX
ManageIQ - Sprint 278 Review - Slide Deck
byManageIQ
 
PPTX
Testing Strategies for Agile Teams in 2026
byMatt Calder
 
PDF
Inside An AI Powered ERP System for Process Manufacturers
byBatchMaster Software Pvt. Ltd.
 
PDF
REST in Peace (Laravel Senegal Meetup 2025)
byWendell Adriel
 
PPTX
EMR Software for Pulmonology Clinics Why EasyClinic Supports Accuracy, Contin...
byEasy Clinic
 
PDF
Prasenjit Bhaumik Plano - Specializing In Web Applications
byPrasenjit Bhaumik Plano
 
PPTX
Google Trips Data Scraping for Tourism & Demand Analysis.pptx
byWeb Data Crawler
 
PDF
IT Asset Lifecycle Management Solution | Infraon
byEverestIMS Technologies Pvt. Ltd
 
PPTX
Dreamforce ‘25 Tour: Boost Productivity with AI-Assisted API Development
byPriya Shaw
 
PPTX
UAE-Based Insurer Transforms Operations with InsureEdge
byInsurance Tech Services
 
PPTX
A practical and beginner-friendly guide to understanding data structures in J...
bynaingseihahs
 
PPTX
Best Way to Convert PST File to MBOX Format
bykyla142000
 
PDF
DSD-INT 2025 Modelling cold water infiltration with the Groundwater Energy (G...
byDeltares
 
Aviation Fleet Management Software for Airlines & Operators
bySISGAIN
 
Cloud Engineering Services | Infysion Technologies
bysolutioninginfysion
 
From Sound Workflow Nets to LTLf Declarative Specifications
byLucaBarbaro3
 
How to Extract Google Maps Business Leads Efficiently
bywakasoftteam
 
DSD-INT 2025 Multi-scale modeling to simulate Land Subsidence in the Central ...
byDeltares
 
Best Application Development Consulting Company in Houston
byDesss Applying Technologies
 
2026 Social Media Trends_ Must-Know Strategies for Dubai Business Owners.pdf
byMAQ MAQ
 
ManageIQ - Sprint 278 Review - Slide Deck
byManageIQ
 
Testing Strategies for Agile Teams in 2026
byMatt Calder
 
Inside An AI Powered ERP System for Process Manufacturers
byBatchMaster Software Pvt. Ltd.
 
REST in Peace (Laravel Senegal Meetup 2025)
byWendell Adriel
 
EMR Software for Pulmonology Clinics Why EasyClinic Supports Accuracy, Contin...
byEasy Clinic
 
Prasenjit Bhaumik Plano - Specializing In Web Applications
byPrasenjit Bhaumik Plano
 
Google Trips Data Scraping for Tourism & Demand Analysis.pptx
byWeb Data Crawler
 
IT Asset Lifecycle Management Solution | Infraon
byEverestIMS Technologies Pvt. Ltd
 
Dreamforce ‘25 Tour: Boost Productivity with AI-Assisted API Development
byPriya Shaw
 
UAE-Based Insurer Transforms Operations with InsureEdge
byInsurance Tech Services
 
A practical and beginner-friendly guide to understanding data structures in J...
bynaingseihahs
 
Best Way to Convert PST File to MBOX Format
bykyla142000
 
DSD-INT 2025 Modelling cold water infiltration with the Groundwater Energy (G...
byDeltares
 

Event Sourcing - what could go wrong - Devoxx BE

  • 1.
    Event Sourcing what couldpossibly go wrong? Andrzej Ludwikowski
  • 2.
    About me ➔ ➔ ludwikowski.info ➔github.com/aludwiko ➔ @aludwikowski
  • 3.
    What is EventSourcing? DB ShoppingCart { items=[itemA, itemB] }
  • 4.
    What is EventSourcing? DB DB ShoppingCart { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  • 5.
    What is EventSourcing? DB DB ShoppingCart { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
  • 6.
    History ● 9000 BC,Mesopotamian Clay Tablets, e.g. for market transactions
  • 7.
    History ● 2005, EventSourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
  • 8.
    Why Event Sourcing? ●complete log of every state change ● debugging ● performance ● scalability ● microservices integration pattern
  • 9.
    ES and CQRS Client QueryService Data access Queries Read model Read model Read models Command Service Domain Event store Commands
  • 10.
    ES and CQRSlevel 1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Transaction Command Service Domain Event store Commands
  • 11.
    ES and CQRSlevel 1 ● Entry-level, synchronous & transactional event sourcing ● Implementing event sourcing using a relational database
  • 12.
    ES and CQRSlevel 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
  • 13.
    ES and CQRSlevel 2 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction Command Service Domain Event store Commands
  • 14.
    ES and CQRSlevel 2 +/- performance +/- scalability - eventual consistency - increased event store load ? - lags
  • 15.
    ES and CQRSlevel 3 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector Transaction event bus Command Service Domain Event store Commands
  • 16.
    ES and CQRSlevel 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Event store Commands
  • 17.
    ES and CQRSlevel 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Event store Commands
  • 18.
    ES and CQRSlevel 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction At-least-once delivery Command Service Domain Event store Commands
  • 19.
    ES and CQRSlevel 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 20.
    ES and CQRSlevel 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 21.
    ES and CQRSlevel 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 22.
    ES and CQRSlevel 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 23.
    ES and CQRSlevel 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 24.
    ES and CQRSlevel 3.1.1 Command Service Domain Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction ?
  • 25.
    ES and CQRSlevel 3.2 Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
  • 26.
    ES and CQRSlevel 3.x + performance + scalability - eventual consistency - complex implementation - locking vs Single Writer
  • 27.
    ES and CQRSalternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of event store (ES based on Kafka)?
  • 28.
    Not covered butworth to check ● Command Sourcing ● Event Collaboration
  • 29.
    ES implementation? ● custom ●library ● framework
  • 30.
    ES from domainperspective ● commands, events, state ● 2 methods ○ process(command: Command): List[Event] ○ apply(event: Event): State
  • 31.
    ES from applicationperspective ● snapshotting ● fail-over/recover ● persistence ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  • 32.
    import javax.persistence.*; import java.util.List; @Entity publicclass Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
  • 33.
    import javax.persistence.*; import java.util.List; @Entity publicclass Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
  • 34.
    import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository= "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
  • 35.
    import akka.Done import com.lightbend.lagom.scaladsl.* importplay.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
  • 36.
    ES packaging ● Keepyour domain clean with ES
  • 37.
    import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* importinfo.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} case class User (userId: UserId, name: String, email: Email, funds: Funds) { def process(command: UserCommand): Either[List[UserEvent]] = command match { case c: UpdateEmail => updateEmail(c) case c: DepositFunds => deposit(c) case c: WithdrawFunds => withdraw(c) ... } def apply(event: UserEvent): User = ??? //pattern matching }
  • 38.
    ES packaging ● snapshotting ●fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
  • 39.
    ES packaging ● domainlogic ● domain validation ● 0 ES framework imports
  • 40.
    library vs. framework ●Akka Persistence vs. Lagom ● Akka Persistence Typed
  • 41.
  • 42.
    Kalix Fundamentals Self-Managed On-Prem Application BusinessLogic Frameworks Database Transport Security Infrastructure Kubernetes Operating System Virtualization Servers, Storage, Networking In the Cloud Application Business Logic Frameworks Database Transport Security Kubernetes Operating System Virtualization Servers, Storage, Networking Infrastructure Major challenges remain: • Architectural Complexity • Skill Availability • Time to Market • Cost Issues Self-Managed Managed by Cloud / Service Provider 42
  • 43.
    Kalix Fundamentals Self-Managed On-Prem Application BusinessLogic Frameworks Database Transport Security Infrastructure Kubernetes Operating System Virtualization Servers, Storage, Networking In the Cloud Application Business Logic Frameworks Database Transport Security Kubernetes Operating System Virtualization Servers, Storage, Networking Infrastructure Self-Managed Managed by Cloud / Service Provider 43 Kalix Application Business Logic Frameworks Database Transport Security Kubernetes Operating System Virtualization Servers, Storage, Networking Infrastructure
  • 44.
    Event store ● file ●RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  • 45.
    Event store forAkka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
  • 46.
    Cassandra perfect forES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
  • 47.
    ScyllaDB ? ● Cassandrawithout JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
  • 48.
    Event serialization ● plaintext ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
  • 49.
    Plain text Binary human-readabledeserialization required
  • 50.
    Plain text Binary human-readabledeserialization required precision issues (JSON IEEE 754, DoS) -
  • 51.
    Plain text Binary human-readabledeserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
  • 52.
    Plain text Binary human-readabledeserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
  • 53.
    Plain text Binary human-readabledeserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
  • 54.
    Binary ● java serialization ●Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 55.
    Binary ● java serialization ●Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 56.
    Binary ● java serialization ●Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 57.
    Binary ● java serialization ●Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
  • 58.
    Multi-language support ● Avro ○C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
  • 59.
  • 60.
  • 61.
    ● backward -V2 can read V1 Full compatibility Application Event store V1 V2
  • 62.
    ● forward -V2 can read V3 Full compatibility Application Event store V1, V2 V2 Application Application V2 V3
  • 63.
    ● forward -V2 can read V3 Full compatibility Event store Read model Read model Read models Projector V2 V3
  • 64.
    Schema evolution -full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
  • 65.
    Protobuf schema management //user-events.proto messageUserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  • 66.
    Protobuf schema management packageuser.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
  • 67.
    Protobuf schema management ●def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
  • 68.
    Protobuf schema management +clean domain - a lot of boilerplate code
  • 69.
    Avro schema management packageuser.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  • 70.
    Avro deserialization Bytes DeserializationObject Reader Schema Writer Schema
  • 71.
    Avro writer schemasource ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
  • 72.
    Avro schema management packageuser.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
  • 73.
    Protocol Buffers vs.Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
  • 74.
    Avro schema management +less boilerplate code +/- clean domain - reader & writer schema distribution
  • 75.
    Avro + less boilerplatecode +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
  • 76.
    Avro vs. ProtocolBuffers ● The best serialization strategy for Event Sourcing
  • 77.
    Event payload ● deltaevent ● rich event (event enrichment) ● + metadata ○ seq_num ○ created_at ○ event_id ○ command_id ○ correlation_id
  • 80.
    State replay time ●snapshotting ● write-through cache
  • 81.
  • 83.
    Immutable vs. mutablestate? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
  • 84.
  • 85.
    Updating all aggregates User(id) Command(user_id)Event(user_id) Event(user_id) Event(user_id)
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
    Event + seq_no Event+ seq_no Handling duplicates Events Read model Read model Read models Projector event bus Event + seq_no read model update + seq_no Event store
  • 91.
    Broken read model Events admodel ead model Read models Projector event bus Event store
  • 92.
    Broken read model Events admodel ead model Read models Projector event bus read model update + offset (manual offset management) Event store
  • 93.
    Multi aggregate transactionalupdate ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
  • 94.
  • 95.
    Pessimistic compensation action Useraccount Cinema show charged
  • 96.
    Pessimistic compensation action Useraccount Cinema show charged booked
  • 97.
    Pessimistic compensation action Useraccount Cinema show charged booked sold out
  • 98.
    Pessimistic compensation action Useraccount Cinema show charged booked booked sold out
  • 99.
    Pessimistic compensation action Useraccount Cinema show charged booked booked sold out
  • 100.
    Pessimistic compensation action Useraccount Cinema show charged booked booked sold out refund
  • 101.
    Optimistic compensation action Useraccount Cinema show charged booked sold out
  • 102.
    Optimistic compensation action Useraccount Cinema show booked booked sold out overbooked
  • 103.
    Optimistic compensation action Useraccount Cinema show charged booked booked sold out overbooked ?
  • 104.
    Saga Command Service Domain Event store Client Query Service Dataaccess Commands Queries Read model Read model Read models Updater event bus Transaction
  • 105.
    Saga - choreography CommandService Domain Event store Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction
  • 106.
    Saga - orchestration CommandService Domain Events Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Transaction Command Service Domain Event store Commands
  • 107.
    Saga ● should bepersistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
  • 108.
    Saga ● Sagas withES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern ● Microservice Patterns
  • 109.
    ES with RODO/GDPR ●“right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
  • 110.
    ES and CQRSlevel 3.2 Event store Client Query Service Data access Commands Queries Read model Read model Read models Projector event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
  • 111.
    Cluster = splitbrain 1 5 4 3 Load balancer 2
  • 112.
    Cluster = splitbrain 1 5 4 3 Load balancer 2 User(1) Command(1)
  • 113.
    Cluster = splitbrain 1 5 4 3 Load balancer 2
  • 114.
    Cluster = splitbrain 1 5 4 3 Load balancer 2 User(1)
  • 115.
    Cluster = splitbrain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
  • 116.
    Cluster = splitbrain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
  • 117.
    Cluster best practises ●remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
  • 118.
  • 119.
    Summary ● carefully chooseES lib/framework ● there is no perfect database for event sourcing ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
  • 121.
  • 122.
    About me ➔ ➔ ludwikowski.info ➔github.com/aludwiko ➔ @aludwikowski