Event Sourcing
- what could go wrong?
Andrzej Ludwikowski
Event Sourcing - what could go wrong?
Andrzej Ludwikowski
Rome | March 22 - 23, 2019
About me
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski
SoftwareMill
SoftwareMill
SoftwareMill
SoftwareMill
● Hibernate Envers
● Alpakka Kafka
● Sttp
● Scala Clippy
● Quicklens
● MacWire
What is Event Sourcing?
DB
Order {
items=[itemA, itemB]
}
What is Event Sourcing?
DB
DB
Order {
items=[itemA, itemB]
}
ItemAdded(itemA)
ItemAdded(itemC)
ItemRemoved(itemC)
ItemAdded(itemB)
History
● 9000 BC, Mesopotamian Clay Tablets,
e.g. for market transactions
History
● 2005, Event Sourcing
“Enterprise applications that use Event Sourcing
are rarer, but I have seen a few applications (or
parts of applications) that use it.”
Why Event Sourcing?
● complete log of every state change
● debugging
● performance
● scalability
ES and CQRS
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
ES and CQRS level 1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Transaction
ES and CQRS level 1
● Entry-level, synchronous & transactional event sourcing
● slick-eventsourcing
ES and CQRS level 1
+ easy to implement
+ easy to reason about
+ 0 eventual consistency
- performance
- scalability
ES and CQRS level 2
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
Transaction
ES and CQRS level 2
+/- performance
+/- scalability
- eventual consistency
- increased events DB load
- lags
ES and CQRS level 3
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
Transaction
event
bus
ES and CQRS level 3.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
At-least-once delivery
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
?
ES and CQRS level 3.2
Events
Client
Query Service
Data access
Commands
Queries
Read
modelRead
modelRead
models
Updater
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
`
Sharded
Cluster
ES and CQRS level 3.x
+ performance
+ scalability
- eventual consistency
- complex implementation
ES and CQRS alternatives
● Change Capture Data (CDC) logging instead of event bus?
● event bus instead of DB?
Not covered but worth to check
● Command Sourcing
● Event Collaboration
ES implementation?
● custom
● library
● framework
ES from domain perspective
● commands, events, state
● 2 methods on state
○ process(command: Command): List[Event]
○ apply(event: Event): State
ES from application perspective
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.
import javax.persistence.*;
import java.util.List;
@Entity
public class Issue {
@EmbeddedId
private IssueId id;
private String name;
private IssueStatus status;
@OneToMany(cascade = CascadeType.MERGE)
private List<IssueComment> comments;
...
public void changeStatusTo(IssueStatus newStatus) {
if (this.status == IssueStatus.DONE
&& newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW
&& newStatus == IssueStatus.DONE) {
throw new RuntimeException(String.format("Cannot change issue status from %s to %s",
this.status, newStatus));
}
this.status = newStatus;
}
...
}
import org.axonframework.commandhandling.*
import org.axonframework.eventsourcing.*
@Aggregate(repository = "userAggregateRepository")
public class User {
@AggregateIdentifier
private UserId userId;
private String passwordHash;
@CommandHandler
public boolean handle(AuthenticateUserCommand cmd) {
boolean success = this.passwordHash.equals(hashOf(cmd.getPassword()));
if (success) {
apply(new UserAuthenticatedEvent(userId));
}
return success;
}
@EventSourcingHandler
public void on(UserCreatedEvent event) {
this.userId = event.getUserId();
this.passwordHash = event.getPassword();
}
private String hashOf(char[] password) {
return DigestUtils.sha1(String.valueOf(password));
}
}
import akka.Done
import com.lightbend.lagom.scaladsl.*
import play.api.libs.json.{Format, Json}
import com.example.auction.utils.JsonFormats._
class UserEntity extends PersistentEntity {
override def initialState = None
override def behavior: Behavior = {
case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onReadOnlyCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists")
}
case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done))
}.onEvent {
case (UserCreated(name), state) => Some(User(name))
}
}
}
ES packaging
● github.com/aludwiko/event-sourcing-akka-persistence
import java.time.Instant
import info.ludwikowski.es.user.domain.UserCommand.*
import info.ludwikowski.es.user.domain.UserEvent.*
import scala.util.{Failure, Success, Try}
final case class User (userId: UserId, name: String, email: Email) {
def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching
def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching
}
object User {
def from(u: UserCreated): User = User(u.userId, u.name, u.email)
}
ES packaging
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.
ES packaging
● domain logic
● domain validation
● 0 ES framework imports
library vs. framework
● Akka Persistence vs. Lagom
Event storage
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra
Event storage for Akka Persistence
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra
akka-persistence-jdbc trap
val theTag = s"%$tag%"
sql"""
SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber",
"#$message", "#$tags"
FROM (
SELECT * FROM #$theTableName
WHERE "#$tags" LIKE $theTag
AND "#$ordering" > $theOffset
AND "#$ordering" <= $maxOffset
ORDER BY "#$ordering"
)
WHERE rownum <= $max"""
akka-persistence-jdbc trap
SELECT * FROM events_journal
WHERE tags LIKE ‘%some_tag%’;
Cassandra perfect for ES?
● partitioning by design
● replication by design
● leaderless (no single point of failure)
● optimised for writes (2 nodes = 100 000 tx/s)
● near-linear horizontal scaling
ScyllaDB ?
● Cassandra without JVM
○ same protocol, SSTable compatibility
● C++ and Seastar lib
● up to 1,000,000 IOPS/node
● not fully supported by Akka Persistence
Event serialization
● plain text
○ JSON
○ XML
○ YAML
● binary
○ java serialization
○ Avro
○ Protocol Buffers (Protobuf)
○ Thrift
○ Kryo
Plain text Binary
human-readable deserialization required
Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
storage consumption compress
Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
storage consumption compress
slow fast
Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
storage consumption compress
slow fast
poor schema evolution support full schema evolution support
Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo
Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo
Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo
Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo
Multi-language support
● Avro
○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala
● Protocol Buffers (Protobuf)
○ even more than Avro
Speed
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
Size
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki
Full compatibility
Application
Events
● backward - V2 can read V1
V1
V2
● forward - V2 can read V3
Full compatibility
Application
Events
V1, V2
V2
Application
Application
V2
V3
Schema evolution - full compatibility
Protocol Buffers Avro
Add field + (optional) + (default value)
Remove field + + (default value)
Rename field + + (aliases)
https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
Protobuf schema management
//user-events.proto
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)
Protobuf schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)
Protobuf schema management
● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated
● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
Protobuf schema management
+ clean domain
- a lot of boilerplate code
Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}
Avro deserialization
Bytes Deserialization Object
Reader SchemaWriter Schema
Avro writer schema source
● add schema to the payload
● custom solution
○ schema in /resources
○ schema in external storage (must be fault-tolerant)
○ Darwin project
● Schema Registry
Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}
Protocol Buffers vs. Avro
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}
Avro schema management
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution
Avro
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution
Protobuf
+ clean domain
- a lot of boilerplate code
Memory consumption
Immutable vs. mutable state?
● add/remove ImmutableList 17.496 ops/s
● add/remove TreeMap 2201.731 ops/s
Fixing state
● healing command
Updating all aggregates
User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)
Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
At-least-once delivery
https://www.seriouspuzzles.com/unicorns-in-fairy-land-500pc-jigsaw-puzzle-by-eurographics/
Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
At-least-once delivery
Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
idempotent updates
Event + seq_noEvent + seq_no
Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
Event + seq_no
read model update +
seq_no
Broken read model
Events
ad model
ead model
Read
models
Updater
event
bus
Broken read model
Events
ad model
ead model
Read
models
Updater
event
bus
read model update + offset
(manual offset management)
Multi aggregate transactional update
● rethink aggregates boundaries
● compensating action
○ optimistic
○ pessimistic
Pessimistic compensation action
User account
Cinema hall
Pessimistic compensation action
User account
Cinema hall
charged
Pessimistic compensation action
User account
Cinema hall
charged
booked
Pessimistic compensation action
User account
Cinema hall
charged
booked sold out
Pessimistic compensation action
User account
Cinema hall
charged
booked
booked sold out
Pessimistic compensation action
User account
Cinema hall
charged
booked
booked sold out
Pessimistic compensation action
User account
Cinema hall
charged
booked
booked sold out
refund
Optimistic compensation action
User account
Cinema hall
charged
booked sold out
Optimistic compensation action
User account
Cinema hall booked
booked sold out
overbooked
Optimistic compensation action
User account
Cinema hall
charged
booked
booked sold out
overbooked
?
Saga, post-processing
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
Saga, post-processing
Command Service
Domain
Events
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
Saga, post-processing
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
Saga, post-processing
● should be persistable
● events order should be irrelevant
● time window limitation
● compensating action must be commutative
Saga, post-processing
● Sagas with ES
● DDD, Saga & Event-sourcing
● Applying Saga Pattern
ES with RODO/GDPR
● “right to forget” with:
○ data shredding (and/or deleting)
■ events, state, views, read models
○ retention policy
■ message brokers, backups, logs
○ data before RODO migration
ES and CQRS level 3.2
Events
Client
Query Service
Data access
Commands
Queries
Read
modelRead
modelRead
models
Updater
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
Sharding
Clustering
Cluster = split brain
1
5 4
3
Load balancer
2
Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)
Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)
Command(1)
Cluster best practises
● remember about the split brain
● very good monitoring & alerting
● a lot of failover tests
● cluster also on dev/staging
● keep it as small as possible (code base, number of nodes, etc.)
Summary
● carefully choose ES lib/framework
● understand event/command/state schema evolution
● eventual consistency is your friend
● scaling is complex
● database inside-out
● log-based processing mindset
Rate me please :)
Thank you and Q&A
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski
WE
WANT
YOU

Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019