Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019

Event Sourcing
- what could go wrong?
Andrzej Ludwikowski

Event Sourcing - what could go wrong?
Andrzej Ludwikowski
Rome | March 22 - 23, 2019

About me
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski

SoftwareMill
● Hibernate Envers
● Alpakka Kafka
● Sttp
● Scala Clippy
● Quicklens
● MacWire

What is Event Sourcing?
DB
Order {
items=[itemA, itemB]
}

What is Event Sourcing?
DB
DB
Order {
items=[itemA, itemB]
}
ItemAdded(itemA)
ItemAdded(itemC)
ItemRemoved(itemC)
ItemAdded(itemB)

History
● 9000 BC, Mesopotamian Clay Tablets,
e.g. for market transactions

History
● 2005, Event Sourcing
“Enterprise applications that use Event Sourcing
are rarer, but I have seen a few applications (or
parts of applications) that use it.”

Why Event Sourcing?
● complete log of every state change
● debugging
● performance
● scalability

ES and CQRS
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models

ES and CQRS level 1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Transaction

ES and CQRS level 1
● Entry-level, synchronous & transactional event sourcing
● slick-eventsourcing

ES and CQRS level 1
+ easy to implement
+ easy to reason about
+ 0 eventual consistency
- performance
- scalability

ES and CQRS level 2
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
Transaction

ES and CQRS level 2
+/- performance
+/- scalability
- eventual consistency
- increased events DB load
- lags

ES and CQRS level 3
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
Transaction
event
bus

ES and CQRS level 3.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
At-least-once delivery

Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction
?

Events
Client
Query Service
Data access
Commands
Queries
Read
modelRead
modelRead
models
Updater
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
`
Sharded
Cluster

ES and CQRS level 3.x
+ performance
+ scalability
- eventual consistency
- complex implementation

ES and CQRS alternatives
● Change Capture Data (CDC) logging instead of event bus?
● event bus instead of DB?

Not covered but worth to check
● Command Sourcing
● Event Collaboration

ES implementation?
● custom
● library
● framework

ES from domain perspective
● commands, events, state
● 2 methods on state
○ process(command: Command): List[Event]
○ apply(event: Event): State

ES from application perspective
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.

import javax.persistence.*;
import java.util.List;
@Entity
public class Issue {
@EmbeddedId
private IssueId id;
private String name;
private IssueStatus status;
@OneToMany(cascade = CascadeType.MERGE)
private List<IssueComment> comments;
...
public void changeStatusTo(IssueStatus newStatus) {
if (this.status == IssueStatus.DONE
&& newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW
&& newStatus == IssueStatus.DONE) {
throw new RuntimeException(String.format("Cannot change issue status from %s to %s",
this.status, newStatus));
}
this.status = newStatus;
}
...
}

import org.axonframework.commandhandling.*
import org.axonframework.eventsourcing.*
@Aggregate(repository = "userAggregateRepository")
public class User {
@AggregateIdentifier
private UserId userId;
private String passwordHash;
@CommandHandler
public boolean handle(AuthenticateUserCommand cmd) {
boolean success = this.passwordHash.equals(hashOf(cmd.getPassword()));
if (success) {
apply(new UserAuthenticatedEvent(userId));
}
return success;
}
@EventSourcingHandler
public void on(UserCreatedEvent event) {
this.userId = event.getUserId();
this.passwordHash = event.getPassword();
}
private String hashOf(char[] password) {
return DigestUtils.sha1(String.valueOf(password));
}
}

import akka.Done
import com.lightbend.lagom.scaladsl.*
import play.api.libs.json.{Format, Json}
import com.example.auction.utils.JsonFormats._
class UserEntity extends PersistentEntity {
override def initialState = None
override def behavior: Behavior = {
case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onReadOnlyCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists")
}
case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done))
}.onEvent {
case (UserCreated(name), state) => Some(User(name))
}
}
}

ES packaging
● github.com/aludwiko/event-sourcing-akka-persistence

import java.time.Instant
import info.ludwikowski.es.user.domain.UserCommand.*
import info.ludwikowski.es.user.domain.UserEvent.*
import scala.util.{Failure, Success, Try}
final case class User (userId: UserId, name: String, email: Email) {
def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching
def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching
}
object User {
def from(u: UserCreated): User = User(u.userId, u.name, u.email)
}

ES packaging
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.

ES packaging
● domain logic
● domain validation
● 0 ES framework imports

library vs. framework
● Akka Persistence vs. Lagom

Event storage
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra

Event storage for Akka Persistence
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra

akka-persistence-jdbc trap
val theTag = s"%$tag%"
sql"""
SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber",
"#$message", "#$tags"
FROM (
SELECT * FROM #$theTableName
WHERE "#$tags" LIKE $theTag
AND "#$ordering" > $theOffset
AND "#$ordering" <= $maxOffset
ORDER BY "#$ordering"
)
WHERE rownum <= $max"""

akka-persistence-jdbc trap
SELECT * FROM events_journal
WHERE tags LIKE ‘%some_tag%’;

Cassandra perfect for ES?
● partitioning by design
● replication by design
● leaderless (no single point of failure)
● optimised for writes (2 nodes = 100 000 tx/s)
● near-linear horizontal scaling

ScyllaDB ?
● Cassandra without JVM
○ same protocol, SSTable compatibility
● C++ and Seastar lib
● up to 1,000,000 IOPS/node
● not fully supported by Akka Persistence

Event serialization
● plain text
○ JSON
○ XML
○ YAML
● binary
○ java serialization
○ Avro
○ Protocol Buffers (Protobuf)
○ Thrift
○ Kryo

Plain text Binary
human-readable deserialization required

Plain text Binary
precision issues (JSON IEEE 754, DoS) -

Plain text Binary
storage consumption compress

Plain text Binary
slow fast

Plain text Binary
slow fast
poor schema evolution support full schema evolution support

Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

Multi-language support
● Avro
○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala
● Protocol Buffers (Protobuf)
○ even more than Avro

Speed
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki

Size
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki

Full compatibility
Application
Events
● backward - V2 can read V1
V1
V2

● forward - V2 can read V3
Full compatibility
Application
Events
V1, V2
V2
Application
Application
V2
V3

Schema evolution - full compatibility
Protocol Buffers Avro
Add field + (optional) + (default value)
Remove field + + (default value)
Rename field + + (aliases)
https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

Protobuf schema management
//user-events.proto
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)

package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)

● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated
● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent

+ clean domain
- a lot of boilerplate code

Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"name" : "email",
"type" : "string"
} ]
}

Avro deserialization
Bytes Deserialization Object
Reader SchemaWriter Schema

Avro writer schema source
● add schema to the payload
● custom solution
○ schema in /resources
○ schema in external storage (must be fault-tolerant)
○ Darwin project
● Schema Registry

Protocol Buffers vs. Avro
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"name" : "operationId",
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"name" : "email",
"type" : "string"
} ]
}
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}

Avro schema management
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution

Avro
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution
Protobuf
+ clean domain
- a lot of boilerplate code

Immutable vs. mutable state?
● add/remove ImmutableList 17.496 ops/s
● add/remove TreeMap 2201.731 ops/s

Fixing state
● healing command

Updating all aggregates
User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)

Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
At-least-once delivery

https://www.seriouspuzzles.com/unicorns-in-fairy-land-500pc-jigsaw-puzzle-by-eurographics/

Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
idempotent updates

Event + seq_noEvent + seq_no
Handling duplicates
Events
Read
modelRead
modelRead
models
Updater
event
bus
Event + seq_no
read model update +
seq_no

Broken read model
Events
ad model
ead model
Read
models
Updater
event
bus

Broken read model
Events
ad model
ead model
Read
models
Updater
event
bus
read model update + offset
(manual offset management)

Multi aggregate transactional update
● rethink aggregates boundaries
● compensating action
○ optimistic
○ pessimistic

Pessimistic compensation action
User account
Cinema hall

User account
Cinema hall
charged

User account
Cinema hall
charged
booked

User account
Cinema hall
charged
booked sold out

User account
Cinema hall
charged
booked
booked sold out

User account
Cinema hall
charged
booked
booked sold out
refund

Optimistic compensation action
User account
Cinema hall
charged
booked sold out

User account
Cinema hall booked
booked sold out
overbooked

User account
Cinema hall
charged
booked
booked sold out
overbooked
?

Saga, post-processing
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

Command Service
Domain
Events
Query Service
Data access
Commands Queries
Read
modelRead
modelRead
models
Updater
event
bus
Transaction

● should be persistable
● events order should be irrelevant
● time window limitation
● compensating action must be commutative

● Sagas with ES
● DDD, Saga & Event-sourcing
● Applying Saga Pattern

ES with RODO/GDPR
● “right to forget” with:
○ data shredding (and/or deleting)
■ events, state, views, read models
○ retention policy
■ message brokers, backups, logs
○ data before RODO migration

Events
Client
Query Service
Data access
Commands
Queries
Read
modelRead
modelRead
models
Updater
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
Sharding
Clustering

Cluster = split brain
1
5 4
3
Load balancer
2

1
5 4
3
Load balancer
2
User(1)
Command(1)

1
5 4
3
Load balancer
2
User(1)

1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)

1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)
Command(1)

Cluster best practises
● remember about the split brain
● very good monitoring & alerting
● a lot of failover tests
● cluster also on dev/staging
● keep it as small as possible (code base, number of nodes, etc.)

Summary
● carefully choose ES lib/framework
● understand event/command/state schema evolution
● eventual consistency is your friend
● scaling is complex
● database inside-out
● log-based processing mindset

Thank you and Q&A
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski

Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019

More Related Content

Similar to Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019

More from Codemotion

Recently uploaded

Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019