CQRS & EVENT SOURCING
IN THE WILD
MICHIEL ROOK
➤ Java, PHP & Scala
developer
➤ Consultant, trainer,
speaker
➤ Dutch Web Alliance
➤ make.io
➤ Maintainer of Phing
➤ @michieltcs
YOU
RAISE YOUR HAND
IF YOU HAVE
heard about CQRS / Event Sourcing
RAISE YOUR HAND
IF YOU HAVE
heard about CQRS / Event Sourcing
followed a tutorial, built a hobby project
RAISE YOUR HAND
IF YOU HAVE
heard about CQRS / Event Sourcing
followed a tutorial, built a hobby project
used it in production
RAISE YOUR HAND
IF YOU HAVE
TOPICS
➤ Quick recap
➤ Replays and rebuilds
➤ Event versioning
➤ Concurrency
➤ Scale
QUICK RECAP
' Event Sourcing ensures that all
changes to application state are stored
as a sequence of events.
-Martin Fowler
ACTIVE RECORD VS. EVENT SOURCING
Account
number
Balance
12345678 €€ 50,00
... ...
Money Withdrawn
Account number 12345678
Amount €€ 50,00
Money Deposited
Account number 12345678
Amount €€ 100,00
Account Created
Account number 12345678
COMMANDS TO EVENTS
Deposit Money
Account number 12345678
Amount €€ 100,00
Money Deposited
Account number 12345678
Amount €€ 100,00
class DepositMoney {

public $accountNumber;

public $amount;

}
class MoneyDeposited {

public $accountNumber;

public $amount;

public $timestamp;

}
function depositMoney(DepositMoney $command) {

$this->apply(new MoneyDeposited($command->accountNumber,

$command->amount, date()));

}
AGGREGATES
class BankAccount {

public $accountNumber;

public $balance;



// command handlers...



public function applyAccountCreated(AccountCreated $event) {

$this->accountNumber = $event->accountNumber;

$this->balance = 0;

}



public function applyMoneyDeposited(MoneyDeposited $event) {

$this->balance += $event->amount;

}



public function applyMoneyWithdrawn(MoneyWithdrawn $event) {

$this->balance -= $event->amount;

}

}
AGGREGATE STATE
Account
number
Balance
12345678 €€ 0,00
Money Withdrawn
Account number 12345678
Amount €€ 50,00
Money Deposited
Account number 12345678
Amount €€ 100,00
Account Created
Account number 56789012
Account
number
Balance
12345678 €€ 100,00
Account
number
Balance
12345678 €€ 50,00
CQRS
➤ Command Query Responsibility Segregation
➤ Separate writes (events) and reads (UI)
➤ Optimize for reads!
Domain
Event Bus
Event Handlers
Command
Repository
Data Layer
Database Database
Event
Store
commands
events
events
queries DTOs
Aggregates
PROS AND CONS
➤ Domain fit
➤ Testing
➤ Audit trail
➤ Scalability
➤ Complexity
➤ Library support / maturity
DISCLAIMER
REPLAYS AND REBUILDS
PROJECTIONS AND READ MODELS
User Registered
User Registered
User Unregistered Number of active users?
PROJECTIONS AND READ MODELS
User Registered
User Deactivated
User Reactivated
Number of active users?
User Registered
User Unregistered
PROJECTIONS AND READ MODELS
User Registered Event Handler
Number of
active users
+1
User Unregistered Event Handler
Number of
active users
-1
PROJECTIONS AND READ MODELS
Events Event Handler(s) Storage
ELASTICSEARCH
class CompanyRegistered {

public $companyId;

public $name;

public $street;

public $city;

}



function handleCompanyRegistered(CompanyRegistered $event) {

$this->elasticsearch->index([

'index' => 'companies',

'type' => 'company',

'id' => $event->companyId,

'body' => [

'name' => $event->name,

'address' => [

'street' => $event->street,

'city' => $event->city

]

]

]);

}
READ MODEL UPDATES
➤ New type
➤ New structure
➤ Based on existing events
➤ Generate from scratch?
REBUILDING
Stop
application
Remove
old read
model
Loop over
events
Apply to
read model
Start
application
ZERO DOWNTIME
Loop over
existing
events
Apply to
new read
model
Apply
queued
events
Start using
new read
model
New events Queue
ELASTICSEARCH: MAPPING CHANGE
Create
index with
new
mapping
Reindex
existing
documents
Apply
queued
events
Start using
new index
New events Queue
CHALLENGE: LONG RUNNING REBUILDS
➤ Alternatives
➤ Distributed
➤ In memory
➤ Partial
➤ Background
CHALLENGE: SIDE EFFECTS
User Registered
User Id 123abc
Email Address test@example.net
Event Handler
Exclude during replays!
CHALLENGE: TRANSACTIONS
Event Handler
Event Handler
Event
Event Handler
Event Handler ?
CHALLENGE: EVENTUAL CONSISTENCY
➤ Asynchronous event handlers
➤ Reads eventually return the same value
➤ Compare with ACID
EVENT VERSIONING
DILEMMA
➤ New business requirements
➤ Refactoring
➤ New view on events
NEW EVENTS / VERSIONS
➤ No longer relevant
➤ Renamed
➤ Additional or renamed field(s)
➤ Too coarse, too fine
SUPPORT YOUR LEGACY?
➤ Events are immutable
➤ Correct (incorrect) old events with new events
UPCASTING
Event Store
UserRegistered_V1
Upcaster
UserRegistered_V2
Event Handler
UPCASTING
class UserRegistered_V1 {

public $userId;

public $name;

public $timestamp;

}



class UserRegistered_V2 {

public $userId;

public $name;

public $date;

}



function upcast($event): array {

if (!$event instanceof UserRegistered_V1) {

return [];

}

return [

new UserRegistered_V2($event->userId, $event->name,

$event->timestamp->format("Y-m-d"))

];

}
REWRITING HISTORY
Load (subset of) events
Deserialize
Modify
Serialize
Save/replace
THINGS TO BE AWARE OF
Upcasting
➤ Performance
➤ Complexity
➤ Existing projections not
automatically updated
Rewriting events
➤ Running code that
depends on old structure
➤ Breaking serialization
➤ Changing wrong events
CONCURRENCY
CONCURRENT COMMANDS
Withdraw Money
Account number 12345678
Amount €€ 50,00
Deposit Money
Account number 12345678
Amount €€ 100,00
?
PESSIMISTIC LOCKING
Withdraw Money
Account number 12345678
Amount €€ 50,00
Deposit Money
Account number 12345678
Amount €€ 100,00
Account
number
Balance
12345678 €€ 100,00
Account
number
Balance
12345678 €€ 50,00
wait for lock
lock
OPTIMISTIC LOCKING
Withdraw Money
Account number 12345678
Amount €€ 50,00
version 1
Deposit Money
Account number 12345678
Amount €€ 100,00
version 1
Account
number
Balance
12345678 €€ 100,00
version 2
ConcurrencyException
SCALE
PERFORMANCE
➤ Server
➤ Database
➤ Framework
➤ Language
STORAGE
➤ #events
➤ #aggregates
➤ #events_per_aggregate
➤ Serializer
➤ Speed
➤ Payload size
➤ Costs
SNAPSHOTS
Events
1 Account Created
2 Money Deposited
3 Money Withdrawn
4 Money Deposited
SNAPSHOT
5 Money Withdrawn
Events
1 Account Created
2 Money Deposited
3 Money Withdrawn
4 Money Deposited
5 Money Withdrawn
SHARDING
➤ Aggregate Id, Type, Event Timestamp, ...
➤ Rebalancing
➤ Distribution
ARCHIVING EVENTS
➤ Reduce working set
➤ Inactive / deleted aggregates
➤ Historic / irrelevant events
➤ Cheaper storage
FRAMEWORK COMPARISON
Framework Upcasting Snapshots Replaying
Broadway
(PHP)
No (PR) No (PR) Not in core
Prooph
(PHP)
MessageFactory
Yes, triggers on
event count
Example code,
off line
Axon
(Java/Scala)
Upcaster /
UpcasterChain
Yes, triggers on
event count
Yes,
ReplayingCluster
Akka
Persistence
(Java/Scala)
Event Adapter
Yes, decided by
actor
Yes
BROADWAY VS PROOPH
class TodoItem extends EventSourcedAggregateRoot {

private $itemId;



public function __construct(PostTodo $command) {

$this->apply(new TodoPosted($command->getItemId()));

}



public function getAggregateRootId()

{

return $this->itemId;

}



public function applyTodoPosted(TodoPosted $event) {

echo "Posted: " . $event->getItemId() . "n";

}

}
BROADWAY VS PROOPH
class TodoItem extends AggregateRoot {

private $itemId;



public function __construct(PostTodo $command) {

$this->recordThat(new TodoPosted($command->getItemId()));

}



protected function aggregateId() {

return $this->itemId;

}



public function whenTodoPosted(TodoPosted $event) {

echo "Posted: " . $event->getItemId() . "n";

}

}
THANK YOU!

CQRS & event sourcing in the wild

  • 1.
    CQRS & EVENTSOURCING IN THE WILD MICHIEL ROOK
  • 2.
    ➤ Java, PHP& Scala developer ➤ Consultant, trainer, speaker ➤ Dutch Web Alliance ➤ make.io ➤ Maintainer of Phing ➤ @michieltcs
  • 3.
  • 4.
  • 5.
    heard about CQRS/ Event Sourcing RAISE YOUR HAND IF YOU HAVE
  • 6.
    heard about CQRS/ Event Sourcing followed a tutorial, built a hobby project RAISE YOUR HAND IF YOU HAVE
  • 7.
    heard about CQRS/ Event Sourcing followed a tutorial, built a hobby project used it in production RAISE YOUR HAND IF YOU HAVE
  • 8.
    TOPICS ➤ Quick recap ➤Replays and rebuilds ➤ Event versioning ➤ Concurrency ➤ Scale
  • 9.
  • 10.
    ' Event Sourcing ensuresthat all changes to application state are stored as a sequence of events. -Martin Fowler
  • 11.
    ACTIVE RECORD VS.EVENT SOURCING Account number Balance 12345678 €€ 50,00 ... ... Money Withdrawn Account number 12345678 Amount €€ 50,00 Money Deposited Account number 12345678 Amount €€ 100,00 Account Created Account number 12345678
  • 12.
    COMMANDS TO EVENTS DepositMoney Account number 12345678 Amount €€ 100,00 Money Deposited Account number 12345678 Amount €€ 100,00 class DepositMoney {
 public $accountNumber;
 public $amount;
 } class MoneyDeposited {
 public $accountNumber;
 public $amount;
 public $timestamp;
 } function depositMoney(DepositMoney $command) {
 $this->apply(new MoneyDeposited($command->accountNumber,
 $command->amount, date()));
 }
  • 13.
    AGGREGATES class BankAccount {
 public$accountNumber;
 public $balance;
 
 // command handlers...
 
 public function applyAccountCreated(AccountCreated $event) {
 $this->accountNumber = $event->accountNumber;
 $this->balance = 0;
 }
 
 public function applyMoneyDeposited(MoneyDeposited $event) {
 $this->balance += $event->amount;
 }
 
 public function applyMoneyWithdrawn(MoneyWithdrawn $event) {
 $this->balance -= $event->amount;
 }
 }
  • 14.
    AGGREGATE STATE Account number Balance 12345678 €€ 0,00 MoneyWithdrawn Account number 12345678 Amount €€ 50,00 Money Deposited Account number 12345678 Amount €€ 100,00 Account Created Account number 56789012 Account number Balance 12345678 €€ 100,00 Account number Balance 12345678 €€ 50,00
  • 15.
    CQRS ➤ Command QueryResponsibility Segregation ➤ Separate writes (events) and reads (UI) ➤ Optimize for reads!
  • 16.
    Domain Event Bus Event Handlers Command Repository DataLayer Database Database Event Store commands events events queries DTOs Aggregates
  • 17.
    PROS AND CONS ➤Domain fit ➤ Testing ➤ Audit trail ➤ Scalability ➤ Complexity ➤ Library support / maturity
  • 18.
  • 19.
  • 20.
    PROJECTIONS AND READMODELS User Registered User Registered User Unregistered Number of active users?
  • 21.
    PROJECTIONS AND READMODELS User Registered User Deactivated User Reactivated Number of active users? User Registered User Unregistered
  • 22.
    PROJECTIONS AND READMODELS User Registered Event Handler Number of active users +1 User Unregistered Event Handler Number of active users -1
  • 23.
    PROJECTIONS AND READMODELS Events Event Handler(s) Storage
  • 24.
    ELASTICSEARCH class CompanyRegistered {
 public$companyId;
 public $name;
 public $street;
 public $city;
 }
 
 function handleCompanyRegistered(CompanyRegistered $event) {
 $this->elasticsearch->index([
 'index' => 'companies',
 'type' => 'company',
 'id' => $event->companyId,
 'body' => [
 'name' => $event->name,
 'address' => [
 'street' => $event->street,
 'city' => $event->city
 ]
 ]
 ]);
 }
  • 25.
    READ MODEL UPDATES ➤New type ➤ New structure ➤ Based on existing events ➤ Generate from scratch?
  • 26.
  • 27.
    ZERO DOWNTIME Loop over existing events Applyto new read model Apply queued events Start using new read model New events Queue
  • 28.
    ELASTICSEARCH: MAPPING CHANGE Create indexwith new mapping Reindex existing documents Apply queued events Start using new index New events Queue
  • 29.
    CHALLENGE: LONG RUNNINGREBUILDS ➤ Alternatives ➤ Distributed ➤ In memory ➤ Partial ➤ Background
  • 30.
    CHALLENGE: SIDE EFFECTS UserRegistered User Id 123abc Email Address test@example.net Event Handler Exclude during replays!
  • 31.
    CHALLENGE: TRANSACTIONS Event Handler EventHandler Event Event Handler Event Handler ?
  • 32.
    CHALLENGE: EVENTUAL CONSISTENCY ➤Asynchronous event handlers ➤ Reads eventually return the same value ➤ Compare with ACID
  • 33.
  • 34.
    DILEMMA ➤ New businessrequirements ➤ Refactoring ➤ New view on events
  • 35.
    NEW EVENTS /VERSIONS ➤ No longer relevant ➤ Renamed ➤ Additional or renamed field(s) ➤ Too coarse, too fine
  • 36.
    SUPPORT YOUR LEGACY? ➤Events are immutable ➤ Correct (incorrect) old events with new events
  • 37.
  • 38.
    UPCASTING class UserRegistered_V1 {
 public$userId;
 public $name;
 public $timestamp;
 }
 
 class UserRegistered_V2 {
 public $userId;
 public $name;
 public $date;
 }
 
 function upcast($event): array {
 if (!$event instanceof UserRegistered_V1) {
 return [];
 }
 return [
 new UserRegistered_V2($event->userId, $event->name,
 $event->timestamp->format("Y-m-d"))
 ];
 }
  • 39.
    REWRITING HISTORY Load (subsetof) events Deserialize Modify Serialize Save/replace
  • 40.
    THINGS TO BEAWARE OF Upcasting ➤ Performance ➤ Complexity ➤ Existing projections not automatically updated Rewriting events ➤ Running code that depends on old structure ➤ Breaking serialization ➤ Changing wrong events
  • 41.
  • 42.
    CONCURRENT COMMANDS Withdraw Money Accountnumber 12345678 Amount €€ 50,00 Deposit Money Account number 12345678 Amount €€ 100,00 ?
  • 43.
    PESSIMISTIC LOCKING Withdraw Money Accountnumber 12345678 Amount €€ 50,00 Deposit Money Account number 12345678 Amount €€ 100,00 Account number Balance 12345678 €€ 100,00 Account number Balance 12345678 €€ 50,00 wait for lock lock
  • 44.
    OPTIMISTIC LOCKING Withdraw Money Accountnumber 12345678 Amount €€ 50,00 version 1 Deposit Money Account number 12345678 Amount €€ 100,00 version 1 Account number Balance 12345678 €€ 100,00 version 2 ConcurrencyException
  • 45.
  • 46.
  • 47.
    STORAGE ➤ #events ➤ #aggregates ➤#events_per_aggregate ➤ Serializer ➤ Speed ➤ Payload size ➤ Costs
  • 48.
    SNAPSHOTS Events 1 Account Created 2Money Deposited 3 Money Withdrawn 4 Money Deposited SNAPSHOT 5 Money Withdrawn Events 1 Account Created 2 Money Deposited 3 Money Withdrawn 4 Money Deposited 5 Money Withdrawn
  • 49.
    SHARDING ➤ Aggregate Id,Type, Event Timestamp, ... ➤ Rebalancing ➤ Distribution
  • 50.
    ARCHIVING EVENTS ➤ Reduceworking set ➤ Inactive / deleted aggregates ➤ Historic / irrelevant events ➤ Cheaper storage
  • 51.
    FRAMEWORK COMPARISON Framework UpcastingSnapshots Replaying Broadway (PHP) No (PR) No (PR) Not in core Prooph (PHP) MessageFactory Yes, triggers on event count Example code, off line Axon (Java/Scala) Upcaster / UpcasterChain Yes, triggers on event count Yes, ReplayingCluster Akka Persistence (Java/Scala) Event Adapter Yes, decided by actor Yes
  • 52.
    BROADWAY VS PROOPH classTodoItem extends EventSourcedAggregateRoot {
 private $itemId;
 
 public function __construct(PostTodo $command) {
 $this->apply(new TodoPosted($command->getItemId()));
 }
 
 public function getAggregateRootId()
 {
 return $this->itemId;
 }
 
 public function applyTodoPosted(TodoPosted $event) {
 echo "Posted: " . $event->getItemId() . "n";
 }
 }
  • 53.
    BROADWAY VS PROOPH classTodoItem extends AggregateRoot {
 private $itemId;
 
 public function __construct(PostTodo $command) {
 $this->recordThat(new TodoPosted($command->getItemId()));
 }
 
 protected function aggregateId() {
 return $this->itemId;
 }
 
 public function whenTodoPosted(TodoPosted $event) {
 echo "Posted: " . $event->getItemId() . "n";
 }
 }
  • 54.