This document discusses using CQRS, EVS, and an ESB to build distributed enterprise applications, using a shipment management application as an example. It describes the patterns, including using CQRS to separate reads from writes, an event store to record changes as events, and an ESB to facilitate communication between services. Technologies used include ASP.NET, RabbitMQ, and a sharded MongoDB cluster to store domain data, views, events, and metadata for subscriptions and metrics. The solution aims to handle high throughput from many agencies while maintaining data availability and partitioning data across the MongoDB shards.

1. CQRS & EVS with
Building of a modern distributed enterprise application

2. CQRS & EVS with
• My name is Lluis Fernández.
• I’m involved in a tech group of freelancers named
B-Team.
• We offer consultancy on distributed system
architectures.
• Contact:
llffernandez@gmail.com
b-team_b-team@googlegroups.com

3. • What’s this about.
• The patterns: CQRS, EVS and ESB.
• All together: An example of distributed
enterprise application.
CQRS & EVS with

4. CQRS & EVS with
What’s this about?

5. CQRS & EVS with
• NoSQL databases, like MongoDB, have
contributed to the adoption and implementation
of archetypes and patterns to develop
distributed enterprise applications.
• High read/write rates, huge data stores
management, highly scalable read-only
replication and schema-less document
orientation, are well suited for those kind of
patterns and techniques.

6. CQRS & EVS with
• This presentation is about how some of these
patterns and archetypes had been applied during
the implementation of real-word use case: a
Shipment Management application for one of
the most important Messaging and Packages
Delivery companies in Spain.

7. CQRS & EVS with
CQRS.

8. CQRS & EVS with
• CQRS stands for Command-Query Responsibility
Segregation.
• Commands are expected to do writes of small
portions of data.
• Queries are expected to do reads of big portions
of data.
• CQRS encourages the separation of these
different workloads.

9. CQRS & EVS with
CQRS facade
Commands
Queries
Shipment
management
core services
View data
synchronization
Regular
operations on
Master data
Master DB
(R/W)
Views DB
(Readonly)
Replication
Corporate
shipment
management
web app.
Packages
Add package
Add destination
Add origin
Add delivery
Assign courier
Pending
deriveries
Lost packages
Packages by
shipment
Packages by
shipment

10. CQRS & EVS with
• Reduces contention and collision between
queries and commands persistence-level
operations.
• Allows the scaling-out of queries, usually the
most common and expensive operations.
• Allows the use of heterogeneous view data
stores.

11. CQRS & EVS with
Shipment management core services
Simple automatic synchronization
(replication)
Queries
Commands
Datasets optimized
for expected queries
(grouping, filtering,
joins, etc.)
Infrastructure-provided
synchronization mechanism.
No optimization:
Master data = View data
Regular datasets
(packages,
deliveries, etc.)
These synchronization
processes introduce
staleness of data
provided to the queries.
Views DB (Readonly) Master DB (R/W)
View data
synchronization
Regular
operations on
Master data
Packages
DeliveriesCustomers
Packages by
shipment
Package
tracking

12. CQRS & EVS with
Queries
Commands
Shipment management core services
Queries
Views DB (Readonly)
View data
synchronization
View data
synchronization
Regular
operations on
Master data
Master DB (R/W)
More copies of view
data or more different
views.
Also, copies of view data
can be heterogeneous.
Views DB (Readonly)
Packages by
shipment
Package
tracking
Packages by
shipment
Package
tracking
Packages
Deliveries
Customers

13. CQRS & EVS with
• MongoDB replica sets and ReadPreference can
be used to segregate master and view data:
 Commands write on the PRIMARY.
 View data is also written on the PRIMARY.
 Queries read from SECONDARIES.
• Separate master and view data collections in
different databases (MongoDb write locking).
• Schema optimization and indexing in collections
storing view data also improve performance.

14. CQRS & EVS with
Shipment management core services
Queries are launched against
SECONDARIES (SECONDARY or
SECONDARY_PREFERRED read
preference)
MongoDB replication
Commands
Queries
View data synchronization
Packages per shipment
view synchronizer
Data optimized for views:
 Schema design.
 Indexing.
Multiple SECONDARIES
allows queries scaling.
Regular
operations on
Master data
Commands are launched against
PRIMARIES (PRIMARY or
PRIMARY_PREFERRED read preference)
View data ALSO goes
to PRIMARIES
Master
data
(affected by
commands)
goes to
PRIMARIES
PRIMARY
(Master and Views data)
SECONDARIES
(View data)
Packages by
shipment
Package
tracking
Packages
DeliveriesCustomers
Packages by
shipment
Package
tracking
Packages
DeliveriesCustomers

15. CQRS & EVS with
PRIMARY
(Master and
View data)
SECONDARY
(View data)
Packages
MongoDB replication
Packages per
shipment
Queries
Packages per
shipment
Shipment management
core services
Master and View
data are stored in
PRIMARY, in
different databases.
View data is replicated to
SECONDARY
Commands
Packages per shipment
view synchronizer
View data
synchronization

16. CQRS & EVS with
• Caveats:
Staleness of data read by queries.
Availability loss and overloads of PRIMARY in case
of SECONDARIES failures.
Balancer in sharding environments can introduce
additional staleness and duplication in data being
read.

17. CQRS & EVS with
EVS.

18. CQRS & EVS with
• EVS stands for EVent Sourcing.
• Closely related to DDD (Domain Driven Design) and
CQRS.
• Events are changes to the state of a domain model
subset of an application.
• Events occurred in the past, and are also referred as
Domain events.
• EVS consists in expressing the state of a domain
model subset as a sequence of events.

19. CQRS & EVS with
Shipment managment core service
EVS engine
Add package
Add package
Create
Add package
Assign courier
Add delivery
Add delivery
Add delivery
Last event
Store
Incoming Domain Events Last event
Delivery
Package
Customer
Courier
Promo
Rebuild
Retrieve
Event store
Apply
ALL retrieved events are
replayed to rebuild domain
model subset state
Last event is aplied after
domain model subset
rebuild, and stored in the
Event Store.

20. CQRS & EVS with
• Events are stored in a Event Store, that can be
huge (ideally infinite), depending on the domain
subset’s lifetime.
• To keep growth of the Event Store under control:
Store a point-in-time state of the domain model
subset (take a snapshot of it).
Store only events occurred after the last snapshot.

21. CQRS & EVS with
Shipment management core service
EVS engine
Add package
Add package
Create
Add package
Assign courier
Add delivery
Add delivery
Add delivery
Last event
Store
Incoming Domain Events Last event
Delivery
Package
Customer
Courier
Promo
Rebuild
Retrieve
Event store
Snapshot
Apply
Store snapshot
ONLY events after snapshot
are used to rebuild domain
model subset state.
A snapshot is taken after
replaying some events.
Remaining events (events
after th snapshot) are used
to rebuild the state of the
domain model subset and
remain stored in the Event
Store.

22. CQRS & EVS with
• Single collection event store:
Both snapshots and events conform a single
document => Single collection.
findAndModify can be well suited to add events
and update or create snapshots in a single
atomic operation.
Usable in scenarios with low frequency rate of
events.

23. CQRS & EVS with
Event
Event
Event
Event
Event
Event
Event
Event
Store
snapshot
AggregateId
AggregateId
Root
Entity
Value
object
Value
object
Entity
Root
Entity
Value
object
Value
object
Entity
Component / service
EVS
engine
StoreRetrieve
Event store
Event
Event
Documents include both
snapshot and events
pending to apply.
Events are added with a
$push operation in a
findAndModify atomic
operation.
Due to growth of
events array,
documents can
be moved often,
impacting
performance

24. CQRS & EVS with
• Multiple collection event store:
Snapshots and events are persisted in different
collections.
May require some locking and operations isolation
management in multithreaded environments.

25. CQRS & EVS with
Event
Event
Event
Event
Event
Event
Event
Event
Store
snapshot
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
AggregateId
Root
Entity
Value
object
Value
object
Entity
Root
Entity
Value
object
Value
object
Entity
Component / service
EVS
engine
StoreRetrieve
Event store
Snapshots and events are
stored in their respective
collection, separated from
each other.
Snapshots
and events
are stored in
a single
atomic
update
operation.

26. CQRS & EVS with
ESB.

27. CQRS & EVS with
• ESB = Enterprise Service Bus. It is an
infrastructure piece of software than a pattern
itself.
• Implements different messaging patterns:
Publish / Subscribe
Request / Reply
Dead letter channel.
Others.

28. CQRS & EVS with
Publisher Subscriber
ServiceService Service Service
Subscriber Subcriber
ESB config DB
Message published and
delivered to all subscribers.
ESB configuration data
persistence: subsciptions
database.
Requester DB Service DB Service DB
Request message, delivered to replier.
Replier info in message header.
Reply message
delivered to
requester.

29. CQRS & EVS with
• MongoDB is applicable as a persistence layer for
subscriptions store.
• In complex applications, with many services
interconnected:
Persistence layer for ESB usage statistics and
metrics (latency, msg/s, queue length, etc.).
Persistence layer for Dead Letter Channel
subscribers (logger for all services connected to
ESB).

30. CQRS & EVS with
Publisher Subscriber
ServiceMonitor Service Service
Subscriber Subcriber
ESB config DB
Metrics, heartbeat, statistics
sample message.
Statistics DB Service DB Service DB
Dead letter channel
message (error on
component captured)
Statistics database.
Dead letter channel
database (logger).

31. CQRS & EVS with
• Two main messaging streams: data and control.
• Control messages:
Monitoring metrics samples.
Start / pause / stop service processing.
Heartbeat messages (ping to services).
• Message traffic is lead by a transport tier.
• To prevent contention and service saturation, a
buffering or queuing system is recommended.

32. CQRS & EVS with
Service 2 Service 3
Control ControlProcess Process
Service 1
ControlProcess
Bus service
Process (Message Routing) Control
Action handling Action handling Action handling
Data
messages
Control
messages
Control
handling
Control
handling
Control
handling
Durable direct RabbitMQ exchange (Control)
Durable direct RabbitMQ exchange (Command, Events, Documents)
Service contracts
serialized as JSON
messages
Bus config DB
Service 3
Service 2
Mirrored RabbitMQ
queues
Mirrored RabbitMQ
queues
Mirrored RabbitMQ
queues

33. CQRS & EVS with
All together: An example of
distributed application.

34. CQRS & EVS with
• Main functional requirements:
Allow management of shipments generated online
by agencies.
Allow management of agency and corporate
customers.
Allow offline loading of shipments generated by
large customers.
Integrate with other line-of-business systems
(CRM, legacy applications, etc.).

35. CQRS & EVS with
• Main QoS requirements:
Nearly 550 agencies.
3K online users during office time.
Several thousands of customers.
Storage needs based on 50K shipments / day
(online and offline workloads) and 45 days of
operational shipment lifetime.
Estimation of 20 state changes per shipment
during its lifetime.

36. CQRS & EVS with
• Solution components:
Frontend: MVC UI.
A layer of CQRS WS interfaces frontend and
backend.
Backend: services (some implementing EVS)
interconnected by ESB.
A MongoDB Sharding Cluster as the persistence
tier for them all.

37. CQRS & EVS with
• Technologies stack:
UI: ASP.NET MVC + JQuery.
CQRS + REST facade: ASP.NET MVC
ESB + Services: .NET Framework based
Windows Services.
Messaging transport: RabbitMQ cluster.
Persistence: MongoDb sharding cluster.

38. CQRS & EVS with
Services
Commands
Events
Documents
Commands
Events
Documents
Commands
Events
Documents
Commands
Events
Documents
MongoDb Sharding Cluster
DDD
Services
Publishers Subscribers
EVS
Services
Queries
Commands
CQRS REST
WS
Domain data:
 Events.
 Aggregates.
 Snapshots.
Domain data:
 Entities.
 Value objects.
 Aggregates.
Domain data:
 Entities.
 Value objects.
 Aggregates.
 Specifications.
Master dataViews
Service contracts:
 Events.
 Commands.
 Documents.

39. CQRS & EVS with
Web Frontend
Web applications
CQRS WS
Load balancing
RabbitMQ client
MongoDB
IIS
NLB
.NET Framework
ASP.NET MVC
MongoDb Router
Services tier
Backend and core services
Application services
Integration services
.NET Framework
SQL Server Client
Cliente RabbitMQ
MongoDb Router (mongos)
Cluster config
Config 1
Config 2
Config 3
Shard 2
Node 1 Node 2
Arbiter
Shard 1
Node 1 Node 2
Arbiter
Persistence tier
CQRS view data.
Services Domain models persistence
ESB subscriptions DB persistence
MongoDB sharded cluster
(2 shards)
MongoDb Router
(mongos).
A router per
appserver.
They cache a local
copy of the cluster
config DB.
Two-phase commit replication
servers.
If one or two config servers fail,
cluster configuration DB goes
readonly (chuncks cannot be moved
between shards, no balancing), but
data remains writable and available.
Data partition and distribution
Sharded collections: distribution is
done in a per-collection basis.
Data is partitioned and distributed
across shards, attending to a
sharding (partition) key.
Each partition unit is called a
chunk.
EVS collections can be partitioned
by a hash on document _id.
Sharding keys for collections
persisting view data must be
choosed for each single case, to
guarantee queries performance.

40. CQRS & EVS with
Than you for your attention.