MongoDB is popular choise for data persistence which had no support for ACID transactions up until v4. Even having such support, transactions incurs higher performance cost and can lead to deadlocks. In this talk I will explain challenges of achieving data consistency in systems that does not support AСID transactions, how to communicate state changes in a reliable manner and the approach to ensure eventual consistency of data with examples in C#.

1. EVENT-DRIVEN SYSTEMS
BACKED BY MONGODB
bit.ly/2DUYXNF

2. ANDRII LITVINOV
Platform engineer at Synergy
Sports Technology

3. CHALLENGES
• Duplicate records matching
• Game statistics calculations
• Reports read model calculations

4. Orders Payments
M
e
s
s
a
g
e
b
r
o
k
e
r
Place
Order
Order
Order
Placed
Order
Placed
Payment
Accepted
Payment
Payment
Accepted

5. M
e
s
s
a
g
e
b
r
o
k
e
r
Orders
Crash
Order
Placed
✕
Order
Placed
?Payments
✓
Order
Place
Order

6. KEY TAKEAWAY:
PERSIST STATE AND
PUBLISH EVENT
ATOMICALLY

7. EVENT SOURCING
Service 1
Event 1
Event 1
Event 1
Event 1
Event
Service 2
Event 1
Event 1
Event 1
Event 1
Event

8. APPLICATION EVENTS
Service
M
e
s
s
a
g
e
b
r
o
k
e
r
Entity EventLocal
transaction
Event
Publisher

9. TRANSACTION LOG TAILING
Service
M
e
s
s
a
g
e
b
r
o
k
e
r
Event
Publisher
Transaction Log
Entity

10. DEMO

11. OUR CONSIDERATIONS
• Go with Oplog tailing
• Event sourcing requires major redesign
• MongoDB 3.6 Change Streams just released
• MongoDB 3.4
• No ACID transactions support so far

12. OPLOG TAILING
• More efforts to implement { w: "majority" }
• Proven, reliable, widely used in production
• Emit event from Primary { w: 1 }
Pros
Cons
• Private undocumented API
• Advised against by MongoDB

13. CHANGE STREAMS 3.6
• Single collection per connection
• Not yet widely used in production
• { w: "majority" } out of the box
• Stable, well-documented API
Pros
Cons
• No way to emit from Primary { w: 1}
• Recommended by MongoDB

16. TAKEAWAY:
FOR PRIMARY-SECONDARY-
ARBITER (PSA) REPLICA SET
USE OPLOG TAILING

17. CHANGE STREAMS 4.0
• { w: "majority" } out of the box
• Option to resume from a given Timestamp
• Subscribe to database or entire deployment
Pros
Cons
• Only supports { w: "majority" }, no { w: 1}
• Recommended by MongoDB

18. TAKEAWAY:
FOR MONGODB 4.0 REPLICA
SET WITH 3+ DATA-
BEARING NODES USE
CHANGE STREAMS

19. var @object = (BsonDocument) operation["o"];
switch ((string) operation["op"])
{
case "i":
{
EmitDomainEvents(@object);
break;
}
case "u":
{
if (@object.TryGetValue("$set", out var set))
EmitDomainEvents((BsonDocument) set);
else
EmitDomainEvents(@object);
break;
}
}

20. var handler = container.GetInstance<IEventHandler<OrderPlaced>>();
Lifestyle.Singleton.CreateRegistration(() =>
new EventHandlersConsumer("payments", database,
new Dictionary<string, Func<DomainEvent, Task>>
{
{
nameof(OrderPlaced),
@event => handler.Handle((OrderPlaced) @event)
}
}, logger), container));

21. await events
.Find(filter.And(
filter.In("event._t", handlers.Keys),
filter.Gt(envelope => envelope.Timestamp, checkpoint.Position)))
.Sort(Builders<EventEnvelope>.Sort.Ascending(envelope => envelope.Timestamp))
.ForEachAsync(async envelope =>
{
if (handlers.TryGetValue(envelope.Event.GetType().Name, out var handler))
{
if (!string.IsNullOrEmpty(envelope.CorrelationId))
TraceContext.Set(envelope.CorrelationId, envelope.EventId);
await handler.Invoke(envelope.Event);
}
await SaveCheckpoint(checkpoint, envelope.Timestamp);
}, cancellationToken);

22. public async Task Handle(OrderPlaced @event)
{
if (@event.Amount >= 200)
{
var delay = new Random().Next(3000, 30000);
await Task.Delay(delay);
}
var payment = new Payment(ObjectId.GenerateNewId(), @event.SourceId);
payment.Process(@event.Amount);
await payments.Create(payment);
}

23. public void Process(decimal amount)
{
Amount = amount;
if (Amount >= 300)
{
Status = PaymentStatus.Rejected;
RecordEvent(new PaymentRejected(Id, OrderId, Amount));
}
else
{
Status = PaymentStatus.Accepted;
RecordEvent(new PaymentAccepted(Id, OrderId, Amount));
}
}

24. var futureEvent = observable.FirstOfType<OrderFulfilled, OrderDiscarded>(orderId);
var command = await request.Body.ReadAs<PlaceOrder>();
await handler.Handle(command);
try
{
await futureEvent;
response.StatusCode = (int) HttpStatusCode.Created;
}
catch (TimeoutException)
{
response.StatusCode = (int) HttpStatusCode.Accepted;
}
var order = await orders.Find(o => o.Id == orderId).FirstAsync();
await response.Body.Write(order);

25. SUMMARY
• Use change streams with MongoDB 4.0
and 3+ data-bearing nodes
• Write state and publish event atomically
• Use oplog tailing prior MongoDB 4.0
or with 2 data-bearing nodes

26. RESOURCES
Replication
https://www.slideshare.net/Stripe_talks/mongo-db-oplog
https://www.compose.com/articles/the-mongodb-oplog-and-node-js/
https://stackoverflow.com/questions/17878835/what-does-the-h-in-oplog-rs-document-stand-for
https://stackoverflow.com/questions/26935419/what-do-the-oplog-fields-actually-mean
Event-Driven-Architecture
https://www.nginx.com/blog/event-driven-data-management-microservices/
https://microservices.io/
https://skillsmatter.com/skillscasts/12115-keynote-not-just-events-developing-asynchronous-microservices
Domain model
http://dddcommunity.org/library/vernon_2011/

27. THANK YOU! QUESTIONS?
https://twitter.com/andriilitvinov
https://github.com/andrii-litvinov
https://www.upwork.com/fl/andriilitvinov