Wait, isn't Cross-Shard Transactions a Sharding feature? What does it have to with Replication? Well, in order to make sharded transactions work, we needed a distributed commit protocol to handle various failovers. One key stage of this protocol is making sure that all shards are “prepared” before they commit. As a result, the replication system needed to make a promise: a transaction that is in the “prepare” state will survive failovers. This means that we don’t corrupt your transactions and we don’t deadlock your system. We worry about those things so that you don’t have to.In this talk, we will follow the lifetime of a sharded transaction and show you how we keep your data safe. You’ll hear about design decisions, implementation details, and key behaviors from two of the engineers that built Prepare Support for Sharded Transactions.

2. Introducing...

3. Introducing...
This is Jon Leaf.He knows nothing.

5. Army Transfer
@mother_of_acorns @lonely_boi_8
10,000 X ?

6. army.update(
{_id: "Daenerys's Army"},
{$inc: {soldiers: 10000}});
Single Replica Set Transactions
army.update(
{_id: ”Jon's Army"},
{$inc: {soldiers: -10000}});
session.startTransaction();
session.commitTransaction(
{writeConcern: {w:’majority’}});
Transaction
Committed!
Transaction
Committed!

7. Definition of commit
Committed Transaction: When all operations in a MongoDB
transaction are visible outside of the transaction
Majority Committed: When an operation is replicated to a majority
of the replica set

8. Jon Leaf needs sharding

9. Sharded Transactions: A Cautionary Tale
army.update(
{_id: ”Jon's Army"},
{$inc: {soldiers: -10000}});
army.update(
{_id: ”Daenerys's Army"},
{$inc: {soldiers: 10000}});
session.startTransaction();session.commitTransaction();
Transaction
Committed!
Transaction
Failed!

10. Sharded Transactions: A Cautionary Tale
@mother_of_acorns@lonely_boi_8

11. Atomicity in Transactions
This violates a key principle for transactions:
Transactions must be All or Nothing

12. Committing Transactions
There are two possibilities when committing on multiple shards:
• All shards successfully commit
• One or more shards fail to commit
We need something to help us deal with both cases

13. The Prepare State
A transaction in the prepared state is guaranteed to be able to
commit

14. The Prepare State
A transaction in the prepared state is guaranteed to be able to
commit
All participating shards must prepare the transaction on a majority
before any shard can commit

15. The Prepare State
A transaction in the prepared state is guaranteed to be able to
commit
All participating shards must prepare the transaction on a majority
before any shard can commit
If any shard fails to prepare, then no shard will commit

16. prepareTransaction(
{writeConcern: {w: “majority”}});
Aborted Cross-Shard Transaction
army.update(
{_id: ”Jon's Army"},
{$inc: {soldiers: -10000}});
army.update(
{_id: ”Daenerys's Army"},
{$inc: {soldiers: 10000}});
session.startTransaction();session.commitTransaction();
Transaction
Aborted!

17. prepareTransaction(
{writeConcern: {w: “majority”}});
commitTransaction();
Committed Cross-Shard Transaction
army.update(
{_id: ”Jon's Army"},
{$inc: {soldiers: -10000}});
army.update(
{_id: ”Daenerys's Army"},
{$inc: {soldiers: 10000}});
session.startTransaction();session.commitTransaction();
Transaction
Committed!
Transaction
Committed!
Transaction
Committed!

18. Successful Cross-Shard Army Transfer!
@mother_of_acorns@lonely_boi_8

19. Transaction in Prepared State

20. The Prepare State
The storage engine only stores in-memory information about a
prepared transaction

21. The Prepare State
The storage engine only stores in-memory information about a
prepared transaction
Replication is in charge of the durability guarantees for prepare

22. Durability of Prepare
Replication promises that we:
• Don't lose your prepared transactions
• Don't corrupt your prepared data

23. Prepare Conflicts: Read Concern Local/Majority
army.find(
{_id: ”Jon's Army"});
{_id: “Jon’s Army”,
soldiers: 20000}
{_id: “Jon’s Army”,
soldiers: 10000}
{_id: “Jon’s Army”,
soldiers: 20000}
*without causal consistency

24. Prepare Conflicts: Other Reads and All Writes
army.find(
{_id: ”Jon's Army"});
commitTransaction();
{_id: “Jon’s Army”,
soldiers: 20000}
{_id: “Jon’s Army”,
soldiers: 10000}
{_id: “Jon’s Army”,
soldiers: 10000}

25. Steady state works!

26. But what about failover?

27. What do we say to failover?

28. Failovers and Reconfigs
● Elections
● Startup Recovery
● Initial Sync
● Rollback

29. Elections

30. Elections with Prepared Transactions
commitTransaction();
Transaction
Committed!

31. Elections with Prepared Transactions
Failed to
Prepare!

32. Failovers and Reconfigs
● Elections
● Startup Recovery
● Initial Sync
● Rollback

33. Startup Recovery

34. Startup Recovery
Consistent
Point In Time

35. Session State
aeff42 Prepared
bdfe32 Prepared
Session State
aeff42 Prepared
Session State
aeff42 Prepared
bdfe32 Committed
Session State
aeff42 Prepared
bdfe32 Committed
Session State
aeff42 Prepared
bdfe32 Committed
Startup Recovery with Prepared Transactions
Consistent
Point In Time

36. Startup Recovery with Prepared Transactions
This is safe because:
1. We cannot change prepared transactions
2. Prepare conflicts on the primary will prevent any conflicting
writes from succeeding
3. We don’t accept reads while in recovery

37. Failovers and Reconfigs
● Elections
● Startup Recovery
● Initial Sync
● Rollback

38. Initial Sync with Prepared Transactions
Session State
aeff42 Prepared
Session State
aeff42 Prepared

39. Initial Sync with Prepared Transactions
Uses the same algorithm as startup recovery

40. Failovers and Reconfigs
● Elections
● Startup Recovery
● Initial Sync
● Rollback

41. Rollback
When you play the game of
transactions, you commit or abort.
There is no middle ground.
- Cersei Leafister

42. What We Mean By "Rollback"

43. Rollback (4.0+)
Stable Timestamp – Consistent majority committed point in time for
replication and storage
Common Point – Point in time after which the rollback node and the
new primary diverge

44. Rollback (4.0+)
Stable Timestamp – Consistent majority committed point in time
for replication and storage
Common Point – Point in time after which the rollback node and the
new primary diverge

45. Rollback (4.0+)
Stable Timestamp – Consistent majority committed point in time for
replication and storage
Common Point – Point in time after which the rollback node
and the new primary diverge

46. Rollback Algorithm
stableTS
common
point
New primary
branch
Rolling back
branch

47. Rollback Algorithm
stableTS
common
point
New primary
branch
Rolling back
branch

48. Rollback Algorithm
stableTS
common
point
New primary
branch

49. Rollback with Prepared Transactions
stableTS
common
point
New primary
branch
Rolling back
branch

50. How do we handle all these scenarios?
The same way

51. Rollback with Prepared Transactions
stableTS
common
point
New primary
branch
Rolling back
branch
Session State
aeff42 Prepared
bdfe32 Prepared
Session State
aeff42 Prepared
Session State
aeff42 Prepared
bdfe32 Prepared
Session State
aeff42 Prepared
bdfe32 Prepared

52. Your data is safe!
We didn't lose your prepared transactions
We didn't corrupt your prepared data

53. Takeaways
Why we need the prepare state
Why prepare needs to be durable
How replication makes prepared transactions
durable