MongoDB Transactions are a major new piece of functionality for users. Early releases of the MongoDB database focused on applications without rigorous transactional semantics, as is common in NoSQL databases. However, some MongoDB users desire advanced transactional features, including multi-document transactions, point-in-time reads, and the choice of snapshot or read-committed isolation.

1. MongoDB:
A New Transactional Model
keith.bostic@mongodb.com

2. KEITH BOSTIC
Distinguished Engineer,
MongoDB
keith.bostic@mongodb.com

3. Review & Motivations

4. Relational vs document data models

5. Goals: performance and complexity

6. Goals: new applications, programmer
models

7. Goals: checkboxes and vendor lock-in

8. Application
developers our
focus

9. But not our only audience

10. “ACID transactions are a key capability for
business critical transactional systems,
specifically around commerce processing.
No other database has both the power of
NoSQL and cross collection ACID
transaction support. This combination will
make it easy for developers to write
mission critical applications leveraging the
power of MongoDB.”
– Dharmesh Panchmatia,
Director of E-commerce, Cisco Systems

11. ACID

12. Transactions: Acid

13. Transactions: aCid

14. Detour:
Consistency, Availability, Partition
tolerance
C
PA

15. Transactions: acId

16. Transactions: aciD

17. MongoDB World talk!
Transactions and Durability:
Putting the “D” in ACID
– Sue LoVerso, Senior Engineer, MongoDB

18. The Path

19. Multi-year, all-hands company effort
• The storage layer,

20. Multi-year, all-hands company effort
• The storage layer,
• Sharding architecture,

21. Multi-year, all-hands company effort
• The storage layer,
• Sharding architecture,
• Introducing a global logical
clock,

22. Multi-year, all-hands company effort
• The storage layer,
• Sharding architecture,
• Introducing a global logical
clock,
• New replication consensus
protocol,

23. Multi-year, all-hands company effort
• The storage layer,
• Sharding architecture,
• Introducing a global logical
clock,
• New replication consensus
protocol,
• Metadata management,
… and that’s just part of the
list!

24. Multi-year, all-hands company effort
• The storage layer,
• Sharding architecture,
• Introducing a global logical clock,
• New replication consensus
protocol,
• Metadata management,
… and that’s just part of the list!
• Not forgetting the driver teams,
documentation and education.

25. MongoDB 2.6: mmapv1 with ACID
2.6
P
S
S
S

26. Detour: the storage engine
The storage engine is part of the MongoDB database server.
Think of it as “owning” the data.
networking, sharding, replication, drivers
analytics, middleware, query optimizer
storage engine
You
are
here.

27. Detour: the storage engine
Single
Node

28. MongoDB’s pluggable storage architecture
• MMAPv1
• RocksDB (Facebook)
• TokuMX (TokuTek)
• WiredTiger

29. MongoDB 3.0: a new storage engine
F Keys &
Values
E Keys &
Values
G Keys &
Values
Write-heavy workloads
Document level locking
Compaction
Encryption
Compression
In-Memory
2.6 3.0
B-D
Keys
E-I
Keys
J-N
Keys

30. MongoDB 3.0: transactional features
2.6 3.0

31. MongoDB 3.2
P
S
S
S
“majority” writes
readConcern
2.6 3.0 3.2

32. MongoDB 3.4
2.6 3.0 3.43.2

33. MongoDB 3.6: transactional features
P
S
S
S
logical sessions
global clock (Lamport)
… leads to causal consistency
2.6 3.0 3.63.43.2

34. MongoDB 3.6: enhanced consistency
P
S
S
S
consistent secondary reads
retryable writes
read concern majority
2.6 3.0 3.63.43.2

35. MongoDB University video
Implementation of Cluster-
Wide Causal Consistency in
MongoDB
– Misha Tyulenev, Engineer, MongoDB

36. MongoDB 4.0: multi-document transactions
P
S
S
S
2.6 3.0 3.63.43.2 4.0

37. MongoDB 4.0: prepared transactions
Commit?
Yes
Yes
Commit!
Phase One Phase Two
2.6 3.0 3.63.43.2 4.0

38. MongoDB 4.X
P
S
S
S
PS
S
2.6 3.0 3.63.43.2 4.0 4.X

39. MongoDB World talk!
What’s Next? The Path to
Sharded Transactions
– Andy Schwerin, VP, Engineering, MongoDB

40. Why is this work hard?

41. Reason #1: single-node vs. group ordering
WiredTigerMongoDB

42. Replication example
... 37 38 39
... 37 38 39
... 37 38 39
... 37 38 39
Primary Oplog
Secondary reads

43. “People assume that time is a strict
progression of cause to effect, but
actually, from a non-linear, non-
subjective point of view, it is more
like a big ball of wibbily-wobbly
timey-wimey … stuff.”
– Doctor Who

44. Multi-threaded secondary oplog application
... 37 38 39 40 4241
Primary oplog order
Secondary apply order
40 39 41 38
OpLog

45. Reason #2: increasing entanglement
Getting the right answer requires communication.
C
PA

46. Timestamps

47. MongoDB’s timestamp
... 40 4138 39 42
ACID for a set of systems
durability based on replication

48. WiredTiger’s transaction ID
... 40 4138 39 42
ACID for a single-node
durability based on checkpoints and journaling

49. MongoDB timestamp overrides
WiredTiger ID
64-bit value
fast comparisons
strictly increasing
MSB hex string API

50. Timestamp stored with each MVCC update
FRUIT
mango
apple
banana
TXN ID
TIMESTAMP

51. Multi-threaded secondary oplog application
... 37 38 39 40 4241
Primary oplog order
Secondary timestamp order
... 37 38 39 40 4241
Set the timestamp at update or commit
OpLog

52. Overlapping reads with oplog application
Secondary timestamp order
... 37 38 39 40 4241
Timestamp #1 Timestamp #2
3.6
4.0
Blocking
Parallel

53. Queries can be “as of” a timestamp
set at transaction begin
largest LTE value

54. We need an “oldest” timestamp

55. Durability can be “as of” a timestamp

56. Replication rollback for a single server
... 37 38 39 40 4241
Primary Oplog
... 37 38 39 117 119118
Corrected (Secondary) Oplog
Replacements

57. We need a “stable” timestamp

58. Sydney & New York

59. KEITH BOSTIC
Distinguished Engineer,
MongoDB
Keith.bostic@mongodb.com