MongoDB can be configured to meet the requirements of active-active applications across multiple data centers. There are three main deployment patterns: 1) active-passive with one data center as primary, 2) partitioned databases with each data center owning a partition, and 3) multi-master with each data center acting as a master. The document discusses how to tune MongoDB for performance, consistency, availability, and durability using features like sharding, read preference, write concern, and causal consistency.

1. Active-Active Application
Architectures
Become a MongoDB Multi-Data Center Master

2. Jay Runkel
Master Solutions Architect, MongoDB
jay.runkel@mongodb.com
@jayrunkel

3. Today’s Challenge

4. Organizations Want Active Active

5. Replica Sets Appear to be a
Mismatch?

6. Can MongoDB used for Active Active
Applications?
Hmmm….
What are all the requirements?

7. Active Active Requirements
Performance/Latency – read and write locally. Millisecond performance.
Consistency – reads see results of previous writes, readers in various regions
get same results, no conflict resolution
Availability – operation continues through loss of nodes, networks, data
centers, etc.
Durability – data replicated among nodes. Data is not lost when failures occur

8. Requirement 1: Latency
Read ReadWrite Write

9. Active DR solution many not meet latency requirements
10ms
2 ms
45 ms
35 ms 75 ms
105 ms
85 ms

10. Active Active provides consistent low latency
10ms
2 ms
10 ms
4 ms 2ms
10 ms
4 ms
2ms

11. Requirement 2: Consistency
Write {x:1} Write {x:2}
{x:0} {x:0} {x:0} {x:0}

12. Requirement 2: Consistency
Write {x:1} Write {x:2}
{x:0} {x:1} {x:0} {x:2}
Reads return the results of previous writes?
{x:0}

13. Requirement 2: Consistency
Write {x:1} Write {x:2}
{x:0} {x:1} {x:0} {x:2}
Reads in different regions return the same results?
{x:1} {x:2}

14. Requirement 2: Consistency
Write {x:1} Write {x:2}
{x:1} {x:1} {x:2} {x:2}
No conflict resolution?
{x:1} {x:2}

15. Requirement 2: Consistency
Write {x:1} Write {x:2}
{x:1} {x:1} {x:1} {x:1}
No Data Loss?
{x:1} {x:2}

16. Requirement 3: Availability
Resilience from loss of a node

17. Requirement 3: Availability
Resilience from loss of a region

18. Requirement 3: Availability
Resilience from loss of network

19. Requirement 4: Durability
Write {x:1}
{x:0} {x:1} {x:0} {x:0}
Writes survive loss of nodes, data center, network?

20. “You can't always get what you want
But if you try sometimes you might find
You get what you need”
- Rolling Stones

21. Trade Offs
Performance
0 ms 1 sec
Durability
1 node 2 nodes 3 nodes n nodes
Consistency
eventual causal linearizable
Read
Availability primary
(with
secondary
reads)~100%
Write
Availability primary ~100%retryable
strong readConcern: majority

22. Agenda For Remainder of This Talk?
1. MongoDB Architecture Review
a. MongoDB Replica Sets
b. Trading Off:
• Consistency
• Availability
• Durability
• Performance
2. Three Multi-Data Center Deployment Patterns

23. MongoDB Architecture

24. Tuning MongoDB to meet Active Active
Requirements
Active Active Requirements MongoDB Capability
Performance Zone Sharding
Availability Secondary Reads
Retryable Writes
Consistency Read Preference
Read Concern
Durability Write Concern

25. Replica Sets
Replica Set – 2 to 50 copies
Self-healing shard
Data Center Aware
Addresses availability considerations:
High Availability
Disaster Recovery
Maintenance
Workload Isolation: operational & analytics

26. Sharding: Scaling MongoDB

27. Tuning Read and Write Availability
Read Preference
Retryable Writes

28. Node 1
(Primary)
Node 2
(Secondary)
Node 3
(Secondary)
Application
Driver
Read
Read Preference: Primary
Default
All reads from Primary

29. Node 1
Node 2
(Secondary)
Node 3
(Secondary)
Application
Driver
Read Preference: Primary
{x: 100}
Node 1
(Primary)
Read
Can’t read, if no primary

30. Node 1
Node 2
(Secondary)
Node 3
(Secondary)
Application
Driver
Read Preference: PrimaryPreferred
{x: 100}
Node 1
(Primary)
Read
Read from secondary, if no primary
100% read availability
SecondaryPreferred also
Read Read

31. Node 1
Node 2
(Secondary)
Node 3
(Secondary)
Application
Driver
Write Availability → Retryable Writes
{x: 100}
Node 1
(Primary)
Write
Can’t write, if no primary
Retryable writes - retries failed
writes

32. Tuning Consistency
Primary vs. Secondary Reads
Read Concern

33. Node 1
(Primary)
Node 2
(Secondary)
Node 3
(Secondary)
Application
Driver
Read
Primary Reads → Strong Consistency
Reads return all previous writes

34. Node 1
(Primary)
Node 2
(Secondary)
Node 3
(Secondary)
Application
Driver
Secondary Reads → Eventually Consistent
Read Read
Reads may reflect a past state of
the data

35. Challenge #1 - Reading Rolled Back Data
Node 1
Node 2 Node 3
Application
Driver
{x: 101}
{x: 100}
{x : 100}{x: 100}
write
{x: 101}
Application
Driver
{x: 101}
Node 1
(Primary)

36. Challenge #1 - Reading Rolled Back Data
Node 2 Node 3
Application
Driver
{x: 101}
{x : 100}{x: 100}
Application
Driver
{x: 101}
{x: 100}
{x: 100}

37. Node 1
Node 2 Node 3
Application
Driver
{x: 101}
{x: 101}
{x : 100}{x: 100}
write
Application
Driver
{x: 100}
ReadConcern: Majority

38. Challenge #2 - Secondary Reads Don’t
Reflect My Writes
Node 1
Node 2 Node 3
Application
Driver
{x: 101}
{x: 100}
{x : 100}{x: 100}
write
{x: 101}
{x : 100}
read

39. Causal Consistency
Node 1
Node 2 Node 3
Application
Driver
{x: 101}
{x: 100}
{x : 100}{x: 100}
write
{x: 101}
read
{x: 101}

40. Tuning Durability
Write Concern

41. Node 1
Node 2 Node 3
Application
Driver
Replica Set Write Durability
{x: 100}
{x: 100}
{x : 100}{x: 100}
writeack

42. Node 1
Node 2 Node 3
Application
Driver
Replica Set Write Durability
{x: 100}
{x: 100}
write
Node 1
(Primary)

43. Node 1
Node 2 Node 3
Application
Driver
Replica Set Write Durability
{x: 100}
{x: 100}
write
Node 1
(Primary)

44. Node 1
(Primary)
Node 2
(Primary)
Node 3
(Secondary)
Application
Driver
Write Durability
{x : 100}
Node 1
(Primary)
{x : 100}

45. Write Durability is Configured via Write Concern
• Intelligent write receipt/confirmation
– Specifieds the the number of nodes that must have written the write to
disk
– Default number is 1
• NOT a distributed transaction
db.test.insert({x:100},{writeConcern:{w:2}})

46. Node 1
Node 2 Node 3
Application
Driver
Write Concern Majority
{x: 100}
{x: 100}
{x : 100}{x: 100}
writeack
db.test.insert({x:100},
{writeConcern:{w:”majority”}})

47. MongoDB Retryable Writes
Write failure handling moved from the app to the database for transient network errors or
primary elections
– Driver automatically retries failed write
– With a unique transaction identifier, server enforces exactly-once
processing semantics
• Properties
– Supports idempotent & non-idempotent operations, and errors caused
by time-outs
• Delivers always-on, global availability of write operations
– Overcomes the complexity imposed by multi-master, eventually
consistent systems

48. Zone Sharding

49. Configuring MongoDB
Performance
0 ms 1 sec
Durability
1 node 2 nodes 3 nodes n nodes
Consistency
eventual causal linearizable
Read
Availability primary
(with
secondary
reads)~100%
Write
Availability primary ~100%retryable
strong readConcern: majority

50. Performance Optimized
Performance
0 ms 1 sec
Durability
1 node 2 nodes 3 nodes n nodes
Consistency
eventual causal linearizable
Read
Availability primary
(with
secondary
reads)~100%
Write
Availability primary ~100%retryable
strong readConcern: majority

51. Durability and Consistency Optimized
Performance
0 ms 1 sec
Durability
1 node 2 nodes 3 nodes n nodes
Consistency
eventual causal linearizable
Read
Availability primary
(with
secondary
reads)~100%
Write
Availability primary ~100%retryable
strong readConcern: majority

52. MongoDB Multi-Data
Center Deployments

53. Writes: Multi-Data Center
3 Alternatives:
Two-phase commit across data centers
• Consistent
• Slow
Multi master
• Each data item can be written to any data center
• Inconsistent - Conflict resolution and data loss
• Fast
Sharded Database
• Multiple masters; each document/partition has a single
master
• Multiple masters can be deployed in various data centers
• Consistent
• Fast

54. Writes: Multi-Data Center
3 Alternatives:
Two-phase commit across data centers
• Consistent
• Slow
Multi master
• Each data item can be written to any data center
• Inconsistent - Conflict resolution and data loss
• Fast
Sharded Database
• Multiple masters; each document/partition has a single
master
• Multiple masters can be deployed in various data centers
• Consistent
• Fast

55. MongoDB Multi Data Center Architecture
Options
Active DR
Partitioned (Sharded) Database
“Multi-master”

56. Option 1: Active DR

57. Active DR

58. Multi Region

59. Multi Region
Tune ReadPreference, ReadConcern, WriteConcern, Causal Consistency

60. Option 2: Sharded Database

61. Routing Requests To Nearest Region
Request Router

62. Sharded Database (zone sharding)
Writes
•Each data center owns a partition of
the data
•External routing process routes
requests to the data center owning
the request’s data
•Application writes to local primary
•Occasional cross data writes may
occur if routing process isn’t perfect
Reads
•Read from local primary for
consistency
•Global data access using “nearest"
read preference. Eventually
consistent.

63. Zone Sharding
Request Router

64. Zone Sharding
Request Router

65. Zone Sharding
Request Router

66. Zone Sharding
Request Router

67. Location-Aware Data Distribution
Shard Key: {regionCode} {userId}
Zone Start End
West West, MinKey West, MaxKey
East East, MinKey East, MaxKey
∞…
East Zone
… … ∞ ∞… … ∞
West Zone
{
_id : ObjectID(“abc…”),
regionCode: “West”,
userId: 12345,
…
}

68. Read Global/Write Local

69. Option 3: Multi-Master Database

70. How to implement Multi-Master
1. Insert only
• Updates → Inserts
• Aggregate on read
2. Always write to local primary
3. All reads are scatter gather

71. Updates
db.carts.find({shopCartId: 1234})
db.carts.update(
{shopCartId: 1234},
{$push: {items: “socks12”}})
{
shopCartId: 1234,
items: [“shoe32”,
“coat43”]
}
ShoppingCart Collection

72. Insert only
db.carts.aggregate(
[{$match: {shopCartId: 1234,
op: “add”}},
{$group: {_id: “$shopCardId”,
items: {$push:
“$items”}}}]
db.carts.insert(
{shopCartId: 1234,
op: “add”,
items: “socks12”}
)
{
shopCartId: 1234,
op: “add”,
items: “shoe32”
}
{
shopCartId: 1234,
op: “add”,
items: “coat32”
}
CartOperations Collection

73. Multi-Master Writes (inserts)
Request Router
App Server assigned data center field

74. Mult-master zone sharding configuration
Shard Key: {dc} {shopCartId}
Zone Start End
Virginia Virginia, MinKey Virginia, MaxKey
Oregon Oregon, MinKey Oregon, MaxKey
∞…
Virginia Zone
… … ∞ ∞… … ∞
Oregon Zone
{
shopCartId: 1234,
dc: “Virginia”,
op: “add”,
items: “shoe32”
}

75. Insert only
db.carts.aggregate(
[{$match: {shopCartId: 1234,
op: “add”}},
{$group: {_id: “$shopCartId”,
items: {$push : “$items”}}}]
db.carts.insert(
{shopCartId: 1234,
dc: “Oregon”,
op: “add”,
items : “socks12”}
)
{
shopCartId: 1234,
dc: “Oregon”,
op: “add”,
items: “shoe32”
}
{
shopCartId: 1234,
dc: “Virginia”
op: “add”,
items: “coat32”
}
CartOperations Collection

76. Multi-Master Reads (scatter gather)

77. Multi-Data Center Approach Summary
•
•
•

78. Tune to application requirements
Performance
0 ms 1 sec
Durability
1 node 2 nodes 3 nodes n nodes
Consistency
eventual causal linearizable
Read
Availability primary
(with
secondary
reads)~100%
Write
Availability primary ~100%retryable
strong readConcern: majority