The document outlines the design and goals of an advanced replication system focusing on high availability, consistent data, automatic failover, and dynamic read capabilities across multi-region deployments. It details operational aspects including node redundancy, record operations, election processes for primary nodes, and client discovery methods. Additionally, the document highlights replication metrics and exceptional conditions that need to be addressed for maintaining system integrity and performance.

1. Advanced
Replication
Internals

2. Design and Goals
Goals
■ Highly Available
■ Consistent Data
■ Automatic Failover
■ Multi-Region/DC
■ Dynamic Reads
Design
● All DBs, each node
● Quorum/Election
● Smart clients
● Source selection
● Read Preferences
● Record operations
● Asynchronous
● Write/Replication
acknowledgements

3. Goal: High Availability
● Node Redundancy: Duplicate Data
● Record Write Operations
● Apply Write Operations
● Use capped collection called "oplog"

4. Replication Operations
Insert
● oplog entry (fields):
○ op, o
{
"ns" : "test.gamma",
"op" : "i", "v" : 2,
"ts" : Timestamp(1350504342, 5),
"o" : { "_id" : 2, "x" : "hi"} }

5. Replication Operations
Update
● oplog entry (fields):
○ o = update, o2 = query
{
"ns" : "test.tags",
"op" : "u", "v" : 2,
"ts": Timestamp(1368049619, 1),
"o2" : { "_id" : 1 },
"o" : { "$set" : { "tags.4" : "e" } } }

6. Operation Transformation
● Idempotent (update by _id)
● Multi-update/delete (results in many ops)
● Array modifications (replacement)

7. Interchangeable
● All members maintain oplog + dbs
● All able to take over, or be used for same
functions

8. Replication Process
● Record oplog entry on write
● Idempotent entries
● Pulled by replicas
1. Read over network
2. Buffer locally
3. Apply in batch
4. Repeat

9. Read + Apply Decoupled
● Background oplog reader thread
● Pool of oplog applier threads (by collection)
Repl Source
Applier
Thread
Pool
16
Buffer
DB4
DB3
DB1 DB2
Local Oplog
Network
Batch
Com
plete

10. Replication Metrics
"network": {
"bytes": 103830503805,
"readersCreated": 2248,
"getmores": {
"totalMillis": 257461206,
"num": 2152267 },
"ops": 7285440 }
"buffer": {
"sizeBytes": 0,
"maxSizeBytes": 268435456,
"count": 0},
"preload": { "docs": {
"totalMillis":0,"num":0},
"indexes": {
"totalMillis": 23142318,
"num": 14560667 } },
"apply": {
"batches": {
"totalMillis": 231847,
"num": 1797105},
"ops": 7285440 },
"oplog": {
"insertBytes": 106866610253,
"insert": {
"totalMillis": 1756725,
"num": 7285440 } }

11. Good Replication States
● Initial Sync
○ Record oplog start position
○ Clone/copy all dbs
○ Set minvalid, apply oplog since start
○ Build indexes
● Replication Batch: MinValid

12. Goal: Consistent Data
● Single Master
● Quorum (majority)
● Ordered Oplog

13. Consistent Data
Why a single master?

14. Election Events
Election events:
● Primary failure
● Stepdown (manual)
● Reconfigure
● Quorum loss

15. Election Nomination
Disqualifications
A replica will nominate itself unless:
● Priority:0 or arbiter
● Not freshest
● Just stepped down (in unelectable state)
● Would be vetoed by anyone because
○ There is a Primary already
○ They don't have us in their config
○ Higher priority member out there
● Higher config version out there

16. The Election
Nomination:
● If it looks like a tie, sleep random time
(unless first node)
Voting:
● If all goes well, only one nominee
● All voting members vote for one nominee
● Majority of votes wins

17. Goal: Automatic Failover
● Single Master
● Smart Clients
● Discovery

18. Discovery
isMaster command:
setName: <name>,
ismaster: true, secondary: false, arbiterOnly:
hosts: [ <visible nodes> ],
passives: [ <prio:0 nodes> ],
arbiters: [ <nodes> ],
primary: <active primary>,
tags: {<tags>},
me: <me>

19. Failover Scenario
Client
P
S
S
Discovery (isMaster)Active Primary

20. Failover Scenario
Client
P
S
S
Active Primary
P
Failed Primary

21. Failover Scenario
Client
Failed
P
S
Discovery (isMaster)
Active Primary

22. Replication Source Select'n
● Select closest source
○ Limit to non-hidden or slave delayed
○ If nothing, try again with hidden/slave delayed
○ Select node with fastest "ping" time
○ Must be fresher
● Choose source when
○ Starting
○ Any error with existing source (network, query)
○ Any member is 30s ahead of current source
● Manual override
○ replSetSyncSource -- good until we choose again

23. Goal: Datacenter Aware
● Dynamic replication topologies
● Beachhead data center server
P

24. Goal: Dynamic Reads
Controls for consistency
● Default to Primary
● Non-primary allowed
● Based on
○ Locality (ping/tags)
○ Tags
Client
S
P
S
Tags: A,
B
Tags: B, C

25. Asynchronous Replication
● Important considerations
● Additional requirements
● System/Application controls

26. Write Propagation
● Write Concern
● Replication requirements
● Timing
● Dynamic requirements

27. Exceptional Conditions
● Multiple Primaries
● Rollback
● Too stale

28. Design and Goals
Goals
■ Highly Available
■ Consistent Data
■ Automatic Failover
■ Multi-Region/DC
■ Dynamic Reads
Design
● All DBs, each node
● Quorum/Election
● Smart clients
● Source selection
● Read Preferences
● Record operations
● Asynchronous
● Write/Replication
acknowledgements

29. Thanks
Questions?