Replication, Durability, and Disaster Recovery

Replication& Durability

@spf13 AKASteve Francia15+ yearsbuilding theinternet Father, husband, skateboarderChief Solutions Architect @responsible for drivers,integrations, web & docs

Agenda• Intro to replication• How MongoDB does Replication• Conﬁguring a ReplicaSet• Advanced Replication• Durability• High Availability Scenarios

Replication

Use cases

Use cases• High Availability (auto-failover)

Use cases• High Availability (auto-failover)• Read Scaling (extra copies to read from)

Use cases• High Availability (auto-failover)• Read Scaling (extra copies to read from)• Backups Delayed Copy (fat ﬁnger) • Online, Time (PiT) backups • Point in

Use cases• High Availability (auto-failover)• Read Scaling (extra copies to read from)• Backups Delayed Copy (fat ﬁnger) • Online, Time (PiT) backups • Point in• Use (hidden) replica for secondary workload • Analytics • Data-processingexternal systems • Integration with

Types of outage

Types of outagePlanned • Hardware upgrade • O/S or ﬁle-system tuning • Relocation of data to new ﬁle-system / storage • Software upgrade

Types of outagePlanned • Hardware upgrade • O/S or ﬁle-system tuning • Relocation of data to new ﬁle-system / storage • Software upgradeUnplanned • Hardware failure • Data center failure • Region outage • Human error • Application corruption

Replica Set features

Replica Set features• A cluster of N servers

Replica Set features• A cluster of N servers• Any (one) node can be primary

Replica Set features• A cluster of N servers• Any (one) node can be primary• Consensus election of primary

Replica Set features• A cluster of N servers• Any (one) node can be primary• Consensus election of primary• Automatic failover

Replica Set features• A cluster of N servers• Any (one) node can be primary• Consensus election of primary• Automatic failover• Automatic recovery

Replica Set features• A cluster of N servers• Any (one) node can be primary• Consensus election of primary• Automatic failover• Automatic recovery• All writes to primary

Replica Set features• A cluster of N servers• Any (one) node can be primary• Consensus election of primary• Automatic failover• Automatic recovery• All writes to primary• Reads can be to primary (default) or a secondary

How MongoDBReplication

How MongoDB Replication works Member 1 Member 3 Member 2• Set is made up of 2 or more nodes

How MongoDB Replication works Member 1 Member 3 Member 2 Primary• Election establishes the PRIMARY• Data replication from PRIMARY to

How MongoDB Replication works negotiate new master Member 1 Member 3 Member 2 DOWN• PRIMARY may fail• Automatic election of new PRIMARY if

How MongoDB Replication works Member 3 Member 1 Primary Member 2 DOWN• New PRIMARY elected• Replica Set re-established

How MongoDB Replication works Member 3 Member 1 Primary Member 2 Recovering• Automatic recovery

How MongoDB Replication works Member 3 Member 1 Primary Member 2• Replica Set re-established

How Is DataReplicated?

How Is Data Replicated? to the• Change operations are writtenoplog • The oplog is a capped collection (ﬁxed size) •Must have enough space to allow new secondaries to catch up (from scratch or from a backup) •Must have enough space to cope with any applicable slaveDelay

How Is Data Replicated? to the• Change operations are written oplog • The oplog is a capped collection (ﬁxed size) •Must have enough space to allow new secondaries to catch up (from scratch or from a backup) •Must have enough space to cope with any applicable slaveDelay• Secondaries query the primary’s oplog and apply what they ﬁnd • All replicas contain an oplog

Conﬁguringa ReplicaSet

Creating a Replica Set$ ./mongod --replSet <name>> cfg = { _id : "<name>", members : [ { _id : 0, host : "sf1.acme.com" }, { _id : 1, host : "sf2.acme.com" }, { _id : 2, host : "sf3.acme.com" } ]}> use admin> rs.initiate(cfg)

Managing a Replica Set

Managing a Replica Setrs.conf() Shell helper: get current conﬁguration

Managing a Replica Setrs.conf() Shell helper: get current conﬁgurationrs.initiate(<cfg>); Shell helper: initiate replica set

Managing a Replica Setrs.conf() Shell helper: get current configurationrs.initiate(<cfg>); Shell helper: initiate replica setrs.reconfig(<cfg>) Shell helper: reconfigure a replica set

Managing a Replica Setrs.conf() Shell helper: get current configurationrs.initiate(<cfg>); Shell helper: initiate replica setrs.reconfig(<cfg>) Shell helper: reconfigure a replica setrs.add("hostname:<port>") Shell helper: add a new member

Managing a Replica Setrs.conf() Shell helper: get current configurationrs.initiate(<cfg>); Shell helper: initiate replica setrs.reconfig(<cfg>) Shell helper: reconfigure a replica setrs.add("hostname:<port>") Shell helper: add a new memberrs.remove("hostname:<port>") Shell helper: remove a member

Managing a Replica Set

Managing a Replica Set rs.status() Reports status of the replica set from one nodes point of view

Managing a Replica Set rs.status() Reports status of the replica set from one nodes point of view rs.stepDown(<secs>) Request the primary to step down

Managing a Replica Set rs.status() Reports status of the replica set from one nodes point of view rs.stepDown(<secs>) Request the primary to step down rs.freeze(<secs>) Prevents any changes to the current replica set conﬁguration (primary/secondary status) Use during backups

AdvancedReplication

Lots of Features• Delayed• Hidden• Priorities• Tags

Slave Delay

Slave Delay• Lags behind master by conﬁgurable time delay

Slave Delay• Lags behind master by conﬁgurable time delay• Automatically hidden from clients

Slave Delay• Lags behind master by conﬁgurable time delay• Automatically hidden from clients• Protects against operator errors • Fat ﬁngering • Application corrupts data

Other member types

Other member types• Arbiters • Don’t store a copy of the data • Vote in elections • Used as a tie breaker

Other member types• Arbiters • Don’t store a copy of the data • Vote in elections • Used as a tie breaker• Hidden • Not reported in isMaster • Hidden from slaveOk reads

Priorities

Priorities• Priority: a number between 0 and 100• Used during an election: • Most up to date • Highest priority • Less than 10s behind failed Primary• Allows weighting of members during failover

Priorities - example A B C D E p:10 p:10 p:1 p:1 p:0

Priorities - example A B C D E p:10 p:10 p:1 p:1 p:0• Assuming all members are up to date

Priorities - example A B C D E p:10 p:10 p:1 p:1 p:0• Assuming all members are up to date• Members A or B will be chosen ﬁrst • Highest priority

Priorities - example A B C D E p:10 p:10 p:1 p:1 p:0• Assuming all members are up to date• Members A or B will be chosen ﬁrst • Highest priority• Members C or D will be chosen when: • A and B are unavailable • A and B are not up to date

Priorities - example A B C D E p:10 p:10 p:1 p:1 p:0• Assuming all members are up to date• Members A or B will be chosen ﬁrst • Highest priority• Members C or D will be chosen when: • A and B are unavailable • A and B are not up to date• Member E is never chosen • priority:0 means it cannot be elected

Durabilit y

Durability Options

Durability Options•Fire and forget

Durability Options•Fire and forget•Write Concern

Write Concern &If a write requires areturn tripWhat the return tripshould depend on

Write Concern

Write Concernw:the number of servers to replicate to (ormajority)

Write Concernw:the number of servers to replicate to (ormajority)wtimeout:timeout in ms waiting for replication

Write Concernw:the number of servers to replicate to (ormajority)wtimeout:timeout in ms waiting for replicationj:wait for journal sync

Write Concernw:the number of servers to replicate to (ormajority)wtimeout:timeout in ms waiting for replicationj:wait for journal synctags:ensure replication to n nodes of given tag

Fire and Forget Driver Primary write apply in memory•Operations are applied in memory•No waiting for persistence to disk•MongoDB clients do not block waiting to conﬁrm the operation completed

Wait for error Driver Primary write getLastError apply in memory•Operations are applied in memory•No waiting for persistence to disk•MongoDB clients do block waiting to conﬁrm the operation completed

Wait for journal sync Driver write Primary getLastError apply in memory j:true Write to journal•Operations are applied in memory•Wait for persistence to journal•MongoDB clients do block waiting to conﬁrm the operation completed

Wait for fsync Driver Primary write getLastError apply in memory fsync:true write to journal (if enabled) fsync•Operations are applied in memory•Wait for persistence to journal•Wait for persistence to disk•MongoDB clients do block waiting to conﬁrm the operation completed

Wait for replication Driver Primary Secondary write getLastError apply in memory w:2 replicate•Operations are applied in memory•No waiting for persistence to disk•Waiting for replication to n nodes•MongoDB clients do block waiting to conﬁrm the operation completed

Tagging• Control over where data is written to.• Each member can have one or more tags: tags: {dc: "stockholm"} tags: {dc: "stockholm", ip: "192.168", rack: "row3-rk7"}• Replica set defines rules for where data resides• Rules defined in RS config... can change without change application code

Tagging - example { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}

HighAvailabilityScenarios

Single Node • Downtime inevitable • If node crashes human intervention might be needed • Should absolutely run with journaling to prevent data loss /

Replica Set 1 • Single datacenterArbiter • Single switch & power • One node failure • Automatic recovery of single node crash • Points of failure: • Power • Network • Datacenter

Replica Set 2 • Single datacenterArbiter • Multiple power/network zones • Automatic recovery of single node crash • w=2 not viable as losing 1 node means no writes • Points of failure: • Datacenter • Two node failure

Replica Set 3 • Single datacenter • Multiple power/network zones • Automatic recovery of single node crash • w=2 viable as 2/3 online • Points of failure: • Datacenter • Two node failure

When disaster

Replica Set 4 • Multi datacenter • DR node for safety • Cant do multi data center durable write safely since only 1 node in distant DC

Replica Set 5 • Three data centers • Can survive full data center loss • Can do w= { dc : 2 } to guarantee write in 2 data centers

SetUse? Data Protection High Availability Notes size Must use --journal to protect X One No No against crashes On loss of one member, surviving Two Yes No member is read only On loss of one member, surviving Three Yes Yes - 1 failure two members can elect a new primary * On loss of two members, X Four Yes Yes - 1 failure* surviving two members are read only On loss of two members, surviving Five Yes Yes - 2 failures three members can elect a new primary Typical

http://spf13.com http://github.com/s @spf13Questions? download at mongodb.org We’re hiring!! Contact us at jobs@10gen.com

http://architects.dzone.com	993
http://spf13.com	208
http://understeer.hatenablog.com	205
http://java.dzone.com	114
http://blog.sambervalley.com	96
http://www.oschina.net	58
http://lanyrd.com	34
http://feeds.feedburner.com	23
http://www.dzone.com	2
http://us-w1.rockmelt.com	1

Replication, Durability, and Disaster Recovery

by Steve Francia on Jun 25, 2012

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 1,734

Statistics

Replication, Durability, and Disaster Recovery Presentation Transcript