In this session we will cover wide area replica sets and using tags for backup. Attendees should be well versed in basic replication and familiar with concepts in the morning's basic replication talk. No beginner topics will be covered in this session

1. #MongoDBDays - @m4rcsch
Advanced Replication
Marc Schwering
Solutions Architect, 10gen

2. Roles & Configuration

3. Replica Set Roles

4. Configuration Options
> conf = {
_id : "mySet",
members : [
{_id : 0, host : "A"},
{_id : 1, host : "B"},
{_id : 2, host : "C", "arbiter" : true}
]
}
> rs.initiate(conf)

5. Simple Setup Demo

6. Behind the Curtain

7. Implementation details
• Heartbeat every 2 seconds
– Times out in 10 seconds
• Local DB (not replicated)
– system.replset
– oplog.rs
• Capped collection
• Idempotent version of operation stored

8. Op(erations) Log

9. Op(erations) Log is
idempotent
> db.replsettest.insert({_id:1,value:1})
{ "ts" : Timestamp(1350539727000, 1), "h" :
NumberLong("6375186941486301201"), "op" : "i", "ns" :
"test.replsettest", "o" : { "_id" : 1, "value" : 1 } }
> db.replsettest.update({_id:1},{$inc:{value:10}})
{ "ts" : Timestamp(1350539786000, 1), "h" :
NumberLong("5484673652472424968"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 1 },
"o" : { "$set" : { "value" : 11 } } }

10. oplog and multi-updates

11. Single operation can have
many entries
> db.replsettest.update({},{$set:{name : ”foo”}, false, true})
{ "ts" : Timestamp(1350540395000, 1), "h" : NumberLong("-
4727576249368135876"), "op" : "u", "ns" : "test.replsettest", "o2" : {
"_id" : 2 }, "o" : { "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 2), "h" : NumberLong("-
7292949613259260138"), "op" : "u", "ns" : "test.replsettest", "o2" : {
"_id" : 3 }, "o" : { "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 3), "h" : NumberLong("-
1888768148831990635"), "op" : "u", "ns" : "test.replsettest", "o2" : {
"_id" : 1 }, "o" : { "$set" : { "name" : "foo" } } }

12. Operations

13. Maintenance and Upgrade
• No downtime
• Rolling upgrade/maintenance
– Start with Secondary
– Primary last
– Commands:
• rs.stepDown(<secs>)
• db.version()
• db.serverBuildInfo()

14. Upgrade Demo

15. Replica Set – 1 Data Center
• Single datacenter
• Single switch & power
• Points of failure:
– Power
– Network
– Data center
– Two node failure
• Automatic recovery of
single node crash

16. Replica Set – 2 Data Centers
• Multi data center
• DR node for safety
• Can’t do multi data
center durable write
safely since only 1
node in distant DC

17. Replica Set – 2 Data Centers
• Analytics
• Disaster Recovery
• Batch Jobs
• Options
– low or zero priority
– hidden
– slaveDelay

18. Replica Set – 3 Data Centers
• Three data centers
• Can survive full data
center loss
• Can do w= { dc : 2 } to
guarantee write in 2
data centers (with tags)

19. Replica Set – 3+ Data Centers
Secondary
Primary
Secondary
Secondary
Secondary delayed
Secondary
Secondary

20. Commands
• Managing
– rs.conf()
– rs.initiate(<conf>) & rs.reconfig(<conf>)
– rs.add(host:<port>) & rs.addArb(host:<port>)
– rs.status()
– rs.stepDown(<secs>)
• Minority reconfig
– rs.reconfig( cfg, { force : true} )

21. Options
• Priorities
• Hidden
• Slave Delay
• Disable indexes (on secondaries)
• Default write concerns

22. Developing with Replica
Sets

23. Strong Consistency

24. Delayed Consistency

25. Write Concern
• Network acknowledgement
• Wait for error
• Wait for journal sync
• Wait for replication
– number
– majority
– Tags

26. Write Concern Demo

27. Datacenter awareness (Tagging)
• Control where data is written to, and read from
• Each member can have one or more tags
– tags: {dc: "ny"}
– tags: {dc: "ny", subnet: "192.168", rack:
"row3rk7"}
• Replica set defines rules for write concerns
• Rules can change without changing app code

28. Tagging Example
{
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}],
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "someDCs"})
> db.getLastErrorObj({"someDCs"})

29. Wait for Replication

30. Write Concern with timeout
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.getLastErrorObj({"allDCs"},100);
> db.getLastErrorObj({”someDCs"},500);
> db.getLastErrorObj(1,500);

31. Read Preference Modes
• 5 modes (new in 2.2)
– primary (only) - Default
– primaryPreferred
– secondary
– secondaryPreferred
– Nearest
When more than one node is possible, closest node is used
for reads (all modes but primary)

32. Tagged Read Preference
• Custom read preferences
• Control where you read from by (node) tags
– E.g. { "disk": "ssd", "use": "reporting" }
• Use in conjunction with standard read
preferences
– Except primary

33. Tags
{"dc.va": "rack1", disk:"ssd", ssd: "installed" }
{"dc.va": "rack2", disk:"raid"}
{"dc.gto": "rack1", disk:"ssd", ssd: "installed" }
{"dc.gto": "rack2", disk:"raid”}
> conf.settings = { getLastErrorModes: { MultipleDC : { "dc.va":
1, "dc.gto": 1}}
> conf.settings = {
"getLastErrorModes" : {
"ssd" : {
"ssd" : 1
},...

34. Tagged Read Preference
{ disk: "ssd" }
JAVA:
ReadPreference tagged_pref =
ReadPreference.secondaryPreferred(
new BasicDBObject("disk", "ssd")
);
DBObject result =
coll.findOne(query, null, tagged_pref);

35. Tagged Read Preference
• Grouping / Failover
{dc : "LON", loc : "EU"}
{dc : "FRA", loc : "EU"}
{dc : "NY", loc : "US”}
DBObject t1 = new BasicDBObject("dc", "LON");
DBObject t2 = new BasicDBObject("loc", "EU");
ReadPreference pref =
ReadPreference.primaryPreferred(t1, t2);

36. Conclusion

37. Best practices and tips
• Odd number of set members
• Read from the primary except for
– Geographically distribution
– Analytics (separate workload)
• Use logical names not IP Addresses in configs
• Set WriteConcern appropriately for what you are
doing
• Monitor secondaries for lag (Alerts in MMS)

38. #MongoDBDays - @m4rcsch
Thank You
Marc Schwering
Solutions Architect, 10gen