1. Sridhar Nanjundeswaran, 10gen
sridhar@10gen.com
@snanjund
October 26, 2012
1

2. • Introduction to Replica Sets
• Developing with Replica Sets
• Operational Considerations
• Behind the Curtain
2

3. Introduction to Replica Sets
Why?
What is it?
Configuration Options

4. • How many have faced node failures?
• How many have been woken to do fail overs?
• How many have experienced issues due to n/w
latency?
• Different uses for data
– Normal processing
– Simple analytics
4

5. Node 1 Node 2
Node 3
5

6. Node 1 Node 2
RS Init
Node 3
6

7. Node 1 Node 2
Heartbeat
Election
Node 3
7

8. Node 1 Node 2
Secondary Heartbeat Secondary
Node 3
Primary Replication
Replication
8

9. Primary Election
Node 1 Node 2
Secondary Heartbeat Secondary
Node 3
Primary
9

10. Replication
Node 1 Node 2
Secondary Heartbeat Primary
Node 3
Primary
10

11. Replication
Node 1 Node 2
Secondary Heartbeat Primary
Node 3
Recovery
11

12. Replication
Node 1 Node 2
Secondary Heartbeat Primary
Node 3
Secondary Replication
12

13. Node 1 Node 2
Secondary Heartbeat Arbiterr
Node 3
Primary
Replication
13

14. > conf = {
_id : "mySet",
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay : 3600}
]
}
> rs.initiate(conf)
14

15. > conf = {
_id : "mySet”, Primary DC
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay : 3600}
]
}
> rs.initiate(conf)
15

16. > conf = {
_id : "mySet”,
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2}, Secondary DC
{_id : 2, host : "C”}, Default priority = 1
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay : 3600}
]
}
> rs.initiate(conf)
16

17. > conf = {
_id : "mySet”,
Analytics node
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay : 3600}
]
}
> rs.initiate(conf)
17

18. > conf = {
_id : "mySet”,
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay : 3600}
]
}
Backup node
> rs.initiate(conf)
18

19. Developing with Replica Sets
Consistency
Write Preference
Read Preference

20. Application
Insert
Read
Primary v1
Secondary
20

21. Application 1
Insert
Read
Primary v1
Secondary
Read
Application 2
21

22. Application 1
Insert
Read
Primary v1
Secondary
Does not
Read
exist
Application 2
22

23. Application 1
Insert Read from primary is
Read always consistent
Primary v1
Secondary
Read
Application 2
23

24. Application 1
Insert
Read
Primary v1
Replicate
Secondary v1
Read
Application 2
24

25. Application 1
Insert
Read
Primary v1
Replicate
Secondary v1
Read Read
Application 2
25

26. Application 1
Insert
Read
Primary v1
Replicate
Secondary v1
Reads V1
Read Read
Application 2
26

27. Application 1
Insert Update
Read
Primary v1 v2
Replicate
Secondary v1
Read Read
Application 2
27

28. Application 1
Insert Update
Read Read
Primary v1 v2
Replicate
Secondary v1
Read Read
Application 2
28

29. Application 1
Insert Update
Read Read
Primary v1 v2
Replicate
Secondary v1
Read Read Read
Application 2
29

30. Application 1
Insert Update
Read Read
Primary v1 v2
Replicate
Secondary v1
Read Read Read
Application 2
30

31. Application 1
Insert Update
Read Read
Primary v1 v2
Replicate Replicate
Secondary v1 v2
Read Read Read
Application 2
31

32. Application 1
Insert Update
Read Read
Primary v1 v2
Replicate Replicate
Secondary v1 v2
Read Read Read Read
Application 2
32

33. Application 1
Insert Update
Read Read
Primary v1 v2
Replicate Replicate
Secondary v1 v2
Read Read Read Read
Application 2
33

34. • Fire and forget
• Wait for error
• Wait for journal sync
• Wait for replication
34

35. Driver Primary
write
apply in memory
35

36. Driver Primary
write
getLastError
apply in memory
36

37. Driver Primary
write
getLastError
apply in memory
j:true
Write to journal
37

38. Driver Primary Secondary
write
getLastError
apply in memory
W:majority
replicate
38

39. • Since 2.0.0
• Control over where data is written to
• Each member can have one or more tags e.g.
– Dc : "ny”
– dc: "ny", ip: "192.168", rack: "row3rk7”
• Replica set defines rules for where data resides
• Rules can change without changing app code
39

40. {
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}]
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "allDCs"})
40

41. Driver Primary (sf) Secondary (ny) Secondary
write (cloud)
getLastError apply in
memory
W:allDCs
replicate
replicate
41

42. • 5 modes (new in 2.2)
– PRIMARY(only) - Default
– PRIMARYPREFERRED
– SECONDARY (only)
– SECONDARYPREFERRED
– NEAREST
42

43. • Custom read preferences
• Control where you read from
– E.g. { "disk": "ssd", "use": "reporting" }
• Use in conjunction with standard read
preferences
– Except primary
43

44. Operational Considerations
Upgrade/Maintenance
Common Deployment Scenarios

45. • No downtime
• Rolling upgrade/maintenance
– Start with Secondary
– Primary last
45

46. • Single datacenter
• Single switch & power
Member 1 • Points of failure:
– Power
Member 2 – Network
– Datacenter
Member 3 – Two node failure
• Automatic recovery of
single node crash
46

47. • Multi datacenter
DC
Member 1
• DR node for safety
1 • Can’t do multi data
Member 2 center durable write
safely since only 1
node in distant DC
DC
Member 3
2
47

48. • Three data centers
DC • Can survive full data
Member 1
1
center loss
Member 2
• Can do w= { dc : 2 } to
guarantee write in 2
DC Member 3
2
data centers (with
Member 4
tags)
DC Member 5 -
3 DR
48

49. Behind the Curtain
Schema
Oplog

50. • Local DB (not replicated)
– system.replset
– oplog.rs
• Capped collection
• Idempotent version of operation stored
50

51. • Heartbeat every 2 seconds
– Times out in 10 seconds
• Missed heartbeat considered node down
51

52. > db.replsettest.insert({_id:1,value:1})
{ "ts" : Timestamp(1350539727000, 1), "h" :
NumberLong("6375186941486301201"), "op" : "i", "ns" :
"test.replsettest", "o" : { "_id" : 1, "value" : 1 } }
> db.replsettest.update({_id:1},{$inc:{value:10}})
{ "ts" : Timestamp(1350539786000, 1), "h" :
NumberLong("5484673652472424968"), "op" : "u", "ns" :
"test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "value" :
11 } } }
52

53. > db.replsettest.update({},{$set:{name : ”foo”}, false, true})
{ "ts" : Timestamp(1350540395000, 1), "h" : NumberLong("-
4727576249368135876"), "op" : "u", "ns" : "test.replsettest", "o2" : {
"_id" : 2 }, "o" : { "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 2), "h" : NumberLong("-
7292949613259260138"), "op" : "u", "ns" : "test.replsettest", "o2" : {
"_id" : 3 }, "o" : { "$set" : { "name" : "foo" } } }
{ "ts" : Timestamp(1350540395000, 3), "h" : NumberLong("-
1888768148831990635"), "op" : "u", "ns" : "test.replsettest", "o2" : {
"_id" : 1 }, "o" : { "$set" : { "name" : "foo" } } }
53

54. • Full read preference support in mongos
– Drivers too
• Improved RS lag logging
• rs.syncFrom command
• buildIndexes setting
• replIndexPrefetch setting
54

55. • Use replica sets
• Easy to setup
– Try on a single machine
• Check doc page for RS tutorials
– http://docs.mongodb.org/manual/replication/#t
utorials
55

56. download at mongodb.org
We’re Hiring !
sridhar@10gen.com @snanjund
conferences, appearances, and meetups
http://www.10gen.com/events
Facebook | Twitter | LinkedIn
http://bit.ly/mongofb @mongodb http://linkd.in/joinmongo
© Copyright 2010 10gen Inc.
56