The document discusses distributed systems, highlighting their benefits such as scalability and redundancy, as well as challenges like crashes and message loss. It emphasizes the CAP theorem, which states that a distributed system can only guarantee two of three properties: consistency, availability, and partition tolerance. The presentation provides insights on managing writes and reads, including error handling, data consistency, and application-specific considerations.

1. David Golden YAPC::NA 2016

2. What is a
distributed system?

3. Data processing spread
over time & space

4. Node A
(primary)
Node B
(secondary)
Node C
(secondary)
RWRO
‘Replica Set’

5. DB
RW
memcached
RW
memcached

6. Node A
Node B Node C
RWRW
etcd
RW

7. Why use a
distributed system?

8. Scale

9. Scale
Performance

10. Scale
Performance
Redundancy

11. How do you break a
distributed system?

14. Crash

15. Crash
Packet loss

16. Crash
Packet loss
Garbage collection

17. Crash
Packet loss
Garbage collection
Process swapped out

18. DB memcached

19. DB memcached
A
1
Update...

20. DB memcached
A
1
A
2
Update...

21. DB memcached
A
1
A
A
3
2
Update...

22. DB memcached
A A
Read...
A

23. How can that go wrong?

24. DB memcached
A
A
1
2

25. DB memcached
A
A
GC
PAUSE
1
2
3

26. DB memcached
A
A
GC
PAUSE
B
5
6
B
4
B

27. DB memcached
A
A
GC
PAUSE
B
B
B
7
GC
DONE

28. DB memcached
A
A
GC
PAUSE
B
B
B
7
GC
DONE
A
8

29. DB memcached
A
A
B
A
BA

30. DB + memcached is not
a ‘good’ system

31. What makes a good
distributed system?

33. CAPTheorem

34. Consistency
Availability
Partition tolerance
Pick two!

35. Appears to be a single-copy of the data to an
outside observer.
Weaker models exist, e.g.‘eventual consistency’.
Consistency

36. Node failures don’t prevent survivors from
operating.
Availability

37. Partition: network can lose arbitrarily many
messages from one node to another
Tolerant: other properties remain true
Partition tolerance

38. Can’t avoid partitions!
CP or AP only!

40. CAP → PAC/ELC

41. # pac/elc system design
if ( partition ) {
pick(“availability”, “consistency”)
}
else {
pick(“low latency”, “consistency”)
}

42. Still simplistic

43. Reads vs writes

44. Majority-side of a partition
Can write (appears consistent)
Can read (available)
Minority-side of a partition
Can’t write (not available)
Can read (available – but stale)

45. ‘Practical
Consistency’

46. Do I know when a write is committed?
How do I read only committed and/or
current data?

47. Thinking about writes...

48. Durability
Convergence
Error recovery

49. Do we know when
writes are durable?

50. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
Write goes to primary
A

51. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
Replicates to majority → committed
A

52. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Partition separates primary

53. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Write to primary can’t replicate
But do we find out?
B X

54. # write concern
MongoDB->connect( $url,
{ w => 1 }
);
MongoDB->connect( $url,
{ w => ‘majority’ }
);

55. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
OK!
{w => 1}

56. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
{w => ‘majority’}

57. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
{w => ‘majority’}

58. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
OK!
{w => ‘majority’}

59. How will the system
converge on recovery?

60. Node B
(secondary)
Node A
(primary)
Node C
(primary)
A
A
A
Old primary steps down
New primary elected
B

61. Node B
(secondary)
Node A
(secondary)
Node C
(primary)
A
A
A
New writes occur and replicate
B C
C

62. Node B
(secondary)
Node A
(secondary)
Node C
(primary)
A
A
A
Partition heals
Returning node rolls back history
B
C
CX

63. Node B
(secondary)
Node A
(secondary)
Node C
(primary)
A
A
A
Returning node catches up with primary
C
C
C

64. • Rollback
• Conflict records
• Conflict-free replicated data type (CRDT)
(e.g.“add to set”)

65. What do we do with a
write error?

66. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Write to primary can’t replicate
B X
{w => ‘majority’}

67. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Majority write concern will timeout with an error
Retry?
Ignore?
B X
Error!
{w => ‘majority’}

68. Answers are specific to
your application!

69. Thinking about reads...

70. Recency
Durability
Latency

71. Do we care if we read
the latest write?

72. Do we care if data we
read rolls back?

73. Trade recency for
durability

74. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Write to primary can’t replicate
B X

75. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Dirty read from partitioned primary
B X
B

76. # read concern (3.2+)
MongoDB->connect( $url,
{ read_concern_level => ‘local’ }
);
MongoDB->connect( $url,
{ read_concern_level => ‘majority’ }
);

77. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Without partition, majority read
concern lags replication
A
B
B
B
{read_concern_level => ‘majority’}

78. Trade recency for
latency

79. Round-trip time

80. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
RTT for each
data center
US-EASTUS-WEST
100ms100ms 10ms

81. # read preference
MongoDB->connect( $url,
{ read_pref_mode => ‘primary’ }
);
MongoDB->connect( $url,
{ read_pref_mode => ‘nearest’ }
);

82. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Primary write
starts replicating
US-EASTUS-WEST
B
B
B

83. Node B
(secondary)
Node A
(primary)
Node C
(secondary)
A
A
A
Meanwhile,
read from
nearest node
US-EASTUS-WEST
B
B
B
A
10ms

84. Still another ‘gotcha’...

85. Partition detection race!

86. Node B
(new primary)
Node A
(‘lame-duck’ primary)
Node C
(secondary)
A
A
A
Doesn’t know
about new primary
Hasn’t stepped down yet
Has discovered
new primary
Just elected primary

87. Node B
(new primary)
Node A
(‘lame-duck’ primary)
Node C
(secondary)
A
A
A
Writes can commit
via new primary
B
B
Doesn’t know
about new primary
Hasn’t stepped down yet

88. Node B
(new primary)
Node A
(‘lame-duck’ primary)
Node C
(secondary)
A
A
A
‘Lame-duck’ primary
returns old
committed data
A
B
B

89. What if this isn’t OK?

91. ‘Quorum read’

92. Read-via-write

93. # find_one_and_update (CAS*)
$mc = MongoDB->connect( $url,
{ w => ‘majority’ }
);
...
$doc = $coll->find_one_and_update(
{ _id => $id },
{ ‘$inc’ => { _dummy => 1 } },
);

94. Take aways...

95. CAP is simplistic
Reality is complex

96. Needs are
application specific

97. When writing, consider...
Durability
Convergence
Error recovery

98. When reading, consider...
Recency vs durability
Recency vs latency
Nuclear option

99. Questions?
Email: david@mongodb.com
Twitter/IRC: @xdg

100. Photo credits:
• 9:00 AM: Mitch Martinez https://www.youtube.com/watch?v=ScDYZXIcihc
• Camel race: Jason Mrachina https://www.flickr.com/photos/w4nd3rl0st/5944487713 by-nc-nd
• Partition: MarcVenezia - Own work Picture taken during a personal trip in Middle-East, CC BY-SA 3.0, https://
commons.wikimedia.org/w/index.php?curid=11469551
• Eric Brewer: ByVera de Kok - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?
curid=39817816
• Daniel Abadi: @daniel_abadi profile picture https://pbs.twimg.com/profile_images/254073578/head2.jpg
• Fireball: By Photo courtesy of National Nuclear Security Administration / Nevada Site Office -This image is
available from the National Nuclear Security Administration Nevada Site Office Photo Library under number
XX-34. Public Domain, https://commons.wikimedia.org/w/index.php?curid=190949