Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1p8xJSS. Michael Nygard discusses several loopholes in the CAP theorem that can be used to engineer practical, real-world systems with desirable features. Filmed at qconlondon.com. Michael Nygard has written and co-authored several books, including "Release It!", "Beautiful Architecture", "97 Things Every Software Architect Should Know" and "Java Developer’s Reference". An expert in highly-available, highly-scalable commerce systems, Michael has been professional programmer and architect for over 15 years.

1. EXPLOITING LOOPHOLES
IN CAP
MichaelT. Nygard
Cognitect, Inc.

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/cap-loopholes

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. About CAP
!
a.k.a. Brewer’s Conjecture
!
a.k.a. Theorem that Shipped
1,000 Launches

5. “Brewer's conjecture and the
feasibility of consistent, available,
partition-tolerant web services.”
!
Seth Gilbert and Nancy Lynch.
SIGACT News 33, 2 (June 2002), 51-59. DOI=10.1145/564585.564601
http://doi.acm.org/10.1145/564585.564601

7. Consistency
Availability
Partition-Tolerance

8. Consistency
Availability
Partition-Tolerance
ChooseTwo

9. BEWARE BAD LOGIC
⊭¬C → A
C ∩ P → ¬A
A ∩ P → ¬C

10. CAP
Gödel’s IncompletenessTheorem
Heisenberg’s Uncertainty Principle

11. CAP
GIT
HUP

12. They’re a drag!

13. The Network

14. Asynchronous
Message-passing
Network

16. Consistency

17. n2
→ write 12
← ack
→ read
← 12
→ read
← 12
→ write 20
← ack

18. n2
write 12
read
read
write 20
n11
write 12
read
read
write 20
n17
write 12
write 9
read
write 20
×

19. Atomic
Linearizable

21. Availability

22. n2
→ write 12
← ack
→ read
← 12
→ read
→ write 20

24. Partitioning

25. G1 = {n1, n2, n3, n4, n5,
n6, n7, n16, n17,
n18, n19, n20}
G2 = {n8, n9, n10, n11,
n12, n13, n14,
n15}

26. G1 = {n1, n2, n3, n4, n5,
n6, n7, n16, n17,
n18, n19, n20}
G2 = {n8, n9, n10, n11,
n12, n13, n14,
n15}

28. Theorem

29. Shared atomic object
Network divided into {G1, G2}
All messages between G1 and G2 are lost
Asynchronous message-passing network

30. Suppose algorithm A meets all 3 of C,A, & P.

31. v0
time
write
v1
G1
α1

32. v0
time
read
←v0
G2
α2

33. v0
time
write
v1
G1
read
←v0
G2
α = α1 + α2

35. loophole
noun
!
1.A way of escaping a difficulty, especially an
omission or ambiguity in the wording of a
contract or law that provides a means of
evading compliance.
!
2.A small hole or slit in a wall, especially one
through which small arms may be fired.

36. Loophole 1

37. HQ9+

38. H Prints “Hello,World!”
Q Prints source text
9 Prints lyrics to 99
bottles+ Increments the register

39. Distributed HQ9+

40. H Prints “Hello,World!”
Q Prints source text
9 Prints lyrics to 99 bottles
+
Increments the distributed
register

42. Loophole 2

43. Write Once,
ImmutableThereafter

44. v0
time
G1
G2
read
←v0
read
←v0
read
←v0
read
←v0
read
←v0
read
←v0
read
←v0

45. A = v0
time
read A
←v0
G1
read B
←w0
G2
new
C, X
read B
←w0
read C
←X
read A
←v0
B = w0

46. A = v0
time
read A
←v0
G1
read B
←w0
G2
new
C, X
read B
←w0
read C
←X
read A
←v0
B = w0

47. A bit of trickery?

48. Loophole 3

49. An older definition
of consistency

50. The data base consists of entities which are
related in certain ways. These relationships
are best thought of as assertions about the
data.

51. Examples of such assertions are:
“Names is an index for
Telephone_numbers.”
“The value of Count_of_X gives the
number of employees in department X.”

52. The data base is said to be consistent if it
satisfies all its assertions. In some cases,
the data base must become temporarily
inconsistent in order to transform it to a
new consistent state.
From "Granularity of Locks and Degrees of Consistency in a
Shared Data Base",
J.N. Gray, R.A. Lorie, G.R. Putzolu, I.L.Traiger, 1976
From "Granularity of Locks and Degrees of Consistency in a
Shared Data Base",
J.N. Gray, R.A. Lorie, G.R. Putzolu, I.L.Traiger, 1976

53. Consistency is a predicate C on entities and
their values.The predicate is generally not
known to the system but is embodied in the
structure of the transactions.
From "Transactions and Consistency in Distributed
Database Systems",
I.L.Traiger, J.N. Gray, C.A. Galtieri, and B.G. Lindsay, 1982

54. Can this kind of consistency
be maintained in a
distributed system?

55. V = v0
time
G1
G2
readV
←v0
read X
←x1
readV
←v0
write
V, v1
write
X, x1
read X
←x1
X = x0
write
X, x1
write
V, v1

56. C
R
D
T
Commutative
Replicated
Data
Type

57. Loophole 4

58. Partition A: <Ca, Ga, a1, a2, …, an>
C Consistency predicate over a
G Subset of nodes in network
a Value of variable i

59. Partition A: <Ca, Ga, a1, a2, …, an>
C Consistency predicate over a
G Subset of nodes in network
a Value of variable i

60. Partition A: <Ca, Ga, a1, a2, …, an>
Partition B: <Cb, Gb, b1, b2, …, bm>

61. Ga Gb
WAN
LOHRs
LOHRs

62. Loophole 5

63. Bounded Consistency

64. Core Nebula

65. RDBMS
Memcached

66. Item Display
A & P
Heavy caching
Bid History
C & P
Strong consistency

67. Loophole 6

68. Stop building
distributed systems

69. Loophole 7

70. Get a better network!

71. Asynchronous message passing
!
That’s UDP!

72. Semi-synchronous network
Lost messages are detected after time t
(by a missed acknowledgement)

73. “Delayed-t Consistency”
A partial ordering P orders all writes, and all reads
with respect to writes.
!
The value of every read is the one written by the
previous write, where “previous” is under P.
!
The order in P is consistent with the order of read
and write requests at each node.
!
If all messages are delivered and an operation θ

74. “Delayed-t Consistency”

75. “Eventual Consistency”

76. Loophole 7
×

77. Loophole 8
Use the Force

78. Relativistic Quantum FieldTheory

80. GPS

81. Loophole 9

82. Redefine availability

83. Normal
Operation
Partition
Detected
Query Available Available
Alter Available
Not
available

84. ASYMMETRY OFTIME
Send
Request

85. Send
Request
100 ms 200 ms
ASYMMETRY OFTIME

86. Send
Request
100 ms 200 ms 300 ms 400 ms 500 ms 600 ms
ASYMMETRY OFTIME

87. Send
Request
100 ms 200 ms 300 ms 400 ms 500 ms 600 ms
Time
Out
ASYMMETRY OFTIME

88. 9000 100 200 300 400 500 600 700 800
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Time Elapsed (ms)
ConfidenceofResponsebeforeTimeout

89. Send
Request
100 ms 200 ms 300 ms 400 ms 500 ms 600 ms
Time
Out
Response
Arrives
ASYMMETRY OFTIME

90. To the observer, there is no
difference between “too
slow” and “not there”.

91. P A C E L C

92. P
A
C
E
L
C

93. Partition?
Availability
Consistency
L
C
Yes
vs
A
C

94. Partition?
Availability
Consistency
Latency
Consistency
Yes No
vs vs
L
C

95. Loophole 10

96. OBSERVABLE CONSISTENCY

97. Porky Pig’s Window Shade
If Porky Pig is looking at the window
shade, it will be down.
If he is looking away from the window
shade, it will be up.

98. FIRST DIMENSION
X1 = {looking, not looking}

99. SECOND DIMENSION
X1 = {looking, not looking}
X2 = {shade open, shade closed}

100. FORBIDDEN STATES
X1 = {looking, not looking}
X2 = {shade open, shade closed}

101. Back to “consistency” as a
predicate over the state space

102. time
t11 t12

103. Back to CAP

104. None of these make
CAP “untrue”
Some of them operate under
different assumptions.

105. Some of them are totally
impractical.
Some of them are in
production today.

106. Finally, I’ll close with this bit of
code:
QHH9Q+++

107. mtnygard@cognitect.com
@mtnygard
MichaelT. Nygard
Cognitect, Inc.

108. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/cap-
loopholes