A (hopefully) accessible introduction to some of the key mathematical concepts that make distributed and streaming computation possible.

1. Monoids and Sketches
and CRDTs, oh my!
Kevin Scaldeferri
OSB 2016

2. How Do I Math
with Big Data?

3. This document and the information herein (including any information that may be incorporated by reference) is
provided for informational purposes only and should not be construed as an offer, commitment, promise or
obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code,
functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated
or disclosed without New Relic’s express written permission.
Such information may contain forward-looking statements within the meaning of federal securities laws. Any
statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals,
or other characterizations of future events is a forward-looking statement. These forward-looking statements can
often be identified as such because the context of the statement will include words such as “believes,” “anticipates,”
“expects” or words of similar import.
Actual results may differ materially from those expressed in these forward-looking statements, which speak only as
of the date hereof, and are subject to change at any time without notice. Existing and prospective investors,
customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on
this forward-looking information. The achievement or success of the matters covered by such forward-looking
statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial
risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or
achievements to differ materially from those expressed or implied in any forward-looking statement. Further
information on factors that could affect such forward-looking statements is included in the filings we make with the
SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations
website at ir.newrelic.com or the SEC’s website at www.sec.gov.
New Relic assumes no obligation and does not intend to update these forward-looking statements, except as
required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to
the information provided.

5. How?

6. Monoids and Sketches
and CRDTs, oh my!

7. Monoids
超音波システム研究所 / http://bit.ly/26bBTQ1 / CC BY 3.0

8. Wikipedia
A monoid is an algebraic
structure with a single
associative binary
operation and an
identity element.
http://bit.ly/1Wlrigv / CC0

9. It’s just a thing
you can “add”

10. interface Monoid[T] {
// (x + y) + z = x + (y + z)
T add(T x, T y);
// 0 + x = x = x + 0
T unit();
}

11. interface Monoid[T] {
// (x + y) + z = x + (y + z)
T add(T x, T y);
// 0 + x = x = x + 0
T unit();
}

12. interface Monoid[T] {
// (x + y) + z = x + (y + z)
T add(T x, T y);
// 0 + x = x = x + 0
T unit();
}

13. interface Monoid[T] {
// (x + y) + z = x + (y + z)
T add(T x, T y);
// 0 + x = x = x + 0
T unit();
}

14. interface Monoid[T] {
// (x + y) + z = x + (y + z)
T add(T x, T y);
// 0 + x = x = x + 0
T unit();
}

15. One data type can
have multiple monoids!

16. Operation Unit
Sum 0
Product 1
Max -∞
Min +∞

17. Live Demo!

18. More Monoids
Count Boolean And
Lists & String
Concatenation
Boolean Or
Set Union
Function
Composition

19. Tuple Monoids
Monoid[U] & Monoid[V]
➜
Monoid[(U,V)]

20. Derived Monoids
Count & Sum ➜ Average
Count & Sum & SumOfSquares ➜
StdDev

21. Sets don’t
scale
Dan Morgan / http://bit.ly/1UiFhGs / CC BY 2.0

22. Sketches
=
Monoids
+
Physics

23. Counting by Flipping Coins
HHT
T
T
HHHHHT
HT
T
HHT
HT
T
T
T
T
T
HT
T
T
T
T
T
HT

24. Unique Count by Hashing
0111101001
1110101100
0010010010
0100100011
1000111000
0100011011
1100100110
1111011011
0011100001
1001011100
1110100101
1001110101
1010111001
1011110111
0000101001
0100101001
0100110000
0011110100
1011011010
0010011011

25. Set
Cardinality
(uniqueCount)
≈
HyperLogLog
Aldo Schumann / http://bit.ly/1Yqzvme / public domain

26. Set
Membership

27. interface ExtensionalSet[T] {
Iterator[T] iterator()
}

28. interface IntensionalSet[T] {
boolean isMember(T t);
}

29. Intensional
Sets
≈
Bloom Filters

30. HashSet

31. A
HashSet

32. A
HashSet

33. A
HashSet

34. A
BHashSet

35. A
BHashSet

36. A B
HashSet

37. A B
CHashSet

38. A B
CHashSet

39. A B
C
Ohnoes!
HashSet

40. A B
C
HashSet

41. A B
C
D?HashSet

42. A B
C
D?HashSet

43. A B
C
D?
Nopes!
HashSet

44. A B
C
E?HashSet

45. A B
C
E?HashSet

46. A B
C
E?
Hmmm
HashSet

47. A B
C
E?==
HashSet

48. A B
C
E?==
Nope!
HashSet

49. BloomFilter
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

50. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ABloomFilter

51. 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0
ABloomFilter

52. 0 0 1 0 1 0 0 0 1 1 0 0 1 0 1 0
A BBloomFilter

53. 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0
A B CBloomFilter

54. 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0
A B C
D?
BloomFilter

55. 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0
A B C
D?
Nope!
BloomFilter

56. 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0
A B C
A?
BloomFilter

57. 0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0
A B C
A?
Yes*
BloomFilter

58. BloomFilter Monoid
0 0 1 0 1 0 1 0 1 1 0 0 1 0 1 0
0 1 1 0 0 0 0 1 0 1 0 0 0 0 0 1
0 1 1 0 1 0 1 1 1 1 0 0 1 0 1 1
+
=

59. Circling Back:
BloomFilters are a scalable
approximation to Sets

60. CountMinSketch

61. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CountMinSketch

62. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CountMinSketch

63. 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0
0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
CountMinSketch

64. 10 0 0 0 0 0 0 0 1 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0
0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0
BCountMinSketch

65. 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 2 0
0 0 1 0 2 0 0 0 0 0 0 0 0 0 0 0
B CCountMinSketch

66. 0 0 0 0 0 0 1 0 1 2 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 0
0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0
B CCountMinSketch

67. 0 0 0 0 0 0 1 0 1 2 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 0
0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0
B C
D?
CountMinSketch

68. 0 0 0 0 0 0 1 0 1 2 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 0
0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0
B C
D? Min(2,1,0) = 0
CountMinSketch

69. 0 0 0 0 0 0 1 0 1 2 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 0
0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0
B C
A?
CountMinSketch

70. 0 0 0 0 0 0 1 0 1 2 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0
A
0 0 0 0 0 0 0 0 0 0 0 0 1 0 3 0
0 0 2 0 2 0 0 0 0 0 0 0 0 0 0 0
B C
A? Min(2,2,3) = 2
CountMinSketch

71. CountMinSketch
Frequency of Occurrence

72. Funnels
% of users who do A, then B
Size(A ∪ B) ≈ HyperLogLog
Size(A ∩ B) / Size(A ∪ B) ≈
MinHash
pedrik / http://bit.ly/25WzP1H / CC BY 2.0

73. What About
Streaming Data?

74. Streaming is
Distributed-in-Time
Computation

75. What About
Mutable Data?

76. CRDTs

77. Conflict-Free
Replicated
Data
Types

78. Available,
Eventually Consistent
Data Structures

79. How Can Two
People Count?

80. 0
0
Shared Counter

81. 0
0
Shared Counter
(+5)
5
5

82. 0
0
Shared Counter
(+5)
5
5
(-4)
(-3)
1 -2
2 -2

83. 0
0
Op-based Counter
(+5)
5
5
(-4)
(-3)
1 -2
2 -2

84. 0
0
Op-based Counter
(+5)
5
5 10
Oops!

85. {}
{}
Naive Sets

86. {}
{}
Naive Sets
(+X)
{X}
(+X)
{X}
{X} {X}

87. {}
{}
Naive Sets
(+X)
{X}
(+X)
{X}
{X} {X}
(-X)
{}
{}

88. {}
{}
Naive Sets
(+X)
{X}
(+X)
{X}
{X} {X}
(-X)
{}
{}
Oops!

90. {}
{}
Observed-Remove Sets
(+Xa)
{Xa}
(+Xb)
{Xb}
{Xb} {XaXb}
(-Xa)
{}
{Xb}

91. 0
0
State-based Counter

92. 0
0
State-based Counter
(+5)
{a=5}=5
{a=5}=5

93. 0
0
{a=9}=9
State-based Counter
(+5) (+4)
(+3)
{a=5}=5
{a=5}=5 {a=5,b=3}=8 {a=9,b=3}=12
{a=9,b=3}=12

94. 0
0
{a=9}=9
State-based Counter
(+5) (+4)
{a=5}=5
???{a=9}=9

95. 0
0
Increment-only Counter
(+5) (+4)
{a=5}=5
{a=9}=9{a=9}=9
{a=9}=9

96. 0
0 {a=+5,-4}=1
{a=+5,-4}=1
PN Counter
(+5) (-4)
{a=+5}=5
{a=+8,-4}=4{a=+5,-4}=1
(+3)
{a=+8,-4}=4

97. 0
0 {a:2:1}=1
{a:2:1}=1
Versioned State
(+5) (-4)
{a:1:5}=5
{a:3:4}=4{a:2:1}=1
(+3)
{a:3:4}=4

98. Replace exactly-once,
in-order delivery
with an idempotent
merge strategy

99. Summing Up
Monoids allow computations to be done
across many machines and merged
Sketches allow approximate results when the
exact answers are computationally infeasible
CRDTs give an approach for mutable
distributed data

100. Thank You
kevin@scaldeferri.com
@kscaldef