This document provides an overview of Blueflood, an open source system for simple metrics processing. It describes key concepts such as metrics, locators, shards, granularity, slots, and how Blueflood ingests metrics data, condenses it through roll ups, and allows querying of metrics. The document outlines Blueflood's goals of providing fast graphs, supporting multiple tenants cheaply and maintainably, and its use of Cassandra for storage.

1. Blueflood
Simple Metrics Processing
Gary Dusbabek • Rackspace • Berlin Buzzwords 2013
Monday, June 3, 13

2. Outline
Description
Motivation
Concepts
Guts
Monday, June 3, 13

3. Monday, June 3, 13

4. First things first...
Monday, June 3, 13

5. First things first...
We are making this open source
Monday, June 3, 13

6. First things first...
We are making this open source
Soon
Monday, June 3, 13

7. Open Sores
Monday, June 3, 13

8. Open Soars
Monday, June 3, 13

9. What is Blueflood?
Monday, June 3, 13

10. Three things:
Monday, June 3, 13

11. Monday, June 3, 13

12. Ingest metrics
Monday, June 3, 13

13. Ingest metrics
Condense metrics
Monday, June 3, 13

14. Ingest metrics
Condense metrics
Query metrics
Monday, June 3, 13

15. Ingest signals
Condense signals
Query signals
Monday, June 3, 13

16. Written in Java
Monday, June 3, 13

17. Cassandra for data
Monday, June 3, 13

18. Motivation
Fast Graphs
Accept Multiple Tenants
Cheap(ish)
Maintainable
Monday, June 3, 13

19. Fast Graphs
Monday, June 3, 13

20. Primary use?
Monday, June 3, 13

21. Primary use?
Dashboards & Graphs
Monday, June 3, 13

22. Get data quickly!!!
Monday, June 3, 13

23. Monday, June 3, 13

24. Return as few
data points as
possible
Monday, June 3, 13

25. Return as few
data points as
possible
Precompute
when possible
Monday, June 3, 13

26. Monday, June 3, 13

27. Sweet Spot:
Monday, June 3, 13

28. Sweet Spot:
300-400 data points
Monday, June 3, 13

29. Sweet Spot:
300-400 data points
Can’t fit much more into a graph
Monday, June 3, 13

30. Support Multiple
Tenants
Monday, June 3, 13

31. Mainly custom
retention policies:
Different TTLs
across tenants
Monday, June 3, 13

32. Cheapish
Monday, June 3, 13

33. Avoid new hardware
Monday, June 3, 13

34. Avoid new hardware
Cassandra nodes use lots of
disk, but no CPU.
Monday, June 3, 13

35. Avoid new hardware
Cassandra nodes use lots of
disk, but no CPU.
Let’s use that CPU!
Monday, June 3, 13

36. MaintenanceMonday, June 3, 13

37. Graphs are
not our
primary
product.
Monday, June 3, 13

38. Graphs are
not our
primary
product.
This
cannot be
a burden.
Monday, June 3, 13

39. Require
little tuning
Monday, June 3, 13

40. Require
little tuning
Scales
horizontally.
Monday, June 3, 13

41. Three jobs
One code baseMonday, June 3, 13

42. Ingest
Roll Up
Query
Monday, June 3, 13

43. ConceptsMonday, June 3, 13

44. Metric
Monday, June 3, 13

45. Signal
Metric
Monday, June 3, 13

46. Has data
Metric
Monday, June 3, 13

47. Can have a type
Metric
Monday, June 3, 13

48. Can also
have units
Metric
Monday, June 3, 13

49. Is uniquely
identifiable
Metric
Monday, June 3, 13

50. Locator
Monday, June 3, 13

51. Uniquely identifies
a metric
Locator
Monday, June 3, 13

52. Treated opaquely
by the system
Locator
Monday, June 3, 13

53. You should embed
data in it
Locator
Monday, June 3, 13

54. Example:
123:web-0:disk0:bytes-free
Monday, June 3, 13

55. A metric exists as a row
a:b:c
Locator
Monday, June 3, 13

56. A metric exists as a row
t1 data
,
a:b:c
Locator
Monday, June 3, 13

57. A metric exists as a row
t1 data
,
t2 data
,
a:b:c
Locator
Monday, June 3, 13

58. A metric exists as a row
t1 data
,
t2 data
,
t3 data
,
a:b:c
Locator
Monday, June 3, 13

59. A metric exists as a row
t1 data
,
t2 data
,
t3 data
,
t4 data
,
a:b:c
Locator
Monday, June 3, 13

60. A metric exists as a row
t1 data
,
t2 data
,
t3 data
,
t4 data
,
t5a:b:c
Locator
Monday, June 3, 13

61. Shard
Monday, June 3, 13

62. Partitions the
metric space to
share rollup
responsibilities
Monday, June 3, 13

63. When set to N,
every metric
hashes to 0..N-1
Shard
Monday, June 3, 13

64. We use 128 shards
Shard
Monday, June 3, 13

65. Each node owns
one or more
shards
Shard
Monday, June 3, 13

66. A shard is owned
by one or more
nodes
Shard
Monday, June 3, 13

67. Has nothing to do
with query
Shard
Monday, June 3, 13

68. Very little to do
with data ingestion
Shard
Monday, June 3, 13

69. A single node actively rolls up a
shard
Monday, June 3, 13

70. Zookeeper manages this
Monday, June 3, 13

71. Zookeeper manages this
(we’d like this to go away)
Monday, June 3, 13

72. OK for Zookeeper to fail because
rollup operations are idempotent
Monday, June 3, 13

73. Granularity
(or Resolution)
Monday, June 3, 13

74. A way of dividing
up time
Granularity
Monday, June 3, 13

75. Time
Full . . . . . . . . . . . . . . . . . . . . . . . .
Data can arrived spaced evenly...
Monday, June 3, 13

76. Time
Full .. . . . . . .... . .. . . . .. . . . . .. . . . . .
... or not.
Monday, June 3, 13

77. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
.
Monday, June 3, 13

78. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. .
Monday, June 3, 13

79. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . .
Monday, June 3, 13

80. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . .
Monday, June 3, 13

81. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . .
Monday, June 3, 13

82. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . .
Monday, June 3, 13

83. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . .
Monday, June 3, 13

84. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . .
Monday, June 3, 13

85. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . .
Monday, June 3, 13

86. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . .
Monday, June 3, 13

87. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . .
Monday, June 3, 13

88. Time
Full
5min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . .
Monday, June 3, 13

89. Time
Full
5min
20min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . . .
.
Monday, June 3, 13

90. Time
Full
5min
20min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . . .
. .
Monday, June 3, 13

91. Time
Full
5min
20min
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . . .
. . .
Monday, June 3, 13

92. Time
Full
5min
20min
1h
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . . .
. . .
.
Monday, June 3, 13

93. Time
Full
5min
20min
1h
.. . . . . . .... . .. . . . .. . . . . .. . . . . .
. . . . . . . . . . . .
. . .
.
4h
24h
etc.
Can’t demonstrate
4h and 24h and
maintain scale
Monday, June 3, 13

94. Imagine a two week
period divided into
chunks, each 5m long
Slots
Monday, June 3, 13

95. 288 of them in a day
Slots
Monday, June 3, 13

96. 4032 in one two
week period
Slots
Monday, June 3, 13

97. Number them
0..4031
Slots
Monday, June 3, 13

98. They repeat every
two weeks
Slots
Monday, June 3, 13

99. 4029
Monday, June 3, 13

100. 4029
4030
Monday, June 3, 13

101. 4029
4030
4031
Monday, June 3, 13

102. 4029
4030
4031
0
Monday, June 3, 13

103. 4029
4030
4031
0
1
Monday, June 3, 13

104. Same concept for
other granularities
Slots
Monday, June 3, 13

105. Just fewer slots as
granularities become
coarser
Slots
Monday, June 3, 13

106. 5m = 4032 slots over two weeks
Monday, June 3, 13

107. 5m = 4032 slots over two weeks
20m = 1008 slots over two weeks
Monday, June 3, 13

108. 5m = 4032 slots over two weeks
20m = 1008 slots over two weeks
1h = 336 slots over two weeks
Monday, June 3, 13

109. 5m = 4032 slots over two weeks
20m = 1008 slots over two weeks
1h = 336 slots over two weeks
4h = 84 slots over two weeks
Monday, June 3, 13

110. 5m = 4032 slots over two weeks
20m = 1008 slots over two weeks
1h = 336 slots over two weeks
4h = 84 slots over two weeks
24h = 14 slots over two weeks (duh!)
Monday, June 3, 13

111. Every timestamp
hashes to one slot
Slots
Monday, June 3, 13

112. Can be used to update a
map indicating if a metric has
been seen over a time
period
Slots
Monday, June 3, 13

113. Just keep track of the
last slot during which
you saw it.
Slots
Monday, June 3, 13

114. Tracked in a column
family with a 48h TTL
Slots
Monday, June 3, 13

115. Metric is “forgotten”
after 48h
Slots
Monday, June 3, 13

116. Doesn’t get rolled up
any more
Slots
Monday, June 3, 13

117. IngestionMonday, June 3, 13

118. Metric has attributes
Monday, June 3, 13

119. Metric has attributes
locator (id)
Monday, June 3, 13

120. Metric has attributes
locator (id)
value
Monday, June 3, 13

121. Metric has attributes
locator (id)
value
collection time
Monday, June 3, 13

122. Metric has attributes
locator (id)
value
collection time
time to live (TTL)
Monday, June 3, 13

123. Metric has attributes
locator (id)
value
collection time
time to live (TTL)
type
Monday, June 3, 13

124. Metric has attributes
locator (id)
value
collection time
time to live (TTL)
type
unit
Monday, June 3, 13

125. Metric has attributes
locator (id)
value
collection time
time to live (TTL)
type
unit
Monday, June 3, 13

126. Metrics arrive somehow
Monday, June 3, 13

127. Metrics arrive somehow
Passe through transforms
Monday, June 3, 13

128. Metrics arrive somehow
Passe through transforms
(you can augment these)
Monday, June 3, 13

129. Metrics arrive somehow
Passe through transforms
(you can augment these)
Written to the full-
resolution database
Monday, June 3, 13

130. Metrics arrive somehow
Passe through transforms
(you can augment these)
Written to the full-
resolution database
Written to the discovery
database
Monday, June 3, 13

131. Metrics arrive somehow
Passe through transforms
(you can augment these)
Written to the full-
resolution database
Written to the discovery
database
(so we know what metrics are active for a given
shard)
Monday, June 3, 13

132. Metrics arrive somehow
Passe through transforms
(you can augment these)
Written to the full-
resolution database
Written to the discovery
database
(so we know what metrics are active for a given
shard)
Shard+slot state is updated
(marked dirty, so we know what time periods
need to be rolled up)
Monday, June 3, 13

133. Ingestion is designed to
be pluggable
Monday, June 3, 13

134. Ingestion is designed to
be pluggable
If you’re a coder you
can swap in other
things:
Monday, June 3, 13

135. Ingestion is designed to
be pluggable
If you’re a coder you
can swap in other
things:
Transports
Monday, June 3, 13

136. Ingestion is designed to
be pluggable
If you’re a coder you
can swap in other
things:
Transports
Transforms
Monday, June 3, 13

137. Ingestion is designed to
be pluggable
If you’re a coder you
can swap in other
things:
Transports
Transforms
Or just live with the
defaults
Monday, June 3, 13

138. Monday, June 3, 13

139. late data is ok
Monday, June 3, 13

140. late data is ok
tolerated unless later
than 24h
Monday, June 3, 13

141. Roll Ups
Monday, June 3, 13

142. roll up is scheduled when a slot has not
received data for 5 minutes
Monday, June 3, 13

143. roll up is scheduled when a slot has not
received data for 5 minutes
(usually because time has moved beyond it)
Monday, June 3, 13

144. roll up is scheduled when a slot has not
received data for 5 minutes
(usually because time has moved beyond it)
select out all locators updated during that slot
Monday, June 3, 13

145. roll up is scheduled when a slot has not
received data for 5 minutes
(usually because time has moved beyond it)
select out all locators updated during that slot
for each locator get all datapoints during the
range of that slot
Monday, June 3, 13

146. roll up is scheduled when a slot has not
received data for 5 minutes
(usually because time has moved beyond it)
select out all locators updated during that slot
for each locator get all datapoints during the
range of that slot
do maths
Monday, June 3, 13

147. roll up is scheduled when a slot has not
received data for 5 minutes
(usually because time has moved beyond it)
select out all locators updated during that slot
for each locator get all datapoints during the
range of that slot
do maths
save in coarser granularity
Monday, June 3, 13

148. roll up is scheduled when a slot has not
received data for 5 minutes
(usually because time has moved beyond it)
select out all locators updated during that slot
for each locator get all datapoints during the
range of that slot
do maths
save in coarser granularity
repeat per granularity
Monday, June 3, 13

149. Monday, June 3, 13

150. Smart scheduling
Monday, June 3, 13

151. Smart scheduling
Don’t rollup a 1h range if its 20m ranges have
not been computed.
Monday, June 3, 13

152. Monday, June 3, 13

153. Don’t want to get behind
Monday, June 3, 13

154. Don’t want to get behind
If you can’t process all rollups within 5 minutes,
you need more processing
Monday, June 3, 13

155. Query API
Monday, June 3, 13

156. GetByPoints(locator, from, to, numPoints)
Monday, June 3, 13

157. GetByPoints(locator, from, to, numPoints)
“Give me N data points”
Monday, June 3, 13

158. GetByPoints(locator, from, to, numPoints)
“Give me N data points”
Automatically chooses resolution for best fit
Monday, June 3, 13

159. GetByResolution(locator, start, stop, resolution)
Monday, June 3, 13

160. GetByResolution(locator, start, stop, resolution)
Most control
Monday, June 3, 13

161. GetByResolution(locator, start, stop, resolution)
Most control
Possibility of return more data than you need
Monday, June 3, 13

162. Query: Straight Cassandra reads
Helps with SLA across tenants
Monday, June 3, 13

163. @gdusbabek
Monday, June 3, 13

164. Image Credits
All images came from the Flickr Commons Collection
http://flickr.com/commons
chalk drawing http://www.flickr.com/photos/stevendepolo/4705141484
outline http://www.flickr.com/photos/adactio/3563013647
pencil http://www.flickr.com/photos/isox4/4841242881
sore http://www.flickr.com/photos/yortw/5436427109
soar http://www.flickr.com/photos/eyeno/6183027047
amber http://www.flickr.com/photos/mikaelmiettinen/4219852860
wall http://www.flickr.com/photos/vinothchandar/8093281752
truck http://www.flickr.com/photos/amalakar/8111811112
packages http://www.flickr.com/photos/cushinglibrary/3729414657
gummi bears http://www.flickr.com/photos/28misguidedsouls/5649609098
skyscraper http://www.flickr.com/photos/nathanmac87/5341060061
traffic light http://www.flickr.com/photos/emrank/2435273839
money http://www.flickr.com/photos/heyrocker/117059817
lightpost http://www.flickr.com/photos/jacreative/134129950
harpsicord http://www.flickr.com/photos/dalbera/2739071156
carboat http://www.flickr.com/photos/mbtrama/3826879277
pens http://www.flickr.com/photos/freddy-click-boy/3098136909/
river http://www.flickr.com/photos/makelessnoise/240072417/
map http://www.flickr.com/photos/bulle_de/4672972586
shard http://www.flickr.com/photos/pauljill/4964306570/
butterflies http://www.flickr.com/photos/webtreatsetc/5265396307/
camel http://www.flickr.com/photos/seattlemunicipalarchives/3797940791/
sand dunes http://www.flickr.com/photos/mikebaird/8517511072/
sprockets http://www.flickr.com/photos/mwichary/2665559632/
snake http://www.flickr.com/photos/mattpandor4/8254089051/
train http://www.flickr.com/photos/scjn/2554491487/sizes/o/
steps http://www.flickr.com/photos/borkurdotnet/363738205/
cinnamon roll http://www.flickr.com/photos/sbogdanich/8090569452/
question mark http://www.flickr.com/photos/bilal-kamoon/6835060992
thank you http://www.flickr.com/photos/nateone/3768979925/
Monday, June 3, 13