Video available at: https://www.youtube.com/watch?v=z4JTjUp3NC0 To scale the building of decision trees on large amounts of Indeed job search data, we created a system called Imhotep. In addition to being a crucial tool for building these machine learning models, Imhotep has proven to be applicable to many different analytics problems. The core of Imhotep is a distributed system that manages the parallel execution of queries across a set of time-sharded inverted indices. This talk covers Imhotep’s primitive operations that allow us to build decision trees, drill into data, build graphs, and even execute sql-like queries in IQL (Imhotep Query Language). We will also discuss what makes Imhotep fast, highly available, and fault tolerant.

1. go.indeed.com/IndeedEngTalks

2. Imhotep
Large Scale Analytics and
Machine Learning at Indeed

3. Jeff Plaisance
Engineering Manager

4. I help
people
get jobs.

5. Indeed is a
Search Engine for Jobs

7. Indeed is a data driven
organization

8. Indeed is a data driven
organization
Data driven organizations
need great tools

9. What does Imhotep allow you to do?
● Decision Tree Building
● Analytics

10. What does Imhotep allow you to do?
● Decision Tree Building
● Analytics

11. Indeed’s Analytics Philosophy
Analytics systems should be:
1. Interactive
2. Not Sampled
3. Not Approximate

12. Imhotep answers questions
What was the weekly average query time in the
last quarter from people doing the query
“software”?

13. Imhotep answers questions
What percent of jobsearch results pages are for
page 2 and beyond?

14. Imhotep answers questions
What are the 5 most common queries in each
country?

15. Total Job Searches From 2014-03-09
to 2014-03-23
?

17. Query

18. Query Location

19. Query Location
Impression

20. Document
query: “indeed software engineer”
location: “austin”
impressions: 10
clicks: 2
time: 2014-03-17T12:00:00

21. Shard
0 21 3 4
5 76 8 9
10 1211 13 14

22. Shard
0 21 3 4
5 76 8 9
10 1211 13 14

23. Server
2014/03/02 2014/03/09 2014/03/11
2014/03/12 2014/03/22 2014/03/24
Documents Documents Documents
Documents Documents Documents

24. Server
2014/03/02 2014/03/09 2014/03/11
2014/03/12 2014/03/22 2014/03/24
Documents Documents Documents
Documents Documents Documents

25. Cluster
2014-03-02
Server A
2014-03-03
Server B
2014-03-04
Server C

26. Cluster
2014-03-02 2014-03-03
Server B
2014-03-04
Server CServer A

27. Cluster
2014-03-02 2014-03-03
Server B
2014-03-04
Server C
Client
Session
Server A

28. Total Job Searches From 2014-03-09
to 2014-03-23
secret

29. Total Job Searches From 2014-03-09
to 2014-03-23 Per Day
2014-03-09 2014-03-16 2014-03-23

30. Metrics
● 64 bit integers
● Exactly one value per doc
● Random access by doc id

31. Metrics
● Time
● Clicks
● Impressions
● Revenue
● … or anything else that is a number

32. Groups
● Documents are placed into numbered
groups
● Every document starts in group 1
● Group 0 means “filtered out”

33. Groups
● Groups are stateful and scoped to a session
● Regroup operations update group for each
doc in shard

34. width
Metric Regroup
● Iterate over doc_id->metric lookup
● Set group to
(value - start)/ bucket_width
● Useful for making graphs (buckets on x-axis)
1 2 3 4 5
start end

35. Get Group Stats
● For each group, sums a metric for all docs in
that group

36. Bucket By Day
1. Regroup on time metric
2. Get Group Stats for count metric (always 1)

37. Total Job Searches From 2014-03-09
to 2014-03-23 Per Day
2014-03-09 2014-03-16 2014-03-23

38. Total and US Job Searches From
2014-03-09 to 2014-03-23 Per Day
2014-03-09 2014-03-16 2014-03-23

39. Inverted Indexes

40. Inverted Index
● Like index in the back of a book
● words = terms, page numbers = doc ids
● Term list is sorted
● Doc list for each term is sorted

41. doc id query country impressions clicks
0 software Canada 10 1
1 blank Canada 10 0
2 sales US 5 0
3 software US 8 1
4 blank US 10 1
Standard Index

42. Constructing an Inverted Index
query country impression clicks
doc id blank sales software Canada US 5 8 10 0 1
0 ✔ ✔ ✔ ✔
1 ✔ ✔ ✔ ✔
2 ✔ ✔ ✔ ✔
3 ✔ ✔ ✔ ✔
4 ✔ ✔ ✔ ✔

43. Constructing an Inverted Index
field term 0 1 2 3 4
query blank ✔ ✔
sales ✔
software ✔ ✔
country Canada ✔ ✔
US ✔ ✔ ✔
impressions 5 ✔
8 ✔
10 ✔ ✔ ✔
clicks 0 ✔ ✔
1 ✔ ✔ ✔

44. Inverted Index
field term doc list
query blank 1, 4
sales 2
software 0, 3
country Canada 0, 1
US 2, 3, 4
impressions 5 2
8 3
10 0, 1, 4
clicks 0 1, 2
1 0, 3, 4

45. Inverted Indexes
Allow you to:
● Quickly find all documents containing
a term
● Intersect several terms to perform
boolean queries

46. Lucene
● Open source inverted index implementation
● Reasonably fast
● Widely used, well tested

47. Global and US Job Searches From
2014-03-09 to 2014-03-23 Per Day
2014-03-09 2014-03-16 2014-03-23

48. field term doc list
query blank 1, 4
sales 2
software 0, 3
country Canada 0, 1
US 2, 3, 4
impressions 5 2
8 3
10 0, 1, 4
clicks 0 1, 2
1 0, 3, 4
Searches in the US only

49. field term doc list
query blank 1, 4
sales 2
software 0, 3
country Canada 0, 1
US 2, 3, 4
impressions 5 2
8 3
10 0, 1, 4
clicks 0 1, 2
1 0, 3, 4
Searches in the US only

50. Searches in the US only
field term doc list
country Canada 0, 1
US 2, 3, 4

51. Searches in the US only
Query Regroup
● Regroup all docs which do not match a
boolean query to group zero
field term doc list
country Canada 0, 1
US 2, 3, 4

52. Term Regroup
Splits docs in a group into one of two new
groups based on presence/absence of a term
country:US everything else
1
32

53. Multiterm Regroup
Generalization of term regroup to N terms and
N+1 new groups
country:US everything elsecountry:CA country:FR
52 3 4
1

54. Total and US Job Searches From
2014-03-09 to 2014-03-23 Per Day
2014-03-09 2014-03-16 2014-03-23

55. Inverted Index Compression
Size of Organic Dataset for last 5 months
● Original: 102 TB
● Inverted: 51 TB

56. Inverted Index Optimizations
● Compressed data structures
○ Better use of RAM and processor cache
○ Better use of memory bandwidth
○ Increased CPU usage and time
● Micro optimizations matter!

57. Delta / Varint Encoding
● Doc id lists are sorted
● Delta between a doc id and the previous doc
id is sufficient
● Deltas are usually small integers
● Use less bits for small integers and more bits
for large integers

58. Delta Encoding
field term doc list
query nursing 34, 86, 247, 301, 674, 714

59. Delta Encoding
field term doc list
query nursing 34, 86, 247, 301, 674, 714
34, 52, 161, 54, 373, 40

60. Small Integer Compression
● Golomb/Rice
● Varint
● Binary Packing
● PForDelta

61. Small Integer Compression
● Golomb/Rice
● Varint
● Bit Packing
● PForDelta

62. Varint Encoding
9838

63. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838

64. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838

65. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838

66. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838

67. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
? 1 1 0 1 1 1 0

68. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9838
? 1 1 0 1 1 1 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0

69. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
? 1 1 0 1 1 1 0

70. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0

71. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0

72. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0
? 1 0 0 1 1 0 0

73. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0
? 1 0 0 1 1 0 0

74. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0
? 1 0 0 1 1 0 0

75. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0
0 1 0 0 1 1 0 0

76. Varint Encoding
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1 0 1 1 1 0
9838
1 1 1 0 1 1 1 0
0 1 0 0 1 1 0 0

77. Inverted Index Compression
Size of Organic Dataset for last 5 months
● Original: 102 TB
● Inverted: 51 TB
● Delta / Varint: 17 TB

78. Flamdex
● Two files per field (terms/docs)
● Can add fields without rebuilding index
● Faster varint decoding
● No TF or positions (or wasted time decoding
them)

79. Varints
Pros:
● Compression
● Can fit more of index in RAM
● Higher information throughput per byte read
from disk

80. Varints
Cons:
● Decodes one byte at a time
● Lots of branch mispredictions
● Not fast to decode

81. Vectorized Varint Decoding
01001010 11001000 01110001 01001110
10011011 01101010 10110101 00010111
01110110 10001101 10110011 11000001

82. Vectorized Varint Decoding
01001010 11001000 01110001 01001110
10011011 01101010 10110101 00010111
01110110 10001101 10110011 11000001
pmovmskb: Extract top bit of each byte

83. Vectorized Varint Decoding
01001010 11001000 01110001 01001110
10011011 01101010 10110101 00010111
01110110 10001101 10110011 11000001
pmovmskb: Extract top bit of each byte
010010100111

84. Vectorized Varint Decoding
01001010 11001000 01110001 01001110
10011011 01101010 10110101 00010111
01110110 10001101 10110011 11000001
pmovmskb: Extract top bit of each byte
010010100111
Lookup in 4096 entry lookup table

85. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

86. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

87. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

88. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

89. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

90. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

91. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

92. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

93. 010010100111
Pattern of leading bits determines:
● how many varints to decode
● sizes and offsets of varints
● length of longest varint in bytes
● number of bytes to consume

94. 010010100111
Decoding options for:
● up to twelve 1 byte varints
● six 1-2 byte varints
● four 1-3 byte varints
● two 1-5 byte varints

95. Vectorized Varint Decoding
● Decode six 1-2 byte varints in parallel
● Need to pad out all 1 byte varints to 2 bytes
pshufb: Intel SSSE3 instruction to shuffle
bytes

96. Vectorized Varint Decoding
01001010 11001000 01110001 01001110
10011011 01101010 10110101 00010111
01110110 10001101 10110011 11000001
Decode 6 varints from 9 bytes

97. Vectorized Varint Decoding
01001010 11001000 01110001 01001110
10011011 01101010 10110101 00010111
01110110 10001101 10110011 11000001
Pad out 1 byte ints to 2 bytes

98. Vectorized Varint Decoding
01001010 00000000 11001000 01110001
01001110 00000000 10011011 01101010
10110101 00010111 01110110 00000000
Pad out 1 byte ints to 2 bytes

99. Vectorized Varint Decoding
01001010 00000000 11001000 01110001
01001110 00000000 10011011 01101010
10110101 00010111 01110110 00000000
Reverse bytes in 2 byte varints

100. Vectorized Varint Decoding
00000000 01001010 01110001 11001000
00000000 01001110 01101010 10011011
00010111 10110101 00000000 01110110
Reverse bytes in 2 byte varints

101. Vectorized Varint Decoding
00000000 01001010 01110001 11001000
00000000 01001110 01101010 10011011
00010111 10110101 00000000 01110110
Mask out leading purple 1’s

102. Vectorized Varint Decoding
00000000 01001010 01110001 01001000
00000000 01001110 01101010 00011011
00010111 00110101 00000000 01110110
Mask out leading purple 1’s

103. Vectorized Varint Decoding
00000000 01001010 01110001 01001000
00000000 01001110 01101010 00011011
00010111 00110101 00000000 01110110
Shift top bytes of each varint 1 bit right
(mask/shift/or)

104. Vectorized Varint Decoding
00000000 01001010 00111000 11001000
00000000 01001110 00110101 00011011
00001011 10110101 00000000 01110110
Shift top bytes of each varint 1 bit right
(mask/shift/or)

105. Vectorized Varint Decoding
00000000 01001010 00111000 11001000
00000000 01001110 00110101 00011011
00001011 10110101 00000000 01110110
● ~10 instructions
● No branches
● Less than 2 instructions per varint

106. Vectorized Varint Decoding
00000000 01001010 00111000 11001000
00000000 01001110 00110101 00011011
00001011 10110101 00000000 01110110
● Imhotep spends ~40% of its CPU time
decoding varints

107. Vectorized Varint Decoding
00000000 01001010 00111000 11001000
00000000 01001110 00110101 00011011
00001011 10110101 00000000 01110110
● Imhotep spends ~40% of its CPU time
decoding varints
● Vectorized decoder ~3-5x faster
○ Decompresses at 1.5 GB per second
○ ~2x overall system performance

108. Top 5 Locations

109. Term Stats
atlanta 49
austin 14
boston 25
chicago 28
dallas 13
houston 36
new york 68
san francisco 54

110. Term Stats Iterator
● For each term in a field, sum metrics across
all docs containing that term

111. Term Stats Iterator
● For each term in a field, sum metrics across
all docs containing that term
● How do we compute this across many
machines?

112. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

113. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

114. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

115. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

116. dallas 5
boston 12
austin 3
atlanta 16
chicago 9
boston 13
austin 7
atlanta 21
atlanta 49
dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

117. atlanta 49
dallas 5
boston 12
austin 3
atlanta 16
chicago 9
boston 13
austin 7
atlanta 21
dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

118. dallas 5
boston 12
austin 3
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21
atlanta 49atlanta 49

119. dallas 5
boston 12
austin 3
dallas 8
chicago 19
austin 4
chicago 9
boston 13
austin 7
atlanta 21
atlanta 49atlanta 49

120. chicago 9
boston 13
austin 7
atlanta 49atlanta 49
dallas 5
boston 12
austin 3
dallas 8
chicago 19
austin 4

121. austin 14
atlanta 49
chicago 9
boston 13
austin 7
dallas 5
boston 12
austin 3
dallas 8
chicago 19
austin 4

122. austin 14
atlanta 49
chicago 9
boston 13
austin 7
dallas 5
boston 12
austin 3
dallas 8
chicago 19
austin 4

123. dallas 5
boston 12
austin 14
atlanta 49
chicago 9
boston 13
austin 7
dallas 8
chicago 19
austin 4

124. dallas 8
chicago 19
dallas 5
boston 12
austin 14
atlanta 49
chicago 9
boston 13
austin 7

125. chicago 9
boston 13
dallas 8
chicago 19
dallas 5
boston 12
austin 14
atlanta 49

126. chicago 9
boston 13
dallas 8
chicago 19
dallas 5
boston 12
boston 25
austin 14
atlanta 49

127. boston 25
austin 14
atlanta 49
chicago 9
boston 13
dallas 8
chicago 19
dallas 5
boston 12

128. dallas 5
boston 25
austin 14
atlanta 49
chicago 9
boston 13
dallas 8
chicago 19

129. chicago 9dallas 5
boston 25
austin 14
atlanta 49
dallas 8
chicago 19

130. chicago 9dallas 5
chicago 28
boston 25
austin 14
atlanta 49
dallas 8
chicago 19

131. chicago 28
boston 25
austin 14
atlanta 49
chicago 9dallas 5
dallas 8
chicago 19

132. dallas 8
chicago 28
boston 25
austin 14
atlanta 49
chicago 9dallas 5

133. dallas 8
chicago 28
boston 25
austin 14
atlanta 49
dallas 5

134. dallas 8
dallas 13
chicago 28
boston 25
austin 14
atlanta 49
dallas 5

135. dallas 5 dallas 8
dallas 13
chicago 28
boston 25
austin 14
atlanta 49

136. dallas 8
dallas 13
chicago 28
boston 25
austin 14
atlanta 49

137. dallas 13
chicago 28
boston 25
austin 14
atlanta 49

138. Term Stats
1-6
TS 1 TS 2 TS 3 TS 4 TS 5 TS 6

139. TS 1-6 TS 7-12 TS 13-18

140. TS 1-6 TS 7-12 TS 13-18
Term Stats 1-
18

141. Amdahl’s Law
● The speedup of a program using multiple
processors is limited by the time needed for
the sequential fraction of the program

142. Amdahl’s Law
● Sequential part of FTGS is last step in
merge
● Can we distribute some part of the final
merge?

143. Hash Partition + Interleave
● Send all stats for each unique term to the
same thread based on a hash of the term
● Interleave merged terms

144. TS 1-6 TS 7-12 TS 13-18
Term Stats 1-
18

145. Shard Distribution

146. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

147. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

148. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

149. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

150. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

151. dallas 5
boston 12
austin 3
atlanta 16
dallas 8
chicago 19
austin 4
atlanta 12
chicago 9
boston 13
austin 7
atlanta 21

152. dallas 5
boston 12
atlanta 16
dallas 8
atlanta 12
boston 13
atlanta 21

153. dallas 5
boston 12
atlanta 16
dallas 8
atlanta 12
boston 13
atlanta 21

154. dallas 5
boston 12
atlanta 16
dallas 8
atlanta 12
boston 13
atlanta 21
atlanta 49

155. dallas 5
boston 12 dallas 8 boston 13
boston 25
atlanta 49

156. dallas 5 dallas 8
dallas 13
boston 25
atlanta 49

157. dallas 13
boston 25
atlanta 49

158. dallas 13
boston 25
atlanta 49
chicago 28
austin 14

159. dallas 13
boston 25
atlanta 49
chicago 28
austin 14

160. dallas 13
boston 25
atlanta 49
chicago 28
austin 14

161. atlanta 49
dallas 13
boston 25
atlanta 49
chicago 28
austin 14

162. atlanta 49
dallas 13
boston 25
atlanta 49
chicago 28
austin 14

163. dallas 13
boston 25
atlanta 49
chicago 28
austin 14

164. austin 14
atlanta 49
dallas 13
boston 25
chicago 28
austin 14

165. austin 14
atlanta 49
dallas 13
boston 25
chicago 28
austin 14

166. chicago 28
dallas 13
boston 25
austin 14
atlanta 49

167. boston 25
austin 14
atlanta 49
chicago 28
dallas 13
boston 25

168. boston 25
austin 14
atlanta 49
chicago 28
dallas 13
boston 25

169. dallas 13
boston 25
austin 14
atlanta 49
chicago 28

170. chicago 28
boston 25
austin 14
atlanta 49
dallas 13 chicago 28

171. chicago 28
boston 25
austin 14
atlanta 49
dallas 13 chicago 28

172. chicago 28
boston 25
austin 14
atlanta 49
dallas 13

173. dallas 13
dallas 13
chicago 28
boston 25
austin 14
atlanta 49

174. dallas 13
dallas 13
chicago 28
boston 25
austin 14
atlanta 49

175. dallas 13
chicago 28
boston 25
austin 14
atlanta 49

176. Shard Distribution
● Lots of datasets for different event types
● Each dataset is split into one shard per
(hour/day)
● Each shard has 2 replicas for fault tolerance
● How do we assign shards to machines?

177. Shard Distribution Considerations
● Space
● Load
● Hot Spots
● Adding/Removing machines

178. Homogeneous vs. Heterogeneous
Systems
● Must decide how or if you will handle
heterogeneous hardware
● Cannot balance for both space and load on
heterogeneous hardware

179. 1 TB
3 TB
Homogeneous vs. Heterogeneous

180. Homogeneous vs. Heterogeneous
12 shards
50% capacity used
4 shards
50% capacity used

181. Homogeneous vs. Heterogeneous
12 shards
50% capacity used
4 shards
50% capacity used
read hotspot

182. Homogeneous vs. Heterogeneous
8 shards
33% capacity used
8 shards
100% capacity used
wasted space

183. Hot Spots
When accessing any subset of a dataset,
evenly spread the load across CPUs, drives,
network cards

184. Hot Spots
When accessing any subset of a dataset,
evenly spread the load across CPUs, drives,
network cards
This is hard

185. Hot Spots
Maybe random is good enough?

186. Hot Spots
Maybe random is good enough?
On average about 10% more data read from
the most loaded machine than the least

187. Two Choice Randomized Load
Balancing
● 2 replicas of each shard to choose from
● Greedily choose the machine that currently
has the least load from this client

188. Two Choice Randomized Load
Balancing
● 2 replicas of each shard to choose from
● Greedily choose the machine that currently
has the least load from this client
● On average about 1% more data read from
the most loaded machine than the least

189. Rendezvous Hashing
● Assignment of a shard to machines
determined only by the machines that exist
in the cluster
● Hash all pairs of shard ID and machine ID
and pick the largest two

190. Rendezvous Hashing
Shard ID: organic.2014-03-02T06:00:00
H(Shard ID + m1
) = 0.592624
H(Shard ID + m2
) = 0.294647
H(Shard ID + m3
) = 0.736681
H(Shard ID + m4
) = 0.647578
H(Shard ID + m5
) = 0.835598

191. Rendezvous Hashing
0
1
m5
m3
m4
m1
m2

192. Rendezvous Hashing
0
1
m5
m3
m4
m1
m2

193. Rendezvous Hashing
0
1
m5
m3
m4
m1
m2

194. Rendezvous Hashing
● No coordination required - deterministic
algorithm used to determine assignment
● No centralized storage for shard to machine
assignment

195. Rendezvous Hashing

196. Rendezvous Hashing

197. Rendezvous Hashing

198. Rendezvous Hashing

199. Rendezvous Hashing

200. Rendezvous Hashing

201. Rendezvous Hashing

202. Rendezvous Hashing

203. Rendezvous Hashing

204. Rendezvous Hashing

205. Expected max hash for a shard is
Rendezvous Hashing

206. Expected max hash for a shard is
Probability that new machine will get shard
Rendezvous Hashing

207. Imhotep answers questions
What was the weekly average query time in the
last quarter from people doing the query
“software”?

208. 1. Query Regroup on query:software
2. Metric Regroup on time, width 7 days
3. Get Group Stats on query time and count,
divide after summing

209. Ramses

210. Imhotep answers questions
What percent of jobsearch results pages are for
page 2 and beyond?

211. 1. Get Group Stats on count
2. Query Regroup on “-page:1”
3. Get Group Stats on count
4. Divide -page:1 count by total count

212. Ramses

213. Imhotep answers questions
What are the 5 most common queries in each
country?

214. 1. Multiterm Regroup on all values of country
2. Term Group Stats Iteration on query

215. IQL
select count()
from jobsearch
‘2014-01-01’
‘2014-03-26’
group by country, query[5]

216. IQL
select count()
from jobsearch
‘2014-01-01’
‘2014-03-26’
group by country, query[5]
Metrics

217. select count()
from jobsearch
‘2014-01-01’
‘2014-03-26’
group by country, query[5]
IQL
Dataset

218. select count()
from jobsearch
‘2014-01-01’
‘2014-03-26’
group by country, query[5]
IQL
Regroup

219. select count()
from jobsearch
‘2014-01-01’
‘2014-03-26’
group by country, query[5]
IQL
Term Group
Stats

220. Imhotep
Large Scale Analytics and Machine
Learning

221. Imhotep
Large Scale Analytics and Machine
Learning
● Varint Decoding:
High Performance Vector Instructions
● Stream Merging: Hash Partition +
Interleave
● Shard Distribution: Rendezvous Hashing

222. We’re Open Sourcing
Imhotep

223. How You Can Use Imhotep
Data Ingestion
● TSV Uploader
● Hadoop
Data Access
● Imhotep Primitives
● IQL

224. Next @IndeedEng Talk
Large Scale Interactive Analytics
with Imhotep
Tom Bergman, Product Manager
Zak Cocos, Manager of Marketing Sciences
April 30, 2014
http://engineering.indeed.com/talks

225. Q&A

226. More Questions?
David James

227. Next @IndeedEng Talk
Large Scale Interactive Analytics
with Imhotep
Tom Bergman, Product Manager
Zak Cocos, Manager of Marketing Sciences
April 30, 2014
http://engineering.indeed.com/talks