See the video here: https://www.youtube.com/watch?v=o6lSeNatVFM
A look at the elements required by Elasticsearch to turn a simple inverted index into an auto-clustering, horizontally scalable real time search and analytics engine. The talk will start from first principles, explaining how an inverted index works, how to make an inverted index suitable for real time search, how to scale that out, and how to add reliability and failover to the cluster.
Strategies for Landing an Oracle DBA Job as a Fresher
Â
Scaling real-time search and analytics with Elasticsearch
1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Clinton Gormley
@clintongormley
Scaling real time
search and analytics with
elasticsearch
2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch.org/guide
4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
⢠real-time
6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
⢠real-time
⢠distributed
7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
⢠real-time
⢠distributed
⢠search
8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
⢠real-time
⢠distributed
⢠search
⢠analytics
9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?
10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?
11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how does it work?
12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 1:
making text searchable
13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
where content like
â%darling%buds%â
15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
slow & inďŹexible
16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
 1 Doc
 2 Doc
 3
breathe
brings
buds
but
by
can
âŚ
damasked
darling
date
day
deaf
death
declines
delight
sorted list of
unique terms
18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
 1 Doc
 2 Doc
 3
breathe
brings
buds
but
by
can
âŚ
damasked
darling
date
day
deaf
death
declines
delight
where
they
occur
19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
 1 Doc
 2 Doc
 3
breathe
brings
buds
but
by
can
âŚ
damasked
darling
date
day
deaf
death
declines
delight
20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
 1 Doc
 2 Doc
 3
breathe
brings
buds
but
by
can
âŚ
damasked
darling
date
day
deaf
death
declines
delight
21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies  relevance
24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
⢠text length
Âť relevance
25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
⢠text length
Âť relevance
Âť doc weight
26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
⢠text length
⢠term positions
Âť relevance
Âť doc weight
27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
⢠text length
⢠term positions
Âť relevance
Âť doc weight
Âť word proximity
28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
⢠text length
⢠term positions
⢠char offsets
Âť relevance
Âť doc weight
Âť word proximity
29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
⢠term frequencies
⢠text length
⢠term positions
⢠char offsets
Âť relevance
Âť doc weight
Âť word proximity
Âť highlighting
30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
not just for text
31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
numbers, dates, bools, enums
geopoints, geoshapes, etc
32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 2:
analytics
33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for search
map values â doc_ids
34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for search
map values â doc_ids
for analytics
map doc_ids â values
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
cache values in memory
called âďŹelddataâ
37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
data access from RAM
very fast
38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-ďŹy analytics
in the context of
a userâs query
39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-ďŹy analytics
relevant analytics
for each user
40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
calculate metrics
count, min, max, sum, avg,
percentiles, cardinality,
stddev, variance, sum of squares
!
41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped by
popular terms, signiďŹcant terms,
ranges, dates, geolocation, etc
42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped by
groups can
⌠contain subgroups
⌠which contain subgroups
etc
43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 3:
building the inverted index
44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
⢠cache friendly
47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
⢠cache friendly
⢠reads from RAM
48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
⢠cache friendly
⢠reads from RAM
⢠fielddata never changes
49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
⢠cache friendly
⢠reads from RAM
⢠fielddata never changes
⢠compressible
50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
⢠cache friendly
⢠reads from RAM
⢠fielddata never changes
⢠compressible
⢠no locking
51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but, immutableâŚ
52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 4:
dynamic inverted index
53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
commit
segment
54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit
56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit
58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
⢠write new segment
61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
⢠write new segment
⢠write new commit point
62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
⢠write new segment
⢠write new commit point
⢠fsync
63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
⢠write new segment
⢠write new commit point
⢠fsync
⢠clear buffer
64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
⢠write new segment
⢠write new commit point
⢠fsync
⢠clear buffer
⢠reopen index
65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
⢠write new segment
⢠write new commit point
⢠fsync â expensive!
⢠clear buffer
⢠reopen index
66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 5:
near real-time search
67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
ďŹush
segment
68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
ďŹush
70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
ďŹush
72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
commit
74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene ďŹush
76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene ďŹush
⢠write new segment
⢠clear buffer
⢠reopen index
!
77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene ďŹush
⢠write new segment
⢠clear buffer
⢠reopen index
⢠no fsync
78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene ďŹush
⢠write new segment
⢠clear buffer
⢠reopen index
⢠no fsync â lightweight
79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
data not safe until fsyncâed!
80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:
donât lose data
81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:
donât lose data
â transaction log
82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
ďŹush
segment
translog
83. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
84. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
ďŹush
translog
85. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
86. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit
87. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit point
88. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch ârefreshâ
⢠lucene âflushâ
⢠makes changes searchable
⢠lightweight
!
89. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠lucene âcommitâ
⢠clears transaction log
⢠persists changes
⢠heavy
!
elasticsearch âďŹushâ
90. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
refresh every second
91. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!
92. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!
near real-time analytics!
93. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
94. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠slow searches
⢠poor term frequencies
⢠poor compression
!
!
too many segments
95. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 7:
reduce segments
96. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
97. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
98. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
99. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
100. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
101. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
102. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
103. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
104. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
105. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
106. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
107. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable
108. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
merge process
⢠many small â one big
⢠removes deleted docs
⢠runs in background
⢠throttled
109. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
110. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âAny wonder it broke downâ by Brian Snelson is licensed under CC BY 2.0
111. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
sometimes you
need another truck
112. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 8:
scale out, not up
113. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your data
114. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your data
transparent in elasticsearch
115. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
many segments
116. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments â
117. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
ssssssss
many shards
ss
â
118. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
one index
IIssssssss
many shards
ss
â
â
119. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
ânodeâ
running instance of elasticsearch
â one server
120. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âshardâ
bucket of data
lives on one node
physical worker unit
121. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âindexâ
logical namespace
points to one or more shards
122. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âindexâ
logical namespace
points to one or more shards
shard = hash(_id) % no_of_shards
123. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
PUT doc _id:1
hash(1) % 3 shard_2
node_A
shard_0
node_B
shard_1
node_C
shard_2
124. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET doc _id:2
hash(2) % 3 shard_0
node_A
shard_0
node_B
shard_1
node_C
shard_2
125. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Search all docs
shard = hash(_id) % no_of_shards
node_A
shard_0
node_B
shard_1
node_C
shard_2
126. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 9:
scaling elastically
127. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
start small
node_A
shard_0
shard_1
shard_2
128. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add more nodes
node_A
shard_0
shard_1
shard_2
node_B node_C
129. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shards migrate
node_A
shard_0
shard_1
shard_2
node_B
shard_1
node_C
shard_2
130. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalanced
node_A
shard_0
node_B
shard_1
node_C
shard_2
131. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add new index
node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
132. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
133. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
more hardware?
134. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
more hardware?
more hardware failure
135. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
at 3am on sundayâŚ
node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
136. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
boom!
node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2
137. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 10:
add redundancy
138. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for every shard
âŚmake a copy
139. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âprimary shardâ
main shard
140. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âreplica shard(s)â
copy of primary shard
141. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one node
node_A
P0
P1
P2
142. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
143. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
R0
R1
R2
144. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
redundancy
node_A
P0
P1
P2
node_B
R0
R1
R2
145. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
R0
R1
R2
node_C
146. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
R0
R1
R2
R1
node_C
P0
147. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalanced
node_A
P0
P1
P2
node_B
R0
R1
R2
node_C
P0
148. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lose a node
node_A
P0
P1
P2
node_B
R0
R2
R1
node_C
P0
149. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica primary
node_A
P0
P1
P2
node_B
R0
R2
150. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica primary
node_A
P0
P1
P2
node_B
P0
R2
151. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
allocate replicas
node_A
P0
P1
P2
node_B
P0
R2
R0
R1
152. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalanced
node_A
P0
P1
P2
node_B
P0
R2
R0
R1
153. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
primary shard
⢠just a role
⢠receives doc changes first
⢠forwards new doc to replicas in parallel
⢠number of primaries fixed
154. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica shard
⢠copy of primary shard
⢠serves read/search requests
⢠number of replicas can be changed
⢠more replicas â more read throughputâ¨
*if you have more hardware*
155. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
156. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
butâŚ
who controls all this?
157. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 11:
the master node
158. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âMaster Yodaâ by Gonzalo MartĂn is licensed under CC BY-SA 2.0
159. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
ânodeâ
running instance of elastic search
160. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
ânodeâ
running instance of elastic search
node_A
161. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âclusterâ
one or more nodes
with same cluster name
working together
162. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âclusterâ
node_A node_B
163. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a cluster
with multicast/unicast
164. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a cluster
with multicast/unicast
165. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routing
send request to any node
166. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routing
forwards to correct node
167. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?
168. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?
every node knows where
every document is
169. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
every node knows where
every document is
170. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
cluster level information
171. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
cluster level information
indices shards nodes
172. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
can only be updated by
the master node
173. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A
master node
elected when cluster forms
174. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
175. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
node_C
176. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
177. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
just a role
178. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
re-elected if master fails
179. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master node
node_A
re-elected if master fails
180. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master node
re-elected if master fails
181. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master node
only manages
cluster level changes
182. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master node
not doc-levelâ¨
get/put/search
183. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
the result?
184. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
distributed
real-time
search & analytics
185. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
which works in the same way
on your laptopâŚ
186. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
âŚas on your
1,000 node cluster
187. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
who is using it?
188. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠full text search
⢠highlighted search snippets
⢠search-as-you-type
⢠did-you-mean suggestions
189. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠combine visitor logs with â¨
social network data
⢠real-time feedback to editors
190. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠combines full text search with
geolocation
⢠uses more-like-this to ďŹnd â¨
related questions and answers
191. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠search repositories, users, â¨
issues, pull requests
⢠search 130 billion lines of code
⢠track all alerts, events, logs
192. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
⢠index and analyse â¨
5TB of log data every day
193. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
thank you
@clintongormley
elasticsearch.org/downloads
elasticsearch.com/support
elasticsearch.com/jobs