10 minutes talk about how Elasticsearch is working. It explains master node responsabilities, and how things work inside a shard. It gives good insight at how refresh, flush and optimize operation impact on elasticsear performance. It also explains how indexing and search work in this distributed database.

1. Elasticsearch
Search and Analyze data

2. NoSQL: Text Search and Document

3. Distributed

4. Distributed
Data Nodes

5. Distributed
Data Nodes
Shard

6. Distributed
Master Nodes

7. Distributed
Client Nodes

8. Master Node

9. Responsabilities
● Shard allocation
● Act on node joining
and leaving
● Broadcast cluster
state to every node

10. Shard allocation is based on:
● balancing shards accross all data nodes
● Disk availability on data node
● Fault tolerance
● Physical configuration (VM, rack, availability zone)
● Filters for node decomissioning

11. Discovery
● Master election
● Discovering Nodes

12. Zen Discovery
1. Uses Unicast
2. Try to contact hosts received by configuration
3. Retrieve cluster state
4. Send a Join to the master

13. Cloud Supported Discovery
● Azure
● AWS
● Google Cloud Computing

14. Master Election
● Any non client node can be configured to be master eligible
● Minimum master eligible nodes before election to prevent 2 masterss
● By default only master eligible and data nodes participate

15. Failure detection
● Master pings all nodes periodically
● All nodes ping master periodically
● Only master can update cluster state

16. Data Node

17. Distributed
P1 R2 R1 R3 P3 P3
R1 P1 P2 R3 R1 R3
R3 R2 R2 R2 R1 P2
Index 1
Index 2

18. Inside a shard: Segments

19. Inside a segment: Inverted index
Term Doc 1 Doc 2 Doc 3 Doc 4
brown X X X
fox X X X
quick X X

20. Inside a segment: Inverted index
Term Doc 1 Doc 2 Doc 3 Doc 4 Doc 5
brown X X X
fox X X X X
quick X X
write
READ

21. Lock would kill performance!

22. Immutable Segment
● Partial inverted index in segment
● Immutability make it cacheable
● Filters on top of segment become immutable and
cacheable

23. Immutable Segment
● Partial inverted index in segment
● Immutability make it cacheable
● Filters on top of segment become immutable and
cacheable
how do I update my index??

24. Indexing document
S1S2S3
DISK
Memory
S1S2S3
Commit point
In memory buffer
Translog

25. Refresh (every sec by default)
S1S2S3
DISK
Memory
S1S2S3
Commit point
In memory buffer
S4
Translog

26. Flush (every 5 secs)
S1S2S3
DISK
Memory
S1S2S3
Commit point
In memory buffer
S4
Translog
S4
fsync

27. Merge (background)
S1S2S3
Commit point
S4 S5
S1S2S3
DISK

28. Merge flushed to disk
S1
Commit point
S5
S1
DISK
S5

29. Search in segment
S5 S1
Diff
1

30. Search in segment
S5 S1
Diff
12

31. Indexing in the cluster

32. Indexing
R2 P2 R1 R2 P1
1
R1

33. Indexing
R2 P2 R1 R2 P1
1
2
R1

34. Indexing
R2 P2 R1 R2 P1
1
2
3
R1
3

35. Write scale by
adding shards

36. Searching in the cluster

37. Search
R2 P2 R1 R2 P1
1
R1

38. Search: Query Phase
R2 P2 R1 R2 P1
1
R1
22

39. Search: Query Phase
R2 P2 R1 R2 P1
1
R1
22
3 3
Return IDs
and Score

40. Search: Fetch Phase
R2 P2 R1 R2 P1
1
R1
22
3 3
Multi get

41. Search scale by
adding Replica

42. Thank you !