The talk covers how Elasticsearch, Lucene and to some extent search engines in general actually work under the hood. We'll start at the "bottom" (or close enough!) of the many abstraction levels, and gradually move upwards towards the user-visible layers, studying the various internal data structures and behaviors as we ascend. Elasticsearch provides APIs that are very easy to use, and it will get you started and take you far without much effort. However, to get the most of it, it helps to have some knowledge about the underlying algorithms and data structures. This understanding enables you to make full use of its substantial set of features such that you can improve your users search experiences, while at the same time keep your systems performant, reliable and updated in (near) real time.

1. Elasticsearch
from the Bottom Up
Njal Karevoll
njal@found.no
@nkvoll

2. Elasticsearch
from the Bottom Up
Njal Karevoll
njal@found.no
@nkvoll

3. Who?
Co-founder of Found AS - Hosted Elasticsearch: found.no
8+ years search, 3+ Elasticsearch
Herding hundreds of Elasticsearch clusters

4. Motivation
• Why isn't my search for *foo-bar* matching “foo-bar"?
• Why can adding more documents shrink the index?
• Why is Elasticsearch using so much memory?

32. Segments are immutable

33. Deletes?

34. Compress all the things!

35. Cache all the things!

76. Search by index terms
Text analysis gives us terms

77. Search by segment
Uses several data structures

78. Immutable segments

79. Shard == Lucene Index

80. Elasticsearch Index
abstracts Lucene Indexes

81. … across nodes in a cluster

82. Learn More!
found.no/foundation
Follow
@foundsays