Elasticsearch is an open-source search and analytics engine that allows for searching both structured and unstructured data in (near) real-time. The document discusses how Elasticsearch uses Lucene's inverted index architecture under the hood and can be used as a plug-and-play replacement for other search engines. It then provides examples of how the company uses Elasticsearch for centralized logging, log monitoring, network monitoring, and generating comparison reports by modeling data as graphs in Elasticsearch.

1. “Where in the world is
Elastic”
Innovation Session
Data & Analytics Team
1

2. Try SQL???
SELECT *,IF(`discount`>10%,1,0)
`has_discounts`,
(3959 * acos ( cos ( radians(cur_lat) ) * cos(
radians( lat ) ) * cos( radians( lng ) - radians
(cur_lng) ) + sin ( radians(cur_lat) ) * sin( radians
( lat ) ) ) ) AS distance
FROM `Restaurants`
WHERE `type`=’Pizza’
AND `price` <= 1500
AND `distance` < 500
ORDER BY `WiFi` DESC,
`has_discounts` DESC
2
● Requirements
o Pizzas
o Under 1500
o Within 500m
● (Optional)
o Wifi
o Discount

3. What If
3
501m
1501
500m

4. ElasticSearch Approach
{
"bool": {
"must": {
"multi_match": {"query": "pizza", "fields":
["type^2", "restaurant"]},
},
"must_not": {},
"should": {
"term": {"features": "wifi"},
"range": {"discounts": {"gt": 10}}
}
}
}
4
● Requirements
o Pizzas
o Under 1500
o Within 500m
● (Optional)
o Wifi
o Discount

5. Via ElasticSearch
5
{
"gauss": {
"location": {
"origin": "<lat>,<lon>",
"offset": "0.5km",
"decay": 0.5
}
"price": {
"origin": 0,
"offset": 1500,
"decay": 0.5
}
}
}
501m
1501
500m

6. Outline
● Introducing ElasticSearch
● Naive comparison
● Lucene: The Architecture
● Plug & Play
● How we use ElasticSearch
● Summary
6

7. Introducing ElasticSearch
“Your data, your search”
7

8. ElasticSearch
● Open-Source Search & Analytics engine
o Structured & Unstructured Data
o (Near) Real Time
o Analytics capabilities (facets)
o REST based
● Distributed
o Designed for the Cloud
o Designed for Big Data
● Lightweight
● Popular: ~200K dl/month
8

9. Naïve Comparison
ElasticSearch vs. Solr.
9

10. 10
● Based on Lucene
● Full-text search
● Structured and Unstructured
● Queries, filters, caches,
Facets
● Cloud-ready
● Download size
● Zookeeper Vs ES own algorithm
● Release process (LGTM vs
Apache)
● Config vs Magic
Commonalities
Differences

11. Architecture behind
ElasticSearch
Inverted Index at Apache Lucene
11

12. Elasticsearch Storage Architecture
● Analysis process
12

13. Elasticsearch Storage Architecture
● Inverted Indexes
13

14. Plug & Play
14

15. ElasticSearch Cluster
15
Cluster State
Index Mappings
Shard Routing Tables
Nodes’ metadata
Node 1 Node 2 Node 3
Shard C
Shard A Shard B
Shard A
Shard C
Shard B
ElasticSearch
Index
Lucene Index
Segments
Inverted Index Replicas
Zone A Zone B

16. Index Request
16

17. Search Request
17

18. How we use
ElasticSearch
Use-cases at Leapset
18

19. 19
“Can you check the CouchDB
errors from last Monday
between 1.00 and 1.05 a.m?”
You gotta be
kidding me,
Ale!!!!

20. Centralized Logs
20

21. Log Monitor with Kibana
21 *GrayLog by LPG is also powered by Elasticsearch

22. “Can you detect the slow
queries on last Sunday
night?”
22

23. Network Monitoring - PacketBeat +
Kibana
● A packet sniffer that collects and
scans packets
● ElasticSearch
o As the storage backend
o As the index/search
backend
Up & Running at TST2 master DB
23

24. “Can you compare
restaurant sales within X
distance in a given hour?”
24

25. Graph Model
25

26. Inside story: Comparison Reports
Technology Stack
● Search Backend: Elasticsearch
o Vertex centric indices
▪ Sort & index edges per
vertex
o Enables efficient focused
traversals
▪ Only retrieve edges that
matter
o Uses pushdown predicates
for quick index driven
retrieval
26

27. Summary
27

28. 28