This is Elasticsearch crash course for Data Engineers at Sentifi AG

1. Elasticsearch
Crash Course for Data Engineers
Duy Do (@duydo)

2. About
● A Father, A Husband, A Software Engineer
● Founder of Vietnamese Elasticsearch Community
● Author of Vietnamese Elasticsearch Analysis Plugin
● Technical Consultant at Sentifi AG
● Co-Founder at Krom
● Follow me @duydo

3. Elasticsearch is Everywhere

5. What is Elasticsearch?

6. Elasticsearch is a distributed
search and analytics engine,
designed for horizontal
scalability with easy
management.

7. Basic Terms
● Cluster is a collection of nodes.
● Node is a single server, part of a
cluster.
● Index is a collection of shards ~
database.
● Shard is a collection of
documents.
● Type is a category/partition of an
index ~ table in database.
● Document is a Json object ~
record in database.

8. Distributed & Scalable

9. Shards & Replicas

10. One node, One shard
Node 1
employees
P0
PUT /employees
{
“settings”: {
“number_of_shards”: 1,
“number_of_replicas”: 0
}
}

11. Two nodes, One shard
Node 1
employees
P0
PUT /employees
{
“settings”: {
“number_of_shards”: 1,
“number_of_replicas”: 0
}
}
Node 2

12. One node, Two shards
Node 1
employees
P0
PUT /employees
{
“settings”: {
“number_of_shards”: 2,
“number_of_replicas”: 0
}
}
P1

13. Two Nodes, Two Shards
Node 1
employees
P0
PUT /employees
{
“settings”: {
“number_of_shards”: 2,
“number_of_replicas”: 0
}
}
Node 2
employees
P1P1

14. Two nodes, Two shards, One replica of each shard
Node 1
employees
P0
PUT /employees
{
“settings”: {
“number_of_shards”: 2,
“number_of_replicas”: 1
}
}
R1
Node 2
employees
P1 R0

15. Index Management

16. Create Index
PUT /employees
{
“settings”: {...},
“mappings”: {
“type_one”: {...},
“type_two”: {...}
},
“aliases”: {
“alias_one”: {...},
“alias_two”: {...}
}
}

17. Index Settings
PUT /employees/_settings
{
“number_of_replicas”: 1
}

18. Index Mappings
PUT /employees/_mappings
{
“employee”: {
“properties”: {
“name”: {“type”: “string”},
“gender”: {“type”: “string”, “index”: “not_analyzed”},
“email”: {“type”: “string”, “index”: “not_analyzed”},
“dob”: {“type”: “date”},
“country”: {“type”: “string”, “index”: “not_analyzed”},
“salary”: {“type”: “double”},
}
}
}

19. Delete Index
DELETE /employees

20. Put Data In, Get Data Out

21. Index a Document with ID
PUT /employees/employee/1
{
“name”: “Duy Do”,
“email”: “duy.do@sentifi.com”,
“dob”: “1984-06-20”,
“country”: “VN”
“gender”: “male”,
“salary”: 100.0
}

22. Index a Document without ID
POST /employees/employee/
{
“name”: “Duy Do”,
“email”: “duy.do@sentifi.com”,
“dob”: “1984-06-20”,
“country”: “VN”
“gender”: “male”,
“salary”: 100.0
}

23. Retrieve a Document
GET /employees/employee/1

24. Update a Document
POST /employees/employee/1/_update
{
“doc”:{
“salary”: 500.0
}
}

25. Delete a Document
DELETE /employees/employee/1

26. Searching

27. Structured Search
Date, Times, Numbers, Text
● Finding Exact Values
● Finding Multiple Exact Values
● Ranges
● Working with Null Values
● Combining Filters

28. Finding Exact Values
GET /employees/employee/_search
{
“query”: {
“term”: {
“country”: “VN”
}
}
}
SQL: SELECT * FROM employee WHERE country = ‘VN’;

29. Finding Multiple Exact Values
GET /employees/employee/_search
{
“query”: {
“terms”: {
“country”: [“VN”, “US”]
}
}
}
SQL: SELECT * FROM employee WHERE country = ‘VN’ OR country = ‘US’;

30. Ranges
GET /employees/employee/_search
{
“query”: {
“range”: {
“dob”: {“gt”: “1984-01-01”, “lt”: “2000-01-01”}
}
}
}
SQL: SELECT * FROM employee WHERE dob BETWEENS ‘1984-01-01’ AND ‘2000-01-01’;

31. Working with Null values
GET /employees/employee/_search
{
“query”: {
“filtered”: {
“filter”: {
“exists”: {“field”: “email”}
}
}
}
}
SELECT * FROM employee WHERE email IS NOT NULL;

32. Working with Null Values
GET /employees/employee/_search
{
“query”: {
“filtered”: {
“filter”: {
“missing”: {“field”: “email”}
}
}
}
}
SELECT * FROM employee WHERE email IS NULL;

33. Combining Filters
GET /employees/employee/_search
{
“query”: {
“filtered”: {
“filter”: {
“bool”: {
“must”:[{“exists”: {“field”: “email”}}],
“must_not”:[{“term”: {“gender”: “female”}}],
“should”:[{“terms”: {“country”: [“VN”, “US”]}}]
}
}
}
}
}

34. Combining Filters
SQL:
SELECT * FROM employee
WHERE email IS NOT NULL
AND gender != ‘female’
AND (country = ‘VN’ OR country = ‘US’);

35. More Queries
● Prefix
● Wildcard
● Regex
● Fuzzy
● Type
● Ids
● ...

36. Full-Text Search
Relevance, Analysis
● Match Query
● Combining Queries
● Boosting Query Clauses

37. Match Query - Single Word
GET /employees/employee/_search
{
“query”: {
“match”: {
“name”: {
“query”: “Duy”
}
}
}
}

38. Match Query - Multi Words
GET /employees/employee/_search
{
“query”: {
“match”: {
“name”: {
“query”: “Duy Do”,
“operator”: “and”
}
}
}
}

39. Combining Queries
GET /employees/employee/_search
{
“query”: {
“bool”: {
“must”:[{“match”: {“name”: “Do”}}],
“must_not”:[{“term”: {“gender”: “female”}}],
“should”:[{“terms”: {“country”: [“VN”, “US”]}}]
}
}
}

40. Boosting Query Clauses
GET /employees/employee/_search
{
“query”: {
“bool”: {
“must”:[{“term”: {“gender”: “female”}}], # default boost 1
“should”:[
{“term”: {“country”: {“query”:“VN”, “boost”:3}}} # the most
important
{“term”: {“country”: {“query”:“US”, “boost”:2}}} # important than
#1 but not as important as #2
],
}
}
}

41. More Queries
● Multi Match
● Common Terms
● Query Strings
● ...

42. Analytics

43. Aggregations
Analyze & Summarize
● How many needles in the
haystack?
● What is the average length of
the needles?
● What is the median length of
the needles, broken down by
manufacturer?
● How many needles are added
to the haystacks each month?
● What are the most popular
needle manufacturers?
● ...

44. Buckets & Metrics
SELECT COUNT(country) # a metric
FROM employee
GROUP BY country # a bucket
GET /employees/employee/_search
{
“aggs”: {
“by_country”: {
“terms”: {“field”: “country”}
}
}
}

45. Bucket is a collection of
documents that meet certain
criteria.

46. Metric is simple mathematical
operations such as: min, max,
mean, sum and avg.

47. Combination
Buckets & Metrics
● Partitions employees by
country (bucket)
● Then partitions each country
bucket by gender (bucket)
● Finally calculate the average
salary for each gender bucket
(metric)

48. Combination Query
GET /employees/employee/_search
{
“aggs”: {
“by_country”: { “terms”: {“field”: “country”},
“aggs”: {
“by_gender”: { “terms”: {“field”: “gender”},
“aggs”: {
“avg_salary”: {“avg”: “field”: “salary”}
}
}
}
}
}
}

49. More Aggregations
● Histogram
● Date Histogram
● Date Range
● Filter/Filters
● Missing
● Geo Distance
● Nested
● ...

50. Best Practices

51. Indexing
● Use bulk indexing APIs.
● Tune your bulk size 5-10MB.
● Partitions your time series data
by time period (monthly, weekly,
daily).
● Use aliases for your indices.
● Turn off refresh, replicas while
indexing. Turn on once it’s done
● Multiple shards for parallel
indexing.
● Multiple replicas for parallel
reading.

52. Mapping
● Disable _all field
● Keep _source field, do not store
any field.
● Use not_analyzed if possible

53. Query
● Use filters instead of queries
if possible.
● Consider orders and scope of
your filters.
● Do not use string query.
● Do not load too many results
with single query, use scroll
API instead.

54. Tools

55. Kibana for Discovery, Visualization

56. Sense for Query

57. Marvel for Monitoring