This document provides an overview of Elasticsearch concepts and best practices. It discusses Elasticsearch documentation, cluster sizing recommendations from 3 nodes to 100s of nodes, default installations, monitoring capabilities, auto-generated mappings, template structures, dynamic settings, dedicated node types, shard sizing between 15-50GB, specific cluster sizing tools, indexing operations like rollover and split, search optimizations, and considerations for multi-cluster architectures. Customer stories from Elastic Support are recommended to learn lessons from real-world use cases.

1. 1
Michael Yuan
March 20, 2019
Scale with confidence

2. 2
From your laptop to
lots of nodes, and
multiple use cases.
So let's go...

3. 3
100,000+
Community
Members
150M+
Product
Downloads
3,700+
Subscription
Customers
Statistics since 2012, founding of Elastic

4. 4
840 Pages of Elasticsearch Documentation

5. 5
30 minute presentation

6. 6
2.142857143 seconds per page

7. 7
Cluster Size
3
nodes
10
nodes
100’s
nodes

8. 8
Use the Default Installation
https://www.elastic.co/subscriptions

9. 9
Monitor all the things
https://www.elastic.co/guide/en/elastic-stack-overview/6.6/xpack-monitoring.html

10. 10
Cluster Monitoring

11. 11
Node Monitoring

12. 12
Index Monitoring

13. 13
Kibana Monitoring

14. 14
Logstash Monitoring

15. 15
Beats Monitoring

16. 16
APM Monitoring

17. 17
Mappings
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

18. 18
Auto generated mappings
A Great Default, Can enforce Policy in Production
{
"line" : 4,
"play" : "HENRY IV",
”speech" : "So shaken as we are, so wan with
care,"
}
Sample document Auto-generated mapping
"line": {
"type": "long"
},
"play": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
”speech": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
Things to watch for
• Numeric default to
long where int is
sufficient
• ‘play’ field mapped
as both text and
keyword
• ‘speech’ should only
be type text

19. 19
Base template structure
Elasticsearch Template
curl –XPUT localhost:9200/_template/template_1
{
"index_patterns" : ["windows-*"],
"order" : 0,
"settings" : {},
"mappings" : {}
}

20. 20
Add Dynamic setting
Elasticsearch Template
curl –XPUT localhost:9200/_template/template_1
{
"index_patterns" : ["windows-*"],
"order" : 0,
"settings" : {...},
"mappings" : {
"docs" : {
"dynamic": true, // default value
"properties": {...}
}
}
}

21. 21
Dynamic template settings
Elasticsearch Template
• "dynamic": true
‒ Newly detected fields are added to the mapping. (default)
https://www.elastic.co/guide/en/elasticsearch/reference/master/dynamic.html

22. 22
Dynamic template settings
Elasticsearch Template
• "dynamic": true
‒ Newly detected fields are added to the mapping. (default)
• "dynamic": false
‒ Newly detected fields are ignored.
https://www.elastic.co/guide/en/elasticsearch/reference/master/dynamic.html

23. 23
Dynamic template settings
Elasticsearch Template
• "dynamic": true
‒ Newly detected fields are added to the mapping. (default)
• "dynamic": false
‒ Newly detected fields are ignored.
• "dynamic": strict
‒ If new fields are detected, an exception is thrown and the document is rejected. New fields
must be explicitly added to the mapping.
https://www.elastic.co/guide/en/elasticsearch/reference/master/dynamic.html

24. 24
Cluster Size
3
nodes
10
nodes
100’s
nodes

25. 25
Dedicated Nodes
https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-node.html

26. 26
To Dedicate or not to Dedicate?
• Master Nodes
• Ingest Nodes
• Coordinator Nodes (a.k.a. Client Nodes)
• Machine Learning Nodes
• Data Nodes - Hot / Warm / Cold

27. 27
Demo

28. 28
Shard Size
https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

29. 29
15 to 50GB’ish per Shards

30. 30
But why 15 to 50GB’ish?

31. 31
Cluster Size
3
nodes
10
nodes
100’s
nodes

32. 32
Specific Cluster Sizing
https://github.com/elastic/rally
https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right
https://www.elastic.co/blog/announcing-rally-benchmarking-for-elasticsearch

33. 33

34. 34
• Logging:
‒ Rollover API - https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html
‒ Shrink API - https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html
‒ Split API - https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-split-index.html
• Metrics
‒ Rollups API - https://www.elastic.co/guide/en/elasticsearch/reference/master/rollups-api.html
• Search:
‒ Query Profiler - https://www.elastic.co/guide/en/elasticsearch/reference/master/_profiling_queries.html
‒ Force Merge - https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-forcemerge.html
And there is much more...

35. 35
• Think of the impact of:
• Cross Cluster Search
• Cross Cluster Replication
• Elastic Cloud and Elastic Cloud Enterprise
• ABAC
Does a monolithic cluster make sense?

36. 36
"Here, Hold My Beer."
Customer Stories to
Learn from as Told by
Support
https://www.elastic.co/elasticon/conf/2018/sf/here-hold-my-beer-customer-stories-to-learn-from-as-told-by-support

37. 37
Thank you...
Come and find us at the
AMA Booth