What is Elastic - Elasticsearch - Kibana - Elasticsearch indexes - Shard and Replica And a lot of exciting topics to be discussed

1. Elastic 101 - Get started
ENG. ISMAIL ANJRINI
ELASTIC CERTIFIED ENGINEER
CURRENT VERSION 7.6Elastic-Saudi-Arabia

2. ELASTIC STACK (ELK)
CURRENT VERSION 7.6

3. About Me
Ismail Anjrini
More than 15 years experience
Elasticsearch Certified Engineer
CURRENT VERSION 7.6

4. Elasticsearch
CURRENT VERSION 7.6

5. Kibana
CURRENT VERSION 7.6

6. Beats
CURRENT VERSION 7.6

7. Logstash
CURRENT VERSION 7.6

8. Lucene
Apache Lucene is an open source project available for free
Lucene is a Java library
Elasticsearch is built over Lucene and provides a JSON based REST API to refer to Lucene features
Elasticsearch provides a distributed system on top of Lucene
CURRENT VERSION 7.6

9. CURRENT VERSION 7.6

10. Elasticsearch Installation
Download and unzip Elasticsearch
Unix
◦ bin/elasticsearch
Windows
◦ binelasticsearch.bat
http://localhost:9200
Done
CURRENT VERSION 7.6

11. Elasticsearch Installation
Install elasticsearch as windows service
◦ elasticsearch-service.bat install
◦ Run the windows service
◦ Wait
◦ http://localhost:9200/
◦ Done
CURRENT VERSION 7.6

12. Kibana Installation
Download and unzip Elasticsearch
Unix
◦ bin/kibana
Windows
◦ binkibana.bat
http://localhost:5601
Done
CURRENT VERSION 7.6

13. Kibana Installation
Install Kibana as task
◦ Create new Task
◦ Run the new task
◦ Wait for a minute
◦ http://localhost:5601/
◦ Done
CURRENT VERSION 7.6

14. CURRENT VERSION 7.6
CONCEPTS

15. Index
An index is a collection of documents that have somewhat similar characteristics
◦ Customer data
◦ Orders
◦ Log
◦ …
An index is identified by a name (that must be all lowercase)
◦ Index name is used to refer to the index when performing indexing, search, update, and delete
operations against the documents in it
CURRENT VERSION 7.6

16. Document
A document is a basic unit of information that can be indexed
◦ Single customer
◦ Single log line
The document is expressed in JSON
CURRENT VERSION 7.6

17. Shards & Replicas
Each shard is in itself a fully-functional and independent "index" that can be hosted on any node
in the cluster
When you create an index, you can simply define the number of shards that you want
Sharding is important for two primary reasons:
◦ It allows you to horizontally split/scale your content volume
◦ It allows you to distribute and parallelize operations across shards (potentially on multiple nodes) thus
increasing performance/throughput
CURRENT VERSION 7.6

18. Shards
CURRENT VERSION 7.6

19. Shards
CURRENT VERSION 7.6

20. Shards routing
CURRENT VERSION 7.6

21. Shards & Replicas
Replication is important for two primary reasons:
◦ It provides high availability in case a shard/node fails.
◦ It allows to scale out your search volume/throughput since searches can be executed on all replicas in
parallel
CURRENT VERSION 7.6

22. Replicas
CURRENT VERSION 7.6

23. CURRENT VERSION 7.6
INDEX

24. Index
An index is like a ‘table’ in a relational database.
It has a mapping which defines multiple types.
An index is a logical namespace:
◦ Maps to one or more primary shards
◦ Can have zero or more replica shards
RDBMS
ES
Database
?
Table
Index
Columns/Rows
Document
CURRENT VERSION 7.6

25. Index settings
number_of_shards
number_of_replicas
refresh_interval
CURRENT VERSION 7.6

26. Index Operations – create index
We can update number of shards/number of replicas for existed indexes
CURRENT VERSION 7.6

27. Index Operations – mapping
CURRENT VERSION 7.6

28. Index Operations – mapping
PUT names
{
"mappings":
{
"properties":
{
"name":
{
"type": "keyword“
},
"name_text": { "type": "text" }
}
}
}
CURRENT VERSION 7.6

29. Index Operations – mapping
CURRENT VERSION 7.6

30. Index Operations – list all indexes
GET _cat/indices
GET /_cat/indices/twi*?v
GET /_cat/indices/?v&health=green|yellow|red&h=col1,col2
CURRENT VERSION 7.6

31. Index Operations – read index details
GET big-index
GET big-index?format=yaml|json
CURRENT VERSION 7.6

32. Index Operations – create document
POST big-index/_doc/1
{
"name": "Ismail Anjrini",
"age": 27
}
POST big-index/_doc/2
{
"name": "Fadi Abdul Wahab",
"age": 45,
"country": "Saudi Arabia"
}
CURRENT VERSION 7.6

33. Index Operations – POST vs PUT
POST big-index/_doc/
{
"name": "Kasem",
"age": 46
}
PUT big-index/_doc/
{
"name": "Riyadh",
"age": 33
}
CURRENT VERSION 7.6

34. Index Operations – read document
GET big-index/_doc/2
CURRENT VERSION 7.6

35. Index Operations – update document
POST big-index/_update/1
{
"doc":
{
"name":"Ismail Hassan Anjrini" ,
"country": "Syria"
}
}
CURRENT VERSION 7.6

36. Index Operations – delete document
DELETE big-index/_doc/1 PUT big-index/_doc/1
{
"name":"Ismail Anjrini",
"age": 27
}
CURRENT VERSION 7.6

37. Index Operations - Index aliases
An index alias is a secondary name used to refer to one or more existing indices
POST index-1/_alias/index-alias
POST index-2/_alias/index-alias
POST index-3/_alias/index-alias
CURRENT VERSION 7.6

38. Index Operations - Index aliases
filter: If specified, the index alias only applies to documents returned by the filter.
POST index-*/_alias/index-Egypt
{
"filter":
{
"term":
{
"nationality": "egypt"
}
}
}
CURRENT VERSION 7.6

39. Index Operations - Index aliases
DELETE index-1/_alias/index-alias
DELETE index-*/_alias/index-alias
GET index-alias/_search
GET index-alias/_search
CURRENT VERSION 7.6

40. Index Template
Index templates define settings and mappings that you can automatically apply when creating
new indices
Elasticsearch applies templates to new indices based on an index pattern that matches the index
name
Changes to index templates do not affect existing indices
Settings and mappings specified in create index API requests override any settings or mappings
specified in an index template
CURRENT VERSION 7.6

41. Index Template
CURRENT VERSION 7.6
PUT elastic-log-sys1

42. Index Template - Order
Multiple index templates can potentially match an index
Both the settings and mappings are merged into the final configuration of the index
The order of the merging can be controlled using the order parameter
With lower order being applied first, and higher orders overriding them
CURRENT VERSION 7.6

43. Index Template - Order
CURRENT VERSION 7.6
PUT elastic-log-sys1

44. Index Operations - Reindex
Reindex the current data in old-index to new-index
It does not copy the settings/fields settings from the source index to destination
CURRENT VERSION 7.6

45. Index Operations - Reindex
version_type: internal or empty:
◦ Update any document that have the same _id regardless the version number in the target index
◦ Increase the version number for the documents with the same _id
CURRENT VERSION 7.6

46. Index Operations - Reindex
CURRENT VERSION 7.6

47. Index Operations - Reindex
version_type: external
◦ Elasticsearch to preserve the version from the source
◦ Create any documents that are missing
◦ The _id value is not matched
◦ Update any documents that have an older version in the destination index than they do in the source
index
◦ The document with older version will get the same version number from the source index
CURRENT VERSION 7.6

48. Index Operations - Reindex
Created index-1
Add data to index-1
Delete new-index-1
CURRENT VERSION 7.6

49. Index Operations - Reindex
Add document to index-1
Do reindex
CURRENT VERSION 7.6

50. Index Operations - Reindex
op_type: create
◦ _reindex to only create missing documents in the target index
◦ All existing documents will cause a version conflict
max_docs
◦ To limit the number of processed documents from source to dest
CURRENT VERSION 7.6

51. CURRENT VERSION 7.6