I researched on Elasticsearch and gave my colleague an intro on Elasticsearch Internal. I focused on the contents that are more relevant to our use cases.

1. Elasticsearch Internal - Shards
Jiaming (Jason) Zhang
31 May 2017

2. Recap

3. Recap
Launch Elasticsearch (ES) and Kibana (a visualization tool for ES) locally
brew install elasticsearch
# Automatically start Elasticsearch when log in
# By default, Elasticsearch will be available at http://localhost:9200
brew services start elasticsearch
###################################
brew install kibana
# Automatically start Kibana when log in
# By default, Kibana will be available at http://localhost:5601
brew services start kibana
###################################
# Verify whether both service is started
brew services list | grep -E "elasticsearch|kibana"

4. Elasticsearch Internal - Shards

5. Elasticsearch Internal - Shards
A Elasticsearch cluster has 2 indices. Here is the setting for one of them:
# Index: entity_companies
{
"number_of_primary_shards": 3,
"number_of_replicas": 2
}
How many shards does the cluster has?

6. Elasticsearch Internal - Shards
A Elasticsearch cluster has 2 indices. Here is the setting for one of them:
# Index: entity_companies
{
"number_of_primary_shards": 3,
"number_of_replicas": 2
}
How many shards does the cluster has? It also depends on the other index.

7. Elasticsearch Internal - Shards
A Elasticsearch cluster has 2 indices. Here is the setting for one of them:
# Index: entity_companies
{
"number_of_primary_shards": 3,
"number_of_replicas": 2
}
How many shards does the cluster has? It also depends on the other index.
How many shards does this index has?

8. Elasticsearch Internal - Shards
A Elasticsearch cluster has 2 indices. Here is the setting for one of them:
# Index: entity_companies
{
"number_of_primary_shards": 3,
"number_of_replicas": 2
}
How many shards does the cluster has? It also depends on the other index.
How many shards does this index has? It also depends on # of nodes the cluster has.

9. Elasticsearch Internal - Shards
A Elasticsearch cluster has 2 indices. Here is the setting for one of them:
# Index: entity_companies
{
"number_of_primary_shards": 3,
"number_of_replicas": 2
}
How many shards does the cluster has? It also depends on the other index.
How many shards does this index has? It also depends on # of nodes the cluster has.
How many shards does this index expects to has?

10. Elasticsearch Internal - Shards
A Elasticsearch cluster has 2 indices. Here is the setting for one of them:
# Index: entity_companies
{
"number_of_primary_shards": 3,
"number_of_replicas": 2
}
How many shards does the cluster has? It also depends on the other index.
How many shards does this index has? It also depends on # of nodes the cluster has.
How many shards does this index expects to has? 9 shards = 3 primary shards * (1 origin + 2 replicas)

11. Elasticsearch Internal - Shards
Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the
cluster.

12. Elasticsearch Internal - Shards
Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the
cluster.
Thus, # of primary shards effectively determines how many data an index can hold.
Elasticsearch Reference - Basic Concepts

13. Elasticsearch Internal - Shards
What factors determine how many data an index can store?

14. Elasticsearch Internal - Shards
What factors determine how many data an index can store?
Here are the factors that determine how many data an index can store:
# of primary shards
Max # of documents a shard can have is bounded by max # of documents a Lucene index can have
(around 2 billions). [1]
Hardware (e.g. # of nodes, disk space, CPU)
Document size
Use case (e.g. query, expected response time)
[1] Elasticsearch Reference - Basic Concept

15. Elasticsearch Internal - Shards
Create index entity_companies w/ default settings
PUT entity_companies

16. Elasticsearch Internal - Shards
Create index entity_companies w/ default settings
PUT entity_companies
Get index entity_companies metadata
GET _cat/indices/entity_companies?v&h=health,index,pri,rep
health index pri rep
yellow entity_companies 5 1

17. Elasticsearch Internal - Shards
Create index entity_companies w/ default settings
PUT entity_companies
Get index entity_companies metadata
GET _cat/indices/entity_companies?v&h=health,index,pri,rep
health index pri rep
yellow entity_companies 5 1
By default, an index has 5 primary shards and 1 replica.

18. Elasticsearch Internal - Shards
Create index entity_people w/ specified settings
PUT entity_people
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}
}

19. Elasticsearch Internal - Shards
Create index entity_people w/ specified settings
PUT entity_people
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}
}
Update number_of_replicas after index entity_people is created
PUT entity_people/_settings
{
"index" : { "number_of_replicas" : 2 }
}

20. Elasticsearch Internal - Shards
Update number_of_shards after index entity_people is created (THIS WILL FAIL)
PUT entity_people/_settings
{
"index" : { "number_of_shards" : 2 }
}
Response: 400
Reason: Can't update non dynamic settings [[index.number_of_shards]]
for open indices [[entity_people/JGW2oY98RZeZJM8mFN5h_w]]

21. Elasticsearch Internal - Shards
Why doesn't Elasticsearch support changing # of primary shards on-the-fly?

22. Elasticsearch Internal - Shards
Why doesn't Elasticsearch support changing # of primary shards on-the-fly?
The answer is related to how Elasticsearch determines which shard a document should be saved in.

23. Elasticsearch Internal - Shards
Why doesn't Elasticsearch support changing # of primary shards on-the-fly?
The answer is related to how Elasticsearch determines which shard a document should be saved in.
... if the number of primary shards ever changed in the future, all previous routing values would be
invalid and documents would never be found.
Elasticsearch Guide - Routing Value

24. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards?

25. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards? - Reindexing all documents

26. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards? - Reindexing all documents
Reindexing is needed when we want to update certain immutable settings of the index, like increasing # of
primary shards or changing existed field mapping.

27. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards? - Reindexing all documents
Reindexing is needed when we want to update certain immutable settings of the index, like increasing # of
primary shards or changing existed field mapping.
Reindexing simply means to (1) cycle through all documents in the existed index and (2) re-insert them to the
new index with the desired new settings. Reindex API was introduced to Elasticsearch in 2016.

28. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards? - Reindexing all documents
Reindexing is needed when we want to update certain immutable settings of the index, like increasing # of
primary shards or changing existed field mapping.
Reindexing simply means to (1) cycle through all documents in the existed index and (2) re-insert them to the
new index with the desired new settings. Reindex API was introduced to Elasticsearch in 2016.
POST _reindex
{ "source": { "index": "entity_companies" },
"dest": { "index": "entity_companies_v2" }}

29. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards? - Reindexing all documents
Reindexing is needed when we want to update certain immutable settings of the index, like increasing # of
primary shards or changing existed field mapping.
Reindexing simply means to (1) cycle through all documents in the existed index and (2) re-insert them to the
new index with the desired new settings. Reindex API was introduced to Elasticsearch in 2016.
POST _reindex
{ "source": { "index": "entity_companies" },
"dest": { "index": "entity_companies_v2" }}
Reindex API also supports
1. Selective Reindex Operation
2. Using Scripts with the Reindexing API
3. Reindexing for Mapping Changes

30. Elasticsearch Internal - Shards
What should I do if I really need to increase # of primary shards? - Reindexing all documents
Reindexing is needed when we want to update certain immutable settings of the index, like increasing # of
primary shards or changing existed field mapping.
Reindexing simply means to (1) cycle through all documents in the existed index and (2) re-insert them to the
new index with the desired new settings. Reindex API was introduced to Elasticsearch in 2016.
POST _reindex
{ "source": { "index": "entity_companies" },
"dest": { "index": "entity_companies_v2" }}
Reindex API also supports
1. Selective Reindex Operation
2. Using Scripts with the Reindexing API
3. Reindexing for Mapping Changes
Reindex Your Documents with Ease and without Third-Party Scripts using Elasticsearch

31. Elasticsearch Internal - Shards
Use index alias if you need to regularly reindex your documents

32. Elasticsearch Internal - Shards
Let's put it together

33. Elasticsearch Internal - Shards
Let's put it together
Step 1: Create index w/ new settings
PUT entity_companies_v2
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}
}

34. Elasticsearch Internal - Shards
Let's put it together
Step 1: Create index w/ new settings
PUT entity_companies_v2
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}
}
Step 2: Connect alias to the new index
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "entity_companies_v2", "alias" : "entity_companies" } }
]
}

35. Elasticsearch Internal - Shards
Let's put it together
Step 3: Reindex all document from old index to the new one
POST _reindex
{
"source": { "index": "entity_companies" },
"dest": { "index": "entity_companies_v2" }
}

36. Elasticsearch Internal - Shards
Let's put it together
Step 3: Reindex all document from old index to the new one
POST _reindex
{
"source": { "index": "entity_companies" },
"dest": { "index": "entity_companies_v2" }
}
Step 4: Remove alias's connection to the old index
POST /_aliases
{
"actions" : [
{ "remove" : { "index" : "entity_companies_v2", "alias" : "entity_companies" } }
]
}
Elasticsearch Reference - Indices Aliases

37. Reading
If you find this topic interesting, here are a few extra readings:
Optimizing Elasticsearch: How Many Shards per Index?
Elasticsearch - The Definitive Guide (by ClintonGormley & Zachary Tong)
Chapter 2 Life Inside a Cluster: This chapter explains what Elasticsearch's internal looks like. Shard
is the most important concept to understand Elasticsearch's internal. It also explains what happen
when we add more nodes to the cluster or a cluster node fails.
Chapter 4 Distributed Document Store: This chapter explains how shards communicates w/ each
other when a create/update/delete/query request is made as well as other topic related to
Elasticsearch's distributed nature