Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Elasticsearch & Docker
Rafał Kuć – Sematext Group, Inc.
@kucrafal @sematext sematext.com
Running High Performance
Fault To...
About me…
Sematext consultant & engineer
Solr.pl co-founder
Father and husband :)
Next 30 minutes
You Are Probably Familiar With This
Development
You Are Probably Familiar With This
Development Test
You Are Probably Familiar With This
Development Test QA
You Are Probably Familiar With This
Development Test QA
Production environment
And The Problems That Come With It
Resources not utilized
And The Problems That Come With It
Resources not utilized
Overprovisioned
Servers
And The Problems That Come With It
Resources not utilized
Overprovisioned
Servers
≠ ≠
The solution
Development Test QA Production
Container Technologies
What is Docker?
Lightweight
Based on
Open Standards
Secure
Containers vs Virtual Machines
Hardware
Traditional Virtual Machine
Containers vs Virtual Machines
Hardware
Host Operating System
Traditional Virtual Machine
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Traditional Virtual Machine
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Guest OS Guest OS
Traditional Virtual Machine
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Guest OS Guest OS
Libraries Libraries
Traditional...
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application...
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application...
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application...
Containers vs Virtual Machines
Hardware
Host Operating System
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application...
What is Elasticsearch?
Reasonable
defaults { JSON }
Distributed
by design
http://www.dailypets.co.uk/2007/06/17/kittens-re...
Running Official Elasticsearch Container
$ docker run -d elasticsearch
Running Official Elasticsearch Container
$ docker run -d elasticsearch == docker run -d elasticsearch:latest
Running Official Elasticsearch Container
$ docker run -d elasticsearch:1.7
$ docker run -d elasticsearch == docker run -d ...
Running Official Elasticsearch Container
$ docker run -d elasticsearch == docker run -d elasticsearch:latest
$ docker run ...
Running Official Elasticsearch Container
$ docker run -d elasticsearch == docker run -d elasticsearch:latest
$ docker run ...
Container Constraints
$ docker run -d -m 2G elasticsearch
http://docs.docker.com/engine/reference/run/
Container Constraints
$ docker run -d -m 2G elasticsearch
$ docker run -d -m 2G --memory-swappiness=0 elasticsearch
http:/...
Container Constraints
$ docker run -d -m 2G elasticsearch
$ docker run -d -m 2G --memory-swappiness=0 elasticsearch
$ dock...
Container Constraints
$ docker run -d -m 2G elasticsearch
$ docker run -d -m 2G --memory-swappiness=0 elasticsearch
$ dock...
Creating Optimized Image
Dockerfile:
FROM elasticsearch
ADD ./elasticsearch.yml /usr/share/elasticsearch/config/
Creating Optimized Image
Dockerfile:
FROM elasticsearch
ADD ./elasticsearch.yml /usr/share/elasticsearch/config/
$ docker ...
Creating Optimized Image
Dockerfile:
FROM elasticsearch
ADD ./elasticsearch.yml /usr/share/elasticsearch/config/
$ docker ...
Dealing With Network
$ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
Dealing With Network
$ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
$ docker run -d elasticsearch
-Dnetwork.publi...
Dealing With Network
$ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
$ docker run -d elasticsearch
-Dnetwork.publi...
Dealing With Network
$ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
$ docker run -d elasticsearch
-Dnetwork.publi...
Network - Good Practices
Separate network for Elasticsearch cluster
Network - Good Practices
Separate network for Elasticsearch cluster
Common host names for containers
$ docker run -d -h es...
Network - Good Practices
Separate network for Elasticsearch cluster
Common host names for containers
$ docker run -d -h es...
Network - Good Practices
Separate network for Elasticsearch cluster
Common host names for containers
$ docker run -d -h es...
Dealing With Storage
By default in /usr/share/elasticsearch/data
Dealing With Storage
By default in /usr/share/elasticsearch/data
By default not persisted
Dealing With Storage
By default in /usr/share/elasticsearch/data
By default not persisted
$ docker run -d
-v /opt/elastics...
Dealing With Storage
$ docker run -d
-v /opt/elasticsearch/data:/usr/share/elasticsearch/data
elasticsearch
By default in ...
Data-Only Docker Volumes
Bypasses Union File System
Data-Only Docker Volumes
Bypasses Union File System
Can be shared between containers
Data-Only Docker Volumes
Bypasses Union File System
Can be shared between containers
Data volumes persist if the container...
Data-Only Docker Volumes
Bypasses Union File System
Can be shared between containers
Data volumes persist if the container...
Data-Only Docker Volumes
Bypasses Union File System
Can be shared between containers
Data volumes persist if the container...
Highly Available Cluster
Master only
Master only
Master only
Data only
Data only
Data only
Data only
Data only
Data only
C...
Highly Available Cluster
Master only
Master only
Master only
Data only
Data only
Data only
Data only
Data only
Data only
C...
Highly Available Cluster
Master only
Master only
Master only
Data only
Data only
Data only
Data only
Data only
Data only
C...
Master Nodes & Docker
$ docker run -d elasticsearch
-Dnode.master=true
-Dnode.data=false
-Dnode.client=false
Client Nodes & Docker
$ docker run -d elasticsearch
-Dnode.master=false
-Dnode.data=false
-Dnode.client=true
Data Nodes & Docker
$ docker run -d elasticsearch
-Dnode.master=false
-Dnode.data=true
-Dnode.client=false
Scaling
Elasticsearch Node Elasticsearch Node
Elasticsearch Node Elasticsearch Node
Scaling
curl -XPUT 'http://localhost:9200/devops/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 4,
"number_of_rep...
Scaling
P P
P P
Scaling
curl -XPUT 'http://localhost:9200/devops/_settings' -d '{
"index.number_of_replicas" : 1
}'
Scaling
P P
P P
R
R R
R
Scaling
curl -XPUT 'http://localhost:9200/devops/_settings' -d '{
"index.number_of_replicas" : 2
}'
Scaling
P P
P P
R R
R R
R R
R R
Scaling
curl -XPUT 'http://localhost:9200/devops/_settings' -d '{
"index.number_of_replicas" : 1
}'
Scaling
P P
P P
R
R R
R
Scaling
P P
P P
R
R R
R
Scaling
P P PP
UnassignedR
R
R
R
RAM Buffer
indices.memory.index_buffer_size: 10%
indices.memory.min_index_buffer_size: 48mb
indices.memory.max_index_buffe...
RAM Buffer
indices.memory.index_buffer_size: 10%
indices.memory.min_index_buffer_size: 48mb
indices.memory.max_index_buffe...
Time-Based Data?
2015-11-23
TODAY
WEEK
Time-Based Data?
curl -XPOST 'http://localhost:9200/_aliases' -d '{
"actions" : [
{ "add" : {"index":"2015-11-23","alias":...
Time-Based Data?
2015-11-23 2015-11-24
TODAY
WEEK
Time-Based Data?
2015-11-23 2015-11-24 2014-11-25
TODAY
WEEK
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
Multiple Tiers
curl -XPUT 'localhost:9200/data_2015-11-23' -d '{
"settings": {
"index.routing.allocation.include.tag" : "h...
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
data_2015-11-23
data_2015-11-23
Multiple Tiers
curl -XPUT 'localhost:9200/data_2015-11-23/_settings' -d '{
"settings": {
"index.routing.allocation.exclude...
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
data_2015-11-23
data_2015-11-23
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
data_2015-11-23
data_2015-11-23
data_2015-11-24
data_2015-11-24
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
data_2015-11-23
data_2015-11-23
data_2015-11-25
data_2015-11-25
da...
Multiple Tenants
Multiple Tenants
Hot
Hot
Cold
Cold
Cold
Cold
Multiple Tenants
Hot
Hot
Cold
Cold
Cold
Cold
R
O
U
T
I
N
G
Indexing Without Routing
Shard 1 Shard 2 Shard 3 Shard 4
Shard 5 Shard 6 Shard 7 Shard 8
Elasticsearch
Application
userA
u...
Indexing With Routing
Shard 1 Shard 2 Shard 3 Shard 4
Shard 5 Shard 6 Shard 7 Shard 8
Elasticsearch
Application
Querying Without Routing
Shard 1 Shard 2 Shard 3 Shard 4
Shard 5 Shard 6 Shard 7 Shard 8
Elasticsearch
Application
Querying With Routing
Shard 1 Shard 2 Shard 3 Shard 4
Shard 5 Shard 6 Shard 7 Shard 8
Elasticsearch
Application
Routing vs No Routing
Queries without routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Media...
Routing vs No Routing
Queries without routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Media...
Monitoring
https://sematext.com/spm/integrations/docker-monitoring.html
https://github.com/sematext/spm-agent-docker
Short summary
http://www.soothetube.com/2013/12/29/thats-all-folks/
We Are Hiring!
Dig Search?
Dig Analytics?
Dig Big Data?
Dig Performance?
Dig Logging?
Dig working with, and in, open–sourc...
Rafał Kuć
@kucrafal
rafal.kuc@sematext.com
Sematext
@sematext
http://sematext.com
http://blog.sematext.com
Thank You !
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Docker up and running
Next
Upcoming SlideShare
Docker up and running
Next
Download to read offline and view in fullscreen.

Share

Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker

Download to read offline

Sematext engineer Rafal Kuc (@kucrafal) walks through the details of running high-performance, fault tolerant Elasticsearch clusters on Docker. Topics include: Containers vs. Virtual Machines, running the official Elasticsearch container, container constraints, good network practices, dealing with storage, data-only Docker volumes, scaling, time-based data, multiple tiers and tenants, indexing with and without routing, querying with and without routing, routing vs. no routing, and monitoring. Talk was delivered at DevOps Days Warsaw 2015.

Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker

  1. 1. Elasticsearch & Docker Rafał Kuć – Sematext Group, Inc. @kucrafal @sematext sematext.com Running High Performance Fault Tolerant Elasticsearch Clusters On Docker
  2. 2. About me… Sematext consultant & engineer Solr.pl co-founder Father and husband :)
  3. 3. Next 30 minutes
  4. 4. You Are Probably Familiar With This Development
  5. 5. You Are Probably Familiar With This Development Test
  6. 6. You Are Probably Familiar With This Development Test QA
  7. 7. You Are Probably Familiar With This Development Test QA Production environment
  8. 8. And The Problems That Come With It Resources not utilized
  9. 9. And The Problems That Come With It Resources not utilized Overprovisioned Servers
  10. 10. And The Problems That Come With It Resources not utilized Overprovisioned Servers ≠ ≠
  11. 11. The solution Development Test QA Production
  12. 12. Container Technologies
  13. 13. What is Docker? Lightweight Based on Open Standards Secure
  14. 14. Containers vs Virtual Machines Hardware Traditional Virtual Machine
  15. 15. Containers vs Virtual Machines Hardware Host Operating System Traditional Virtual Machine
  16. 16. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Traditional Virtual Machine
  17. 17. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Traditional Virtual Machine
  18. 18. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Traditional Virtual Machine
  19. 19. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Traditional Virtual Machine
  20. 20. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Hardware Host Operating System Traditional Virtual MachineContainer
  21. 21. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Hardware Host Operating System Docker Engine Traditional Virtual MachineContainer
  22. 22. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Hardware Host Operating System Docker Engine Libraries Libraries Application 1 Application 2 Traditional Virtual MachineContainer
  23. 23. What is Elasticsearch? Reasonable defaults { JSON } Distributed by design http://www.dailypets.co.uk/2007/06/17/kittens-rest-at-half-time/
  24. 24. Running Official Elasticsearch Container $ docker run -d elasticsearch
  25. 25. Running Official Elasticsearch Container $ docker run -d elasticsearch == docker run -d elasticsearch:latest
  26. 26. Running Official Elasticsearch Container $ docker run -d elasticsearch:1.7 $ docker run -d elasticsearch == docker run -d elasticsearch:latest
  27. 27. Running Official Elasticsearch Container $ docker run -d elasticsearch == docker run -d elasticsearch:latest $ docker run --name es_1 -h es_master_1 elasticsearch $ docker run -d elasticsearch:1.7
  28. 28. Running Official Elasticsearch Container $ docker run -d elasticsearch == docker run -d elasticsearch:latest $ docker run --name es_1 -h es_master_1 elasticsearch $ docker run -d elasticsearch:1.7
  29. 29. Container Constraints $ docker run -d -m 2G elasticsearch http://docs.docker.com/engine/reference/run/
  30. 30. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch http://docs.docker.com/engine/reference/run/
  31. 31. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch http://docs.docker.com/engine/reference/run/
  32. 32. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch http://docs.docker.com/engine/reference/run/ $ docker run -d --cpu-period=50000 --cpu-quota=25000 elasticsearch
  33. 33. Creating Optimized Image Dockerfile: FROM elasticsearch ADD ./elasticsearch.yml /usr/share/elasticsearch/config/
  34. 34. Creating Optimized Image Dockerfile: FROM elasticsearch ADD ./elasticsearch.yml /usr/share/elasticsearch/config/ $ docker build -t devops/example .
  35. 35. Creating Optimized Image Dockerfile: FROM elasticsearch ADD ./elasticsearch.yml /usr/share/elasticsearch/config/ $ docker build -t devops/example . Sending build context to Docker daemon 3.072 kB Step 1 : FROM elasticsearch ---> 8112755253f1 Step 2 : ADD ./elasticsearch.yml /usr/share/elasticsearch/config/ ---> Using cache ---> c9ca48a22e58 Successfully built c9ca48a22e58
  36. 36. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
  37. 37. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch $ docker run -d elasticsearch -Dnetwork.publish_host=192.168.1.1
  38. 38. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch $ docker run -d elasticsearch -Dnetwork.publish_host=192.168.1.1 $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Dnetwork.publish_host=192.168.1.1
  39. 39. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch $ docker run -d elasticsearch -Dnetwork.publish_host=192.168.1.1 $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Dnetwork.publish_host=192.168.1.1 $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Dnetwork.publish_host=0.0.0.0
  40. 40. Network - Good Practices Separate network for Elasticsearch cluster
  41. 41. Network - Good Practices Separate network for Elasticsearch cluster Common host names for containers $ docker run -d -h es_node_1 elasticsearch
  42. 42. Network - Good Practices Separate network for Elasticsearch cluster Common host names for containers $ docker run -d -h es_node_1 elasticsearch Expose 9200 & 9300 ports only for client nodes
  43. 43. Network - Good Practices Separate network for Elasticsearch cluster Common host names for containers $ docker run -d -h es_node_1 elasticsearch Expose 9200 & 9300 ports only for client nodes Elasticsearch data & client nodes point to masters only
  44. 44. Dealing With Storage By default in /usr/share/elasticsearch/data
  45. 45. Dealing With Storage By default in /usr/share/elasticsearch/data By default not persisted
  46. 46. Dealing With Storage By default in /usr/share/elasticsearch/data By default not persisted $ docker run -d -v /opt/elasticsearch/data:/usr/share/elasticsearch/data elasticsearch
  47. 47. Dealing With Storage $ docker run -d -v /opt/elasticsearch/data:/usr/share/elasticsearch/data elasticsearch By default in /usr/share/elasticsearch/data By default not persisted Use data only containers Permissions
  48. 48. Data-Only Docker Volumes Bypasses Union File System
  49. 49. Data-Only Docker Volumes Bypasses Union File System Can be shared between containers
  50. 50. Data-Only Docker Volumes Bypasses Union File System Can be shared between containers Data volumes persist if the container itself is deleted
  51. 51. Data-Only Docker Volumes Bypasses Union File System Can be shared between containers Data volumes persist if the container itself is deleted $ docker create -v /mnt/es/data:/usr/share/elasticsearch/data --name esdata elasticsearch Permissions
  52. 52. Data-Only Docker Volumes Bypasses Union File System Can be shared between containers Data volumes persist if the container itself is deleted $ docker create -v /mnt/es/data:/usr/share/elasticsearch/data --name esdata elasticsearch $ docker run --volumes-from esdata elasticsearch
  53. 53. Highly Available Cluster Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only
  54. 54. Highly Available Cluster Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only minimum_master_nodes = N/2 + 1
  55. 55. Highly Available Cluster Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only minimum_master_nodes = N/2 + 1 recovery.after.nodes recovery.expected.nodes cluster.routing.allocation.node_concurrent_ recoveries index.unassigned.node_left.delayed_timeout index.priority
  56. 56. Master Nodes & Docker $ docker run -d elasticsearch -Dnode.master=true -Dnode.data=false -Dnode.client=false
  57. 57. Client Nodes & Docker $ docker run -d elasticsearch -Dnode.master=false -Dnode.data=false -Dnode.client=true
  58. 58. Data Nodes & Docker $ docker run -d elasticsearch -Dnode.master=false -Dnode.data=true -Dnode.client=false
  59. 59. Scaling Elasticsearch Node Elasticsearch Node Elasticsearch Node Elasticsearch Node
  60. 60. Scaling curl -XPUT 'http://localhost:9200/devops/' -d '{ "settings" : { "index" : { "number_of_shards" : 4, "number_of_replicas" : 0 } } }'
  61. 61. Scaling P P P P
  62. 62. Scaling curl -XPUT 'http://localhost:9200/devops/_settings' -d '{ "index.number_of_replicas" : 1 }'
  63. 63. Scaling P P P P R R R R
  64. 64. Scaling curl -XPUT 'http://localhost:9200/devops/_settings' -d '{ "index.number_of_replicas" : 2 }'
  65. 65. Scaling P P P P R R R R R R R R
  66. 66. Scaling curl -XPUT 'http://localhost:9200/devops/_settings' -d '{ "index.number_of_replicas" : 1 }'
  67. 67. Scaling P P P P R R R R
  68. 68. Scaling P P P P R R R R
  69. 69. Scaling P P PP UnassignedR R R R
  70. 70. RAM Buffer indices.memory.index_buffer_size: 10% indices.memory.min_index_buffer_size: 48mb indices.memory.max_index_buffer_size (unbounded) indices.memory.min_shard_index_buffer_size: 4mb
  71. 71. RAM Buffer indices.memory.index_buffer_size: 10% indices.memory.min_index_buffer_size: 48mb indices.memory.max_index_buffer_size (unbounded) indices.memory.min_shard_index_buffer_size: 4mb Higher Indexing Throughput Lower Indexing Throughput defaults ><
  72. 72. Time-Based Data? 2015-11-23 TODAY WEEK
  73. 73. Time-Based Data? curl -XPOST 'http://localhost:9200/_aliases' -d '{ "actions" : [ { "add" : {"index":"2015-11-23","alias":"today"} }, { "add" : {"index":"2015-11-23","alias":"week"} } ]}'
  74. 74. Time-Based Data? 2015-11-23 2015-11-24 TODAY WEEK
  75. 75. Time-Based Data? 2015-11-23 2015-11-24 2014-11-25 TODAY WEEK
  76. 76. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold
  77. 77. Multiple Tiers curl -XPUT 'localhost:9200/data_2015-11-23' -d '{ "settings": { "index.routing.allocation.include.tag" : "hot" } }'
  78. 78. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23
  79. 79. Multiple Tiers curl -XPUT 'localhost:9200/data_2015-11-23/_settings' -d '{ "settings": { "index.routing.allocation.exclude.tag" : "hot", "index.routing.allocation.include.tag" : "cold", } }'
  80. 80. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23
  81. 81. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23 data_2015-11-24 data_2015-11-24
  82. 82. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23 data_2015-11-25 data_2015-11-25 data_2015-11-24 data_2015-11-24
  83. 83. Multiple Tenants
  84. 84. Multiple Tenants Hot Hot Cold Cold Cold Cold
  85. 85. Multiple Tenants Hot Hot Cold Cold Cold Cold R O U T I N G
  86. 86. Indexing Without Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application userA userA userA userA userAuserA userA userA
  87. 87. Indexing With Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application
  88. 88. Querying Without Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application
  89. 89. Querying With Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application
  90. 90. Routing vs No Routing Queries without routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 1 3169ms 19,0/min 5214ms 2692ms 95 – 99%
  91. 91. Routing vs No Routing Queries without routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 1 3169ms 19,0/min 5214ms 2692ms 95 – 99% Queries with routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 10 196ms 50,6/sec 642ms 29ms 25 – 40% 20 218ms 91,2/sec 718ms 11ms 10 – 15%
  92. 92. Monitoring https://sematext.com/spm/integrations/docker-monitoring.html https://github.com/sematext/spm-agent-docker
  93. 93. Short summary http://www.soothetube.com/2013/12/29/thats-all-folks/
  94. 94. We Are Hiring! Dig Search? Dig Analytics? Dig Big Data? Dig Performance? Dig Logging? Dig working with, and in, open–source? We’re hiring worldwide! http://sematext.com/about/jobs.html
  95. 95. Rafał Kuć @kucrafal rafal.kuc@sematext.com Sematext @sematext http://sematext.com http://blog.sematext.com Thank You !
  • hteixe

    Jun. 19, 2019
  • jpietrocini

    Oct. 22, 2018
  • eayoub

    Aug. 29, 2018
  • OkharyadiSaputra

    Jun. 23, 2018
  • BartomiejRybak

    Mar. 30, 2018
  • kennethowino9

    Dec. 6, 2017
  • GeorgianCocis

    Nov. 30, 2017
  • TianpoGao

    Oct. 9, 2017
  • stouneua

    Sep. 28, 2017
  • perng1220

    Aug. 28, 2017
  • motoskia

    Aug. 14, 2017
  • SiyuChen10

    May. 18, 2017
  • frantzcy

    Apr. 29, 2017
  • kunlqt

    Apr. 3, 2017
  • NickitaMoroz

    Mar. 20, 2017
  • ChinhVoCong

    Dec. 21, 2016
  • MengQi1

    Dec. 18, 2016
  • ShahanRehan

    Dec. 10, 2016
  • ImmaValls

    Dec. 4, 2016
  • KingMasterCnC

    Nov. 28, 2016

Sematext engineer Rafal Kuc (@kucrafal) walks through the details of running high-performance, fault tolerant Elasticsearch clusters on Docker. Topics include: Containers vs. Virtual Machines, running the official Elasticsearch container, container constraints, good network practices, dealing with storage, data-only Docker volumes, scaling, time-based data, multiple tiers and tenants, indexing with and without routing, querying with and without routing, routing vs. no routing, and monitoring. Talk was delivered at DevOps Days Warsaw 2015.

Views

Total views

37,195

On Slideshare

0

From embeds

0

Number of embeds

19,277

Actions

Downloads

492

Shares

0

Comments

0

Likes

85

×