The document details a developer meetup focused on Elasticsearch and related technologies, featuring discussions on various topics such as configuration, performance optimization, and monitoring. Key points include shard allocation strategies, zone awareness for data nodes, and the transition from transport clients to high-level REST clients in Elasticsearch. Additionally, it covers integrations with Prometheus for monitoring and alerting systems using Grafana.

1. Developer Meetup #33
LINE Elasticsearch +α
Kotaro WATANABE, Development Support 2018/04/25

2. Developer Meetup #33
l (Kotaro WATANABE)
l (Development Support)
l Infra Engineer (2016/04~)
l
l LINE
l LINE LIVE
l LINE
l LINE
l LINE
l Clova
l Etc.

3. Developer Meetup #33
l
l Elasticsearch
l Prometheus
l HBase
l MySQL
l Redis
l Memcached
l Kafka
l Hadoop
l Rabbitmq
l Tomcat
l Nginx
l Etc.
LINE

4. Developer Meetup #33
l
l Elasticsearch
l Prometheus
l HBase
l MySQL
l Redis
l Memcached
l Kafka
l Hadoop
l Rabbitmq
l Tomcat
l Nginx
l Etc.
LINE

5. Developer Meetup #33
l LINE
l Elasticsearch
l Elasticsearch
l
l
l

6. Developer Meetup #33
l LINE Elasticsearch
LINE

7. ˆLINE

8. Developer Meetup #33
l LINE
l
l 2017 7 26
l LINE : @linedelima
l 1100
l Elasticsearch
LINE

9. Developer Meetup #33
LINE ( )

10. Developer Meetup #33
LINE ( )

11. Developer Meetup #33
LINE

12. Developer Meetup #33
l Java Client
l Transport Client
l Indices
l block ( )
l item ( )
l shop ( )
l Dashboard
l Imenezes/cerebro

13. ˆElasticsearch

14. Developer Meetup #33
LINE
l Topics
1.
2. Admin Elasticsearch
3.

15. Developer Meetup #33
1.
l
l Elasticsearch
l 1 4000rps
l 2

16. Developer Meetup #33

17. Developer Meetup #33
l
l Master node
l
l VM 3
l discovery.zen.minimum_master_nodes: 2
l Primary shard Replica shard
l Primary shard: 5 (Data node )
l Replica shard: 1
l 1node 2 shard

18. Developer Meetup #33

19. Developer Meetup #33
l 20
l Master node
l Master node Data node
l discovery.zen.minimum_master_nodes
l (master_eligible_nodes / 2) + 1 = 2
l batch indexing rebalance allocation
l
l Shard Allocation Settings
l 5Mbps 20Mbps

20. Developer Meetup #33
2. Admin Elasticsearch
l Admin Elasticsearch
l Data node
l Forced Awareness
l /path/to/elasticsearch.yml
l e.g. /etc/elasticsearch/elasticsearch.yml
# for service zone
node.attr.zone: service
cluster.routing.allocation.awareness.force.zone.values: service,admin
cluster.routing.allocation.awareness.attributes: zone
# for admin zone
node.attr.zone: admin
cluster.routing.allocation.awareness.force.zone.values: service,admin
cluster.routing.allocation.awareness.attributes: zone

21. Developer Meetup #33
zone

22. Developer Meetup #33
zone
l zone index
l zone shard replica
l Primary shard Replica shard
l shard rebalance
l zone index
l zone Elasticsearch zone
l admin

23. Developer Meetup #33
3.
l
l
l 4000qps
l search Thread queue

24. Developer Meetup #33
l
l Thread Pool
l int((available processors * 3) / 2) + 1
l available processors 32
l (32 * 3 / 2) + 1 = 49 ( Thread Pool )
l processors (PM : 40)
l (40 * 3) / 2 + 1 = 61
l Queue (1000 → 3000)
l thread_pool.search.queue_size: 3000
l ES5 ES6

25. Developer Meetup #33

26. Developer Meetup #33
(2018/1/2 21:00~23:00)

27. Developer Meetup #33
(2018/1/9 21:00~23:00)

28. Developer Meetup #33
l Master node Data node
l service admin zone
l Data Node(PM)
l

29. ˆElasticsearch

30. Developer Meetup #33
l
l
l
l IMON (Monitoring + Alerting)
l Java
l PMC

31. Developer Meetup #33
l Prometheus (Alerting) + Grafana (Monitoring) ← NEW
l line/promgen
l LINE Prometheus
l
l
l
l IMON (Monitoring + Alerting)
l Java
l PMC

32. Developer Meetup #33
Prometheus Architecture
: https://prometheus.io/docs/introduction/overview/#architecture

33. Developer Meetup #33
Prometheus + Grafana + Promgen
l Prometheus
l Pull
l agent
l *_exporter
l http
l EXPORTERS AND INTEGRATIONS
l Grafana
l Prometheus DataSource
l exporter Overview Dashboard
l Promgen
l line/promgen
l LINE Prometheus

34. Developer Meetup #33
Elasticsearch Prometheus
l elasticsearch_exporter
l justwatchcom/elasticsearch_exporter
l GET /_nodes/stats Prometheus
exporter
l Promgen
Prometheus
l Alertmanager
push post
l
l e.g. elaseticsearch_cluster_health_status{color=“red”} == 1

35. Developer Meetup #33

36. Developer Meetup #33
Promgen( )

37. Developer Meetup #33
l Prometheus + Grafana
l
l line/promgen
l LINE Engineering Blog
l PromCon 2017: Prometheus as an (internal) Service
l Prometheus Casual Talks

38. ˆ

39. Developer Meetup #33
l Client
l Transport Client High Revel REST Client
l Elasticsearch
l 5 6
l

40. Developer Meetup #33
l Elasticsearch5.x 6.x
l Java Client (TransportClient → High Level Rest Client)

41. Developer Meetup #33
5.X 6.X
l Elasticsearch
l Upgrade Elasticsearch
l 6.x Rolling Upgrade 5.6
l 5.x 5.6 Rolling Upgrade
l
l 5.x -> 5.6 -> 6.x
l Upgrade
l …
l 1index 1type
l _all
l Etc.
l Kibana Upgrade-Assistant

42. Developer Meetup #33
Java Client
l Transport Client …
l Deprecate Elasticsearch7.0
l Remove: Elasticsearch8.0
l Client
l High Level Rest Client
l Migration Guide
l
l Transport Client ES
l
l 5.6.x 6.x

43. ˆ

44. Developer Meetup #33
THANK YOU