渡邊 紘太朗 (LINE株式会社 開発支援室) LINEデリマでのElasticsearchの運用と監視の話 LINE Developer Meetup in Tokyo #27 -Elastic-の発表資料です https://line.connpass.com/event/76226/

1. Developer Meetup #27
LINE Elasticsearch
Kotaro WATANABE, Development Support 2018/01/30

2. Developer Meetup #27
l (Kotaro WATANABE)
l (Development Support)
l Infra Engineer (2016/04~)
l
l LINE
l LINE LIVE
l LINE
l LINE
l LINE
l Clova
l Etc.

3. Developer Meetup #27
l LINE
l Elasticsearch
l Elasticsearch
l
l

4. Developer Meetup #27
l LINE Elasticsearch
LINE

5. ˆLINE

6. Developer Meetup #27
l LINE
l
l 2017 7 26
l LINE : @linedelima
l 1100
l Elasticsearch
LINE

7. Developer Meetup #27
LINE ( )

8. Developer Meetup #27
LINE ( )

9. Developer Meetup #27
LINE

10. Developer Meetup #27
l Java Client
l Transport Client
l Indices
l block ( )
l item ( )
l shop ( )
l Dashboard
l Imenezes/cerebro

11. ˆElasticsearch

12. Developer Meetup #27
LINE
l Topics
1.
2. Admin Elasticsearch
3.

13. Developer Meetup #27
1.
l
l Elasticsearch
l 1 4000rps
l 2

14. Developer Meetup #27

15. Developer Meetup #27
l
l Master node
l
l VM 3
l discovery.zen.minimum_master_nodes: 2
l Primary shard Replica shard
l Primary shard: 5 (Data node )
l Replica shard: 1
l 1node 2 shard

16. Developer Meetup #27

17. Developer Meetup #27
l 20
l Master node
l Master node Data node
l discovery.zen.minimum_master_nodes
l (master_eligible_nodes / 2) + 1 = 2
l batch indexing rebalance allocation
l
l Shard Allocation Settings
l 5Mbps 20Mbps

18. Developer Meetup #27
2. Admin Elasticsearch
l Admin Elasticsearch
l Data node
l Forced Awareness
l /path/to/elasticsearch.yml
l e.g. /etc/elasticsearch/elasticsearch.yml
# for service zone
node.attr.zone: service
cluster.routing.allocation.awareness.force.zone.values: service,admin
cluster.routing.allocation.awareness.attributes: zone
# for admin zone
node.attr.zone: admin
cluster.routing.allocation.awareness.force.zone.values: service,admin
cluster.routing.allocation.awareness.attributes: zone

19. Developer Meetup #27
zone

20. Developer Meetup #27
zone
l zone index
l zone shard replica
l Primary shard Replica shard
l shard rebalance
l zone index
l zone Elasticsearch zone
l admin

21. Developer Meetup #27
3.
l
l
l 4000qps
l search Thread queue

22. Developer Meetup #27
l
l Thread Pool
l int((available processors * 3) / 2) + 1
l available processors 32
l (32 * 3 / 2) + 1 = 49 ( Thread Pool )
l processors (PM : 40)
l (40 * 3) / 2 + 1 = 61
l Queue (1000 → 3000)
l thread_pool.search.queue_size: 3000
l ES5 ES6

23. Developer Meetup #27

24. Developer Meetup #27
(2018/1/2 21:00~23:00)

25. Developer Meetup #27
(2018/1/9 21:00~23:00)

26. Developer Meetup #27
l Master node Data node
l service admin zone
l Data Node(PM)
l

27. ˆElasticsearch

28. Developer Meetup #27
l
l
l
l IMON (Monitoring + Alerting)
l Java
l PMC

29. Developer Meetup #27
l Prometheus (Alerting) + Grafana (Monitoring) ← NEW
l line/promgen
l LINE Prometheus
l
l
l
l IMON (Monitoring + Alerting)
l Java
l PMC

30. Developer Meetup #27
Prometheus Architecture
: https://prometheus.io/docs/introduction/overview/#architecture

31. Developer Meetup #27
Prometheus + Grafana + Promgen
l Prometheus
l Pull
l agent
l *_exporter
l http
l EXPORTERS AND INTEGRATIONS
l Grafana
l Prometheus DataSource
l exporter Overview Dashboard
l Promgen
l line/promgen
l LINE Prometheus

32. Developer Meetup #27
Elasticsearch Prometheus
l elasticsearch_exporter
l justwatchcom/elasticsearch_exporter
l GET /_nodes/stats Prometheus
exporter
l Promgen
Prometheus
l Alertmanager
push post
l
l e.g. elaseticsearch_cluster_health_status{color=“red”} == 1

33. Developer Meetup #27

34. Developer Meetup #27
Promgen( )

35. Developer Meetup #27
l Prometheus + Grafana
l
l line/promgen
l LINE Engineering Blog
l PromCon 2017: Prometheus as an (internal) Service
l Prometheus Casual Talks

36. ˆ

37. Developer Meetup #27
l Client
l Transport Client High Revel REST Client
l Elasticsearch
l 5 6
l

38. ˆ

39. Developer Meetup #27
THANK YOU