Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ELK
Ansible + Vagrant 10 ELK
rueiancsie@gmail.com
2015/10/17
•
• API
•
• ...
• DAU
•
•
•
• Nagios
Zabbix
• elastic.co
elasticsearch
elasticsearch
• elasticsearch shard inverted
index : http://
www.slideshare.net...
ELK
• ELK = elasticsearch + logstash + kibana
• elastic.co
elasticsearch logstash
kibana
• elasticsaerch logstash kibana
E...
logstash
• logstash
( RSS)
elasticsearch
• logstash Java
Fluentd
https://www.elastic.co/products/logstash
logstash-forwarder
• logstash-forwarder elastic.co logstash
go
logstash
• logstash-forwarder
elastic.co filebeat
https://gi...
kibana
• kibana Hapi Node.js
Web UI elasticsearch
• kibana Grafana Grafana
Graphite v2.5
elasticsearch
https://www.elastic...
kibana
beats
• logstash elastic.co
beats
• packetbeat HTTP
Thrift-RPC Mysql PostgreSQL MongoDB Redis
Memcache protocol
• topbeat ...
Marvel
• Marvel Web UI elasticsearch
elasticsearch cluster
CPU
Index
https://www.elastic.co/products/marvel
Marvel
watcher
•
• watcher elastic.co elasticsearch
email slack webhook
• watcher
https://www.elastic.co/products/watcher
logstash
https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
logstash
elasticsearch
elasticsaerch
https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
logstash
Redis RabbitMQ
logstash
logstash
packetbeat & topbeat
packetbeat elasticsaerch
https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash...
packetbeat & topbeat
Redis logstash
elasticsearch logstash
https://www.elastic.co/guide/en/beats/packetbeat/current/packet...
Marvel Watcher
• elastic.co Marvel Watcher
elasticsearch
• elasticsearch
https://www.elastic.co/guide/en/watcher/watcher-1...
nginx
nodejs
nodejs
nodejs
postgres
postgres
elastic
search
elastic
search
elastic
search
elastic
search
monitor
logstash
...
elasticsearch
• elasticsearch
• Index shard 5 5 elasticsearch
shard
• ES_HEAP_SIZE elasticsearch
• 64000
1024 elasticsearc...
elasticsearch
•
elasticsearch
• Index Curator
• Index Curator
elasticsearch
repository ( S3)
https://www.elastic.co/guide/...
elasticseach
• elasticsearch
elasticsearch
• logstash
statsD Graphite Graphite
1s 1m
ELK
• Ansible + Vagrant ELK
https://github.com/rueian/ansible-elk-example
• vagrant up
kibana
README
• Vagrant
ELK
nginx +
nodejs
postgres
elastic
search
elastic
search
redis +
logstash
logstash
logstash
elastic
search
kibana
redis +...
• [ ] Fluentd: An data collector for unified logging layer
• [ ] StatsD: Simple daemon for easy stats aggregation
• [ ] Gra...
Thanks
Any Questions?
rueiancsie@gmail.com
Upcoming SlideShare
Loading in …5
×

Elasitcsearch + Logstash + Kibana 日誌監控

4,204 views

Published on

介紹 elasticsearch, logstash, kibana, logstash-forwarder, topbeat, packetbeat 等工具,以及如何部署,進行 log analysis & monitoring。
並且附上一個使用 Ansible + Vagrant 部署 10 個節點的 ELK 系統的完整範例。
https://github.com/rueian/ansible-elk-example

Published in: Data & Analytics

Elasitcsearch + Logstash + Kibana 日誌監控

  1. 1. ELK Ansible + Vagrant 10 ELK rueiancsie@gmail.com 2015/10/17
  2. 2. • • API • • ...
  3. 3. • DAU • • •
  4. 4. • Nagios Zabbix • elastic.co elasticsearch elasticsearch • elasticsearch shard inverted index : http:// www.slideshare.net/rueian3/elasticsearch-45855699
  5. 5. ELK • ELK = elasticsearch + logstash + kibana • elastic.co elasticsearch logstash kibana • elasticsaerch logstash kibana ELK logstash-forwarder beats watcher
  6. 6. logstash • logstash ( RSS) elasticsearch • logstash Java Fluentd https://www.elastic.co/products/logstash
  7. 7. logstash-forwarder • logstash-forwarder elastic.co logstash go logstash • logstash-forwarder elastic.co filebeat https://github.com/elastic/logstash-forwarder
  8. 8. kibana • kibana Hapi Node.js Web UI elasticsearch • kibana Grafana Grafana Graphite v2.5 elasticsearch https://www.elastic.co/products/kibana
  9. 9. kibana
  10. 10. beats • logstash elastic.co beats • packetbeat HTTP Thrift-RPC Mysql PostgreSQL MongoDB Redis Memcache protocol • topbeat process CPU • filebeat logstash-forwarder https://www.elastic.co/products/beats
  11. 11. Marvel • Marvel Web UI elasticsearch elasticsearch cluster CPU Index https://www.elastic.co/products/marvel
  12. 12. Marvel
  13. 13. watcher • • watcher elastic.co elasticsearch email slack webhook • watcher https://www.elastic.co/products/watcher
  14. 14. logstash https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
  15. 15. logstash elasticsearch elasticsaerch https://www.elastic.co/guide/en/logstash/2.0/deploying-and-scaling.html
  16. 16. logstash Redis RabbitMQ
  17. 17. logstash logstash
  18. 18. packetbeat & topbeat packetbeat elasticsaerch https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash.html
  19. 19. packetbeat & topbeat Redis logstash elasticsearch logstash https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-logstash.html
  20. 20. Marvel Watcher • elastic.co Marvel Watcher elasticsearch • elasticsearch https://www.elastic.co/guide/en/watcher/watcher-1.0/installing-watcher.html https://www.elastic.co/guide/en/marvel/current/_installation.html
  21. 21. nginx nodejs nodejs nodejs postgres postgres elastic search elastic search elastic search elastic search monitor logstash redis borker logstash logstash logstash-forwarder logstash redis borker packetbeat & topbeat elastic search monitor
  22. 22. elasticsearch • elasticsearch • Index shard 5 5 elasticsearch shard • ES_HEAP_SIZE elasticsearch • 64000 1024 elasticsearch • JVM swapping elasticsearch elastic.co swapping 1. sudo swapoff -a swapping 2. sysctl vm.swappiness 0 3. elasticsearch bootstrap.mlockall: true https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html
  23. 23. elasticsearch • elasticsearch • Index Curator • Index Curator elasticsearch repository ( S3) https://www.elastic.co/guide/en/elasticsearch/client/curator/current/_features.html
  24. 24. elasticseach • elasticsearch elasticsearch • logstash statsD Graphite Graphite 1s 1m
  25. 25. ELK • Ansible + Vagrant ELK https://github.com/rueian/ansible-elk-example • vagrant up kibana README • Vagrant
  26. 26. ELK nginx + nodejs postgres elastic search elastic search redis + logstash logstash logstash elastic search kibana redis + logstash logstash-forwarder packetbeat ELK redis + logstash topbeat
  27. 27. • [ ] Fluentd: An data collector for unified logging layer • [ ] StatsD: Simple daemon for easy stats aggregation • [ ] Grafana: Gorgeous metric viz, dashboards & editors for Graphite, InfluxDB & OpenTSDB • [ ] Graphite: Scalable Realtime Graphing • [ ] Nagios: The Industry Standard In IT Infrastructure Monitoring • [ ] Zabbix: The Enterprise-class Monitoring Solution for Everyone • [ ] QBox: Optimizing Elasticsearch: How Many Shards per Index? • [ ] QBox: What is Elasticsearch, and How Can I Use It? • [ ] Elasticsearch • [ ] Ansible: Application Deployment + Configuration Management + Continuous Delivery • [ ] Vagrant: Create and configure lightweight, reproducible, and portable development environments
  28. 28. Thanks Any Questions? rueiancsie@gmail.com

×