The ELK Stack - Get to Know Logs

3,215 views

Published on

This presentation deals with logging in the course of mobile development, namely describing the open source logging environment built with ELK stack (ElasticSearch, Logstash and Kibana).

Presentation by Igor Rudyk (Software Engineer, GlobalLogic, Lviv), delivered at Mobile TechTalk Lviv on April 28, 2015.

More details - http://globallogic.com.ua/mobile-techtalk-lviv-2015-report

Published in: Engineering

The ELK Stack - Get to Know Logs

  1. 1. ©2015 GlobalLogic Inc. CONFIDENTIAL The ELK stack - get to know logs Igor Rudyk DevOps / System Integrator
  2. 2. 2 CONFIDENTIAL Agenda • Introduction. What is ELK, and why do we need it? • The ELK stack Logstash ElasticSearch Kibana •Architecture •Demo
  3. 3. 3 CONFIDENTIAL Can you check the errors from yesterday between 9:09 and 9:27
  4. 4. 4 CONFIDENTIAL So what’s a log
  5. 5. 5 CONFIDENTIAL Log = timestamp + data
  6. 6. 6 CONFIDENTIAL Lifecycle of a log Delete Real Real
  7. 7. 7 CONFIDENTIAL Default problem • Multiple log time formats •Apr 28 20:21:59 •[27/Apr/2015:07:05:28 +0000] •071012 09:27:32 •Mon, 27-Apr-15 06:27:02 UTC •2015-04-28 20:07:51 +0000 • Starts not with timestamp or without timestamp •Error messages with really unhelpful info •No rotation •No scaling Tools? • grep • awk / sed / cut •less / tail •vi / vim •regular expression •...
  8. 8. 8 CONFIDENTIAL Logging Solutions Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial Logstash Logstash shipper or logstash-forwarder (Lumberjack) RPM installation Logstash shipper or logstash- forwarder (Lumberjack, encrypted transport is the default) RPM installation Output plugins Central server-master with a hot- standby in case of failure Codecs plugins Grok debugger ElasticSearch, MongoDB, AWS S3 and much more Kibana, graylog2 Riemann Kibana, graylog2 NO fluentd Input plugins Install from source or via gem Output plugins Load-balance between multiple hosts or have a master with a hot- standby in case of failure Plugins Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected Kibana, graylog2 Riemann Kibana, graylog2 NO splunk Splunk Universal Forwarder RPM installation Splunk Universal Forwarder SSL security Splunk Splunk Splunk Splunk Splunk YES Graylog2 Graylog2 Logstash Graylog2 Logstash Graylog2 ElasticSearch Kibana, graylog2 graylog2 graylog2 NO loggly loggly loggly loggly Hosted loggly loggly loggly YES
  9. 9. 9 CONFIDENTIAL What is ELK, and why do we need it? • ELK is a stack of programs that help dealing with logs. • Includes: – Aggregation of logs – Search capabilities – Aggregation of statistics – Visualizations
  10. 10. 10 CONFIDENTIAL Logstash Unstructured Filters Outputs Documents
  11. 11. 11 CONFIDENTIAL Inputs ➔ Logs: ● Lumberjack - resilient, compressed, secure (logstash-forwarder) ● Remote syslog ● Files ➔ Devices: ● Event log, Collectd ● Netflow, WMI ➔ Event Queue: ● Redis, RabbitMQ ● Kafka, ZeroMQ ➔ Streaming APIs: ● Twitter ➔ Email (IMAP) ➔ Amazon S3, ganglia, sqs, varnishlog, etc ➔ .... http://logstash.net/docs/1.4.2/ - Full list
  12. 12. 12 CONFIDENTIAL Filters➔ grep ➔ date ➔ json ➔ grok ➔ .... http://logstash.net/docs/1.4.2/ - Full list Why Do I like Logstash? It uses Grok filter for parsing standard and non standard logs: Log Line: 27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0 Pattern: %{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}
  13. 13. 13 CONFIDENTIAL Outputs ➔ Storage: ● ElasticSearch ● MongoDB ● S3 ● Graphite ● File ● ... ➔ Notification: ● Zabbix ● Nagios ● Riemann ● PagerDuty ● Email ➔ Event Queue: ● Redis, RabbitMQ ● Kafka, ZeroMQ ● tcp/udp ➔ SaaS: ● AWS CloudWatch ● Hipchat ● Jira ➔ .... http://logstash.net/docs/1.4.2/ - Full list
  14. 14. 14 CONFIDENTIAL Logstash - Forwarder (Shipper) Configuration file{ "network": { "servers": [[logstash_indexers]] "timeout": 15, "ssl ca": "logstash-forwarder.crt" }, "files": [ { "paths": [ "/usr/share/tomcat7/logs/*.json.log" ], "fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } }, { "paths": [ "/usr/share/tomcat7/logs/*.activities.log" ], "fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } } ] }
  15. 15. 15 CONFIDENTIAL Logstash-Indexer Configuration file input { lumberjack { codec => json{} port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } output { elasticsearch { host => "127.0.0.1" protocol => "http" cluster => "[[elasticsearch_cluster_name]]" manage_template => false index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" } }
  16. 16. 16 CONFIDENTIAL ElasticSearch Configuration file (yaml-based configuration) cluster.name: [[elasticsearch_cluster_name]] node.name: "[[node_name]]" node.master: false / true node.data: false / true index.number_of_replicas: 1 #Security discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [[elasticsearch_servers]] action.disable_close_all_indices: true action.disable_delete_all_indices: true action.disable_shutdown: true script.disable_dynamic: true
  17. 17. 17 CONFIDENTIAL The ELK stack General Architecture Logstash ElasticSearch Kibana
  18. 18. 18 CONFIDENTIAL Kibana ElasticSearch Logstash-Forwarder Logstash-Indexer Logstash-Indexer Logstash-Indexer The ELK stack Our Scaled Architecture ElasticSearch ElasticSearch Kibana Kibana
  19. 19. ©2015 GlobalLogic Inc. CONFIDENTIAL DEMO

×