©2015 GlobalLogic Inc. CONFIDENTIAL
The ELK stack - get to know logs
Igor Rudyk
DevOps / System Integrator
2 CONFIDENTIAL
Agenda
• Introduction. What is ELK, and why do we need it?
• The ELK stack
Logstash
ElasticSearch
Kibana...
3 CONFIDENTIAL
Can you check the errors from yesterday between 9:09 and 9:27
4 CONFIDENTIAL
So what’s a log
5 CONFIDENTIAL
Log = timestamp + data
6 CONFIDENTIAL
Lifecycle of a log
Delete
Real Real
7 CONFIDENTIAL
Default problem
• Multiple log time formats
•Apr 28 20:21:59
•[27/Apr/2015:07:05:28 +0000]
•071012 09:27:32...
8 CONFIDENTIAL
Logging Solutions
Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial
L...
9 CONFIDENTIAL
What is ELK, and why do we need it?
• ELK is a stack of programs that help dealing with logs.
• Includes:
–...
10 CONFIDENTIAL
Logstash
Unstructured
Filters Outputs
Documents
11 CONFIDENTIAL
Inputs
➔ Logs:
● Lumberjack - resilient, compressed, secure (logstash-forwarder)
● Remote syslog
● Files
➔...
12 CONFIDENTIAL
Filters➔ grep
➔ date
➔ json
➔ grok
➔ ....
http://logstash.net/docs/1.4.2/ - Full list
Why Do I like Logsta...
13 CONFIDENTIAL
Outputs
➔ Storage:
● ElasticSearch
● MongoDB
● S3
● Graphite
● File
● ...
➔ Notification:
● Zabbix
● Nagio...
14 CONFIDENTIAL
Logstash - Forwarder (Shipper)
Configuration file{
"network": {
"servers": [[logstash_indexers]]
"timeout"...
15 CONFIDENTIAL
Logstash-Indexer
Configuration file
input {
lumberjack {
codec => json{}
port => 5000
type => "logs"
ssl_c...
16 CONFIDENTIAL
ElasticSearch
Configuration file (yaml-based configuration)
cluster.name: [[elasticsearch_cluster_name]]
n...
17 CONFIDENTIAL
The ELK stack
General Architecture
Logstash ElasticSearch Kibana
18 CONFIDENTIAL
Kibana
ElasticSearch
Logstash-Forwarder
Logstash-Indexer
Logstash-Indexer
Logstash-Indexer
The ELK stack
O...
©2015 GlobalLogic Inc. CONFIDENTIAL
DEMO
Upcoming SlideShare
Loading in …5
×

The ELK Stack - Get to Know Logs

3,510 views

Published on

This presentation deals with logging in the course of mobile development, namely describing the open source logging environment built with ELK stack (ElasticSearch, Logstash and Kibana).

Presentation by Igor Rudyk (Software Engineer, GlobalLogic, Lviv), delivered at Mobile TechTalk Lviv on April 28, 2015.

More details - http://globallogic.com.ua/mobile-techtalk-lviv-2015-report

Published in: Engineering

The ELK Stack - Get to Know Logs

  1. 1. ©2015 GlobalLogic Inc. CONFIDENTIAL The ELK stack - get to know logs Igor Rudyk DevOps / System Integrator
  2. 2. 2 CONFIDENTIAL Agenda • Introduction. What is ELK, and why do we need it? • The ELK stack Logstash ElasticSearch Kibana •Architecture •Demo
  3. 3. 3 CONFIDENTIAL Can you check the errors from yesterday between 9:09 and 9:27
  4. 4. 4 CONFIDENTIAL So what’s a log
  5. 5. 5 CONFIDENTIAL Log = timestamp + data
  6. 6. 6 CONFIDENTIAL Lifecycle of a log Delete Real Real
  7. 7. 7 CONFIDENTIAL Default problem • Multiple log time formats •Apr 28 20:21:59 •[27/Apr/2015:07:05:28 +0000] •071012 09:27:32 •Mon, 27-Apr-15 06:27:02 UTC •2015-04-28 20:07:51 +0000 • Starts not with timestamp or without timestamp •Error messages with really unhelpful info •No rotation •No scaling Tools? • grep • awk / sed / cut •less / tail •vi / vim •regular expression •...
  8. 8. 8 CONFIDENTIAL Logging Solutions Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial Logstash Logstash shipper or logstash-forwarder (Lumberjack) RPM installation Logstash shipper or logstash- forwarder (Lumberjack, encrypted transport is the default) RPM installation Output plugins Central server-master with a hot- standby in case of failure Codecs plugins Grok debugger ElasticSearch, MongoDB, AWS S3 and much more Kibana, graylog2 Riemann Kibana, graylog2 NO fluentd Input plugins Install from source or via gem Output plugins Load-balance between multiple hosts or have a master with a hot- standby in case of failure Plugins Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected Kibana, graylog2 Riemann Kibana, graylog2 NO splunk Splunk Universal Forwarder RPM installation Splunk Universal Forwarder SSL security Splunk Splunk Splunk Splunk Splunk YES Graylog2 Graylog2 Logstash Graylog2 Logstash Graylog2 ElasticSearch Kibana, graylog2 graylog2 graylog2 NO loggly loggly loggly loggly Hosted loggly loggly loggly YES
  9. 9. 9 CONFIDENTIAL What is ELK, and why do we need it? • ELK is a stack of programs that help dealing with logs. • Includes: – Aggregation of logs – Search capabilities – Aggregation of statistics – Visualizations
  10. 10. 10 CONFIDENTIAL Logstash Unstructured Filters Outputs Documents
  11. 11. 11 CONFIDENTIAL Inputs ➔ Logs: ● Lumberjack - resilient, compressed, secure (logstash-forwarder) ● Remote syslog ● Files ➔ Devices: ● Event log, Collectd ● Netflow, WMI ➔ Event Queue: ● Redis, RabbitMQ ● Kafka, ZeroMQ ➔ Streaming APIs: ● Twitter ➔ Email (IMAP) ➔ Amazon S3, ganglia, sqs, varnishlog, etc ➔ .... http://logstash.net/docs/1.4.2/ - Full list
  12. 12. 12 CONFIDENTIAL Filters➔ grep ➔ date ➔ json ➔ grok ➔ .... http://logstash.net/docs/1.4.2/ - Full list Why Do I like Logstash? It uses Grok filter for parsing standard and non standard logs: Log Line: 27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0 Pattern: %{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}
  13. 13. 13 CONFIDENTIAL Outputs ➔ Storage: ● ElasticSearch ● MongoDB ● S3 ● Graphite ● File ● ... ➔ Notification: ● Zabbix ● Nagios ● Riemann ● PagerDuty ● Email ➔ Event Queue: ● Redis, RabbitMQ ● Kafka, ZeroMQ ● tcp/udp ➔ SaaS: ● AWS CloudWatch ● Hipchat ● Jira ➔ .... http://logstash.net/docs/1.4.2/ - Full list
  14. 14. 14 CONFIDENTIAL Logstash - Forwarder (Shipper) Configuration file{ "network": { "servers": [[logstash_indexers]] "timeout": 15, "ssl ca": "logstash-forwarder.crt" }, "files": [ { "paths": [ "/usr/share/tomcat7/logs/*.json.log" ], "fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } }, { "paths": [ "/usr/share/tomcat7/logs/*.activities.log" ], "fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } } ] }
  15. 15. 15 CONFIDENTIAL Logstash-Indexer Configuration file input { lumberjack { codec => json{} port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } output { elasticsearch { host => "127.0.0.1" protocol => "http" cluster => "[[elasticsearch_cluster_name]]" manage_template => false index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" } }
  16. 16. 16 CONFIDENTIAL ElasticSearch Configuration file (yaml-based configuration) cluster.name: [[elasticsearch_cluster_name]] node.name: "[[node_name]]" node.master: false / true node.data: false / true index.number_of_replicas: 1 #Security discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [[elasticsearch_servers]] action.disable_close_all_indices: true action.disable_delete_all_indices: true action.disable_shutdown: true script.disable_dynamic: true
  17. 17. 17 CONFIDENTIAL The ELK stack General Architecture Logstash ElasticSearch Kibana
  18. 18. 18 CONFIDENTIAL Kibana ElasticSearch Logstash-Forwarder Logstash-Indexer Logstash-Indexer Logstash-Indexer The ELK stack Our Scaled Architecture ElasticSearch ElasticSearch Kibana Kibana
  19. 19. ©2015 GlobalLogic Inc. CONFIDENTIAL DEMO

×