ELK (Elasticsearch, Logstash, Kibana) is an open source toolset for centralized logging, where Logstash collects, parses, and filters logs, Elasticsearch stores and indexes logs for search, and Kibana visualizes logs. Logstash processes logs through an input, filter, output pipeline using plugins. It can interpret various log formats and event types. Elasticsearch allows real-time search and scaling through replication/sharding. Kibana provides browser-based dashboards and visualization of Elasticsearch query results.

1. Log Management with ELK
ELASTICSEARCH, LOGSTASH, KIBANA FOR CENTRALIZED LOGS

2. Purpose
Centralized Log Management
◦ Collect, Parse and Filter using Logstash
◦ Store, Index and Search using Elasticsearch
◦ Visualize using Kibana
Full open source stack
◦ Use for free
◦ Support plan from Elasticsearch company

3. Elasticsearch
Real-time search engine
◦ Based on Apache Solr/Lucene
◦ Pure Java
◦ Document database
◦ Advanced text indexing
◦ Fuzzy search
◦ Replication/Sharding for true scalability

4. Logstash
JRuby Based log processor
Pluggable event pipeline
◦ Input plugins
◦ Filter plugins
◦ Codec plugins
◦ Output plugins
DevOps Comunity
◦ Mix of developers, operations and system administrators

5. Kibana
Browser based dashboard for ElasticSearch
Visualization of query results
◦ Time Charts
◦ Filter any field
◦ Compare subsets

6. Logstash pipeline
Define input, filters and outputs
Simple configuration file
Ruby syntax

7. Logstash not just for logs
Interpretes different log formats
◦ Syslog messages
◦ Log4j with full details
◦ Apache log files
Other event types too
◦ Ganglia server monitoring events
◦ SNMP events
◦ Windows EventLog
Pre-proces before sending
◦ lumberjack

8. Logstash Inputs
Rsyslog via TCP/UDP
Log4j appender
JMX Listener
Logstash-forwarder
File tails
SNMP
Ganglia
…

9. Logstash Filters

10. Grok Filter
Readable regex
Predefined patterns for common log data
Extract to properties
◦ Indexed properties

11. Metrics Filter
Aggregate metrics
◦ Event rate using sliding windows
◦ 1 min
◦ 5 min
◦ 15 min
◦ Min/max/stddev/percentiles

12. Logstash Outputs

13. Statsd output
Node.js based
◦ Counters
◦ Timers
◦ Graphite frontend

14. Alert outputs
Send alerts
◦ Email
◦ Pagerduty
◦ XMPP/Jabber
◦ Hipchat
◦ Nagios
Use treshold from metrics filter

15. ElasticSearch output
Auto-creates new index per day
Index all recognized fields
Full text index, customizable indexer, mapper

16. Kibana dashboard

17. Scalability
Easy deployment using chef/puppet/docker