The document outlines log scaling and analytics using Logstash, addressing common logging challenges such as load on databases and difficulties in log format management. It describes Logstash as an open-source tool that processes, filters, and outputs logs to various destinations while integrating with Elasticsearch for indexing and Kibana for analytics. Additionally, it provides details on the architecture, plugins, and configuration necessary for effective log management and data analysis.

1. Log Scaling and Analytics With
Logstash
Richard Viet
Principal Engineer
Cloud Elements

2. Problems

Logging to a database or filesystem

Logging has placed a load on the database and
filesystem.

Multiple log formats

No easy way to search logs

No easy method to gather statistics

3. Logstash

Open source, Apache licence

Written in JRuby. Runs on jvm.

Plugins easily written in Ruby.

Part of the Elasticsearch family.

www.logstash.net

4. Logstash

Scalable: Elasticsearch for indexing, search
and retrieval

Process multiple log formats

Receive logs from multiple sources

Output logs to multiple destinations

Kibana provides web interface for search and
analytics

Easily extended with plugins written in Ruby

5. Logstash Architecture
Shipper
Broker Indexer
Search
Storage
Shipper
Shipper
Web
Interface

6. Logstash Pipeline

Input → filters → output

Separate threads

Filters are applied in order of config file

Outputs processed in order of config file

7. Logstash Plugins

Input – read input stream
– File input
– Log4j
– Redis
– Syslog

Codecs – decoding log messages
– Json
– Multiline

8. Logstash Plugins

Filters – processing messages
– Csv – define fields in a csv
– Date – define date field formats
– Mutate – change date type
– Xml – extract xml
– Grok – parses arbitrary text

9. Logstash Plugins

Output
– Elasticsearch
– Elasticsearch_http
– Mongodb
– Email
– Nagios

10. Indexer

Send message to Elasticsearch for indexing

An index is created for each day

Each index split into 5 shards by default

Original message is stored

Each field indexed

11. Elasticsearch

Apache Lucene search engine

An elasticsearch index is made up of multiple
shards

Each shard is a lucene index

Primary shard and at least one replica

Shards are moved between servers when
servers are added or removed

12. Elasticsearch Configuration

Self discovery
– Multicast
• Simplest if all nodes on same network
– Unicast
• Provide a list of servers
– Combination

13. Elasticsearch

Adding more nodes improves indexing and
search time.

Primary node is indexed first then replicas

Number of shards determined when index is
created.

Number of replicas is configurable

14. Elasticsearch

Adding more nodes improves indexing and
search time.

Primary node is indexed first then replicas

Number of shards determined when index is
created.

Number of replicas is configurable

15. Kibana

Browser based analytics for time-stamped data

Included in the logstash jar

Connect to the logstash server port 9292

Sends multiple requests to avoid overloading
the server.

16. Log4j to Logstash
App
Logstash Redis
Elasticsearch
Cluster
App
App Logstash

17. Logstash Log4j Server

Configure logstash as a Log4j server
input {
log4j {
mode => "server"
port => 9501
}
}

18. Send to a broker

Configure broker
output {
stdout {}
redis {
host => "redis1"
data_type => "list"
key => "logstash"
}
}

19. Indexing
input {
redis {
host => “redis”
data_type => “list”
key => “logstash”
}}
output {
elasticsearch {
cluster => “logstash”
host => "elasticsearch"
port => "9200"
}}

20. Scaling
Broker
Indexer
Search
Storage
Shipper
Web
Interface
Broker
Indexer

21. Sending to Broker
output {
stdout {}
redis {
host => ["redis1", “redis2”]
data_type => "list"
shuffle_hosts => true
key => "logstash"
}
}

22. Indexing
input {
redis {
host => “redis1”
data_type => “list”
key => “logstash”
redis {
host => “redis2”
data_type => “list”
key => “logstash”
}
}
output { ...

23. Quick Start

Logstash, elasticsearch and kibana configured
to run from the logstash jar

Download and untar

bin/logstash agent -f config.file

bin/logstash web