The document discusses using Logstash, Elasticsearch, and Kibana as tools for log collection, parsing, storage, and search. It provides configuration examples for processing logs from HTTPd and Tomcat servers and outlines various best practices for logging implementations. Additionally, it includes resources for further learning and emphasizes the importance of security and flexibility in logging applications.

1. Using Logstash, ElasticSearch and Kibana
Alejandro E Brito Monedero
@ae_bm
2013 / 05 / 23

2. Business as usual

4. So many hosts to check
Is there life out there?
http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg

5. Time to play whack a log
http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif

6. http://brotality.com/wp-content/uploads/2012/12/madness.jpg

7. I need a new toy
http://www.youtube.com/watch?v=8L6Dpq5kY_A
Video time

8. Logstash ✔ collects logs
✔ parses logs
✔ stores logs
✔ indexes logs
✔ searches logs
✔ and fixes timestamps
You only need
● JVM
● logstash.jar

9. $ log_producer | grep ... | sed … | awk … | tee output
| sort | uniq -c | sort -n
Log source Logstash
Logstash
(optional)
ElasticSearch
DB
statsd
Pipes

10. Inputs Filters Outputs
File
Redis
Syslog
Lumberjack
Rabbitmq
SQS
…
Alter
Date
Grok
Multiline
Grep
...
AMQP
Cloudwatch
Elasticsearch
Mongodb
Redis
File
...
Plugins
Not here yet? JRuby to the rescue

11. ElasticSearch
Distributed RESTful search server
● Near real-time search
● RESTful API
● Easy to scale horizontally
● HA
● Full text search
● YAML config file / JSON format!!
● Document oriented JSON
Getting started: Logstash JAR includes it / download and set cluster.name
This is where it will be worth to spend some time tuning

12. Kibana
✔ Nice UI
✔ Better than the old frontend logstash
included
✔ Ruby / framework Sinatra
Web frontend to search / graph and more

13. Original plan
Apache
(ligthweight shipper)
Tomcat
(ligthweight shipper)
broker logstash
ElasticSearch
Kibana

14. After a few workarounds
Apache
(logstash shipper)
Tomcat
(logstash shipper)
Logstash ElasticSearch
Kibana
SSH tunnels

15. Example config 1/3
Logstash-httpd.conf
input {
file {
type => "httpd"
path => ["/var/log/httpd/*-logstash.log"]
exclude => ["*.gz"]
start_position => "beginning"
format => "json_event"
}
}
output {
tcp {
host => "0.0.0.0"
mode => "server"
port => 1666
}
}
Logstash-server.conf
input {
tcp {
type => "httpd"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "1666"
}
tcp {
type => "app"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "2666"
}
}
output {
elasticsearch {
cluster => "logstash"
}
}

16. Example config 2/3
Logstash-tomcat.conf
filter {
# Tomcat
# Remove blank lines
grep {
type => "tomcat"
match => [ "@message", "(.+)" ]
drop => true
add_tag => [ "no_blank_lines" ]
}
# make the multilines be treated like a single line
multiline {
type => "tomcat"
pattern => "^dddd"
negate => true
what => "previous"
}

17. Example config 3/3
Logstash-tomcat.conf
# mark the exceptions (multiline)
grep {
type => "tomcat"
tags => [ "multiline" ]
match => [ "@message", ".+Exception: .+" ]
drop => false
add_tag => [ "java_exception" ]
}
# get the log level, operation id, module and timestamp as separated fields
grok {
type => "tomcat"
pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."
add_tag => [ "groked" ]
}
# fix the timestamp
date {
type => "tomcat"
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]
add_tag => [ "timestamp_fix" ]
}
}

18. I need a new toy
Demo

19. Some remarks
● Don't forget about security
● The applications should be flexible enough for allowing to
publish their logs using brokers or other methods beyond
files and syslog
● Logging in JSON format is a nice to have
● Share the log visualization
● Use the brokers Luke
● If you develop internalize this
http://www.masterzen.fr/2013/01/13/the-10-
commandments-of-logging/

20. Extras
● http://logstash.net/
● http://www.logstashbook.com/code/ only $10.09
● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns
● http://grokdebug.herokuapp.com/
● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)
●
http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
● http://kibana.org/
●
https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html
● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/
●
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/

21. Do you want to join the <some fancy words here> team?
I am not hiring, but I can tell you about some places where
it is better to stay away
Have a nice day
All the images, videos and stuff are property of their respective owners, look at the cat
and don't sue me
http://stuffpoint.com/cats/image/41633/cute-cat-picture/