Using Logstash, elasticsearch & kibana

27,517 views
26,785 views

Published on

A tale of my adventures to process logs in a production environment. Soon I will link the demo video (in spanish)

Published in: Technology, Education
0 Comments
38 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
27,517
On SlideShare
0
From Embeds
0
Number of Embeds
32
Actions
Shares
0
Downloads
481
Comments
0
Likes
38
Embeds 0
No embeds

No notes for slide

Using Logstash, elasticsearch & kibana

  1. 1. Using Logstash, ElasticSearch and KibanaAlejandro E Brito Monedero@ae_bm2013 / 05 / 23
  2. 2. Business as usual
  3. 3. So many hosts to checkIs there life out there?http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
  4. 4. Time to play whack a loghttp://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
  5. 5. http://brotality.com/wp-content/uploads/2012/12/madness.jpg
  6. 6. I need a new toyhttp://www.youtube.com/watch?v=8L6Dpq5kY_AVideo time
  7. 7. Logstash ✔ collects logs✔ parses logs✔ stores logs✔ indexes logs✔ searches logs✔ and fixes timestampsYou only need● JVM● logstash.jar
  8. 8. $ log_producer | grep ... | sed … | awk … | tee output | sort | uniq -c | sort -nLog source LogstashLogstash(optional)ElasticSearchDBstatsdPipes
  9. 9. Inputs Filters OutputsFileRedisSyslogLumberjackRabbitmqSQS…AlterDateGrokMultilineGrep...AMQPCloudwatchElasticsearchMongodbRedisFile...PluginsNot here yet? JRuby to the rescue
  10. 10. ElasticSearchDistributed RESTful search server● Near real-time search● RESTful API● Easy to scale horizontally● HA● Full text search● YAML config file / JSON format!!● Document oriented JSONGetting started: Logstash JAR includes it / download and set cluster.nameThis is where it will be worth to spend some time tuning
  11. 11. Kibana✔ Nice UI✔ Better than the old frontend logstashincluded✔ Ruby / framework SinatraWeb frontend to search / graph and more
  12. 12. Original planApache(ligthweight shipper)Tomcat(ligthweight shipper)broker logstashElasticSearchKibana
  13. 13. After a few workaroundsApache(logstash shipper)Tomcat(logstash shipper)Logstash ElasticSearchKibanaSSH tunnels
  14. 14. Example config 1/3Logstash-httpd.confinput {file {type => "httpd"path => ["/var/log/httpd/*-logstash.log"]exclude => ["*.gz"]start_position => "beginning"format => "json_event"}}output {tcp {host => "0.0.0.0"mode => "server"port => 1666}}Logstash-server.confinput {tcp {type => "httpd"format => "json_event"host => "127.0.0.1"mode => "client"port => "1666"}tcp {type => "app"format => "json_event"host => "127.0.0.1"mode => "client"port => "2666"}}output {elasticsearch {cluster => "logstash"}}
  15. 15. Example config 2/3Logstash-tomcat.conffilter {# Tomcat# Remove blank linesgrep {type => "tomcat"match => [ "@message", "(.+)" ]drop => trueadd_tag => [ "no_blank_lines" ]}# make the multilines be treated like a single linemultiline {type => "tomcat"pattern => "^dddd"negate => truewhat => "previous"}
  16. 16. Example config 3/3Logstash-tomcat.conf# mark the exceptions (multiline)grep {type => "tomcat"tags => [ "multiline" ]match => [ "@message", ".+Exception: .+" ]drop => falseadd_tag => [ "java_exception" ]}# get the log level, operation id, module and timestamp as separated fieldsgrok {type => "tomcat"pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."add_tag => [ "groked" ]}# fix the timestampdate {type => "tomcat"match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]add_tag => [ "timestamp_fix" ]}}
  17. 17. I need a new toyDemo
  18. 18. Some remarks● Dont forget about security● The applications should be flexible enough for allowing topublish their logs using brokers or other methods beyondfiles and syslog● Logging in JSON format is a nice to have● Share the log visualization● Use the brokers Luke● If you develop internalize thishttp://www.masterzen.fr/2013/01/13/the-10-commandments-of-logging/
  19. 19. Extras● http://logstash.net/● http://www.logstashbook.com/code/ only $10.09● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns● http://grokdebug.herokuapp.com/● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)●http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/● http://kibana.org/●https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/●http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
  20. 20. Do you want to join the <some fancy words here> team?I am not hiring, but I can tell you about some places whereit is better to stay awayHave a nice dayAll the images, videos and stuff are property of their respective owners, look at the catand dont sue mehttp://stuffpoint.com/cats/image/41633/cute-cat-picture/

×