Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Using Logstash, ElasticSearch and KibanaAlejandro E Brito Monedero@ae_bm2013 / 05 / 23
Business as usual
So many hosts to checkIs there life out there?http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_L...
Time to play whack a loghttp://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
http://brotality.com/wp-content/uploads/2012/12/madness.jpg
I need a new toyhttp://www.youtube.com/watch?v=8L6Dpq5kY_AVideo time
Logstash ✔ collects logs✔ parses logs✔ stores logs✔ indexes logs✔ searches logs✔ and fixes timestampsYou only need● JVM● l...
$ log_producer | grep ... | sed … | awk … | tee output | sort | uniq -c | sort -nLog source LogstashLogstash(optional)Elas...
Inputs Filters OutputsFileRedisSyslogLumberjackRabbitmqSQS…AlterDateGrokMultilineGrep...AMQPCloudwatchElasticsearchMongodb...
ElasticSearchDistributed RESTful search server● Near real-time search● RESTful API● Easy to scale horizontally● HA● Full t...
Kibana✔ Nice UI✔ Better than the old frontend logstashincluded✔ Ruby / framework SinatraWeb frontend to search / graph and...
Original planApache(ligthweight shipper)Tomcat(ligthweight shipper)broker logstashElasticSearchKibana
After a few workaroundsApache(logstash shipper)Tomcat(logstash shipper)Logstash ElasticSearchKibanaSSH tunnels
Example config 1/3Logstash-httpd.confinput {file {type => "httpd"path => ["/var/log/httpd/*-logstash.log"]exclude => ["*.g...
Example config 2/3Logstash-tomcat.conffilter {# Tomcat# Remove blank linesgrep {type => "tomcat"match => [ "@message", "(....
Example config 3/3Logstash-tomcat.conf# mark the exceptions (multiline)grep {type => "tomcat"tags => [ "multiline" ]match ...
I need a new toyDemo
Some remarks● Dont forget about security● The applications should be flexible enough for allowing topublish their logs usi...
Extras● http://logstash.net/● http://www.logstashbook.com/code/ only $10.09● https://github.com/logstash/logstash/blob/v1....
Do you want to join the <some fancy words here> team?I am not hiring, but I can tell you about some places whereit is bett...
Using Logstash, elasticsearch & kibana
Upcoming SlideShare
Loading in …5
×

Using Logstash, elasticsearch & kibana

29,388 views

Published on

A tale of my adventures to process logs in a production environment. Soon I will link the demo video (in spanish)

Published in: Technology, Education
  • Be the first to comment

Using Logstash, elasticsearch & kibana

  1. 1. Using Logstash, ElasticSearch and KibanaAlejandro E Brito Monedero@ae_bm2013 / 05 / 23
  2. 2. Business as usual
  3. 3. So many hosts to checkIs there life out there?http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
  4. 4. Time to play whack a loghttp://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
  5. 5. http://brotality.com/wp-content/uploads/2012/12/madness.jpg
  6. 6. I need a new toyhttp://www.youtube.com/watch?v=8L6Dpq5kY_AVideo time
  7. 7. Logstash ✔ collects logs✔ parses logs✔ stores logs✔ indexes logs✔ searches logs✔ and fixes timestampsYou only need● JVM● logstash.jar
  8. 8. $ log_producer | grep ... | sed … | awk … | tee output | sort | uniq -c | sort -nLog source LogstashLogstash(optional)ElasticSearchDBstatsdPipes
  9. 9. Inputs Filters OutputsFileRedisSyslogLumberjackRabbitmqSQS…AlterDateGrokMultilineGrep...AMQPCloudwatchElasticsearchMongodbRedisFile...PluginsNot here yet? JRuby to the rescue
  10. 10. ElasticSearchDistributed RESTful search server● Near real-time search● RESTful API● Easy to scale horizontally● HA● Full text search● YAML config file / JSON format!!● Document oriented JSONGetting started: Logstash JAR includes it / download and set cluster.nameThis is where it will be worth to spend some time tuning
  11. 11. Kibana✔ Nice UI✔ Better than the old frontend logstashincluded✔ Ruby / framework SinatraWeb frontend to search / graph and more
  12. 12. Original planApache(ligthweight shipper)Tomcat(ligthweight shipper)broker logstashElasticSearchKibana
  13. 13. After a few workaroundsApache(logstash shipper)Tomcat(logstash shipper)Logstash ElasticSearchKibanaSSH tunnels
  14. 14. Example config 1/3Logstash-httpd.confinput {file {type => "httpd"path => ["/var/log/httpd/*-logstash.log"]exclude => ["*.gz"]start_position => "beginning"format => "json_event"}}output {tcp {host => "0.0.0.0"mode => "server"port => 1666}}Logstash-server.confinput {tcp {type => "httpd"format => "json_event"host => "127.0.0.1"mode => "client"port => "1666"}tcp {type => "app"format => "json_event"host => "127.0.0.1"mode => "client"port => "2666"}}output {elasticsearch {cluster => "logstash"}}
  15. 15. Example config 2/3Logstash-tomcat.conffilter {# Tomcat# Remove blank linesgrep {type => "tomcat"match => [ "@message", "(.+)" ]drop => trueadd_tag => [ "no_blank_lines" ]}# make the multilines be treated like a single linemultiline {type => "tomcat"pattern => "^dddd"negate => truewhat => "previous"}
  16. 16. Example config 3/3Logstash-tomcat.conf# mark the exceptions (multiline)grep {type => "tomcat"tags => [ "multiline" ]match => [ "@message", ".+Exception: .+" ]drop => falseadd_tag => [ "java_exception" ]}# get the log level, operation id, module and timestamp as separated fieldsgrok {type => "tomcat"pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."add_tag => [ "groked" ]}# fix the timestampdate {type => "tomcat"match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]add_tag => [ "timestamp_fix" ]}}
  17. 17. I need a new toyDemo
  18. 18. Some remarks● Dont forget about security● The applications should be flexible enough for allowing topublish their logs using brokers or other methods beyondfiles and syslog● Logging in JSON format is a nice to have● Share the log visualization● Use the brokers Luke● If you develop internalize thishttp://www.masterzen.fr/2013/01/13/the-10-commandments-of-logging/
  19. 19. Extras● http://logstash.net/● http://www.logstashbook.com/code/ only $10.09● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns● http://grokdebug.herokuapp.com/● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)●http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/● http://kibana.org/●https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/●http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
  20. 20. Do you want to join the <some fancy words here> team?I am not hiring, but I can tell you about some places whereit is better to stay awayHave a nice dayAll the images, videos and stuff are property of their respective owners, look at the catand dont sue mehttp://stuffpoint.com/cats/image/41633/cute-cat-picture/

×