Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Using Logstash, ElasticSearch and Kibana
Alejandro E Brito Monedero
@ae_bm
2013 / 05 / 23
Business as usual
So many hosts to check
Is there life out there?
http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above...
Time to play whack a log
http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
http://brotality.com/wp-content/uploads/2012/12/madness.jpg
I need a new toy
http://www.youtube.com/watch?v=8L6Dpq5kY_A
Video time
Logstash ✔ collects logs
✔ parses logs
✔ stores logs
✔ indexes logs
✔ searches logs
✔ and fixes timestamps
You only need
●...
$ log_producer | grep ... | sed … | awk … | tee output 
| sort | uniq -c | sort -n
Log source Logstash
Logstash
(optional)...
Inputs Filters Outputs
File
Redis
Syslog
Lumberjack
Rabbitmq
SQS
…
Alter
Date
Grok
Multiline
Grep
...
AMQP
Cloudwatch
Elas...
ElasticSearch
Distributed RESTful search server
● Near real-time search
● RESTful API
● Easy to scale horizontally
● HA
● ...
Kibana
✔ Nice UI
✔ Better than the old frontend logstash
included
✔ Ruby / framework Sinatra
Web frontend to search / grap...
Original plan
Apache
(ligthweight shipper)
Tomcat
(ligthweight shipper)
broker logstash
ElasticSearch
Kibana
After a few workarounds
Apache
(logstash shipper)
Tomcat
(logstash shipper)
Logstash ElasticSearch
Kibana
SSH tunnels
Example config 1/3
Logstash-httpd.conf
input {
file {
type => "httpd"
path => ["/var/log/httpd/*-logstash.log"]
exclude =>...
Example config 2/3
Logstash-tomcat.conf
filter {
# Tomcat
# Remove blank lines
grep {
type => "tomcat"
match => [ "@messag...
Example config 3/3
Logstash-tomcat.conf
# mark the exceptions (multiline)
grep {
type => "tomcat"
tags => [ "multiline" ]
...
I need a new toy
Demo
Some remarks
● Don't forget about security
● The applications should be flexible enough for allowing to
publish their logs...
Extras
● http://logstash.net/
● http://www.logstashbook.com/code/ only $10.09
● https://github.com/logstash/logstash/blob/...
Do you want to join the <some fancy words here> team?
I am not hiring, but I can tell you about some places where
it is be...
Using Logstash, elasticsearch & kibana
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
Next
Upcoming SlideShare
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
Next
Download to read offline and view in fullscreen.

41

Share

Using Logstash, elasticsearch & kibana

Download to read offline

A tale of my adventures to process logs in a production environment. Soon I will link the demo video (in spanish)

Related Books

Free with a 30 day trial from Scribd

See all

Using Logstash, elasticsearch & kibana

  1. 1. Using Logstash, ElasticSearch and Kibana Alejandro E Brito Monedero @ae_bm 2013 / 05 / 23
  2. 2. Business as usual
  3. 3. So many hosts to check Is there life out there? http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
  4. 4. Time to play whack a log http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
  5. 5. http://brotality.com/wp-content/uploads/2012/12/madness.jpg
  6. 6. I need a new toy http://www.youtube.com/watch?v=8L6Dpq5kY_A Video time
  7. 7. Logstash ✔ collects logs ✔ parses logs ✔ stores logs ✔ indexes logs ✔ searches logs ✔ and fixes timestamps You only need ● JVM ● logstash.jar
  8. 8. $ log_producer | grep ... | sed … | awk … | tee output | sort | uniq -c | sort -n Log source Logstash Logstash (optional) ElasticSearch DB statsd Pipes
  9. 9. Inputs Filters Outputs File Redis Syslog Lumberjack Rabbitmq SQS … Alter Date Grok Multiline Grep ... AMQP Cloudwatch Elasticsearch Mongodb Redis File ... Plugins Not here yet? JRuby to the rescue
  10. 10. ElasticSearch Distributed RESTful search server ● Near real-time search ● RESTful API ● Easy to scale horizontally ● HA ● Full text search ● YAML config file / JSON format!! ● Document oriented JSON Getting started: Logstash JAR includes it / download and set cluster.name This is where it will be worth to spend some time tuning
  11. 11. Kibana ✔ Nice UI ✔ Better than the old frontend logstash included ✔ Ruby / framework Sinatra Web frontend to search / graph and more
  12. 12. Original plan Apache (ligthweight shipper) Tomcat (ligthweight shipper) broker logstash ElasticSearch Kibana
  13. 13. After a few workarounds Apache (logstash shipper) Tomcat (logstash shipper) Logstash ElasticSearch Kibana SSH tunnels
  14. 14. Example config 1/3 Logstash-httpd.conf input { file { type => "httpd" path => ["/var/log/httpd/*-logstash.log"] exclude => ["*.gz"] start_position => "beginning" format => "json_event" } } output { tcp { host => "0.0.0.0" mode => "server" port => 1666 } } Logstash-server.conf input { tcp { type => "httpd" format => "json_event" host => "127.0.0.1" mode => "client" port => "1666" } tcp { type => "app" format => "json_event" host => "127.0.0.1" mode => "client" port => "2666" } } output { elasticsearch { cluster => "logstash" } }
  15. 15. Example config 2/3 Logstash-tomcat.conf filter { # Tomcat # Remove blank lines grep { type => "tomcat" match => [ "@message", "(.+)" ] drop => true add_tag => [ "no_blank_lines" ] } # make the multilines be treated like a single line multiline { type => "tomcat" pattern => "^dddd" negate => true what => "previous" }
  16. 16. Example config 3/3 Logstash-tomcat.conf # mark the exceptions (multiline) grep { type => "tomcat" tags => [ "multiline" ] match => [ "@message", ".+Exception: .+" ] drop => false add_tag => [ "java_exception" ] } # get the log level, operation id, module and timestamp as separated fields grok { type => "tomcat" pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..." add_tag => [ "groked" ] } # fix the timestamp date { type => "tomcat" match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ] add_tag => [ "timestamp_fix" ] } }
  17. 17. I need a new toy Demo
  18. 18. Some remarks ● Don't forget about security ● The applications should be flexible enough for allowing to publish their logs using brokers or other methods beyond files and syslog ● Logging in JSON format is a nice to have ● Share the log visualization ● Use the brokers Luke ● If you develop internalize this http://www.masterzen.fr/2013/01/13/the-10- commandments-of-logging/
  19. 19. Extras ● http://logstash.net/ ● http://www.logstashbook.com/code/ only $10.09 ● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns ● http://grokdebug.herokuapp.com/ ● http://www.infoq.com/articles/review-the-logstash-book (better diagrams) ● http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/ ● http://kibana.org/ ● https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html ● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ ● http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
  20. 20. Do you want to join the <some fancy words here> team? I am not hiring, but I can tell you about some places where it is better to stay away Have a nice day All the images, videos and stuff are property of their respective owners, look at the cat and don't sue me http://stuffpoint.com/cats/image/41633/cute-cat-picture/
  • JupiterTsou

    Oct. 3, 2018
  • PeiLingHung1

    Aug. 26, 2018
  • lantince

    Jan. 19, 2018
  • alxcho

    Jul. 13, 2016
  • kangjinlee3720

    Jun. 24, 2015
  • ssuser1b3f5c

    Jun. 8, 2015
  • NurettinOnurYAKIT

    May. 9, 2015
  • ssuserc961e4

    Mar. 20, 2015
  • Codefor

    Mar. 11, 2015
  • tinwinaung

    Mar. 11, 2015
  • feiwong50767

    Jan. 25, 2015
  • tcvn1985

    Nov. 26, 2014
  • bunkertor

    Nov. 26, 2014
  • fromzy

    Nov. 3, 2014
  • ssusere75594

    Nov. 2, 2014
  • johan.moreau

    Sep. 10, 2014
  • gwyddyon

    Sep. 7, 2014
  • zzzhc

    Jul. 9, 2014
  • AaronJung

    May. 6, 2014
  • johnwuzi

    Apr. 9, 2014

A tale of my adventures to process logs in a production environment. Soon I will link the demo video (in spanish)

Views

Total views

30,653

On Slideshare

0

From embeds

0

Number of embeds

52

Actions

Downloads

547

Shares

0

Comments

0

Likes

41

×