Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Logstash and friends

12,006 views

Published on

Internal training @ inuits

Published in: Technology, Education

Logstash and friends

  1. 1. ; Logstash and friendsLogstash and friends Julien PivottoJulien Pivotto Techies Teach TechiesTechies Teach Techies September 2, 2013September 2, 2013
  2. 2. ; Introduction Logstash Kibana Conclusion 1 Introduction 2 Logstash Missions Inputs Filters Output 3 Kibana 4 Conclusion Julien Pivotto Logstash and friends
  3. 3. ; Introduction Logstash Kibana Conclusion Logging • Recording of eventsRecording of events • Voice of your systems and applicationsVoice of your systems and applications • It tells you almost everythingIt tells you almost everything • It is a source of knowledgeIt is a source of knowledge Julien Pivotto Logstash and friends
  4. 4. ; Introduction Logstash Kibana Conclusion Logging is useful • Understanding outagesUnderstanding outages Julien Pivotto Logstash and friends
  5. 5. ; Introduction Logstash Kibana Conclusion Logging is useful • Understanding outagesUnderstanding outages • not only when it’s wrongnot only when it’s wrong • you can extract metricsyou can extract metrics • no logs means somethingno logs means something • it tells you what, why, who, whenit tells you what, why, who, when Julien Pivotto Logstash and friends
  6. 6. ; Introduction Logstash Kibana Conclusion Logging in the wild • SyslogSyslog • |tee /var/log/myapp.log|tee /var/log/myapp.log • Cron + MAILTO=Cron + MAILTO= • &>/dev/null&>/dev/null Julien Pivotto Logstash and friends
  7. 7. ; Introduction Logstash Kibana Conclusion Logging in the past • Logging to files on each serverLogging to files on each server • Using syslog protocolUsing syslog protocol • DecentralizedDecentralized • Reading requires SSH accessReading requires SSH access • Not developer friendlyNot developer friendly Julien Pivotto Logstash and friends
  8. 8. ; Introduction Logstash Kibana Conclusion The tools nowadays • Jenkins, Icinga, Graphite, ForemanJenkins, Icinga, Graphite, Foreman • Nice web interfacesNice web interfaces • CentralizedCentralized • Easy to useEasy to use Julien Pivotto Logstash and friends
  9. 9. ; Introduction Logstash Kibana Conclusion Requirements • Scalable toolsScalable tools • Configured by text filesConfigured by text files • Playing with existing toolsPlaying with existing tools • ScalableScalable • Following the Unix philosophyFollowing the Unix philosophy Julien Pivotto Logstash and friends
  10. 10. ; Introduction Logstash Kibana Conclusion 3 separate tools • Elasticsearch, distributed search & analytics engineElasticsearch, distributed search & analytics engine • Logstash, logs managementLogstash, logs management • Kibana, very nice webui to ES and LogstashKibana, very nice webui to ES and Logstash Julien Pivotto Logstash and friends
  11. 11. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Logstash Julien Pivotto Logstash and friends
  12. 12. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Shipping the logs • Some applications can only write to filesSome applications can only write to files • But you need them on the main logstash serverBut you need them on the main logstash server • Logstash can act as a daemon to ship the logsLogstash can act as a daemon to ship the logs • Destinations can be syslog, redis,. . .Destinations can be syslog, redis,. . . • Then you can act on your logsThen you can act on your logs Julien Pivotto Logstash and friends
  13. 13. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Collecting the logs • You can plug logstash to a lot of data sourcesYou can plug logstash to a lot of data sources • It can be passive or activeIt can be passive or active • Listening on a UDP port vs checking mailsListening on a UDP port vs checking mails • All your logs are managed by one applicationAll your logs are managed by one application • It creates fields from the logsIt creates fields from the logs Julien Pivotto Logstash and friends
  14. 14. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Filtering the logs • Making sense of a log messageMaking sense of a log message • Finding what is importantFinding what is important • Adding and removing fieldsAdding and removing fields Julien Pivotto Logstash and friends
  15. 15. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Storing the logs • Output to ElasticsearchOutput to Elasticsearch • Sending information to statsdSending information to statsd • Sending to your inbox, to icinga or filesSending to your inbox, to icinga or files Julien Pivotto Logstash and friends
  16. 16. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output What is an "event"? • Several fieldsSeveral fields • Several tagsSeveral tags • A type (syslog message, irc message,. . . )A type (syslog message, irc message,. . . ) • A @message fieldA @message field • A timestampA timestamp Julien Pivotto Logstash and friends
  17. 17. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Event Julien Pivotto Logstash and friends
  18. 18. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output http://www.flickr.com/photos/quinnanya/7237788632/ Julien Pivotto Logstash and friends
  19. 19. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output UDP and TCP input • Compatible with rsyslog protocolCompatible with rsyslog protocol • Each syslog talks with logstash directlyEach syslog talks with logstash directly • Allow you to use the syslog toolchains: logger, rsyslogAllow you to use the syslog toolchains: logger, rsyslog • UDP is shoot and forgetUDP is shoot and forget Julien Pivotto Logstash and friends
  20. 20. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output UDP and TCP input Logstash configuration input { udp { type => syslog port => 5544 } tcp { type => syslog port => 5544 } } Julien Pivotto Logstash and friends
  21. 21. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output UDP and TCP input Rsyslog configuration *.* @logstash.example.com:5544 • IIn /etc/rsyslog.conf • TThat line will forward all the logs to logstash • LLogstash will make useful fields out of it: priority, severity, program. . . Julien Pivotto Logstash and friends
  22. 22. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output File • Enable you to use logstash with every applicationEnable you to use logstash with every application • Useful to ship the logsUseful to ship the logs • Acts as a tail -n 0 -FActs as a tail -n 0 -F • It works even if you use logrotateIt works even if you use logrotate Julien Pivotto Logstash and friends
  23. 23. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output File input { file { path => "/var/log/legacyapp.log" type => "legacylog" } } Julien Pivotto Logstash and friends
  24. 24. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grok • Extract fields from textExtract fields from text • Useful to read messagesUseful to read messages • A lot of pre-existing patternsA lot of pre-existing patterns • Uses Regex to find out fieldsUses Regex to find out fields Julien Pivotto Logstash and friends
  25. 25. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grok Input text Invalid user oracle from 85.249.144.18 Grok pattern Invalid user %{USERNAME:login} from %{IP:ip} Result { "login": [ [ "oracle" ] ], "ip": [ [ "85.249.144.18" ] ] } Julien Pivotto Logstash and friends
  26. 26. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grok filter { grok { type => "syslog" pattern => ["(?m)<%{POSINT:syslog_pri}>..." add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{@source_host}" ] add_tag => "syslog-%{syslog_program}" } } Julien Pivotto Logstash and friends
  27. 27. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grep • Allows you to grep interresting messagesAllows you to grep interresting messages • Useful to countUseful to count Julien Pivotto Logstash and friends
  28. 28. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grep filter { grep { add_field => ["outputirc", "A puppet package has been deployed"] add_tag => "outputirc" drop => false match => [ "syslog_program", "yum" ] match => [ "@source_host", "puppetmaster" ] match => [ "@message", "puppet-tree" ] } } Julien Pivotto Logstash and friends
  29. 29. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Geoip filter{ geoip { tags => ["syslog-httpd"] source => ["client"] } } • Transform ip address into geo dataTransform ip address into geo data • Useful to filter by country/map the dataUseful to filter by country/map the data Julien Pivotto Logstash and friends
  30. 30. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Elasticsearch • Version of elasticsearch <=> version of logstashVersion of elasticsearch <=> version of logstash • Unless you use the elasticsearch_http outputUnless you use the elasticsearch_http output output { elasticsearch { } } Julien Pivotto Logstash and friends
  31. 31. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output IRC output { irc { channels => ["#example"] host => "chat.freenode.net" nick => "loggy" port => 6667 tags => "outputirc" user => "loggy" format => "%{outputirc}" } } Julien Pivotto Logstash and friends
  32. 32. ; Introduction Logstash Kibana Conclusion statsd output { statsd { host => ’127.0.0.1’ sender => "logstash" increment => [ "httpd.%{http_host}.r.%{response}", "httpd.response.%{response}"] count => ["apache.%{http_host}.bytes", "%{bytes}" ] timing => ["apache.%{http_host}", "%{duration_msec}"] tags => ’grokked-apache’ } } Julien Pivotto Logstash and friends
  33. 33. ; Introduction Logstash Kibana Conclusion Kibana • Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES • Kibana 1 was written in PHPKibana 1 was written in PHP • Kibana 2 was written in RubyKibana 2 was written in Ruby • Kibana 3 is written in AngularJSKibana 3 is written in AngularJS Julien Pivotto Logstash and friends
  34. 34. ; Introduction Logstash Kibana Conclusion Kibana 3 • Everything happens in the browserEverything happens in the browser • The browser is connected to ElasticsearchThe browser is connected to Elasticsearch • You can save dashboards into ESYou can save dashboards into ES • You can write/template dashboards to filesYou can write/template dashboards to files Julien Pivotto Logstash and friends
  35. 35. ; Introduction Logstash Kibana Conclusion Installing kibana3 git clone https://github.com/elasticsearch/kibana.git ssh -NL 9200:127.0.0.1:9200 elasticsearch & python -m SimpleHTTPServer Julien Pivotto Logstash and friends
  36. 36. ; Introduction Logstash Kibana Conclusion Kibana queries Example of a kibana query @fields.syslog_program:"httpd" AND @fields.http_host:"test.example.com" AND @fields.response:"404" • LLucene query syntax • SSimple and effective • PPoint & click web interface Julien Pivotto Logstash and friends
  37. 37. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  38. 38. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  39. 39. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  40. 40. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  41. 41. ; Introduction Logstash Kibana Conclusion Conclusion • Logstash is a small daemonLogstash is a small daemon • Simple to package & deploy (jar file)Simple to package & deploy (jar file) • Scalable thanks to ElasticsearchScalable thanks to Elasticsearch • Developer friendly thanks to KibanaDeveloper friendly thanks to Kibana Julien Pivotto Logstash and friends

×