Logstash and friends

8,418
-1

Published on

Internal training @ inuits

Published in: Technology, Education

Logstash and friends

  1. 1. ; Logstash and friendsLogstash and friends Julien PivottoJulien Pivotto Techies Teach TechiesTechies Teach Techies September 2, 2013September 2, 2013
  2. 2. ; Introduction Logstash Kibana Conclusion 1 Introduction 2 Logstash Missions Inputs Filters Output 3 Kibana 4 Conclusion Julien Pivotto Logstash and friends
  3. 3. ; Introduction Logstash Kibana Conclusion Logging • Recording of eventsRecording of events • Voice of your systems and applicationsVoice of your systems and applications • It tells you almost everythingIt tells you almost everything • It is a source of knowledgeIt is a source of knowledge Julien Pivotto Logstash and friends
  4. 4. ; Introduction Logstash Kibana Conclusion Logging is useful • Understanding outagesUnderstanding outages Julien Pivotto Logstash and friends
  5. 5. ; Introduction Logstash Kibana Conclusion Logging is useful • Understanding outagesUnderstanding outages • not only when it’s wrongnot only when it’s wrong • you can extract metricsyou can extract metrics • no logs means somethingno logs means something • it tells you what, why, who, whenit tells you what, why, who, when Julien Pivotto Logstash and friends
  6. 6. ; Introduction Logstash Kibana Conclusion Logging in the wild • SyslogSyslog • |tee /var/log/myapp.log|tee /var/log/myapp.log • Cron + MAILTO=Cron + MAILTO= • &>/dev/null&>/dev/null Julien Pivotto Logstash and friends
  7. 7. ; Introduction Logstash Kibana Conclusion Logging in the past • Logging to files on each serverLogging to files on each server • Using syslog protocolUsing syslog protocol • DecentralizedDecentralized • Reading requires SSH accessReading requires SSH access • Not developer friendlyNot developer friendly Julien Pivotto Logstash and friends
  8. 8. ; Introduction Logstash Kibana Conclusion The tools nowadays • Jenkins, Icinga, Graphite, ForemanJenkins, Icinga, Graphite, Foreman • Nice web interfacesNice web interfaces • CentralizedCentralized • Easy to useEasy to use Julien Pivotto Logstash and friends
  9. 9. ; Introduction Logstash Kibana Conclusion Requirements • Scalable toolsScalable tools • Configured by text filesConfigured by text files • Playing with existing toolsPlaying with existing tools • ScalableScalable • Following the Unix philosophyFollowing the Unix philosophy Julien Pivotto Logstash and friends
  10. 10. ; Introduction Logstash Kibana Conclusion 3 separate tools • Elasticsearch, distributed search & analytics engineElasticsearch, distributed search & analytics engine • Logstash, logs managementLogstash, logs management • Kibana, very nice webui to ES and LogstashKibana, very nice webui to ES and Logstash Julien Pivotto Logstash and friends
  11. 11. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Logstash Julien Pivotto Logstash and friends
  12. 12. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Shipping the logs • Some applications can only write to filesSome applications can only write to files • But you need them on the main logstash serverBut you need them on the main logstash server • Logstash can act as a daemon to ship the logsLogstash can act as a daemon to ship the logs • Destinations can be syslog, redis,. . .Destinations can be syslog, redis,. . . • Then you can act on your logsThen you can act on your logs Julien Pivotto Logstash and friends
  13. 13. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Collecting the logs • You can plug logstash to a lot of data sourcesYou can plug logstash to a lot of data sources • It can be passive or activeIt can be passive or active • Listening on a UDP port vs checking mailsListening on a UDP port vs checking mails • All your logs are managed by one applicationAll your logs are managed by one application • It creates fields from the logsIt creates fields from the logs Julien Pivotto Logstash and friends
  14. 14. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Filtering the logs • Making sense of a log messageMaking sense of a log message • Finding what is importantFinding what is important • Adding and removing fieldsAdding and removing fields Julien Pivotto Logstash and friends
  15. 15. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Storing the logs • Output to ElasticsearchOutput to Elasticsearch • Sending information to statsdSending information to statsd • Sending to your inbox, to icinga or filesSending to your inbox, to icinga or files Julien Pivotto Logstash and friends
  16. 16. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output What is an "event"? • Several fieldsSeveral fields • Several tagsSeveral tags • A type (syslog message, irc message,. . . )A type (syslog message, irc message,. . . ) • A @message fieldA @message field • A timestampA timestamp Julien Pivotto Logstash and friends
  17. 17. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Event Julien Pivotto Logstash and friends
  18. 18. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output http://www.flickr.com/photos/quinnanya/7237788632/ Julien Pivotto Logstash and friends
  19. 19. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output UDP and TCP input • Compatible with rsyslog protocolCompatible with rsyslog protocol • Each syslog talks with logstash directlyEach syslog talks with logstash directly • Allow you to use the syslog toolchains: logger, rsyslogAllow you to use the syslog toolchains: logger, rsyslog • UDP is shoot and forgetUDP is shoot and forget Julien Pivotto Logstash and friends
  20. 20. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output UDP and TCP input Logstash configuration input { udp { type => syslog port => 5544 } tcp { type => syslog port => 5544 } } Julien Pivotto Logstash and friends
  21. 21. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output UDP and TCP input Rsyslog configuration *.* @logstash.example.com:5544 • IIn /etc/rsyslog.conf • TThat line will forward all the logs to logstash • LLogstash will make useful fields out of it: priority, severity, program. . . Julien Pivotto Logstash and friends
  22. 22. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output File • Enable you to use logstash with every applicationEnable you to use logstash with every application • Useful to ship the logsUseful to ship the logs • Acts as a tail -n 0 -FActs as a tail -n 0 -F • It works even if you use logrotateIt works even if you use logrotate Julien Pivotto Logstash and friends
  23. 23. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output File input { file { path => "/var/log/legacyapp.log" type => "legacylog" } } Julien Pivotto Logstash and friends
  24. 24. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grok • Extract fields from textExtract fields from text • Useful to read messagesUseful to read messages • A lot of pre-existing patternsA lot of pre-existing patterns • Uses Regex to find out fieldsUses Regex to find out fields Julien Pivotto Logstash and friends
  25. 25. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grok Input text Invalid user oracle from 85.249.144.18 Grok pattern Invalid user %{USERNAME:login} from %{IP:ip} Result { "login": [ [ "oracle" ] ], "ip": [ [ "85.249.144.18" ] ] } Julien Pivotto Logstash and friends
  26. 26. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grok filter { grok { type => "syslog" pattern => ["(?m)<%{POSINT:syslog_pri}>..." add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{@source_host}" ] add_tag => "syslog-%{syslog_program}" } } Julien Pivotto Logstash and friends
  27. 27. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grep • Allows you to grep interresting messagesAllows you to grep interresting messages • Useful to countUseful to count Julien Pivotto Logstash and friends
  28. 28. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Grep filter { grep { add_field => ["outputirc", "A puppet package has been deployed"] add_tag => "outputirc" drop => false match => [ "syslog_program", "yum" ] match => [ "@source_host", "puppetmaster" ] match => [ "@message", "puppet-tree" ] } } Julien Pivotto Logstash and friends
  29. 29. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Geoip filter{ geoip { tags => ["syslog-httpd"] source => ["client"] } } • Transform ip address into geo dataTransform ip address into geo data • Useful to filter by country/map the dataUseful to filter by country/map the data Julien Pivotto Logstash and friends
  30. 30. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output Elasticsearch • Version of elasticsearch <=> version of logstashVersion of elasticsearch <=> version of logstash • Unless you use the elasticsearch_http outputUnless you use the elasticsearch_http output output { elasticsearch { } } Julien Pivotto Logstash and friends
  31. 31. ; Introduction Logstash Kibana Conclusion Missions Inputs Filters Output IRC output { irc { channels => ["#example"] host => "chat.freenode.net" nick => "loggy" port => 6667 tags => "outputirc" user => "loggy" format => "%{outputirc}" } } Julien Pivotto Logstash and friends
  32. 32. ; Introduction Logstash Kibana Conclusion statsd output { statsd { host => ’127.0.0.1’ sender => "logstash" increment => [ "httpd.%{http_host}.r.%{response}", "httpd.response.%{response}"] count => ["apache.%{http_host}.bytes", "%{bytes}" ] timing => ["apache.%{http_host}", "%{duration_msec}"] tags => ’grokked-apache’ } } Julien Pivotto Logstash and friends
  33. 33. ; Introduction Logstash Kibana Conclusion Kibana • Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES • Kibana 1 was written in PHPKibana 1 was written in PHP • Kibana 2 was written in RubyKibana 2 was written in Ruby • Kibana 3 is written in AngularJSKibana 3 is written in AngularJS Julien Pivotto Logstash and friends
  34. 34. ; Introduction Logstash Kibana Conclusion Kibana 3 • Everything happens in the browserEverything happens in the browser • The browser is connected to ElasticsearchThe browser is connected to Elasticsearch • You can save dashboards into ESYou can save dashboards into ES • You can write/template dashboards to filesYou can write/template dashboards to files Julien Pivotto Logstash and friends
  35. 35. ; Introduction Logstash Kibana Conclusion Installing kibana3 git clone https://github.com/elasticsearch/kibana.git ssh -NL 9200:127.0.0.1:9200 elasticsearch & python -m SimpleHTTPServer Julien Pivotto Logstash and friends
  36. 36. ; Introduction Logstash Kibana Conclusion Kibana queries Example of a kibana query @fields.syslog_program:"httpd" AND @fields.http_host:"test.example.com" AND @fields.response:"404" • LLucene query syntax • SSimple and effective • PPoint & click web interface Julien Pivotto Logstash and friends
  37. 37. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  38. 38. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  39. 39. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  40. 40. ; Introduction Logstash Kibana Conclusion Kibana Julien Pivotto Logstash and friends
  41. 41. ; Introduction Logstash Kibana Conclusion Conclusion • Logstash is a small daemonLogstash is a small daemon • Simple to package & deploy (jar file)Simple to package & deploy (jar file) • Scalable thanks to ElasticsearchScalable thanks to Elasticsearch • Developer friendly thanks to KibanaDeveloper friendly thanks to Kibana Julien Pivotto Logstash and friends

×