Monitoring with ElasticSearch

3,730 views

Published on

Monitoring with Elasticsearch, short talk for the ElasticSearch Belux Meetup Pre Fosdem

Published in: Technology

Monitoring with ElasticSearch

  1. 1. Monitoring with the ELKMonitoring with the ELK StackStack Kris Buytaert @krisbuytaert
  2. 2. KrisKris BuytaertBuytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Organising too many confs , #devopsdays,Organising too many confs , #devopsdays, #loadays, #cfgmgmtcamp ...#loadays, #cfgmgmtcamp ... ● Evangelizing devopsEvangelizing devops
  3. 3. #devops=~C(L)AMS#devops=~C(L)AMS ● CultureCulture ● (Lean)(Lean) ● AutomationAutomation ● Monitoring and MeasurementMonitoring and Measurement ● SharingSharing Damon Edwards and John WillisDamon Edwards and John Willis Gene KimGene Kim
  4. 4. #monitoringsucks#monitoringsucks ● John Vincent (@lusis), june 2011John Vincent (@lusis), june 2011 ● A sub #devops movementA sub #devops movement ● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
  5. 5. #monitoringlove#monitoringlove • • Ulf Mansson #devopsdays Rome 2011Ulf Mansson #devopsdays Rome 2011 • A new era of toolingA new era of tooling • #monitoringlove hacksessions @inuits#monitoringlove hacksessions @inuits • #monitorama#monitorama
  6. 6. What we wantWhat we want ● Small , well suited componentsSmall , well suited components • CollectCollect • Transport / MangleTransport / Mangle • StoreStore • AnalyseAnalyse • Act / AlertAct / Alert • VisualizeVisualize •
  7. 7. What do YOU do withWhat do YOU do with your logfiles ?your logfiles ?
  8. 8. Logs & CollectionLogs & Collection ● Syslog,Syslog, ● RsyslogRsyslog ● Syslog-ngSyslog-ng ● Log4jLog4j ● Graylog2Graylog2 ● ELSA (Enterprise LogELSA (Enterprise Log Search and Archive)Search and Archive) ● ELK StackELK Stack
  9. 9. LogstashLogstash ● Not your average centralized logging toolNot your average centralized logging tool ● Elasticsearch backedElasticsearch backed ● ShipperShipper ● IndexerIndexer ● WebWeb
  10. 10. ● Collect fromCollect from anywhereanywhere ● FilterFilter ● Send anywhereSend anywhere ● QueingQueing
  11. 11. Collect logsCollect logs ● rsyslog -> rsyslog inputrsyslog -> rsyslog input ● dedicated shippers (logstash-shipper,dedicated shippers (logstash-shipper, lumberjack)lumberjack) • Direct or via queueDirect or via queue ● Write logs -> json format -> redis -> json inputWrite logs -> json format -> redis -> json input ● * -> your favourite logstash input* -> your favourite logstash input ●
  12. 12. FiltersFilters ● GrokGrok ● MutateMutate mutate {mutate { # Lowercase some values that are always in uppercase# Lowercase some values that are always in uppercase lowercase => [ "EventType", "FileName", "Hostname", "Severity" ]lowercase => [ "EventType", "FileName", "Hostname", "Severity" ] }} ● dropdrop ● TagsTags
  13. 13. GrokGrok grok {grok { match => [match => [ "message", "%{IPORHOST:host} - %{USER:remote_user} [%"message", "%{IPORHOST:host} - %{USER:remote_user} [% {MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}] % {QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} %{QS:request} %{INT:status} %{INT:bytes_sent} %{QS:http_referer} % {QS:http_user_agent}"{QS:http_user_agent}" ]] add_field => ["grok_type", "nginx-access"]add_field => ["grok_type", "nginx-access"] add_tag => ["grokked"]add_tag => ["grokked"] }} }}
  14. 14. OutputsOutputs ● Mostly ElasticSearchMostly ElasticSearch ● Plenty of outputsPlenty of outputs • EmailEmail • NagiosNagios • RiemannRiemann • StatsdStatsd • GraphiteGraphite
  15. 15. Same tool used by devs to debugSame tool used by devs to debug as by ops to debugas by ops to debug
  16. 16. Long Term MetricsLong Term Metrics ● Disk space is cheapDisk space is cheap ● But some people don't care about those logsBut some people don't care about those logs after X weeks / months / yearsafter X weeks / months / years ● Send statistics via statsd to graphite ,Send statistics via statsd to graphite , ● Keep graphite data for long term storage,Keep graphite data for long term storage, ● Purge elasticsearch contentPurge elasticsearch content (curator is incomplete, working on patches)(curator is incomplete, working on patches)
  17. 17. GraphiteGraphite ● Graphing at ScaleGraphing at Scale ● Graphing at EaseGraphing at Ease ● Any metric is a graphAny metric is a graph ● echo "somestring $somevalue $timestamp" |echo "somestring $somevalue $timestamp" | nc <%= graphitehost %> 2003nc <%= graphitehost %> 2003
  18. 18. Friends of GraphiteFriends of Graphite ● CCollection :ollection : ● Statsd,Statsd, ● Collectd + CarbonpluginCollectd + Carbonplugin ● JmxtransJmxtrans ● LogsterLogster ● DashboardsDashboards ● TattleTattle ● GdashGdash ● GrafanaGrafana
  19. 19. Alerting on Events/LogsAlerting on Events/Logs ● Logstash -> icingaLogstash -> icinga output {output { if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc-if [message] =~ /(error|ERROR|CRITICAL)/ and [logsource] =~ /edc- app/ and [program] =~ /^edc-/ {app/ and [program] =~ /^edc-/ { nagios_nsca {nagios_nsca { host => "10.0.64.28"host => "10.0.64.28" nagios_status => "1"nagios_status => "1" nagios_host => "%{logsource}"nagios_host => "%{logsource}" nagios_service => "Log check - %{program}"nagios_service => "Log check - %{program}" }} }}
  20. 20. Buy this guy a beer
  21. 21. Remember, it's notRemember, it's not about the toolsabout the tools
  22. 22. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ InuitsInuits Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×