Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Log Monitoring Simplified - Get the best out of Graylog2 & Icinga 2

Presentation from OSMC 2014
http://www.netways.de/en/osmc/osmc_2014/program/

Bernd Ahlers, Graylog2
Michael Friedrich, Icinga

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Log Monitoring Simplified - Get the best out of Graylog2 & Icinga 2

  1. 1. Bernd Ahlers Michael Friedrich Log Monitoring Simplified Get the best out of Graylog2 & Icinga 2
  2. 2. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA BEFORE WE START …
  3. 3. Agenda
  4. 4. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA AGENDA •Introduction •Tools •Log History •Logs & Monitoring •Demo •„The Future“ •Resources •Q&A
  5. 5. Introduction
  6. 6. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA WHO‘S WHO Bernd Ahlers @berndahlers German, 34, Graylog2 Developer Graylog2 Team since 2014 Developer @ TORCH GmbH Michael Friedrich @dnsmichi Austrian, 31, Icinga Developer Icinga Team since May 2009 Application Developer @ NETWAYS
  7. 7. Tools: Graylog2
  8. 8. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 •Started as open source project by Lennart Koopmann in 2010 –Developed entirely in his free time –Free & open source log management tool •TORCH GmbH founded as company behind Graylog2 in late 2012 –after seeing massive growth and worldwide distribution in large scale setups •Team of 8 engineers working full-time on it
  9. 9. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2 •Big rewrite of Graylog2 started in 2012 •Finished with releasing a final v0.20.0 in February 2014 •Addresses what we learnt from our first customers and all users •Unified REST API communication –easy extending and integrating with other products, tools and scripts •New web interface focusing on powerful analytics •Current stable version: 0.91.3
  10. 10. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: GRAYLOG2
  11. 11. Tools: Icinga 2
  12. 12. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 •Monitoring core engine –Checks, alerts, notifications –Backend interfaces for frontend visualization •Scalable for high performance & real-time monitoring –check_interval = 1s •Dynamic configuration format •Cluster & remote clients, SSL x509 & IPv4/6
  13. 13. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA TOOLS: ICINGA 2 •Modular feature set & connectors –DB IDO, Livestatus, Perfdata, Graphite, Gelf •Supports Monitoring Plugins API •Rewritten from scratch –Stable version: 2.2.0 (17.11.2014)
  14. 14. Log History
  15. 15. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY •Logs everywhere •How to collect them? –Splunk (4500$+ for 1GB/day) –Syslog-ng + Custom scripts •Purpose of your collection? –Regex for log parsing –Filters –Alerts? Notifications? Correlation? –Reporting •#devops Stack –Graylog2, Logstash (ELK) + $monitoring + $metrics + $cfgmgmt
  16. 16. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG HISTORY •Problems with remote syslog checks –Failure: where‘s the context? –Pattern matching –Seek files (state history, rate calculation) –Configuration inside Icinga/Plugin •Collect them –Central log cluster (failover) –Correlate events from other servers –Defined streams and alert triggers –Defined input types (e.g. GELF) –Query alert API from Icinga
  17. 17. Logs & Monitoring
  18. 18. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA LOG & MONITORING •Monitor your logs –Call check plugin or receive passive events –Generate alerts based on thresholds (configuration) –Notifications based on alerts –Visualize the current state & history for SLA reporting –Trigger event handlers (e.g. iptables on flood) •Popular plugins –check_logfiles –check_splunk •Collector APIs & Hooks –Graylog2 alert API & alert callback plugin –Logstash Nagios output
  19. 19. Logs & Monitoring: Strategy
  20. 20. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA STRATEGY •Out-of-the-box support or external addons? •Add hook to streams for passive event sending? •Query a defined API for alerts? •Visualize alerts, and where? (we want dashboards!) •Re-usable & customizable URL for notifications •Combine Log Events & Monitoring notifications and handlers
  21. 21. Logs & Monitoring: Push
  22. 22. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA PUSH: GRAYLOG2 ALARM CALLBACK •Requirements –Icinga API (Command Pipe) –Graylog2 Plugin Alarm Callback http://www.graylog2.org/resources/documentation/general/streams http://www.graylog2.org/resources/documentation/general/plugins •Ideas –Exec Callback+NSCA http://bashinglinux.wordpress.com/2013/05/26/graylog2-and-nagios-integration-2/ –(Ab)Use the notification plugin http://everythingshouldbevirtual.com/graylog2-streams-via-email –Custom Rake Plugin http://gallaman.blogspot.de/2012/04/marrying-graylog2-and-nagios.html •Solution –There is no simple & secure unified Core API (yet) –Use local Icinga2 client & poll check plugin instead
  23. 23. Logs & Monitoring: Poll
  24. 24. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK •Requirements –Graylog2 REST API –Icinga Check Plugin •Ideas –Wrapper for Python API calls? https://github.com/qmetric/graylog2-api-tools –Compile check_graylog2_stream? https://github.com/emind-systems/check_graylog2_stream •Solution –New Icinga Plugin by Graylog2 https://github.com/Graylog2/check-graylog2-stream
  25. 25. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK
  26. 26. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA POLL: ICINGA CHECK # ./check-graylog2-stream usage: -condition="<ID>": Condition ID, set only to check a single alert (optional) -password="<password>": API password (mandatory) -stream="<ID>": Stream ID (mandatory) -url="http://localhost:12900": URL to Graylog2 api (optional) -user="<username>": API username (mandatory)
  27. 27. Combining Graylog2 & Icinga 2
  28. 28. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA COMBINING GRAYLOG2 & ICINGA 2 •Events triggered by Icinga 2 –Check results –State changes –Notifications •Sent to Graylog2 using `GelfWriter` feature # icinga2 feature enable gelf && service icinga2 restart •Visualize in Graylog2 –Filter based on type (e.g. state != OK) –Alert streams based on counts, etc
  29. 29. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA NOTIFICATIONS •„Default Monitoring Alerts are awful“ http://holyhandgrenade.org/blog/2012/11/default-monitoring-alerts-are-awful/ –You want to see what‘s wrong. No additional click on your mobile. •Icinga 2 triggers a notification –Fetch additional information from Graylog2 API –Include ‚notes_url‘ with stream id in notification •Requirements –Custom notification script –Stream ids as custom attributes –Icinga2 v2.2 Apply For Rules
  30. 30. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA MONITOR THE MONITORING CORE •Check Plugin –Query Graylog2 Alert Stream API for Icinga 2 alerts –Use Stream ID for notifications & notes_url •See what‘s happening in Icinga 2 –Restrict views based on user roles –Debug plugin & check problems –Combine cluster mal-function log –Filter events –Additional dashboard
  31. 31. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA GRAYLOG2: GELFWRITER VISUALIZED
  32. 32. Demo
  33. 33. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA DEMO •Graylog2 0.91.x •Icinga 2 2.2.0 •check-graylog2-stream Plugin •Configuration –Graylog2 icinga2 stream & alert –Icinga2 check plugin & host/service/notification apply rules
  34. 34. „The FUTURE“
  35. 35. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA „THE FUTURE“ •Build your own stack •Combine existing interfaces into one –Graylog2 streams in Icinga Web 2 (ask Tom!) –Icinga 2 Events in Graylog2 (more? We want more!) •Correlate your monitoring events with events & logs of any kind •Think about –Simple and secure event receiver –Auto-Discover checkable objects from log alerts –Alert stream rules for monitoring
  36. 36. RESOURCES
  37. 37. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA •Code https://github.com/graylog2 https://github.com/icinga/icinga2 •Vagrant Box icinga2x-graylog2 New @ https://github.com/icinga/icinga-vagrant/ •Documentation http://www.graylog2.org/resources/documentation http://docs.icinga.org/icinga2/latest
  38. 38. WWW.GRAYLOG2.ORG | WWW.ICINGA.ORG #OSMC #GRAYLOG2 #ICINGA Q&A Web www.{graylog2,icinga}.org Releases github.com/{graylog2,Icinga} IRC #graylog2 #icinga on FreeNode Support support.{graylog2,icinga}.org Twitter twitter.com/{graylog2,icinga} …….. Everywhere! ? Questions & Answers

    Be the first to comment

    Login to see the comments

  • JoostHekman

    Nov. 22, 2014
  • daolong549

    Dec. 17, 2014
  • jonashagstrom

    Feb. 6, 2015
  • MehdiSayad

    Jun. 10, 2015
  • JunshanHe

    Nov. 18, 2015
  • DannySheehan

    Apr. 25, 2016
  • viperbjpn

    Jun. 2, 2016
  • bbkingtw

    Jun. 18, 2016
  • NguynNgcSng

    Jun. 29, 2016
  • lljokell

    Jun. 21, 2017

Presentation from OSMC 2014 http://www.netways.de/en/osmc/osmc_2014/program/ Bernd Ahlers, Graylog2 Michael Friedrich, Icinga

Views

Total views

38,209

On Slideshare

0

From embeds

0

Number of embeds

22,929

Actions

Downloads

131

Shares

0

Comments

0

Likes

10

×