Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Current State of Logmanagement with Icinga - Icinga Camp Stockholm 2019

269 views

Published on

Talk by Thomas Widhalm:
You want to collect your logs in one place and be alerted when there’s something going on an active check can’t find out? This talk is about the different ways of how to connect Icinga to the Elastic Stack from easily done to very sophisticated. As a bonus you will learn about how to parse Icinga logs with Logstash.

Published in: Technology
  • DOWNLOAD THI5 BOOKS 1NTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Current State of Logmanagement with Icinga - Icinga Camp Stockholm 2019

  1. 1. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Icinga 2 and Logs
  2. 2. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Thomas Widhalm ( @widhalmt ) • Lead Support Engineer @ Netways • Specialised in Icinga and Elastic Stack • Collector of Star Wars Lego and Camo patterns About me
  3. 3. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm About me
  4. 4. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Logs
  5. 5. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Every IT infrastructure has lots of them • Many admins don‘t really care • Focused view of one box • Filters and parsing on the fly – Different levels of knowledge – Rerun all filters every time you change something • Full harddisks or now long time storage • Only used when something bad happened Logs
  6. 6. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Different kinds of logs
  7. 7. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Logs of monitored objects • Logs of Icinga • Logs of alerts and notifications • Logs of logmanagement Logs and Icinga
  8. 8. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Give more thorough insight • Allow monitoring of otherwise inaccessible objects • Different ways of monitoring – Plugin / Agent – Logmanagement Logs of monitored objects
  9. 9. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Show how Icinga is doing • Help with monitoring and debugging • Very useful for post mortems and support tickets Logs of Icinga
  10. 10. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Part of Icinga logs • Can be used for SLA / umbrella monitoring • Basically show how monitored objects are doing Logs of alerts and notifications
  11. 11. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Often can‘t be processed by logmanagement itself (Loops!) • Show problems in logmanagement infrastructure Logs of logmanagement
  12. 12. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Toolset
  13. 13. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • No native way of monitoring logs • Plugins for monitoring logs – Statusmonitoring, no searching – Only single hosts Icinga
  14. 14. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Collects Logdata from many sources • Stores data in a central database • Monitoring addon – Not free – Cumbersome configuration • Connectors to many receivers – Cumbersome configuration – All but flexible Elastic Stack
  15. 15. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Search Server (based on Apache Lucene) • Elastic, highly available, load balanced very resilient • Extremely scalable • REST-API for communication Components of Elastic Stack: Elasticsearch
  16. 16. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Receive Logs from many sources – Syslog – Beats (Agents) • Send to many targets – Elasticsearch – Icinga • Parse, disect, transform, filter, enrich Logs Components of Elastic Stack: Logstash
  17. 17. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Webinterface for Elastic Stack • Search and filter logs • Build Dashboards for Screens or interactive drill drown Components of Elastic Stack: Kibana
  18. 18. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Lightweight agents • Collect Filelogs (syslog) or Event Log • More specialised beats available – Icingabeat – MySQL-beat – Redisbeat Components of Elastic Stack: Beats
  19. 19. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Common problems
  20. 20. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Hardly any „problem event“ has a corresponding „ok again“ event • Hearing nothing from your hosts: – Everything is fine – Too dead to talk End of the world or end of the problem?
  21. 21. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Automatically return to „OK“ after a while – Send notifications – Enrich with active monitoring • Have someone check – Can create lots of work Best effort
  22. 22. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Approaches
  23. 23. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Use Elastic Stack to collect and store Logmessages • Use Icinga for alerting • Different ways of connecting Combined forces
  24. 24. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Cumbersome configuration – Still easier than full blown logmanagement? • Not part of monitoring plugins • Independent from everything else – Use to avoid loops – Monitor logmanagement infrastructure Ye goode olde check_logfiles
  25. 25. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Full blown logmanagement solution • Several ways of ingestion – Reads logs from filesystem – Receives logs from „icingabeat“ agent • Several ways of monitoring – „icinga“ output to API – Icinga Web module „elasticsearch“ Elastic Stack & Icinga
  26. 26. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm $ yum install java-1.8.0-openjdk-devel $ /usr/share/logstash/bin/logstash-plugin install logstash-output-icinga Elastic Stack & Icinga
  27. 27. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm output { icinga { host => 'fornax.icinga-book.local' user => 'root' password => '***' action => 'process-check-result' action_config => { exit_status => 0 plugin_output => "%{[message]}" } icinga_host => '%{[host]}' icinga_service => 'logevent' } } Elastic Stack & Icinga
  28. 28. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Use all actions of the API – Process check result – Add hosts – Set downtimes • Decide which data to use from what logevent in Logstash config • Get results into Icinga in almost no time • Use passive checks with automatic recovery Elastic Stack & Icinga
  29. 29. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Icinga Web Modules „elasticsearch“
  30. 30. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Configure connection to Elasticsearch • Set filters to identify logs – Objectname in Icinga = Objectname in logs! – Enforce Icinga Web permissions on logs • Give Icinga users quick access to logs without allowing access to Kibana Elastic Stack & Icinga
  31. 31. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm # icingacli elasticsearch check --instance elastic01-hot --crit 5 --warn 3 --index logstash* --filter "beat.hostname=qa,source=/var/log/httpd/*.log" --from -5m OK - 0 hits Elastic Stack and Icinga
  32. 32. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Ready-to-use ruleset – https://github.com/Icinga/icinga-logstash-pipeline • Ingest Icinga Logs • Have data parsed from logs – „eventtype“ for every type of event – Data like endpointnames, objectnames etc extracted – Numbers like queuelengths extracted Elastic Stack and Icinga
  33. 33. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Elastic Stack and Icinga
  34. 34. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Elastic Stack and Icinga
  35. 35. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Use as a standalone Logstash pipeline – Input and output for Redis are provided – Clone git repo into configuration directory (and use *conf files) – Rest of minimal config is provided in Readme • Collect logs from masters, satellites, agents • Get the whole picture what‘s going on in your monitoring Elastic Stack and Icinga
  36. 36. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Filter for severity
  37. 37. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Restrict to facility or severity • Watch for spikes in logs • View message just like in the logfile Elastic Stack and Icinga
  38. 38. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm View details of an event
  39. 39. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Get Facility/Severity • „Eventtype“ for every kind of logmessage • Get related object (split into host, service, notification etc.) • Get message specific details (pluginoutput, exitcodes etc.) • Use all these fields for filters or graphs Elastic Stack and Icinga
  40. 40. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Use dashboards
  41. 41. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • Use dashboards for screenwalls – See problems arising before they get critical – Get fresh status update during problems • Use them interactively – Klick on parts of graphs to create quick filters – Make fast drilldowns – Filter every item on the dashboard at once (graphs, event lists) Elastic Stack and Icinga
  42. 42. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Detect anomalies
  43. 43. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • See unusual spikes in event flows – Big benefit even for experienced log-greppers • Get consolidated logs from all (or some) nodes – Drill down to the problem at hand Elastic Stack and Icinga
  44. 44. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm Questions and Answers
  45. 45. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm • thomas.widhalm@netways.de • thomas.widhalm@icinga.com • Twitter: @widhalmt • GnuPG: B50D AF2B 22A6 94E8 C195 9C89 DAAC 19AE A84C B603 Contact
  46. 46. 2019-09-03 | Icinga Camp Stockholm | Thomas Widhalm netways.de blog.netways.de git.netways.de sales@netways.de netways netways netways +49 911 92885 - 66 Contact

×