OSMC 2017 | Ops and dev stories- Integrate everything into your monitoring stack by Carsten Köbke and Michael Friedrich
There are many tools and possibilities available in the Icinga monitoring ecosystem. Setup Icinga 2 within a distributed architecture, put Icinga Web 2 on top and visualise alerts. Further you’ve already setup your preferred graphing solution (Graphite, InfluxDB, etc.) and Grafana greets you with a shiny metrics dashboard. Log events are processed and Elastic Stack or Graylog add more possibilities to correlate them with monitoring alerts. Having so many tools requires you to know how to connect them, or simply integrate an existing data source or frontend into your all-in-one user dashboard. Be it Graphs in your detail view, additional log events on a critical service view or yet a global map for location based alerts. Maybe you’ll also want to provide as much details in your notifications as possible, think of fancy Grafana graphs. This talk dives into existing and possible integrations, and explains why you sometimes just write your own integration. Carsten is the author of the Icinga Web 2 Grafana module while Michael focusses on Dashing and log processing. We’ll catch up with war stories (“meh, nothing exists”) to hero stories (“hooray, users love my integration”) and hope to motivate everyone out there doing the same. We will continue with practical development on Friday during the OSMC hackathon.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to OSMC 2017 | Ops and dev stories- Integrate everything into your monitoring stack by Carsten Köbke and Michael Friedrich
Similar to OSMC 2017 | Ops and dev stories- Integrate everything into your monitoring stack by Carsten Köbke and Michael Friedrich (20)
Recently uploaded
Recently uploaded (20)
OSMC 2017 | Ops and dev stories- Integrate everything into your monitoring stack by Carsten Köbke and Michael Friedrich
- 1. Ops and Dev Stories Integrate everything into your monitoring stack #OSMC | 22nd November 2017 | Nuremberg
- 8. METRICS
- 9. METRICS Overview •„Time series data bases“ (TSDB) •RRDTool -> PNP •Graphite •InfluxDB
- 11. METRICS Overview •Manage default templates •Custom dashboards •Annotations with external sources
- 14. METRICS Icinga Web 2 •Grafana Render API •Download •Direct •Detail View Image
- 15. METRICS Icinga Web 2 •Additional Parameters for Grafana API •Host/Service •Check Command •Custom Attributes
- 17. METRICS Icinga Web 2 •„Community in the making“ •Respond to bugs/feature requests •Review PRs •Write documentation for the first success
- 18. MAPS
- 21. MAPS Openstreetmap •Host object needs “geolocation“ attribute vars.geolocation = “<longitude>,<latitude>“ object Host "be-host" { vars.geolocation = "52.524372,13.389856" } Reference: https://www.icinga.com/2017/08/16/integrate-maps-into-icinga/
- 23. MAPS Openstreetmap •Fetch geolocation via JS call •Points as JSON data •Render with leaflet.js
- 24. MAPS Openstreetmap •More „Goodies“ •Host Popup with service list/states •Worst state calculation •Default zoom config
- 26. DASHBOARDS
- 30. List of problems: How many problems do you have?
- 31. DASHBOARDS Icinga Web 2 •Reduce problems to NUMBERS •Add static details & short readable lists •No clicks – view it in the office
- 33. DASHBOARDS Dashing •Saw it once in the NETWAYS office •Open tickets, current tasks, any problems •Colors for state indication
- 34. DASHBOARDS Dashing •Ruby, HTML, CSS, JS, Coffeescript •WTF •Docs – wtf •Ok, let‘s just copy an existing one
- 36. DASHBOARDS Dashing •Dashboard •Add Widget types •Define widget data as „data-id“ •Static HTML (browser drag&drop via JS)
- 37. DASHBOARDS Dashing •Jobs •Run every X seconds •Get data from source •Send event to „data-id“
- 39. TODO: Screenshot
- 40. DASHBOARDS Dashing •Use „dashing-icinga2“ as basis •Use Linux or a PI client. No Windows. •Development is SUPER HARD •Wrote some development docs
- 41. TICKET SYSTEMS
- 42. TICKET SYSTEMS Overview •Notifications create tickets •Ticket Update => Acknowledge in Icinga •Ticket Close => Update comment in Icinga
- 43. TICKET SYSTEMS Overview •OTRS: SystemMonitoring •Request Tracker Modules •HP ITSM Scripts •.......
- 44. TICKET SYSTEMS Overview •Doing more with Request Tracker lately •Replace the N-word module •Use Icinga 2 REST API actions everywhere •Let‘s do it on the hackathon J
- 45. LOGS & EVENTS
- 46. Who is using Elastic Stack or Graylog?
- 49. LOGS & EVENTS Overview •Beats •filebeat and icingabeat •Icinga 2 •GELF Writer •Elasticsearch Writer (2.8)
- 50. Host is down in Icinga Web 2 + Access to syslogs & events = Central view for faster problem analysis
- 51. Nice idea J Doesn‘t exist yet. Write a module?
- 52. LOGS & EVENTS Elastic Stack •HostActions Hook •Not embedded (would refresh too often) •Query Elasticsearch/Graylog REST API •List entries •Filters, fields
- 53. LOGS & EVENTS Elastic Stack •VW sponsored Elasticsearch integration •Join Bernd‘s talk later
- 54. LOGS & EVENTS Graylog •How about Graylog? •“Streams“ with pre-filtered data •Query data (same as monitoring list) •Apply filters •Pagination
- 55. LOGS & EVENTS Graylog •DEV code from Elasticsearch VW project •Icinga Vagrant boxes with PHP 7.1 •LOTS OF TIME (actually, not) •An early prototype
- 58. IDEAS
- 59. IDEAS Ours •Workflow integration (toolbars, widgets) •Central view on your monitoring •Icinga Web 2 modules/integrations •More dashboards •“Everything which has an API“
- 60. Your ideas for the OSMC hackathon (no promises ;))
- 61. IDEAS Overview •Access state data from http://localhost/icingaweb2/monitoring/list/ hosts?format=json •Client side JS which does „something“
- 62. REFERENCES Everything mentioned in this talk • https://github.com/Mikesch-mp/icingaweb2-module-grafana • https://github.com/nbuchwitz/icingaweb2-module-map • https://github.com/dnsmichi/icingaweb2-module-osmc • https://www.youtube.com/watch?v=GR1QguEdilg