For the monitoring of our systems, we make extensive use of Icinga, its director, and the business process monitoring module. We also make broad use of automation (at least we try to!). In this talk we would like to tell you how we automated the monitoring of our services using our self-written Ansible collections. We will cover how we developed the Ansible components and how we use them. We’ll also show you what we plan to do with them in the future.

1. Still directing the Director and more!
Page 1 / 22

2. Contents
The Monitoring History
The Director-module
The Business Process-module
The Icinga-Collection
Q&A
Page 2 / 22

3. About us
Daniel Uhlmann
Sebastian Gumprich
Page 3 / 22

4. Directing the Director in 2019
Anyone remember this guy?
Page 4 / 22

5. What we did with our architecture
Central kubernetes cluster with icinga instances
at least 45 Icinga instances
Page 5 / 22

6. What problems we faced with our code
No exsisting upstream modules
instead use Ansible uri-Tasks
Page 6 / 22

7. What we did with our Ansible code
- name: create service template if it does not exist
uri:
headers:
Accept: application/json
url: "{{ icinga_host }}/director/service"
method: POST
user: "{{ icinga_user }}"
password: "{{ icinga_pass }}"
body: '{"check_command":"{{ item.item.check_command }}",
"object_name":"{{ item.item.name }}",
"object_type": "template", "use_agent":{{ item.item.use_agent }}}'
body_format: json
return_content: yes
register: service_template_created
with_items: "{{ service_template_presence.results }}"
when: "'error' in item.content"
changed_when: service_template_created.status == 201
failed_when: service_template_created.status != 201
Page 7 / 22

8. What we're doing now with the code
- name: Create servicetemplate with event command
t_systems_mms.icinga_director.icinga_service_template:
state: present
url: "{{ icinga_url }}"
url_username: "{{ icinga_user }}"
url_password: "{{ icinga_pass }}"
object_name: apache_check_servicetemplate
use_agent: false
event_command: restart_httpd
notes: "example note"
notes_url: "'http://url1' 'http://url2'"
Page 8 / 22

9. What we accomplished
Date Feature
04/2020 internal development started
05/2020 Modules on Github in v1.0.0
05/20 - 12/20 command_template, host_template, zones and endpoints, service
01/21 inclusion into Ansible
02/21 facts/info modules for all objects
03/21 usergroups, timeperiod_template
07/21 inventory-plugin
Page 9 / 22

10. Facts-/Info-modules
> ansible localhost -m t_systems_mms.icinga_director.icinga_host_info -a "query=customer-test-web01"
localhost | SUCCESS => {
"changed": false,
"objects": [
{
"address": "172.29.23.203",
"display_name": "customer-test-web01",
"imports": [
"StandardServer"
],
"object_name": "customer-test-web01",
"object_type": "object"
}
]
}
Page 10 / 22

11. Inventory Plugin
> ansible-inventory -i test.icinga_director_inventory.yaml --graph
@all:
|--@customer_abn:
| |--customer-abn-internet
| |--customer-abn-web01
|--@customer_dev:
| |--customer-dev-app01
| |--customer-dev-internet
| |--customer-dev-web01
|--@customer_live:
| |--customer-live-app01
| |--customer-live-db01
| |--customer-live-db02
| |--customer-live-internet
Page 11 / 22

12. Inventory Plugin
object Host "customer-live-web01" {
display_name = "customer-live-web01"
address = "172.29.23.204"
check_command = "hostalive"
max_check_attempts = "3"
check_period = "24/7"
retry_interval = 1m
enable_notifications = true
enable_active_checks = true
enable_passive_checks = false
enable_event_handler = false
enable_perfdata = true
volatile = false
vars.HostOS = "Linux"
}
Page 12 / 22

13. Inventory Plugin
> ansible customer-live-web01 -m debug
-a "var=hostvars[inventory_hostname]['vars']['HostOS']"
customer-live-web01 | SUCCESS => {
"hostvars[inventory_hostname]['vars']['HostOS']": "Linux"
}
Page 13 / 22

14. Icinga Director Stats
Page 14 / 22

15. Let's talk about the Business Processes Plugin (BP)
Optional Icinga2 module
Build custom process-based dashboards
Get visualized Overview over your monitored objects
Page 15 / 22

16. Example Business Process Node
Page 16 / 22

17. Collection development
First things first: manually play around with every feature
Check the requests made by the extension (we need those for the upload or
check part f.e)
Our first draft was a Ansible role:
{% if node.operator == 'or' -%}
{{ node.name }} = {{ checks|join(' | ') }}
{% elif node.operator == 'not' -%}
{{ node.name }} = {{ checks|join(' ! ') }}
{% elif node.operator == 'and' -%}
{{ node.name }} = {{ checks|join(' & ') }}
{% elif node.operator is defined -%}
{{ node.name }} = {{ node.operator }}: {{ checks|join(' + ') }}
{% endif %}
Page 17 / 22

18. Configuration overview
---
- hosts: localhost
roles:
- t_systems_mms.icinga_business_process.ansible_icinga_business_process
vars:
icinga_business_processes:
- title: project-live
description: Project Live Environment
owner: "{{ icinga_user }}"
menu: yes
statetype: soft
nodes:
- name: loadbalancer
displayname: Loadbalancer Pools
operator: and
visible: false
checks:
- name: lb01
type: service
service: lb_pool_project-live
- name: lb02
type: service
service: lb_pool_project-live
Page 18 / 22

19. The future
At this time all functionalities are already implemented
Create service templates as check commands to integrate monitoring-alerts
- service_apply_object:
- "check_bpm"
imports:
- domon-business-process
assign_filter: 'host.name="pseudo-host"'
vars:
config: $bpm_servive_name
process: $bpm_process
Page 19 / 22

20. Yet another Ansible Icinga collection?
Page 20 / 22

21. Yet another Ansible Icinga collection?
One Collection to rule them all!
https://github.com/T-Systems-MMS/ansible-collection-icinga
Page 21 / 22

22. Ending + Questions
Sebastian Gumprich
Sebastian.Gumprich@t-systems.com
https://github.com/rndmh3ro
Daniel Uhlmann
Daniel.Uhlmann@t-systems.com
https://github.com/xFuture603
https://twitter.com/xFuturecs
Page 22 / 22