I will show how we designed our new central monitoring system based on Icinga2. Currently we are in the process of migrating our projects to this new system. While doing this, our teams learn a lot of new stuff and discover solutions to their specific problems. After describing, how we share our acquired knowledge to all teams, I will give a closer insight in one particular approach where we used Ansible to provision all our Icinga configuration via the Director interface.

1. DIRECTING THE DIRECTORDIRECTING THE DIRECTOR

2. T-Systems
15 years experience
monitoring
automation
databases
performance & debugging
ABOUT MEABOUT ME

3. WHAT CAN YOU EXPECTWHAT CAN YOU EXPECT
a little bit of history
implementing a monitoring system
the user perspective
some Ansible code
(live demo)

4. STARTING POINTSTARTING POINT

5. STARTING POINTSTARTING POINT
three years ago

6. STARTING POINTSTARTING POINT
three years ago
one central monitoring system

7. STARTING POINTSTARTING POINT
three years ago
one central monitoring system
central team handles all changes

8. STARTING POINTSTARTING POINT
three years ago
one central monitoring system
central team handles all changes
monitoring is requested and then implemeneted

9. WHAT DO WE WANT?WHAT DO WE WANT?
Monitoring as a Service …

10. WHAT DO WE WANT?WHAT DO WE WANT?
Monitoring as a Service …
… for our projects

11. WHAT DO WE WANT?WHAT DO WE WANT?
Monitoring as a Service …
… for our projects
… and our customers

14. THE FIRST TRY ™THE FIRST TRY ™

15. CURRENT STATECURRENT STATE

17. DOCKER WITHOUT BASH SCRIPTSDOCKER WITHOUT BASH SCRIPTS
Possible through Packer and Ansible

18. SHOW ME SOME CODESHOW ME SOME CODE

19. KUBERNETESKUBERNETES
did you just say cloud?

20. KUBERNETESKUBERNETES
did you just say cloud?
don’t ask me, I’m just a happy user

21. NOW WE HAVE A SAAS MONITORINGNOW WE HAVE A SAAS MONITORING
central team keeps up with all updates
also found some bugs in icinga and other software
a good self-service solution

22. THERE IS MORE TO ITTHERE IS MORE TO IT
central team supports our projects with consulting
shared knowledgebase
and a library of default checks and dashboards
also helping with migrations

23. BUT DO YOU ACTUALLY USE THIS?BUT DO YOU ACTUALLY USE THIS?

24. BUT DO YOU ACTUALLY USE THIS?BUT DO YOU ACTUALLY USE THIS?
currently running 37 instances

25. BUT DO YOU ACTUALLY USE THIS?BUT DO YOU ACTUALLY USE THIS?
currently running 37 instances
which support 94 different projects

26. BUT DO YOU ACTUALLY USE THIS?BUT DO YOU ACTUALLY USE THIS?
currently running 37 instances
which support 94 different projects
but still most old projects on SNMP/NRPE :(

27. DECENTRALIZED CHALLENGESDECENTRALIZED CHALLENGES
teams need synchronization
individual solutions need discussions
talk …

28. DECENTRALIZED CHALLENGESDECENTRALIZED CHALLENGES
teams need synchronization
individual solutions need discussions
talk … A LOT

29. OLD HABBITS DIE SLOWLYOLD HABBITS DIE SLOWLY
SNMP and NRPE still deeply embedded
self-service alone is insufficient, also need enablement
Icinga Director is a good starting point

30. BUT THEN, THINGS HAPPENBUT THEN, THINGS HAPPEN

31. API HEAVENAPI HEAVEN

32. IF YOU BUILD IT, THEY WILL COMEIF YOU BUILD IT, THEY WILL COME
Hmm, we are con guring our servers with
Ansible. Why not con gure Icinga too?

33. LOOKING INTO UPSTREAMLOOKING INTO UPSTREAM
searching for an Ansible module to manage our
configuration
there is an icinga2_host module, but no module uses
the Director API
:(

34. ROLL OUR OWNROLL OUR OWN
our implementation uses Ansible uri module

35. - name: see if service template already exists
uri:
headers:
Accept: application/json
url: "{{ icinga_host }}/director/service?
name={{ item.name }}"
method: GET
user: "{{ icinga_user }}"
password: "{{ icinga_pass }}"
return_content: yes
register: service_template_presence
with_items: "{{ checks }}"
failed_when: false

36. - name: create service template if it does not exist
uri:
headers:
Accept: application/json
url: "{{ icinga_host }}/director/service"
method: POST
user: "{{ icinga_user }}"
password: "{{ icinga_pass }}"
body: '{"check_command":"{{ item.item.check_command }}","obje
body_format: json
return_content: yes
register: service_template_created
with_items: "{{ service_template_presence.results }}"
when: "'error' in item.content"
changed_when: service_template_created.status == 201
failed when: service template created status != 201

37. - name: modify service template if it does exist
uri:
headers:
Accept: application/json
url: "{{ icinga_host }}/director/service?name={{ item.item.nam
method: POST
user: "{{ icinga_user }}"
password: "{{ icinga_pass }}"
body: '{"check_command":"{{ item.item.check_command }}","obje
body_format: json
return_content: yes
register: service_template_modified
with_items: "{{ service_template_presence.results }}"
when: "'error' not in item.content"
changed_when: service_template_modified.status == 200
failed when: service template modified status != 200

38. IMPLEMENTED FOR ALL ICINGA OBJECTSIMPLEMENTED FOR ALL ICINGA OBJECTS
Hosts
Services
Vars
Apply Rules
Users
Templates

39. CREATING AN APPLY RULECREATING AN APPLY RULE

40. - object_name: tomcat_user_processes
object_type: apply
imports:
- check_user_procs
assign_filter: "host.vars.tomcat_port=true"
vars:
username: "tomcat"
warning: "50"
critical: "80"

41. CREATED A ANSIBLE ROLE FOR SHARINGCREATED A ANSIBLE ROLE FOR SHARING

42. DISCOVERED SOME PROBLEMSDISCOVERED SOME PROBLEMS
complex configuration in a single file
no delete feature implemented
assign_filter has to be specified for every Rule

46. .
├── apply_rules
│ └── srv
│ ├── all.yml
│ ├── web.yml
│ ├── db.yml
│ ├── int
│ │ ├── all.yml
│ │ ├── db
│ │ │ └── 01.yml
├── icinga_command.yml
├── icinga_hostgroups.yml
├── icinga_hosttemplates.yml
├── icinga_notification.yml
├── icinga_service_template.yml
├── icinga timeperiods yml

47. WHAT IS GAINEDWHAT IS GAINED
hierarchical host configuration
short and concise configuration files
reuse of initial ansible tasks
deleting objects now possible

48. WHAT IS LEFTWHAT IS LEFT
needs to get faster
use a full fledged Ansible module
publish this to upstream

49. WAIT, THERE’S MOREWAIT, THERE’S MORE

50. SETTING DOWNTIMES WITH ANSIBLESETTING DOWNTIMES WITH ANSIBLE
- hosts: webserver
tasks:
- name: set downtime
icinga_downtime:
dt_task: "add"
type: "Service"
comment: "Deployment Application"
duration: "7200"
author: "ansible-playbook"
service:
- "tomcat_status_check"
- "tomcat_user_open_files"

51. DELETING DOWNTIMES WITH ANSIBLEDELETING DOWNTIMES WITH ANSIBLE
- hosts: webserver
tasks:
- name: remove downtime
icinga_downtime:
dt_task: "remove"
type: "Service"
comment: "Deployment Application"
duration: "7200"
author: "ansible-playbook"
service:
- "tomcat_status_check"
- "tomcat_user_open_files"

52. TALKING ABOUT NOTIFICATIONS …TALKING ABOUT NOTIFICATIONS …
we want all our production hosts and services to notify
us on failure 24x7

53. TALKING ABOUT NOTIFICATIONS …TALKING ABOUT NOTIFICATIONS …
we want all our production hosts and services to notify
us on failure 24x7
really all services?

54. TALKING ABOUT NOTIFICATIONS …TALKING ABOUT NOTIFICATIONS …
we want all our production hosts and services to notify
us on failure 24x7
really all services?
what about ntp or sssd

56. ICINGA TEXT CONFIGURATIONICINGA TEXT CONFIGURATION
assign where "Live" in host.groups && !(
match("oracle_soft_parse_ratio_*", service.name) ||
match("oracle_connected_users_*", service.name) ||
match("oracle_switch_interval_*", service.name))

57. NOW THE JSON VERSIONNOW THE JSON VERSION
assign_filter: "host.groups=%22Live%22&!(
service.name=%22oracle_soft_parse_ratio_%2A%22|
service.name=%22oracle_connected_users_%2A%22|
service.name=%22oracle_switch_interval_%2A%22)"

59. ACCEPTING THE REALITYACCEPTING THE REALITY
that will not work for long
really error prone
one typo can mix up all of our notifications!

60. ACCEPTING THE REALITYACCEPTING THE REALITY
that will not work for long
really error prone
one typo can mix up all of our notifications!
so this needs to be auto-generated

61. INPUTINPUT
voice_exempt:
- hostname: db
servicename: oracle_soft
- hostname: web
servicename: http_hits

62. PROCESSPROCESS
- name: generate single strings for exeptions
set_fact:
args:
voice_exempt_strings:
"{{ voice_exempt_strings|default([]) +
[ '("' + item.hostname + '" in host.display_name and
"' + item.servicename + '" in service.display_name)'
] }}"
with_items: "{{voice_exempt}}"

63. OUTPUTOUTPUT
# echo with debug module
msg: '( "db" in host.display_name and
"oracle_soft" in service.display_name)
&&
( "web" in host.display_name and
"http_hits" in service.display_name)
)'

64. RESULTRESULT
easy configuration
understandable array for exceptions

65. RESULTRESULT
easy configuration
understandable array for exceptions
not so understandable rule for transforming :/

66. RESULTRESULT
easy configuration
understandable array for exceptions
not so understandable rule for transforming :/
that we never need to touch :)

67. WHAT DID WE LEARNWHAT DID WE LEARN
encourage sharing (internal and external)
Ansible is awesome
trust!

68. DEMO TIMEDEMO TIME

69. THANKS!THANKS!
QUESTIONS?QUESTIONS?