This document discusses Adobe's use of Icinga2 for infrastructure monitoring. It provides an overview of Adobe's Online Experience Management team and environments, why they chose Icinga2, their current Icinga2 configuration and statistics, and how they design configurations for maintainability. It also covers notifications, performance data collection with Graphite, and using Docker to easily deploy Icinga2.

1. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Monitoringwith Icinga2
Jordan Jethwa | Web Technologist, Online Experience Management

2. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Thank you
Icinga Development, QA, and Support
§ Commitment to Open Source values
§ Technology
§ Release schedules
Community
§ Mailing list, IRC, and Monitoring Portal users

3. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Online Experience Management
About us
§ DevOps!
§ Develop platform code that runs on Adobe Experience Manager (AEM)
§ Support and maintain infrastructure
§ Automation, monitoring, and metrics
Stakeholders
§ Content authors across multiple worldwide business units
§ Teams with integrations, etc
Priorities
§ Accelerate content development activities
§ Reduce technical and maintenance debt
§ Provide a unified and consistent experience
3

4. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Online Experience Management (continued)
Adobe@Adobe
§ Showcase technologies from Adobe
§ Collaborate with engineering teams
§ Innovate
Environments
§ Complex
§ Landscapes (dev -> prod)
§ Platforms (legacy vs new, intranet, etc)
§ Differing SLAs
4

5. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Ops day to day stack slice
5

6. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Icinga2 Case Study
Online Experience Management
6

7. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Move to DevOps (~2 years ago)
An opportunity to evaluate, modernize, and experiment
§ Methodology
§ Technology stack
Scalability
§ An ever growing infrastructure
§ More hands in the cookie jar: Dev + Ops
§ More frequent code and infrastructure changes
Flexibility
§ All the things

8. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Why Icinga2?
A few other evaluations
§ Nagios - NOPE!
§ New Relic, Datadog – Cons: Hosted, Java Agent byte-code injection
§ Sensu – Cons: Set up and maintenance
Icinga2
§ Object-based configuration format (macros, assignments, conditionals, and functions!)
§ Dependencies
§ Metrics and built-in graphite export
§ Icinga Web 2
§ Compatible with Monitoring Plugins (https://www.monitoring-plugins.org/)
§ Icinga Exchange (https://exchange.icinga.org/)
§ REST API – Manage objects, retrieve information, perform actions like rescheduling checks

9. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Current Icinga2 stats
9
220+
hosts
2100+
checks
5m
intervals
~ 1s
execution time
1 server (failover)

10. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
ConfigurationDesign
Focus on maintainability, flexibility, and automation
Structure:
§ Tree-based
§ One file per configuration object (mostly)
Groups:
§ Not just for visualization!
Templates:
§ Apply a set of generic attributes to multiple objects
Assignments – Match object attributes:
§ HostGroups and ServiceGroups
§ Custom variables
10
/etc/icinga2/conf.d/
|—- api-users.conf
|—- app.conf
|—- commands/
|—- downtimes/
|—- groups/
|—-—-—- hostGroups/
|—-—-—- serviceGroups/
|—- hosts/
|—-—-—- platform0/
|—-—-—-—-—- myhost.conf
|—-—-—-—-—- myhost/
|—-—-—-—-—-—-—- myhost_specific0.conf
|—- notifications/
|—- services/
|—-—-—- ssh.conf
|—-—-—- platform0_specific/
|—-—-—-—-—-—-—- platform_specific0.conf
|—- templates/
|—-—-—- hosts/
|—-—-—- notifications/
|—- users.conf

11. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
ConfigurationDesign (continued)
Keep it simple and modular
Objects:
§ Rely on inheritance
§ assign and ignore
§ Human readable
Host object automation:
§ Current: Script generates from CMDB but, CMDB entry is manual L
§ Target: Chef
11
+-----------------------------+
|object Host "example_host" { |
| import "rhel6-5" |
| import "aem6-1" |
| |
| display_name "Example Host"|
| address = "192.168.0.22" |
| |
| vars.landcape = "prod" |
| vars.location = "eu-west-1"|
|} |
+-----------------------------+

12. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Guess the chart
12
0
10
20
30
40
50
60
70
80
90
100

13. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Millions of notifications go unread everyday
13
0
10
20
30
40
50
60
70
80
90
100
Notification fatigue
Productivity Notifications
You can make a difference
Reduce notifications:
§ Notification intervals based on SLAs
§ Use dependencies!
§ Scheduled downtimes
§ Route using user groups
Notification alternatives/additions:
§ Slack (https://github.com/jjethwa/icinga2-slack-
notification)
§ PagerDuty
§ Push notifications (Pushover, etc)

14. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Performance Data
14
It’s all about metrics
Enable performance data writer:
§ No
Enable Graphite Carbon writer:
§ Pa
Grafana:
§ Visualization – See trends, correlate data
§ Dashboards – Impress execs J
Bonus round:
§ Icinga Web 2 graphite module
(https://github.com/findmypast/icingaweb
2-module-graphite)
# icinga2 feature enable perfdata
# icinga2 feature enable graphite

15. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Docker
Zero to Icinga2 in less than a minute
§ x
§ debian:jessie
Resources:
§ Docker Hub: https://hub.docker.com/r/jordan/icinga2/
§ GitHub: https://github.com/jjethwa/icinga2
Contributing a patch:
§ Submit an issue describing your proposed change to the repo in question.
§ I will respond to your issue promptly.
§ Fork the desired repo, develop and test your code changes.
§ Submit a pull request.
15
# docker pull jordan/icinga2
# sudo docker run -p 80:80 -t jordan/icinga2:latest

16. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Upcoming activities
§ Host object generation - Replace current CMDB script
§ Plug n Play Icinga2 satellites - Gather metrics from different office networks
16