Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monitoring with Icinga2 at Adobe

2,751 views

Published on

Talk given by Jordan Jethwa at Icinga Camp San Francisco 2016 - https://www.icinga.org/community/events/archive/2016-archive/icinga-camp-san-francisco/

Published in: Technology

Monitoring with Icinga2 at Adobe

  1. 1. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Monitoringwith Icinga2 Jordan Jethwa | Web Technologist, Online Experience Management
  2. 2. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Thank you Icinga Development, QA, and Support § Commitment to Open Source values § Technology § Release schedules Community § Mailing list, IRC, and Monitoring Portal users
  3. 3. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Online Experience Management About us § DevOps! § Develop platform code that runs on Adobe Experience Manager (AEM) § Support and maintain infrastructure § Automation, monitoring, and metrics Stakeholders § Content authors across multiple worldwide business units § Teams with integrations, etc Priorities § Accelerate content development activities § Reduce technical and maintenance debt § Provide a unified and consistent experience 3
  4. 4. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Online Experience Management (continued) Adobe@Adobe § Showcase technologies from Adobe § Collaborate with engineering teams § Innovate Environments § Complex § Landscapes (dev -> prod) § Platforms (legacy vs new, intranet, etc) § Differing SLAs 4
  5. 5. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Ops day to day stack slice 5
  6. 6. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Icinga2 Case Study Online Experience Management 6
  7. 7. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Move to DevOps (~2 years ago) An opportunity to evaluate, modernize, and experiment § Methodology § Technology stack Scalability § An ever growing infrastructure § More hands in the cookie jar: Dev + Ops § More frequent code and infrastructure changes Flexibility § All the things
  8. 8. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Why Icinga2? A few other evaluations § Nagios - NOPE! § New Relic, Datadog – Cons: Hosted, Java Agent byte-code injection § Sensu – Cons: Set up and maintenance Icinga2 § Object-based configuration format (macros, assignments, conditionals, and functions!) § Dependencies § Metrics and built-in graphite export § Icinga Web 2 § Compatible with Monitoring Plugins (https://www.monitoring-plugins.org/) § Icinga Exchange (https://exchange.icinga.org/) § REST API – Manage objects, retrieve information, perform actions like rescheduling checks
  9. 9. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Current Icinga2 stats 9 220+ hosts 2100+ checks 5m intervals ~ 1s execution time 1 server (failover)
  10. 10. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. ConfigurationDesign Focus on maintainability, flexibility, and automation Structure: § Tree-based § One file per configuration object (mostly) Groups: § Not just for visualization! Templates: § Apply a set of generic attributes to multiple objects Assignments – Match object attributes: § HostGroups and ServiceGroups § Custom variables 10 /etc/icinga2/conf.d/ |—- api-users.conf |—- app.conf |—- commands/ |—- downtimes/ |—- groups/ |—-—-—- hostGroups/ |—-—-—- serviceGroups/ |—- hosts/ |—-—-—- platform0/ |—-—-—-—-—- myhost.conf |—-—-—-—-—- myhost/ |—-—-—-—-—-—-—- myhost_specific0.conf |—- notifications/ |—- services/ |—-—-—- ssh.conf |—-—-—- platform0_specific/ |—-—-—-—-—-—-—- platform_specific0.conf |—- templates/ |—-—-—- hosts/ |—-—-—- notifications/ |—- users.conf
  11. 11. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. ConfigurationDesign (continued) Keep it simple and modular Objects: § Rely on inheritance § assign and ignore § Human readable Host object automation: § Current: Script generates from CMDB but, CMDB entry is manual L § Target: Chef 11 +-----------------------------+ |object Host "example_host" { | | import "rhel6-5" | | import "aem6-1" | | | | display_name "Example Host"| | address = "192.168.0.22" | | | | vars.landcape = "prod" | | vars.location = "eu-west-1"| |} | +-----------------------------+
  12. 12. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Guess the chart 12 0 10 20 30 40 50 60 70 80 90 100
  13. 13. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Millions of notifications go unread everyday 13 0 10 20 30 40 50 60 70 80 90 100 Notification fatigue Productivity Notifications You can make a difference Reduce notifications: § Notification intervals based on SLAs § Use dependencies! § Scheduled downtimes § Route using user groups Notification alternatives/additions: § Slack (https://github.com/jjethwa/icinga2-slack- notification) § PagerDuty § Push notifications (Pushover, etc)
  14. 14. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Performance Data 14 It’s all about metrics Enable performance data writer: § No Enable Graphite Carbon writer: § Pa Grafana: § Visualization – See trends, correlate data § Dashboards – Impress execs J Bonus round: § Icinga Web 2 graphite module (https://github.com/findmypast/icingaweb 2-module-graphite) # icinga2 feature enable perfdata # icinga2 feature enable graphite
  15. 15. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Docker Zero to Icinga2 in less than a minute § x § debian:jessie Resources: § Docker Hub: https://hub.docker.com/r/jordan/icinga2/ § GitHub: https://github.com/jjethwa/icinga2 Contributing a patch: § Submit an issue describing your proposed change to the repo in question. § I will respond to your issue promptly. § Fork the desired repo, develop and test your code changes. § Submit a pull request. 15 # docker pull jordan/icinga2 # sudo docker run -p 80:80 -t jordan/icinga2:latest
  16. 16. © 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Upcoming activities § Host object generation - Replace current CMDB script § Plug n Play Icinga2 satellites - Gather metrics from different office networks 16

×