Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Project 3M: Meaningful Monitoring and Messaging

62 views

Published on

Sensu has become a critical component to keeping the modern visual effects studio of Industrial, Light & Magic in the business of creating the beautiful movies of our world and realizing the dreams we all enjoy on the big screen.

In this talk from Sensu Summit 2018, Christopher J. Caillouet,
Senior Dev|Ops Production Engineer at Industrial Light & Magic, looks behind the curtain and sees how the intelligence and uptime they gain by leveraging Sensu in the ILM monitoring infrastructure enables reliability and stable delivery within a large scale and geographically distributed set of datacenters.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Project 3M: Meaningful Monitoring and Messaging

  1. 1. MEANINGFUL MONITORING & MESSAGING PROJECT 3M
  2. 2. PRESENTER Christopher Caillouet ILM Dev|Ops Engineering
  3. 3. What do businesses want from monitoring?
  4. 4. BUSINESS INTELLIGENCE THROUGH MONITORING Meaningful Monitoring & Messaging
  5. 5. ●100% Open Source!! (Spend dollars for significant value) ●Ansible deployed infrastructure ●Ansible deployed services ●Dev, Staging and Production environments ●Sensors and Templates authored via code (YAML & Ruby) ●Monitoring occurs within datacenter local to monitored object ●Data visualization can occur across global footprint ●Scalable approach as we grow in sensor/service/studio count ●Flexible tool set for diverse consumption of platform PROJECT 3M DIRECTIVES
  6. 6. DEV STAGING PRODUCTION OUR PIPELINE APPROACH
  7. 7. Meaningful Monitoring & Messaging PLATFORM BREAKDOWN
  8. 8. Sens u
  9. 9. Ansible Leveraging our extensive internal set of playbooks during handled events is building our path toward automated remediation of issues
  10. 10. Jenkins We are always in need of intelligence around build times and job status metrics to ensure we are able to deliver code to meet the needs of the studio
  11. 11. SSL Keeping us in touch with the validation and expiration of the various certificates we have throughout our services and systems
  12. 12. CPU Systemd SNMP Zendesk Docker Vsphere Gitlab HTTP Varnish PostgreSQL Mongodb Consul Google Sheets Memory Supervisor Graylog LDAP Kubernetes Openstack Tomcat Memcached Elasticsearch RabbitMQ Redis Vault Hipchat BUT WAIT… ...THERE’ S MORE!
  13. 13. CONSUL
  14. 14. Consu l
  15. 15. Elasti c SERVER DATA SERVER DATA SERVER DATA SERVER DATA TIMESERIES&LOGDATA
  16. 16. GRAFANA
  17. 17. Grafan a SENSU DATA CONSUL DATA ELASTIC DATA CUSTOM DATA INTERSECTIONSOFDATA
  18. 18. Ansible Filename: your_service.yml --- - name: Configure 3m monitoring for your_service hosts: your_service.lucasfilm.com become: true vars: mmm_service_name: "your_service" mmm_service_port: 80 mmm_service_process_name: "nginx" roles: - 3m_service - custom_role Sampl e
  19. 19. Ansible Filename: your_service.yml Would give you… 1.Service Discovery in Hashicorp Consul a.Consul infrastructure monitored within Sensu 2.Centralized logging via Sensu handlers into ELK 3.Sensu monitoring of PING, CPU, RAM, HDD for base role a.additional Ansible roles added for plugin based check(s) b.additional Ansible roles added for custom authored check(s) c.additional or custom Sensu handlers i. elasticsearch, chat, email, SMS… and more! 4.Notifications for alerts where you want to get them 5.Customizable Grafana dashboard for your service Sampl e
  20. 20. QUESTIONS, YOU HAVE?
  21. 21. THANK YOU!
  22. 22. THANK YOU!
  23. 23. Filename: /etc/consul.d/3m_your_service.json { "services": [ { "name": "your_service", "tags": ["3m"], "port": 80, "check":{ "args": ["curl", "localhost:80"], "interval": "20s" } } ] } consul. d
  24. 24. Filename: /etc/sensu/conf.d/client.json { "client": { "name": "your_service.lucasfilm.com", "address": "1.2.3.4", "environment": "prod", "subscriptions": ["3m"] } } sensu.d
  25. 25. THANK YOU!

×