Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monitor OpenStack Environments from the bottom up and front to back

Talk given by Thomas Stocking at Icinga Camp San Francisco 2016 - https://www.icinga.org/community/events/archive/2016-archive/icinga-camp-san-francisco/

  • Login to see the comments

Monitor OpenStack Environments from the bottom up and front to back

  1. 1. Monitor OpenStack Environments From the Bottom Up and Front to Back Thomas Stocking, Director of Systems Engineering, Founder. GroundWork Open Source Inc. February 16, 2016 | Icinga Camp
  2. 2. What’s ahead? • Overview—the impact of virtualization on IT operations • How OpenStack fits into the virtualization landscape • Monitoring the changing landscape of IT infrastructure • New monitoring concepts • Selecting the right tools to fit the right process • Conclusion 2 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. CONFIDENTIAL
  3. 3. Start with the facts: OpenStack is for real! 3 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. OpenStack deployments are not just happening in a far away land… It’s all open source. 5000 active members, and growing. Not just a geek movement. Serious deployments for IT operations, not just in the Silicon Valley. It’s disruptive and requires serious retooling for IT operations. Many corporate sponsors. Let’s review the challenges and discover which tools best fit the new realities…
  4. 4. Once upon a time… SysAdmin task: Add to IT infrastructure { Deploy servers into data center Provisioning applications Define monitoring for each element Monitoring: SSH checks and port gets } repeat Checking for up down and send email 4 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. MONITOR SERVER A P P S A P P S SERVER A P P S A P P S SERVER A P P S SWITCH SWITCH FIREWALL ROUTER PORT GET SSH SNMP SNMP SNMP
  5. 5. VIRTUALIZATION Then came along compute virtualization… Data center compute optimization • Server resources were virtualized to improve efficiency which was <15%. • Products like ESX allowed resource optimization not disturbing the provisioning process. Rollout • VM machines were provisioned. • Applications installed. • Monitoring defined as before. API was added to the virtualization manager (example Vsphere API) 5 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. MONITOR SERVER A P P S A P P S SERVER A P P S A P P S SERVER A P P S SWITCH SWITCH FIREWALL ROUTER PORT GET SSH SNMP SNMP SNMP REST API
  6. 6. MONITOR SNMP VIRTUALIZATION But it didn’t stop there… Software defined everything • Compute (hypervisors) • Storage (SDS) • Network (SDN) Hybrid Cloud Public/Private • Amazon Web Services • Rackspace • Azure Change created blind spots in coverage. Suddenly, SSH/SNMP methods don’t cover everything. 6 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. SERVER A P P S A P P S SERVER A P P S A P P S SERVER A P P S SWITCH SWITCH FIREWALL ROUTER PORT GET SSH REST API REST API SNMP SNMP VIRTUALIZED NETWORK VIRTUALIZED NETWORK SWITCHES SWITCHESSWITCHES
  7. 7. Files Devices VMs Shared I/O Fog Network & Storage The infrastructure landscape completely changed. 7 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. SILOVIEWSILOVIEWSILOVIEWSILOVIEWSILOVIEW HYPERVISORMANAGERVIEW
  8. 8. VIRTUALIZATION And the moment you think you’ve seen it all… DevOps is pushing the envelope even further… Linux Containers are the new kids in town API for deployment: Monitoring is somebody else’s job 8 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. MONITOR SERVER A P P S A P P S SERVER A P P S A P P S SERVER A P P S SWITCH SWITCH FIREWALL ROUTER PORT GET SSH VIRTUALIZED NETWORK VIRTUALIZED NETWORK SWITCHES SWITCHESSWITCHES REST API REST API DOCKER HOST DOCKER HOST DOCKER HOST Application Isolation Rapid Deployment Elastic Scalability
  9. 9. ESX NSX vSan Nova Neutron Cinder Glance EC2 VPC S3 Linux KVM Network NFS Diverse Virtualization Stacks Don’t panic. What happened over the last 5 years: • Virtualize everything • Private and public clouds • API-centric world is not just for applications • REST API is a standard • CORBA, SOAP and proprietary API’s are now classified as dinosaurs • Browser based UI/JavaScript is king • Learn to speak REST and JSON 9 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. vSphere API OpenStack API AWS API oVirt API
  10. 10. Sounds really good, doesn’t it? • Service-oriented architecture • Pluggable hypervisors, network, storage to support a wide range of technologies • Elastic Compute Units for better virtualization efficiency • Standardized API’s to all services But what’s the reality? How does OpenStack fit in? 10 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Object Store–Swift Dashboard– Horizon Image– Glance Compute/Hyperviso r–Nova Storage–Cinder Network– Quantum-Neutron Identity– Keystone Façade Service API API API API API API
  11. 11. OpenStack Profile Editor 11 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
  12. 12. OpenStack Profile Editor 12 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
  13. 13. Performance Data Virtual Infrastructure Container Metrics Provisioning/Config uration ManageIQ/ Cloud Forms Ansible API Manager Stack API HP HelionAPI VMWare VRealize Mirantis Fuel API API API is good, but comprehensive coverage is even better... 13www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Dashboard Object Store Image Computer Storage Network Identity OpenStack API Legacy Network Servers ServersServers Storage Racks Applications, Infrastructure Checks SSH/SNMP/port get Unified View Standardized Data Collection
  14. 14. Unified View Silo Tools • Impossible to correlate • Naming mismatch • Over-monitoring • Encourages “It’s not my problem, check your system” • No Big Picture dashboards 14 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Need For • Stack monitoring bottom to top • Aliasing of names • Combine best of breed collectors • Correlated metrics across all infrastructure • Dashboard for each customer/client
  15. 15. Stack monitoring for OpenStack 15 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Function Source Host name Alias Network, Storage Netflow SNMP Dsw1-422 os-eng-h1 Hardware IPMI SNMP Drac-server1 os-eng-h1 Identity server check-mysql check-port server1:port os-eng-h1 Operating System check-proc check-mem check-load server1 os-eng-h1 OpenStack-API Hypervisor CPU/Mem metrics, Network storage 10.10.0.1 os-eng-h1 OpenStack API VM CPU/mem, Network, storage 10.10.10.123 eng-slicer-1 Operating system check-proc check-mem check-load eng-cent6-actg eng-slicer-1 DockerHost Memory, CPU eng-cent6-actg eng-slicer-1 Container Memory, CPU, Procs FAC3443DA77 eng-slicer-1 Application Check_https 172.28.102.51 eng-slicer-1 HYPERVISORVMAPP
  16. 16. Automation/Continuous configuration discovery Adding virtual machines, containers is automatic… Monitoring is like a flight recorder—collecting all active data “Continuous configuration discovery” by… Synchronization of configuration over APIs Auto-registration of agents/Download of plugins Collection of data availability and performance 16 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. V C HYPERVISOR Management Monitoring System Configuration collector Data collector Plugins Dashboards D A T A C C V V V V V C so is the monitoring
  17. 17. OpenStack Monitoring: Tools Selection • Fuel—OpenStack deployment • Murano—Application deployment • Fuel Plugin—Configuration generate • Fuel Plugin—Configure monitoring 17 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. OpenStack API Icinga2 setup OpenStack Linux OS Platform Hypervisor Hardware IPMI Port Agent OpenStack ControllerKeystone OpenStack StorageCinder Unified Monitoring View Data Management CollectorCollectorCollector Fuel Murano Application Software Catalog Icinga 2 Docker Node 2 Node 3 Docker VM 1 OPEN TSDB Icinga2 VM 1 VM 2 VM 3 VM 4 OPEN TSDB Docker
  18. 18. Performance Data Monitoring Systems 18 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Centralized Data Collector • Monitoring send Perf Data to single API Expandable Storage Cluster • Expand on demand Dashboard to visualize and drill down • Historic, raw performance data • Group alike metrics to find outliers Grafana Dashboards
  19. 19. Reference Architecture Monitoring 19 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Aliasing and data normalization in backend A P I Data Collector Hub Collector Collector Collector Collect Data Integration Normalization Aliasing A P I Integrate A P I Visualize Growing number of API’s require integration Hub
  20. 20. Benefits Automation • Reduced cost of maintenance • Dynamic configuration • Streamlined operations Personnel • System Administrator and DevOps functions merging 20 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide. Capacity & Resource Planning • Complete Bottom-to-Top (Network-to-Application) data collection • Allocate resources at the right level Hardware • No vendor lock-in • Transparency through API
  21. 21. Lessons learned… • Virtualization and containerization require new monitoring techniques • Legacy hardware/software will be around for awhile, so don’t throw tools away • Everything API needs integration • Aliasing is hard, but may be easier than cross silo cooperation • Use the best tool for the task • Don’t over-monitor to cover a gap 21 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
  22. 22. Conclusion • Open source tools are the driving force for innovation • DevOps just select the best tools to do the job • A single monitoring tool to “rule them all” doesn’t exist • Integration is a complex task—don’t expect teams to agree on naming, process, and workflow • Automation and pragmatism will prevail as DevOps has demonstrated • “Virtualize everything” will continue • Automation and continuous discovery is necessary for rapid scale out • OpenStack is the open source virtualization platform, but monitoring coverage needs major improvements 22 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.
  23. 23. Thank you. Thomas Stocking tstocking@gwos.com 23 www.gwos.com ©2016 Groundwork Open Source, Inc. All rights reserved worldwide.

×