Oreilly solinea-managing-openstack

1,432 views
1,181 views

Published on

Published in: Technology

Oreilly solinea-managing-openstack

  1. 1. Accelerating the adoption of Cloud Computing Beyond Installation: Managing Your OpenStack Cloud May 6th, 2014
  2. 2. !  Ken Pepple is the co-founder and Chief Technology Officer of Solinea !  Prior to founding Solinea, he led the introduction of Internap's OpenStack-based public cloud services while serving as their Director of Cloud Development !  Code contributor since Bexar release of OpenStack !  Author of O'Reilly Media's "Deploying OpenStack" and several other books 2 Speakers
  3. 3. Introduction !  Installing OpenStack gets all the attention … !  … but distributions like Red Hat OSP and Cloudscaling are attacking this problem !  They will beat it. Then what ? !  The reality is that OpenStack management is what we should be focusing on. !  Installation is 2 – 3 weeks … management is forever 3
  4. 4. OpenStack Architecture OpenStack Object Store OpenStack Image Service OpenStack Compute OpenStack Dashboard OpenStack Identity Service OpenStack Compute API / Admin API keystone (service & admin APIs) nova-api (OS, EC2, Metadata, Admin) nova-consoleauth nova-cert/ objectstore nova-console nova-*proxy VNC/Spice OpenStack Object API http://www.solinea.com Queue nova-compute nova-scheduler nova database OpenStack Compute API OpenStack Image API Horizon OpenStack Image API identity backend swift-proxy objectcontaineraccount object store account DB container DB OpenStack Object API HTTP(S) OpenStack Object API OpenStack Identity API OpenStack Identity API OpenStack Identity API OpenStack Image API OpenStack Identity API OpenStack Image API catalog backend token backend OpenStack Identity API hypervisor libvirt, XenAPI, etc. HTTP(S) Amazon Web Services EC2 API Internet / Enterprise Network OpenStack Network Service glance-api glance database OpenStack Block Storage OpenStack Block Storage API cinder-api cinder-volume neutron-server neutron plugin(s) OpenStack Identity API cinder-scheduler cinder database OpenStack Network API network provider OpenStack Block Storage API OpenStack Network API policy backend Queue OpenStack Network API neutron database neutron agent(s) nova-conductor memcached OpenStack Identity API ⁃ OpenStack Command Line Tools (nova-client, swift-client, etc.) ⁃ Cloud Management Tools (Rightscale, Enstratius, etc.) ⁃ GUI tools (Cyberduck, iPhone client, etc.) volume provider cinder-backup OpenStack Object API OpenStack Identity API Queue OpenStack Block Storage API OpenStack Orchestration heat-api heat-engine heat database Queue cloudwatch-api OpenStack Orchestration API OpenStack Identity API OpenStack Compute API OpenStack Bock Storage API OpenStack Network API glance-api OpenStack Database trove-api trove-taskmgr trove database trove-conductor OpenStack Identity API OpenStack Database API OpenStack Block Storage API OpenStack Orchestration API OpenStack Compute API Agent Queue 4 * Ceilometer omitted for clarity
  5. 5. OpenStack Deployment 5
  6. 6. OpenStack Management Basics !  Development and test cluster –  Smaller, but representative –  Same set and version of services –  Reproduce problems, test fixes and practice upgrades !  Configuration management system –  Chef, Puppet, Ansible, SaltStack, etc. –  Your OpenStack distribution already uses one –  Pick one and stick with it – everything falls under it !  Skilled and trained staff –  Experienced Linux admins with virtualization skills –  Network architects that understand cloud –  Trained for OpenStack 6
  7. 7. Developing Toolkit for Management !  Troubleshooting tools –  Operating system level tools –  OpenStack specific tools !  Administration tools –  OpenStack specific tools !  Monitoring tools –  Monitoring platforms –  Log management tools !  Specialized tools –  Image creation 7
  8. 8. Troubleshooting Tools !  Tools used to investigate or fix problems within your stack !  Mostly Linux tools but some are specific to OpenStack !  These need to span virtualization, networking and normal system administration 8
  9. 9. Troubleshooting Hypervisor !  Vary by hypervisor, each one has it’s own tooling !  Map VM to hypervisor by OpenStack CLI with nova show! !  Investigate hypervisor through virsh tools !  Also can access backing store for VM through hypervisor mount point or Cinder volume 9
  10. 10. VM Troubleshooting # nova list! +---------------------+-------+---------+------------+-------------+-------------------------------------+! | ID | Name | Status | Task State | Power State | Networks |! +---------------------+-------+---------+------------+-------------+-------------------------------------+! | f94b097d-b030-473b- | ken | ACTIVE | - | Running | rdonet=192.168.90.11 |! +---------------------+-------+---------+------------+-------------+-------------------------------------+! # nova show f94b097d-b030-473b-86a3-d501091c650b! +--------------------------------------+------------------------------------------------------------+! | Property | Value |! +--------------------------------------+------------------------------------------------------------+! | OS-EXT-AZ:availability_zone | nova |! | OS-EXT-SRV-ATTR:host | localhost.localdomain |! | OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain |! | OS-EXT-SRV-ATTR:instance_name | instance-0000000e |! | OS-EXT-STS:power_state | 1 |! | OS-EXT-STS:task_state | - |! | OS-EXT-STS:vm_state | active |! | OS-SRV-USG:launched_at | 2014-05-06T06:13:01.000000 |! | created | 2014-05-06T06:11:55Z |! | flavor | m1.small (2) |! | hostId | 7e31bda83a3586907464e8e68f83a035bf9fa500d9579b2b807fa9f0 |! | id | f94b097d-b030-473b-86a3-d501091c650b |! | image | cirros-0.3.2-x86_64 (f66d54e8-f8bd-4220-930f-86b6b44dfe4d) |! | rdonet network | 192.168.90.11 |! | security_groups | default |! | status | ACTIVE |! +--------------------------------------+------------------------------------------------------------+! # vish list! Id Name State! ----------------------------------------------------! 1 instance-0000000e running! 10
  11. 11. Troubleshooting Backing Store (Ephemeral) # cd /var/lib/nova/! # ls! buckets CA images instances keys networks tmp! # cd instances/! # ll! total 16! drwxr-xr-x. 2 nova nova 4096 May 6 09:03 13e86b72-7e14-43f5-ab2f- e7abf117213f! drwxr-xr-x. 2 nova nova 4096 May 2 11:25 _base! -rw-r--r--. 1 nova nova 45 May 5 23:18 compute_nodes! drwxr-xr-x. 2 nova nova 4096 Apr 30 19:28 locks! # cd 13e86b72-7e14-43f5-ab2f-e7abf117213f/! # ll! total 208! -rw-rw----. 1 qemu qemu 0 May 6 09:04 console.log! -rw-r--r--. 1 qemu qemu 262656 May 6 09:03 disk! -rw-r--r--. 1 nova nova 79 May 6 09:03 disk.info! -rw-r--r--. 1 nova nova 1529 May 6 09:04 libvirt.xml! # file disk! disk: Qemu Image, Format: Qcow , Version: 2! 11 The ‘disk’ file is our qcow image. The XML file is the KVM template.
  12. 12. Troubleshooting Network !  Combination of Linux, OpenvSwitch and OpenStack tools !  OpenStack tools will show logical configuration of Neutron’s ports, routers and subnets –  neutron port-list, net-list and router- list! !  OpenvSwitch will map internal and external bridges –  ovs-vsctl and ovs-dpctl! !  Linux tools will show you inside VLAN and Linux namespaces –  ip netns, iptables and tcpdump! 12
  13. 13. Network Troubleshooting: Router # ip netns show! qdhcp-8a496b23-ef2c-4170-9919-611d9a12180f! qrouter-e439ff2a-1973-4cda-86a4-20c977eec828! qdhcp-16b5549e-3a1a-4254-b122-f7507f003597! ! ! # ip netns exec qrouter-e439ff2a-1973-4cda-86a4-20c977eec828 ifconfig! qg-bbe18331-0c Link encap:Ethernet HWaddr FA:16:3E:47:63:D0! inet addr:192.168.57.132 Bcast:192.168.57.255 Mask:255.255.255.0! inet6 addr: fe80::f816:3eff:fe47:63d0/64 Scope:Link! UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1! RX packets:1335 errors:0 dropped:0 overruns:0 frame:0! TX packets:145 errors:0 dropped:0 overruns:0 carrier:0! collisions:0 txqueuelen:1000! RX bytes:197195 (192.5 KiB) TX bytes:13110 (12.8 KiB)! ! qr-c4e2b047-4a Link encap:Ethernet HWaddr FA:16:3E:FD:4E:A9! inet addr:192.168.90.1 Bcast:192.168.90.255 Mask:255.255.255.0! inet6 addr: fe80::f816:3eff:fefd:4ea9/64 Scope:Link! UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1! RX packets:364 errors:0 dropped:0 overruns:0 frame:0! TX packets:309 errors:0 dropped:0 overruns:0 carrier:0! collisions:0 txqueuelen:1000! RX bytes:34760 (33.9 KiB) TX bytes:36569 (35.7 KiB)! ! ! ! # ip netns exec qrouter-e439ff2a-1973-4cda-86a4-20c977eec828 netstat -nr! Kernel IP routing table! Destination Gateway Genmask Flags MSS Window irtt Iface! 192.168.57.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-bbe18331-0c! 192.168.90.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-c4e2b047-4a! 13
  14. 14. Troubleshooting OVS Bridges # ovs-vsctl show! 06667946-811b-4c7b-97a5-eafc8386e9ff! Bridge br-int! Port "qvo246622d1-02"! tag: 2! Interface "qvo246622d1-02"! Port "tap3dfc8b70-ee"! tag: 1! Interface "tap3dfc8b70-ee"! Port "tapecef7610-4f"! tag: 2! Interface "tapecef7610-4f"! Port "tapc4e2b047-4a"! tag: 2! Interface "tapc4e2b047-4a"! Port br-int! Interface br-int! type: internal! Bridge br-ex! Port br-ex! Interface br-ex! type: internal! Port "eth1"! Interface "eth1"! Port "tapbbe18331-0c"! Interface "tapbbe18331-0c"! ovs_version: "1.11.0"! 14 Neutron’s integration bridge connecting VMs Neutron’s external bridge Physical NIC for internet access
  15. 15. Monitoring !  Metering is not monitoring –  Ceilometer isn’t a monitoring solution !  Horizon doesn’t save history !  Monitor for FCAPS: fault, configuration, accounting, performance and security !  Needs to be instrumented at multiple levels –  Hardware/Operating System, OpenStack, VM –  Although VM monitoring may be left to the user !  Needs to be used across all elements and processes 15
  16. 16. Operating System Monitoring !  Required set of information as any other set of systems –  CPU, memory, availability, etc. !  Process level information –  RabbitMQ, database, OpenStack processes, etc. !  Should rely on host sending information to monitoring server (not ping model) !  Ideally has APIs and strong discovery to aid automation 16
  17. 17. !  Installed as part of many distributions !  Open source !  Easy installation and usage !  API is an add-on module Nagios 17
  18. 18. Nagios OpenStack Plugin !  Add service checks for some OpenStack services –  Glance –  Keystone –  Nova –  Swift API and dispersion !  Available in most Linux distributions –  # sudo apt-get install nagios-plugins-openstack! !  More information and checks available at http://openstack.prov12n.com/monitoring- openstack-nagios-3/ 18
  19. 19. !  Open source monitoring tool used at several large service provider clouds !  Strong API and discovery modes !  Templates can be applied to host groups for monitoring Zabbix Console 19
  20. 20. Zabbix Templates !  Templates created for each type of server –  Compute nodes, controllers, Swift object servers, etc. !  Each template checks processes running and configuration management running –  Should issue commands against processes not rely on process table to catch hung processes !  All nodes also get default OS template !  Alerting set for pagerduty 20
  21. 21. Log Management !  More than just for error viewing !  Primary source of OpenStack data !  Useful for –  Finding OpenStack bugs –  Understanding event timings (spin new VM) –  Visualizing cluster level statistics (VMs running) –  Creating dashboards !  Can be challenging to store, query and interpret data –  Clusters can generate GBs per day –  Use dedicated tools and data stores –  May be required for legal / audit reasons 21
  22. 22. !  Commercial log management solution !  Visualization, ad hoc queries, post processing and add- ons !  Easy to setup dashboards !  Supported with relatively easy installer Splunk 22
  23. 23. !  Open source alternative to Splunk !  Requires more complicated setup to parse logs correctly !  Provides ad hoc queries as well as dashboards !  Active community Logstash, Kibana and ElasticSearch 23
  24. 24. Interesting Uses for Log Data !  VMs –  CPUs/Instances by hypervisor (scheduler efficiency) –  Total vCPUs/CPUs in cluster available versus used –  Spawn success and failures –  Spawn time –  Top Users of VMs/vCPUs !  Authentication –  Tokens generated versus invalidated –  Failed authentications !  Errors –  All error messages / stack traces create alert !  Logs that have stopped (zombie processes) 24
  25. 25. “Canary” Scripts !  Highest level check for cloud infrastructure: “Can we spin a new VM ?” –  Custom written script that starts VM, attaches block storage, assign IP address, pings outside world then terminates –  Logs to all actions with timings into log management solution !  Run every 5 to 15 minutes !  Also can be run interactively !  This should be written for your own site 25
  26. 26. Specialized Tools !  Many sites will want to be able to create their own custom images –  CI/CD “golden images” !  Several commercial and open source alternatives –  CohesiveFT Server3 –  Elasticbox (https://www.elasticbox.com/) –  Packer (http://www.packer.io/) !  All provide ability to create images with specified software pre-installed via command line 26
  27. 27. Manageability Improvements in Icehouse !  Nova live upgrade !  Swift discovery API 27
  28. 28. Rolling (“live”) Upgrades !  Ability to upgrade a running cluster to new release !  Upgrade controller(s) first then individual compute nodes !  Requires several pre-conditions –  Neutron upgraded first –  Nova-conductor being used to isolate DB schemas –  Set icehouse compatibility mode /etc/nova/nova.conf
 # Set a version cap for messages sent to compute services. If
 # you plan to do a live upgrade from havana to icehouse, you
 # should set this option to "icehouse-compat" before beginning
 # the live upgrade procedure. (string value)
 compute=icehouse-compat
 28
  29. 29. !  API calls to /info will return information about the cluster !  Users now able to take advantage of the unique features available in each cluster !  Turned on by default but can be disabled # swift capabilities! Core: swift! Options:! account_listing_limit: 10000! container_listing_limit: 10000! max_account_name_length: 256! max_container_name_length: 256! max_file_size: 5368709122! max_meta_count: 90! max_meta_name_length: 128! max_meta_value_length: 256! max_object_name_length: 1024! strict_cors_mode: True! version: 1.13.1! Additional middleware: keystoneauth! Additional middleware: staticweb! Additional middleware: tempurl! Options:! methods: ['GET', 'HEAD', 'PUT']! 29 Swift Discoverability

×