Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Neutron upgrades

96 views

Published on

This presentation covers the governance requirements to use upgrade tags and how neutron is in compliance with those tags

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Neutron upgrades

  1. 1. Neutron Upgrade Victor Morales
  2. 2. Agenda • Upgrade - https://governance.openstack.org/reference/tags/assert_supports-upgrade.html • Rolling upgrade - https://governance.openstack.org/reference/tags/assert_supports-rolling- upgrade.html • Zero downtime upgrade - https://governance.openstack.org/reference/tags/assert_supports- zero-downtime-upgrade.html • Plan - https://review.openstack.org/#/c/386685/ • Meetings - https://wiki.openstack.org/wiki/Meetings/Neutron-Upgrades-Subteam • Zero impact upgrade - https://governance.openstack.org/reference/tags/assert_supports-zero- impact-upgrade.html
  3. 3. Upgrade – Part 1 That means it supports a controlled and planned upgrade process from release to release. Requirements: • Configuration from release N-1 is supported in release N. • oslo-config-generator https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L401-L442 • Database schema updates are stable and ordered such that moving a database (with actual data in it) from release N-1 to N is possible without data loss. • Independent Sub-Project Tables (networking-odl, networking-onos, networking-calico, etc) • Offline/Online Migration http://docs.openstack.org/developer/neutron/devref/alembic_migrations.html • Expand - Rules are strictly additive and can be applied while neutron-server is running (creating a new table, adding a new table column, adding a new index, etc.) • Contract - Rules that are not safe to apply while neutron-server is running (column or table removal, moving data from one part of the database into another, introducing or modifying constraints, etc.) • https://review.openstack.org/#/c/400239/ Forbid contract migration scripts for Ocata
  4. 4. Upgrade – Part 2 • A procedure for general upgrades of the project is defined and does not change substantially from cycle to cycle. http://docs.openstack.org/developer/neutron/devref/upgrade.html • Upgrade Strategy • All services are shut down, code upgraded, then all services are started again. • Services are upgraded gradually, based on operator service windows(Rolling Upgrade) • Provides an upgrade impact section on the release notes page that highlights anything that must be done by operators for each cycle outside the normal upgrade procedures. https://docs.openstack.org/releasenotes/neutron/unreleased.html#id41
  5. 5. Upgrade – Part 3 • Full stack integration testing is performed on every proposed commit to validate that cold upgrades from the previous stable release are not broken. https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml#L7709-L7738 • Grenade https://docs.openstack.org/developer/grenade/ 1. Get 2 Devstacks (base & target) 2. Install base devstack 3. Perform some sanity checking (currently tempest smoke) to ensure this is right 4. Allow projects to create resources that should survive upgrade - see projects/*/resources.sh 5. Shut down all services 6. Verify resources are still working during shutdown 7. Upgrade and restart all services 8. Verify resources are still working after upgrade 9. Perform some sanity checking (currently tempest smoke) to ensure everything seems good.
  6. 6. Rolling Upgrade That means that the code is installed and deployed on many distributed systems and can be upgraded avoiding a significant downtime. Requirements: • Supports Upgrade. • The project has a defined plan that allows operators to roll out new code to subsets of services, eliminating the need to restart all services on new code simultaneously. In other words, “restarting all API services together” is a reasonable restriction. http://docs.openstack.org/developer/neutron/devref/upgrade.html 1. Server Upgrade 2. Agents Upgrade 3. Networking backends. https://www.youtube.com/watch?v=UQLBw1_VGcU
  7. 7. Zero Downtime Upgrade – Plan • Isolate layer that has access to database (Oslo-Versioned Objects). https://docs.google.com/spreadsheets/d/1FeeQlQITsZSj_wpOXiLbS36dirb_arX0XEWBdFVPMB8/edit#gid=1434170112 • Forbid new contract scripts in Ocata. The idea is to reach a point where unsafe database operations (dropping tables and/or columns) become safe to execute it. https://review.openstack.org/#/c/400239/ • Online data migration[Draft] https://review.openstack.org/#/c/432494/ • Neutron PTG Upgrades session https://etherpad.openstack.org/p/neutron-ptg-pike-upgrades
  8. 8. Oslo-Versioned Objects (Approach) https://www.youtube.com/watch?v=ivcNI7EHyAY
  9. 9. Olso-versioned Objects (NeutronDbObject) @six.add_metaclass(abc.ABCMeta) class NeutronObject(obj_base.VersionedObject, obj_base.VersionedObjectDictCompat, obj_base.ComparableVersionedObject): @six.add_metaclass(DeclarativeObject) class NeutronDbObject(NeutronObject):
  10. 10. Oslo-Versioned Objects(Example) @obj_base.VersionedObjectRegistry.register class QosPolicy(rbac_db.NeutronRbacObject): # Version 1.0: Initial version # Version 1.1: QosDscpMarkingRule introduced # Version 1.2: Added QosMinimumBandwidthRule # Version 1.3: Added standard attributes (created_at, revision, etc) # Version 1.4: Changed tenant_id to project_id VERSION = '1.4’ … def obj_make_compatible(self, primitive, target_version): … _target_version = versionutils.convert_version_to_tuple(target_version) names = [] if _target_version >= (1, 0): names.append(rule_obj_impl.QosBandwidthLimitRule.obj_name())
  11. 11. Zero Downtime Upgrade That means that the code is installed and deployed on many distributed systems and can be upgraded without downtime (control plane entirely). Requirements: • Supports Rolling Upgrades. • Requires services to completely eliminate API downtime of the control plane during the upgrade. In other words, “restarting all API services together” is not reasonable. • While all requests to the control plane must be eventually processed, performance degradation during the upgrade is acceptable.
  12. 12. Neutron – Zero Impact Upgrade That means that the code is installed and deployed on many distributed systems and can be seamlessly upgraded without downtime. Requirements: • Supports Zero Downtime Upgrades. • Requires services to completely eliminate any perceivable performance penalty during the upgrade process.
  13. 13. Q & A

×