Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OpenStack Summit Vancouver: Lessons learned on upgrades

846 views

Published on

Deploying OpenStack in production at any scale, upgrade support is one of the requirements to have a successful deployment. Without upgrade management, adeployment will have bugs and security issues from day 1. Also in longer term, it will miss the latest features that OpenStack offers.

  • Be the first to comment

  • Be the first to like this

OpenStack Summit Vancouver: Lessons learned on upgrades

  1. 1. The importance of HA and automation tools Frédéric Lepied Engineering Manager flepied@redhat.com Lessons Learned On Upgrades Senior Software Engineer emilien@redhat.com Emilien Macchi
  2. 2. Red Hat Cloud Innovation Practice Engineering Frédéric Lepied: RCIP Engineering manager Emilien Macchi: installer team / Puppet PTL
  3. 3. Disclaimer The examples are taken from former eNovance products and not Red Hat ones.
  4. 4. OpenStack is a wonderful place, but upgrades are not easy.
  5. 5. What is a successful upgrade? • No need of new hardware • The less interruption possible • Minor & Major upgrade support • Efficient, fast, reproducible process
  6. 6. Roadmap • Redundant architecture • Enough free capacity • Image based deployment • Automation tooling
  7. 7. Redundant Architecture
  8. 8. Enough free capacity • Have enough compute resources to migrate instances • Have some spare in case of failure
  9. 9. Image based workflow (recommended) • Build your images once • Install using your images • Upgrade using your images
  10. 10. Build and archive your images • Build your image in a CI • Use packaging tools (yum, apt, …) • Compression & archive • Stamp with versioning • Use Cloud Storage (Swift, Ceph) Image based deployment
  11. 11. Limit the number of images • More images = more pain • Single image with: • all packages installed • all services disabled at boot Image based deployment
  12. 12. Prohibit packaging tools • Keep systems: • consistent • reproducible • auditable • Speed-up configuration management • Allow to re-enable the tools Image based deployment
  13. 13. Upgrade your system with a tool • APT / YUM: • too slow at scale (~20 min / node) • need to manage your repositories • Using eDeploy: • very fast at scale (~20 s / node) • allow rollbacks Image based deployment
  14. 14. Automation tooling • Control system upgrade • Configuration management • Orchestration • Automate the workflow
  15. 15. Control system upgrade We need: • one command to upgrade one system • no service restarted or reloaded • possibility to rollback What we use: • eDeploy : tool to upgrade images with rsync Automation tooling
  16. 16. Configuration management • Puppet, Chef, Ansible, whatever you like • “The best tool is the one you already use.” • But: • … you need to update your config • … do not manage packages Automation tooling
  17. 17. Orchestrator • Puppet and Chef are good for configuration • But you need to orchestrate multiple systems: • restart services in the right order • upgrade the system at the right time Automation tooling
  18. 18. Upgrade workflow Automation tooling • Pre-upgrade actions • Resources evacuation • Stop OpenStack services • Stop Infra / system services • Upgrade packages • Start Infra / system services • Start OpenStack services • Post-upgrade actions
  19. 19. Example: upgrade a compute node • evacuate virtual machines • disable nova compute service • system upgrade • update config • service libvirtd restart • service openstack-nova-compute restart • enable nova-compute service • test the service Automation tooling
  20. 20. Ansible snippet example (hypervisor) - name: evacuate compute node script: evacuate-compute.sh tags: 2 - name: restart nova-compute service: name={{ item }} state=restarted with_items: - "{{ libvirt }}" - "{{ nova_compute }}" tags: 8 - name: enable nova-compute service script: enable-compute.sh tags: 9 Automation tooling
  21. 21. Automate the workflow Automation tooling • Upgrades are repetitive • Prepare an upgrade without effort • Prepare Ansible Playbooks with snippets • Compose Playbooks by computing: • what is upgraded in the image • which service is running on a node
  22. 22. Ansible best practices • Use tags in snippets to define ordering • Run HA nodes in serial • Run compute nodes in parallel • Use a script for hypervisor evacuation • Allow to continue to roll playbooks after a failure • a snippet for each service to upgrade Automation tooling
  23. 23. Generate Ansible playbooks per role Automation tooling
  24. 24. • Your OpenStack needs HA • Make sure you have free capacity • Image based upgrade is a good option • Orchestration and Configuration Management are key Conclusion
  25. 25. Thank you! http://tinyurl.com/ansible-snippets @EmilienMacchi @flepied

×