This document outlines best practices for upgrading CloudStack-based cloud infrastructure. It recommends proper planning to prevent issues, upgrading components in stages, extensive testing of upgrades in labs before production, and having rollback strategies. It also provides an overview of upgrading CloudStack management servers and databases using a parallel build approach to simplify rollbacks.

1. The Cloud Specialists
CloudStack Upgrades Best Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo

2. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
A b o u t M e
• Cloud Architect @ ShapeBlue
• Background:
• Cloud and virtualization architect with 19
years experience from the service provider,
financial and manufacturing industries.
• Specialize in:
• Cloud infrastructure architecture and
engineering.
• Virtualization - VMware vSphere, Citrix
XenServer, KVM.
• Automation and configuration management.
• Involved with CloudStack since version 2.1.

3. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public & private
clouds. They are the leading global CloudStack
services company.”
A b o u t S h a p e B l u e

4. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

5. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

6. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

7. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Upgrading CloudStack

8. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Upgrades - general advice
• 5p’s – proper planning prevents poor performance.
• Divide and conquer – don’t plan too many changes at the same time.
• Do as much work as possible in advance:
• Prepare all builds.
• Upgrade hypervisors if required.
• Ensure system VM templates are uploaded.
• RTFM!
• Ensure infrastructure is healthy.
• Keep your users informed – and disable their access until upgraded systems healthy.

9. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Planning - customer impact
• CloudStack GUI and API will be offline during the
upgrade.
• All VMs continue to run.
• Virtual Routers
• Will continue to function post upgrade, but can not be changed…
• To upgrade these they need to be restarted
• System VMs: are offline with the GUI, also need restarted.

10. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Lab Upgrade Testing
• Production equivalent lab upgrade test – try to match:
• Hardware
• OS and S/W versions
• Storage – at least protocol
• Prepare:
• VMs with production type workloads
• Guest VMs with similar networking models utilised and disks
attached.
• Post upgrade tests:
• Functional, regression and user acceptance tests.
• User actions
• VM, network and storage lifecycle actions
• Don’t forget your integrated systems!
• Rinse and repeat….

11. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Production DB upgrade test
• Warning: carry out in a network isolated location!
• Single upgraded CS server with MySQL installed.
• Import production DB, then:
• Cloudstack-setup-databases
• Cloudstack-setup-management
• Troubleshoot and fix MySQL upgrade problems,
document for production run.

12. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Strategy – parallel builds
• Standard in-place upgrade works well, but:
• Doesn’t allow for technology refresh.
• Rollbacks are trickier and involves snapshots, backup
restores as well as loss of logs.
• Overall higher risk.
• Building a parallel upgrade infrastructure has
some benefits:
• Old CS management servers can simply be disabled.
• Upgrade done on brand new VMs / servers built in
advance.
• Rollbacks are much simpler – switch off new servers,
switch old ones back on.
• The cost of this is minimal, but it does mean a few extra
steps.

13. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Strategy – parallel build overview
• New management server builds
• Just RPM install
• Do not seed system VM template
• Do not carry out cloudstack-setup-databases
• Do not run cloudstack-setup-management
• Don’t start services
• New MySQL servers
• Ensure all permissions used in your existing MySQL
servers is replicated on the new ones.
• Create new empty databases
• Configure master-slave and ensure the empty
databases are replicated.

14. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Agent upgrades
• VMware:
• Minor – but watch out for password updates.
• XenServer:
• Minor - watch out for changes to vhd-utils and scripts.
• KVM:
• Only hypervisor which requires agent update.
• If large number of nodes – automate!
• Test automation well in advance.
• Remember to automate backup, upgrade and rollback!
• Use e.g. Ansible, Puppet or Chef.

15. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Sacrificial infrastructure
• Optional - plan and prepare sacrificial infrastructure:
• For any hypervisor where agent upgrade is required , i.e. KVM. It is
easier to roll back a subset of KVM hosts rather than your whole
infrastructure.
• Create test cases which won’t impact end users:
• Accounts
• VMs
• Networks – with every service offering you may use.
• These can be used for the initial testing before decision is made to
especially do VR restarts for all end users.

16. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
U p g r a d e P r o c e s s u s i n g p a r a l l e l b u i l d sStep0
RTFM!
Build parallel
upgrade
infrastructure
Upload System
VM templates
Step1
Confirm system
VM templates
in place
Step2
Stop and
disable existing
CloudStack
management
services
Step3
Back up all
your existing
databases.
Disable MySQL
on the original
DB servers
Step4
Copy and
import the
databases to
the new
MySQL master
server.
Step5
Update
the
"host"
global
setting
Step6
Upgrade
hypervisor
s
Step7
Configure
and start
first
managem
ent server
Step8
Monitor
startup
Ensure all
HV check
in – esp.
KVM!
Step9
Restart
CPVM
then
SSVM.
Restart
VRs
Step10
Configure
and start
additional
mgmt
servers

17. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Rollback
• Only roll back when all other options exhausted!
• All changes since upgrade will be lost – take timing
and customer access into account.
Step1
Disable new CS
mgmt servers
Disable new DB
servers
Step2On hypervisor
infrastructure
remove all VMs
created since
upgrade
Step3
Enable and start
the original
MySQL servers
Enable and start
the original
CloudStack mgmt.
servers
Step4
Restart system
VMs and VRs if
they don't auto
start

18. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Questions?

19. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Some references
• This talk is based on the blog post in http://www.shapeblue.com/cloudstack-
upgrades-best-practices/ - this has more technical detail.
• CloudStack upgrades – reference the release notes – e.g.
http://docs.cloudstack.apache.org/projects/cloudstack-release-
notes/en/4.9.2.0/upgrade/upgrade-4.7.html

20. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
M o r e i n f o r m a t i o n
• Slide deck: http://www.slideshare.net/shapeblue
• Blog: http://shapeblue.com/blog
http://dsonstebo.wordpress.com
• Email: dag.sonstebo@shapeblue.com
• Twitter: @dagsonstebo
• Web: http://shapeblue.com

21. The Cloud Specialists
CloudStack Upgrades Best
Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo