CloudStack upgrade best practices - Dag Sonstebo - Shapeblue

1. The Cloud Specialists
CloudStack Upgrades Best Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo

2. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
A b o u t M e
• Cloud Architect @ ShapeBlue
• Background:
• Cloud and virtualization architect with 19
years experience from the service provider,
financial and manufacturing industries.
• Specialize in:
• Cloud infrastructure architecture and
engineering.
• Virtualization - VMware vSphere, Citrix
XenServer, KVM.
• Automation and configuration management.

3. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public & private
clouds. They are the leading global CloudStack
services company.”
A b o u t S h a p e B l u e

4. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

5. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

6. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

7. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Upgrading CloudStack

8. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Upgrades - general advice
• 5p’s – proper planning prevents poor performance.
• Divide and conquer – don’t plan too many changes at the same time.
• Do as much work as possible in advance:
• Prepare all builds.
• Upgrade hypervisors if required.
• Ensure system VM templates are uploaded.
• RTFM!
• Ensure infrastructure is healthy.
• Keep your users informed – and disable their access until upgraded systems healthy.

9. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Planning - customer impact
• CloudStack GUI and API will be offline during the
upgrade.
• All VMs continue to run.
• Virtual Routers
• Will continue to function post upgrade, but can not be changed…
• To upgrade these they need to be restarted
• System VMs: are offline with the GUI, also need restarted.

10. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Lab Upgrade Testing
• Production equivalent lab upgrade test – try to match:
• Hardware
• OS and S/W versions
• Storage – at least protocol
• Prepare:
• VMs with production type workloads
• Guest VMs with similar networking models utilised and disks
attached.
• Post upgrade tests:
• Functional, regression and user acceptance tests.
• User actions
• VM, network and storage lifecycle actions
• Don’t forget your integrated systems!
• Rinse and repeat….

11. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Production DB upgrade test
• Warning: carry out in a network isolated location!
• Single upgraded CS server with MySQL installed.
• Import production DB, then:
• Cloudstack-setup-databases
• Cloudstack-setup-management
• Troubleshoot and fix MySQL upgrade problems,
document for production run.

12. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Strategy – parallel builds
• Standard in-place upgrade works well, but:
• Doesn’t allow for technology refresh.
• Rollbacks are trickier and involves snapshots, backup
restores as well as loss of logs.
• Overall higher risk.
• Building a parallel upgrade infrastructure has
some benefits:
• Old CS management servers can simply be disabled.
• Upgrade done on brand new VMs / servers built in
advance.
• Rollbacks are much simpler – switch off new servers,
switch old ones back on.
• The cost of this is minimal, but it does mean a few extra
steps.

13. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Strategy – parallel builds
• Management server builds
• Do not seed system VM template
• Do not carry out cloudstack-setup-databases
• Do not run cloudstack-setup-management
• MySQL
• Ensure all permissions used in your existing MySQL
servers is replicated on the new ones.
• Create new empty databases
• Configure master-slave and ensure the empty
databases are replicated.

14. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Agent upgrades
• Vmware:
• Minor – but watch out for password updates.
• XenServer:
• Minor - watch out for changes to vhd-utils and scripts.
• KVM:
• Only hypervisor which requires agent update.
• If large number of nodes – automate!
• Test automation well in advance.
• Remember to automate backup, upgrade and rollback!
• Use e.g. Ansible, Puppet or Chef.

15. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Sacrificial infrastructure
• Optional - plan and prepare sacrificial infrastructure:
• For any hypervisor where agent upgrade is required , i.e. KVM. It is
easier to roll back a subset of KVM hosts rather than your whole
infrastructure.
• Create test cases which won’t impact end users:
• Accounts
• VMs
• Networks – with every service offering you may use.
• These can be used for the initial testing before decision is made to
especially do VR restarts for all end users.

16. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
U p g r a d e P r o c e s s u s i n g p a r a l l e l b u i l d s
Step 0
RTFM!
Build parallel
upgrade
infrastructure
Upload System
VM templates
Step 1
Confirm system
VM templates
in place
Step 2
Stop and
disable existing
CloudStack
management
services
Step 3
Back up all
your existing
databases.
Disable MySQL
on the original
DB servers
Step 4
Copy and
import the
databases to
the new
MySQL master
server.
Step 5
Update
the
"host"
global
setting
Step 6
Upgrade
hypervisor
s
Step 7 Configure
and start
first
managem
ent server
Step 8
Monitor
startup
Ensure all
HV check
in – esp.
KVM!
Step 9
Restart
CPVM
then
SSVM.
Restart
VRs
Step 10
Configure
and start
additional
mgmt
servers

17. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Rollback
• Only roll back when all other options exhausted!
• All changes since upgrade will be lost – take timing
and customer access into account.
Step 1
Disable new CS
mgmt servers
Disable new DB
servers
Step 2
On hypervisor
infrastructure
remove all VMs
created since
upgrade
Step 3
Enable and start
the original
MySQL servers
Enable and start
the original
CloudStack mgmt.
servers
Step 4
Restart system
VMs and VRs if
they don't auto
start

18. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Questions?

19. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
B y t h e w a y … .
• Next CloudStack European User Group:
Wednesday, April 12 @ Interoute Prague
Interoute Czech s.r.o. Siemensova Street 2717/4, Prague 5, 155
00, Czech Republic
https://www.eventbrite.co.uk/e/cloudstack-european-user-
group-tickets-32354983627?aff=es2

20. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
M o r e i n f o r m a t i o n
• Slide deck: http://www.slideshare.net/shapeblue
• Blog: http://shapeblue.com/blog
http://dsonstebo.wordpress.com
• Email: dag.sonstebo@shapeblue.com
• Twitter: @dagsonstebo
• Web: http://shapeblue.com

21. The Cloud Specialists
CloudStack Upgrades Best
Practices
Dag Sonstebo
Cloud Architect
dag.sonstebo@shapeblue.com
Twitter: @dagsonstebo