One of the main shortcomings of CloudStack compared to VMware is the lack of built-in support for disaster recovery. In this workshop, Venko shows how you can build a cloud using Apache CloudStack and StorPool storage that features disaster recovery capabilities and off-site backups. He shows a reference architecture of the solution, followed by a live demo on how to set backup policies, backup virtual machines and recover the VMs in a site-down event. The workshop allows an open discussion about the practical aspects of implementing a DR service, the difficulties of building such solutions and how to address them. ----------------------------------------- The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.

1. Backup and Disaster
Recovery with
CloudStack and StorPool
Venko Moyankov
Solutions Architect at StorPool Storage
Workshop

2. Agenda
● How Disaster Recovery works with ACS and StorPool
● Demo
○ Demo setup
○ Failover
○ Failback
● Open Discussion

3. DR setup

4. CloudStack View - Multiple Zones?

5. CloudStack View - Single Zone

6. Key StorPool Features for DR
• Asynchronous replication between the primary storage clusters
(snapshots)
• Bypass secondary storage
• Multi-cluster configuration (single namespace, single API endpoint)
• Periodic snapshots (VolumeCare)

7. The Demo Setup
● 2 x KVM hosts
● 2 x StorPool clusters
● in a multi-cluster
● ACS management server
● Secondary storage
● Layer-2 network

8. Normal Operations
• Pod-B is disabled (optional)
• All user VMs, system VMs and VRs are running on KVM-1
• Bypass secondary storage enabled (sp.bypass.secondary.storage)
• snapshots are stored on the primary storage only.
• Templates are "cached" on the primary storage.
• VolumeCare (snapshot scheduler):
• Creates a snapshot of every volume, every 5 minutes
• Sends the snapshots to the remote storage cluster
• Retention policy: keep 1 local and 1 remote snapshot
• ACS mgmt server uses the API of StorPool-1

9. Failover - Prep steps
1. Fence KVM-1 host (make sure no VMs are running there)
2. Switch ACS use API of StorPool-2 (on site B)
3. Mark all VMs at site A as stopped:
cmk stop virtualmachine id=${vmid} forced=true
4. Delete template refs:
mysql> delete from template_spool_ref
5. Enable Pod-B and disable Pod-A
6. Destroy the system VMs and VRs (on Pod-A)
cmk stop systemvm id=${vmid} forced=true
cmk destroy systemvm id=${vmid}

10. The Demo Setup
● 2 x KVM hosts
● 2 x StorPool clusters
● in a multi-cluster
● ACS management server
● Secondary storage
● Layer-2 network

11. Failover - Restart VMs
./start-vm-on-dr.py [-v] [-a] vm [vm ...]
1. get the volumes used by the VM
2. find the latest snapshot(s)
3. create a new volume(s) from the latest snapshot
4. update ACS volume path to point to the new volume
5. start the VM

12. Failback
1. delete any old/stale volumes in StorPool-1
2. enable Pod-A
3. live migrate VMs to Pod-A