The document previews ShapeBlue's CloudStack Backup and Recovery Framework, which aims to provide a vendor-agnostic API and UI in CloudStack for third-party backup and recovery solutions. The framework abstracts vendor specifics through plugins so solutions can deliver features like scheduled, ad-hoc, and policy-based backups as well as VM and volume restoration. An example plugin for Veeam Backup & Replication is provided. The framework and initial plugins are targeted for an open source release in Q4.

1. The Cloud Specialists
CloudStack Backup and
Recovery Framework
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus
paul.angus@shapeblue.com
A P r e vi e w o f

2. The Cloud Specialists
ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public
& private clouds. They are the leading
independent global CloudStack services
company”
Ab o u t S h a p e B l u e

3. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Paul Angus – VP Technology
• Global authority on CloudStack & cloud infrastructure design.
• 20+ years C-Level experience.
• Apache CloudStack project committer & PMC member
• Specialising in deployment of CloudStack
and surrounding infrastructure especially the user story
• USP, Georgian Ministry of Justice, Orange, TomTom,
PaddyPower, Ascenty, BSkyB, SAP, British Telecom
Ab ou t m e

4. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

5. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

6. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

7. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• What is a snapshot? (and what’s snapshots got to do with this
anyway?)
• Why is this a ‘framework’? (and why is it in quotes?)
• What does this ‘framework’ do then? (enough with the quotes)
• What will the framework work with?
• An Example would be nice ..
Cl ou d S t ac k B ac k u p an d Rec over y F r am ewor k

8. The Cloud Specialists
ShapeBlue.com @ShapeBlue
What’s is a Snapshot?

9. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
VM Snapshot
• A hypervisor-driven point-in-time image of a virtual machine’s
disks. The exact mechanism of this is dependant on the
hypervisor.
Volume Snapshot
• A point-in-time image of a specific volume – The process usually
involves taking a VM snapshot and then copying the required
volume to secondary storage and the deleting the VM snapshot
W h at ’ s a S n ap s h ot ?

10. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Features of volume snapshots include:
• Alternate storage location of image (from primary storage)
• Ability to schedule the volume snapshots
• Ability to set a retention policy of n copies.
• Ability to create new VMs from the snapshot image (pseudo
restoration)
• Ability to export volume snapshots from the system
O K , b u t wh at ’ s s n ap s h ot s g ot t o d o wi t h t h i s ?

11. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
So snapshots are used by many (everyone) as a form of
backup
but…
O K , b u t wh at ’ s s n ap s h ot s g ot t o d o wi t h t h i s ?

12. The Cloud Specialists
ShapeBlue.com @ShapeBlue
What’s wrong with
snapshots as backups?

13. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h a t ’ s w r o n g w i t h s n a p s h o t s ?
• Not consistent across volumes
• A VM with separate disks for say; OS, logs and
data, will have each disk processed at different
times
• Only crash consistent within a volume
and they aren’t application aware
• A database will not be quiesced before snapshot is
taken
• Transaction logs won’t be processed
• VMware tools or XenTools may help with some
basics

14. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h a t ’ s w r o n g w i t h s n a p s h o t s ?
• They can be very slow.
• Data has to be transferred from primary storage to
secondary storage
• In the case of vSphere, this data must go via the
SSVM to be compressed into a OVA while being
moved to secondary storage.
• CloudStack locks out other actions while a
snapshot is taking place. Workarounds have their
own issues.

15. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h a t ’ s w r o n g w i t h s n a p s h o t s ?
• Restoration of a VM requires a number of steps
(and did I mention it’s slow)
• User must make a template from snapshot
(VM image copied via SSVM from one part of secondary
storage to another – 2x network transfers)
• User then creates VM from template
(VM image copied from secondary to primary to create
template on primary storage, then primary storage based
template copied to make actual VM. 2x network assuming
network based primary storage and 1x on-disk copy)
• To restore a VM ‘exactly as it was’ requires considerable
operator intervention

16. The Cloud Specialists
ShapeBlue.com @ShapeBlue
Why is this a ‘framework’

17. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
The CloudStack Backup & Recovery Framework
• This is a framework because, by itself it doesn’t ‘do’ anything.
• Plugins are required to drive physical actions
• A dummy plugin has been created for testing the B&R framework API
Oh. This is going to be a short/rubbish presentation then.
W h y i s t h i s a ‘ f r a m e w o r k ’ ?

18. The Cloud Specialists
ShapeBlue.com @ShapeBlue
What does this ‘framework’
do then?

19. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
The CloudStack Backup and Recovery Framework
Creates a vendor agnostic API and UI in CloudStack for end users to leverage 3rd
party backup and recovery solutions.
The B&R Framework abstracts the specifics of B&R solutions, such that through the
use of a plugin, a 3rd party B&R solution can deliver ….
W h at d o e s t h i s ‘ f r am ewor k ’ d o t h en ?

20. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
For the Cloud Operator:
The B&R framework enables cloud operators to leverage their choice of enterprise
grade backup and recovery technology. Giving them:
• Greater control over the backend backup and recovery processes
• Hardware and software accelerated performance
• Reducing/elimination of operator overhead
• More features to offer users
• Additional chargeable services
W h at d o e s t h i s ‘ f r am ewor k ’ d o t h en ?

21. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
For the Cloud Consumer:
• Improved performance over snapshots
• A feature set that consumers expect from backup software
• Proper restoration options
• Consistent API and user experience independent of B&R vendor
• Piece of mind of a recognised industry named B&R solution
W h at d o e s t h i s ‘ f r am ewor k ’ d o t h en ?

22. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Specifically:
• Support for:
• Operator created ‘policy based’ backups (RPOs)
• Scheduled backups (similar to snapshot scheduling)
• Ad hoc backups
• Backups as first class citizens
• ‘In-place’ restoration of entire VM including data disks
• Restoration of VM backup after original VM is destroyed
• Restoration of individual VM volumes (can then be attached to another VM)
• Full integration into the CloudStack usage service, recording both protected and
on-disk data metrics
W h at d o e s t h i s ‘ f r am ewor k ’ d o t h en ?

23. The Cloud Specialists
ShapeBlue.com @ShapeBlue
What will the framework
work with?

24. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Dummy Plugin
• Built for compile time testing of the APIs.
Veeam Backup & Replication 9.5 (for VMware)
• Plugin written against Veeam Backup & Replication 9.5 and Apache CloudStack
4.12
• Tested against VMware 6.5
• Requires Veeam Backup & Replication Console, Veeam Backup Enterprise
Manager, OpenSSH and PowerShell on the Veeam Server
W h at w i l l t h e f r am ew or k w or k w i t h ?

25. The Cloud Specialists
ShapeBlue.com @ShapeBlue
Example(s) ?

26. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Backup APIs so far*
• createVMBackup(A)
• deleteBackupPolicydeleteVM
• BackupimportBackupPolicy (A)
• listBackupPolicies
• listBackupProviders
• listVMBackupRestorePoints
• listVMBackupsrestoreVMBackup (A)
• restoreVolumeFromBackup (A)
• startVMBackup (A)
*Still subject to change during development
Th e B a c k u p an d Rec over y AP I

27. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Veeam B & R P l u g i n
‘Policy Based’ Backups
• listBackupPolices external=true
Returns ‘template’ jobs.
• importBackupPolicy Creates a policy (with
a name and description) in CloudStack
and maps it to a Veeam template job.
ie.
GOLD – 6hr RPO
SILVER – 24hr RPO
BRONZE – 7 day RPO

28. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Veeam B & R P l u g i n
‘Policy Based’ Backups
• Add VM to the policy using
createVMBackup*
Creates a Veeam backup job for the VM
with the properties of the template job
linked.

29. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Veeam B & R P l u g i n
‘Ad-hoc’ & Scheduled Backups
• CloudStack will send command to carry
out an ad-hoc run of the policy based job
or create one as required.
• Ad-hoc and scheduled jobs will be driven
by CloudStack
• One VM to one Veeam job mapping allows
for simpler accounting and unified
incremental backups

30. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Veeam B & R P l u g i n
Listing Backups
listVMBackups
• Lists backup jobs for a given VM. Under
Veeam approach, this is a one-to-one
relationship
listVMBackupRestorePoints*
• Veeam thinks in terms of restore points as
being the image of a VM at any point in
time. This API lists the restore points
available for a given VM backup job (VM)
*This is quite Veeam centric and potentially confusing, as the
overall feature is rounded out, the abstraction naming will be
reviewed.

31. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
VM and Volume Restoration
• Veeam ‘in-place’ restoration is more in-place-of than in-the-original-place
• The Veeam plugin takes care of the fact the replacement VM is in an alternate
location to the ‘original’
• Current implementation automatically attaches a restored volume to a VM (option
to restore to secondary storage will be added)
Veeam B & R P l u g i n

32. C l i c k t o e d i t
The Cloud Specialists
ShapeBlue.com @ShapeBlue
Veeam B & R P l u g i n
‘Ad-hoc’ & Scheduled Backups
• CloudStack will send command to carry
out an ad-hoc run of the policy based job
or create one as required.
• Ad-hoc and scheduled jobs will be driven
by CloudStack
• One VM to one Veeam job mapping allows
for simpler accounting and unified
incremental backups

33. The Cloud Specialists
ShapeBlue.com @ShapeBlue
That’s all cool. Where is it?

34. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Framework & Veeam Plugin
• Expect to see a PR in Q4
• APIs largely done
• Work left:
• Scheduling of backups
• Ad-hoc backups
• Changing of underlying policy
• Deletion of backups/restore points
UI
• Expect to see a PR late Q4
Ti m e l i n e

35. The Cloud Specialists
ShapeBlue.com @ShapeBlue
Questions?

36. The Cloud Specialists
CloudStack Backup and
Recovery Framework
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus
paul.angus@shapeblue.com
A P r e vi e w o f