BackROLL: An open-source KVM Backup solution fully
integrated inside Apache Cloudstack
From November 14 to 16, 2022
Table of contents
§ Bringing an easy-to-use KVM backup solution to the community
What's our vision of a production grade backup solution ?
§ Key features
All the features you need to build a backup strategy
§ Integration within your workflow
Our design of the BackRoll API was built around integration with a wide range of workflows
§ Integration with the CloudStack ecosystem (plugin)
Integrate our tool with CloudStack to make backup processes transparent to users and administrators
§ Roadmap
What's next ? Planned new features and our vision of the future of BackRoll
§ Q & A
Ask us anything you want about us or BackRoll’s project J
§ Who are we
About DIMSI and your speakers
§ What is BackROLL ?
All the features you need to build a backup strategy
Your fellow speakers (Greg, Quentin, Pierre)
Grégoire LAMODIERE
Co-Founder & solution architect
Backroll evangelist
Quentin ROCCIA
Senior SRE and Devops Engineer
Backroll guru
Pierre CHARTON
Senior C# / Java developer, Pierre
provides Backroll integration within
Apache Cloudstack, main contributor of
Backroll plugin
Senior backend developer
Backroll integrator
Automation addict, Quentin is in charge of
maintaining Cloudstack infrastructure up and
running. He also helps our developers to
speed-up code integration and deployment
Senior solution architect, Grégoire helps our
customers to define Cloud-based, hybrid and
OnPremise strategies
Who are we ?
We are a small independent French cloud
provider that keeps its focus on small and
medium sized companies
We also provide for companies using
Microsoft Dynamics CRM, and bring them
our expertise on project management and
development to meet their business needs
We think that even a small team such as ours
can bring great knowledge and success to
our customers
Our human-size teams put people at the
heart of our digital services
ABOUT DIMSI
In 2022, DIMSI became a mission-driven company
Foundation
2006
Area of expertise
Company Size
4 sites :
§ Paris
§ Lorient
§ Sousse
§ Hong-Kong
Offerings
Microsoft
Power Platform
Cloud & Services
Modern Apps
Lorient
Paris
Sousse
Hong-Kong
What’s Backroll ?
What prompted us to create BackROLL
• No similar open-source community project
• No turnkey solution available
• No KVM backup software yet integrated into Apache CloudStack
• You may have noticed that we are KVM/CloudStack enthusiasts :)
What we wanted
• No agent to install on virtual machines nor on hosts
• Quiescent backup support
• Backup deduplication
• Incremental backups
• Fully integrated into CloudStack
Already proven safeguards
• Currently in production for all internal and customer Apache Cloudstack instances at DIMSI
• Has already been used for client requests for data restoration
• Used to build our disaster recovery plan
• No performance impact during backups
Open Source with Apache2 license
• The same licensing model as Apache Cloudstack
• Everyone can check source code and bring their ideas to improve the software
Aiming at becoming a community driven project !
Architecture
Modern web UI
The various components of BackROLL
Modern web UI
• Front-end
The web UI, offers the same functionality as the API
• Back-end
Contains API and core logic, it receives all API calls that will trigger tasks (Celery)
• Workers
Used to run the Celery tasks (VM list, backup list, host list, backup job, and so on...), can be stand-alone or clustered to
maximize task parallelization
• Flower
Flower is used to monitor tasks in real-time and get logs via web interface / web API
• Beat
Beat is the scheduler, allowing us to create scheduled backup policies and run redundant and automated tasks (like backup
cleaning every night)
• Redis
Redis is used as a database to store keys specific for every task, allowing tracking and keeping records.
Bring an easy-to-use KVM backup solution to the community
1. Modern web UI
Designed to be easy to use
Easily manage your backups
§ Easily track the progress and logs of your backup tasks
§ Manage virtual machine backups, pools or backup policies via a
graphical interface
§ Schedule backup and restore tasks
§ Built in VueJS 3
The web interface is optional, but still convenient for non-tech users
2. RESTful API
That follows best practices
Can be used in your existing workflows
§ Stateless
§ Uniform interface
§ Layered system
§ Can be used with scripts or tools like Ansible
The learning curve is minimal if you are already used to working with
other REST APIs
3. Containerized application
Reduces maintenance requirements
§ Deploy and update using a docker-compose template
§ One click deployment
§ Customize deployment template to meet your needs
Reduce maintenance hassle as everything needed is embedded in the
docker image
What does an API call look like?
Modern web UI
What does an API call look like?
Modern web UI
How does a backup work?
Modern web UI
What you can already do
A sample of what BackROLL is capable of
Instant start of backup or restore task of a VM / Pool
Everything is handled in the background
Nothing to do on the VM side, even the needed restart is handled by BackROLL
Backup / restore task scheduling
If you prefer to schedule your tasks to run at night, it is possible!
Self-management of lock for each VM, no way to go wrong
Whenever a VM is linked to a task, BackROLL locks out any further operations on the virtual machine
Retention management and automatic cleaning
Only keep data you need
Operation history and webhook notification
To keep track of what has been done
Caveats and limitations
For a better understanding
• It is not currently possible to backup a shutdown VM that is managed by cloudstack
• Restore and redeploy (to another VMs) task is currently in work-in-progress state
• Only supports Slack for webhook notifications
• Only NFS storage has been used to store the backups
• You cannot select a specific disk you'd like to backup / restore, current task state is all-or-nothing
• You cannot backup a VM that is currently using a snapshot disk file
• Integration of BackROLL within CloudStack is in work-in-progress state
Integration within your workflow
ANALYSE
Revue de l’architecture
Prérequis
Document de conception d’infrastructure
complété incluant la volumétrie de
données, le nombre d’utilisateurs, etc.
Livrable
Rapport de l'examen d'architecture
accompagné des préconisations DIMSI
délivré par
un Solution Architecte D365
CONCEPTION
Revue de la conception
Prérequis
Document de spécifications fonctionnels
et Document de Design Technique
complétés
Plan d‘ intégration & Interfaces complété
Livrable
Document d’examen de la conception :
Conclusions et recommandations.
PERSONNALISATION
Revue de la personnalisation
Prérequis
Document de conception technique
détaillées.
Accès à tous les codes sources de
l'application.
Définition d'un périmètre pour l'examen
de codes.
(L'étude portera sur un périmètre restreint
représentatif d'un processus métier clé)
Livrable
Document d'examen de la
personnalisation : Conclusion et
recommandations
DÉPLOIEMENT
Revue de performances
Prérequis
Disponibilité du document de de
déploiement.
Plan de production complétés
Environnement iso production et
opérationnel (non accessibles aux
utilisateurs)
Définitions des scénarios de tests.
Document d'exploitation.
Livrable
Document d’examen des performances et
de l'exploitation, conclusions et
recommandations.
Integration with the CloudStack ecosystem (plugin)
Cloudstack Back and restore
framework
Cloudstack Backroll Plugin
How we’ve built Backroll plugin for Cloudstack
(http://docs.cloudstack.apache.org/en/latest/adminguide/backup_and_recovery.
html)
We are developping our plugin using the Cloudstack Back and Restore
framework
How to setup the Backroll plugin
• Set 'backroll' to the "backup.framework.provider.plugin" global configuration variable
• Set these 3 variables:
• "backup.plugin.backroll.config.url"
• "backup.plugin.backroll.config.appname"
• "backup.plugin.backroll.config.password"
Code sample, authent, etc
To do: diagram to describe components
• Allows Cloudstack users to
use 3rd party backup
solutions
• Abstracts the API commands
required for common backup
and recovery operations
Concepts
• 'SLA' Backups
• Adhoc and user scheduled
backups
Key features
• Authentication
• List of existing backup policies
• Start a backup for registred virtual machines
• Restore virtual machines from backups
• Delete backups
Live demo
Modern web UI
Roadmap
Modern web UI
What's next ?
www.dimsi.fr | contact@dimsi.fr | (+33) 01 75 43 89 66
4 Avenue Laurent Cély Hall D,
92600 Asnières-sur-Seine
Thank you !

Backroll: Production Grade KVM Backup Solution Integrated in CloudStack

  • 1.
    BackROLL: An open-sourceKVM Backup solution fully integrated inside Apache Cloudstack From November 14 to 16, 2022
  • 2.
    Table of contents §Bringing an easy-to-use KVM backup solution to the community What's our vision of a production grade backup solution ? § Key features All the features you need to build a backup strategy § Integration within your workflow Our design of the BackRoll API was built around integration with a wide range of workflows § Integration with the CloudStack ecosystem (plugin) Integrate our tool with CloudStack to make backup processes transparent to users and administrators § Roadmap What's next ? Planned new features and our vision of the future of BackRoll § Q & A Ask us anything you want about us or BackRoll’s project J § Who are we About DIMSI and your speakers § What is BackROLL ? All the features you need to build a backup strategy
  • 3.
    Your fellow speakers(Greg, Quentin, Pierre) Grégoire LAMODIERE Co-Founder & solution architect Backroll evangelist Quentin ROCCIA Senior SRE and Devops Engineer Backroll guru Pierre CHARTON Senior C# / Java developer, Pierre provides Backroll integration within Apache Cloudstack, main contributor of Backroll plugin Senior backend developer Backroll integrator Automation addict, Quentin is in charge of maintaining Cloudstack infrastructure up and running. He also helps our developers to speed-up code integration and deployment Senior solution architect, Grégoire helps our customers to define Cloud-based, hybrid and OnPremise strategies
  • 4.
    Who are we? We are a small independent French cloud provider that keeps its focus on small and medium sized companies We also provide for companies using Microsoft Dynamics CRM, and bring them our expertise on project management and development to meet their business needs We think that even a small team such as ours can bring great knowledge and success to our customers Our human-size teams put people at the heart of our digital services
  • 5.
    ABOUT DIMSI In 2022,DIMSI became a mission-driven company Foundation 2006 Area of expertise Company Size 4 sites : § Paris § Lorient § Sousse § Hong-Kong Offerings Microsoft Power Platform Cloud & Services Modern Apps Lorient Paris Sousse Hong-Kong
  • 6.
    What’s Backroll ? Whatprompted us to create BackROLL • No similar open-source community project • No turnkey solution available • No KVM backup software yet integrated into Apache CloudStack • You may have noticed that we are KVM/CloudStack enthusiasts :) What we wanted • No agent to install on virtual machines nor on hosts • Quiescent backup support • Backup deduplication • Incremental backups • Fully integrated into CloudStack Already proven safeguards • Currently in production for all internal and customer Apache Cloudstack instances at DIMSI • Has already been used for client requests for data restoration • Used to build our disaster recovery plan • No performance impact during backups Open Source with Apache2 license • The same licensing model as Apache Cloudstack • Everyone can check source code and bring their ideas to improve the software Aiming at becoming a community driven project !
  • 7.
  • 8.
    The various componentsof BackROLL Modern web UI • Front-end The web UI, offers the same functionality as the API • Back-end Contains API and core logic, it receives all API calls that will trigger tasks (Celery) • Workers Used to run the Celery tasks (VM list, backup list, host list, backup job, and so on...), can be stand-alone or clustered to maximize task parallelization • Flower Flower is used to monitor tasks in real-time and get logs via web interface / web API • Beat Beat is the scheduler, allowing us to create scheduled backup policies and run redundant and automated tasks (like backup cleaning every night) • Redis Redis is used as a database to store keys specific for every task, allowing tracking and keeping records.
  • 9.
    Bring an easy-to-useKVM backup solution to the community 1. Modern web UI Designed to be easy to use Easily manage your backups § Easily track the progress and logs of your backup tasks § Manage virtual machine backups, pools or backup policies via a graphical interface § Schedule backup and restore tasks § Built in VueJS 3 The web interface is optional, but still convenient for non-tech users 2. RESTful API That follows best practices Can be used in your existing workflows § Stateless § Uniform interface § Layered system § Can be used with scripts or tools like Ansible The learning curve is minimal if you are already used to working with other REST APIs 3. Containerized application Reduces maintenance requirements § Deploy and update using a docker-compose template § One click deployment § Customize deployment template to meet your needs Reduce maintenance hassle as everything needed is embedded in the docker image
  • 10.
    What does anAPI call look like? Modern web UI
  • 11.
    What does anAPI call look like? Modern web UI
  • 12.
    How does abackup work? Modern web UI
  • 13.
    What you canalready do A sample of what BackROLL is capable of Instant start of backup or restore task of a VM / Pool Everything is handled in the background Nothing to do on the VM side, even the needed restart is handled by BackROLL Backup / restore task scheduling If you prefer to schedule your tasks to run at night, it is possible! Self-management of lock for each VM, no way to go wrong Whenever a VM is linked to a task, BackROLL locks out any further operations on the virtual machine Retention management and automatic cleaning Only keep data you need Operation history and webhook notification To keep track of what has been done
  • 14.
    Caveats and limitations Fora better understanding • It is not currently possible to backup a shutdown VM that is managed by cloudstack • Restore and redeploy (to another VMs) task is currently in work-in-progress state • Only supports Slack for webhook notifications • Only NFS storage has been used to store the backups • You cannot select a specific disk you'd like to backup / restore, current task state is all-or-nothing • You cannot backup a VM that is currently using a snapshot disk file • Integration of BackROLL within CloudStack is in work-in-progress state
  • 15.
    Integration within yourworkflow ANALYSE Revue de l’architecture Prérequis Document de conception d’infrastructure complété incluant la volumétrie de données, le nombre d’utilisateurs, etc. Livrable Rapport de l'examen d'architecture accompagné des préconisations DIMSI délivré par un Solution Architecte D365 CONCEPTION Revue de la conception Prérequis Document de spécifications fonctionnels et Document de Design Technique complétés Plan d‘ intégration & Interfaces complété Livrable Document d’examen de la conception : Conclusions et recommandations. PERSONNALISATION Revue de la personnalisation Prérequis Document de conception technique détaillées. Accès à tous les codes sources de l'application. Définition d'un périmètre pour l'examen de codes. (L'étude portera sur un périmètre restreint représentatif d'un processus métier clé) Livrable Document d'examen de la personnalisation : Conclusion et recommandations DÉPLOIEMENT Revue de performances Prérequis Disponibilité du document de de déploiement. Plan de production complétés Environnement iso production et opérationnel (non accessibles aux utilisateurs) Définitions des scénarios de tests. Document d'exploitation. Livrable Document d’examen des performances et de l'exploitation, conclusions et recommandations.
  • 16.
    Integration with theCloudStack ecosystem (plugin) Cloudstack Back and restore framework Cloudstack Backroll Plugin How we’ve built Backroll plugin for Cloudstack (http://docs.cloudstack.apache.org/en/latest/adminguide/backup_and_recovery. html) We are developping our plugin using the Cloudstack Back and Restore framework How to setup the Backroll plugin • Set 'backroll' to the "backup.framework.provider.plugin" global configuration variable • Set these 3 variables: • "backup.plugin.backroll.config.url" • "backup.plugin.backroll.config.appname" • "backup.plugin.backroll.config.password" Code sample, authent, etc To do: diagram to describe components • Allows Cloudstack users to use 3rd party backup solutions • Abstracts the API commands required for common backup and recovery operations Concepts • 'SLA' Backups • Adhoc and user scheduled backups Key features • Authentication • List of existing backup policies • Start a backup for registred virtual machines • Restore virtual machines from backups • Delete backups
  • 17.
  • 18.
  • 19.
    www.dimsi.fr | contact@dimsi.fr| (+33) 01 75 43 89 66 4 Avenue Laurent Cély Hall D, 92600 Asnières-sur-Seine Thank you !