VM backup beyond Bacula/Bareos with backy
•
0 likes•185 views
The Flying Circus is an Operations-as-a-Service platform that supports project development teams to run their custom-develop software for clients. Earlier in 2014 we experienced a major data loss and had to perform massive disaster recovery. Unfortunately our Bacula setup was not up to the task and it took us longer and more effort to restore the data than we and our customers expected. In this case study I’d like to present our public and very honest root cause analysis on how we managed to lose a lot of VMs’ data, how the restore happened, what we learned and how we’re trying to get better. After investigating our options for the future we decided to move away from Bacula’s file and VTL-oriented model and are currently implementing a solution based on CoW-filesystems (ZFS/btrfs), block-layer snapshots and diffing, and a small utility to glue things together.
Recommended
Recommended
More Related Content
What's hot
What's hot (16)
Viewers also liked
Viewers also liked (20)
Similar to VM backup beyond Bacula/Bareos with backy
Similar to VM backup beyond Bacula/Bareos with backy (20)
Recently uploaded
Recently uploaded (20)
VM backup beyond Bacula/Bareos with backy
- 1. backy VM backup beyond Bacula/Bareos Christian Theune @theuni ct@flyingcircus.io
- 2. Mea Culpa
- 3. And I almost missed it — again
- 4. Backup!!11!!
- 5. • flyingcircus.io • DevOps as a Service • custom, mission-critical web applications
- 6. Part I - Oh the Pain
- 7. The story unfolds …
- 14. Restore script bottleneck: global lock
- 15. Undetected inconsistency in important customer database
- 16. Bacula: complexity and the VTL
- 17. Not “everything” backed up.
- 18. 24 hours are not a sufficient RPO in quite a few cases
- 19. Paper cuts • Hard link farms • Boot loaders • The director as a “most valuable bottleneck”
- 20. Recap • Restore fiddly to script • Undetected inconsistency that was hard to deal with • Blind spots • Daily Interval • Overall complexity, performance and the VTL • Paper cuts
- 21. Part II - Make a wish
- 22. Simplicity • Restore with basic Unix tools • No VTL • Not mixing data of different VMs
- 23. Reliability • Verification / Scrubbing / (Repair) • High frequency • Integration with storage snapshots • Not inventing new formats
- 24. Operability • Avoid bottlenecks / head-of-line blocking • Efficient deltas for large files (ZODB) • Parallelisation (multiple jobs and multiple servers) • Simple scripting and environment-specific integration • Coordination: pre/post actions on storage, hypervisor, VM …
- 25. Operability II • Simple Nagios integration to ensure we notice RPO/ SLA failures • RTO-compliance during mass-restore • Self-service for customers to restore files or VMs
- 26. Part III - Let’s do this!
- 27. –Probably someone, maybe me “One size fits all … not” It’s all about size
- 28. It’s all about size: backy
- 29. It’s all about size: Bacula
- 30. It’s all about size: Bareos
- 31. The giants we stand on
- 32. Limits • Not a general purpose backup system. • No tapes. No weird hardware. • Restore without tools. • Dead-simple configuration.
- 33. Let’s take a tour
- 35. simple!?!
- 36. Hello CLI
- 37. Running a single backup
- 40. Hello, daemon!
- 42. Scheduling
- 43. Scheduling
- 44. Purging
- 45. Purging
- 46. Scrubbing • partial, random verification during backup against source • btrfs scrubbing • Raid-6
- 47. Deleting a VM • rewrite config, reload master • rm -rf
- 48. Monitoring • old state is uninteresting • do I have to act?
- 49. Monitoring
- 51. What did we leave out? • Physical host backup • Guesstimating achievable backup storage ratio
- 52. Future • trim-ready - waiting for our whole stack (Guest, Hypervisor, Ceph, …) to pass this through • Hot reload of scheduler • Ensuring we can move VM backup directories between different backup hosts
- 53. Having your backup and eating it!
- 55. Image Sources • https://www.flickr.com/photos/mpa-berlin/ 14337541104/ • https://www.flickr.com/photos/ seattlemunicipalarchives/4777122561 • https://www.flickr.com/photos/jkroll/15314415946/ • https://www.flickr.com/photos/dvids/6956044669/ • https://www.flickr.com/photos/flowtastic/7354146628/
- 56. Image Sources • https://www.flickr.com/photos/galeria_stefbu/ 4781641072/in/pool-fotoszene/ • https://www.flickr.com/photos/dlography/6982668385/ • https://www.flickr.com/photos/ 127437845@N04/15142216255 • https://www.flickr.com/photos/ clement127/15440591160
- 57. Image Sources • https://www.flickr.com/photos/ clement127/15999160179 • https://www.flickr.com/photos/ 63433965@N04/5814096531/ • private pictures