Downloaded 292 times
Uploaded byDocker, Inc.
Live migrating a container: pros, cons and gotchas
The document discusses the complexities and considerations involved in live migrating containers, including reasons for migration like load balancing and kernel updates, as well as methods like memory pre-copy and post-copy. It highlights the pros and cons of these methods, emphasizing challenges such as unpredictable migration times and the risks of node failures. Additionally, it introduces tools like CRIU and p.haul, which facilitate container migration by managing state saving and restoring processes.
In this document
Powered by AI
Introduction to live migration, discussing its pros and cons, the agenda highlights the reasons for migration and complexities involved.
Overview of the live migration process comprising state saving, transferring, restoring, and explaining pre-copy and post-copy techniques.
Live migration can enhance performance by achieving load balancing, facilitating hardware upgrades, and is crucial when updating kernels.
Reasons to avoid live migration, including network traffic balancing and strategies like crash-driven updates and planned downtime.
Overview of the live migration process comprising state saving, transferring, restoring, and explaining pre-copy and post-copy techniques. Details of memory management during migration, explaining pre-copy and post-copy methods, their pros and cons for containers.
Key challenges ('gotchas') in live migration, such as CPU compatibility, kernel module checks, and issues related to memory and state restoration.
Discusses CRIU and P.Haul tools that facilitate live container migration, outlining their functionalities and integration process.
Provides resources for more information on CRIU and P.Haul, including websites, contact information, and social media links.
More Related Content
Similar to Live migrating a container: pros, cons and gotchas
Live migrating a container: pros, cons and gotchas
- 1. Live migrating acontainer: pros, cons and gotchas Pavel Emelyanov Principal engineer @ Virtuozzo
- 2. AgendaAgenda • Why youmight want to live migrate a container • Why (and how) to avoid live migration • Why is container live migration so complex 2
- 3. Migration in anutshelMigration in a nutshel • Save state • Copy state • Restore from state 3
- 4. Why you mightwant to live migrate a containerWhy you might want to live migrate a container • Spectacular • Load balancing • Updating kernel – Can avoid live migration, just C/R • Updaring or replacing hardware 4
- 5. Why to avoidlive migrationWhy to avoid live migration 5
- 6. How to avoidlive migrationHow to avoid live migration • Balance network traffic • Microservices • Crash-driven updates • Planned downtime 6
- 7. Making live migrationliveMaking live migration live • State saving, transfering and restoring happens with tasks frozen • (Big) memory transfer should not be done at that time • Memory pre-copy • Memory post-copy 7
- 8. Pre-copyPre-copy • Track memorychanges, copy memory while tasks are running, goto again • Pros: – Safe: once migrated, source node can disappear • Cons: – Unpredictable: iterations may take long – Non-guaranteed: “dirty” memory next round may remain big 8
- 9. Post-copyPost-copy • Migrate allbut memory, turn on “network swap” on destination • Pros: – Predictable: time to migrate can be well estimated • Cons: – Unsafe: src node death means death of container on destination 9
- 10. Live migration atlengthLive migration at length • Memory pre-copy (iteratively, optional) • Freeze + Save state • Copy state • Restore from state + Unfreeze and resume • Memory post-copy (optional) 10
- 11.
- 12. Things to workwithThings to work with • VM – Environment: virtual hardware, paravirt – CPU – Memory • Container – Environment: cgroups, namespaces – Processes and other animals – Memory 12
- 13. Memory pre-copyMemory pre-copy •VM – All memory at hands – Plain address space • Container – Memory ● is scatered over the processes ● can be (or can be not) shared ● can be (or can be not) mapped to disk files 13
- 14. Save stateSave state •VM – Hardware state ● Tree of ~100 objects ● Fixed amount of data per each • Container – State of all objects ● Graph of up to ~1000 objects ● All have different amount of data, different reading API 14
- 15. Restore from stateRestorefrom state • VM – Copy memory in place, write state into devices • Container – Creation of many small objects – Not all have sane API for creation ● Creation sequence can be non-trivial 15
- 16. Memory post-copyMemory post-copy •UserfaultFD from Andrea Archangeli • VM – Merged into 4.2 • Container – Non-cooperative work of uffd monitor and client, need further patching 16
- 17. And we alsoneed this, this and this!And we also need this, this and this! • Check for CPUs compatibility • Check and load necessary kernel modules (iptables, filesystems) • Non-shared filesystem should be copied • Roll-back on source node if something fails in between – Keep tasks frozen after dump, kill after restore 17
- 18. ImplementationImplementation • CRIU – Save& restore state – Memory pre/post copy • P.Haul – Checks – Orchestrate all C/R steps – Deal with filesystem 18
- 19. P.Haul goalsP.Haul goals •Provide engine for containers live miration using CRIU • Perform necessary pre-checks (e.g. CPU compatibility) • Organize memory pre-copy and/or post-copy • Take care of file-system migration (if needed) 19
- 20. Under the hoodUnderthe hood 20 CRIU CRIUp.haul p.hauldocker -d docker -d migrate src dst check (CPUs, kernels) pre-dump memory dump other images restore memory lazy mem FS FS copy done pre-copypost-copy kill freeze time
- 21. More infoMore info •http://criu.org • http://criu.org/P.Haul • criu@openvz.org • +CriuOrg / @__criu__ • https://github.com/xemul/(criu|p.haul) 21
- 22.