CRIU: Time and Space Travel for Linux Containers
•Download as ODP, PDF•
7 likes•1,711 views
This talk describes CRIU (checkpoint/restore in userspace) software, used to checkpoint, restore, and live migrate Linux containers and processes. It describes the live migration, compares it to that of VM, and shows other uses for checkpoint/restore.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to CRIU: Time and Space Travel for Linux Containers
Similar to CRIU: Time and Space Travel for Linux Containers (20)
More from Kirill Kolyshkin
More from Kirill Kolyshkin (8)
Recently uploaded
Recently uploaded (20)
CRIU: Time and Space Travel for Linux Containers
- 1. CRIU: time and space travel for Linux containers CRIU: time and space travel for Linux containers Kirill Kolyshkin ContainerDays NYC, 30 Oct 2015
- 2. AgendaAgenda • Why would we want to migrate containers • Why wouldn't we want to migrate containers • How complex is to migrate containers 2
- 3. Live migration at a glanceLive migration at a glance • Save the state • Transfer the state • Restore the state 3
- 4. Container live migrationContainer live migration 4
- 5. Why would we want to migrate containers?Why would we want to migrate containers? • It's awesome! • Load balancing in a cluster • Kernel upgrade – Can be done without migration • Hardware upgrade 5
- 6. Why wouldn't we want to live migrate containers?Why wouldn't we want to live migrate containers? 6
- 7. How to avoid live migrating containersHow to avoid live migrating containers • Incoming traffic load balancing • Microservices • Crash-driven upgrades • Scheduled downtimes 7
- 8. How to make live migration really live?How to make live migration really live? • Need to get rid of migrating memory while the container is frozen • Two ways: – Pre-copy the memory – Post-copy the memory 8
- 9. Live migration in more detailsLive migration in more details • Pre-copy: collect and transfer the memory (might be iterative) • Freeze the container • Save its state • Copy the state • Restore • Unfreeze • Post-copy: swap in the memory over the network 9
- 10. Obstacles, booby traps, and rakesObstacles, booby traps, and rakes 10 VS
- 11. What do we need to migrateWhat do we need to migrate • Virtual Machine – Environment (i.e. virtual hardware) – CPU state – Memory • Container – Environment (cgroups, namespaces) – Processes and stuff – Memory 11
- 12. Collect and copy the memoryCollect and copy the memory • Virtual Machine – All memory is at hand • Container – Memory is spread through the processes – Different types of memory (shared/private, backed by a file or not) – Need to collect the processes first ● Only then collect the memory 12
- 13. FreezingFreezing • Virtual Machine – Suspend all CPUs • Container – Walk the tree (/proc), catch the processes and freeze those – Freeze cgroup helps a bit 13
- 14. Saving the stateSaving the state • Virtual Machine – Hardware state, tree, 300K, ~70 objects • Container – State of all objects, graph, 160K, ~1000 objects – Not all objects have decent API to get the state 14
- 15. Copying the stateCopying the state • Virtual Machine – Can read and copy at once, easy to serialize • Container – Not easy to serialize as it's a graph not a tree 15
- 16. Restoring the stateRestoring the state • VM: recreate the memory, state of CPUs and virtual hardware • Containers – In-kernel: create a myriad of small objects – In CRIU: same, but there might not be a convenient API ● Over 1000 syscalls ● Need to sort it all out 16
- 17. FreezeFreeze • VM: resume the virtual CPUs • Container – Either SIGCONT through the tree – Or “unfreeze” the cgroup – Problem: need to wake processes in the proper order 17
- 18. Post-memory migration: network swap devicePost-memory migration: network swap device • Not yet ready for neither VMs nor CTs • userfaultfd by Andrea Arcangeli of Red Hat – a file descriptor to inform about page fault and get a memory back – merged into 4.2 kernel – work in progress to use it for KVM/QEMU • Container – Userfault FD is not sufficient for CRIU case 18
- 19. ImplementationImplementation • https://criu.org • criu@openvz.org • plus.google.com/+CriuOrg • @__criu__ • github: xemul/criu 19
- 20. CRIU uses beyond the live migrationCRIU uses beyond the live migration • HPC jobs: periodic checkpoints • Slow boot services speed up • That magical SAVE button e.g. in games • Software testing speed up • Reverse debugging 20
- 21. Live migrationLive migration • P.Haul – Process hauler – http://criu.org/P.Haul – Uses CRIU for c/r 21
- 22. That's all Folks! Kirill Kolyshkin kir@openvz.org That's all Folks! Kirill Kolyshkin kir@openvz.org
Editor's Notes
- It's not about CRIU per se, as I can talk for a whole day about it, and you are probably not interested. It's about one of it's applications, which is containers live migration. I'm going to tell why and when it is useful, why it's not, and what are the obstacles if you decide to do it. <number>
- What is live migration? Live migration is very well described in science fiction, it's just its called teleportation there. An object is analyzed, information about its bits and pieces are communicated to the other side, and it's assembled there at the destination. It's pretty much the same for containers, except for the fact it's already implemented. <number>
- It is already implemented in OpenVZ, for about 10 years, in the kernel, as a kernel modules. For the last 4 years we are working on re-implementing that feature using a different «engine», developing the functionality of analyzing, decomposing and then re-composing the processes not as kernel modules, but as a user-space application. <number>
- Why would we want to migrate containers?: First, It looks awesome, totally mind blowing. If you take an inexperienced user and show them a set of processes with all the bells and whistles and stuff being moved from one physical server to another without being stopped --- it looks cool! Live migration can also be used to balance a load between a few machines. <number>
- Of course live migration is a complex technology, and it is error-prone and people are afraid of using it because of various possible side effects, good or bad. So, there are ways to avoid live migration. <number>
- One method is to balance not the processes using the resources, but the reason why they start to do it. For example, incoming network traffic – you can use some frontend to load balance, if your architecture allows it. Another method is microservices – you run services that don't have much context, much state, so you can stop anything and run it on a different machine pretty fast and without losing anything. Again, if your architecture allows it. This is a paradigm of OpenStack, Docker, and some Docker-based projects such as Kubernetes. Third option is somewhat peculiar, but is still being used. You wait until there's a major problem with the machine, and then you reboot and upgrade. Obvious option is to plan a downtime. <number>
- Anyway, live migration is also a way to go, and once we start using it we'll see that during migration a lot of time is spent on moving the memory over the network. To make the migration really live, to have a really uninterrupted service, you need to exclude this memory migration from the period of time when the container is frozen. There are two options for that. First one is to copy all or most of the memory before freezing the container. Second is not to migrate the memory. <number>
- Once we take into account this need to pre- or post-migrate the memory, the live migration is becoming more complicated. <number>
- There is some specifics in implementing such a technology for containers. As live migration for VMs exist for a while, while for containers it's relatively new. So to better understand the details, let's compare containers and VMs. Let's do it step by step. <number>
- All the virtual hardware a hypervisor gives to the guest OS, virtual CPU state and memory state. It's sort of like the same for Cts, but named differently. Instead of virtual hardware we have cgroups and namespaces. Instead of CPUs we have processes. <number>
- Not a problem for VM, as a hypervisor manages VM memory and knows everything about it. For Cts, there are many different types of memory – shared or private, backed by a file or not backed by a file, etc etc <number>
- There are two ways to catch the processes. First, we follow the steps of ps utility, get the processes one by one, stop them, make sure the ones we haven't stopped yet might fork and their children might fork. A second option is to use freeze cgroup. If you put processes inside such a cgroup you can later say «freeze!» and it will. In such case this freezing will be done by the kernel who is good at it. <number>
- For VM running a fresh install of say Fedora Linux, excluding the memory it will be about 300K of data and less than 100 objects. For CT, this is way more fine grained – open files, sockets, and everything those processes might have used. Plus, some of those objects might be shared, like files – so we have a graph rather than a tree. It takes somethat less space (comparable to VM), but the number of objects is two orders of magnitude greater! The second problem is not a fundamental one, but rather a specifics of the CRIU implementation. If we would do checkpoint from the kernel, we would know everything, every state of every object. But as we are doing it from the userspace we need some API to get such state. <number>
- For containers, receiving side can't get it from a socket as there might be some objects depending on the objects that are not yet copied <number>
- For CTs, we have a set of objects to be restored, and we have relations between those objects, a graph, and we have some rules, some restrictions on how to create these objects with their relations. It's not like we can create an object and then tie it to some other objects. We also have a state to which we want to go. So we need to solve this task, figure out a sequence to recreate all this. <number>
- To install a font: Open Fonts by clicking the Start button , clicking Control Panel, clicking Appearance and Personalization, and then clicking Fonts. Click File, and then click Install New Font. ... In the Add Fonts dialog box, under Drives, click the drive where the font that you want to install is located. http://windows.microsoft.com/en-us/windows-vista/install-or-uninstall-fonts <number>
- If a page is missing, the kernel won't kill the process but send a special message over that file descriptor so the listening process can get this memory and give it to the kernel Userfaultfd is not working as it for CRIU for a few reasons: - with QEMU, it's the same process initialing and handling the page fault,with CRIU it's different processes - not all memory types are currently supported . - an app can remap its memory, currently unsupported - fork() is not supported, child wil have pages with zeroes <number>
- Vibrant community, version 1.7.2 was released this week. Mostly driven by Odin, but also Google, Canonical, Red Hat, SuSE Debian, Samsung, Huawei, Docker… Integrated with OpenVZ (future version), LXC, LXD, Docker/Rocket libcontainer. Linux kernel developers are aware and helpful <number>
- For slow boot, we tried starting Eclipse GUI, took 30s to start, 1.5s to restore. <number>
- Project logo is the little humpbacked horse (a magic pony) <number>