Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CRIU:
time and space travel
for Linux containers
CRIU:
time and space travel
for Linux containers
Kirill Kolyshkin
Contain...
AgendaAgenda
• Why would we want to migrate containers
• Why wouldn't we want to migrate containers
• How complex is to mi...
Live migration at a glanceLive migration at a glance
• Save the state
• Transfer the state
• Restore the state
3
Container live migrationContainer live migration
4
Why would we want to migrate containers?Why would we want to migrate containers?
• It's awesome!
• Load balancing in a clu...
Why wouldn't we want to live migrate containers?Why wouldn't we want to live migrate containers?
6
How to avoid live migrating containersHow to avoid live migrating containers
• Incoming traffic load balancing
• Microserv...
How to make live migration really live?How to make live migration really live?
• Need to get rid of migrating memory while...
Live migration in more detailsLive migration in more details
• Pre-copy: collect and transfer the memory (might be iterati...
Obstacles, booby traps, and rakesObstacles, booby traps, and rakes
10
VS
What do we need to migrateWhat do we need to migrate
• Virtual Machine
– Environment (i.e. virtual hardware)
– CPU state
–...
Collect and copy the memoryCollect and copy the memory
• Virtual Machine
– All memory is at hand
• Container
– Memory is s...
FreezingFreezing
• Virtual Machine
– Suspend all CPUs
• Container
– Walk the tree (/proc), catch the processes and freeze ...
Saving the stateSaving the state
• Virtual Machine
– Hardware state, tree, 300K, ~70 objects
• Container
– State of all ob...
Copying the stateCopying the state
• Virtual Machine
– Can read and copy at once, easy to serialize
• Container
– Not easy...
Restoring the stateRestoring the state
• VM: recreate the memory, state of CPUs and virtual hardware
• Containers
– In-ker...
FreezeFreeze
• VM: resume the virtual CPUs
• Container
– Either SIGCONT through the tree
– Or “unfreeze” the cgroup
– Prob...
Post-memory migration: network swap devicePost-memory migration: network swap device
• Not yet ready for neither VMs nor C...
ImplementationImplementation
• https://criu.org
• criu@openvz.org
• plus.google.com/+CriuOrg
• @__criu__
• github: xemul/c...
CRIU uses beyond the live migrationCRIU uses beyond the live migration
• HPC jobs: periodic checkpoints
• Slow boot servic...
Live migrationLive migration
• P.Haul
– Process hauler
– http://criu.org/P.Haul
– Uses CRIU for c/r
21
That's all Folks!
Kirill Kolyshkin
kir@openvz.org
That's all Folks!
Kirill Kolyshkin
kir@openvz.org
Upcoming SlideShare
Loading in …5
×

CRIU: Time and Space Travel for Linux Containers

1,216 views

Published on

This talk describes CRIU (checkpoint/restore in userspace) software, used to checkpoint, restore, and live migrate Linux containers and processes. It describes the live migration, compares it to that of VM, and shows other uses for checkpoint/restore.

Published in: Software
  • Be the first to comment

CRIU: Time and Space Travel for Linux Containers

  1. 1. CRIU: time and space travel for Linux containers CRIU: time and space travel for Linux containers Kirill Kolyshkin ContainerDays NYC, 30 Oct 2015
  2. 2. AgendaAgenda • Why would we want to migrate containers • Why wouldn't we want to migrate containers • How complex is to migrate containers 2
  3. 3. Live migration at a glanceLive migration at a glance • Save the state • Transfer the state • Restore the state 3
  4. 4. Container live migrationContainer live migration 4
  5. 5. Why would we want to migrate containers?Why would we want to migrate containers? • It's awesome! • Load balancing in a cluster • Kernel upgrade – Can be done without migration • Hardware upgrade 5
  6. 6. Why wouldn't we want to live migrate containers?Why wouldn't we want to live migrate containers? 6
  7. 7. How to avoid live migrating containersHow to avoid live migrating containers • Incoming traffic load balancing • Microservices • Crash-driven upgrades • Scheduled downtimes 7
  8. 8. How to make live migration really live?How to make live migration really live? • Need to get rid of migrating memory while the container is frozen • Two ways: – Pre-copy the memory – Post-copy the memory 8
  9. 9. Live migration in more detailsLive migration in more details • Pre-copy: collect and transfer the memory (might be iterative) • Freeze the container • Save its state • Copy the state • Restore • Unfreeze • Post-copy: swap in the memory over the network 9
  10. 10. Obstacles, booby traps, and rakesObstacles, booby traps, and rakes 10 VS
  11. 11. What do we need to migrateWhat do we need to migrate • Virtual Machine – Environment (i.e. virtual hardware) – CPU state – Memory • Container – Environment (cgroups, namespaces) – Processes and stuff – Memory 11
  12. 12. Collect and copy the memoryCollect and copy the memory • Virtual Machine – All memory is at hand • Container – Memory is spread through the processes – Different types of memory (shared/private, backed by a file or not) – Need to collect the processes first ● Only then collect the memory 12
  13. 13. FreezingFreezing • Virtual Machine – Suspend all CPUs • Container – Walk the tree (/proc), catch the processes and freeze those – Freeze cgroup helps a bit 13
  14. 14. Saving the stateSaving the state • Virtual Machine – Hardware state, tree, 300K, ~70 objects • Container – State of all objects, graph, 160K, ~1000 objects – Not all objects have decent API to get the state 14
  15. 15. Copying the stateCopying the state • Virtual Machine – Can read and copy at once, easy to serialize • Container – Not easy to serialize as it's a graph not a tree 15
  16. 16. Restoring the stateRestoring the state • VM: recreate the memory, state of CPUs and virtual hardware • Containers – In-kernel: create a myriad of small objects – In CRIU: same, but there might not be a convenient API ● Over 1000 syscalls ● Need to sort it all out 16
  17. 17. FreezeFreeze • VM: resume the virtual CPUs • Container – Either SIGCONT through the tree – Or “unfreeze” the cgroup – Problem: need to wake processes in the proper order 17
  18. 18. Post-memory migration: network swap devicePost-memory migration: network swap device • Not yet ready for neither VMs nor CTs • userfaultfd by Andrea Arcangeli of Red Hat – a file descriptor to inform about page fault and get a memory back – merged into 4.2 kernel – work in progress to use it for KVM/QEMU • Container – Userfault FD is not sufficient for CRIU case 18
  19. 19. ImplementationImplementation • https://criu.org • criu@openvz.org • plus.google.com/+CriuOrg • @__criu__ • github: xemul/criu 19
  20. 20. CRIU uses beyond the live migrationCRIU uses beyond the live migration • HPC jobs: periodic checkpoints • Slow boot services speed up • That magical SAVE button e.g. in games • Software testing speed up • Reverse debugging 20
  21. 21. Live migrationLive migration • P.Haul – Process hauler – http://criu.org/P.Haul – Uses CRIU for c/r 21
  22. 22. That's all Folks! Kirill Kolyshkin kir@openvz.org That's all Folks! Kirill Kolyshkin kir@openvz.org

×