Seven problems of Linux Containers


Published on

OpenVZ, which has turned 7 recently, is an implementation of lightweight virtualization technology for Linux, something which is also referred to as LXC or just containers. The talk gives an insight into 7 different problems with containers and how they were solved. While most of these problems and solutions belongs in the Linux kernel, kernel knowledge is not expected from the audience.

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Seven problems of Linux Containers

  1. 1. || || criu.orgSeven Problemsof Linux ContainersKir Kolyshkin<>28 April 2013 LinuxFest Northwest
  2. 2. || || criu.orgSeventy Seven Problemsof Linux ContainersKir Kolyshkin<>28 April 2013 LinuxFest Northwest(of which I am going to cover six)
  3. 3. || || criu.orgProblem 1: Effective virtualization●Virtualization is partitioning●Historical way: $M mainframes●Modern way: virtual machines●Problem: performance overhead●Partial solution: hardware support(Intel VT, AMD V)
  4. 4. || || criu.orgSolution: isolation●Run many isolated userspace instanceson top of sone single (Linux) kernel●All processes see each other– files, process information, network,shared memory, users, etc.●Make them unsee it!
  5. 5. || ||
  6. 6. || || criu.orgOne historical way to unseechroot()
  7. 7. || || criu.orgNamespaces●Implemented in the Linux kernel– PID– net– IPC– UTS– mnt– user●clone() with CLONE_NEW* flags
  8. 8. || || criu.orgProblem 2: Shared resources●All containers share the same set of resources(CPU, RAM, disk, various kernel things ...)●Need fair distribution of goods so everyonegets their share●Need DoS prevention●Need prioritization– “All animals are equal, but some animals are moreequal than others” -- George Orwell
  9. 9. || ||
  10. 10. || || criu.orgSolution: OpenVZ resource controls●OpenVZ:– user beancounters●controls 20 parameters– hierarchical CPU scheduler– disk quota per containers– I/O priorities per-container●Dynamic control, can “resize” runtime
  11. 11. || || criu.orgSolution: cgroups●Cgroups is a mechanism to control resourcesper hierarchical groups of processes●Cgroups is nothing without controllers:– blkio, cpu, cpuacct, cpuset, devices, freezer,memory, net_cls, net_prio●Cgroups are orthogonal to namespaces●Still a work in progress (kernel memory)
  12. 12. || || criu.orgProblem 3: easy resources●User Beancounters are complicated:–– user has to set all these parameters– some of which are interdependent●We created a collection of valid configs,●... wrote a whole book about UBC●... and a set of tools to help
  13. 13. || ||
  14. 14. || || criu.orgSolution: VSwap●Only two primary parameters: RAM and swap– others still exist, but no longer required to set●Swap is virtual, no actual I/O is performed●Slow down to emulate real swap●Only when actual global RAM shortageoccurs,virtual swap goes into the real swap●Currently only available in OpenVZ kernel
  15. 15. || || criu.orgProblem 4: fast live migration●We can migrate an OpenVZ containerfrom one physical server to anotherwithout a shutdown●We want to do it fast even for huge containers– huge disk: use shared storage– huge RAM: ???
  16. 16. || || criu.orgNormal migration process●(Assuming shared storage)●1 Freeze the container●2 Dump its complete state to a dump file●3 Copy dump file to destination server●4 Undump●5 Unfreeze●Problem: huge dump file
  17. 17. || || criu.orgSolution 1: network swap●1 Dump the minimal memory, lock the rest●2 Restore the minimal memory,mark the rest as swapped out●3 Set up network swap from the source●4 Unfreeze. Missing RAM will be “swapped in”●5 Migrate the rest of RAM and kill it on source
  18. 18. || ||
  19. 19. || || criu.orgSolution 1: network swap●1 Dump the minimal memory, lock the rest●2 Copy, undump what we have,mark the rest as swapped out●3 Set up network swap served from the source●4 Unfreeze. Missing RAM will be “swapped in”●5 Migrate the rest of RAM and kill it on source●PROBLEM? Reliability, no way to rollback
  20. 20. || || criu.orgSolution 2: Iterative RAM migration●1 Ask kernel to track modified pages●2 Copy all memory to destination system●3 Ask kernel for list of modified pages●4 Copy those pages●5 GOTO 3 until satisfied●6 Freeze and do migration as usual
  21. 21. || || criu.orgProblem 5: upstreaming●OpenVZ was developed separately●Then we wanted to merge it upstream(i.e. to vanilla Linux kernel)●Problem?
  22. 22. || ||
  23. 23. || || criu.orgProblem 5: upstreaming●OpenVZ was developed separately●Then we wanted to merge it upstream(i.e. to vanilla Linux kernel)●Problem:●upstream devs are not accepting our work
  24. 24. || || criu.orgSolution 1: rewrite from scratch●User Beancounters -> CGroups●Did 2 rewrites for PID namespaceuntil it finally got accepted●Network namespace redone●It works!●about 1500 patches got landed to vanilla●II Parallels made it to top10 contributors
  25. 25. || || criu.orgSolution 2: CRIU●We tried hard to merge checkpoint/restore●Other people tried hard too, no luck●Cant make it to the kernel, lets go userspace●With minimal kernel intervention whenrequired●Kernel exports most of information already, solets just add missing bits and pieces
  26. 26. || || criu.orgCRIU●Checkpoint / Restore (mostly) In UserspaceTools currently at version 0.4●Will do 1.0 release this year●Kernel 3.8 has about 120 patches from us– 95% of needed features are there●Memory snapshot recently made it to -mm tree
  27. 27. || ||
  28. 28. || || criu.orgProblem 6: common file system●Container is just a directory on host,all CTs reside on the same FS●File system journal is a bottleneck●Lots of small-size files I/O on CT backup●No sub-tree disk quota support in upstream●No per-container snapshots●Live migration: rsync -- changed inodes●File system type and properties are fixed
  29. 29. || || criu.orgSolution 1: LVM●Only works only on top of block device●Hard to manage (e.g. how to migrate hugevolume?)●No dynamic allocation●Complicated management
  30. 30. || || criu.orgSolution 2: loop device●VFS operations leads to double page-caching– (already fixed in the recent kernels)●No dynamic allocation, max space is used●Limited feature set
  31. 31. || || criu.orgSolution 3: ploop●Basic idea: same as loop, just better●Modular design:– various image formats (qcow2 in TODO)– various I/O backends●More features:– live resize– instant live snapshots– write tracker to help in live migration
  32. 32. || || criu.orgAny problems questions?●●Twitter: @kolyshkin
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.