LINUX
CONTAINERS
AT SCALE

CHALLENGES IN A VERY DENSE ENVIRONMENT
fabio.kung@gmail.com
fabio@heroku.com
こんにちは COMSYS 2013

FABIO KUNG
Technical Lead, Runtime Systems at Heroku
heroku scale web=3 worker=2
lxc-at-scale.herokuapp.com
millions of (web) applications
one of the biggest deployments of Linux containers (LXC)
12FACTOR.NET
portable
modern cloud platforms
scale up and down
two regions in production: us-east and eu-west
many different Availability Zones
CHALLENGES
MILLIONS OF APPS
IDLING
STATE MANAGEMENT AND
CONVERGENCE
HRK SAEWB3WRE=
EOU CL E= OKR5
HRK SAEWB3WRE=
EOU CL E= OKR5
ARBITRARY CODE
EXECUTION
GIT PUSH HEROKU MASTER
$gtrmt i eoe v
hrk
eou
gthrk.o:yp.i (ec)
i@eoucmmapgt fth
hrk
eou
gthrk.o:yp.i (uh
i@eoucmmapgt ps)...
CONTAINERS ARE JUST LINUX
$hrk rnbs
eou u ah
Rnig`ah atce t tria..u,rn1
unn bs` tahd o emnl. p u.
~$hsnm
otae
2df7-724d-71...
ATTACHED PROCESSES
TTY, attach/detach (re-entry)
MIXED WORKLOADS
CPU bound vs I/O bound tasks [1]
RESOURCE MANAGEMENT
INSTANCE SIZE
DENSITY
STATIC VS DYNAMIC
LIMITS
CPU, MEMORY, NETWORKING, OS
LINUX WASN'T BORN WITH
CONTAINERS
COMPARE
CPU
GOOD
shares/CFS, hard caps, pin
MEMORY
GOOD
limits, overcommit, swap usage
TOOLS
BAD/HARD
cgroup/namespace awareness
/ r cvs / g o p
po
cru
Tracing and troubleshooting:
perftools
/rc<I>shd
po/PD/ce...
I/O
BAD/HARD
blkio
filesystem quotas
statistics
throttling
NETWORKING
BAD/HARD
isolation
abuse/fraud
limits
statistics
L3 routing, L2 switching
USER MANAGEMENT
ROOT IN CONTAINERS
apt-get install libmy-dev
USER NAMESPACES
UNIQUE UIDS
often required by non-cgroup aware tools
PID=1 SPECIAL MEANING
can not die
signals are filtered
upstart vs systemd vs in-house
DISK IMAGES
HOW CONTAINERS ARE LAUNCHED
1.
2.
3.
4.
5.

download
extract
make it safe
set up the container filesystem
run
DOWNLOAD IS A PROBLEM
1-10GB
SOLUTION: SPLIT
base image, root filesystem (stack image): large, shared
application files (slug): smaller, private to eac...
FORMATS
directory based, tarballs, squashfs, ...
filesystem w/ incremental snapshots:
AUFS, btrfs, zfs, ...
block level: d...
STANDARD?
https://github.com/containers/container-rfc
We lost the standards game for virtual
machine images, but it feels like this
...
ROUTING
containers are constantly moving
STATE REPLICATION
VIRTUAL NETWORKING
1. Ongaro, Diego, Alan L. Cox and Scortt Rixner. "Scheduling I/O in
Machine Monitors". Rice University. 2 Aug 2012
<http:/...
THANK YOU!

ありがとうございました
fabio.kung@gmail.com
fabio@heroku.com
Linux Containers at scale: challenges in a very dense environment
Linux Containers at scale: challenges in a very dense environment
Linux Containers at scale: challenges in a very dense environment
Linux Containers at scale: challenges in a very dense environment
Linux Containers at scale: challenges in a very dense environment
Linux Containers at scale: challenges in a very dense environment
Upcoming SlideShare
Loading in …5
×

Linux Containers at scale: challenges in a very dense environment

2,867 views

Published on

Talk I gave at ComSys2013 (http://www.ipsj.or.jp/sig/os/index.php?ComSys2013) about challenges of running LXC containers at Heroku.

Published in: Technology, News & Politics

Linux Containers at scale: challenges in a very dense environment

  1. 1. LINUX CONTAINERS AT SCALE CHALLENGES IN A VERY DENSE ENVIRONMENT fabio.kung@gmail.com fabio@heroku.com
  2. 2. こんにちは COMSYS 2013 FABIO KUNG Technical Lead, Runtime Systems at Heroku
  3. 3. heroku scale web=3 worker=2
  4. 4. lxc-at-scale.herokuapp.com
  5. 5. millions of (web) applications
  6. 6. one of the biggest deployments of Linux containers (LXC)
  7. 7. 12FACTOR.NET portable modern cloud platforms scale up and down
  8. 8. two regions in production: us-east and eu-west
  9. 9. many different Availability Zones
  10. 10. CHALLENGES
  11. 11. MILLIONS OF APPS
  12. 12. IDLING
  13. 13. STATE MANAGEMENT AND CONVERGENCE
  14. 14. HRK SAEWB3WRE= EOU CL E= OKR5
  15. 15. HRK SAEWB3WRE= EOU CL E= OKR5
  16. 16. ARBITRARY CODE EXECUTION
  17. 17. GIT PUSH HEROKU MASTER $gtrmt i eoe v hrk eou gthrk.o:yp.i (ec) i@eoucmmapgt fth hrk eou gthrk.o:yp.i (uh i@eoucmmapgt ps) $gtps hrk mse i uh eou atr Cutn ojcs 1 dn. onig bet: , oe Wiigojcs 10 (/) 11bts|0btss dn. rtn bet: 0% 11, 8 ye ye/, oe Ttl1(et 0,rue 0(et 0 oa dla ) esd dla ) --- Rb apdtce --> uy p eetd --- CmiigRb --> opln uy .. . T gthrk.o:yp.i o i@eoucmmapgt 9deb.21a mse - mse 1f0.f5b7 atr > atr
  18. 18. CONTAINERS ARE JUST LINUX $hrk rnbs eou u ah Rnig`ah atce t tria..u,rn1 unn bs` tahd o emnl. p u. ~$hsnm otae 2df7-724d-713599c6 a1bc52-9dbb-0917e4 ~$p ax s uf UE SR PD%P %E I CU MM VZ RSTY S S T nbd ooy 1 00 00 238 15 ? . . 06 46 u50 407 2 00 00 146 21 ? . . 91 06 u50 407 4 00 00 136 11 ? . . 51 12 SA SAT TT TR S 0:2 21 S 0:2 21 R + 0:2 21 TM CMAD IE OMN 00 p-u :0 srn 00 bs :0 ah 00 p ax :0 _ s uf ~$l / s ap bn dv ec hm lb lb4 ls+on po si tp ur vr p i e t oe i i6 otfud rc bn m s a ~$
  19. 19. ATTACHED PROCESSES TTY, attach/detach (re-entry)
  20. 20. MIXED WORKLOADS CPU bound vs I/O bound tasks [1]
  21. 21. RESOURCE MANAGEMENT
  22. 22. INSTANCE SIZE
  23. 23. DENSITY STATIC VS DYNAMIC
  24. 24. LIMITS CPU, MEMORY, NETWORKING, OS
  25. 25. LINUX WASN'T BORN WITH CONTAINERS
  26. 26. COMPARE
  27. 27. CPU GOOD shares/CFS, hard caps, pin
  28. 28. MEMORY GOOD limits, overcommit, swap usage
  29. 29. TOOLS BAD/HARD cgroup/namespace awareness / r cvs / g o p po cru Tracing and troubleshooting: perftools /rc<I>shd po/PD/ce ...
  30. 30. I/O BAD/HARD blkio filesystem quotas statistics throttling
  31. 31. NETWORKING BAD/HARD isolation abuse/fraud limits statistics L3 routing, L2 switching
  32. 32. USER MANAGEMENT
  33. 33. ROOT IN CONTAINERS apt-get install libmy-dev
  34. 34. USER NAMESPACES
  35. 35. UNIQUE UIDS often required by non-cgroup aware tools
  36. 36. PID=1 SPECIAL MEANING
  37. 37. can not die signals are filtered upstart vs systemd vs in-house
  38. 38. DISK IMAGES
  39. 39. HOW CONTAINERS ARE LAUNCHED 1. 2. 3. 4. 5. download extract make it safe set up the container filesystem run
  40. 40. DOWNLOAD IS A PROBLEM 1-10GB
  41. 41. SOLUTION: SPLIT base image, root filesystem (stack image): large, shared application files (slug): smaller, private to each container
  42. 42. FORMATS directory based, tarballs, squashfs, ... filesystem w/ incremental snapshots: AUFS, btrfs, zfs, ... block level: dm, lvm, ... VM image formats: qcow2, vmdk, vdi, ...
  43. 43. STANDARD?
  44. 44. https://github.com/containers/container-rfc We lost the standards game for virtual machine images, but it feels like this community is tight nit enough we might be able to do something for Linux Containers. -- Alex Polvi (coreos.com)
  45. 45. ROUTING
  46. 46. containers are constantly moving
  47. 47. STATE REPLICATION
  48. 48. VIRTUAL NETWORKING
  49. 49. 1. Ongaro, Diego, Alan L. Cox and Scortt Rixner. "Scheduling I/O in Machine Monitors". Rice University. 2 Aug 2012 <http://pages.cs.wisc.edu/~remzi/Classes/838/Spring2013/Pape vee08.pdf>
  50. 50. THANK YOU! ありがとうございました fabio.kung@gmail.com fabio@heroku.com

×