Your SlideShare is downloading. ×
Linux Containers at scale: challenges in a very dense environment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Linux Containers at scale: challenges in a very dense environment

2,054

Published on

Talk I gave at ComSys2013 (http://www.ipsj.or.jp/sig/os/index.php?ComSys2013) about challenges of running LXC containers at Heroku.

Talk I gave at ComSys2013 (http://www.ipsj.or.jp/sig/os/index.php?ComSys2013) about challenges of running LXC containers at Heroku.

Published in: Technology, News & Politics
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,054
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. LINUX CONTAINERS AT SCALE CHALLENGES IN A VERY DENSE ENVIRONMENT fabio.kung@gmail.com fabio@heroku.com
  • 2. こんにちは COMSYS 2013 FABIO KUNG Technical Lead, Runtime Systems at Heroku
  • 3. heroku scale web=3 worker=2
  • 4. lxc-at-scale.herokuapp.com
  • 5. millions of (web) applications
  • 6. one of the biggest deployments of Linux containers (LXC)
  • 7. 12FACTOR.NET portable modern cloud platforms scale up and down
  • 8. two regions in production: us-east and eu-west
  • 9. many different Availability Zones
  • 10. CHALLENGES
  • 11. MILLIONS OF APPS
  • 12. IDLING
  • 13. STATE MANAGEMENT AND CONVERGENCE
  • 14. HRK SAEWB3WRE= EOU CL E= OKR5
  • 15. HRK SAEWB3WRE= EOU CL E= OKR5
  • 16. ARBITRARY CODE EXECUTION
  • 17. GIT PUSH HEROKU MASTER $gtrmt i eoe v hrk eou gthrk.o:yp.i (ec) i@eoucmmapgt fth hrk eou gthrk.o:yp.i (uh i@eoucmmapgt ps) $gtps hrk mse i uh eou atr Cutn ojcs 1 dn. onig bet: , oe Wiigojcs 10 (/) 11bts|0btss dn. rtn bet: 0% 11, 8 ye ye/, oe Ttl1(et 0,rue 0(et 0 oa dla ) esd dla ) --- Rb apdtce --> uy p eetd --- CmiigRb --> opln uy .. . T gthrk.o:yp.i o i@eoucmmapgt 9deb.21a mse - mse 1f0.f5b7 atr > atr
  • 18. CONTAINERS ARE JUST LINUX $hrk rnbs eou u ah Rnig`ah atce t tria..u,rn1 unn bs` tahd o emnl. p u. ~$hsnm otae 2df7-724d-713599c6 a1bc52-9dbb-0917e4 ~$p ax s uf UE SR PD%P %E I CU MM VZ RSTY S S T nbd ooy 1 00 00 238 15 ? . . 06 46 u50 407 2 00 00 146 21 ? . . 91 06 u50 407 4 00 00 136 11 ? . . 51 12 SA SAT TT TR S 0:2 21 S 0:2 21 R + 0:2 21 TM CMAD IE OMN 00 p-u :0 srn 00 bs :0 ah 00 p ax :0 _ s uf ~$l / s ap bn dv ec hm lb lb4 ls+on po si tp ur vr p i e t oe i i6 otfud rc bn m s a ~$
  • 19. ATTACHED PROCESSES TTY, attach/detach (re-entry)
  • 20. MIXED WORKLOADS CPU bound vs I/O bound tasks [1]
  • 21. RESOURCE MANAGEMENT
  • 22. INSTANCE SIZE
  • 23. DENSITY STATIC VS DYNAMIC
  • 24. LIMITS CPU, MEMORY, NETWORKING, OS
  • 25. LINUX WASN'T BORN WITH CONTAINERS
  • 26. COMPARE
  • 27. CPU GOOD shares/CFS, hard caps, pin
  • 28. MEMORY GOOD limits, overcommit, swap usage
  • 29. TOOLS BAD/HARD cgroup/namespace awareness / r cvs / g o p po cru Tracing and troubleshooting: perftools /rc<I>shd po/PD/ce ...
  • 30. I/O BAD/HARD blkio filesystem quotas statistics throttling
  • 31. NETWORKING BAD/HARD isolation abuse/fraud limits statistics L3 routing, L2 switching
  • 32. USER MANAGEMENT
  • 33. ROOT IN CONTAINERS apt-get install libmy-dev
  • 34. USER NAMESPACES
  • 35. UNIQUE UIDS often required by non-cgroup aware tools
  • 36. PID=1 SPECIAL MEANING
  • 37. can not die signals are filtered upstart vs systemd vs in-house
  • 38. DISK IMAGES
  • 39. HOW CONTAINERS ARE LAUNCHED 1. 2. 3. 4. 5. download extract make it safe set up the container filesystem run
  • 40. DOWNLOAD IS A PROBLEM 1-10GB
  • 41. SOLUTION: SPLIT base image, root filesystem (stack image): large, shared application files (slug): smaller, private to each container
  • 42. FORMATS directory based, tarballs, squashfs, ... filesystem w/ incremental snapshots: AUFS, btrfs, zfs, ... block level: dm, lvm, ... VM image formats: qcow2, vmdk, vdi, ...
  • 43. STANDARD?
  • 44. https://github.com/containers/container-rfc We lost the standards game for virtual machine images, but it feels like this community is tight nit enough we might be able to do something for Linux Containers. -- Alex Polvi (coreos.com)
  • 45. ROUTING
  • 46. containers are constantly moving
  • 47. STATE REPLICATION
  • 48. VIRTUAL NETWORKING
  • 49. 1. Ongaro, Diego, Alan L. Cox and Scortt Rixner. "Scheduling I/O in Machine Monitors". Rice University. 2 Aug 2012 <http://pages.cs.wisc.edu/~remzi/Classes/838/Spring2013/Pape vee08.pdf>
  • 50. THANK YOU! ありがとうございました fabio.kung@gmail.com fabio@heroku.com

×