Uploaded on

Xuli, the chief evangelist of Qiniu Cloud Storage presented these slides on Docker X OpenStack Meetup Shanghai.

Xuli, the chief evangelist of Qiniu Cloud Storage presented these slides on Docker X OpenStack Meetup Shanghai.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
416
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
24
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. The Docker Way xuli@qiniu.com
  • 2. Agenda • Docker: What, Why and How • Story: Qiniu meet Docker • Q & A
  • 3. Container do Virtualization
  • 4. Container vs. VM Containers  are  isolated,   but  share  OS  and,  where   appropriate,  bins/libraries App   A   ! ! ! ! ! Hypervisor  (Type  2) Host  OS Server ! ! ! ! Guest   OS   ! ! ! ! ! ! ! Bins/   Libs   ! ! ! App   A’   ! ! ! ! ! ! ! ! ! Guest   OS   ! ! ! ! ! ! ! Bins/   Libs   ! ! ! App   B   ! ! ! ! ! ! ! ! ! Guest   OS   ! ! ! ! ! ! ! Bins/   Libs   ! ! ! VM ! ! ! ! Guest   OS   ! ! ! ! ! ! ! Guest   OS   ! ! ! …result  is  significantly  faster  deployment,     much  less  overhead,  easier  migration,   faster  restart App  A’ Docker   Host  OS Server ! ! ! ! Bins/Libs   ! ! ! App  A ! ! ! ! Bins/Libs   ! ! ! App  B App  B’ App  B’ Container App  B’
  • 5. • The same see and shared for each Linux-process: • Linux Kernel • File System • Network System • PID, UID, IPC, etc.. • Memory, Disk, CPU, etc..
  • 6. Virtualization • Isolated: • FS, net, pid, uid, uts, mnt, ipc namespace • Constrained: • Memory, CPU, Network IO, Disk IO, Disk Space
  • 7. The process groups that meet the above restrictions are called "lightweight virtual machine" or Container
  • 8. Why Docker ?
  • 9. The Challenge Static website Web frontend User DB Queue Analytics DB Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 3 postgresql + pgv8 + v8 hadoop + hive + thrift + OpenJDK Ruby + Rails + sass + Unicorn Redis + redis-sentinel Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql- client DevelopmentVM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers Multiplicity  of  Stacks Multiplicity  of   hardware   environments Production Cluster Customer Data Center Do  services  and  apps   interact  appropriately? Can  I  migrate  smoothly   and  quickly?
  • 10. N x N compatibility nightmare Static website Web frontend Background workers User DB Analytics DB Queue Developm ent VM QA Server Single Prod Server Onsite Cluster Public Cloud Contribut or’s laptop Customer Servers ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  • 11. An Inspiration.. and history: Cargo Transport Pre-1960 Multiplicity  of  Goods Multipilicity  of  methods   for  transporting/storing Do  I  worry  about  how   goods  interact  (e.g.   coffee  beans  next  to   spices) Can  I  transport  quickly   and  smoothly   (e.g.  from  boat  to  train  to   truck)
  • 12. Also a matrix from hell ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  • 13. Solution: Intermodal Shipping Container Multiplicity  of  Goods Multiplicity  of  methods   for  transporting/ storing Do  I  worry  about  how   goods  interact  (e.g.   coffee  beans  next  to   spices) Can  I  transport  quickly   and  smoothly   (e.g.  from  boat  to  train   to  truck) …in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.
  • 14. http://www.gatesnotes.com/About-Bill-Gates/Best-Books-2013
  • 15. A shipping container system for code Static website Web frontendUser DB Queue Analytics DB Multiplicity  of  Stacks Multiplicity  of   hardware   environments Do  services  and  apps   interact  appropriately? Can  I  migrate  smoothly   and  quickly …that can be manipulated using standard operations and run consistently on virtually any hardware platform An engine that enables any payload to be encapsulated as a lightweight, portable, self- sufficient container…
  • 16. This eliminated the N x N problem Static website Web frontend Background workers User DB Analytics DB Queue Develop ment VM QA Server Single Prod Server Onsite Cluster Public Cloud Contribut or’s laptop Custome r Servers
  • 17. Build, Ship, Run
  • 18. – developers “build once, run anywhere”
  • 19. – operations “configure once, run anything”
  • 20. What can I use Docker for? • Continuous Integration / Deployment • Packaging and deploying applications • Build your own PAAS • Deploy applications at hyperscale
  • 21. Work with Puppet or Chef • Chef and Puppet are state management tools • Docker images are version contronlled and layered • Small, self-contained and lightweight
  • 22. standardized, interoperable, automatable
  • 23. Why not virtual machines? • Speed of deployment • Portability • Size aka cached layering FTW • Density & Performance • Cost
  • 24. Why are Docker containers lightweight? ! ! ! ! Bins/   Libs   ! ! ! App   A Original  App   ! (No  OS  to  take   up  space,  resources,   or  require  restart) App  Δ   ! ! ! ! Bins/ App   A   ! ! ! ! ! ! ! ! ! Bins/   Libs   ! ! ! App   A’   ! ! ! ! ! ! ! ! ! Guest   OS   ! ! ! ! ! ! ! Bins/   Libs   ! ! ! Modified  App   ! Copy  on  write   capabilities  allow  us  to   only  save  the  diffs   Between  container  A   and  container  A’VMs   Every  app,  every  copy  of  an   app,  and  every  slight  modification   of  the  app  requires  a  new  virtual  server App   A   ! ! ! ! ! ! ! ! ! Guest   OS   ! ! ! ! ! ! ! ! ! ! Bins/   Libs   ! ! ! Copy  of  App   ! No  OS.  Can  Share   bins/libs App   A ! ! ! ! Guest   OS   ! ! ! ! ! ! ! ! ! ! Guest   OS   ! ! ! ! ! ! VMs Containers
  • 25. VM for IAAS, Container for PAAS
  • 26. Technology Stack • Runs on most Linux distros • Boot2Docker for OSX and windows • Uses Linux Kernel features • Storage is provided by Union File systems • Container format
  • 27. Docker Architecture Linux Kernel Cgroups Namespaces Layered FS LXC (Linux Containers) Docker
  • 28. Docker Basics • Image • Container • Registry (like http://index.docker.io)
  • 29. Images • Images are the source of containers • Every container is launched from an image • Images are "built" and layered • Images are stored in a registry
  • 30. Layers • Each image is made up of layers • Like Linux the bottom layer is a root file system • Docker uses union file system mounting • Top layer is writeable and created when a container is launched
  • 31. Containers • Containers are launched from images • They contain one or more running processes • Can be started, stopped, restarted and killed
  • 32. Images vs. Containers • An images is a stopped container • You make an image by committing a container • Images are readonly, containers are writable • Images are building, containers are running
  • 33. Build (Image) Ship (by Registry) Run (Container)
  • 34. Workflow ! ! ! ! Source  Code   Repository Dockerfile   For     A Docker  Engine ! ! Docker   Container   Image   Registry Build Docker   Host  2  OS    (Linux) Container  A Container  B Container  C Container  A Push Search Pull Run Host  1    OS  (Linux)
  • 35. Changes and Updates Docker  Engine ! ! Docker   Container   Image   Registry Docker  Engine Push Update ! ! ! ! Bins/   Libs   ! ! ! App   A App  Δ   ! ! ! ! Bins/ Base     Container   Image Host  is  now  running  A’’   Container  Mod   A’’ App  Δ   ! ! ! ! Bins/ ! ! ! ! Bins/   Libs   ! ! ! App   A ! ! ! ! Bins/ ! ! ! ! Bins/   Libs   ! ! ! App   A’’ Host  running  A  wants  to  upgrade  to  A’’.  Requests   update.  Gets  only  diffs   Container  Mod   A’
  • 36. Commands • $ docker search ubuntu • $ docker pull <user>/<repo> ! • $ docker run ubuntu /bin/echo hello world ! • $ id=$(docker ps -q -l) • $ docker stop $id • $ docker start $id ! • $ docker commit $CONTAINER_ID <user>/<repo> • $ docker push <user>/<repo>
  • 37. Dockfile Automating Builds FROM ubuntu MAINTAINER YOUR_NAME YOUR_EMAIL RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get install -y memcached ENTRYPOINT ["memcached"] USER daemon EXPOSE 11211 $ docker build -t memcached - < Dockerfile $ docker push <user>/<repo>
  • 38. Resources • https://www.docker.io/ • https://github.com/dotcloud/docker • http://bit.ly/dockersources • http://lwn.net/Articles/199643/ • http://lwn.net/Articles/236038/ • http://en.wikipedia.org/wiki/Operating_system-level_virtualization • https://linuxcontainers.org/ • http://en.wikipedia.org/wiki/Cgroups • http://en.wikipedia.org/wiki/Aufs
  • 39. Qiniu meet Docker • Qiniu Cloud Storage service full-stack Go (since 2011) • Docker is also written in Go (since 2011) • Both Gopher ! • Docker’s containers is user-level, not good for disk use • Qiniu Cloud Storage is os-level, disk-based • How could they can be together?
  • 40. 第三⽅方云主机 第三⽅方云引擎or www.qiniu.com 七⽜牛云存储 七⽜牛云分发七⽜牛云处理 终端上传 静态⽂文件 ⾳音频转码 视频转码 图⽚片处理 管道链式处理 电信 联通 移动 教育⺴⽹网 海外 ⾃自定义数据处理 其他 ⽂文档转换 ⾃自有服务器or
  • 41. blue.qiniudn.com yellow.qiniudn.com green.qiniudn.com balck.qiniudn.com
  • 42. Q & A • Twitter: why404 • Wechat: why404
  • 43. THANK YOU !