Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker Ecosystem on Azure

8,514 views

Published on

Docker Ecosystem on Microsoft Azure

Published in: Technology
  • Be the first to comment

Docker Ecosystem on Azure

  1. 1. Patrick Chanezon & Tim Park Microsoft @chanezon @timpark
  2. 2. French Polyglot Server Side San Francisco Developer Relations @chanezon
  3. 3. 3 Devops TED Working Group
  4. 4. 5 Cloud Market PublicHybridPrivate IT Pros Devops DevelopersArchitects
  5. 5. History of containerization 1960’s mainframe 1990’s hardware virtualization 1990’s OS virt precursors: BSD Jails, Solaris zones 2006 Cloud IaaS 2009 platform virtualization (PaaS) 2013 Docker See @bcantrill’s deck http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production
  6. 6. 7
  7. 7. Linux Container Ecosystem
  8. 8. Containers
  9. 9. Engaging with the ecosystem
  10. 10. It looks like a VM Private process space Can run stuff as root Private network interface and IP address Custom routes, iptables rules, etc. Can mount filesystems and more
  11. 11. Faster to boot, less overhead than a VM $ time docker run ubuntu echo hello world hello world real 0m0.258s Disk usage: less than 100 kB Memory usage: less than 1.5 MB
  12. 12. Isolation using Linux kernel features namespaces  pid  mnt  net  uts  ipc  user cgroups  memory  cpu  blkio  devices
  13. 13. How does it work? Copy-on-write storage  Create a new machine instantly (Instead of copying its whole filesystem)  Storage keeps track of what has changed  Multiple storage plugins available (AUFS, device mapper, BTRFS, overlayfs, VFS)
  14. 14. Union Filesystems (AUFS, overlayfs) Copy-on-write block devices Snapshotting filesystems Provisioning Superfast Supercheap Average Cheap Fast Cheap Changing small files Superfast Supercheap Fast Costly Fast Cheap Changing large files Slow (first time) Inefficient (copy-up!) Fast Cheap Fast Cheap Diffing Superfast Slow Superfast Memory usage Efficient Inefficient (at high densities) Inefficient (but may improve) Drawbacks Random quirks AUFS not mainline Overlayfs bleeding edge Higher disk usage Great performance (except diffing) ZFS not mainline BTRFS not as nice Bottom line Ideal for PAAS, CI/CD, high density things Works everywhere, but slow and inefficient Will be great once memory usage is fixed Storage options
  15. 15. Initial goals  LXC container engine to build a PaaS  Containers as lightweight VMs (complete with syslog, cron, ssh...)  Part of a bigger puzzle (other parts: builders, load balancers...)
  16. 16. Docker now  Build, ship, and run any app, anywhere
  17. 17. Say again?  Build: package your application in a container  Ship: move that container from a machine to another  Run: execute that container (i.e. your application)  Any application: anything that runs on Linux  Anywhere: local VM, cloud instance, bare metal...
  18. 18. build
  19. 19. Dockerfiles  Recipe to build a container  Start FROM a base image  RUN commands on top of it  Easy to learn, easy to use
  20. 20. Authoring images with a Dockerfile  Minimal learning curve  Rebuilds are easy  Caching system makes rebuilds faster  Single file to define the whole environment  Single file to define the whole component
  21. 21. FROM ubuntu:14.04 RUN apt-get update RUN apt-get install -y nginx RUN echo 'Hi, I am in your container!' >/usr/share/nginx/html/index.html CMD nginx -g "daemon off;" EXPOSE 80 docker build -t jpetazzo/web . docker run -d -P jpetazzo/web
  22. 22. Docker language stacks
  23. 23. « docker build » goodness  Takes a snapshot after each step  Re-uses those snapshots in future builds  Doesn't re-run slow steps (package install...) when it is not necessary
  24. 24. ship
  25. 25. Docker Hub  docker push an image to the Hub  docker pull this image from any machine
  26. 26. Why does this matter?
  27. 27. Deploy reliably & consistently  Images are self-contained, independent from host  If it works locally, it will work on the server  With exactly the same behavior  Regardless of versions  Regardless of distros  Regardless of dependencies
  28. 28. run
  29. 29. Execution is fast and lightweight  Containers have no* overhead  Lies, damn lies, and other benchmarks: http://qiita.com/syoyo/items/bea48de8d7c6d8c73435 http://www.slideshare.net/BodenRussell/kvm-and-docker-lxc-benchmarking-with-openstack *For some definitions of « no »
  30. 30. Is there really no overhead at all?  Processes are isolated, but run straight on the host  Code path in containers = code path on native  CPU performance = native performance  Memory performance = a few % shaved off for (optional) accounting  Network and disk I/O performance = small overhead; can be reduced to zero
  31. 31. Deploy almost anywhere
  32. 32. Devops
  33. 33. Separation of concerns: Dave the Developer  Inside my container:  my code  my libraries  my package manager  my app  my data
  34. 34. Separation of concerns: Oscar the Ops guy  Outside the container:  logging  remote access  network configuration  monitoring
  35. 35. Docker Components
  36. 36. Docker: the cast  Docker Engine  Docker Hub  Docker, the community  Docker Inc, the company
  37. 37. Docker Engine  Open Source engine to commoditize LXC  Uses copy-on-write for quick provisioning  Written in Go, runs as a daemon, comes with a CLI  Everything exposed through a REST API  Allows to build images in standard, reproducible way  Allows to share images through registries  Defines standard format for containers (stack of layers; 1 layer = tarball+metadata)
  38. 38. … Open Source?  Nothing up the sleeve, everything on the table  Public GitHub repository: https://github.com/docker/docker  Bug reports: GitHub issue tracker  Mailing lists: docker-user, docker-dev (Google groups)  IRC channels: #docker, #docker-dev (Freenode)  New features: GitHub pull requests (see CONTRIBUTING.md)  Docker Governance Advisory Board (elected by contributors)
  39. 39. Docker Hub •Collection of services to make Docker more useful.  Library of official base images  Public registry (push/pull your images for free)  Private registry (push/pull secret images for $)  Automated builds (link github/bitbucket repo; trigger build on commit)  More to come!
  40. 40. Docker, the community  >700 contributors  ~20 core maintainers  >40,000 Dockerized projects on GitHub  >60,000 repositories on Docker Hub  >25000 meetup members, >140 cities, >50 countries  >2,000,000 downloads of boot2docker
  41. 41. Docker Inc, the company  Headcount: ~70  Revenue:  t-shirts and stickers featuring the cool blue whale  SAAS delivered through Docker Hub  Support & Training
  42. 42. Developing with Docker
  43. 43. One-time setup  On your dev env (Linux, OS X, Windows)  boot2docker (25 MB VM image)  Natively (if you run Linux)  On your servers (Linux)  Packages (Ubuntu, Debian, Fedora, Gentoo, Arch...)  Single binary install (Golang FTW!)  Easy provisioning on Azure, Rackspace, Digital Ocean...  Special distros: CoreOS, Project Atomic, Ubuntu Core
  44. 44. Azure deployment VMNAME=jpetazzo IMAGE=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140724-en-us-30GB USER=jpetazzo PASSWORD=1234abcdABCD@ LOCATION="West US" azure vm docker create $VMNAME $IMAGE $USER $PASSWORD -l "$LOCATION" export DOCKER_HOST=tcp://$VMNAME.cloudapp.net:4243 docker --tls version azure vm endpoint create $VMNAME 80
  45. 45. Running multiple containers
  46. 46. Fig  Run your stack with one command: fig up  Describe your stack with one file: fig.yml  Example: Python+Redis webapp
  47. 47. web: build: . command: python app.py ports: - "5000:5000" volumes: - .:/code links: - redis:redis redis: image: redis
  48. 48. Per-project setup  Write Dockerfiles  Write fig.yml file(s)  Test the result (i.e.: Make sure that « git clone ; fig up » works on new Docker machine works fine
  49. 49. Per-developer setup  Make sure that they have Docker (boot2docker or other method)  git clone ; fig up  Done
  50. 50. Development workflow  Edit code  Iterate locally or in a container (use volumes to share code between local machine and container)  When ready to test « the real thing », fig up
  51. 51. Going to production  There are many options  full 45-minutes talk about « Docker to production »
  52. 52. Implementing CI/CD  Each time I commit some code, I want to:  build a container with that code  test that container  if the test is successful, promote that container
  53. 53. Docker Hub to the rescue  Automated builds let you link github/bitbucket repositories to Docker Hub repositories  Each time you push to github/bitbucket:  Docker Hub fetches your changes,  builds new containers images,  pushes those images to the registry.
  54. 54. Coming next on Docker Hub...  Security notifications  Automated deployment to Docker hosts  Docker Hub Enterprise (all those features, on your infrastructure)
  55. 55. Summary •With Docker, I can:  put my software in containers  run those containers anywhere  write recipes to automatically build containers  use Fig to effortlessly start stacks of containers  automate testing, building, hosting of images, using the Docker Hub
  56. 56. Advanced concepts  naming  give a unique name to your containers  links  connect containers together  volumes  separate code and data  share data between containers
  57. 57. Docker future • Docker machine: provisioning hosts • Docker swarm: clustering • Docker compose: composing apps • Docker plugin model: for weave, flocker
  58. 58. Linux Container Ecosystem
  59. 59. Weave
  60. 60. Flocker
  61. 61. CoreOS
  62. 62. Fleet
  63. 63. Docker & etcd
  64. 64. Cluster Architecture
  65. 65. Deis (http://deis.io) • Open source PaaS platform that builds on CoreOS. • Replicates the popular Heroku devops workflow. • Primary mechanism for pushing applications is through git. • Developer experience is not unlike Azure Websites… • …but is built on Linux so full support for open source stacks. • Enables us to win migrations from Salesforce to Azure. • Hackfest in November to enable Deis for Tagboard. • Enables us to win startups that expect this workflow.
  66. 66. tpark:www$ git push deis master • Git pushes master to deis git remote on endpoint • Deis senses static web application • Selects Heroku Buildpack • Uses buildpack to build application Docker container. • Pushes this container to a private Docker registry. • Orchestrates the creation or update of this container on the cluster. • Updates routing mesh to route to these containers.
  67. 67. Router Mesh deis-1 deis-2 deis-3 deis-4 www CoreOS CoreOS CoreOS CoreOS
  68. 68. tpark:www$ deis scale www=3 • Deis pushes the container to two more cluster nodes. • Updates routing mesh to pass traffic to these nodes.
  69. 69. Router Mesh deis-1 deis-2 deis-3 deis-4 www www www
  70. 70. tpark:api$ git push deis master • Git pushes master to deis git remote on endpoint • Deis senses node.js application • Selects Heroku node.js Buildpack • Uses buildpack to build application Docker container. • Pushes this container to a private Docker registry. • Orchestrates the creation or update of this container on the cluster. • Updates routing mesh to route to these containers.
  71. 71. Router Mesh deis-1 deis-2 deis-3 deis-4 www api www api www api
  72. 72. Router Mesh deis-1 deis-2 deis-3 deis-4 www api www api www api
  73. 73. Router Mesh deis-1 deis-2 deis-3 deis-4 www api www api www api
  74. 74. tpark:api$ deis config:set DATABASE_URL=postgres://user:pass@example.com:54 32/db • Applications in Deis are configured through environmental variables. • MUST READ: http://12factor.net/ • Key point: Code is separated from config. • Enables generic containers that are configured at runtime. • Every app container spun up by Deis will have a copy of these config environmental variables.
  75. 75. tpark:api$ deis logs • Deis automatically rolls and consolidates logs from all containers.
  76. 76. Router Mesh deis-1 deis-2 deis-3 deis-4 www api www api www api
  77. 77. Router Mesh deis-1 deis-2 deis-3 deis-4 www api www api www api
  78. 78. Kubernetes (http://kubernetes.io)
  79. 79. Kubernetes Master / Scheduler host-1 host-2 host-3 host-n ….. Container Agent Container Agent Container Agent Container Agent Linux Linux Linux Linux
  80. 80. Kubernetes Scheduler host-1 host-2 host-3 host-n ….. Container Agent Container Agent Container Agent Container Agent Linux Linux Linux Linux Container Container
  81. 81. Kubernetes host-1 Container host-2 host-3 host-4 host-n … Container Container Container Container ContainerContainer Container Container
  82. 82. Kubernetes host-1 host-2 host-3 host-4 host-n … Frontend Worker my_app pod MyAppMyApp MyApp Replication Controller 3
  83. 83. Kubernetes host-1 host-2 host-3 host-4 host-n … Frontend Worker my_app pod MyAppMyApp MyApp Replication Controller 3
  84. 84. Kubernetes host-1 host-2 host-3 host-4 host-n … MyAppMyApp MyApp Replication Controller Pod Pod Pod Pod PodPod Pod Pod Replication Controller
  85. 85. Kubernetes host-1 host-2 host-3 host-4 host-n … MyApp staging MyApp staging MyApp staging MyApp prod MyApp prod MyApp prod MyApp prod MyApp prod MyApp Production Service { environment: prod } MyApp Staging Service { environment: staging } Labels and Services
  86. 86. • https://github.com/chanezon/azure-linux • Docker container to get started docker run –ti chanezon/linux • CoreOS cluster, fleet • Deis • Weave • Docker machine • Deploy Java app
  87. 87. • Strategic Engagement team • We build Epic stuff with customers • What can we do together? • When do you want to start? • patric@microsoft.com
  88. 88. 10 3 http://www.slideshare.net/chanezon/tackling-complexity-in-giant-systems-approaches-from-several-cloud- providers https://msopentech.com/ https://github.com/timfpark/coreos-azure http://www.slideshare.net/jpetazzo/ http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production

×