Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PuppetConf 2016: The Challenges with Container Configuration – David Lutterkort, Puppet

233 views

Published on

Here are the slides from David Lutterkort's PuppetConf 2016 presentation called The Challenges with Container Configuration. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PuppetConf 2016: The Challenges with Container Configuration – David Lutterkort, Puppet

  1. 1. The challenges of container configuration David Lutterkort @lutterkort lutter@puppet.com
  2. 2. Overview ● What is configuration ? ● Immutability ● Build vs Run ● Who configures the scheduler ? ● Conclusions 3
  3. 3. What is configuration ?
  4. 4. package/file/service is only one instance of a more general problem 5
  5. 5. Configuration is any input into infrastructure It needs to be managed over time and at scale 6
  6. 6. Core configuration management features: ❏ describe system aspects in isolation ❏ combine aspects into whole ❏ common format for querying ❏ bridge across entire infrastructure 7
  7. 7. $ docker run -d -e MYSQL_HOST=mysql.example.com -e MYSQL_PORT=3306 --health-cmd /usr/bin/check webapp
  8. 8. Immutability
  9. 9. $ docker run --name example fedora:24 /bin/sh -c ‘while true; do cat /etc/system-release; sleep 1; done’
  10. 10. $ docker run … Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four)
  11. 11. $ docker exec example /bin/sh -c ‘sed -i -e s/24/25/ /etc/system-release’
  12. 12. Fedora release 24 (Twenty Four) Fedora release 24 (Twenty Four) Fedora release 25 (Twenty Four) Fedora release 25 (Twenty Four) Fedora release 25 (Twenty Four) Fedora release 25 (Twenty Four) Fedora release 25 (Twenty Four) Fedora release 25 (Twenty Four) $ docker exec …
  13. 13. $ docker diff example C /run A /run/secrets C /etc C /etc/system-release
  14. 14. Containers are not immutable by default Only as immutable as packages 15
  15. 15. $ docker run --read-only --name example fedora:24 /bin/sh -c ‘while true; do cat /etc/system-release; sleep 1; done’
  16. 16. $ docker exec example /bin/sh -c ‘sed -i -e s/24/25/ /etc/system-release’ sed: couldn't open temporary file /etc/sed5OCs5t: Read-only file system
  17. 17. $ docker diff example C /run A /run/secrets
  18. 18. Suggestion Enable --read-only whenever possible 19
  19. 19. require 'rubygems' require 'sinatra' require 'haml' # Handle GET-request (Show the upload form) get "/upload" do haml :upload end # Handle POST-request (Receive and save the uploaded file) post "/upload" do File.open('uploads/' + params['myfile'][:filename], "w") do |f| f.write(params['myfile'][:tempfile].read) end return "The file was successfully uploaded!" end
  20. 20. $ docker run -d --read-only lutter/lolcat
  21. 21. require 'rubygems' require 'sinatra' require 'haml' # Handle GET-request (Show the upload form) get "/upload" do haml :upload end # Handle POST-request (Receive and save the uploaded file) post "/upload" do File.open('uploads/' + params['myfile'][:filename], "w") do |f| f.write(params['myfile'][:tempfile].read) end return "The file was successfully uploaded!" end
  22. 22. $ docker run -d --read-only -v /srv/lolcat/uploads:/app/uploads lutter/lolcat
  23. 23. require 'rubygems' require 'sinatra' require 'haml' # Handle GET-request (Show the upload form) get "/upload" do haml :upload end # Handle POST-request (Receive and save the uploaded file) post "/upload" do File.open('uploads/' + params['myfile'][:filename], "w") do |f| f.write(params['myfile'][:tempfile].read) end return "The file was successfully uploaded!" end
  24. 24. $ docker run -d --read-only -v /srv/lolcat/uploads:/app/uploads --tmpfs /tmp lutter/lolcat
  25. 25. Suggestion Use --tmpfs where needed 26
  26. 26. Without technical controls you only have social guarantees of immutability 27
  27. 27. How do you know the correct invocation for an image ? 28
  28. 28. Build vs Run
  29. 29. Given an image ❏ What machine built this image ? ❏ How do you run this image ? ❏ Who supports this image ? ❏ Does the image contain malware ? 30
  30. 30. Given a container ❏ Who built it ? ❏ How was it built ? ❏ What software does it contain ? ❏ Is the software up-to-date ? 31
  31. 31. FROM fedora:24 RUN dnf update -y && dnf install -y ruby rubygem-bundler && dnf clean all COPY . /app RUN cd /app && bundle install --path vendor/bundle WORKDIR /app VOLUME /app/uploads EXPOSE 9292 CMD ["/usr/bin/bundle", "exec", "rackup"]
  32. 32. FROM fedora:24 RUN dnf update -y && dnf install -y ruby rubygem-bundler && dnf clean all COPY . /app RUN cd /app && bundle install --path vendor/bundle WORKDIR /app VOLUME /app/uploads EXPOSE 9292 CMD ["/usr/bin/bundle", "exec", "rackup"] Where did the base image come from ?
  33. 33. FROM fedora:24 RUN dnf update -y && dnf install -y ruby rubygem-bundler && dnf clean all COPY . /app RUN cd /app && bundle install --path vendor/bundle WORKDIR /app VOLUME /app/uploads EXPOSE 9292 CMD ["/usr/bin/bundle", "exec", "rackup"] What repositories and what package versions ?
  34. 34. FROM fedora:24 RUN dnf update -y && dnf install -y ruby rubygem-bundler && dnf clean all COPY . /app RUN cd /app && bundle install --path vendor/bundle WORKDIR /app VOLUME /app/uploads EXPOSE 9292 CMD ["/usr/bin/bundle", "exec", "rackup"] What was in this directory at build time ?
  35. 35. Time is your enemy 36
  36. 36. When do you rebuild images ? 37
  37. 37. Code changes and external factors should trigger rebuild 38
  38. 38. Explain yourself with metadata Docker labels are a great way to do that 39
  39. 39. Name : glibc Version : 2.23.1 Release : 10.fc24 Architecture: x86_64 License : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+ Signature : RSA/SHA256, Thu 18 Aug 2016 09:27:43 AM PDT, Key ID 73bde98381b46521 Source RPM : glibc-2.23.1-10.fc24.src.rpm Build Date : Thu 18 Aug 2016 06:37:42 AM PDT Build Host : buildvm-16.phx2.fedoraproject.org Packager : Fedora Project Vendor : Fedora Project Summary : The GNU libc libraries
  40. 40. $ docker inspect -f "{{json .Config.Volumes}}" lutter/lolcat { "/app/uploads": {} }
  41. 41. $ docker inspect -f "{{json .Config.ExposedPorts}}" lutter/lolcat { "9292/tcp": {} }
  42. 42. LABEL vendor=”ACME Incorporated” com.acme.release-status=”beta” com.acme.version=”0.1.0-beta” com.acme.git.sha=”f260653a”
  43. 43. $ docker inspect -f "{{json .Config.Labels}}" lutter/lolcat | jq { "com.acme.git.sha": "f260653a", "com.acme.release-status": "beta", "com.acme.version": "0.1.0-beta", "vendor": "ACME Incorporated" }
  44. 44. Suggestion Decide upon and enforce metadata standards 45
  45. 45. LABEL com.acme.dockerfile=”/Dockerfile”
  46. 46. $ docker inspect -f "{{json .Config.Labels}}" lutter/alpine | jq { "com.example.dockerfile": "/Dockerfile" }
  47. 47. $ docker run -it lutter/alpine cat /Dockerfile FROM alpine RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile / LABEL com.example.dockerfile="/Dockerfile"
  48. 48. Suggestion Embed your Dockerfile in the image 49
  49. 49. LABEL com.acme.cmd.packages=”apk info -vv”
  50. 50. $ docker run -it lutter/alpine apk info -vv musl-1.1.14-r12 - the musl c library (libc) busybox-1.24.2-r11 - Size optimized toolbox of ... alpine-baselayout-3.0.3-r0 - Alpine base dir ... alpine-keys-1.1-r0 - Public keys for Alpine Linux ... zlib-1.2.8-r2 - A compression/decompression Library bash-4.3.42-r3 - The GNU Bourne Again shell ...
  51. 51. Suggestion Make your images discoverable 52
  52. 52. puppetlabs/puppetlabs-image_build
  53. 53. class { 'nginx': } nginx::resource::vhost { 'default': www_root => '/var/www/html', } file { '/var/www/html/index.html': ensure => present, content => 'Hello Puppet and Docker', } exec { 'Disable Nginx daemon mode': path => '/bin', command => 'echo "daemon off;" >> /etc/nginx/nginx.conf', unless => 'grep "daemon off" /etc/nginx/nginx.conf', }
  54. 54. # metadata.yaml cmd: nginx expose: 80 image_name: puppet/nginx
  55. 55. $ puppet docker build ... $ docker run -d -p 8080:80 acme/nginx-test 83d5fbe370e84d424c71c1c038ad1f5892fec579d28b... $ curl http://127.0.0.1:8080 Hello Puppet and Docker
  56. 56. Who configures the scheduler ?
  57. 57. Schedulers/orchestrators isolate you from ❏ where individual containers run ❏ balancing due to new resources ❏ respawning due to failed resources 58
  58. 58. Schedulers operate on constraints 59
  59. 59. Decisions depend on accurate resource information 60
  60. 60. $ docker daemon --label environment=production --label storage=ssd
  61. 61. $ docker run -d -P --label com.example.environment=production -e constraint:storage==ssd --name db mysql
  62. 62. template: metadata: labels: app: guestbook tier: frontend spec: containers: - name: php-redis image: gcr.io/google-samples/gb-frontend:v4 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns # If your cluster config does not include a dns service, then to # instead access environment variables to find service host # info, comment out the 'value: dns' line above, and uncomment the # line below. # value: env ports: - containerPort: 80
  63. 63. How do you manage properties for all your hosts ? 64
  64. 64. Suggestion Compute host properties dynamically 65
  65. 65. $ facter -y | head -n 20 aio_agent_version: 1.7.0 augeas: version: 1.4.0 disks: sda: model: SanDisk SDSSDA24 size: 223.57 GiB size_bytes: 240057409536 vendor: ATA ... dmi: bios: ... memory: ...
  66. 66. $ docker daemon --label os=$(facter os.family) --label kernel=$(facter kernelversion) --label memory=$(facter memory.system.total_bytes)
  67. 67. https://forge.puppet.com/puppetlabs/docker_platform
  68. 68. class { 'docker': labels => [ "os=${facts[os][family]", "kernel=${facts[kernelversion]}", "memory=${facts[memory][system][total_bytes]}" ], }
  69. 69. Schedulers introduce higher-level primitives 70
  70. 70. Docker networks Kubernetes services and replication controllers Chronos jobs 71
  71. 71. Many interfaces imperative not declarative 72
  72. 72. $ kubectl get pod mypod -o yaml | sed -e ‘s/(image:myimage):.*$/1:v4/’ | kubectl replace -f -
  73. 73. $ docker network create bob ca7b185775966003d38ccbd9bba822fb570766e4bb $ docker network create bob Error response from daemon: network with name bob ...
  74. 74. docker_network { 'bob': ensure => present, driver => 'overlay', subnet => '192.168.1.0/24', gateway => '192.168.1.1', ip_range => '192.168.1.4/32', }
  75. 75. And everything is in YAML 76
  76. 76. “ The language to represent the data should be a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language, where there are well-understood semantics, as well as good tooling. Borg, Omega, and Kubernetes, ACM Queue, Volume 14 Issue 1 | http://queue.acm.org/detail.cfm?id=2898444 77
  77. 77. Code plus data has advantages over data alone 78
  78. 78. https://forge.puppet.com/garethr/kubernete s
  79. 79. kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }
  80. 80. controller_service_pair { 'redis-master': app => 'redis', role => 'master', tier => 'backend', port => 6379, }
  81. 81. Conclusions
  82. 82. The difference between how you think a system behaves and how it actually behaves risks hard-to-debug production issues 83
  83. 83. Container use at scale and over time requires meaningful abstraction 84
  84. 84. Configuration management as a discipline provides tools to build those abstractions and thereby minimize risk 85
  85. 85. 86 Project Blueshift booth Exhibition Hall Docker, Mesos, Kubernetes and Puppet? Don't Panic ! Deepak Giridharagopal, Thur, 4:45pm Pulling the strings to containerize your life Scott Coulton, Fri, 9:50am Running Puppet software in Docker containers Gareth Rushgrove, Fri, 1:30pm

×