Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker and Puppet for Continuous Integration


Published on

Today developers want to change the code, build and deploy often, even several times per day.
New versions of software may need to be tested on different distributions, and with different configurations.
Achieving this with Virtual Machines it’s possible, but it’s very resource and time consuming. Docker provides an incredibly good solution for this, in particular if combined with Continuous Integration tools like Jenkins and Configuration Management tools like Puppet.
This presentation focuses on the opportunities to configure automatically Docker images, use Docker containers as disposable workers during your tests, and even running your Continuous Integration system inside Docker.

Published in: Software

Docker and Puppet for Continuous Integration

  1. 1. Docker and Puppet for Continuous Integration Giacomo Vacca, Founder at RTCSoft,
  2. 2. About me • 15 years “in the trenches cubicles” • Developer of RTC (VoIP, IM, WebRTC) solutions • Founder of RTCSoft in 2015 @giavac
  3. 3. Docker usage scenarios • Run services • Packaging/Deployment mechanism • Fast prototyping • Continuous Integration/ Delivery
  4. 4. Continuous Integration 1/2 • A team of developers work on the same project • Each developer commits several times per day • Each commit is potentially verified (Build, Deploy, Test) • The project is always kept in a stable/releasable state
  5. 5. Continuous Integration 2/2 • Developers work on personal/bug/feature (short-lived) branches • These branches are merged into the “release” branch for the release Image source:
  6. 6. CI components Image source: myself (This is why it sucks)
  7. 7. Server-side apps for Linux • Apps are built, packaged, deployed and tested • Unit testing during the builds • Component/Integration/Load testing after test deployments • Artefacts are uploaded into file servers or deb/yum repos
  8. 8. Old school: “Build and run” Build the app on the runtime machine • PROBLEMS: • Repeat the build on each host • Unnecessary pollution of the runtime host • Maintenance nightmare: the easiest upgrade is problematic • Potential exposure of source code to external attacks
  9. 9. Using a “Build machine” Dedicate a machine to the various builds • PROs: • Keep the runtime environment reasonably clean • Source code is protected inside the private network • CONs: • You’re reducing a multi-dimensional problem into a bi-dimensional one • You’ll pay the price one day!
  10. 10. • Widely adopted, Open Source CI tool • Core + many plugins • Integrates well with git and Docker • Can manage multiple “slaves”
  11. 11. Jenkins and Master worker • “Cron jobs manager with a GUI” • What’s a Jenkins job? • The simplest case: Jenkins runs the jobs inside its own host • “Master” worker, no slaves involved • Still you need to manage Jenkins’ configuration…
  12. 12. Building machineS Assign a VM per building scenario (app/OS) • A step in the right direction • CONs: • Hard to maintain (see Configuration Management later) • VM resources allocated even when the builds are not running • Explosion of number of VM types to use (OS/versions matrix) • VMs are “heavy”
  13. 13. Jenkins and Slaves • Configure a number of “slave workers” • Assign a slave to a job • But you need to maintain all those slaves…
  14. 14. Configuration Management You need something better than that.
  15. 15. Configuration Management intro • Define programmatically the configuration of a machine • Many tools available: Puppet, Ansible, Chef, Salt, etc. • Make configuration predictable AND fix divergence automatically • Track changes during time (it’s all text) • Define a formal language across developers, sysadmin, Ops
  16. 16. • Puppet defines the final state of a host (What, not How) • Has its own declarative syntax • Written in Ruby • Designed for Master-Slave interaction, can be used Standalone • Idempotent
  17. 17. Configuration Management You really, really need it. Even if you like 3-ways diffs and hate the term “idempotence”.
  18. 18. Example of Puppet codepackage { 'nginx': ensure => present, } -> file { '/etc/nginx/nginx.conf': ensure => file, owner => 'root', group => 'root', mode => '0644', content => template('mymodule/nginx.conf.erb'), notify => Service['nginx'], } -> file { '/etc/nginx/conf.d/global.conf': ensure => file, owner => 'root', group => 'root', mode => '0644', content => template('mymodule/nginx_global.conf.erb'), notify => Service['nginx'], } service { 'nginx': ensure => running, enable => true, }
  19. 19. Back to Jenkins and multiple Slaves • We saw the approach with multiple slaves, with VMs as slave • Functionally valid • Cumbersome to maintain • What if we could have “on demand”, lightweight workers? e.g. Docker containers?
  20. 20. • A new form of virtualisation (w/ real HW performance) • A delivery mechanism • Many “images” freely available • “Here’s the ingredients and recipe” VS “Here’s the cake”
  21. 21. Virtual Machines vs Docker Source:
  22. 22. Docker workflow • Build an image (Dockerfile + base images) • docker build -t gvacca/nginx . • Store image in a registry (Public, e.g. Dockerhub or private) • docker push gvacca/nginx • Pull image on a target host • docker pull gvacca/nginx • Run container on a target host • docker run -it gvacca/nginx
  23. 23. Dockerfile - example FROM ubuntu:latest MAINTAINER Giacomo Vacca <> RUN apt-get update # Install nginx RUN apt-get -y -q install nginx # Prepare target dir for website (Must match global.conf) RUN mkdir -p /var/www/html # Configure nginx COPY nginx/nginx.conf /etc/nginx/nginx.conf COPY nginx/global.conf /etc/nginx/conf.d/global.conf EXPOSE 8080 CMD [ "/usr/sbin/nginx" ]
  24. 24. Build a Docker image gvacca@dockerubuntu:~simple_nginx$ sudo docker build -t gvacca/simple-nginx . Sending build context to Docker daemon 7.168 kB Step 1 : FROM ubuntu:14.04 ---> 1d073211c498 … Step 2 : MAINTAINER Giacomo Vacca <> ---> Running in 10a8334becfd ---> 559310abf4fb Removing intermediate container 10a8334becfd Step 3 : RUN apt-get update ---> Running in 873b09debab0 … Step 8 : EXPOSE 8080 ---> Running in a27f73413c02 ---> f5ef8527f2a3 Removing intermediate container a27f73413c02 Step 9 : CMD /usr/sbin/nginx ---> Running in d99de19cc782 ---> df76d20cd00f Removing intermediate container d99de19cc782 Successfully built df76d20cd00f
  25. 25. Run the container! gvacca@dockerubuntu:simple_nginx$ sudo docker run -d --name nginx -p 8080:8080 -v $PWD/website:/var/www/html/website gvacca/simple-nginx add8d371f82f615ebbd121cea804511dcdafac893b14325366744797424fa44c gvacca@dockerubuntu:simple_nginx$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES add8d371f82f gvacca/simple-nginx "/usr/sbin/nginx" 11 seconds ago Up 9 seconds>8080/tcp nginx gvacca@dockerubuntu:simple_nginx$ ps aux |grep docker root 792 0.0 4.7 693136 23740 ? Ssl Nov04 11:20 /usr/bin/docker daemon root 1977 0.0 2.6 152692 13172 ? Sl 11:26 0:00 docker-proxy - proto tcp -host-ip -host-port 8080 -container-ip -container-port 8080
  26. 26. Intermediate topics • Volumes • Network (was ‘link’ before 1.9) • Debugging (‘inspect’) • ARG (after 1.9) • Docker on OSX/Windows (Docker Machine) • Interacting with Kernel modules
  27. 27. Docker orchestration • Docker Composer • Docker Swarm • Google’s Kubernetes • Apache Mesos • Puppet & others
  28. 28. Let’s start exploring interaction opportunities between the tools seen so far…
  29. 29. Puppet to manage Jenkins • • Manage Jenkins as a service • Install plugins automatically, etc • Or build your own module! • Install .war • Configure Jenkins (config.xml) • Configure jobs ($JENKINS_PATH/jobs/JOB/config.xml)
  30. 30. Docker as Slave for Jenkins • Associate specific Docker images to specific Jenkins build jobs • Keep builds clean & reproducible (“Non-event releases”) • Run slave containers on demand - stop them when not needed. • Jenkins Plugin: +Plugin • Other Cloud methods, e.g. Mesos
  31. 31. Jenkins to build Docker images • A Jenkins job: • Checks out a Dockerfile • Builds the image • Upload the new image in a repo • May re-use images for new builds (careful about integrity)
  32. 32. Puppet to manage Docker containers • Module for docker: • Installs Docker and required dependencies • Launches the Docker daemon • Sets DNS, users and other configuration items • Pull images from selected repos • Run containers (image + command)
  33. 33. Run Jenkins inside Docker • Manage Jenkins with a dedicated image + volume with data • From the official Docker registry: docker run -d -p 8080:8080 -p 50000:50000 -v ~:/var/jenkins_home jenkins • TCP 8080: Web UI • TCP 50000: interface to connect slaves
  34. 34. Docker inside Docker But you need the privileged mode
  35. 35. Conclusion • There are many opportunities to automate Continuous Integration • CI for server-side apps poses specific challenges • You can’t ignore Configuration Management • Docker represents a huge opportunity • You need to find your own use case and solution
  36. 36. APPENDIX - Puppet vs Docker • Some stop using Puppet for Configuration Management once they move their apps inside containers • A Dockerfile defines “how” to build an image • A Dockerfile must be distribution-specific (yum or apt?) • Puppet is OS-agnostic: the differences are hidden inside modules
  37. 37. Questions and Answers Thanks
  38. 38. Recommended Books • “The Docker book”, J. Turnbull, Book-Containerization-new-virtualization-ebook/dp/B00LRROTI4 • “Continuous Delivery”, J. Humble, Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/ 0321601912 • “Pro Puppet”, J. Turnbull, Second-Professional-Apress/dp/1430260408
  39. 39. See also… revolution-of-webrtc