Downloaded 75 times
![Building these containers makes you sad
29
agent 0.6.0-dev 35634607e29d 4.252 GB
IMAGE CREATED BY SIZE
35634607e29d /bin/sh -c #(nop) ENTRYPOINT [/bin/sh -c /hom 0 B
465a5f6c26cf /bin/sh -c #(nop) USER [root] 0 B
c5ff91f2b32c /bin/sh -c mkdir /home/weeby/projects 0 B
fc032221787f /bin/sh -c #(nop) USER [weeby] 0 B
ef78532c29c3 /bin/sh -c #(nop) ADD file:abd8470a4e17591d90 2.502 kB
46492ac76bd7 /bin/sh -c #(nop) ADD file:a5adf541d11457b216 113 B
d6427b5956f9 /bin/sh -c mkdir /usr/local/var && chmod 0 B
0f7bfc28bbf1 /bin/sh -c cd /home/weeby/dev/weebyco_cloud/a 310.2 kB
898a4341879c /bin/sh -c cd /home/weeby/dev/weebyco_cloud/c 587.5 kB
7a6065e92b0f /bin/sh -c #(nop) USER [root] 0 B
cd1f2caaa012 /bin/sh -c echo 'cd $HOME/dev && git clon 25.72 MB
2c88826fc9eb /bin/sh -c echo 'cd $HOME/.config/devkit/cach 961.4 MB
25439817982f /bin/sh -c echo 'cd $HOME/.config/devkit/cach 168.1 MB
4556160eddef /bin/sh -c echo 'cd $HOME/dev && git clon 120.4 MB
a08d25dd66ae /bin/sh -c echo 'cd $HOME/dev && git clon 98.14 MB
a035740ba00e /bin/sh -c echo 'cd $HOME/dev && git clon 47.29 MB
0c62eb7db0d2 /bin/sh -c echo 'cd $HOME/dev && git clon 3.135 MB
33c95b02dfa9 /bin/sh -c echo 'cd $HOME/dev/cloud9/package 127.9 MB
763729fa92a4 /bin/sh -c #(nop) USER [weeby] 0 B
2e78c2b40604 /bin/sh -c ln -s /home/weeby/dev/cloud9/bin/c 36 B
6a884b1375f3 /bin/sh -c cd /home/weeby/dev/cloud9 && w 25.04 MB
28c5729feee9 /bin/sh -c #(nop) USER [root] 0 B
80d0e886412a /bin/sh -c cd $HOME/dev/cloud9 0 B
05d56e545a3c /bin/sh -c git clone https://github.com/ajaxo 139.5 MB
0ab202522723 /bin/sh -c echo "y" | android update sdk --no 75.4 MB
ab31fb1a51bb /bin/sh -c wget -qO- https://raw.githubuserco 20.36 MB
4feec10a13dc /bin/sh -c #(nop) USER [weeby] 0 B
8c005817dbcc /bin/sh -c chown -R weeby:weeby /usr/local/li 0 B
07864b99174c /bin/sh -c mkdir /usr/local/lib/node_modules 0 B
04c8e6de638a /bin/sh -c git config --global user.name "wee 64 B
84da5ff19c9b /bin/sh -c git config --global user.email "us 36 B
78935503e47e /bin/sh -c echo "Host github.comntStrictHos 43 B
312ff96ab276 /bin/sh -c chmod 400 /root/.ssh/id_rsa 1.675 kB
7b2614d72c1a /bin/sh -c #(nop) ADD file:38f735b4fb5a6b4471 404 B
d955c233c352 /bin/sh -c #(nop) ADD file:e2e0b14867bf7dbe94 1.675 kB
a559d491dd97 /bin/sh -c mkdir /root/.ssh 0 B
1e18f03f0089 /bin/sh -c echo "Host github.comntStrictHos 2.122 kB
eea093d89545 /bin/sh -c #(nop) ADD file:38f735b4fb5a6b4471 404 B
a1327152885a /bin/sh -c #(nop) ADD file:e2e0b14867bf7dbe94 1.675 kB
99ef3790f6c1 /bin/sh -c #(nop) USER [root] 0 B
31f5f6c9a6dc /bin/sh -c #(nop) ENV JAVA_HOME=/usr/lib/jvm/ 0 B
b613d6e86ae0 /bin/sh -c #(nop) ENV HOME=/home/weeby 0 B
98e4522b2296 /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B
ae86930cd9db /bin/sh -c #(nop) ENV ANT_HOME=/usr/local/apa 0 B
c8309a6f63d0 /bin/sh -c #(nop) ENV NVM_DIR=/home/weeby/.nv 0 B
35009f9ab773 /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B
feb64f27edf5 /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B
167629594dee /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B
0b3c53da27ff /bin/sh -c #(nop) ENV ANDROID_HOME=/usr/local 0 B
d4ddbd91b703 /bin/sh -c mkdir /home/weeby/.ssh/ && mkd 0 B
46312a881da6 /bin/sh -c cd /home/weeby && wget -q http 37.58 MB
a64be5f192ac /bin/sh -c cd /home/weeby && wget -q http 1.375 GB
f806641c7f11 /bin/sh -c cd /home/weeby && wget -q http 172.5 MB
67aee57af54c /bin/sh -c #(nop) USER [weeby] 0 B
c6b9bec31057 /bin/sh -c mkdir /etc/ssh/weeby && touch 2.885 kB
8d74e04926ed /bin/sh -c mkdir /usr/local/android-sdk && 0 B
3ba127c0a460 /bin/sh -c addgroup weeby --gid=1001 && a 334.5 kB
718bde0aab9c /bin/sh -c pip install WTForms watchdog uWSGI 9.972 MB
4698ca611967 /bin/sh -c apt-get update && apt-get inst 572.6 MB
ddeaf9c76769 /bin/sh -c dpkg --add-architecture i386 11 B
45bdcf534dd2 /bin/sh -c add-apt-repository ppa:webupd8team 776.1 kB
a006895ec3ac /bin/sh -c apt-get update && apt-get -y i 61.53 MB
f8ada74e3af2 /bin/sh -c echo "debconf shared/accepted-orac 2.797 MB
14a199c8ef06 /bin/sh -c #(nop) ENV DEBIAN_FRONTEND=noninte 0 B
ba6abd860f82 /bin/sh -c #(nop) MAINTAINER Martin Hunt <mar 0 B
96864a7d2df3 /bin/sh -c #(nop) CMD [/bin/bash] 0 B
809ed259f845 /bin/sh -c apt-get update && apt-get dist-upg 12.39 MB
9387bcc9826e /bin/sh -c sed -i 's/^#s*(deb.*universe)$/ 1.895 kB
897578f527ae /bin/sh -c rm -rf /var/lib/apt/lists/* 0 B
c1f3bdbd8355 /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic 194.5 kB
bfb8b5a2ad34 /bin/sh -c #(nop) ADD file:a889e7d86acdbac15e 192.5 MB
511136ea3c5a 0 B](https://image.slidesharecdn.com/dockerconweeby16x9r4-141210191840-conversion-gate01/75/Scaling-Development-Environments-with-Docker-29-2048.jpg)
![What our team looks like
32
[Insert witty picture of Martin and I here]](https://image.slidesharecdn.com/dockerconweeby16x9r4-141210191840-conversion-gate01/75/Scaling-Development-Environments-with-Docker-32-2048.jpg)
Uploaded byDocker, Inc.
Scaling Development Environments with Docker
This document discusses using Docker to create a scalable development environment. It outlines setting up containers for different development components like the build server, production servers, and tools. Templates are used to configure container dependencies and build processes. The goal is allowing developers to run all components locally for testing and to reproduce the production environment.
More Related Content
![[OpenInfra Days Korea 2018] Day 2 - E5-1: "Invited Talk: Kubicorn - Building ...](https://cdn.slidesharecdn.com/ss_thumbnails/e60955buildingsimplekubernetes-180705043459-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Scaling Development Environments with Docker
![[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview](https://cdn.slidesharecdn.com/ss_thumbnails/deis1-141107133247-conversion-gate02-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Agentic AI Architecture: Redefining Syste...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-agenticaiarchitectureredefiningsystemcommunication-251124030838-e6c70cc2-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-themodernstackbuildingwebaiapplicationswithserverless-251124030844-388cf04f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-digitalaccessibilitywhydevelopersneedtoknowandcarein2025-251127011019-0674441d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...](https://cdn.slidesharecdn.com/ss_thumbnails/md-craftingimmersiveuiwithe2eandagslshaderveronicaputrianggraini-251124030840-0c677f44-thumbnail.jpg?width=640&height=640&fit=bounds)
Scaling Development Environments with Docker
- 1. Scaling Development Environments With Docker 1
- 2. Title Text Dockerception Joe Brown Software Engineer December 4th, 2014 1. From 0 to Docker, on your team 2. Managing container dependencies
- 3. Make mobile gamesin JS, run them just as well as a native app, and make them social with a few clicks. • Rapid feedback of in browser development • Develop once, deploy many times • Make it social in a matter of minutes 3
- 4. This is complicated. 4 OS X Java 7 Xcode NodeJS DevKit Weeby HomeBrew Android SDK Android ANT NDK
- 5. This is complicated. 5 OS X Java 7 Xcode NodeJS DevKit Weeby HomeBrew Android SDK Android ANT NDK
- 6. Virtualization is great 6 Zero installation is easier than any installation
- 7. Virtualization alone doesnot mean reproducibility 7
- 8.
- 9. Ports do notgrow on trees 9
- 10. Reverse proxies area thing. NGINX is a thing. 10
- 11. A way tomanage our system 11
- 12. A way toauthenticate 12
- 13.
- 14. Docker is simple,again. 14
- 15.
- 16. Developing this iscomplicated 16 Production Servers • Docker Registry • Build Server • GAE Game Server • GAE Authentication and Management (Nimbus) • Nginx Router • Manager • Agent (per user)
- 17. Run everything locally,develop your entire stack, with no repercussions of failure 17
- 18.
- 19. Whole picture 19 The beginning of a beautiful development environment
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. Mounted files fordevelopment mode 25
- 26. Running this ishard. Building containers from this is hard. Common docker build directory • Dockerfile • launch.sh • build_container.sh • run_container.sh • etc… 26 Template the Dockerfile • Dockerfile_base.template • Dockerfile_patch.template What about shared dependencies between containers?
- 27. What templates looklike Dockerfile_base.template Dockerfile_patch.template
- 28. How do weconfigure the templates Container Dependencies - Agent 28 • build.sh Builds the specified container, based on the dependencies • tag-and-upload.sh Tag and upload a container build to a docker registry • git-tag.sh Tag the git repository, and wait for the new tag to be built by the build server • Agent (Template) • Manager (Template) • Manager-proxy (Standard) Tools Involved Containers
- 29. Building these containersmakes you sad 29 agent 0.6.0-dev 35634607e29d 4.252 GB IMAGE CREATED BY SIZE 35634607e29d /bin/sh -c #(nop) ENTRYPOINT [/bin/sh -c /hom 0 B 465a5f6c26cf /bin/sh -c #(nop) USER [root] 0 B c5ff91f2b32c /bin/sh -c mkdir /home/weeby/projects 0 B fc032221787f /bin/sh -c #(nop) USER [weeby] 0 B ef78532c29c3 /bin/sh -c #(nop) ADD file:abd8470a4e17591d90 2.502 kB 46492ac76bd7 /bin/sh -c #(nop) ADD file:a5adf541d11457b216 113 B d6427b5956f9 /bin/sh -c mkdir /usr/local/var && chmod 0 B 0f7bfc28bbf1 /bin/sh -c cd /home/weeby/dev/weebyco_cloud/a 310.2 kB 898a4341879c /bin/sh -c cd /home/weeby/dev/weebyco_cloud/c 587.5 kB 7a6065e92b0f /bin/sh -c #(nop) USER [root] 0 B cd1f2caaa012 /bin/sh -c echo 'cd $HOME/dev && git clon 25.72 MB 2c88826fc9eb /bin/sh -c echo 'cd $HOME/.config/devkit/cach 961.4 MB 25439817982f /bin/sh -c echo 'cd $HOME/.config/devkit/cach 168.1 MB 4556160eddef /bin/sh -c echo 'cd $HOME/dev && git clon 120.4 MB a08d25dd66ae /bin/sh -c echo 'cd $HOME/dev && git clon 98.14 MB a035740ba00e /bin/sh -c echo 'cd $HOME/dev && git clon 47.29 MB 0c62eb7db0d2 /bin/sh -c echo 'cd $HOME/dev && git clon 3.135 MB 33c95b02dfa9 /bin/sh -c echo 'cd $HOME/dev/cloud9/package 127.9 MB 763729fa92a4 /bin/sh -c #(nop) USER [weeby] 0 B 2e78c2b40604 /bin/sh -c ln -s /home/weeby/dev/cloud9/bin/c 36 B 6a884b1375f3 /bin/sh -c cd /home/weeby/dev/cloud9 && w 25.04 MB 28c5729feee9 /bin/sh -c #(nop) USER [root] 0 B 80d0e886412a /bin/sh -c cd $HOME/dev/cloud9 0 B 05d56e545a3c /bin/sh -c git clone https://github.com/ajaxo 139.5 MB 0ab202522723 /bin/sh -c echo "y" | android update sdk --no 75.4 MB ab31fb1a51bb /bin/sh -c wget -qO- https://raw.githubuserco 20.36 MB 4feec10a13dc /bin/sh -c #(nop) USER [weeby] 0 B 8c005817dbcc /bin/sh -c chown -R weeby:weeby /usr/local/li 0 B 07864b99174c /bin/sh -c mkdir /usr/local/lib/node_modules 0 B 04c8e6de638a /bin/sh -c git config --global user.name "wee 64 B 84da5ff19c9b /bin/sh -c git config --global user.email "us 36 B 78935503e47e /bin/sh -c echo "Host github.comntStrictHos 43 B 312ff96ab276 /bin/sh -c chmod 400 /root/.ssh/id_rsa 1.675 kB 7b2614d72c1a /bin/sh -c #(nop) ADD file:38f735b4fb5a6b4471 404 B d955c233c352 /bin/sh -c #(nop) ADD file:e2e0b14867bf7dbe94 1.675 kB a559d491dd97 /bin/sh -c mkdir /root/.ssh 0 B 1e18f03f0089 /bin/sh -c echo "Host github.comntStrictHos 2.122 kB eea093d89545 /bin/sh -c #(nop) ADD file:38f735b4fb5a6b4471 404 B a1327152885a /bin/sh -c #(nop) ADD file:e2e0b14867bf7dbe94 1.675 kB 99ef3790f6c1 /bin/sh -c #(nop) USER [root] 0 B 31f5f6c9a6dc /bin/sh -c #(nop) ENV JAVA_HOME=/usr/lib/jvm/ 0 B b613d6e86ae0 /bin/sh -c #(nop) ENV HOME=/home/weeby 0 B 98e4522b2296 /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B ae86930cd9db /bin/sh -c #(nop) ENV ANT_HOME=/usr/local/apa 0 B c8309a6f63d0 /bin/sh -c #(nop) ENV NVM_DIR=/home/weeby/.nv 0 B 35009f9ab773 /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B feb64f27edf5 /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B 167629594dee /bin/sh -c #(nop) ENV PATH=/usr/local/sbin:/u 0 B 0b3c53da27ff /bin/sh -c #(nop) ENV ANDROID_HOME=/usr/local 0 B d4ddbd91b703 /bin/sh -c mkdir /home/weeby/.ssh/ && mkd 0 B 46312a881da6 /bin/sh -c cd /home/weeby && wget -q http 37.58 MB a64be5f192ac /bin/sh -c cd /home/weeby && wget -q http 1.375 GB f806641c7f11 /bin/sh -c cd /home/weeby && wget -q http 172.5 MB 67aee57af54c /bin/sh -c #(nop) USER [weeby] 0 B c6b9bec31057 /bin/sh -c mkdir /etc/ssh/weeby && touch 2.885 kB 8d74e04926ed /bin/sh -c mkdir /usr/local/android-sdk && 0 B 3ba127c0a460 /bin/sh -c addgroup weeby --gid=1001 && a 334.5 kB 718bde0aab9c /bin/sh -c pip install WTForms watchdog uWSGI 9.972 MB 4698ca611967 /bin/sh -c apt-get update && apt-get inst 572.6 MB ddeaf9c76769 /bin/sh -c dpkg --add-architecture i386 11 B 45bdcf534dd2 /bin/sh -c add-apt-repository ppa:webupd8team 776.1 kB a006895ec3ac /bin/sh -c apt-get update && apt-get -y i 61.53 MB f8ada74e3af2 /bin/sh -c echo "debconf shared/accepted-orac 2.797 MB 14a199c8ef06 /bin/sh -c #(nop) ENV DEBIAN_FRONTEND=noninte 0 B ba6abd860f82 /bin/sh -c #(nop) MAINTAINER Martin Hunt <mar 0 B 96864a7d2df3 /bin/sh -c #(nop) CMD [/bin/bash] 0 B 809ed259f845 /bin/sh -c apt-get update && apt-get dist-upg 12.39 MB 9387bcc9826e /bin/sh -c sed -i 's/^#s*(deb.*universe)$/ 1.895 kB 897578f527ae /bin/sh -c rm -rf /var/lib/apt/lists/* 0 B c1f3bdbd8355 /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic 194.5 kB bfb8b5a2ad34 /bin/sh -c #(nop) ADD file:a889e7d86acdbac15e 192.5 MB 511136ea3c5a 0 B
- 30.
- 31. What our deploymentlooks like 31
- 32. What our teamlooks like 32 [Insert witty picture of Martin and I here]
- 33. What development lookslike 33
- 34. What is next? 34 • Scale and load balance • Reduce container load times • Offload running state • Streamlining the development flow Packaging and releasing our build tools for other teams to make use of as well
- 35.
Editor's Notes
- #2 * The title of this talk is “Scaling Development Environments with Docker”, but that does not really cover everything * * Producing scalable, state heavy, workspaces for users * Maintaining multiple projects with many dependencies * Deploying large scale containers rapidly * Anecdote about Maxes project * * Shared code between containers * * Many containers in a single project * * Easy and fast deployment of changes
- #3 * Weeby co is a games company in Mountain View, California * How do you bring new developers onto your project, how can you make a new developer a useful developer * How do you manage dependencies for containers, two main types of containers
- #4 * Our goal is to create the best tools that exist for developing social mobile games * Take a 2 man development team and put them in a place where they can compete with Candy Crush or Dragon Flight * Our javascript games run just as well as a pure native game, and we can – in a few clicks – integrate rich social features into that game * * Send lives, daily bonuses, leaderboards, in game currency * Rapid feedback of in browser development – hit refresh and see your changes immediately * Develop once, deploy anywhere * Make it social in a mater of minutes – all it takes is a few clicks
- #5 * Unfortunately that is not easy to do, if you are a developer or a product manager * Our tools require a lot of dependencies, and require the setup to be done exactly correct NEXT SLIDE * Working with node is not always simple * * Sometimes broken state, difficult on your computer, difficult when its not your computer, even more difficult when you don’t have direct access to the computer * Most of our tools assume you are working on a Mac OS, and if not at least a UNIX OS
- #6 * Unfortunately that is not easy to do, if you are a developer or a product manager * Our tools require a lot of dependencies, and require the setup to be done exactly correct THIS SLIDE * For example: working with node is not always simple * * Sometimes broken state, difficult on your computer, difficult when its not your computer, even more difficult when you don’t have direct access to the computer * Most of our tools assume you are working on a Mac OS, and if not at least a UNIX OS
- #7 * Obviously the most universally accessible solution is virtualization * Move all of our tools off of your computer and on to a server that your computer has access to
- #8 * A lot of different ways to virtualize software, unfortunately most are easy for the user but complicated for the developers * Cannot have a server per user * * Not cost effective * * Doesn’t scale well * * Ton of overhead, as well as long startup and shutdown times
- #9 * With Docker we can condense this entire server in to a container * * Easily distributed and scaled * * Cost effective * What is actually running inside this container * * Agent * * Weeby * * Devkit * Now apparent why a games company needs docker. Providing all of our tools in the most simple to consume way possible, over the web
- #10 * There are a lot of servers running inside containers on our machine, and we have a limited number of ports that we can expose * * Issue of linking to a non standard port in a web browser * * Issue of accessing non standard ports from behind firewalls
- #11 * Easily solved by using NGINX as a reverse proxy * Solves the problem of ports * The system is still not easy for us to use – must spin up containers individually, edit NGINX configurations, and then reload the routing * Manually managed systems are not fun
- #12 * Make a server to manage docker and nginx * * Using python Flask server with beaker for sessions and uWSGI for threading * * Responsible for not only creating new containers, reloading routing configuration, but also monitoring container health, and spinning down inactive containers to save space on the system
- #13 * Easily add an authentication method through NGINX to make all access to our user containers secure * * Sweet docker goodness * * No container is exposed to the outside world, so the only way to access them is through NGINX
- #14 * …Getting toward the end of things now * Need a way to easily persist user projects, simple with docker volumes * Still have this problem of manually managing this “server”, there are a lot of things going on and what is worse than having to keep track of all that
- #15 * Docker is awesome, again * Allows us to completely wrap the manager in a way that it can now be deployed to any system running docker * Problem: What happens when we need to update the manager? * * Shut down manager, no NGINX, no route from user computer to user container
- #16 * Again we were shown how using docker to compartmentalize makes development much simpler * Move NGINX into a separate container, so that we can start a new manager whenever we want * * The procedure is fairly straight forward * * New manager * * Starts up all its internal servers * * Adopts existing user containers * * Tell NGINX container that it is the new master, and routes should point to itself * * Tell the old manager it can now safely shut down
- #17 * It turns out developing this whole system is complicated * There are a ton of things going on inside our manager and agent, and we haven’t even looked at the rest of the stack * What do each of these production servers do? * * Registry is backed by Amazon’s S3, each manager has access to a registry * * Build server because uploading large containers is not possible on all networks * * Google App Engine servers to run our authentication and game logic (leaderboards, push notifications, etc.) * * NGINX container to route containers * * Manager * * Agent
- #18 * Developing servers is always a difficult task because you have to do your best to make sure your dev environment matches your deployment environment * Should something change, an interaction, a dependency, it could cause the entire system to fail * Difficult to keep track of interactions in your head, and verify based on your personal knowledge that a change will not effect some other part of the system * People are bad at programming, failure is always a possibility, perfect test coverage is difficult
- #19 * We rely on Google App Engine for our game server, as well as our producer authentication server * These are separate servers because we want devkit to remain open and usable to anyone with no affiliation to the producer * One generic App Engine container running the local python development tools * Mount the specific server files in to the container at run time
- #20 * Right now our system is relatively simple, we are just looking at local running versions of our app engine servers * Being able to completely isolate them is great because we are able to see a much more accurate depiction of how the servers will run in deployment
- #21 * Next we add in the NGINX routing container and the manager * We have support in these servers to configure important URLs, so we can easily just choose a URL and have it route based on our /etc/hosts file * * Instead of “producer.weeby.co”, it is “producer-dev.weeby.co”
- #22 * How to deal with a bunch of web servers all running on port 80 inside of boot2docker * * Fully configurable URLs, so we can just tell docker that newServer should be running on 8080, nimbus on 8081, etc * Explain the interactions between containers
- #23 * Next we start up the manager container * The agent container is started as part of the log in procedure, which means it is not started here, but it will be in the diagrams for clarity * Really the meat and beans, these are the two servers that see near constant development and improvement
- #24 * This is the complete development environment, without any of the interactions between the containers * Nothing too complicated, each container has a very specific task. Each server has a very concise repository of code
- #25 * Now we are able to see the interactions of all our containers and we can look at our entire system as a whole, from the comfort of our local development machine * What does the flow look like for a user coming into the system for the first time? * How the manager is able to talk back to the reload server * How do we actually edit the servers running inside of these containers?
- #26 * This shows all of the file mounts into the containers that originate in our host development machine * * These mounts allow us to interact with all of the files as though they are local files, using vim or sublime, git, etc * * All the files are in one place, in one repository so that we can easily track the entire deployment * Changes made locally are available inside the containers, so we can develop each server as though it were actually running on our local machine, except each server is actually running in its own isolated environment
- #27 * There is a common way of setting up a docker build * * Use a Dockerfile and a launch script * * We do this, but we also add build-container and run-container scripts so that any special arguments or mounts are easily reproduced * The templating is part of a tool that we started developing for this project * * Dockerfile_base template and Dockerfile_patch template * * We are building 4-5 GB images, and cannot afford to build the entire image on a regular basis * * * Leverage docker’s union filesystem * * Patch builds only run a checkout instead of the entire build proces
- #28 * Our build tools know when to run the base template, based on git diffs on the base template * If there are no diffs, then it runs the patch template to only insert new changes made to the other dependencies
- #29 * Configuring the templates are easy – just define where (relatively) the dependencies are, build tools automatically determine the commit, and automatically inject that in to the dockerfile template * Build tools also automatically determine if it is a minor build or a patch build, by checking for changes made to the dockerfile_base template. If it is a patch, the most recent image is used, and then a checkout is run on the repositories, making patch builds almost instant * Why have a template? Why have two templates?
- #30 * This is just the agent container, it has been reduced in size by about 1 GB since the beginning of the project * Running all of our software takes a lot of space and takes a long time to clone dependencies * Cannot rely on a strong internet connection to allow us to develop (strong meaning ability to upload 4+ GB)
- #31 * Our build server is ridiculously simple, since all of our build tools are packaged in to the main repository * * Build server listens to github webhook requests, looks for tags matching a certain pattern, ‘agent-0.6.0’, and then runs the build tools * * Checkout the new tag, run the build scripts, tag and upload the new image, remove any old images to keep build server storage free * This build server is super resilient ... not only are you checking out the content to build, but you are also checking out the scripts (and the procedure) for building the content * * You know that the build server will work because you can test all of the scripts it will use locally, with the same mentality as running all of the containers locally * * Break it without any repercussions
- #32 * This is spread out across many servers, and is pretty complicated to conceptualize
- #33 * This is Martin and myself, we are the two people developing these tools and building this architecture
- #34 * No way that 2 developers could work on this project without being able to work on the entire thing locally, without being able to examine interactions first hand, and have the ability to break things without causing disruptions in service * Started off taking 30 minutes to build and 30 minutes to upload * * Now takes 30 minutes total for a rebuild and 30 seconds for a patch * * Build server uses our main code repository, and the same build scripts that we use locally * * git hook looking for a new agent or a new manager, automatically gets the next version from our registry, increments, and builds * Full circle of development to build to upload to accessible by a user
- #35 * Where are we going next with this project and these tools * Work with shippable to integrate their build server support into our tools
- #36 * Thank you