"Never upgrade a server again. Never update your code. Instead, create new servers, and throw away the old ones!" That's the idea of immutable servers, or immutable infrastructure. This makes many things easier: rollbacks (you can always bring back the old servers), A/B testing (put old and new servers side by side), security (use the latest and safest base system at each deploy), and more. However, throwing in a bunch of new servers at each one-line CSS change is going to be complicated, not to mention costly. Containers to the rescue! Creating container "golden images" is easy, fast, dare I say painless. Replacing your old containers with new ones is also easy to do; much easier than virtual machines, let alone physical ones. In this talk, we'll quickly recap the pros (and cons) of immutable servers; then explain how to implement that pattern with containers. We will use Docker as an example, but the technique can easily be adapted to Rocket or even plain LXC containers.

1. Immutable
infrastructure
with Docker and
containers
1 / 59

2. Who am I?
Jérôme Petazzoni (@jpetazzo)
French software engineer living in California
Joined Docker (dotCloud) more than 4 years ago
(I was at Docker before it was cool!)
I have built and scaled the dotCloud PaaS
I learned a few things about running containers
(in production)
2 / 59

3. Outline
What is immutable infrastructure?
What are its pros and cons?
How can it be implemented with containers?
Also: demos!
3 / 59

4. Immutable
infrastructure
(a.k.a. immutable servers,
phoenix servers, etc.)
4 / 59

5. Rule 1: never change what's on a server
Don't install new packages
Don't upgrade existing ones
Don't remove or downgrade them
(Even for security vulnerabilities!)
Don't edit configuration files
Don't update your app code
(Even for small or urgent fixes!)
5 / 59

6. Rule 2: if tempted to change something...
See Rule 1
(OK, we will see an exception later.)
6 / 59

7. How do we upgrade?
Create new server from scratch
Apply deployment process*
(scripts, configuration management...)
(Optional: test the new server)
Replace old server with new server
Keep old server around, just in case
* Configuration management helps, but is not mandatory here.
7 / 59

8. WHY?!?
8 / 59

9. Avoid drift
9 / 59

10. Avoid drift
10 / 59

11. Avoid drift
Drift = differences between servers
(when they are supposed to be identical)
Caused by:
provisioning servers at different times
any manual operation
Consequences:
seemingly random failures
same code, different behavior
gets worse with time
11 / 59

12. Coping with drift
Careful replication of manual operations doesn't scale
(and is error-prone)
Automation seems simple at first,
but has to deal with many edge cases
Configuration management helps,
but only deals with what you've defined
12 / 59

13. Automation fails
"Let's use parallel-ssh!" (Or your favorite tool)
What if some servers...
are unreachable
become unreachable during the process
are being provisioned at the same time
What if one of those services is (partially) down?
distro package repositories
code or artifact repositories
13 / 59

14. Config management fails
package{"openssl":ensure=>"installed"}
14 / 59

15. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
15 / 59

16. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
package{"openssl":ensure=>"1.0.1g"}
16 / 59

17. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
package{"openssl":ensure=>"1.0.1g"}
Something went wrong, abort, abort!
17 / 59

18. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
package{"openssl":ensure=>"1.0.1g"}
Something went wrong, abort, abort!
package{"openssl":ensure=>"installed"}
18 / 59

19. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
package{"openssl":ensure=>"1.0.1g"}
Something went wrong, abort, abort!
package{"openssl":ensure=>"installed"}
We didn't roll back to whatever-we-had!
19 / 59

20. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
package{"openssl":ensure=>"1.0.1g"}
Something went wrong, abort, abort!
package{"openssl":ensure=>"installed"}
We didn't roll back to whatever-we-had!
package{"openssl":ensure=>"1.0.1f"}
20 / 59

21. Config management fails
package{"openssl":ensure=>"installed"}
A wild OpenSSL vulnerability appears!
package{"openssl":ensure=>"1.0.1g"}
Something went wrong, abort, abort!
package{"openssl":ensure=>"installed"}
We didn't roll back to whatever-we-had!
package{"openssl":ensure=>"1.0.1f"}
This should do the trick. (Hopefully.)
21 / 59

22. More nightmares
package{"openssl":ensure=>"1.0.1f"}
22 / 59

23. More nightmares
package{"openssl":ensure=>"1.0.1f"}
Package not found on the repos.
23 / 59

24. More nightmares
package{"openssl":ensure=>"1.0.1f"}
Package not found on the repos.
"Well, actually" we want an older version.
24 / 59

25. More nightmares
package{"openssl":ensure=>"1.0.1f"}
Package not found on the repos.
"Well, actually" we want an older version.
Package even less likely to be found on the repos.
25 / 59

26. More nightmares
package{"openssl":ensure=>"1.0.1f"}
Package not found on the repos.
"Well, actually" we want an older version.
Package even less likely to be found on the repos.
"Well, actually" we were using 0.9.8xxx.
When we requested 1.0.1g we upgraded the whole distro.
26 / 59

27. More nightmares
package{"openssl":ensure=>"1.0.1f"}
Package not found on the repos.
"Well, actually" we want an older version.
Package even less likely to be found on the repos.
"Well, actually" we were using 0.9.8xxx.
When we requested 1.0.1g we upgraded the whole distro.
(╯°□°）╯︵ ┻━┻)
27 / 59

28. With immutable servers
We still have the old server
Just put it back into service
(while we figure out the OpenSSL upgrade!)
Also works for any kind of upgrade
that needs to be rolled back
Alright, we have easy rollbacks.
But how does that help with drift?
28 / 59

29. "Trash your servers and burn your code"
(Chad Fowler)
Reprovision your servers regularly
(from scratch)
Ensures that you're always using recent packages
Any manual deviation gets fixed automatically
29 / 59

30. Improvement: golden image
Create a server from scratch
Apply deployment process
Snapshot this server (create an image)
(Optional: create a test server and validate it)
Create multiple identical servers from the image
Avoids uncertainties in the deployment process:
unreachable packages repositories etc.
Allows to keep (for cheap) past versions around.
30 / 59

31. Downsides
(and how to cope)
31 / 59

32. Problem: small changes are cumbersome
E.g. one line of CSS.
Before: manual change, validate, replicate
(a few minutes)
After: manual change, validate, ...
create new golden image from scratch
(one hour)
provision new servers from image
(a few minutes)
switch old/new servers
decommission old servers after a while
32 / 59

33. Solution: automation
All those operations have to happen
But everything after the "validate" step
should be automated
The clock time will still be 1+ hour
The user time will be a few minutes
(just like before)
Note: intermediary golden images can help
(provision from checkpoint instead of from scratch)
33 / 59

34. Problem: debugging is harder
E.g. troubleshoot network issues.
Before:
install tcpdump
fiddle with iptables
accumulate logs and packet captures locally
After:
install tcpdu-oops, the server was re-imaged
fiddle with ipta-oops, ...
logs and traces have to be shipped out
34 / 59

35. Solution 1: drift and self-destruct
Tag a given machine to prevent its "re-imaging"
Schedule it for self-destruct after e.g. 1 week
(shutdown+10000)
That machine is allowed to drift
(you can install your tools on it,
leave logs and traces locally...)
If you need more time, reschedule the self-destruct
35 / 59

36. Solution 1: drift and self-destruct
Tag a given machine to prevent its "re-imaging"
Schedule it for self-destruct after e.g. 1 week
(shutdown+10000)
That machine is allowed to drift
(you can install your tools on it,
leave logs and traces locally...)
If you need more time, reschedule the self-destruct
If you find yourself setting up a cron job to reschedule the self-
destruct, you're doing it wrong!
36 / 59

37. Solution 2: bundle the tools
Install tcpdumpand friends in the golden image
Enable traffic capture with feature switch
(Alternate solution: statistical sampling)
Automate shipping of logs and traces
It's more work in the beginning, but pays in the long run.
37 / 59

38. Problem: storing data
Databases and anything stateful!
Before: just store it locally
After: need to persist it somehow
38 / 59

39. Solution 1: not my problem
"Often you can pass the buck to a service which someone else
maintains, like Amazon's RDS database service."
(Kief Morris)
Easy!
But what if:
there is no such service
I can't use it for $REASONS?
39 / 59

40. Solution 2: state = files
All you need is a mechanism to store files externally.
NAS/SAN (on-prem)
EBS, EFS (AWS)
Ceph, Gluster... (anywhere)
But it's extra work, expensive, and/or slower.
40 / 59

41. Solution 3: ?
41 / 59

42. Solution 3: ?
SPOILER ALERT
42 / 59

43. Solution 3
43 / 59

44. Immutable
containers
44 / 59

45. Let's review our process
Create image:
from scratch
(can take an hour or more)
from checkpoint
(takes a few minutes, more complex)
Deploy it N times
(takes a few minutes)
How do we do that with containers?
45 / 59

46. Building container images
We get the best of both worlds:
from scratch
(clean rebuilds without side-effects)
incremental
(fast rebuilds when changes are minor)
Why and how?
container snapshots are cheap
(seconds versus minutes)
simple DSL to break down the build into steps
(each step = one command = one snapshot)
46 / 59

47. FROMdebian:jessie
MAINTAINERJessicaFrazelle<jess@docker.com>
#Installdependencies
RUNapt-getupdate&&apt-getinstall-y
build-essential
…
--no-install-recommends
#Installnode
RUNcurl-sLhttps://deb.nodesource.com/setup|bash-
RUNapt-getinstall-ynodejs
#Cloneatom
RUNgitclonehttps://github.com/atom/atom/src
WORKDIR/src
RUNgitfetch&&gitcheckout
$(gitdescribe--tags
`gitrev-list--tags--max-count=1`)
RUNscript/build&&script/gruntinstall
#Autorunatom
CMD/usr/local/bin/atom--foreground
47 / 59

48. What happens during the first build?
FROMdebian
RUNapt-getxxx
COPY./src
RUN/src/build
Create a container from debianbase image
Execute apt-getxxxin this container, take a snapshot
Create a container from this snapshot
Copy source files into /src, take a snapshot
Create a container from this snapshot
Execute /src/buildin this container, take a snapshot
The final snapshot is our built image.
48 / 59

49. What happens during subsequent builds?
Before executing each step:
check if we already executed the same step before
(and have a snapshot of its result)
if we do, use the snapshot and continue
otherwise, execute the step normally
(and snapshot the result)
As a result, we zoom through the build process,
until we hit a step that has changed
The end result is the same as a full clean build,
but much faster
49 / 59

50. Demo
50 / 59

51. Running container images
On physical or virtual machines
Run multiple containers per machine
Upgrading is faster
(doesn't have to wait for IaaS VM to come up)
Can reuse local data (Docker concept: "volumes")
Solves the stateful service problem
51 / 59

52. Demo
52 / 59

53. Bonus
Containers can share:
directories (e.g.: logs)
network stack (e.g.: traffic analysis)
... and more!
Logging, backups, metrics collection, troubleshooting...
can be done from "sidekick" containers.
53 / 59

54. Demo
54 / 59

55. Other niceties
Containers filesystem can be made read-only
enforces immutability
exception for data volumes (with noexec)
easier security audit
Cheaper
consolidation
save a few ¢ or $ per server per deploy
(great if your IAAS bills by the hour)
55 / 59

56. Conclusions
56 / 59

57. Immutable containers
All the advantages of immutable servers
(avoid drift, reliable rollbacks...)
Build in seconds instead of minutes/hours
Faster, simpler deployment
Deal with stateful services
Bonus: cheaper, safer, cleaner
57 / 59

58. 58 / 59

59. Thanks!
Questions?
@jpetazzo
@docker
59 / 59