A look at some of the configuration issues that containers introduce, and how to avoid or fix them. Discusses immutable infrastructure, the difference between build-time and runtime configuration, scheduler configuration and more.

1. (without introducing more risk)
The Challenges of Container
Configuration
Puppet
Gareth Rushgrove
New capabilities and associated problems

2. (without introducing more risk)
Gareth Rushgrove
@garethr

3. (without introducing more risk)
Gareth Rushgrove

4. (without introducing more risk)
Configuration
What is it and why should I care

5. (without introducing more risk)
Gareth Rushgrove

6. (without introducing more risk)
synonyms: design, grouping, marshalling
Gareth Rushgrove

7. (without introducing more risk)
Marshalling your containers
Gareth Rushgrove

8. (without introducing more risk)
- Immutability and containers
- Runtime vs build time
- Who configures the orchestrator?
Gareth Rushgrove

9. (without introducing more risk)
Mainly Docker and Kubernetes
examples, but should be
generally applicable
Gareth Rushgrove

10. (without introducing more risk)
Everything is
immutable now?
Assumptions vs reality

11. (without introducing more risk)
Gareth Rushgrove

12. (without introducing more risk)
Gareth Rushgrove
ker run -d ubuntu:16.04 /bin/sh
c "while true; do echo hello world; sleep 1; done"

13. (without introducing more risk)
Gareth Rushgrove
$ docker exec a7a01beb14de touch /tmp/surprise

14. (without introducing more risk)
Gareth Rushgrove
$ docker diff a7a01beb14de
C /tmp
A /tmp/surprise

15. (without introducing more risk)
Containers are not
immutable by default
Gareth Rushgrove

16. (without introducing more risk)
Containers are not
immutable by default
Gareth Rushgrove

17. (without introducing more risk)
Gareth Rushgrove
$ docker run --read-only -d ubuntu:16.04 /bin/sh
-c "while true; do echo hello world; sleep 1; done"

18. (without introducing more risk)
Gareth Rushgrove
$ docker exec 379150b2cf05 touch /tmp/surprise
touch: cannot touch '/tmp/surprise': Read-only file syste

19. (without introducing more risk)
Suggestion
Enable read-only where possible
Gareth Rushgrove

20. (without introducing more risk)
Many applications won’t start with
a read-only filesystem
Gareth Rushgrove

21. (without introducing more risk)
Gareth Rushgrove
1 import logging
2 from logging.handlers import RotatingFileHandler
3
4 from flask import Flask
5
6 app = Flask(__name__)
7
8 @app.route('/')
9 def home():
10 app.logger.info('log request')
11 return "We'll never get to here"
12
13 if __name__ == '__main__':
14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount
15 handler.setLevel(logging.INFO)
16 app.logger.addHandler(handler)
17 app.run(debug=True, host='0.0.0.0')

22. (without introducing more risk)
Gareth Rushgrove
1 import logging
2 from logging.handlers import RotatingFileHandler
3
4 from flask import Flask
5
6 app = Flask(__name__)
7
8 @app.route('/')
9 def home():
10 app.logger.info('log request')
11 return "We'll never get to here"
12
13 if __name__ == '__main__':
14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount
15 handler.setLevel(logging.INFO)
16 app.logger.addHandler(handler)
17 app.run(debug=True, host='0.0.0.0')

23. (without introducing more risk)
Gareth Rushgrove
$ docker run --read-only -p 5000:5000 garethr/flaskapp
Traceback (most recent call last):
File "app.py", line 14, in <module>
handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1)
File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 30] Read-only file system: '/app/app.log'

24. (without introducing more risk)
tmpfs support added in
Docker 1.10
Gareth Rushgrove

25. (without introducing more risk)
Gareth Rushgrove
$ docker run --read-only --tmpfs /tmp
-d ubuntu:16.04 /bin/sh
-c "while true; do echo hello world; sleep 1; done"

26. (without introducing more risk)
Gareth Rushgrove
$ docker exec 222331443d28 touch /tmp/surprise

27. (without introducing more risk)
Gareth Rushgrove
$ docker diff 222331443d28

28. (without introducing more risk)
Gareth Rushgrove
$ docker exec 222331443d28 ls /tmp
surprise

29. (without introducing more risk)
Suggestion
Use tmpfs only where needed
Gareth Rushgrove

30. (without introducing more risk)
Remember
Without technical controls you
only have social guarantees
of immutability
Gareth Rushgrove

31. (without introducing more risk)
Build vs Run
And the relationship between them

32. (without introducing more risk)
Given an image
- What machine built this image?
- Are all the licenses compatible?
- Who supports this image?
- Does this image contain malware?
Gareth Rushgrove

33. (without introducing more risk)
Given a running container
- Who built it?
- How was it built?
- What software does it contain?
- Is the software up-to-date?
Gareth Rushgrove

34. (without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]

35. (without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
Where did this base image come from?

36. (without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
What packages are installed? At what version?
Where are those packages from?

37. (without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
What version of flask is this?

38. (without introducing more risk)
Gareth Rushgrove
FROM ubuntu:16.04
RUN apt-get update &&
apt-get install -y python-pip python-dev build-essent
apt-get clean &&
rm -rf /var/lib/apt/lists/*
RUN pip install flask
COPY . /app
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["app.py"]
What was in this folder at build time?

39. (without introducing more risk)
The importance of time
Gareth Rushgrove

40. (without introducing more risk)
How often are images rebuilt?
Gareth Rushgrove

41. (without introducing more risk)
Rebuilding only on code change
ignores environmental factors
Gareth Rushgrove

42. (without introducing more risk)
Versioning and metadata
Gareth Rushgrove

43. (without introducing more risk)
Gareth Rushgrove
LABEL vendor="ACME Incorporated"
com.example.is-beta
com.example.version="0.0.1-beta"
com.example.release-date="2015-02-12"

44. (without introducing more risk)
Gareth Rushgrove
LABEL vendor="ACME Incorporated"
com.example.is-beta
com.example.version="0.0.1-beta"
com.example.release-date="2015-02-12"
What time? What timezone?

45. (without introducing more risk)
Gareth Rushgrove
$ docker inspect -f "{{json .Config.Labels }}"
4fa6e0f0c678 | jq
{
"vendor": "ACME Incorporated",
"com.example.is-beta": "",
"com.example.version": "0.0.1-beta",
"com.example.release-date": "2015-02-12"
}

46. (without introducing more risk)
Suggestion
Decide upon and enforce
metadata standards
Gareth Rushgrove

47. (without introducing more risk)
Gareth Rushgrove
LABEL com.example.git.repository="https://github.com/pupp
com.example.git.sha="dc123cfb5ed4dca43a84be34a99d7c
com.example.build.time="2016-04-24T15:43:05+00:00"
com.example.build.builder=“jenkins1.example.com"
com.example.docs="https://github.com/puppetlabs/doc
...

48. (without introducing more risk)
Suggestion
Embed Dockerfiles in images
Gareth Rushgrove

49. (without introducing more risk)
Gareth Rushgrove
$ docker inspect -f "{{json .Config.Labels }}"
garethr/alpine
| jq
{
"net.morethanseven.dockerfile": "/Dockerfile",
}

50. (without introducing more risk)
Gareth Rushgrove
$ docker run -i -t garethr/alpine cat /Dockerfile
FROM alpine
LABEL net.morethanseven.dockerfile="/Dockerfile"
RUN apk add --update bash && rm -rf /var/cache/apk/*
COPY Dockerfile /

51. (without introducing more risk)
Suggestion
Provide an API for your containers
Gareth Rushgrove

52. (without introducing more risk)
Gareth Rushgrove
$ docker inspect -f "{{json .Config.Labels }}"
garethr/alpine
| jq
{
"com.example.api.packages": "apk info -vv"
}

53. (without introducing more risk)
Gareth Rushgrove
$ docker run -i -t garethr/alpine apk info -vv
musl-1.1.11-r2 - the musl c library (libc) implementation
busybox-1.23.2-r0 - Size optimized toolbox of many common UNIX
alpine-baselayout-2.3.2-r0 - Alpine base dir structure and init
openrc-0.15.1-r3 - OpenRC manages the services, startup and shu
alpine-conf-3.2.1-r6 - Alpine configuration management scripts

54. (without introducing more risk)DEMO

55. (without introducing more risk)
Who configures the
scheduler?
Higher level configuration

56. (without introducing more risk)
Schedulers/orchestrators abstract
you from
- Where individual containers run
- Balancing due to new resources
- Balancing due to failed resources
Gareth Rushgrove

57. (without introducing more risk)
This results in a constraints
based system
Gareth Rushgrove

58. (without introducing more risk)
Which means those constraints
need to be explicit and correct
Gareth Rushgrove

59. (without introducing more risk)
Gareth Rushgrove
$ docker daemon
--label com.example.environment="production"
--label com.example.storage="ssd"

60. (without introducing more risk)
Gareth Rushgrove
$ docker run -d -P
-e constraint:storage==ssd --name db mysql

61. (without introducing more risk)
Gareth Rushgrove
1 template:
2 metadata:
3 labels:
4 app: guestbook
5 tier: frontend
6 spec:
7 containers:
8 - name: php-redis
9 image: gcr.io/google_samples/gb-frontend:v4
10 resources:
11 requests:
12 cpu: 100m
13 memory: 100Mi
14 env:
15 - name: GET_HOSTS_FROM
16 value: dns
17 # If your cluster config does not include a dns service, th
18 # instead access environment variables to find service host
19 # info, comment out the 'value: dns' line above, and uncomm

62. (without introducing more risk)
How do you manage properties
for all of your hosts?
Gareth Rushgrove

63. (without introducing more risk)
Gareth Rushgrove
$ docker daemon
--label com.example.environment="production"
--label com.example.storage="ssd"
Does this machine really have an SSD?
What if someone swaps the drive?

64. (without introducing more risk)
Suggestion
Use properties of hosts as labels
Gareth Rushgrove

65. (without introducing more risk)
Gareth Rushgrove
$ facter | head -n 20
architecture => x86_64
domain => local
facterversion => 2.4.6
fqdn => Pro.local
gid => staff
hardwareisa => i386
hardwaremodel => x86_64
hostname => Pro
id => garethr
interfaces =>
lo0,gif0,stf0,en0,p2p0,awdl0,en1,en2,bridge0,vboxnet0,vboxnet1,v
ipaddress => 192.168.0.5
ipaddress_en0 => 192.168.0.5

66. (without introducing more risk)
Gareth Rushgrove
$ facter -j os | jq
{
"os": {
"name": "Darwin",
"family": "Darwin",
"release": {
"major": "14",
"minor": "5",
"full": "14.5.0"
}
}
}

67. (without introducing more risk)
Gareth Rushgrove
$ docker daemon
--label net.example.os=`facter operatingsystem`
--label net.example.virtual=`facter is_virtual`
--label net.example.kernel=`facter kernelversion`
...

68. (without introducing more risk)
Orchestrators also tend to
introduce new higher-level
primitives
Gareth Rushgrove

69. (without introducing more risk)
Docker Networks, Kubernetes
Services, ReplicationControllers,
Chronos Jobs
Gareth Rushgrove

70. (without introducing more risk)
Many with imperative interfaces
Gareth Rushgrove

71. (without introducing more risk)
Gareth Rushgrove
$ kubectl get pod mypod -o yaml
| sed 's/(image: myimage):.*$/1:v4/'
| kubectl replace -f -

72. (without introducing more risk)
Gareth Rushgrove
$ docker network create bob
c0a0f4538d259515813b771264688d37aaedb41098379a0d73ec0ca08
$ docker network create bob
Error response from daemon: network with name bob already

73. And everything configured
in YAML
Gareth Rushgrove

74. Code plus data has advantages
over data alone
Gareth Rushgrove

75. The language to represent the data should
be a simple, data-only format such as JSON
or YAML, and programmatic modification of
this data should be done in a real
programming language
Gareth Rushgrove
Borg, Omega, and Kubernetes, ACM Queue,Volume 14, issue 1 http://queue.acm.org/detail.cfm?id=2898444
“

76. Avoid repetition
Combine external inputs
Correctness
Abstractions
Gareth Rushgrove
-
-
-
-

77. Suggestion
Use a higher level programming
tool for generating config data
Gareth Rushgrove

78. (without introducing more risk)
Gareth Rushgrove
jsonnet.org

79. (without introducing more risk)
Gareth Rushgrove
$ cat test.jsonnet
// Example jsonnet file
{
person1: {
name: "Alice",
welcome: "Hello " + self.name + "!",
},
person2: self.person1 { name: "Bob" },
}

80. (without introducing more risk)
Gareth Rushgrove
$ jsonnet test.jsonnet | jq
{
"person1": {
"name": "Alice",
"welcome": "Hello Alice!"
},
"person2": {
"name": "Bob",
"welcome": "Hello Bob!"
}
}

81. (without introducing more risk)
Gareth Rushgrove
$ jsonnet test.jsonnet | json2yaml
---
person1:
name: "Alice"
welcome: "Hello Alice!"
person2:
name: "Bob"
welcome: "Hello Bob!"

82. (without introducing more risk)DEMO

83. (without introducing more risk)
Gareth Rushgrove
garethr/kubernetes

84. (without introducing more risk)
Gareth Rushgrove
kubernetes_pod { 'sample-pod':
ensure => present,
metadata => {
namespace => 'default',
},
spec => {
containers => [{
name => 'container-name',
image => 'nginx',
}]
},
}

85. (without introducing more risk)
Gareth Rushgrove
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: sample-pod
spec:
container:
- image: nginx
name: container-name

86. (without introducing more risk)
Gareth Rushgrove
controller_service_pair { 'redis-master':
app => 'redis',
role => 'master',
tier => 'backend',
port => 6379,
}

87. (without introducing more risk)
Gareth Rushgrove
apiVersion: v1
kind: Service
metadata:
name: redis-master
labels:
app: redis
tier: backend
role: master
spec:
ports:
# the port that this service should serve on
- port: 6379
targetPort: 6379
selector:
app: redis
tier: backend
role: master
---
apiVersion: v1
kind: ReplicationController
metadata:
name: redis-master
# these labels can be applied automatically
# from the labels in the pod template if not set
labels:
app: redis
role: master
tier: backend
spec:
# this replicas value is default
# modify it according to your case
replicas: 1
# selector can be applied automatically
# from the labels in the pod template if not set
# selector:
# app: guestbook
# role: master
# tier: backend

88. (without introducing more risk)DEMO

89. (without introducing more risk)
Conclusions
New technology means old problems

90. (without introducing more risk)
The difference between how you
think something works and how
it actually works risks
hard-to-debug production issues
Gareth Rushgrove

91. (without introducing more risk)
Containers introduce new and old
configuration problems
Gareth Rushgrove

92. (without introducing more risk)
Configuration management
is the discipline aimed at
minimising those risks
Gareth Rushgrove

93. (without introducing more risk)
Start with principles
Gareth Rushgrove

94. (without introducing more risk)
- Identification
- Control
- Status accounting
- Verification
Gareth Rushgrove
Military Handbook Configuration Management Guidance MIL-HDBK-61B

95. (without introducing more risk)
Apply them to your
container based
infrastructure today
Gareth Rushgrove

96. (without introducing more risk)
Questions?
And thanks for listening