Talk from the Oct 11, 2016 LA Docker Meetup

1. SOLVING REAL WORLD PRODUCTION
PROBLEMS WITH DOCKER
OCTOBER 11TH, 2016
DOCKER MEETUP, LOS ANGELES

2. MARC CAMPBELL
@mccode

3. GOALS
• Review Docker features that enable a more reliable, secure
production environment
• Present a secure build-deliver-execute process that includes
Docker in production
• Provide solutions you can start using today

4. “The only difference between a process in a container
and a process not in a container is a few labels on top
of a process that say ‘this is in container X’”
Jérôme Petazzoni, Docker
July 06, 2015

5. SECURE DELIVERY PIPELINE
BUILD DELIVER EXECUTE

6. BUILD DELIVER EXECUTE

7. BUILD DELIVER EXECUTE
BUILD
Choosing and creating container images that
will run in your production environment.

8. THREE DIFFERENT ROLES, EQUALLY IMPORTANT
Operations
Development
Security
Does it work?
Can it be supported?
Can it be safely run?
Is it Elasticsearch? I don’t want “elasticsearch-like”, I want Elasticsearch and
I want version 2.4.1.
Will it send alerts when it breaks? Does it support zero-downtime upgrades?
There are 2,532 Elasticsearch containers in DockerHub. Why this one?
BUILD DELIVER EXECUTE

9. • Development images do not have to be the same as production images
• Prefer library (official) images when possible
BUILD DELIVER EXECUTE
• Always look at the Dockerfile, regardless of pull count
• Be cautious when bind mounting the docker.sock file

10. BUILD DELIVER EXECUTE

11. Best practices
- whitelist (or choose) base images
- don’t trust “pull count” from DockerHub, find and read the dockerfile
- use the most specific tag possible
redis:sha256@abc > redis:3.2.4 > redis:3.2 > redis:3 > redis:latest
- adopt a tagging pattern for your own images
- use security scanning (coreos clair or dockerhub)
- use docker content trust
BUILD DELIVER EXECUTE

12. Monitor images with DockerHub
Security Scanning or CoreOS Clair
The current nginx container on DockerHub has:
13 Critical CVEs
23 Major CVEs
Including 1 CRITICAL OpenSSL CVE
BUILD DELIVER EXECUTE

13. BUILD DELIVER EXECUTE

14. DELIVER
Ensure the images you want to run are the
images you are running
BUILD DELIVER EXECUTE

15. I typed `docker run redis` so now i’m
running redis…right?…right???
BUILD DELIVER EXECUTE

16. What happens when you type `docker run redis`
DOCKER RUN
REDIS
REDIS:LATEST
IMAGE EXISTS?
CREATE REDIS
CONTAINER
PULL
REDIS:LATEST
START REDIS
CONTAINER
NO
YES
BUILD DELIVER EXECUTE

17. BUILD DELIVER EXECUTE
DOCKER RUN
REDIS
DOCKER CLI
DOCKER ENGINE
DOCKER HUB
CREATE NO IMAGE PULL
GET /V2
PARSE
HEADER
Trust Boundaries
401 AUTH
REQUIRED
POST
/LOGIN
GET
/V2/…/MANIFEST
GET
/V2/…/LAYER
IMAGE
COMPLETE
CREATE START

18. Connect to a trusted host
Deliver the content over a secure channel
Sent the content you requested
Verify the author of the content
A.
B.
C.
D.
To securely download data from the Internet
BUILD DELIVER EXECUTE
HTTPS
TLS
Content Addressable IDs
Signed Images
The problems The solutions

19. Downloading and executing software from the Internet is dangerous
Don’t download from untrusted hosts.
e.g.: `docker pull [—insecure-registry] 52.207.178.113/redis:latest`
Don’t download on insecure channels.
e.g.: `docker pull [—insecure-registry] registry.mycompany.com/redis:latest`
Don’t trust the remote server to look up the content.
e.g.: `docker pull redis:latest`
Don’t trust content that isn’t signed by the publisher.
e.g.: `docker pull --disable-content-trust redis:latest`
1.
2.
3.
4.
BUILD DELIVER EXECUTE

20. Docker Content Trust
“Content trust gives you the ability to verify both
the integrity and the publisher of all the data
received from a registry over any channel”
BUILD DELIVER EXECUTE

21. $ docker pull redis
Using default tag: latest
latest: Pulling from library/redis
6a5a5368e0c2: Pull complete
<...>
2bcdfa1b63bf: Pull complete
Digest: sha256:38e873a...912
Status: Downloaded newer image for redis:latest
WITHOUT TRUST:
PULL BY TAG
BUILD DELIVER EXECUTE

22. $ export DOCKER_CONTENT_TRUST=1
$ docker pull redis
Using default tag: latest
Pull (1 of 1): redis:latest@sha256:c4365e...680
sha256:c4365ec...680: Pulling from library/redis
6a5a5368e0c2: Pull complete
<...>
58e3d55f4ce5: Pull complete
Digest: sha256:c4365e...680
Status: Downloaded newer image for
redis@sha256:c4365e...680
Tagging redis@sha256:c4365e...680 as redis:latest
WITH TRUST:
PULL BY SHA
BUILD DELIVER EXECUTE

23. BUILD DELIVER EXECUTE
DEMO
•Create a signed image
•Run a signed image
•Update the image from an untrusted source
•Pull and run the new image

24. BUILD DELIVER EXECUTE

25. EXECUTE
Provide a consistent, secure environment
with continuous auditing
BUILD DELIVER EXECUTE

26. Center For Internet Security
• Use AppArmor / SELinux
• Enable Kernel Auditing
• User namespaces
• /var/lib/docker volume
• Enable an authorization plugin
• Use a centralized log driver
• Prevent registry v1 access
https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_1.11.0_Benchmark_v1.0.0.pdf
BUILD DELIVER EXECUTE

27. Docker Bench for Security
https://dockerbench.com/
docker run -it --net host --pid host --cap-add audit_control
-v /var/lib:/var/lib
-v /var/run/docker.sock:/var/run/docker.sock
-v /usr/lib/systemd:/usr/lib/systemd
-v /etc:/etc --label docker_bench_security
docker/docker-bench-security
BUILD DELIVER EXECUTE

28. 2.1 - Restrict network traffic between containers
2.2 - Set the logging level
2.3 - Allow Docker to make changes to iptables
2.4 - Do not use insecure registries
2.5 - Do not use the aufs storage driver
2.6 - Configure TLS authentication for Docker daemon
* Docker daemon not listening on TCP
2.7 - Set default ulimit as appropriate
* Default ulimit doesn't appear to be set
2.8 - Enable user namespace support
2.9 - Confirm default cgroup usage
2.10 - Do not change base device size until needed
2.11 - Use authorization plugin
2.12 - Configure centralized and remote logging
2.13 - Disable operations on legacy registry (v1)
[WARN]
[PASS]
[PASS]
[PASS]
[WARN]
[INFO]
[INFO]
[INFO]
[INFO]
[WARN]
[PASS]
[PASS]
[WARN]
[WARN]
[WARN]
BUILD DELIVER EXECUTE

29. Review
☑ Choose images carefully
☑ Scan your Dockerfiles
☑ Enable Docker Content Trust
☑ Run Docker Benchmark for Security