How to secure your docker installation

1. Docker security
SOFTWARE FREEDOM DAY, 2019, SZEGED
Janos SUTO, sj@acts.hu

2. Isn't docker secure?
Ars Technica: Infected images mined digital coins
"17 images posted by a single account over10 months may have
generated $90,000."
"For ordinary users, just pulling a Docker image from Docker Hub is like
pulling arbitrary binary data from somewhere, executing it, and hoping
for the best without really knowing what’s in it.”
https://arstechnica.com/information-
technology/2018/06/backdoored-images-downloaded-5-million-times-
finally-removed-from-docker-hub/

3. CVE-2019-5736
"runc through 1.0-rc6, as used in Docker before 18.09.2 and other
products, allows attackers to overwrite the host runc binary (and
consequently obtain host root access) by leveragingthe ability to
execute a command as root within one of these types of containers:
(1) a new container withan attacker-controlled image, or (2) an
existing container, to which the attacker previously had write access,
that can be attached with docker exec. This occurs because of file-
descriptor mishandling, related to /proc/self/exe.

4. Important stuff I won't talk this time
Physical security
Host security (patched OS, only necessary packages, OS hardening, ...)
Networksecurity (open ports, firewalls, strict SSH access, …)
Educating users
...

5. Securing images

6. Official images
Essential base OS repositoriesas the starting point for users
Lead examples of Dockerfile best practices
Security updates are applied in a timely manner
Scanned for vulnerabilities
https://docs.docker.com/docker-hub/official_images/

8. Docker Content Trust (DCT)
Use digital signatures for data sent to and receivedfrom remote
Docker registries.
These signatures allow client-sideor runtime verification of the integrity
and publisher of specific image tags.
Through DCT, image publishers can sign their images and image
consumers can ensure that the images they use are signed.

9. DCT #2
export DOCKER_CONTENT_TRUST="1"
docker pull user/someimage
Error: remote trust data does not exist for docker.io/user/someimage:
notary.docker.io does not have trust data for
docker.io/user/someimage
https://docs.docker.com/engine/security/trust/content_trust/

10. Runtime Enforcement with DCT
Applies to Docker Enterprise only

11. Personal Access Tokens for Docker HUB

12. Use your own registry
Docker registry
Harbor
Quay (automatic security scanning)
JFrog Artifactory (PRO edition)

13. Don't use insecure registries
By default it's not enabled

14. Build your own images
Start from official images
Use a reasonable distro (eg. alpine)
Include only what's really required (eg. --no-install-recommends)
USER someuser
No sudo
No sshd

15. Don't bake any secrets to the image
ENV MYSQL_PASSWORD "aaaa"

16. Scan your images
Microscanner: https://github.com/aquasecurity/microscanner
FROM debian:jessie-slim
RUN apt-get update && apt-get -y install ca-certificates
ADD https://get.aquasec.com/microscanner /
ARG token
RUN chmod +x /microscanner && /microscanner ${token}

17. "vulnerabilities": [
{
"name": "CVE-2017-8398",
"description": "dwarf.c in GNU Binutils 2.28 is vulnerable to an
invalid read of size 1 during dumping of debug information from a
corrupt binary …".
"nvd_score": 5,
"nvd_score_version": "CVSS v2",
"nvd_vectors": "AV:N/AC:L/Au:N/C:N/I:N/A:P",
"nvd_severity": "medium",
"nvd_url": https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-
2017-8398,
…..
},

18. Other image scanner products
Clair
Docker Trusted Registry
JFrog Xray
...

19. Virus scanning
$ docker create --name erlang_scan erlang # Create container from image
$ docker export –output "live_system.tar" erlang_scan # Push the container fs to tar file
$ clamscan live_system.tar
live_system.tar: OK
----------- SCAN SUMMARY -----------
Known viruses: 6590083
Engine version: 0.100.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 1029.54 MB (ratio 0.00:1)
Time: 9.586 sec (0 m 9 s)
$ docker rm erlang_scan
https://medium.com/@cwgem/thoughts-about-docker-security-8e0df4b43650

20. Docker bench security
Checking for best practices:
1. Host configuration
2. Docker daemon configuration
3. Docker daemon configuration files
4. Container Images and Build File
5. Container Runtime
6. Docker Security Operations
7. Docker Swarm Configuration
https://github.com/docker/docker-bench-security

21. [INFO] 2 - Docker daemon configuration
[PASS] 2.1 - Ensure network traffic is restricted between containers on the
default bridge
[PASS] 2.2 - Ensure the logging level is set to 'info'
[PASS] 2.3 - Ensure Docker is allowed to make changes to iptables
[PASS] 2.4 - Ensure insecure registries are not used
[PASS] 2.5 - Ensure aufs storage driver is not used
[INFO] 2.6 - Ensure TLS authentication for Docker daemon is configured
[INFO] * Docker daemon not listening on TCP
[INFO] 2.7 - Ensure the default ulimit is configured appropriately
[INFO] * Default ulimit doesn't appear to be set

22. Securing the daemon

23. Protect the socket
srw-rw----1 root docker 0 Sep 10 21:04 /var/run/docker.sock=
Don't put just anyone to the docker group

24. Accessing docker over the network
TLS encryption (don't enable port 2375)
Certificate authentication
Firewall the docker host
DOCKER_HOST=tcp://docker.yourdomain.com:2376
DOCKER_TLS_VERIFY=1
https://docs.docker.com/engine/security/https/
https://docs.docker.com/engine/security/certificates/

25. Securing containers

26. Resource limits
--memory 2G: The maximum amount of memory the container can use
--memory-swap 2G: The amount of memory the container is allowed to
swap to disk
--shm-size 64M: Size of /dev/shm
--cpus=1.5 how much of the available CPU resources a container can use.
--cpuset-cpus=0,1,2: Limit the specific CPUs or cores a container can use
--gpus device=0,2: nvidia gpu access
--pids-limit: Limit number of processes started inside docker container
--ulimit <options>, eg. --ulimit nproc=256:512
https://docs.docker.com/config/containers/resource_constraints/

27. Stopping a fork bomb
$ docker run --rm --name aaa --pids-limit 30 ubuntu
bash -c ":() { : | : & }; :; while [[ true ]]; do sleep 1; done"
environment: fork: retry: Resource temporarily unavailable
environment: fork: retry: Resource temporarily unavailable
...
bash: fork: retry: Resource temporarily unavailable
environment: fork: retry: Resource temporarily unavailable
environment: fork: retry: Resource temporarily unavailable
environment: fork: Resource temporarily unavailable
bash: fork: Interrupted system call

28. Stopping a fork bomb #2
$ docker stats aaa
NAME CPU % MEM USAGE / LIMIT MEM % PIDS
aaa 0.14% 9.172MiB / 23.41GiB 0.04% 30

29. Make the root fs read-only
$ docker run --rm -ti --read-only ubuntu bash
root@4f8d760aa70b:/# touch /tmp/iii
touch: cannot touch '/tmp/iii': Read-only file system
root@4f8d760aa70b:/#

30. Use tmpfs to allow write access
$ docker run --rm -ti --read-only --tmpfs /tmp ubuntu bash
root@e28b09f46878:/# touch /tmp/akaka
root@e28b09f46878:/#

31. Remove all capabilities ...
$ docker run --rm --cap-drop=ALL nginx
2019/09/22 09:37:45 [emerg] 1#1:
chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not
permitted)
nginx: [emerg] chown("/var/cache/nginx/client_temp", 101) failed (1:
Operation not permitted)
man 7 capabilities

32. … add only what's required
$ docker run --rm --cap-add=chown --cap-drop=ALL nginx
2019/09/22 09:39:28 [emerg] 1#1: bind() to 0.0.0.0:80 failed (13:
Permission denied)
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

33. … add only what's required #2
$ docker run --rm --cap-add=chown --cap-add=net_bind_service --
cap-drop=ALL nginx
2019/09/22 09:43:22 [emerg] 6#6: setgid(101)failed (1: Operation not
permitted)
2019/09/22 09:43:22 [alert] 1#1: workerprocess 6 exitedwithfatal code
2 and cannot be respawned

34. … add only what's required #3
$ docker run --rm --cap-add=chown --cap-add=net_bind_service --
cap-add=setgid --cap-drop=ALL nginx
2019/09/22 09:43:54 [emerg] 6#6: setuid(101)failed (1: Operation not
permitted)
2019/09/22 09:43:54 [alert] 1#1: workerprocess 6 exitedwithfatal code
2 and cannot be respawned

35. … add only what's required #4
$ docker run --rm
--cap-add=chown
--cap-add=net_bind_service
--cap-add=setgid
--cap-add=setuid
--cap-drop=ALL
nginx

36. User remapping
$ docker run --rm -ti -v /etc:/etc ubuntu bash
root@6ac62e5eb40c:/# touch /etc/hello-world
root@6ac62e5eb40c:/# exit
$ ls -la /etc/hello-world
-rw-r--r-- 1 root root 0 Sep 22 11:53 /etc/hello-world

37. User remapping #2
$ whoami
john
$ id -u
1000
$id -g
100

38. User remapping #3
/etc/docker/daemon.json:
{
"userns-remap": "john"
}
/etc/subuid:
john:1000:65536
/etc/subgid:
john:100:65536

39. User remapping #4
$ docker run --rm -ti -v /etc:/etc ubuntu bash
root@deb50f4847e6:/# touch /etc/hello-world2
touch: cannot touch '/etc/hello-world2': Permission denied

40. User remapping #5
$ docker run --rm -ti -v /tmp:/tmp ubuntu bash
root@7b66cc086eb4:/# touch /tmp/aaa
root@7b66cc086eb4:/# ls -la /tmp/aaa
-rw-r--r--1 root root 0 Sep 22 10:13 /tmp/aaa
root@7b66cc086eb4:/# exit
$ ls -la /tmp/aaa
-rw-r--r-- 1 john users 0 Sep 22 12:13 /tmp/aaa
https://ilya-bystrov.github.io/posts/docker-daemon-remapping/

41. Don't use privileged mode
"Privileged mode enables access to all deviceson the host as wellas
set some configuration in AppArmor or SELinux to allow the container
nearly all the same access to the host as processes running outside
containers on the host."

42. Don't use the host's namespaces
$ docker run --userns=host -ti --rm -v /tmp:/tmpubuntu bash
root@a78119823836:/# touch /tmp/hahaha
root@a78119823836:/# ls -la /tmp/hahaha
-rw-r--r--1 root root 0 Oct 3 10:08 /tmp/hahaha
root@a78119823836:/# exit
$ ls -la /tmp/hahaha
-rw-r--r-- 1 root root 0 Oct 3 12:08 /tmp/hahaha

43. Authorization plugin
dockerd --authorization-plugin=someplugin
Could run locally on a Unix domain socket, or anywhere on http(s)

44. Authorization plugin #2
https://docs.docker.com/engine/extend/plugins_authorization/
PoC implementation: https://pastebin.com/SFUWdP08

45. Secrets in container
$ docker run --rm –ti –e SOME_PASSWORD=aaaa ubuntu bash
root@7b66cc086eb4:/#echo $SOME_PASSWORD
aaaa
root@7b66cc086eb4:/#
Environments are often logged!
#less /proc/29487/task/29487/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin^@HOSTN
AME=7b66cc086eb4^@TERM=xterm^@SOME_PASSWORD=aaaa^@H
OME=/root

46. Secrets in config files in container
docker run –v /path/to/1.cfg:/etc/yourapp/1.cfg:royourimage

47. Docker secrets
Container orchestrationsystems offer some basic secret management
Kubernetes: secrets, configmaps (Namespaces, RBAC)
Docker Swarm: secrets
Not for a standalone docker installation :-(
Setup a single node swarm or k8s deployment:-)
https://www.hashicorp.com/resources/securing-container-secrets-vault

48. Elevating privileges
FROM ubuntu:latest
RUN apt-get update &&
apt-get install –y sudo &&
echo "sj ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/sj &&
echo "sj:x:1000:100::/home/sj:/bin/bash" >> /etc/passwd
USER 1000

49. Elevating privileges #2
$ docker run --rm -ti aaa bash
sj@177cd44c70c0:/$ id
uid=1000(sj) gid=100(users) groups=100(users)
sj@177cd44c70c0:/$ sudo bash
root@177cd44c70c0:/# id
uid=0(root) gid=0(root) groups=0(root)

50. Elevating privileges #3
/etc/docker/daemon.json:
{
"no-new-privileges": true
}

51. Elevating privileges #4
$ docker run --rm -ti aaa bash
sj@177cd44c70c0:/$ id
uid=1000(sj) gid=100(users) groups=100(users)
sj@177cd44c70c0:/$ sudo bash
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the
'nosuid' option set or an NFS file system without root privileges?

52. More daemon.json settings
{
"icc": false, // Disable inter container communication
"userland-proxy": false, // Disable userland proxy for loopback traffic
….
}

53. Activity monitoring with sysdig/falco
Notify other systems or humans of abnormal behavior.
https://sysdig.com/opensource/falco/

54. Activity monitoring with sysdig/falco #2
***Actionchange_thread_namespace
Calling setns() to change namespaces...
***Actioncreate_files_below_dev
Creating /dev/created-by-event-generator-sh...
***Actiondb_program_spawn_process
Becomingthe program "mysql" and then running ls
***Actionexec_ls
bin dev etc …
***Actionexfiltration
Reading /etc/shadow and sending to 10.5.2.6:8197...

55. Activity monitoring with sysdig/falco #3
2019-10-03T13:17:21.968443650+0000: Notice Namespace change (setns) by
unexpected program (user=root command=event_generator
parent=<NA> <NA> (id=2f5a7b42362a) container_id=2f5a7b42362a
image=<NA>)
2019-10-03T13:17:22.968679872+0000: Error File created below /dev by
untrusted program (user=root command=event_generator
file=/dev/created-by-event-generator-sh container_id=2f5a7b42362a
image=sysdig/falco-event-generator)
2019-10-03T13:17:23.971571824+0000: Notice Database-related program
spawned process other than itself (user=root program=ls parent=mysqld
container_id=2f5a7b42362a image=sysdig/falco-event-generator)
2019-10-03T13:17:24.972983032+0000: Warning Sensitive file opened for
reading by non-trusted program (user=root program=event_generator
command=event_generator file=/etc/shadow parent=<NA>
gparent=<NA> ggparent=<NA> gggparent=<NA>
container_id=2f5a7b42362a image=sysdig/falco-event-generator)

56. Host encryption
Don't use fscrypt: no support for namespaces
Use LUKS

57. Final words
Apply what makes sense in your environment
At the end of the day security must not kill productivity