This talk will focus on a possible privilege escalation to bypass RBAC rules when running privileged containers without any security policies in place. We will also do a live demo and show how this can be achieved in AWS EKS cluster. Afterwards we will show how to remediate this using PodSecurityPolicies and what to watch for when implementing those in an active cluster.

2. WHO IS AFRAID OF
PRIVILEGED CONTAINERS?_
Marko Bevc

3. CONTAINER
BOUNDARIES_
https://sysdig.com/blog/sysdig-2020-container-security-snapshot/
“By default, root inside the
container runs as root on the host.”
— @lizrice, about Docker

4. Short Docker demo...

5. ABOUT
ME_ ●
Senior IT Consultant at The Scale Factory (DevOps consultancy,
AWS advanced consulting partner and K8s service provider)
●
IT system engineering and design background with extensive
Linux and virtualization experience
●
Certifications and competencies: AWS, CKA, RHEL, HCTA
●
Open source contributor and supporter
●
Fan of automation/simplifying things, hiking, cycling and travelling

6. TOPICS
COVERED_ ●
Containers and orchestration
●
Security mechanisms
●
Escalation methods and remediation
●
Demo
●
Conclusions and takeaways

7. CONTAINERS
ARCHITECTURE_
container
image
pull
CONTAINER
REGISTRY
IMAGE
IMAGE
IMAGE
NAME
SPACES
KERNEL
CONTROL
GROUPS
SECURITY
MODULES
LINUX
KERNEL DRIVERS
CONTAINERISED
PROCESSESCONTAINERISED
PROCESSES
CONTAINERISED
PROCESSES
REGULAR
PROCESSESREGULAR
PROCESSES
REGULAR
PROCESSES
USERLAND
dockerd
containerd
runc
libcontainer
docker (client)
Docker

8. ORCHESTRATING
CONTAINERS_
etcd
API
SERVER
SCHEDULER
CLOUD
CONTROLLER
MANAGER
CLUSTER
CONTROLLER
MANAGER
CONTROL PLANE
CLOUD
PROVIDER
KUBELET PROXY
OPERATING
SYSTEM
VIRTUAL
MACHINE
KUBELET PROXY
OPERATING
SYSTEM
VIRTUAL
MACHINE
NODES

9. SECURITY
MECHANISMS_
• Docker:
–Rootless containers (eg. Podman)
–Drop container privileges (*_CAP), RO host paths
–Daemon endpoint exposure (socket, TCP/TLS)
–Kernel security features: AppArmor, SELinux or
seccomp
• Kubernetes:
–Secure endpoints & worker nodes
–RBAC
–CI/CD pipelines policies and scan (manifests &
containers)
–NetworkPolicy & PodSecurityPolicy

10. MOUNTING
SECRETS_
• “Boring” escalation method, but quite effective
• Context: user authorised to create resources in a namespace
• Access to all currently mounted secrets from
/var/lib/kubelet/pods/*/volumes/kubernetes.io~secret/*
• Run a pod that has bind mounts the host’s kubelet directory:
kubectl run -it --rm --restart=Never --overrides="$(cat overrides.json)"
--image=busybox sh
• Escalation from here depends on what’s running on the node →
• Can also run a DaemonSet and grab secrets from every node
• By default, nothing in EKS that allows further privilege escalation
#1

11. {
"apiVersion": "v1",
"spec": {
"containers": [
{
"name": "shell",
"image": "busybox",
"command": ["sh"],
"stdin": true,
"stdinOnce": true,
"tty": true,
"volumeMounts": [
{
"name": "kubelet-volume",
"mountPath": "/var/lib/kubelet"
}
]
}
],
"volumes": [
{
"name": "kubelet-volume",
"hostPath": {
"path": "/var/lib/kubelet"
}
}
]
}
}
overrides.json

12. BYPASS
RBAC_
●
More interesting/scary escalation method
●
Context: user authorised to create resources in a
namespace and has cluster API access
●
Run a Pod with a common modern Linux distribution as
container image
●
Escape the container
●
Spoiler alerts (DEMO):
– We can elevate our way to node root
– Check EC2 instance user-data
– Inspect containers running on that node
– Grab private key material and Kubelet’s kubeconfig
– Use that for cluster access and gain full control
#2

13. RESOLVE AND
PREVENT_
• Limit unsafe features using PodSecurityPolicy (also other options)
• Pods validated against this PSP are unable to use hostPath
mounting, hostNetwork or privileged mode
• Need a ClusterRole and a ClusterRoleBinding assigned to
user/group that can use this PSP
• Ensures previous escalations are not possible
• AWS EKS default: eks.privileged – amend with caution, as it can
break your cluster!
• Further hardening/locking: make container filesystem read-only,
prevent running as root and drop all capabilities

14. DEMO
CONTEXT_
●
Running currently latest AWS EKS cluster 1.17
●
Managed node groups (or unmanaged)
●
Worker nodes only on private subnets
●
RBAC enabled cluster wide and in place
●
User has limited access: constrained to team
namespace, using RBAC
Now let’s try and escalate our privileges to gain more
control...

15. TIME FOR
DEMO!_

16. CONCLUSIONS_
& TAKEAWAYS
●
Container orchestration platforms are nice (batteries included), but:
– still need to take care of security
– shared security model in the Cloud
●
Out of the box is usually* remote-exploit safe
●
The basics:
– Protect your crown jewels: control plane and worker nodes
– Rootless containers; least privilege Pods, change defaults (EC2
credentials access)
– Enforce pipelines for changes (no kubectl write access)
– Limit blast radius using namespaces
– Scan for vulnerabilities and check for updates
●
Kubernetes RBAC is complex and not enough on its own
●
Everyone should be afraid of privileged pods!

17. ●
Resources:
– https://kubernetes.io/docs/concepts/policy/pod-security-policy/
– https://kubernetes.io/docs/concepts/security/
– https://aws.amazon.com/blogs/opensource/using-pod-security-policies-amazon-eks-clusters/
– https://docs.aws.amazon.com/eks/latest/userguide/restrict-ec2-credential-access.html
– https://docs.aws.amazon.com/eks/latest/userguide/security.html
– https://docs.giantswarm.io/guides/securing-with-rbac-and-psp/
– https://github.com/kris-nova/public-speaking/tree/master/slides/clusterfuck-fosdem-2020
FURTHER
READING_

18. KEEP IN
TOUCH_
https://www.scalefactory.com/
@_MarkoB
@mbevc1
@mbevc1
https://www.linkedin.com/in/marko-bevc/
https://www.scalefactory.com/Web:
Twitter:
GitHub:
GitLab:
LinkedIn: