A talk given at Devoxx Morocco on Wednesday, November 13, 2019. In this talk a very insecure sample (demo) application is used to explain the various security principles application developers can apply when using containers and Kubernetes--from image sourcing, content, scanning to resource controls, attack surface mitigation, and reducing privilege for containers.
What We Won’t Cover
Hardening your host OS
Hardening your cluster
Conﬁguring container runtimes
Securing network trafﬁc
- Secure coding practices
- Protect against exploits
- No embedded secrets/keys
$ docker build -t myawesomeapp .
Host OS container runtime
Container Application Security
Tools, conﬁguration, and runtime capabilities
provided via the container ecosystem used to
wrap an existing security-focused application to
provide more isolation, protection, and overall
security for your code running inside a container.
● NO secrets
○ passwords, API keys, tokens, private keys
● FROM what?
● Minimize content
● Use “FROM SCRATCH” if possible
02 ATTACK SURFACE
As limited as is feasible.
As small as is possible.
The least amount necessary.
Runtime Security: Limiting Resources
> CPU controls
are in the OCI
> Process limit
(how many can I
create) are part
of the OCI spec.
> Memory limits
are a part of the
OCI runtime spec.
exposes them in a
less complex way.
> Disk bandwidth
limits are in the OCI
> Kubernetes does not
enable them, but has
ephemeral disk limits.
ﬁlesystem is possible
Runtime Security: Limiting Attack Surface
> Collections of similar system
> Names like CAP_NET_RAW
> Some are ﬁne grained and
some, like CAP_SYS_ADMIN,
might as well be “the new root”
Linux Security Modules
(LSMs) like AppArmor:
> provide a “language” to
describe a wide-ranging set
of permissions for processes
> container runtimes use a
custom proﬁle written for
you to limit what containers
can do on the system.
SECCOMP stands for
> Allows a proﬁle to be
associated with a process to
allow or deny speciﬁc Linux
system calls for that process.
> Container runtimes have a
default seccomp proﬁle;
custom ones are allowed.
Capabilities AppArmor Seccomp
Runtime Security: Reducing Privilege
Don’t run privileged containers.
Don’t run containers as root or elevate privileges.
> All security controls/limits are disabled when you run with privileged mode.
> Find the speciﬁc required privilege and only enable that feature/access.
> Rarely do your containerized applications truly need root privileges so don’t use it.
> User namespaces are a Linux feature in container runtimes but not yet in Kubernetes.
> Exposing the Docker socket or K8s API into your container may allow escalation to root!
Kubernetes: Controlling Resource Limits
- name: db
- name: wp
● Resource Limits
○ Set per-container in the Pod yaml
○ Note that not all OCI spec memory/CPU options
are exposed in the K8s API speciﬁcation
● Limit Processes
○ Still alpha as of Kubernetes 1.16
○ Cluster operator must enable feature gate
SupportPodPidsLimit=true, and then pass a
--pod-max-pids integer to kubelet
○ Limit is ﬁxed per-pod; no customization possible
● I/O Bandwidth Limits
○ The cgroups i/o settings are not exposed here to
be set per container.
○ K8s does offer resource quotas, and QoS
features—related but not the same features
Kubernetes: Limiting Attack Surface
- name: hello
○ Set per-container via securityContext ; can add/drop caps by name
○ Annotations are used to identify AppArmor proﬁles in Kubernetes
○ Operator must install them on worker nodes; developing new proﬁles? Tools TBD
○ Also set via annotation, but on PodSecurityPolicy; see upcoming example; not default
Kubernetes: Reducing Privilege
- name: sec-ctx-demo
command: [ "sh", "-c", "sleep 1h" ]
add: ["NET_ADMIN", "SYS_TIME"]
● Non-root user
○ Use securityContext for
containers and pod-level control
○ Use PodSecurityPolicy to enforce
● Capabilities (privilege related)
○ Also in securityContext; see
Kubernetes: Cluster Security Enforcement
# This is redundant with non-root + disallow privilege escalation:
# Require the container to run without root privileges.
Kubernetes: Applying Container Security
● PodSecurityPolicy: enforce many good practices cluster-wide! OpenShift is a good
example of a Kubernetes distribution with strong defaults out of the box
● Use the Kubernetes secrets implementation to protect sensitive keys, tokens, materials.
Vendor tools available as well (Hashicorp Vault), and potentially from your cloud provider
● Don’t circumvent security to make your code “easy”: e.g. K8s API access with admin role;
mounting container runtime (e.g. Docker) API with full privilege
● Have a unique workload requirement (multi-tenancy, untrusted code)? Take a look at
RuntimeClass features in Kubernetes to allow custom isolators (gVisor, Kata,
Firecracker, Nabla, etc.)
● Remember that you need visibility and not simply ﬁre-and-forget security! Logging,
audit, vendor tools/open source projects for runtime protection, anomaly detection, etc.
BUT...Container Security is Hard!!
● Use a cloud provider
○ Managed Kubernetes services many times can be created with a set of default tools and
policies for strong controls pre-conﬁgured for you
○ Many managed services integrate with popular vendor tooling
■ e.g. Twistlock, Snyk, Aqua, Datadog, Sysdig, LogDNA and many others
● Use recommended guides and proﬁles publicly available (CIS, NIST,
● Try out emerging tooling
○ Generate seccomp proﬁles by running your application with BPF tracing:
Kubernetes Security Concepts: https://kubernetes.io/docs/concepts/security/overview/
AppArmor documentation: https://kubernetes.io/docs/tutorials/clusters/apparmor/
Resource controls: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
Complete list of Linux capabilities: http://man7.org/linux/man-pages/man7/capabilities.7.html
Demos located at: