Everyone heard about Kubernetes. Everyone wants to use this tool. However, sometimes we forget about security, which is essential throughout the container lifecycle. Therefore, our journey with Kubernetes security should begin in the build stage when writing the code becomes the container image. Kubernetes provides innate security advantages, and together with solid container protection, it will be invincible. During the sessions, we will review all those features and highlight which are mandatory to use. We will discuss the main vulnerabilities which may cause compromising your system. Contacts: LinkedIn - https://www.linkedin.com/in/vshynkar/ GitHub - https://github.com/sqerison ------------------------------------------------------------------------------------- Materials from the video: The policies and docker files examples: https://gist.github.com/sqerison/43365e30ee62298d9757deeab7643a90 The repo with the helm chart used in a demo: https://github.com/sqerison/argo-rollouts-demo Tools that showed in the last section: https://github.com/armosec/kubescape https://github.com/aquasecurity/kube-bench https://github.com/controlplaneio/kubectl-kubesec https://github.com/Shopify/kubeaudit#installation https://github.com/eldadru/ksniff Further learning. A book released by CISA (Cybersecurity and Infrastructure Security Agency): https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF O`REILLY Kubernetes Security: https://kubernetes-security.info/ O`REILLY Container Security: https://info.aquasec.com/container-security-book Thanks for watching!

1. Kubernetes & Container Security
by Volodymyr Shynkar
Senior Lead DevOps Engineer
2021 | intellias.com

2. [devops@stage ~]$ cat ABOUT_ME.md
• 6+ years of commercial DevOps experience. Overall 8+ years of
Engineering
• Member of Technology Office
• Member of the Center of Excellence
• Successfully migrated, rolled out, consulted over 15 projects in
the healthcare, gambling, automotive, e-commerce industries
• Certified SAFe Agile Software Engineer
• Addicted to IoT and Smart Home
• Cyclist, promoter of a healthy lifestyle
Volodymyr Shynkar
Senior Lead DevOps Engineer at Intellias
[devops@stage ~]$

3. Agenda Overview
Container Security
Kubernetes Security
Other Tooling to help

4. 01
Overview

5. Will talk about:
• Scan containers and Pods for vulnerabilities or misconfigurations.
• Run containers and Pods with the least privileges possible.
• Use network separation to control the amount of damage a
compromise can cause.
• Use firewalls to limit unneeded network connectivity and
encryption to protect confidentiality.
• Use strong authentication and authorization to limit user and
administrator access as well as to limit the attack surface.
• Periodically review all Kubernetes settings and use vulnerability
scans to help ensure risks are appropriately accounted for and
security patches are applied.

6. Attack Surface
Analysis for:
• Cloud and Host
• Kubernetes Cluster
• Container (images and running)
Goal: Reduce the attack surface

7. Attack Surface – Cloud & Host
There are at least few things that you do yo achive base
security level:
• Run instances in private network
• Expose services only through external services like LB or Proxy
• Block all external traffic except exposed ports, like 80, 443
• Do not expose SSH. Try to use SSM instead.
• Minimize privilege to applications running on the host
• Optional: for HTTP traffic, use WAF if posiable
Goal: Follow “Principle of least privilege”
Managed K8s services already goes with most of this features enabled

8. 02
Container Security

9. Dockerfile best practices
1.Avoid unnecessary privileges.
1. Avoid running containers as root.
2. Don’t bind to a specific UID.
3. Make executables owned by root and not writable.
2.Reduce attack surface.
1. Leverage multistage builds.
2. Use distroless images, or build your own from scratch.
3. Update your images frequently.
4. Watch out for exposed ports.
3.Prevent confidential data leaks.
1. Never put secrets or credentials in Dockerfile instructions.
2. Prefer COPY over ADD.
3. Be aware of the Docker context, and use .dockerignore.
4.Others.
1. Reduce the number of layers, and order them intelligently.
2. Add metadata and labels.
3. Leverage linters to automatize checks.
4. Scan your images locally during development.
5.Beyond image building.
1. Protect the docker socket and TCP connections.
2. Sign your images, and verify them on runtime.
3. Avoid tag mutability.
4. Don’t run your environment as root.
5. Include a health check.
6. Restrict your application capabilities. Source: https://sysdig.com/blog/dockerfile-best-practices/

10. Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the container - only
app, no source code, and build dependencies
• no backed secrets or bind through volume or
encrypted
Seriously :)

11. Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the container - only
app, no source code, and build dependencies
• no backed secrets or bind through volume or
encrypted
Serious :)

12. Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the container - only
app, no source code, and build dependencies
• no backed secrets or bind through volume or
encrypted
Seriously :)

13. Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the container - only
app, no source code, and build dependencies
• no backed secrets or bind through volume or
encrypted
Seriously :)
More examples:
https://github.com/GoogleContainerTools/distroless

14. Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the
container - only app, no source code, and
build dependencies
• no backed secrets or bind through volume or
encrypted
Seriously :)

15. Let’s start from scratch
First steps
Start from your app
• unprivileged user (rootless)
• read-only
• no shell, cat, grep, less, tail, echo, etc
• focus on fewer data stored inside the container - only
app, no source code, and build dependencies
• no backed secrets or bind through volume or
encrypted
Seriously :)

16. Scan your image
Docker and Snyk recently entered into a partnership to
provide container vulnerability scanning.
Alternative
Source: https://www.docker.com/blog/bringing-docker-scan-to-linux/

17. 03
Kubernetes Security

18. Next to discuss:
• Deployment to the cluster
• Pod Security Policy
• Open Policy Agent
• Network Policy
• Secrets
• Securing the Cluster
Kubernetes Hardening
Remember: follow the “Principle of least privilege”

19. Deployment to the cluster
kubectl apply –f my_app.yaml
helm install my_app charts/my_app

20. How to automate deploy?
There are two approaches:
• Push-based
• Regular CI (Jenkins, Gitlab, GitHub, CircleCI)

21. How to automate deploy?
There are two approaches:
• Pull-based
• GitOps strategy (ArgoCD, Flux)

22. How to manage manage
By only two resources: AppProject and Application

23. By only two resources: AppProject and Application
How to manage manage

24. How to grant access
ArgoCD provided with powerful UI
• Embedded security features
• Support of SAML/OKTA.
• Enhanced experience
• Role-based
• Easy to use

25. Pod Security Policy
PSP are one way to control the security-related attributes of pods,
including container privilege levels.
• Do not run application processes as root
• Do not allow privilege escalation
• Use a read-only root filesystem
• Use the default (masked) /proc filesystem mount
• Do not use the host network or process space
• Drop unused and unnecessary Linux capabilities
• Service Account control

26. Pod Security Policy
When a PSP resource is created, it does nothing. You need to
authorize using RBAC!

27. Pod Security Policies is going to be marked as
deprecated since v1.21 and will be removed in v1.25
The PSP Replacement Policy is the new enhancement:
https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/2579-psp-replacement
PSP Replacement Policy
The way PSPs are applied
to Pods has proven
confusing to nearly everyone
that has attempted to use
them.

28. Open Policy Agent (Gatekeeper)
Gatekeeper controller provides you ability to:
• Required labels
• Required resources section
• Mutate container images to always point to the corporate image registry.
• Set node and pod affinity and anti-affinity selectors to Deployments
• You can enforce anything that you want to see or not to see in configs
I need to control other pod fields or
any fields in other resource? How
can I achieve that?
The answer is:

29. Open Policy Agent (Gatekeeper)
Example: enforce to use only allowed container registries

30. Open Policy Agent (Gatekeeper)
Example: enforce to use only allowed container registries
The other examples can be found here:
https://github.com/open-policy-agent/gatekeeper/tree/master/demo

31. Network Policy
By default, namespaces are not automatically isolated.
For that we have networking policies and RBAC.
With this simple config you will
isolate NS from other NS

32. Network Policy
Network policy will not work with default CNI from
AWS EKS. You need to install Calico.
With this simple config you will allow
traffic from specific namespace

33. Network Policy
With AWS EKS CNI you can use different security
groups per pod which makes network policy redundant
With this simple config you will allow
traffic to a specific port
You can also specify protocol.

34. Secrets
Where do I version control my secrets?
What is solution for this?
• Paper/USB/CDR two fireproof safes?
• Vault (or actually Consul)?

35. Secrets
Where do I version control my secrets?
What is solution for this?
• Sealed Secrets (a Kubernetes controller and a tool for
one-way encrypted Secrets):​
https://github.com/bitnami-labs/sealed-secrets
• Git Crypt - transparent file encryption in git:​
https://www.agwa.name/projects/git-crypt/

36. Sealed Secrets example
As you can see the value of the secret is enrypted

37. Sealed Secrets example
The controller will unseal that into something like:

38. Securing the Cluster
API Server
By default, the API server will listen on what is rightfully
port 8080
• Close the insecure port by arg in API server’s --insecure-port flag to 0
• And --insecure-bind-address is not set.
Any requests to this port bypass authentication and authorization
checks.

39. Securing the Cluster
etcd
The etcd backend database is a critical component and the
to secure within the cluster.
Close the insecure port by arg in API server’s --insecure-port flag to
0
• And --insecure-bind-address is not set.
etcd server should be configured to only trust certificates assigned to API
servers

40. Securing the Cluster
Kubelet
The kubelet is the agent that is responsible launching pods (not
Check this parameters:
• Disable anonymous access with --anonymous-auth=false
• Ensure that requests are authorized by setting --
other than AlwaysAllow

41. Securing the Cluster
Kubernetes Dashboard
The Dashboard has historically been used by attackers to gain
Kubernetes clusters.
Check this parameters:
• Allow only authenticated access. Only known users should be able
Dashboard.
• Use RBAC. Limit the privileges that users have so they can
they need to.
• Don’t expose your Dashboard to the public internet
• Unless you really know what you’re doing.

42. Securing the Cluster
Protocol Direction Port Range Purpose
TCP Inbound 6443 or 8080 if not disabled Kubernetes API server
TCP Inbound 2379-2380 etcd server client API
TCP Inbound 10250 kubelet API
TCP Inbound 10251 kube-scheduler
TCP Inbound 10252 kube-controller-manager
TCP Inbound 10258 cloud-controller-manager (optional)
The following table lists the control plane ports and services.
You can try to curl on each port to check if it’s secured.

43. 04
Other Tooling to help

44. Kubescape
Kubescape is the first tool for testing if Kubernetes deployed securely
Source: https://github.com/armosec/kubescape

45. Kube-bench
The same tool as Kubescape but could be deployed as CronJob and executed
on regular bases
Source: https://github.com/aquasecurity/kube-bench

46. Kubesec
kubectl plugin for scanning Kubernetes pods,
deployments, daemonsets and statefulsets
Source: https://github.com/controlplaneio/kubectl-kubesec
Suggests what should be improved or changed

47. Kubeaudit
kubeaudit is a command line tool to audit Kubernetes clusters for various security
concerns, such as:
• run as non-root
• use a read-only root filesystem
• drop scary capabilities, don't add new ones
• don't run privileged
Source: https://github.com/Shopify/kubeaudit

48. Ksniff
A kubectl plugin that utilize tcpdump and Wireshark to start a remote capture
on any pod in your Kubernetes cluster.
Source: https://github.com/eldadru/ksniff

49. Demo Time

50. Demo Time

51. THANK
YOU!
Let's get connected!