How to Prevent Your Kubernetes Cluster From Being Hacked

How to Prevent Your Kubernetes Cluster
From Being Hacked
ContainerDays Meetup 2022

Who we are
© white duck GmbH 2022
Nico Meisenzahl (Head of DevOps Consulting and Operations,
Cloud Solution Architect, Azure & Developer Technologies MVP, GitLab Hero)
Email: nico.meisenzahl@whiteduck.de
Twitter: @nmeisenzahl
LinkedIn: https://www.linkedin.com/in/nicomeisenzahl/
Philip Welz (Senior Kubernetes & DevOps Engineer,
GitLab Hero, CKA, CKAD & CKS)
Twitter: @philip_welz
LinkedIn: https://www.linkedin.com/in/philip-welz

Security quick wins through the DevOps cycle

You should think about
• ensure secure application / deployment code
• build secure container images
• implement Kubernetes policies
• introduce Kubernetes network policies
• rely on Container Runtime Security
• many more…

Ensure secure application / deployment code
• your cluster is a shared resource
• a single compromised application can be enough
• automate and enforce code checks (shift left!)
• scan your code as well as your dependencies
• application code
• Dockerfiles
• deployment manifests

Examples
• vulnerable code and dependencies
• missing Dockerfile best practices
• unsecure Kubernetes manifests
They all can lead to someone gaining access to your
cluster!

SAST in general
• Static Application Security Testing (SAST) can help
validate you with
• application code
• Dockerfiles
• Kubernetes Manifests
• and more
• use your existing tooling (GitHub, GitLab) for easy start
• there is also a big ecosystem available

Software Bill of Materials (SBOM)
• “list of ingredients” for all your software and dependencies
• supports hierarchy and therefore multi-level dependencies
• without you don’t have the full visibility
• in an ideal world you would only need to care about your
own stuff
• SBOMs can be the baseline for your vulnerability
scanning

Tooling: syft and grype
• syft helps you to create Software Bill of Materials (SBOM)
• https://github.com/anchore/syft
• can also be used via “docker sbom”
• grype allows to scan SBOMs for vulnerability
• https://github.com/anchore/grype
• both
• support a large list of programming languages and ecosystems
• are easy to integrate into CI
• can be integrated with K8s via Operators
• https://github.com/ckotzbauer/sbom-operator
• https://github.com/ckotzbauer/vulnerability-operator

Demo: Vulnerability scanning with SBOM
• we will create an SBOM for an existing container image
using syft
• then analyzing the exported SBOM with grype to detect
possible vulnerabilities

Further K8s-related tooling
• Ensure best practices and detect misconfiguration on clusters
• kube-bench: https://github.com/aquasecurity/kube-bench
• Kubescape: https://github.com/kubescape/kubescape
• Popeye: https://github.com/derailed/popeye
• Scan Dockerfiles, K8s manifests or Helm Charts
• Checkov: https://github.com/bridgecrewio/checkov
• Trivy: https://github.com/aquasecurity/trivy
• kics: https://github.com/Checkmarx/kics

Implement Kubernetes policies
• allows you to enforce what is allowed and what is not
• enables you to define guardrails for your user
• can limit attackers from gaining further access

Examples
• attacker spins up a privileged container to escape
• someone mounts the nodes disk via hostPath
• then gains access to the secret of the underlying cloud identity
• exposing a service via LoadBalancer service

Tooling: Built-in, OPA or Kyverno
• Pod Security Admission
• stable since 1.25
• admission controller to enforce the Pod Security Standards
• define different isolation levels (Profiles) for Pods
• Open Policy Agent Gatekeeper
• “jack of all trades”
• can become quite complex – Rego J
• https://open-policy-agent.github.io/gatekeeper
• Kyverno
• easy to start
• Kubernetes-only (which can be a plus)
• https://kyverno.io

Introduce Kubernetes network policies
• once again: K8s is a shared resource
• you need to define guard rails
• think about
• outgoing traffic into the internet
• traffic between applications and namespaces
• traffic into your internal network and/or cloud resources

Examples
• malicious code gets loaded from the internet
• attack talks to the Cloud provider matadata service
• and retrieve cloud identity secret
• vulnerable application has access to other apps backends
• and retrieves, dumps, changes data

Tooling: There are many…
• grab the one that suits your needs
• go for managed one in managed Clusters
• otherwise: Cilium or Calico
• Cilium
• based on eBPF
• allows Domain name-based policies
• https://editor.cilium.io J
• https://cilium.io
• Calico
• supports standard Linux networking or eBPF "now"
• Domain name-based policies only via Calico Enterprise
• widely adopted (AKS, GKE,…)

Rely on Container Runtime Security
• gives you awareness into your cluster
• without that you won't know what is going on
• alerts you on malicious events and workloads
• based on the defined rules
• real-time enforcement

Examples
• undetected untrusted process within container
• Untrusted shells running inside a container
• container process mounting a sensitive path
• a process making outbound network connections

Tooling: Falco or Tetragon
• Falco
• parses Linux system calls from the kernel at runtime
• “the known one”
• https://falco.org
• Tetragon
• eBPF-based analysis
• Process execution events
• System call activity
• I/O activity including network & file access
• “the new one”
• https://github.com/cilium/tetragon

Demo: Tetragon
• we will connect to privileged pod and execute some
commands and access files
• we will use Tetragon (pre-installed via Helm) and Tetragon
CLI to observe process execution, network
connections and file access

Service Account Token
• got more secure with Kubernetes 1.24
• you now must create a token when you need one
• do not
• share service accounts between applications
• enable higher access levels for the default service account if
not required
• mount service account token if not required
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-
service-account/#use-the-default-service-account-to-access-the-api-
server

Web Application Firewall (WAF)
• helps you to further secure your application
• most of them enforce OWASP 10 by default
• https://owasp.org/www-project-top-ten
• implement further rules based on your needs
• put it in front of your workload as
• dedicated WAF
• as part of your Ingress Controller
• not more than 5 lines to get started with Ingress Nginx

Don’t miss your running workload
• while shifting left, don’t miss running workload
• you have multiple options
• run Trivy Operator
• https://github.com/aquasecurity/trivy-operator
• or Trivy regularly in your cluster with Estafette
• https://github.com/estafette/estafette-vulnerability-scanner
• or with Jetstack version checker
• https://github.com/jetstack/version-checker

Links & further details
• there are many more projects
• we have shown only a very small portion
• “Hijack Kubernetes” demo
• https://github.com/nmeisenzahl/hijack-kubernetes

Questions?
Nico Meisenzahl (Head of DevOps Consulting and Operations,
Cloud Solution Architect, Azure & Developer Technologies MVP, GitLab Hero)
Email: nico.meisenzahl@whiteduck.de
Twitter: @nmeisenzahl
LinkedIn: https://www.linkedin.com/in/nicomeisenzahl/
Philip Welz (Senior Kubernetes & DevOps Engineer,
GitLab Hero, CKA, CKAD & CKS)
Twitter: @philip_welz
LinkedIn: https://www.linkedin.com/in/philip-welz

How to Prevent Your Kubernetes Cluster From Being Hacked

Recommended

Recommended

More Related Content

Similar to How to Prevent Your Kubernetes Cluster From Being Hacked

Similar to How to Prevent Your Kubernetes Cluster From Being Hacked (20)

More from Nico Meisenzahl

More from Nico Meisenzahl (14)

Recently uploaded

Recently uploaded (20)

How to Prevent Your Kubernetes Cluster From Being Hacked