KCD Munich 2022: How to Prevent Your Kubernetes Cluster From Being Hacked

SPONSORS
Speakers: Philip Welz, Nico Meisenzahl
Company: white duck
How to Prevent Your
Kubernetes Cluster From
Being Hacked

Who we are
© white duck GmbH 2022
Nico Meisenzahl (Head of DevOps Consulting and Operations,
Cloud Solution Architect, Azure & Developer Technologies MVP, GitLab Hero)
Email: nico.meisenzahl@whiteduck.de
Twitter: @nmeisenzahl
LinkedIn: https://www.linkedin.com/in/nicomeisenzahl/
Philip Welz (Senior Kubernetes & DevOps Engineer,
GitLab Hero, CKA, CKAD & CKS)
Twitter: @philip_welz
LinkedIn: https://www.linkedin.com/in/philip-welz

Security quick wins through the DevOps cycle

You should think about
• ensure secure application / deployment code
• build secure container images
• implement Kubernetes policies
• introduce Kubernetes network policies
• rely on Container Runtime Security
• many more…

Ensure secure application / deployment code
• your cluster is a shared resource
• a single compromised application can be enough
• automate and enforce code checks (shift left!)
• scan your code as well as your dependencies
• application code
• Dockerfiles
• deployment manifests

Examples
• vulnerable code and dependencies
• missing Dockerfile best practices
• unsecure Kubernetes manifests
They all can lead to someone gaining access to your
cluster!

SAST in general
• Static Application Security Testing (SAST) can help
validate you with
• application code
• Dockerfiles
• Kubernetes Manifests
• and more
• use your existing tooling (GitHub, GitLab) for easy start
• there is also a big ecosystem available

Software Bill of Materials (SBOM)
• “list of ingredients” for all your software and dependencies
• supports hierarchy and therefore multi-level dependencies
• without you don’t have the full visibility
• in an ideal world you would only need to care about your
own stuff
• SBOMs can be the baseline for your vulnerability
scanning

Tooling: syft and grype
• syft helps you to create Software Bill of Materials (SBOM)
• https://github.com/anchore/syft
• can also be used via “docker sbom”
• grype allows to scan SBOMs for vulnerability
• https://github.com/anchore/grype
• both
• support a large list of programming languages and ecosystems
• are easy to integrate into CI
• can be integrated with K8s via Operators
• https://github.com/ckotzbauer/sbom-operator
• https://github.com/ckotzbauer/vulnerability-operator

Demo: Vulnerability scanning with SBOM
• we will create an SBOM for an existing container image
using syft
• then analyzing the exported SBOM with grype to detect
possible vulnerabilities

Further K8s-related tooling
• Ensure best practices and detect misconfiguration on clusters
• kube-bench: https://github.com/aquasecurity/kube-bench
• Kubescape: https://github.com/kubescape/kubescape
• Popeye: https://github.com/derailed/popeye
• Scan Dockerfiles, K8s manifests or Helm Charts
• Checkov: https://github.com/bridgecrewio/checkov
• Trivy: https://github.com/aquasecurity/trivy
• kics: https://github.com/Checkmarx/kics

Implement Kubernetes policies
• allows you to enforce what is allowed and what is not
• enables you to define guardrails for your user
• can limit attackers from gaining further access

Examples
• attacker spins up a privileged container to escape
• someone mounts the nodes disk via hostPath
• then gains access to the secret of the underlying cloud identity
• exposing a service via LoadBalancer service

Tooling: Built-in, OPA or Kyverno
• Pod Security Admission
• stable since 1.25
• admission controller to enforce the Pod Security Standards
• define different isolation levels (profiles) for Pods
• Open Policy Agent Gatekeeper
• “jack of all trades”
• can become quite complex – Rego J
• https://open-policy-agent.github.io/gatekeeper
• Kyverno
• easy to start
• Kubernetes-only (which can be a plus)
• https://kyverno.io

Introduce Kubernetes network policies
• once again: K8s is a shared resource
• you need to define guard rails
• think about
• egress traffic into the internet
• east-west traffic between applications and namespaces
• traffic into your internal network and/or cloud resources

Examples
• malicious code gets loaded from the internet
• attacker talking to the Cloud provider matadata service
• and retrieve cloud identity secret
• vulnerable application has access to other apps backends
• and dumps, changes, deletes data

Tooling: There are many…
• grab the one that suits your needs
• go for a managed solution in managed Clusters
• Cilium
• based on eBPF
• allows Domain name-based policies
• https://editor.cilium.io J
• https://cilium.io
• Calico
• supports standard Linux networking or eBPF "now"
• Domain name-based policies only via Calico Enterprise
• widely adopted (AKS, GKE,…)

Rely on Container Runtime Security
• gives you awareness into your cluster
• without that you won't know what is going on
• alerts you on malicious events and workloads
• based on the defined rules
• real-time enforcement

Examples
• undetected untrusted process within container
• Untrusted shells running inside a container
• container process mounting a sensitive path
• a process making outbound network connections

Tooling: Falco or Tetragon
• Falco
• parses Linux system calls from the kernel at runtime
• “the known one”
• https://falco.org
• Tetragon
• eBPF-based analysis
• Process execution events
• System call activity
• I/O activity including network & file access
• “the new one”
• https://github.com/cilium/tetragon

Demo: Tetragon
• we will inject into a Pod via Log4Shell
• we will use Tetragon (pre-installed via Helm) to observe
the process execution
• and finally block it

Service Account Token
• got more secure with Kubernetes 1.24
• you now must create a token when you need one
• do not
• share service accounts between applications
• enable higher access levels for the default service account if
not required
• mount service account token if not required (prior 1.24)
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-
service-account/#use-the-default-service-account-to-access-the-api-
server

Web Application Firewall (WAF)
• helps you to further secure your application
• most of them enforce OWASP 10 by default
• https://owasp.org/www-project-top-ten
• implement further rules based on your needs
• put it in front of your workload as
• dedicated WAF
• as part of your Ingress Controller
• not more than 5 lines to get started with Ingress Nginx

Don’t miss your running workload
• while shifting left, don’t miss running workload
• you have multiple options
• run Trivy Operator
• https://github.com/aquasecurity/trivy-operator
• or Trivy regularly in your cluster with Estafette
• https://github.com/estafette/estafette-vulnerability-scanner
• or with Jetstack version checker
• https://github.com/jetstack/version-checker

Links & further details
• there are many more projects
• we have shown only a very small portion
• “Hijack Kubernetes” demo
• https://github.com/nmeisenzahl/hijack-kubernetes
• Slides
• https://www.slideshare.net/nmeisenzahl

Questions?
Nico Meisenzahl (Head of DevOps Consulting and Operations,
Cloud Solution Architect, Azure & Developer Technologies MVP, GitLab Hero)
Email: nico.meisenzahl@whiteduck.de
Twitter: @nmeisenzahl
LinkedIn: https://www.linkedin.com/in/nicomeisenzahl/
Philip Welz (Senior Kubernetes & DevOps Engineer,
GitLab Hero, CKA, CKAD & CKS)
Twitter: @philip_welz
LinkedIn: https://www.linkedin.com/in/philip-welz

KCD Munich 2022: How to Prevent Your Kubernetes Cluster From Being Hacked

KCD Munich 2022: How to Prevent Your Kubernetes Cluster From Being Hacked

Recommended

More Related Content

Similar to KCD Munich 2022: How to Prevent Your Kubernetes Cluster From Being Hacked

Similar to KCD Munich 2022: How to Prevent Your Kubernetes Cluster From Being Hacked(20)

More from Nico Meisenzahl

More from Nico Meisenzahl(14)

Recently uploaded

Recently uploaded(20)

KCD Munich 2022: How to Prevent Your Kubernetes Cluster From Being Hacked