Keeping Your Kubernetes Cluster Secure

@OtherDevOpsGene #AllThingsOpen
Keeping your
Kubernetes Cluster
Secure
Gene Gotimer
@OtherDevOpsGene
WEDNESDAY, NOVEMBER 2, 2022

Define and Design the Optimal Survey Experience
KUBERNETES SECURITY
Layers
• Infrastructure
• Hosts
• Cluster
• Build
• Application
• Container images
• Deployment code
• Runtime
• Policies
• Resources
• Network
• System calls and interaction
2

Infrastructure
Build
Runtime
Wrap-up
3

INFRASTRUCTURE
Hardening
Kubernetes Hardening Guidance,
National Security Agency (NSA) and
Cybersecurity and Infrastructure Security Agency (CISA).
• Start with the kubernetes.io article
Kubernetes Security Technical Implementation Guide,
• Start with the stigviewer.com client.
CIS Kubernetes Benchmark,
Center for Internet Security (CIS),
non-government, non-profit.
https://www.cisecurity.org/benchmark/kubernetes/
4

Cluster
configuration
INFRASTRUCTURE
5
Is my Kubernetes cluster installed and configured properly?
• Use Aqua kube-bench
• May not apply to master nodes

INFRASTRUCTURE
6
Cluster
configuration
$ kubectl logs kube-bench-kc82n
[INFO] 3 Worker Node Security Configuration
[INFO] 3.1 Worker Node Configuration Files
[PASS] 3.1.1 Ensure that the kubeconfig file permissions are set to 644 or more restrictive (Manual)
[PASS] 3.1.2 Ensure that the kubelet kubeconfig file ownership is set to root:root (Manual)
[PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Manual)
[PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Manual)
[INFO] 3.2 Kubelet
[PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
[PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Manual)
[PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Manual)
[PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual)
[PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated)
[PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated)
[PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Manual)
[WARN] 3.2.9 Ensure that the --eventRecordQPS argument is set to 0 or a level which ensures appropriate event capture
(Automated)
[PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Manual)
[PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Manual)
[INFO] 3.3 Container Optimized OS
[WARN] 3.3.1 Prefer using Container-Optimized OS when possible (Manual)

Infrastructure
Build
Runtime
Wrap-up
7

Static code analysis
BUILD
8
Are resources configured properly?
• Use Checkov by Bridgecrew
• Scans source code for
• Dockerfiles
• Kubernetes manifests
• Terraform

Static code analysis
BUILD
9
$ checkov -d manifests --quiet --compact
kubernetes scan results:
Passed checks: 1066, Failed checks: 166, Skipped checks: 0
Check: CKV_K8S_20: "Containers should not run with allowPrivilegeEscalation"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_43: "Image should use digest"
Check: CKV_K8S_38: "Ensure that Service Account Tokens are only mounted where necessary"
Check: CKV_K8S_29: "Apply security context to your pods and containers"
Check: CKV_K8S_8: "Liveness Probe Should be Configured"
Check: CKV_K8S_9: "Readiness Probe Should be Configured"
Check: CKV_K8S_31: "Ensure that the seccomp profile is set to docker/default or runtime/default"

Container
image
scanning
BUILD
10
Are there vulnerabilities or misconfigurations on the
container image?
• Use Aqua Security Trivy
• Scan images for vulnerable packages
• Scan infrastructure-as-code for misconfigurations

Container
image
scanning
BUILD
11
$ trivy config manifests/01-carts-dep.yaml
2022-10-22T15:55:21.615Z INFO Misconfiguration scanning is enabled
2022-10-22T15:55:21.806Z INFO Detected config files: 1
01-carts-dep.yaml (kubernetes)
==============================
Tests: 79 (SUCCESSES: 74, FAILURES: 5, EXCEPTIONS: 0)
Failures: 5 (UNKNOWN: 0, LOW: 3, MEDIUM: 2, HIGH: 0, CRITICAL: 0)
MEDIUM: Container 'carts' of Deployment 'carts' should set 'securityContext.allowPrivilegeEscalation' to false
════════════════════════════════════════
A program inside the container can elevate its own privileges and run as root, which might give the program control over the container and node.
See https://avd.aquasec.com/misconfig/ksv001
────────────────────────────────────────
01-carts-dep.yaml:20-45
────────────────────────────────────────
20 ┌ - name: carts
21 │ image: weaveworksdemos/carts:0.4.8
22 │ env:
23 │ - name: JAVA_OPTS
24 │ value: -Xms64m -Xmx128m -XX:+UseG1GC -Djava.security.egd=file:/dev/urandom -Dspring.zipkin.enabled=false
25 │ resources:
26 │ limits:
27 │ cpu: 300m
28 └ memory: 500Mi
..
────────────────────────────────────────

Container image
scanning
BUILD
12
$ trivy image weaveworksdemos/carts:0.4.8 --no-progress
2022-10-22T15:48:40.726Z WARN This OS version is no longer supported by the distribution: alpine 3.4.6
2022-10-22T15:48:40.726Z WARN The vulnerability detection may be insufficient because security updates are not provided
weaveworksdemos/carts:0.4.8 (alpine 3.4.6)
==========================================
Total: 40 (UNKNOWN: 0, LOW: 0, MEDIUM: 23, HIGH: 13, CRITICAL: 4)
┌──────────────┬────────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Installed Version │ Fixed Version │ Title │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ busybox │ CVE-2017-16544 │ HIGH │ 1.24.2-r12 │ 1.24.2-r13 │ busybox: Insufficient sanitization of filenames when │
│ │ │ │ │ │ autocompleting │
│ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2017-16544 │
│ ├────────────────┼──────────┤ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2017-15873 │ MEDIUM │ │ │ busybox: Integer overflow in the get_next_block function │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ freetype │ CVE-2017-8105 │ CRITICAL │ 2.6.3-r0 │ 2.6.3-r1 │ freetype: heap-based buffer overflow related to the │
│ │ │ │ │ │ t1_decoder_parse_charstrings │
│ ├────────────────┤ │ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2017-8287 │ │ │ │ freetype: heap-based buffer overflow related to the │
│ │ │ │ │ │ t1_builder_close_contour function │
│ ├────────────────┼──────────┤ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2016-10244 │ HIGH │ │ │ freetype: parse_charstrings function in type1/t1load.c does │
│ │ │ │ │ │ not ensure that a font contains... │

Container
image
scanning
BUILD
13
Are there vulnerabilities on the container image?
• Use Anchore Grype
• Scan images for vulnerable packages
• Finds some different vulnerabilities than
Aqua Trivy

Container
image
scanning
BUILD
14
$ grype weaveworksdemos/carts:0.4.8
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
busybox 1.24.2-r12 apk CVE-2021-42386 High
busybox 1.24.2-r12 1.24.2-r13 apk CVE-2017-15873 Medium
busybox 1.24.2-r12 apk CVE-2018-1000517 Critical
busybox 1.24.2-r12 apk CVE-2021-42376 Medium
busybox 1.24.2-r12 1.24.2-r13 apk CVE-2017-16544 High
freetype 2.6.3-r0 2.6.3-r1 apk CVE-2016-10244 High
freetype 2.6.3-r0 apk CVE-2022-27404 Critical
freetype 2.6.3-r0 apk CVE-2022-27405 High
freetype 2.6.3-r0 2.6.3-r1 apk CVE-2017-8287 Critical
freetype 2.6.3-r0 apk CVE-2020-15999 Medium

Software
bill of materials
(SBOM)
BUILD
15
What components are in my application and
container image?
• Use Anchore Syft
• OS packages
• Libraries
• Frameworks

Software
bill of materials
(SBOM)
BUILD
16
$ syft weaveworksdemos/carts:0.4.8 --output json --file carts-0.4.8.json
$ grype sbom:carts-0.4.8.json
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
busybox 1.24.2-r12 apk CVE-2018-1000517 Critical
busybox 1.24.2-r12 1.24.2-r13 apk CVE-2017-16544 High
busybox 1.24.2-r12 1.24.2-r13 apk CVE-2017-15873 Medium
freetype 2.6.3-r0 2.6.3-r1 apk CVE-2016-10244 High
freetype 2.6.3-r0 apk CVE-2022-27405 High
...
$ syft weaveworksdemos/carts:0.4.8 --output cyclonedx-json --file carts-0.4.8-dx.json

SBOM
monitoring
BUILD
17
Do any components have newly identified
vulnerabilities?
• Use OWASP Dependency-Track
• Track SBOMs
• Identify vulnerabilities
• Notifications

SBOM
monitoring
BUILD
18

Infrastructure
Build
Runtime
Wrap-up
19

Policy
enforcement
RUNTIME
20
Are my Kubernetes workloads and resources
following my rules?
• Use Open Policy Agent
• Admissions controller
• Compliance rules

Policy
enforcement
RUNTIME
21
$ cat allowed.yaml
apiVersion: v1
kind: Pod
metadata:
name: opa-allowed
spec:
containers:
- name: opa
image: openpolicyagent/opa:0.9.2@sha256:04ff8fce2afd1a3bc26260348e5b290e8d945b1fad4b4c16d22834c2f3a1814a
args:
- "run"
- "--server"
- "--addr=localhost:8080“
$ kubectl apply -f allowed.yaml
pod/opa-allowed created

Policy
enforcement
RUNTIME
22
$ cat disallowed.yaml
apiVersion: v1
kind: Pod
metadata:
name: opa-disallowed
spec:
initContainers:
- name: opainit
image: openpolicyagent/opa:0.9.2
args:
- "run"
- "--server"
- "--addr=localhost:8080"
containers:
- name: opa
image: openpolicyagent/opa:0.9.2
args:
- "run"
- "--server"
- "--addr=localhost:8080“
$ kubectl apply -f disallowed.yaml
Error from server (Forbidden): error when creating "disallowed.yaml": admission webhook "validation.gatekeeper.sh"
denied the request: [container-image-must-have-digest] container <opa> uses an image without a digest
<openpolicyagent/opa:0.9.2>
[container-image-must-have-digest] initContainer <opainit> uses an image without a digest <openpolicyagent/opa:0.9.2>

Resource
limits
RUNTIME
23
Can a few containers hog too much
memory or CPU?
• Set the resource requests and limits
for memory and CPU
• Use Fairwinds Goldilocks
• Watches loads
• Makes request and limit
recommendations

Resource
limits
RUNTIME
24

RUNTIME
Network isolation
Can Kubernetes resources reach others they don’t
need to?
• Use a service mesh or CNI
• Build a network policy
• Network Policy editor
• https://networkpolicy.io
25

RUNTIME
Network isolation
26

Monitor
behavior
RUNTIME
27
Are any workloads doing something unexpected
on the system?
• Use Falco
• Watches system calls
• Privilege escalation
• Ownership and mode changes
• Unexpected network connections

Monitor
behavior
RUNTIME
28
$ kubectl logs -n falco falco-zplnz
Sat Oct 22 19:53:03 2022: Falco version: 0.33.0 (x86_64)
Sat Oct 22 19:53:03 2022: Falco initialized with configuration file: /etc/falco/falco.yaml
Sat Oct 22 19:53:03 2022: Loading rules from file /etc/falco/falco_rules.yaml
Sat Oct 22 19:53:03 2022: Loading rules from file /etc/falco/falco_rules.local.yaml
Sat Oct 22 19:53:03 2022: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Sat Oct 22 19:53:03 2022: Starting health webserver with threadiness 2, listening on port 8765
Sat Oct 22 19:53:03 2022: Enabled event sources: syscall
Sat Oct 22 19:53:03 2022: Opening capture with Kernel module
19:53:09.668585724: Warning Docker or kubernetes client executed in container (user=<NA> user_loginuid=-1
k8s.ns=gatekeeper-system k8s.pod=gatekeeper-update-crds-hook-l5zr2 container=006eacc6d95b parent=<NA> cmdline=kubectl
apply -f crds/ pid=13671 image=openpolicyagent/gatekeeper-crds:v3.10.0)
19:53:09.739647377: Notice Unexpected connection to K8s API Server from container (command=kubectl apply -f crds/
pid=13671 k8s.ns=gatekeeper-system k8s.pod=gatekeeper-update-crds-hook-l5zr2 container=006eacc6d95b
image=openpolicyagent/gatekeeper-crds:v3.10.0 connection=192.168.34.30:45214->10.100.0.1:443)
19:53:21.068878529: Notice Unexpected connection to K8s API Server from container (command=manager --port=8443 --
health-addr=:9090 --prometheus-port=8888 --logtostderr --log-denies=false --emit-admission-events=false --log-
level=INFO --exempt-namespace=gatekeeper-system --operation=webhook --enable-external-data=false --enable-generator-
resource-expansion=false --log-mutations=false --mutation-annotations=false --disable-cert-rotation=false --max-
serving-threads=-1 --tls-min-version=1.3 --metrics-backend=prometheus --operation=mutation-webhook --disable-opa-
builtin={http.send} pid=14628 k8s.ns=gatekeeper-system k8s.pod=gatekeeper-controller-manager-78b8774b7c-wknwm
container=bb34633b4f13 image=openpolicyagent/gatekeeper:v3.10.0 connection=192.168.60.49:34346->10.100.0.1:443)
19:53:21.133021316: Notice Unexpected connection to K8s API Server from container (command=manager --audit-
interval=60 --log-level=INFO --constraint-violations-limit=20 --audit-from-cache=false --audit-chunk-size=500 --
audit-match-kind-only=false --emit-audit-events=false --operation=audit --operation=status --operation=mutation-
status --logtostderr --health-addr=:9090 --prometheus-port=8888 --enable-external-data=false --enable-generator-
resource-expansion=false --metrics-backend=prometheus --disable-cert-rotation=true pid=14691 k8s.ns=gatekeeper-system
k8s.pod=gatekeeper-audit-9b7795dcf-tff44 container=c603d5981dea image=openpolicyagent/gatekeeper:v3.10.0
connection=192.168.62.41:58520->10.100.0.1:443)

Infrastructure
Build
Runtime
Wrap-up
29

WRAP-UP
Key takeaways
• Enforce the principle of least privilege.
• Keep everything up to date.
• Scan your container images frequently,
before and after deployment.
• Monitor your systems for expected and
unexpected behavior.
• And disk space.
30

WRAP-UP
Reading list
Kubernetes Hardening Guidance,
National Security Agency (NSA) and
https://media.defense.gov/2021/Aug/03/2002820425/-1/-
1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF
A Closer Look at NSA/CISA Kubernetes Hardening Guidance,
Jim Angel, Pushkar Joglekar, and Savitha Raghunathan.
https://kubernetes.io/blog/2021/10/05/nsa-cisa-
kubernetes-hardening-guidance/
Kubernetes Security Technical Implementation Guide,
https://public.cyber.mil/stigs/downloads/
CIS Kubernetes Benchmark,
Center for Internet Security (CIS),
https://www.cisecurity.org/benchmark/kubernetes/
31

WRAP-UP
Tools
Aqua Security kube-bench:
https://github.com/aquasecurity/kube-bench
Checkov by Bridgecrew: https://github.com/bridgecrewio/checkov
Aqua Security Trivy: https://github.com/aquasecurity/trivy
Anchore Grype: https://github.com/anchore/grype
Anchore Syft: https://github.com/anchore/syft
OWASP Dependency-Track: https://dependencytrack.org
Open Policy Agent: https://www.openpolicyagent.org
Fairwinds Goldilocks: https://github.com/fairwindsops/goldilocks
Network Policy Editor: https://networkpolicy.io
Falco: https://falco.org
32

Questions?
33

Keeping Your Kubernetes Cluster Secure

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Keeping Your Kubernetes Cluster Secure

Similar to Keeping Your Kubernetes Cluster Secure (20)

More from Gene Gotimer

More from Gene Gotimer (20)

Recently uploaded

Recently uploaded (20)

Keeping Your Kubernetes Cluster Secure

Editor's Notes