Many organizations are shifting to containers and Kubernetes, and that move means learning new ways to secure their environments. Kubernetes clusters have to be hardened at different levels. We have to consider the nodes where the Kubernetes control plane is running. We also need to secure the Kubernetes workloads and check the code that creates them. And we need to inspect the containers we are using for vulnerabilities and watch for unusual behavior.
Gene will show you some open-source tools that can find issues and vulnerabilities at each layer. You will see how they can be used to build your Kubernetes cluster safely and keep it secure.
4. Define and Design the Optimal Survey Experience
INFRASTRUCTURE
Hardening
Kubernetes Hardening Guidance,
National Security Agency (NSA) and
Cybersecurity and Infrastructure Security Agency (CISA).
β’ Start with the kubernetes.io article
Kubernetes Security Technical Implementation Guide,
Cybersecurity and Infrastructure Security Agency (CISA).
β’ Start with the stigviewer.com client.
CIS Kubernetes Benchmark,
Center for Internet Security (CIS),
non-government, non-profit.
https://www.cisecurity.org/benchmark/kubernetes/
@OtherDevOpsGene #AllThingsOpen
4
6. INFRASTRUCTURE
@OtherDevOpsGene #AllThingsOpen
6
Cluster
configuration
$ kubectl logs kube-bench-kc82n
[INFO] 3 Worker Node Security Configuration
[INFO] 3.1 Worker Node Configuration Files
[PASS] 3.1.1 Ensure that the kubeconfig file permissions are set to 644 or more restrictive (Manual)
[PASS] 3.1.2 Ensure that the kubelet kubeconfig file ownership is set to root:root (Manual)
[PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Manual)
[PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Manual)
[INFO] 3.2 Kubelet
[PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
[PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Manual)
[PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Manual)
[PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual)
[PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated)
[PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated)
[PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Manual)
[WARN] 3.2.9 Ensure that the --eventRecordQPS argument is set to 0 or a level which ensures appropriate event capture
(Automated)
[PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Manual)
[PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Manual)
[INFO] 3.3 Container Optimized OS
[WARN] 3.3.1 Prefer using Container-Optimized OS when possible (Manual)
8. Static code analysis
BUILD
@OtherDevOpsGene #AllThingsOpen
8
Are resources configured properly?
β’ Use Checkov by Bridgecrew
β’ Scans source code for
β’ Dockerfiles
β’ Kubernetes manifests
β’ Terraform
9. Static code analysis
BUILD
@OtherDevOpsGene #AllThingsOpen
9
$ checkov -d manifests --quiet --compact
kubernetes scan results:
Passed checks: 1066, Failed checks: 166, Skipped checks: 0
Check: CKV_K8S_20: "Containers should not run with allowPrivilegeEscalation"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_43: "Image should use digest"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_38: "Ensure that Service Account Tokens are only mounted where necessary"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_29: "Apply security context to your pods and containers"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_8: "Liveness Probe Should be Configured"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_9: "Readiness Probe Should be Configured"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
Check: CKV_K8S_31: "Ensure that the seccomp profile is set to docker/default or runtime/default"
FAILED for resource: Deployment.sock-shop.carts
File: /manifests/01-carts-dep.yaml:2-51
22. Policy
enforcement
RUNTIME
@OtherDevOpsGene #AllThingsOpen
22
$ cat disallowed.yaml
apiVersion: v1
kind: Pod
metadata:
name: opa-disallowed
spec:
initContainers:
- name: opainit
image: openpolicyagent/opa:0.9.2
args:
- "run"
- "--server"
- "--addr=localhost:8080"
containers:
- name: opa
image: openpolicyagent/opa:0.9.2
args:
- "run"
- "--server"
- "--addr=localhost:8080β
$ kubectl apply -f disallowed.yaml
Error from server (Forbidden): error when creating "disallowed.yaml": admission webhook "validation.gatekeeper.sh"
denied the request: [container-image-must-have-digest] container <opa> uses an image without a digest
<openpolicyagent/opa:0.9.2>
[container-image-must-have-digest] initContainer <opainit> uses an image without a digest <openpolicyagent/opa:0.9.2>
23. Resource
limits
RUNTIME
@OtherDevOpsGene #AllThingsOpen
23
Can a few containers hog too much
memory or CPU?
β’ Set the resource requests and limits
for memory and CPU
β’ Use Fairwinds Goldilocks
β’ Watches loads
β’ Makes request and limit
recommendations
25. Define and Design the Optimal Survey Experience
RUNTIME
Network isolation
Can Kubernetes resources reach others they donβt
need to?
β’ Use a service mesh or CNI
β’ Build a network policy
β’ Network Policy editor
β’ https://networkpolicy.io
@OtherDevOpsGene #AllThingsOpen
25
26. Define and Design the Optimal Survey Experience
RUNTIME
Network isolation
@OtherDevOpsGene #AllThingsOpen
26
30. Define and Design the Optimal Survey Experience
WRAP-UP
Key takeaways
β’ Enforce the principle of least privilege.
β’ Keep everything up to date.
β’ Scan your container images frequently,
before and after deployment.
β’ Monitor your systems for expected and
unexpected behavior.
β’ And disk space.
@OtherDevOpsGene #AllThingsOpen
30
31. Define and Design the Optimal Survey Experience
WRAP-UP
Reading list
Kubernetes Hardening Guidance,
National Security Agency (NSA) and
Cybersecurity and Infrastructure Security Agency (CISA).
https://media.defense.gov/2021/Aug/03/2002820425/-1/-
1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF
A Closer Look at NSA/CISA Kubernetes Hardening Guidance,
Jim Angel, Pushkar Joglekar, and Savitha Raghunathan.
https://kubernetes.io/blog/2021/10/05/nsa-cisa-
kubernetes-hardening-guidance/
Kubernetes Security Technical Implementation Guide,
Cybersecurity and Infrastructure Security Agency (CISA).
https://public.cyber.mil/stigs/downloads/
CIS Kubernetes Benchmark,
Center for Internet Security (CIS),
https://www.cisecurity.org/benchmark/kubernetes/
@OtherDevOpsGene #AllThingsOpen
31
32. Define and Design the Optimal Survey Experience
WRAP-UP
Tools
Aqua Security kube-bench:
https://github.com/aquasecurity/kube-bench
Checkov by Bridgecrew: https://github.com/bridgecrewio/checkov
Aqua Security Trivy: https://github.com/aquasecurity/trivy
Anchore Grype: https://github.com/anchore/grype
Anchore Syft: https://github.com/anchore/syft
OWASP Dependency-Track: https://dependencytrack.org
Open Policy Agent: https://www.openpolicyagent.org
Fairwinds Goldilocks: https://github.com/fairwindsops/goldilocks
Network Policy Editor: https://networkpolicy.io
Falco: https://falco.org
@OtherDevOpsGene #AllThingsOpen
32
We will look at 10 tools across 3 rough layers of the Kubernetes ecosystem.
All are open-source and/or freely available
Also, some publicly available guidance
Security is a type of quality
You cannot be insecure and have high quality
You cannot have low quality but high security
Kubernetes clusters consist of servers acting as master nodes and worker nodes. The operating system and processes on these servers have to be secured just like any others.
These are the tasks traditionally done by Ops and Security
YAGNI
K8s hosts need the same security as other hosts
Keep the systems up-to-date
Easiest to do. Just regularly run apt-get update or dnf update or yum update
CISA recommends the following remediation timelines:
Critical vulnerabilities should be remediated within 15 calendar days of initial detection.
High vulnerabilities should be remediated within 30 calendar days of initial detection.
Least privilege
Ronald Reagan 1986 β The nine most terrifying words in the English language are "I'm from the Government and I'm here to help."
Google both terms,
See the Kubernetes.io article from Oct 2021.
See the stigviewer.com link
Also, not specific to k8s, but Federal, state, local, tribal and territorial governments, as well as public and private sector critical infrastructure organizations. Can request Cyber Hygiene Services at no cost from CISA.
CIS is not a government agency, but it is non-profit. CIS Benchmarks are free checklists, very similar to STIGs, easier to read.
"checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark."
If you are running in a managed Kubernetes cluster, such as Amazon EKS or Azure AKS, kube-bench does not have access to the master nodes but can still evaluate the worker nodes.
Β
Runs as a kubectl job
Runs as a kubectl job
cd ~/git/ggkube/Book/code
kubectl apply -f kube-bench-job-eks.yaml
kubectl get pods
kubectl logs kube-bench-kc82n
These tasks are typically going to fall to the development teams while they are producing their software for deployment
All of this is a moot point if the application is security swiss cheese.
Least privilege
Checkov by Bridgecrew
Frequent updates, sometimes daily
I use this extensively on Terraform code
Python pip install or use Docker container
pushd ./microservices-demo/deploy/Kubernetes
checkov -d manifests --quiet --compact
Canβt just scan once, vulnerabilities can be found even in existing, previously βsafeβ, containers/code
Installs as package, from script, container, etc.
trivy config manifests/01-carts-dep.yaml
Other options are Clair by Red Hat
Canβt just scan once, vulnerabilities can be found even in existing, previously βsafeβ, containers/code
grype weaveworksdemos/carts:0.4.8
Can feed that back to Grype
Can feed that back to Grype
syft weaveworksdemos/carts:0.4.8 --output json --file carts-0.4.8.json
grype sbom:carts-0.4.8.json
syft weaveworksdemos/carts:0.4.8 --output cyclonedx-json --file carts-0.4.8-dx.json
This is the Ops piece of DevSecOps. After you deploy, the job isnβt finished.
Keep monitoring. It is embarrassing how often companies that spend millions to staff a 24x7 SOC have apps go down when they run out of disk space