This document provides an overview of best practices for securing Kubernetes clusters. It discusses infrastructure protection, Kubernetes internal security, authentication and authorization options, network security, secrets management, container runtime security, and other security tools. Specific recommendations include limiting SSH access, using hardened images, encrypting storage, restricting API access, separating workloads, enabling authentication, implementing role-based access control, using network policies, securely managing secrets, scanning images for vulnerabilities, and auditing events.

1. K8S security – Best practices
By: Sharon Vendrov

2. 0
500
1000
1500
2000
2500
Mac Os X Windows 7 Windows XP Windows 8.1 Windows 10
CVE Sum
CVE Sum
2
Total Number Of Vulnerabilities in 2017 – Source:
CVEdetails.com

3. 3

4. 4

5. Sharon Vendrov
Sr. DevOps Engineer
About Me
5
Storm-runner functional

6.  Infrastructure protection
 K8s internal security
 Authentication & Authorization options
 Network
 Secrets
 Container runtime Security
 Some other security tools and considerations
6
Agenda

7. Infrastructure protection
7
 Limit SSH access to your cluster
 Use hardened images for your cluster ( )
 Encrypt your storage volume
 Avoid from exposing your cluster to the internet
 Limit the access to the K8s API (consider to use bastion machine)
 Create dedicated cluster for each environment (Prod, Stg, Dev)
 Separate sensitive pods into different nodes

8. Kubernetes internal security
8
 Use minimal base docker image
 Don’t use arbitrary base images
 Separate sensitive workloads across instances (using anti-affinity,
taints and tolerations)
 Use namespaces for isolation
 Enforce resource quota (CPU, Memory, Storage)
Image Name node:latest ubuntu:latest alpine:latest scratch
Image Size 670MB~ 110MB~ 4.1MB~ 0

9. Secure kubelet
9
 curl -sk https://<nodeIP>:10250/run/<namespace>/<pod-name>/<container-name> -d
"cmd=ls -la /“
 Protect kubelet by enable authentication and authorization:
start the apiserver with --kubelet-client-certificate and --kubelet-client-key flags
/usr/local/bin/kubelet
--anonymous-auth=false
--authorization-mode=Webhook
--allow-privileged=true
--kubeconfig=/var/lib/kubelet/kubeconfig
--client-ca-file=/var/lib/kubernetes/ca.pem
• Enable kubelet certification rotation (1.8 beta)

10. Authentication & Authorization

11. 11

12. 12

13. Authentication
13
 Static password/token file
 Client certificates x509
 Proxy + headers
 OpenID Connect
 Custom (Web hook)
password,user,uid,”group1,group2,group3”

14. Authentication
14
 Service accounts
 Default service account have full permissions over the cluster, use custom SA instead
 Set “automountServiceAccountToken : false” in your pod spec – when possible

15. Authorization
15
 ABAC
 Difficult to manage and understand
 Requires ssh and root filesystem access on the master
 For permission changes to take effect the cluster API server must be restarted
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1",
"kind": "Policy",
"spec": {
"user": "bob",
"namespace": "projectSpaceX",
"resource": "pods",
"readonly": true
}
}

16. Authorization
16
 RBAC (stable 1.8)
Service Account
User
Role binding Role

17. 17
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: Bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""]
group
resources: ["pods"]
verbs: ["get", "watch", "list"]

18. Authorization
18
 Custom (Web hook)
 Node
Restrict kubelet to perform R/W operation only to his bound pods
--authorization-mode=Node,RBAC
--admission-control=NodeRestriction

19. Network

20. 20

21. Netwok
21
 Limit the access to cloud provider metadata
(http://169.254.169.254/latest/meta-data/)

22. 22
$ curl -s 169.254.169.254/latest/meta-data/iam/security-
credentials/kubernetes-worker-iam-policy
{
"Code" : "Success",
"LastUpdated" : "2017-12-25T00:00:00Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "MyAccessKeyID",
"SecretAccessKey" : "MySecretAccessKey",
"Token" : "MySessionToken",
"Expiration" : "2017-12-25T04:00:00Z"
} @bradgeesaman

23. 23
# Place credentials in ENV vars
$ export AWS_REGION=us-east-1
$ export AWS_ACCESS_KEY_ID=MyAccessKeyID
$ export AWS_SECRET_ACCESS_KEY=MySecretAccessKey
$ export AWS_SESSION_TOKEN=MySessionToken
$ aws ec2 … @bradgeesaman

24. The solution
24
• For AWS use kube2iam or kiam (using docker proxy for requests to the
metadata)
• For GCE use k8s-metadata-proxy
• Limit egress with network policy

25. 25
 Use network policy (GA from 1.7) https://goo.gl/HRtn5B
 Egress rules are beta from 1.8
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: access-nginx
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- podSelector:
matchLabels:
access: "true"
• Istio

26. Network policy guidelines
26
 Label your workloads properly
 Isolate workloads from each other
 Restrict income traffic to the kube-system (except kube-dns)
 Consider limit egress to the internet

27. “The definition of Secret—
something you tell everybody to
tell nobody.”
– The universe

28. Treat your secrets with respect
28
 Don’t store your secrets on Git, it will remain in history even If you
delete it.
 Create dedicated secrets for dev and prod environments
 Secrets are stored at etcd as base64 (almost like plain text) 
encrypt your secrets (K8S encryption –alpha 1.7)
 Use Vault as you secret management (starting from Vault 0.8.3)

29. Security Context
A security context defines privilege and access control settings for a Pod or Container
29
 Discretionary Access Control: Permission to access an object, like a file, is
based on user ID (UID) and group ID (GID).
 Security Enhanced Linux (SELinux): Objects are assigned security labels.
 Running as privileged or unprivileged.
 Linux Capabilities: Give a process some privileges, but not all the privileges
of the root user.
 AppArmor: Use program profiles to restrict the capabilities of individual
programs.
 Seccomp: Limit a process’s access to open file descriptors.
 AllowPrivilegeEscalation: Controls whether a process can gain more
privileges than its parent process.

30. Example: RunasNonRoot
30
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
containers:
- name: sec-ctx-demo
image: gcr.io/google-samples/node-hello:1.0
securityContext:
runAsNonRoot : true

31. 31

32. Example: readOnlyRootFilesystem
32
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
containers:
- name: sec-ctx-demo
image: gcr.io/google-samples/node-hello:1.0
securityContext:
runAsNonRoot : false
readOnlyRootFilesystem : true

33. 33

34. 34

35. Other security tools and considerations
35
 Scan your docker images for vulnerabilities, (Clair CoreOS /Quay.io,
Docker Security Scanning, aqua, Twistlock).
 Use kube-bench (aqua security) or kubernetes-auto-analyzer
(nccgroup) to execute CIS Kubernetes Benchmark
 Enforce cluster wide security policy w/podSecurityPolicy
 Use only trusted private docker registry
 Always tag your images avoid from using “latest”
 Audit events and store them on external storage (beta 1.8)
 Consider using kubeaudit to audit security issue

36. 36

37. Other security considerations
37
 Specify an image with its digest (SHA256)
 Keep up with K8S stable releases
 Implement monitoring and set alerts
 Don’t run “kubectl create –f <some unknown URL to some unknown
yamls>
 Keep updated with new security vulnerabilities from the google
group “kubernetes-announces”
https://groups.google.com/forum/#!forum/kubernetes-announce

38. 38

39. Thanks and credit
39
 My Wife 
 All K8s contributors
 Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman -
https://goo.gl/komeXN
 Running containers securely with Google Container Engine, Alex Mohr and
Jessica Frazelle - https://goo.gl/AFhTyp
 Shipping in Pirate-Infested Waters: Practical Attack and Defense in Kubernetes
[A] - Greg Castle - https://goo.gl/WFDrrv
 Compliance and Identity Management in Kubernetes [I] Marc Boorshtein -
https://goo.gl/Jf7Rkh
 Securing K8s Microservices with Calico Network Policies, Vadim Solvey -
https://goo.gl/rWGGXM

40. Thank You.