This document provides an overview and best practices for securing Kubernetes (K8s) clusters. It discusses common threats like exposed dashboards, APIs, and etcd stores. It also covers risks from within the cluster like compromised nodes and pods or vulnerabilities in container images. The document recommends 10 essential practices for securing K8s like image scanning, role-based access control, security boundaries, upgrades, pod security policies, node hardening, audit logging, and host/container logging. It emphasizes the importance of a security-aware development process and provides resources for further information.

1. Practical Guide to
Securing Kubernets(K8’s)
Dan Hubbard and James Condon
Art into Science : A Conference of Defense
January, 2019

2. overview

3. Scalable Container
Management
Provisioning
Security
Storage
Redundancy
Auto-Scaling

4. Scalable Container
Management
Provisioning
Security
Storage
Redundancy
Auto-Scaling

6. risks and threats

7. THREATS & RISKS FOR CLOUD WORKLOADS
Information
Disclosure
Service LossesAbuse & Nefarious
Use
Data leaks

8. MAJOR THRE AT VECTORS
Outside the Cluster Inside the Cluster
• Management Server UI
• API Service
• etcd
• Kubelet
• Compromised Nodes
• Compromised Pods
• Compromised Accounts

9. EXPOSED DASHBOARDS

10. EXPOSED DASHBOARDS

11. DISCOVERING DASHBOARDS WITH SHODAN

12. EXPOSED DASHBOARDS

13. EXPOSED DASHBOARDS

14. EXPOSED DASHBOARDS

15. EXPOSED DASHBOARDS

16. EXPOSED KUBERNETES API SERVER
• By default API server accepts discovery requests by anonymous users
• Twistlock PoC
• Leak information on all pods and namespaces for metrics server as
anonymous user
• @_evict PoC
• gain cluster-admin on servicecatalog as anonymous user

17. EXPOSED KUBELET

18. EXEC ON RUNNING CONTAINER THROUGH KUBELET
• PoC by Security Engineer @ Handy (K8 v1.9)
• Issue POST request to targeted Pod
• Follow with GET request via SPDY or websocket client

19. REPLAYING KUBELET CREDENTIALS
• SSRF in vulnerable service used by Shopify
• Kubelet credentials leaks via vulnerability
• Credentials replayed to gain root access in any container

20. EXPOSED etcd
Distributed Key Value
Data Store
No Authentication by
Default
REST & gRPC APIs

21. EXPOSED etcd DEMO

22. Pod
Compromise
Application
Vulnerabilities
Supply Chain
Attacks
Known &
Unknown
CVEs
THREATS FROM WITHIN THE CLUSTER

23. • allows containers using subPath volume
mounts to access files or directories outside of
the volume, including the host’s filesystem
CVE-2017-
1002101
• allows containers using certain volumes to
trigger deletion of arbitrary files on the host
filesystem
CVE-2017-
1002102
• Options for accessing host system
Privileged
Containers
LATERAL MOVEMENT: NODES

24. • proxy request handling in kube-
apiserver can leave vulnerable TCP
connections
CVE-2018-
1002105
• Authorization to create pods,
deamonsets, etc.
Overprivileged
Service
Accounts
LATERAL MOVEMENT: CLUSTER

25. securing kubernetes

26. 10
ESSENTIALS
SECURING
K8S
Upgrade
Security Saavy
Dev <-> Saavy
Security
POD Security
PoliciesNode Security
Hardening
Audit Logging
Security
Boundaries
RT Compliance
/ Auditing
Image Scanning
RBAC
Host Logging /
HIDS

27. IMAGE SCANNING
Container vulnerability scans
Scan for poor configurations in containers
Scan for keys in containers
Combine pre-deploy with runtime

28. ROLE BASED ACCESS CONTROL
Critical for division on access
Segregates roles and permissions
Decreases attack surface

29. SECURITY BOUNDARIES
Utilize multiple namespaces
Separate sensitive workloads
Utilize node pools to separate
Ex: kube-public

30. UPGRADE!
CVE-2018-1002105 DEMO
Upgrading should be seamless
No runtime patching
Patch = redeploy
Vulnerabilities != vulnerable often

31. POD SECURITY POLICIES
Huge win in securing K8s
Allow centralized cluster level
security controls / configuration
Controls growing frequently

32. NODE SECURITY HARDENNING
Eliminate logins and “in-place” changes
Read-only file systems
Least Privilege
Consistent deployments
Atomic Deployment and Validation
Run as non-root

33. AUDIT LOGGING
Audit Logging for ALL API requests
API is largest attack surface
Log as much as you can afford
Store, glacier, have them avail/query
Audit logs big forensics firehouse

34. RT COMPLIANCE / CONFIG
Realtime / runtime auditing critical
Infrastructure as code = wider paper cuts
Security vulnerabilities often config’s
Identify, alert, fix, measure (repeat)

35. HOST LOGGING / HIDS / EDR
Ephemeral workloads make logging more
important
Understand process, applications, network
Building net “sensors” hard / blind
Correlate IOC’s + events (ML+)
Opensource + SaaS options
Build / buy centralized warehouse
Auditd, /proc, pcap,etc..

36. SECURITY SAAVY DEV :
DEV SAAVY SECURITY
The Firewall is the
security!
Least Privilege
Immutable for All
Window of
Opportunity
YESTERDAY TOMORROWTODAY

37. dan @ lacework.com
james @ lacework.com

38. questions

39. ABOUT LACEWORK
Automated security for cloud workloads
Purpose built for servers, containers, & Kubernetes
High fidelity detection and alerting
Engineered for massive scale
Unified security platform

40. LACEWORK FOR WORKLOAD SECURITY
Host intrusion
detection
Runtime Container
& K8s security
File integrity
monitoring
Threat & incident
investigations
Private cloud

41. LACEWORK FOR ACCOUNT SECURITY
Compliance monitoring
for AWS, Azure, & GCP
Compliance reporting
for CIS benchmark,
PCI DSS, & SOC 2
Misconfiguration
detection & alerting
Anomaly detection
for API behaviors
& audit logs

42. resources
1.Tesla Exposed Dashboard https://redlock.io/blog/cryptojacking-tesla
2.Weight Watchers Exposed Dashboard https://kromtech.com/blog/security-center/weightwatchers-exposure-a-simple-yet-powerful-lesson-
in-cloud-security
3.Lacework Containers at Risk Report https://info.lacework.com/hubfs/Containers%20At-
Risk_%20A%20Review%20of%2021,000%20Cloud%20Environments.pdf
4.CVE-2018-1002105 Github Page https://github.com/kubernetes/kubernetes/issues/71411
5.Kubelet Reference Page https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/
6.Compromising Kubernetes Through Kubelet Blog https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-backdooring-through-
kubelet-823be5c3d67c
7.Shopify Hack https://hackerone.com/reports/341876
8.Exposed etcd Clusters Blog https://elweb.co/the-security-footgun-in-etcd/
9.Lacework exposed etcd Clusters Blog https://www.lacework.com/etcd-thousands-of-clusters-open/
10.Backdoored Docker Images https://arstechnica.com/information-technology/2018/06/backdoored-images-downloaded-5-million-times-
finally-removed-from-docker-hub/
11.Twistlock Blog on CVE-2017-1002101https://www.twistlock.com/labs-blog/deep-dive-severe-kubernetes-vulnerability-date-cve-2017-
1002101/
12.Attacking and Defending a Kubernetes Cluster Webinar https://vimeo.com/277901517
13. Kubernetes Illustrated Children's Guide: https://youtu.be/4ht22ReBjno