Batten Down the Hatches: A Practical Guide to Securing Kubernetes - RMISC 2019
•
1 like•526 views
Presented at RMISC 2019 & the June CSA Chapter meetup
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Batten Down the Hatches: A Practical Guide to Securing Kubernetes - RMISC 2019
Similar to Batten Down the Hatches: A Practical Guide to Securing Kubernetes - RMISC 2019 (20)
More from Lacework
More from Lacework (10)
Recently uploaded
Recently uploaded (20)
Batten Down the Hatches: A Practical Guide to Securing Kubernetes - RMISC 2019
- 1. @laceworklabs Batten Down the Hatches: A Practical Guide to Securing Kubernetes James Condon CSA June 18th, 2019
- 2. @laceworklabs whoami • James Condon, Director of Research @ Lacework • Former USAF OSI, Mandiant, and ProtectWise • Network Forensics, Incident Response, Threat Intelligence, Cloud Security Twitter: @laceworklabs, @jameswcondon Email: james@lacework.com Blog: www.lacework.com/blog/
- 3. @laceworklabs AGENDA Kubernetes Overview Risks and Threats Securing Kubernetes
- 7. @laceworklabs Master API Server etcd Scheduler Control Manger Proxy Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI BASIC ARCHITECTURE
- 8. @laceworklabs RISKS & THREATS TO KUBERNETES
- 9. @laceworklabs MAJOR THRE AT VECTORS Exposed K8s Components Pod Compromise • UI Dashboard • API Service • etcd • Kubelet • Application Attacks • Supply Chain Attacks • Orchestrator and container CVEs
- 10. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI EXPOSED DASHBOARDS Master API Server etcd Scheduler Control Manger Proxy
- 11. @laceworklabs EXPOSED DASHBOARDS • Web-based cluster management UI • Risks & Threats: • Default service accounts needs RBAC • Can expose CSP keys • Cryptojacking attacks • Information & data leaks
- 13. @laceworklabs DASHBOARD FINDINGS 500+ 75% AWS 10% GCP + Azure Ports 80, 443, 8080
- 14. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI KUBE-APISERVER Master API Server etcd Scheduler Control Manger Proxy
- 15. @laceworklabs EXPOSED API SERVER • Handles all client interactions to the K8s API • REST API • Handles authentication and authorization • Secure & insecure port by default • Risks & Threats • Access to insecure port allows complete access of cluster • CVE-2018-1002105 • Information leaks
- 17. @laceworklabs API SERVER FINDINGS 21K+ 92% AWS, ~3% GCP + Azure ’18: 21K+ for K8s, Meso, OpenShift, & Swarm Cert CNs: kubernetes-master 88%, system:apiserver 4%, apiserver 2%
- 18. @laceworklabs API SERVER FINDINGS (INSECURE PORT) 800+
- 19. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI KUBELET Master API Server etcd Scheduler Control Manger Proxy
- 20. @laceworklabs EXPOSED KUBELET • Daemon on nodes to bridge compute resources, facilitate communicates, and aide in pod health • Risks & Threats • Allows anonymous requests by default • “AlwaysAllow” is the default for authenticated requests rest by default • Contains credentials that can be used to access other components in the cluster
- 21. @laceworklabs EXEC ON RUNNING CONTAINER THROUGH KUBELET • PoC by Security Engineer @ Handy (K8 v1.9) • Issue POST request to targeted Pod • Follow with GET request via SPDY or websocket client
- 22. @laceworklabs REPLAYING KUBELET CREDENTIALS • SSRF in vulnerable service used by Shopify • Kubelet credentials leaks via vulnerability • Credentials replayed to gain root access in any container
- 23. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI etcd Master API Server etcd Scheduler Control Manger Proxy
- 24. @laceworklabs ETCD • Distributed key value datastore • REST & gRPC APIs • Responsible for storing objects, state, etc. • Risks & Threats • No authentication or encryption at rest by default • Maintains cluster secrets • The Luke Hemsworth of unsecured DBs
- 27. @laceworklabs POD COMPROMISE & LATERAL MOVEMENT Pod Compromise Application Vulnerabilities Supply Chain Attacks Known & Unknown CVEs
- 28. @laceworklabs Node Proxy Kubelet Container Runtime Pod 1...n Node Proxy Kubelet Container Runtime Pod 1...n UI Dashboard CLI PODS Master API Server etcd Scheduler Control Manger Proxy
- 31. @laceworklabs • allows containers using subPath volume mounts to access files or directories outside of the volume, including the host’s filesystemCVE-2017-1002101 • Flaw in runc, allows potential container escapeCVE-2019-5736 • Options for accessing host systemPrivileged Containers • Default service accounts are overprivileged and have too much access that an attacker could leverageService Accounts • Authenticated users with permission to exec/attach/portforward could escalated to run additional commands against Kubelet APICVE-2018-1002105 LATERAL MOVEMENT
- 33. @laceworklabs 10 ESSENTIALS SECURING K8S Upgrade Network Security POD Security PoliciesNode Security Hardening Audit Logging Security Boundaries RT Compliance / Auditing Image Security RBAC Host Logging / HIDS
- 34. @laceworklabs NETWORK SECURITY Restrict Open Internet Access TLS, VPN, Bastion Network Policy for pods Host Firewalls
- 35. @laceworklabs IMAGE SECURITY Container vulnerability scans Scan for poor configurations in containers Scan for keys in containers Combine pre-deploy with runtime
- 36. @laceworklabs ROLE BASED ACCESS CONTROL Critical for division on access Segregates roles and permissions Decreases attack surface Reduce default permissions of service accounts
- 37. @laceworklabs SECURITY BOUNDARIES Utilize multiple namespaces Separate sensitive workloads Utilize node pools to separate Ex: kube-public
- 38. @laceworklabs UPGRADE! CVE-2018-1002105 DEMO Upgrading should be seamless No runtime patching Patch = redeploy Vulnerabilities != vulnerable often
- 39. @laceworklabs POD SECURITY POLICIES Huge win in securing K8s Allow centralized cluster level security controls / configuration Controls growing frequently Common Examples: privileged volumes network
- 40. @laceworklabs NODE SECURITY HARDENNING Minimal OS footprint Restricted file system access Upgrades and Patches Disabled root login Kernel Protection Security Defaults
- 41. @laceworklabs AUDIT LOGGING Audit Logging for ALL API requests API is largest attack surface Log as much as you can afford Store, glacier, have them avail/query Audit logs big forensics firehouse
- 42. @laceworklabs RT COMPLIANCE / CONFIG CIS Benchmarks Realtime / runtime auditing critical Infrastructure as code = wider paper cuts Security vulnerabilities often config’s Identify, alert, fix, measure (repeat)
- 43. @laceworklabs HOST LOGGING / HIDS / EDR Ephemeral workloads make logging more important Understand process, applications, network Building net “sensors” hard / blind Correlate IOC’s + events (ML+) Opensource + SaaS options Build / buy centralized warehouse Auditd, /proc, pcap,etc..
- 44. @laceworklabs FINAL THOUGHTS • K8s is complex “5 minutes to deploy, 5 years to learn” • Reported attacks are primarily cryptojacking, pivoting to CSP, and data leak • Misconfiguration and pod compromise are the major vectors • Use traditional security, DevSecOps, and K8s features to harden your cluster
- 45. @laceworklabs resources 1. Tesla Exposed Dashboard https://redlock.io/blog/cryptojacking-tesla 2. Weight Watchers Exposed Dashboard https://kromtech.com/blog/security-center/weightwatchers-exposure-a- simple-yet-powerful-lesson-in-cloud-security 3. Lacework Containers at Risk Report https://info.lacework.com/hubfs/Containers%20At- Risk_%20A%20Review%20of%2021,000%20Cloud%20Environments.pdf 4. CVE-2018-1002105 Github Page https://github.com/kubernetes/kubernetes/issues/71411 5. Kubelet Reference Page https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet- authentication-authorization/ 6. Compromising Kubernetes Through Kubelet Blog https://medium.com/handy-tech/analysis-of-a-kubernetes- hack-backdooring-through-kubelet-823be5c3d67c 7. Shopify Hack https://hackerone.com/reports/341876 8. Exposed etcd Clusters Blog https://elweb.co/the-security-footgun-in-etcd/ 9. Lacework exposed etcd Clusters Blog https://www.lacework.com/etcd-thousands-of-clusters-open/ 10. Backdoored Docker Images https://arstechnica.com/information-technology/2018/06/backdoored-images- downloaded-5-million-times-finally-removed-from-docker-hub/ 11. Twistlock Blog on CVE-2017-1002101https://www.twistlock.com/labs-blog/deep-dive-severe-kubernetes- vulnerability-date-cve-2017-1002101/ 12. Attacking and Defending a Kubernetes Cluster Webinar https://vimeo.com/277901517 13. Kubernetes Illustrated Children's Guide: https://youtu.be/4ht22ReBjno
- 46. @laceworklabs QUESTIONS Twitter: @laceworklabs, @jameswcondon Email: james@lacework.com Blog: www.lacework.com/blog/