Amazon Elastic Container Service for Kubernetes (Amazon EKS) is an AWS service offering a managed Kubernetes control plane for customers to orchestrate their containerized applications on Amazon EC2. In this chalk talk, Micah Hausler, AWS system development engineer, explains how customers can ensure the integrity and auditability of their applications on Amazon EKS. He demonstrates the exploitation of a misconfigured web application container, and he conducts a forensic analysis of what happened in the system.

1. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Audibility in Kubernetes with Amazon
EKS
Micah Hausler
Sr. System Development Engineer
AWS
G R C 3 0 2

2. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Overview of Amazon Elastic Kubernetes Service (Amazon EKS)
Create your cluster
Authentication and authorization in Kubernetes
Install a web application
Exploit your web application
Deep dive into the audit

3. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Tuesday, June 25
SDD326: Security best practices for Amazon EKS
2:30 – 3:30 PM | Level 1, Room 151B, Table 2
5:30 – 6:30 PM | Level 1, Room 151B, Table 2
Wednesday, June 26
SDD326: Security best practices for Amazon EKS
11:00 AM – 12:00 PM | Level 1, Room 151B, Table 2
2:00 – 3:00 PM | Level 1, Room 151B, Table 2
Tuesday, June 25
GRC340: Container runtime security and automation
4:00 – 5:00 PM | Level 1, Room 151B, Table 4

4. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS
managed control
plane
Customer-owned
data plane
AWS Cloud
VPC
VPC
AZ-1 AZ-2 AZ-3

6. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Go to https://eksworkshop.com
Select Start the workshop
Use the on your own guide
Select Launch using eksctl
Once you are waiting on the cluster to create, STOP!

9. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authentication

11. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authorization

12. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
ClusterRole and ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups: ['*']
resources: ['*’]
verbs: ['*']
- nonResourceURLs: ['*’]
verbs: ['*']
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: micah-cluster-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: micahhausler

13. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authentication

14. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2 instance
Service account token mount
Pod (container)

15. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://github.com/micahhausler/reinforce-grc302

17. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Clone the workshop repository
git clone https://github.com/micahhausler/reinforce-grc302.git
cd reinforce-grc302
bash ./scripts/1-cluster-logging.sh

18. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Check the cluster in the
console
After your cluster finishes updating, you
can view cluster control plane logs in
Amazon CloudWatch

19. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
View the default token
bash ./scripts/2-decode-token.sh

20. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bind the default service account
bash ./scripts/3-bind-service-account.sh

21. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Launch a pod
kubectl apply -f manifests/al2-deployment.yaml
kubectl get pods -o wide

22. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Explore the pod
bash ./scripts/5-exec-pod.sh
top
Whoami
ifconfig
hostname
ps aux

23. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2 instance
kubectl exec
Pod (container)

24. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Run a web application
kubectl apply -f ./manifests/webapp.yaml
kubectl get -o wide -f ./manifests/webapp.yaml
kubectl port-forward service/webapp 3000:80
# In a separate tab
curl http://localhost:3000

26. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2 instance
kubectl port-forward
Pod (container)

27. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
The web application has some endpoints
curl http://localhost:3000/serve/

28. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
The web application has some endpoints
curl http://localhost:3000/proxy/?q=https://checkip.amazonaws.com/

29. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
The web application has some problems
curl http://localhost:3000/serve/var/run/secrets/kubernetes.io/serviceaccount/token

31. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
kubectl apply -f ./manifests/exploit.yaml

32. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Review the web application logs
kubectl logs –l app=webapp | jq

34. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Go to Amazon
CloudWatch Logs in
the console
Click the CloudWatch Logs
Insights link

35. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
CloudWatch Logs insights query
fields @timestamp, user.username as username, sourceIPs.0, responseStatus.code, verb,
requestURI
| filter username = "system:serviceaccount:default:default"

36. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Try more examples
cat examples/cwl-insight-queries.txt

37. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

38. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some best practices to follow
Do not give service accounts admin permissions
ServiceAccount tokens never expire, you have to delete them to rotate
Use PodSecurityPolicy – new in Kubernetes 1.13
Save application logs

39. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Other tools to try
audit2rbac: Create RBAC roles from audit logs
https://github.com/liggitt/audit2rbac
CloudWatch Log metric filters
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html
Amazon VPC Flow Logs
AWS CloudTrail

40. Thank you!
© 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Micah Hausler
@micahhausler