Amazon EKS makes it easy to run Kubernetes on AWS without managing master nodes or etcd operators. Kubernetes offers a powerful abstraction layer for managing containerized infrastructure, which presents unique challenges to AWS media customers. In this session, we share lessons from Synamedia, and we discuss its reasons for moving to EKS and the security and governance implications for migrating workloads. Learn about the approach and benefits for establishing security and governance with Open Policy Agent (OPA), which uses Kubernetes validating and mutating admission controllers to establish policy guardrails for container registries, input, load balancers, and other objects within EKS.

1. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Architecting security and governance through
policy guardrails in Amazon EKS
Paavan Mistry
Specialist Solutions Architect – Security
Amazon Web Services
S D D 4 1 1
Stephen Tallamy
Principal Cloud Architect
Synamedia

2. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Agenda
Amazon EKS overview
Amazon EKS data plane security and authentication
Governance and security for Kubernetes and Amazon EKS at Synamedia
Policy guardrails on Amazon EKS using Open Policy Agent

3. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Related events
Tuesday, June 25
GRC340-R – Container runtime security and automation (Builder Session)
4:00 PM – 5:00 PM | Level 1, Room 151B, Table 4
Wednesday, June 26
SDD326-R5 – Security best practices for Amazon EKS (Builder Session)
2:00 PM – 3:00 PM | Level 1, Room 151B, Table 2
Wednesday, June 26
SDD308 – Integrating security testing into your container build pipeline (Workshop)
11:15 AM – 1:15 PM | Level 2, Room 210C

4. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

5. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
We give you the power to choose
Amazon ECS Amazon EKS
Amazon
EC2
AWS
Fargate
Amazon
EC2
AWS
Fargate
1. Choose your
orchestration tool
2. Choose your
launch type
We’re
working
on it #32

6. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Kubernetes architecture
etcd

7. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Shared responsibility model
Customers
Data plane
Amazon EKS
Control planeContainers
AWS Cloud

8. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Amazon EKS compliance
Amazon EKS
Control plane

9. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Amazon EKS security
Customers
Data planeContainers

10. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

11. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

12. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

13. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Data plane security: Container runtime interface (CRI)
”Inside the CNCF Project Security Reviews”: https://www.youtube.com/watch?v=0BkKpsrUo5k

14. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Kubernetes and container security issues
https://aws.amazon.com/security/security-bulletins/
https://aws.amazon.com/blogs/compute/anatomy-of-cve-2019-5736-a-runc-container-escape/

15. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Amazon EKS update life cycle
May 21, 2019
Blog: https://aws.amazon.com/blogs/compute/updates-to-amazon-eks-version-lifecycle/

16. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
1. Implement a documented and operational update
and upgrade program
2. Test, update, and upgrade
Threat model input #1

17. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

18. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
IAM authentication with Amazon EKS
$ kubectl
$ ~/kubelet

19. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
IAM authentication with Amazon EKS
K8s action allowed/denied
Authorization of AWS identity
against Kubernetes RBAC
K8s API
server
Passes AWS identity
Verifies AWS identity
kubectl/kubelet/K
8s client
AWS Identity and Access
Management (IAM)
authentication
https://github.com/kubernetes-sigs/aws-iam-authenticator

20. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
List API endpoints
K8s API
server
Pass AWS
identity token
Remote host

21. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

22. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
VPC
IAM roles for nodes and pods
AWS Key Management
Service
AWS Secrets Manager Amazon GuardDuty Amazon Inspector AWS CloudTrail Amazon CloudWatch AWS Config AWS Systems Manager
“Using AWS Secrets Manager with Amazon EKS”: https://amzn.to/2KeOqzW

23. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
AWS containers roadmap on GitHub
https://github.com/aws/containers-roadmap/projects/1

24. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
AWS containers roadmap on GitHub
https://github.com/aws/containers-roadmap/projects/1

25. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
1. Review cluster IAM, and limit resource access to
least privilege and need to know
2. Use AWS container roadmap for feature requests
Threat model input #2

26. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Container security life cycle
CIS Benchmark
checks
Image
hardening and
scanning
Restrict to
approved
registries
Run signed
images only
Immutability
(Pod Security
Policies/
seccomp/ LSM
policies)
Service-to-
service identity
and mTLS
Anomaly
detection
Amazon EKS now supports Pod Security Policies with v1.13: https://amzn.to/2IrS600

27. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Amazon EKS security: A shared responsibility
Customers
Data planeContainers

28. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

29. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
About Synamedia
The largest global provider of video
solutions for pay TV operators
Trusted by over 200 pay TV
operators
Global innovation with 600 US and
non-US patents

30. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
China DTH
Deployed globally

31. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Transforming the way the world is entertained and informed
Video security Video processing Cloud DVR
Foundation platform Infinite platform

32. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

33. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Deployment approach
Dilemma: How to support various environments while also increasing agility?
App
➢ App teams build for a single
framework/platform
Solution: Abstract the apps from what is under the hood
App
App App
➢ Platform team enables the
framework/platform to be
deployed in different
environments
COE cluster External services
On premisesAmazon EKS

34. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Private subnet
Deployment of OpenShift on AWS
VPC
Node
Node
Public subnet
Network Load Balancer (NLB)
Node
Node
OpenShift workersOpenShift masters
Elastic Load Balancing
Master
Master
Master
Docker registry
Yum repo
Ansible deployer

35. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Successes and challenges
• Large number of production
Kubernetes deployments
• Field experience performing live
upgrades, maintenance, and war
gaming
• Consistent, repeatable deployment
on-prem and in AWS
• Challenge of keeping track with
Kubernetes version (upgrades)
• User-provisioned infrastructure
lacks features (e.g., ASGs)
• Management of guest operating
system is an overhead

36. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Private subnet
Deployment of OpenShift on AWS
VPC
Node
Node
Public subnet
NLB
Node
Node
OpenShift workersOpenShift masters
ELB
Master
Master
Master
Docker registry
Yum repo
Ansible deployer

37. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Private subnet
AWS CloudFormation
Amazon ECR
Amazon EKS AMI
Deployment of Amazon EKS
VPC
Node
Node
Public subnet
NLB
Node
Node
Amazon EKS workers
Amazon EKS

38. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Migrating to Amazon EKS from OpenShift
Own Docker registry to Amazon ECR
OpenShift router Nginx ingress
controller
Docker registry Amazon ECR
OpenShift built-in extensions
Kubernetes
dashboard
Cluster
Autoscaler
Metrics
server
BYO dashboard, autoscaler, metrics server
OpenShift router to ingress controller

39. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

40. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Private link (recap)
Private subnet Private subnet
VPC (“Provider”)
Instance
Instance
Endpoint
NLB
VPC (”Client”)
Instance
Instance

41. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Private subnet Private subnet
Hybrid deployment architecture
AWS Direct Connect
VPC (main)
AWS Cloud
Node
Node
Endpoint
NLB
Public subnet
NLB
VPC (proxy)
Node
Node
Customer data center
Private subnet
On-prem network
Client
Kubernetes workers
Node
Kubernetes workers
Node
NLB
Endpoint

42. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

43. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
“With great power
comes great
responsibility.”
—Uncle Ben, “Spiderman”

44. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great deployment power – Marketplace of containers
Kubernetes
resources
(YAML)
Amazon EKS
My pod
Node

45. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great deployment responsibility – Marketplace of
containers
Kubernetes
resources
(YAML)
Amazon EKS
My pod
Node
NodePort
ELB Attacker
Attacker
Attacker

46. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great deployment responsibility – Marketplace of
containers
Amazon ECR

47. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great integration power – IAM roles
Amazon S3
UI customization
bucket
Billing reports
bucket
Role
IAM
Frontend VM
Backend VM
Role
Amazon Kinesis
Amazon Kinesis Data
Streams
Amazon ElastiCache
Amazon ElastiCache
for Redis

48. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great integration responsibility – IAM roles
Frontend pod
Node
Backend pod
Amazon S3
UI customization
bucket
Billing reports
bucket
Role
IAM
Log DaemonSet
Kinesis
Kinesis Data Streams
ElastiCache
ElastiCache for
Redis

49. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great integration responsibility – IAM roles
Frontend pod
Node
Backend pod
Amazon S3
UI customization
bucket
Billing reports
bucket
IAM Kinesis
Kinesis Data Streams
Log DaemonSet
ElastiCache
ElastiCache for
RedisCredential
Credential
Credential

50. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great power – Service type LoadBalancer
Service
(LoadBalancer)
ELB Users
Pod selector
(app = frontend)
Frontend pod 1
Node
NodePort
Frontend pod 2
Node
NodePort
Frontend pod 3
Node
NodePort

51. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great responsibility – Service type LoadBalancer
Service
(LoadBalancer)
ELB Users
Pod selector
(type = nodejs)
Frontend pod 1
Node
NodePort
Frontend pod 2
Node
NodePort
Frontend pod 3
Node
NodePort
Backend pod 1
Node
NodePort
Generic pod selector

52. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Great responsibility – Service type LoadBalancer
Service
(LoadBalancer)
ELB Users
Pod selector
(app = frontend)
Frontend pod 1
Node
NodePort
Frontend pod 2
Node
NodePort
Frontend pod 3
Node
NodePort
Backend pod 1
Node
NodePortService
(LoadBalancer)
Pod selector
(app = backend)
ELB Users
Incorrect service type

53. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

54. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Admission controllers in Kubernetes

55. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Admission controllers in Kubernetes
https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

56. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Open Policy Agent – CNCF project (incubating)
https://www.youtube.com/watch?v=n94_FNhuzy4

57. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Amazon ECR

58. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.

59. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Styra.com – Open Policy Agent

60. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Rich partner ecosystem

61. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Amazon EKS security resources
• Kubernetes security book: https://kubernetes-security.info/
• Running Kubernetes with Amazon EKS: https://youtu.be/-3FELDeZf_Q
• Deep dive on Amazon EKS: https://youtu.be/EDaGpxZ6Qi0
• Driving continuous security and configuration checks for Amazon EKS with
Alcide Advisor: https://amzn.to/2L8dNDK
• Using AWS Secrets Manager PrivateLink endpoint with Amazon EKS:
https://amzn.to/2KeOqzW

62. © 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
To conclude…
Clearly understand managed systems’ ownership boundaries
and your systems to build the threat model.
“With great power comes great responsibility.”

63. Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates.All rights reserved.
Paavan Mistry
Specialist Solutions Architect - Security
AWS
Stephen Tallamy
Principal Cloud Architect
Synamedia