Advanced Container Security

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jason Umiker
Solutions Architect
Advanced Container Security

Agenda
• Introduction
• A (brief) refresher on security in AWS
• Security considerations of ECS (EC2 and Fargate) and EKS
• Container image best practice and scanning approaches
• Conclusion

Challenges of Containers at Scale
• More transient / dynamic
• More distributed and complex
• More services interdependent over network
• Scheduling / Scaling / Resource Management
• Less isolated
• Share a kernel
• Often share a network and (in case of EKS) a network interface
All these new challenges have solutions / mitigations

ECS EKS
EC2 Fargate EC2 Fargate
1. Choose your
orchestration tool
2. Choose your
launch type
Coming
Soon!
We Give You The Power to Choose:

Shared Responsibility Model
Responsible for
Security “of” the
cloud
Responsible for
Security “in” the
cloud
Network and Firewall Configuration
Identity & Access Management
Customer Data
Compute Storage Database Networking
Regions Availability Zones Edge Locations
Operating System
Applications Platform
AWSCUSTOMER

IAM = Who can do what in the platform/cluster?
People Code / Pipelines
Photo & Licence Photo & Licence

Invest in end-to-end automation via pipelines
• AWS Infrastructure-as-Code
• Code & Container Builds
• Security (DevSecOps)
• Deployments
Make it fast and easy for your
team to do the right thing!
Photo

Why DevSecOps via pipelines?
Because if you don’t
make it fast and easy to
do the right thing then
people will often just go
around the
rules/restrictions to get
their work done!
Photo & Licence

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS IAM vs. Kubernetes RBAC
If using ECS then that is a
native extension of the AWS
platform and is fully managed by
AWS IAM.
If running Kubernetes, either
yourself or via EKS, then you need
to understand and configure BOTH
AWS IAM and Kubernetes RBAC.

K8s action allowed/denied
Authorization of AWS
Identity against Kubernetes
RBAC
K8s API
Passes AWS Identity
Verifies AWS Identity
kubectl
AWS IAM
Authentication
EKS: IAM Authentication + kubectl

Kubernetes RBAC built-in ClusterRoles
Default
ClusterRole
Description
cluster-admin
Allows super-user access to perform any action on any resource. When used in
a ClusterRoleBinding, it gives full control over every resource in the cluster and in all
namespaces. When used in a RoleBinding, it gives full control over every resource in the
rolebinding's namespace, including the namespace itself.
admin
Allows admin access, intended to be granted within a namespace using a RoleBinding. If
used in a RoleBinding, allows read/write access to most resources in a namespace,
including the ability to create roles and rolebindings within the namespace. It does not
allow write access to resource quota or to the namespace itself.
edit
Allows read/write access to most objects in a namespace. It does not allow viewing or
modifying roles or rolebindings.
view
Allows read-only access to see most objects in a namespace. It does not allow viewing
roles or rolebindings. It does not allow viewing secrets, since those are escalating

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes RBAC Basics
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
Kubernetes has Roles which are
defined and apply within a single
namespace (a virtual cluster) and
ClusterRoles which apply cluster-
wide across all namespaces.
You define custom roles
describing resources (such as
pods and nodes) and which verbs
(such as get, update and delete)
are allowed against them.

AWS IAM Role per Task / Pod?
Assigning an IAM Role to a ECS Task
is an included feature in the AWS
Platform and ‘just works’.
If running Kubernetes then you need
to add either kube2iam or kiam to
leverage this functionality.
Assigning an IAM role to an Instance/Task/Function means the right
AWS access key and secret to call the AWS CLI/SDK are transparently
obtained and rotated.

AWS Container Roadmap on GitHub
Captured
19/5/2019
https://github.com/aws/containers-roadmap

Logging and Auditing the Control Plane
ECS is part of the AWS platform and the
control plane logs go to CloudTrail just like
the rest of the platform.
Kubernetes’ control plane logs include an
audit trail. With EKS these logs are not
exposed by default but you can (and should)
enable sending them to CloudWatch Logs.
Logging of the control plane, especially around an audit trail of API
actions, is an important aspect of security.

AWS Security Groups vs. Kube Network Polices
If using ECS then that is an
extension of the AWS platform
and you only need to understand
and configure AWS VPC and
Security Groups.
If running Kubernetes yourself or
EKS then you need to understand
and configure BOTH AWS VPCs /
Security Groups as well as
Kubernetes Network Polices

Default/Root Namespace
lo
eth0
Task Namespace
lo
eth1
Networking with ECS
When using ECS with the aws-vpc
network mode (optional for EC2 mode
but required for Fargate mode) then
each Task gets its own dedicated
Elastic Network Interface (ENI).
Since each Task is 1:1 with an ENI
and each ENI is 1:1 with a Security
Group (SG) that means any
communication in/out of each Task
goes through its SG on both ingress
and egress.

Micro-segmenting with Security Groups
You can use a security group ID as both a source and destination for
other security group rules – both to loop back to itself or referencing other
SGs.
• This enables network segmentation without complex subnetting

Private EKS Control Plane API Endpoints
Worker VPC (your account)
Kubectl
Master VPC (AWS account)
etcd
AZ 1 AZ 2
API Server
etcd
API Server
EKS-owned
ENIs
Public == false
Private == true
prod-cluster-123.eks.amazonaws.com
Private hosted zone
Kubelet
AZ 1
Worker
node
Kube-proxy
Kubelet
AZ 2
Worker
node
Kube-proxy

Networking with EKS
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
10.0.0.1
10.0.0.2
ENI
10.0.0.20
10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet –
10.0.0.0/24
Instance 1 Instance 2
VPC

Installing a Network Policy Provider on
Kubernetes
You first need to add a Network Policy Provider to EKS / Kubernetes in
order to use Network Policies. A popular one covered in our
documentation is Calico.
https://docs.aws.amazon.com/eks/latest/userguide/calico.html

Frontend
Cats Dogs
kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: default-deny
spec:
podSelector:
matchLabels: {}
catsndogs-namespace
Network Policies on Kubernetes

Frontend
Cats Dogs
catsndogs-namespace
kind: NetworkPolicy
metadata:
name: public-to-frontend
spec:
podSelector:
matchLabels:
role: frontend
ingress:
- from:
- ipBlock:
cidr: "0.0.0.0/0"
ports:
- protocol: TCP
port: 80

Frontend
Cats Dogs
catsndogs-namespace
kind: NetworkPolicy
metadata:
name: frontend-to-cats
spec:
podSelector:
matchLabels:
role: cats
ingress:
- from:
- podSelector:
matchLabels:
role: “frontend”
ports:
- protocol: TCP
port: 80

Secure Cloud Edition (CE)
Features:
• Enterprise Support from Tigera
• Host-to-Host IPSEC Encryption
• Flow Logs enriched with Kubernetes Workload Metadata
• Integration between AWS Security Groups and Network Policies

Alternative – Multiple NodeGroups or Clusters
One way that you can both assign both EC2 Instance-level IAM
Roles (without kops or kiam) as well as fully trust Security Group-
based micro-segmentation without Tigera is to have a different set
of worker nodes, or even entirely separate Clusters, for different
services or trust boundaries.
EKS has the concept of a NodeGroup which is a separate Auto
Scaling Group of worker Nodes that can be labelled in such a way
that you can limit which pods/services can be run on them.
https://kubernetes.io/docs/concepts/configuration/assign-pod-node

Wildcard – Service Mesh Instead?
Instead of imposing network-level restrictions like AWS Security Groups
or Kubernetes Network Policies you can instead use a Service Mesh to
both encrypt and authenticate between all of your services allowing for a
flatter more unsegmented underlying network yet still staying safe.
AWS App Mesh Istio

Why Service Mesh?
http/tcp
Service
team A
Service
team B
Common need: Manage inter-service traffic
How to generate uniform logs,
metrics and traces
How to load balance traffic
How to shift traffic between
deployments
How to decouple service teams
How to minimize impact to
application code
How to ensure all traffic is
encrypted
How to ensure that the service
making the request is the one
that SHOULD be making it

Why App Mesh?
http/tcp
Service
team A
Service
team B
Proxy
Sits between all services
Manages and observes traffic
Control plane
Translates intent to proxy config
Distributes proxy config
Control plane

Sidecar proxy with containers
Proxy runs
as a container
Task or pod
External traffic
Application
code as a
container

OSS project
Wide community support, numerous integrations
Stable and production-proven
Graduated Project in Cloud Native Computing Foundation
Started at Lyft in 2016
App Mesh and Istio both use Envoy

App Mesh control plane configures every proxy

Mutual TLS authentication (mTLS)
Client Server
Certificate Authority
Certificate Exchange
Certificate Generation /
Signing for Client & Server
Client Validates Server Cert
and
Server Validates Client Cert
Private Key
Client Cert
Private Key
Server Cert
Client Cert
Server Cert

AWS App Mesh vs. Istio
App Mesh
• Orchestrates Envoy sidecar
• Integrated with Kubernetes
• Does not yet support
encryption and mTLS
• Wider integration with AWS
• Regional Managed Service
• Fully supported by AWS
Istio (on EKS)
• Orchestrates Envoy sidecar
• Integrated with Kubernetes
• Supports encryption and mTLS
• Requires Kubernetes
• State stored in Kubernetes’
control plane and etcd via
Custom Resources
• Best effort support by AWS

AWS AppMesh Roadmap on GitHub
https://github.com/aws/aws-app-mesh-roadmap
Captured

EC2 Mode – Customer Responsibilities
• Instance type and quantity to choose?
• What is the CPU to RAM ratio?
• Excess capacity for scaling and
availability?
• Which OS to choose?
• If Amazon Linux we provide AMIs
• Hardening the OS (e.g. against CIS
benchmark)
• Patching of the OS, Docker, ECS
Agent or kubelet etc.
Photo & Licence

Security Benefits Of Fargate
We do more, you do less.
• Patching (OS, Docker, ECS Agent, etc.)
• Task isolation (via separate Clusters)
• No --privileged mode for containers
• Requires awsvpc network mode so ENI/SG per Task
• No runtime access for users (ssh or interactive
Docker)

EC2-mode ECS Shared Responsibility Model
Customer Data
Storage Database Networking
Regions Edge Locations
Operating System
Images ECS Config
ECS Control Plane
Instance Scaling
Compute
Availability Zones
AWSCUSTOMER

ECS Fargate Shared Responsibility Model
Customer Data
Storage Database Networking
Regions Edge Locations
Operating System
Images ECS Config
ECS Control Plane
Instance Scaling
Compute
Availability Zones
AWSCUSTOMER

Updating EKS
• Kubernetes has a new major version every quarter
• Kubernetes has a new minor version quite regularly
• Sometimes Kubernetes updates are security-related
• EKS has APIs to trigger an update of the control plane
• You then need to update the worker Nodes - both re:
Kubernetes as well as Docker and OS
• Often the workers are in an Autoscaling
Group so this means building updating
AMIs
• We provide a regularly updated EKS Node
AMI as well as scripts to build your own.

AWS Parameter Store, Secrets Manager and
Kubernetes Secrets
AWS has both Parameter Store and Secrets
Manager to store your Secrets. They are
integrated into ECS but you’ll need to call
them within the Pod on Kubernetes via our
CLI or SDK.
Kubernetes’ built-in Secrets functionality
stores secrets in its control plane and puts
them into running Pods via Environment
Variables or files in the filesystem. You can’t
use these outside of the Kubernetes cluster.

Security Best Practices For Container Images
Less is more (secure)
• No secrets in them
• One service per container
- Use sidecars within Task / Pod
• Minimise container footprint
- Include only what is needed at runtime bootfs
kernel
Base image
Image
Image
Container
References
parent
image

Security Best Practices For Container Images
• Use known and trusted base images
- Official on Docker Hub
- Read the Dockerfiles
- Scan the image for CVEs
• Specify USER in Dockerfile
(otherwise it’s root)
• Unique and informative image tags
- Be able to tell which commit at a
glance
bootfs
kernel
Base image
Image
Image
Container
References
parent
image

Image Scanning
• Scan images in your registry
• Docker Hub does this
• On our roadmap to do with Elastic Container Registry (ECR) as well
• Scan images in your build pipeline
• Clair - https://github.com/coreos/clair
• Aqua Microscanner - https://github.com/aquasecurity/microscanner
• Scan images at runtime / running containers (these can also scan in pipeline)
• Aqua - https://www.aquasec.com/solutions/aws-container-security/
• Twistlock - https://www.twistlock.com/solutions/aws-security/
All of the above?

Runtime container security
You can protect against zero-day vulnerabilities not yet even in a CVE database
via partner products from the likes of Aqua and Twistlock.
• Limit what can execute within container(s) via rules engine
• e.g. “Do not run things that were not in the image” or
“Do not run things that are not on this whitelist”
• Ensure only trusted images can be deployed/run in your cluster
• Get visibility into the runtime behaviour of the entire environment
• Detect vulnerable running containers as soon as a CVE is made public

https://aws.amazon.com/containers/partner-solutions/

Summing Up
The customer has many responsibilities in running containers securely in AWS -
especially when running Kubernetes on the platform rather than running ECS.
The key areas to delve into include:
• Identity and Access Management
• Network Topology and Firewalling
• Logging and Auditing
• Encryption and Mutual Authentication between Tasks/Pods
• Patching (container images, container hosts and the Kubernetes control plane)
• Secrets Management
• What is in, and what isn’t in, each container image you run

Advanced Container Security

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Advanced Container Security

Similar to Advanced Container Security (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Advanced Container Security