Csa container-security-in-aws-dw

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved..
Dave Walker, Specialist Solutions Architect, Security and Compliance
18/06/19
Securing Containers on AWS

Agenda
• Container Concept Review
• Container Security in Development
• Container Security in Operations
• Orchestration and Management
Security
• Resources

Containers Concept Review: History
• chroot –The first container
• Changes the root directory of a process to a new directory
• Introduced in 1979 via UnixVersion 7
• Used to create “jails” (originally in BSD)
• LXC – OS-level virtualisation for running multiple isolated Linux systems on a
single kernel
• Introduced in 2008
• Other OS-level Branches
• Eg Solaris Zones
• Docker – Mainstream containers
• Debuted at PyCon in 2013
• Mainstream adoption of containers

Containers Concept Review: Implementation
Server (Host)
Hypervisor
Guest OS
Bins/Libs
App 2
Guest OS
Bins/Libs
App 3
Guest OS
Bins/Libs
App 1
Server (Host)
Hypervisor
Guest OS
App 2
Guest OS
App 3
Guest OS
App 1
Guest OS / Docker Engine
Bins/Libs Bins/LibsBins/Libs
Server (Host)
Operating System (OS)
Guest OS Guest OSGuest OS Libraries
App 1, 2, 3
Bare Metal Virtual Machine Containers

Containers Concept Review: Firecracker
https://github.com/firecracker-microvm/firecracker

© 2019, Amazon Web Services, Inc. or its Affiliates.
Why Containers?
• Speed
• Efficiency
• Easier packaging
• Less risky deployments
• Helps facilitate move to
microservices
• Sidecars
• Helps deploy and run up-to-
30x more services Photo & Licence

Challenges of Containers at Scale
• More transient / dynamic
• More distributed and complex
• More services interdependent over network
• Scheduling / Scaling / Resource Management
• Less isolated
• Share a kernel (unless running in Fargate / Firecracker)
• Often share a network and (in case of EKS) a network interface
All these new challenges have solutions or mitigations.

We Give You The Power To Choose:
ECS EKS
EC2 Fargate EC2 Fargate
1. Choose your
orchestration tool
2. Choose your
launch type
We’re
working
on it
#32

AWS Container Roadmap on GitHub
Captured 19/5/2019
https://github.com/aws/containers-roadmap

AWS Containers Roadmap on GitHub
https://github.com/aws/containers-roadmap

What is Kubernetes?
• Open source container management platform
• Helps you run containers at scale
• Gives you primitives for building modern applications

Kubernetes Architecture
Scheduler Controllers
etcd
API Server
Kubelet
Kube-proxy
Control Plane
Kubelet
Kube-proxy
Kubelet
Kube-proxyData Plane

Shared Responsibility Model
Customers
Data plane
Amazon EKS
Control PlaneContainers

Amazon EKS Compliance
Amazon EKS
Control Plane

Amazon EKS Security: A Shared Responsibility
Customers
Data planeContainers
a. Data plane security lifecycle
b. Container security lifecycle
1. Understand ownership boundaries
2. Build your threat model around:

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved..© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved..
So, you’re a developer …

Where do images come from?
Images are…
…separated by tags…
…(optionally) stored in repositories…
…organized in registries
image:tagrepository/registry/

Where does nginx come from?
Image Name
Default tag: latest
Default repository: library
Default registry: Docker Hub (docker.io)
nginx:latestlibrary/docker.io/

AWS ECR CoreOS Quay JFrog ArtifactoryDocker Registry
Registries
• Public/Private
• Cloud/Self-Hosted
• Open Source/Enterprise

Private Registries
…should be trusted if it’s yours!
• Keep images close to runtime
• Lower latency
• Reduce “man-in-the-middle” attacks
• Controlled maintenance window
• ”Cached” image copy in AWS, even if not the original

Public Registries
…should NOT
be trusted!
Docker Hub has official repositories
• Essential base OS repositories
• Popular runtimes, data stores, and services (PaaS)
• Best Practices examples
• Security Scanned and Updated
https://hub.docker.com/explore/

Marketplace Registries
“Trust butVerify”
Docker Store
• Submitted by StoreVendor Partners
• Published and maintained by company
• Offers “Docker Certified” images
• Search includes Docker Hub (as option)
https://store.docker.com/search

Containerise an app

Selecting a Base Image
Know where your base image comes from
• What base image/OS does nginx use?
• What repository did this base image come from?
• Who published it to that repository?
• When was it last published?
Don’t assume that everything on DockerHub (or Github, or NPM) is secure!

From https://hub.docker.com/_/nginx/

From docker-nginx/mainline/stretch/Dockerfile

Selecting a Base Image Tag
Avoid latest tag on images
• Can introduce security flaws
• Can cause security scan failures
Use a defined tag
• nginx:1.13
https://hub.docker.com/_/nginx/

FROM scratch Base Image
scratch…
• does not contain any files
• is an empty base image
• uses bootfs from kernel
Golang is a good choice
• binary as single file!

Unprivileged Containers
Unless you specify a user,
your containers will run as the
same user as Docker
This means ROOT
Give your container a user!

Minimise Attack Vectors
• Write security-minded code!
• No hard-coded passwords
• Sanitise inputs (OWASPTop 10 etc)
• Test,Test,Test
• AppSec Review
• Encryption SDK

Multi-Stage Builds
• Each Dockerfile command is one layer
• Shell tricks (like &&) to reduce layers?
• Maintain multiple Dockerfiles?
• Dev Dockerfile: build/debug
• Prod Dockerfile: just run
• Solution: Multi-Stage Builds:
• Or build without Docker at all?
• Kaniko / Skaffold
FROM golang:1.7.3
WORKDIR /go/src/scott/app
COPY app.go .
RUN go build -o app .
FROM alpine:3.7
WORKDIR /root/
COPY --from=0 /go/src/scott/app .
CMD ["./app"]

Humans make mistakes…

CI/CD for Image Builds
Source Build Store ImageTest
• Scanning/Linting code & deps (Dockerfile et al)
• Image Scanning in Pipeline (via Test phase)
• Image Signing (Cryptographic Verification)

Build Time Scanners
Twistlock
Clair Black Duck
Trend Micro Deep Security
SmartCheck

Image/Runtime Scanners
Twistlock Aqua Security NeuVector
Clair Black Duck Tenable

https://aws.amazon.com/containers/partner-solutions/

So, you’re in Ops ….

Development has given you a new “image” …
Now what?
• Verify the image
• Check Dockerfile from source
• Validate build pipeline/artifact
• Check scan results
• Create Runtime Configuration

Runtime Configuration
• Shared Responsibility between Dev and Ops
• Meets at the runtime (Task Definition/Pod Specification)
Runtime ConfigDevelopment Operations

Runtime Configuration
• A declarative set of instructions that tells the scheduler how
to run containers.
Examples:
Task Definition Pod Specification

Reduce/Remove SELinux Capabilities
• Linux Most containers do not need root privileges
• Logging handled through runtime
• Network is managed for container
• Do not need SSH to container
• Cron as scheduled container
• Docker drops unnecessary capabilities
• Reduces blast radius of compromised container
CAP_AUDIT_CONTROL
CAP_AUDIT_READ
CAP_BLOCK_SUSPEND
CAP_DAC_READ_SEARCH
CAP_FOWNER
CAP_IPC_LOCK
CAP_IPC_OWNER
CAP_LEASE
CAP_LINUX_IMMUTABLE
CAP_MAC_ADMIN
CAP_MAC_OVERRIDE
CAP_NET_ADMIN
CAP_NET_BROADCAST
CAP_SYS_ADMIN
CAP_SYS_BOOT
CAP_SYS_MODULE
CAP_SYS_NICE
CAP_SYS_PACCT
CAP_SYS_PTRACE
CAP_SYS_RAWIO
CAP_SYS_RESOURCE
CAP_SYS_TIME
CAP_SYS_TTY_CONFIG
CAP_SYSLOG
CAP_WAKE_ALARM
CAP_AUDIT_WRITE
CAP_CHOWN
CAP_DAC_OVERRIDE
CAP_FSETID
CAP_KILL
CAP_MKNOD
CAP_NET_BIND_SERVICE
CAP_NET_RAW
CAP_SETGID
CAP_SETFCAP
CAP_SETPCAP
CAP_SETUID
CAP_SYS_CHROOT

Namespaces
• Linux kernel feature to partition kernel resources to a specific
set of processes
• Isolates from other similar resources
• Mount points (mnt)
• Process IDs (pid)
• Networks (net)
• Users (user)
• InterProcess Communication (ipc)
• Control Groups (cgroup)

Control Groups (cgroups)
• Linux kernel feature to limit and isolate hardware access to a
specific set of processes
• CPU, memory, disk I/O, network
• Includes accounting and prioritization
• Prevents oversubscription on the hostVM
• “By default, a container has no resource constraints and can use as
much of a given resource as the host’s kernel scheduler allows”
(Docker)
• “Noisy Neighbours”

Resource Limits (ulimits)
• “User Limit” - restrict usage of resources per user
• Docker restricts usage of resources per container
• Per process (prlimit)
• Soft/Hard Limit per item
• Soft limit can be increased by container
• Hard limit can be increased with privilege
• Example: number of processes (nproc)
• Avoid the fork-bomb
core
cpu
data
fsize
locks
memlock
msgqueue
nice
nofile
nproc
rss
rtprio
rttime
sigpending
stack

Seccomp
• “Secure Computing Mode”
• Transition a process into a secure state
• Irreversible
• Supported from Docker 1.10
• Controls system calls by container
• Docker-supported seccomp profiles
• Default Profile disables 44 system calls out of 300+
• Custom seccomp profile on containers via Docker security options
• Examples: unconfined, default, <custom>

(Un-)Privileged Containers
• sudo-esque permissions on the underlying host
• Grants
• Access to all devices
• AppArmor/SELinux unrestricted access
• Ability to Change Hard Resource Limits within container
• Most containers should be UN-privileged (default)
• Only needed for some use cases:
• d-in-d (Docker in Docker – specialised use cases)
• Containers that support Containers (e.g. monitoring agents)

Read-Only Containers
• Changes the root filesystem to read-only
• Unable to write to files (or change files)
• Can write logs to stdout and stderr
• Supports immutable workloads
• If something changes, launch new containers

IAM Roles for Instances
• Restricts containers running
on instance to specific set of
permissions
• Shared permissions among
containers on instance

IAM Roles for Tasks/Pods
• Least Privilege
• Moves authorisation to
container level
• Containers on same instance
do not share permissions!
• Small change to IAM Role
• K8s pods require additional
daemon
• jtblin/kube2iam

Network Security for Instances
• ENI on instance
• Traffic bridged to container
• Shared security group among
containers on instance
• Default Network Namespace
• Updates to security groups
based on abstracted IPs

Network Security for Tasks/Pods
• Network namespace for each
task/pod*
• ENIs attached to namespace
• Traffic routed to container
• Isolated security
with each task/pod
• EKS pods may share ENI
• Same instance
• Same subnet
*Pods in the same ReplicaSet share the same Network Namespace

Network Security for Tasks/Pods
Container Network Interface Plugin
• containernetworking/cni
• configures network interfaces in Linux containers
Used with runtimes:
• ECS (awsvpc mode)
• Fargate
• EKS (with aws/amazon-vpc-cni-k8s)

Protecting the Host

Start with the Operating System
• Only include binaries/libraries necessary to run containers
• Minimise the image
• Potential performance boost
• Container Runtime handles many system calls
• Suggested Operating Systems/Images:
(I like http://www.projectatomic.io/ , too)
Optimized AMIs

Remember EC2 and Linux Best Practices
• Restrict communication to your host by using tools outside
the host (same as EC2):
• Security Groups, NACLs, Firewalls,WAF, IDS / IPS
• Minimal number of users with specific permissions
• Fine-grained controls using tools like SELinux,AppArmor
• No passwords on host
• IAM Role
• Secrets Management
• No root access
• AWS Security ProcessesWhitepaper (and others!)

Ongoing Operational Practices
• Patch the host often
• Recurring reviews of settings and configuration
• Monitoring, Logging, Auditing
• Cloudwatch, Splunk
• Datadog, New Relic
• Prometheus
• Alcide
• CIS Benchmark
• GitHub: docker-bench-security
• http://dev-sec.io/
 Launch new immutable hosts

Let someone else worry about hosts?!
• No Host = Less Operations
• Only secure the task, not the host
• Security Groups, NACLs, Firewalls,WAF, IDS / IPS
• No users, no passwords, no root access
• No Patching (Fargate launches new hosts)
• Recurring reviews of settings and configuration
• Monitoring, Logging, Auditing

Securely Deploying Containers

CI/CD for Container Deployments

Wider Security Ecosystem / Next Steps
• Alternative Container Engines
• rkt
• gVisor
• Kata
• IAM on the container level
• ECS/Fargate – IAM forTask Roles
• EKS/K8s – kube2iam, kiam
• Use AWS Security Services, Partners, and SMEs
• Test,Test,Test!

Orchestration and Management
Security

Shared Responsibility Model
Responsible for
Security “of” the
cloud
Responsible for
Security “in” the
cloud
Network and Firewall Configuration
Identity & Access Management
Customer Data
Compute Storage Database Networking
Regions Availability Zones Edge Locations
Operating System
Applications Platform
AWSCUSTOMER

IAM = Who can do what in the platform and/or cluster?
People Code / Pipelines
Photo & Licence Photo & Licence

AWS IAM vs. Kubernetes RBAC
If using ECS then that is a native
extension of the AWS platform and is
fully managed by AWS IAM.
If running Kubernetes, either yourself
or via EKS, then you need to
understand and configure BOTH
AWS IAM and Kubernetes RBAC.

K8s action allowed/denied
EKS: IAM Authentication + kubectl
Authorization of AWS Identity
against Kubernetes RBAC
K8s API
Passes AWS Identity
Verifies AWS Identity
kubectl
AWS IAM
Authentication

Kubernetes RBAC built-in ClusterRoles
Default ClusterRole
Description
cluster-admin
Allows super-user access to perform any action on any resource. When used in
a ClusterRoleBinding, it gives full control over every resource in the cluster and in all
namespaces. When used in a RoleBinding, it gives full control over every resource in the
rolebinding's namespace, including the namespace itself.
admin
Allows admin access, intended to be granted within a namespace using a RoleBinding. If used in
used in a RoleBinding, allows read/write access to most resources in a namespace, including the
including the ability to create roles and rolebindings within the namespace. It does not allow
allow write access to resource quota or to the namespace itself.
edit
Allows read/write access to most objects in a namespace. It does not allow viewing or modifying
modifying roles or rolebindings.
view
Allows read-only access to see most objects in a namespace. It does not allow viewing roles or
roles or rolebindings. It does not allow viewing secrets, since those are escalating

Kubernetes RBAC Basics
Kubernetes has Roles which are
defined and apply within a single
namespace (a virtual cluster) and
ClusterRoles which apply cluster-
wide across all namespaces.
You define custom roles describing
resources (such as pods and nodes)
and which verbs (such as get,
update and delete) are allowed
against them.
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'

AWS IAM Role per Task / Pod?
Assigning an IAM Role to a ECS
Task is an included feature in the
AWS Platform and ‘just works’.
If running Kubernetes then you need
to add either kube2iam or kiam to
leverage this functionality.
Assigning an IAM role to an Instance/Task/Function means the right AWS access key
and secret to call the AWS CLI/SDK are transparently obtained and rotated.

VPC
IAM Roles for Nodes and Pods
API Server
Kubelet
Kube-proxy
Kubelet
Kube-proxy
Kubelet
Kube-proxy
AWS Key Management
Service
AWS Secrets Manager Amazon GuardDuty Amazon Inspector AWS CloudTrail Amazon CloudWatch AWS Config AWS Systems Manager

Logging and Auditing the Control Plane
ECS is part of the AWS platform and the
control plane logs go to CloudTrail just like
the rest of the platform.
Kubernetes’ control plane logs include an
audit trail. When using EKS these logs are
not enabled by default but you can (and
should) enable logging to CloudWatch Logs.
Logging of the control plane, especially around an audit trail of API actions, is an
important aspect of security.

AWS Security Groups vs. Kubernetes Network Polices
If using ECS then that is an
extension of the AWS platform and
you only need to understand and
configure AWS VPC and Security
Groups.
If running Kubernetes yourself or
EKS then you need to understand
and configure BOTH AWS VPCs /
Security Groups as well as
Kubernetes Network Polices

Default/Root Namespace
lo
eth0
Task Namespace
lo
eth1
Networking with ECS
When using ECS with the aws-vpc
network mode (optional for EC2 mode
but required for Fargate mode) then
each Task gets its own dedicated
Elastic Network Interface (ENI).
Since each Task is 1:1 with an ENI
and each ENI is 1:1 with a Security
Group (SG) that means any
communication in/out of each Task
goes through its SG on both ingress
and egress.

Micro-segmenting with Security Groups
You can use a security group ID as both a source and destination for other
security group rules – both to loop back to itself or referencing other SGs.
• This enables network segmentation without complex subnetting

EKS Control Plane API Endpoints – Make Them Private
Worker VPC (your account)
Kubectl
Master VPC (AWS account)
etcd
AZ
1
AZ
2
API Server
etcd
API Server
EKS-owned
ENIs
Public == false
Private == true
prod-cluster-123.eks.amazonaws.com
Private hosted zone
Kubelet
AZ
1
Worker
node
Kube-proxy
Kubelet
AZ
2
Worker
node
Kube-proxy

Networking with EKS
ENI
Secondary IPs:
10.0.0.1
10.0.0.2
10.0.0.1
10.0.0.2
ENI
10.0.0.20
10.0.0.22
Secondary IPs:
10.0.0.20
10.0.0.22
ec2.associateaddress()
VPC Subnet – 10.0.0.0/24
Instance 1 Instance 2
VPC

Installing a Network Policy Provider on Kubernetes
You first need to add a Network Policy Provider to EKS / Kubernetes in order to
use Network Policies. A popular one covered in our documentation is Calico.
https://docs.aws.amazon.com/eks/latest/userguide/calico.html

Frontend
Cats Dogs
kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: default-deny
spec:
podSelector:
matchLabels: {}
catsndogs-namespace
Network Policies on Kubernetes

Frontend
Cats Dogs
catsndogs-namespace
kind: NetworkPolicy
metadata:
name: public-to-frontend
spec:
podSelector:
matchLabels:
role: frontend
ingress:
- from:
- ipBlock:
cidr: "0.0.0.0/0"
ports:
- protocol: TCP
port: 80

Frontend
Cats Dogs
catsndogs-namespace
kind: NetworkPolicy
metadata:
name: frontend-to-cats
spec:
podSelector:
matchLabels:
role: cats
ingress:
- from:
- podSelector:
matchLabels:
role: “frontend”
ports:
- protocol: TCP
port: 80

Secure Cloud Edition (CE)
Features:
• Enterprise Support from Tigera
• Host-to-Host IPSEC Encryption
• Flow Logs enriched with Kubernetes Workload Metadata
• Integration between AWS Security Groups and Network Policies

Secure Cloud Edition (CE)

Alternative – Multiple NodeGroups or EKS Clusters
One way that you can both assign both EC2 Instance-level IAM Roles
(without kops or kiam) as well as fully trust Security Group-based micro-
segmentation without Tigera is to have a different set of worker nodes or
even entirely separate Clusters for different services or trust boundaries.
EKS has the concept of a NodeGroup which is a separate Auto Scaling
Group of worker Nodes that can be labelled in such a way that you can
limit which pods/services can be run on them.
https://kubernetes.io/docs/concepts/configuration/assign-pod-node

Security Benefits Of Fargate
We do more, you do less.
• Patching (OS, Docker, ECS Agent, etc.)
• Task isolation (via Firecracker)
• No --privileged mode for containers
• Requires awsvpc network mode so ENI/SG perTask
• No runtime access for users (ssh or interactive Docker)

EC2-mode ECS Shared Responsibility Model
Customer Data
Storage Database Networking
Regions Edge Locations
Operating System
Images ECSConfig
ECSControl Plane
Instance Scaling
Compute
Availability Zones
AWSCUSTOMER

ECS Fargate Shared Responsibility Model
Customer Data
Storage Database Networking
Regions Edge Locations
Operating System
Images ECSConfig
ECSControl Plane
Instance Scaling
Compute
Availability Zones
AWSCUSTOMER

Updating EKS
• Kubernetes has a new major version every quarter
• Kubernetes has a new minor version quite regularly
• Sometimes Kubernetes updates are security-related
• EKS has APIs to trigger an update of the control plane
• You then need to update the worker Nodes - both re:
Kubernetes as well as Docker and OS
• Often the workers are in an Autoscaling Group
so this means building updating AMIs
• We provide a regularly updated EKS Node AMI
as well as scripts to build your own.

Kubernetes and container security issues
https://aws.amazon.com/security/security-bulletins/
https://aws.amazon.com/blogs/compute/anatomy-of-cve-2019-5736-a-runc-container-escape/

Amazon EKS update lifecycle
May 21, 2019
Blog: https://aws.amazon.com/blogs/compute/updates-to-amazon-eks-version-lifecycle/

AWS Parameter Store, Secrets Manager and Kubernetes
Secrets
AWS has both Parameter Store and Secrets
Manager to store your Secrets. They are
integrated into ECS but you’ll need to call
them within the Pod on Kubernetes via our
CLI or SDK.
Kubernetes’ built-in Secrets functionality
stores secrets in its control plane and puts
them into running Pods via Environment
Variables or files in the filesystem. You can’t
use these outside of the Kubernetes cluster.

Kubernetes-Specific Security
Add-Ons

Admission Controllers in Kubernetes

Admission Controllers in Kubernetes
Source: https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

Open Policy Agent – CNCF Project (Incubating)
Logic Data
Request
Policy Query Policy Decision (Allow/Deny)
Source: OPA Deep Dive https://www.youtube.com/watch?v=n94_FNhuzy4

Security – AWS / Containers
• https://aws.amazon.com/security/
• https://d1.awsstatic.com/whitepapers/Security/Security_Compute_Services
_Whitepaper.pdf
• https://d1.awsstatic.com/whitepapers/AWS_CAF_Security_Perspective.pdf
• https://d1.awsstatic.com/whitepapers/Security/AWS_Security_Best_Practic
es.pdf
• https://docs.docker.com/engine/security/security/
• https://github.com/docker/docker-bench-security
• https://www.infoworld.com/article/3234671/containers/4-container-security-
tools-for-docker-and-kubernetes.html
• https://thenewstack.io/5-docker-security-best-practices/

Security – Partners / OSS / Third Party
• https://www.twistlock.com/
• https://www.aquasec.com/
• https://www.datadoghq.com/
• https://newrelic.com/
• https://sysdig.com/
• https://www.tenable.com/
• https://neuvector.com/
• https://www.blackducksoftware.com/
• https://prometheus.io/
• https://alcide.io/
• Many, many others ….

Data Plane Security - Container Runtime Interface
Kubelet docker-containerd docker-containerd-shim docker-runc Container
docker-containerd-shim docker-runc Container
docker-containerd-shim docker-runc Container
Inside the CNCF Project Security Reviews: https://www.youtube.com/watch?v=0BkKpsrUo5k
CRI

Csa container-security-in-aws-dw

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Csa container-security-in-aws-dw

Similar to Csa container-security-in-aws-dw (20)

More from Cloud Security Alliance, UK chapter

More from Cloud Security Alliance, UK chapter (11)

Recently uploaded

Recently uploaded (20)

Csa container-security-in-aws-dw