Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017

419 views

Published on

While organizations gain agility and scalability when they migrate to containers and microservices, they also benefit from compliance and security, advantages that are often overlooked. In this session, Kelvin Zhu, lead software engineer at Okta, joins Mitch Beaumont, enterprise solutions architect at AWS, to discuss security best practices for containerized infrastructure. Learn how Okta built their development workflow with an emphasis on security through testing and automation. Dive deep into how containers enable automated security and compliance checks throughout the development lifecycle. Also understand best practices for implementing AWS security and secrets management services for any containerized service architecture.

  • Be the first to comment

Architecting Container Infrastructure for Security and Compliance - CON406 - re:Invent 2017

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Architecting Container Infrastructure for Security and Compliance M i t c h B e a u m o n t , S o l u t i o n s A r c h i t e c t , A W S K e l v i n Z h u , P r o d u c t i v i t y T e a m L e a d D e v e l o p e r , O k t a C O N 4 0 6 N o v e m b e r 3 0 , 2 0 1 7
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “If everything seems under control, you're not going fast enough.” —Mario Andretti Picture: 1990 Indy 500 / DoctorIndy / License © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mario Andretti crashed in the 1992 Indianapolis 500. The reason? He lost control of his car under acceleration and smashed into the wall. Mario Andretti crashed in the 1992 Indianapolis 500. The reason? He lost control of his car under acceleration and smashed into the wall. Picture: F1 Collision / TMWolf / License© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image Security Credentials and Secrets Kernel Security Denial of Service Container Breakouts Architecting a secure infrastructure Runtime Security 503
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kernel and Host Security Denial of Service Container Breakouts Image Security Secrets Runtime
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Containers as opposed to VMs Server Host OS Hypervisor Guest OS Guest OS Bins/Libs Bins/Libs Cats App Cats App Server Host OS Container Engine Bins/Libs CatsApp CatsApp Bins/Libs DogsApp DogsApp Virtual Machines Containers
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Containers and VMs Server Host OS Container Engine Bins/Libs CatsApp CatApp Bins/Libs DogsApp DogsApp Hypervisor
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Machine Image builds EC2 instance ECS Optimized AMI • ECS Optimised Amazon Linux • RHEL • Ubuntu • Container Centric OS
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Machine Image builds EC2 instance EC2 instance Foundational AMI • Security best practices • Provisioners • Loggers • Config, and so on ECS Optimized AMI • ECS Optimised Amazon Linux • RHEL • Ubuntu • Container Centric OS
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Machine Image Builds EC2 instance • ECS Optimised Amazon Linux • RHEL • Ubuntu • Container Centric OS EC2 instance Foundational AMI • Frameworks • Applications EC2 instance Full Stack AMI • Security best practices • Provisioners • Loggers • Config, and so on ECS Optimized AMI
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building an AMI pipeline DevOps Persona Code Repository
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building an AMI pipeline DevOps Persona Code Repository CI/CD Pipeline
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building an AMI pipeline DevOps Persona Code Repository CI/CD Pipeline Base AMI AMI Builder ECS Optimized AMI Public Catalog
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building an AMI pipeline DevOps Persona Code Repository CI/CD Pipeline Base AMI AMI Builder ECS Optimized AMI Public Catalog Customer AMI
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building an AMI pipeline DevOps Persona Code Repository CI/CD Pipeline Base AMI AMI Builder ECS Optimized AMI Public Catalog CloudFormation Customer AMI
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kernel and Host Security Denial of Service Container Breakouts Image Security Secrets Runtime
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Limiting resources E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Limiting resources E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Limiting resources E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Limiting resources E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Limiting resources • Define your resource limits up front • It’s not just memory and CPU. • Monitor usage • Leverage Auto Scaling
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kernel and Host Security Denial of Service Container Breakouts Image Security Secrets Runtime
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segmentation E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segmentation – Placement constraints aws ecs put-attributes --cluster mycatsanddogscluster --attributes “name=CDE,value=true,targettype=container-instance,targetId=<blahblahblah>”
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segmentation – Placement constraints aws ecs put-attributes --cluster mycatsanddogscluster --attributes “name=CDE,value=true,targettype=container-instance,targetId=<blahblahblah>” "placementConstraints": [ { "expression":"attribute:CDE==true", "type": "memberOf" }] Task Definition Container Instance
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segmentation- Placement constraints E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e C D E E C S T a s k C a t s C o n t a i n e r E C S T a s k D o g s C o n t a i n e r C o n t a i n e r I n s t a n c e N o n - C D E E C S T a s k D o g s C o n t a i n e r
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segmentation- Placement Constraints E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e C D E E C S T a s k C a t s C o n t a i n e r E C S T a s k D o g s C o n t a i n e r C o n t a i n e r I n s t a n c e N o n - C D E E C S T a s k D o g s C o n t a i n e r This doesn’t need to happen!
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segmentation – Task ENI E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p S e c u r i t y G r o u p E C S T a s k C a t s C o n t a i n e r S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r S e c u r i t y G r o u p
  29. 29. 1. Pre ENI Attachment: The primary ENI (eth0) is in the default namespace Default/Root Global Namespace docker0 eth0 lo
  30. 30. 1. Pre ENI Attachment: The primary ENI (eth0) is in the default namespace 2. ENI Attached: The new ENI (eth1) is in the default namespace. Default/Root Global Namespace Default/Root Global Namespace docker0 docker0 eth0 lo lo eth0 eth1
  31. 31. Default/Root Global Namespace Default/Root Global Namespace Default/Root Global Namespace docker0 docker0 eth0 lo lo eth0 eth1 lo eth0 docker0 ecs0 eth1 lo ve- c1 3. ENI Provisioned: The ECS Agent invokes CNI plugins to move the new ENI into a new namespace and configure it with the addresses and routes. 1. Pre ENI Attachment: The primary ENI (eth0) is in the default namespace 2. ENI Attached: The new ENI (eth1) is in the default namespace.
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Capabilities Kernel Server Cats Container Dogs Container NET_BIND_SERVICE CHOWN • AUDIT_CONTROL • BLOCK_SUSPEND • DAC_OVERRIDE • MKNOD • … • …
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ”LinuxParameters": { ”capabilities": { add: [“AUDIT_CONTROL”, ...], drop: [“MKNOD”, ...] } } docker run --cap-drop ALL --cap-add .. Dropping and adding capabilities
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kernel and Host Security Denial of Service Container Breakouts Image Security Secrets Runtime
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Do you really know your image?
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Layers, binaries, and dependencies, oh my! /usr/bin/passwd /usr/bin/gpasswd /usr/bin/chsh /usr/bin/chfn /usr/bin/newgrp /bin/su /bin/umount /bin/mount $ find / -user root -perm -4000 -print Do I need binaries with SUID flag?
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Layers, binaries, and dependencies, oh my! IMAGE CREATED CREATED BY SIZE a8bdc7fdaa4f 2 weeks ago /bin/sh -c #(nop) CMD ["httpd-foreground"] 0B <missing> 2 weeks ago /bin/sh -c #(nop) EXPOSE 80/tcp 0B <missing> 2 weeks ago /bin/sh -c #(nop) COPY file:761e313354b918... 133B <missing> 2 weeks ago /bin/sh -c set -eux; buildDeps=" bzip2... 9.72MB <missing> 2 weeks ago /bin/sh -c #(nop) ENV APACHE_DIST_URLS=ht... 0B <missing> 2 weeks ago /bin/sh -c #(nop) ENV HTTPD_PATCHES= 0B <missing> 2 weeks ago /bin/sh -c #(nop) ENV HTTPD_SHA256=777753... 0B <missing> 2 weeks ago /bin/sh -c #(nop) ENV HTTPD_VERSION=2.4.29 0B <missing> 4 weeks ago /bin/sh -c apt-get update && apt-get inst... 44.2MB <missing> 4 weeks ago /bin/sh -c { echo 'deb http://deb.debian... 161B <missing> 4 weeks ago /bin/sh -c #(nop) ENV OPENSSL_VERSION=1.0... 0B <missing> 4 weeks ago /bin/sh -c #(nop) ENV NGHTTP2_VERSION=1.1... 0B <missing> 4 weeks ago /bin/sh -c #(nop) WORKDIR /usr/local/apache2 0B <missing> 4 weeks ago /bin/sh -c mkdir -p "$HTTPD_PREFIX" && ch... 0B <missing> 4 weeks ago /bin/sh -c #(nop) ENV PATH=/usr/local/apa... 0B <missing> 4 weeks ago /bin/sh -c #(nop) ENV HTTPD_PREFIX=/usr/l... 0B <missing> 4 weeks ago /bin/sh -c echo 'deb http://deb.debian.org... 55B <missing> 4 weeks ago /bin/sh -c #(nop) CMD ["bash"] 0B <missing> 4 weeks ago /bin/sh -c #(nop) ADD file:55b071e2cfc3ea2... 123MB
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Best practices • Signing container images (Docker content trust) • Set filesystems to be read-only (readonlyRootFilesystem) • Remove setuid/setgid binaries from images (defang) • Set containers to run as non-root user • Consider running static binaries
  39. 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image security Trusted Images • ECR • docker-hub-enterprise • elastic.io/running-a- docker-private-registry- on-ec2 Minimal OS Base Images • Minimum OS (alpine etc) • Docker cis-docker-bench • Image signing Container Vulnerability Scanning • TwistLock • Clair • NueVector
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DevSecOps container pipeline Developers Security Engineers Ops Engineers AWS CodeCommit Task Definition Dockerfile FROM centos:centos7 MAINTAINER cb@demo.com RUN yum -y update RUN yum -y install openssh- server U SER sshduser EXPOSE 5432 ENTRYPOINT sshd
  41. 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DevSecOps container pipeline Docker image AWS CodeBuild Developers Security Engineers Ops Engineers Task Definition AWS CodeCommit Dockerfile FROM centos:centos7 MAINTAINER cb@demo.com RUN yum -y update RUN yum -y install openssh- server U SER sshduser EXPOSE 5432 ENTRYPOINT sshd
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DevSecOps container pipeline Docker image AWS CodeBuild Validate Configuration > Merge >  python ./check_dockerfile.py ./examples/Dockerfile-demo |jq ".warnings.warnings[].message" "yum clean all is not used" "installing SSH in a container is not recommended" "No 'USER' instruction" Developers Security Engineers Ops Engineers AWS CodeCommit Task Definition Dockerfile FROM centos:centos7 MAINTAINER cb@demo.com RUN yum -y update RUN yum -y install openssh- server U SER sshduser EXPOSE 5432 ENTRYPOINT sshd
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DevSecOps container pipeline Docker image AWS CodeBuild Validate Configuration > Merge >  python ./check_dockerfile.py ./examples/Dockerfile-demo |jq ".warnings.warnings[].message" "yum clean all is not used" "installing SSH in a container is not recommended" "No 'USER' instruction" Developers Security Engineers Ops Engineers AWS CodeCommit 0 50 100 150 DEV INT TEST QA PROD Vulnerabilities Low Medium High Scan Docker Image > Publish > Task Definition Dockerfile FROM centos:centos7 MAINTAINER cb@demo.com RUN yum -y update RUN yum -y install openssh- server U SER sshduser EXPOSE 5432 ENTRYPOINT sshd
  44. 44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. DevSecOps container pipeline Amazon ECR Docker image AWS CodeBuild Validate Configuration > Merge >  python ./check_dockerfile.py ./examples/Dockerfile-demo |jq ".warnings.warnings[].message" "yum clean all is not used" "installing SSH in a container is not recommended" "No 'USER' instruction" Developers Security Engineers Ops Engineers AWS CodeCommit 0 50 100 150 DEV INT TEST QA PROD Vulnerabilities Low Medium High Scan Docker Image > Publish > Task Definition Dockerfile FROM centos:centos7 MAINTAINER cb@demo.com RUN yum -y update RUN yum -y install openssh- server U SER sshduser EXPOSE 5432 ENTRYPOINT sshd
  45. 45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kernel and Host Security Denial of Service Container Breakouts Image Security Secrets Runtime
  46. 46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storing secrets in environment variables • Suggested by 12-factor apps (III. Config) • Environment variables can be seen in too many places • Linked containers • ECS API calls • Docker inspect • Can’t be deleted https://12factor.net/
  47. 47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Protecting secrets using IAM roles for tasks Benefits • Simplify usage of AWS SDKs in containers • Credential isolation between tasks/container • Authorization per task/container • Auditability in Amazon CloudTrail with taskArn
  48. 48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM EC2 roles Cats Contai ner Dogs Contai ner C o n t a i n e r I n s t a n c e E C S T a s k E C S T a s k C a t s B u c k e t D o g s B u c k e t
  49. 49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM task roles Cats Contai ner Dogs Contai ner C o n t a i n e r I n s t a n c e E C S T a s k E C S T a s k C a t s B u c k e t D o g s B u c k e t
  50. 50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM roles for tasks, explained 1. ECS agent periodically queries ECS control plane 2. Control plane generates ID token  Auto-rotated 3. ECS agent:  Constructs HTTP URL for each container  Sets AWS_CREDENTIALS_ENDPOINT in HostConfig 4. AWS SDK extracts URL
  51. 51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3-based secrets storage • Secrets stored in S3 bucket • Accessed via IAM roles for EC2 or IAM roles for tasks • Enforce encryption at rest and flight via IAM policies and KMS • Use VPC endpoint for S3 to lock down access from certain VPCs
  52. 52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EC2 systems manager parameter store • Secrets stored in SSM parameter store • Accessed via IAM roles for EC2 or IAM roles for tasks • Enforce encryption at rest and flight via IAM policies and KMS • Govern permission to decrypt using specific KMS through IAM policy
  53. 53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Using IAM roles to retrieve secrets Cats Contai ner C o n t a i n e r I n s t a n c e E C S T a s k Parameter Store secretStringAWS KMS
  54. 54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kernel and Host Security Denial of Service Container Breakouts Image Security Secrets Runtime
  55. 55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VPC flow logs and Task ENI E C S T a s k C a t s C o n t a i n e r C o n t a i n e r I n s t a n c e S e c u r i t y G r o u p S e c u r i t y G r o u p E C S T a s k D o g s C o n t a i n e r S e c u r i t y G r o u p 630247214269 eni-0123456a 10.0.1.221 10.76.2.101 27039 22 6 5 268 1466491141 1466491200 REJECT OK 22 StopTask: { task: “52c…” } $ docker diff / inspect
  56. 56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Access Control Cloud Native Firewalling Runtime Defense Compliance Vulnerability Management
  57. 57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Container Security at Okta K e l v i n Z h u – P r o d u c t i v i t y T e a m L e a d D e v e l o p e r
  58. 58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Topics • How we use containers • How we secure: • Hosts • Containers • Images
  59. 59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What security is to us
  60. 60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Keeping it reliable • Security breaches have the potential to bring down our infrastructure • Improper use of secrets can cause unexpected and damaging effects • Bad security results in costly problems! But with automation it doesn’t have to be hard! • We’ll focus on a few examples of how we apply security principles to our Docker containers
  61. 61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How we use containers
  62. 62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. High-level flow
  63. 63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. High level flow • Developer writes code and commits to GitHub • Bacon Web application receives commit and processes into test runs • Test runs inserted as messages to an AWS SQS queue • ECS cluster of instances pick up messages from SQS queue and run them on Docker containers • Test results reported back to Bacon Web application
  64. 64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container generation and usage
  65. 65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container generation and usage
  66. 66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container generation and usage
  67. 67. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container generation and usage • Individual runs generate artifacts, some of which are Docker artifacts • Docker images uploaded to repositories • Individual Amazon ECS clusters brought up by AWS CloudFormation for either CI or application images • Application images deployed to Amazon ECS services
  68. 68. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployment scale • Two different types of deployments: • CI scales up to thousands and down to just a few containers according to messages in queue • One time use containers that finish a task within an hour • Almost no containers necessary overnight and during weekends • Applications scale up and down according to web traffic • Longer lived containers
  69. 69. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container host security
  70. 70. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Components of host security • AMI • Accessibility • Ease of change
  71. 71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Locking down the AMI • The very packages we install are potential threats! • Viruses • Backdoors • Creation of a hardened AMI • Even if an installed package is unused, if a vulnerability is discovered patching is required! • AMI with only minimum necessary to run application • Follow Center of Internet Security (CIS) checklists to ensure generated AMIs are secure
  72. 72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Locking down the AMI Steps to build, test, and certify an AMI • Define packer template • Test: run CIS • Share foundation AMI to DMZ account • Share foundation AMI to other AWS accounts • Delete foundation AMI from originating AWS account
  73. 73. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Preventing unauthorized access
  74. 74. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Preventing unauthorized access
  75. 75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Preventing unauthorized access • All hosts behind the VPC • Only users with VPN can communicate with hosts • Security groups to limit approved traffic between hosts • SSH keys given only to trusted set of users • Regular rotation to prevent leaking
  76. 76. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ease of change • Always prepare for the worst case • Zero days! • Time is of the essence in minimizing damage • To enable easy patching, AMI creation is hooked into the CI system aws ec2 run-instances --count 1 aws ec2 wait instance-status-ok --instance-id ${INSTANCE_ID} aws ec2 create-image --instance-id ${INSTANCE_ID} aws ec2 wait image-available --image-ids ${IMAGE_ID} aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}
  77. 77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  78. 78. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  79. 79. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  80. 80. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  81. 81. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  82. 82. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  83. 83. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users UpdatePolicy: AutoScalingRollingUpdate MaxBatchSize: 5 MinInstancesInService: 5 PauseTime: “PT10M”
  84. 84. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Malicious users
  85. 85. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ease of change • AWS CloudFormation for each Amazon ECS service • Auto Scaling used to reduce reliance on individual hosts staying alive • If a host becomes compromised or starts failing, terminate it and allow a new one to come up • Auto replacement of all hosts every day ensures manual patches do not occur
  86. 86. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container security
  87. 87. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Credentials on containers
  88. 88. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Credentials on containers • Industry standards • Pull credentials from external protected storage encrypted both on disk and in transit into flat file available within container • Pull credentials from external protected storage encrypted both on disk and in transit into memory (slightly better) • Both approaches allow anyone with access to code run in container to retrieve the secrets!
  89. 89. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Protecting creds • The next step up! • Containers run in protected networks to prevent unauthorized direct access • Standard and parameterized code that is run on containers and modification disallowed • Sanitize logs and output files for any retrieved secrets • Prevents ability to sniff out secrets and prevents leaks
  90. 90. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service access
  91. 91. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Minimizing service access • Containers need access to services, but only the minimal set they actually need! • Task definitions associated with each Amazon ECS task/service type given their own task IAM role • Avoid use of “*” in access policies! • Access advisor – Remove access to unnecessary services • CloudTrail logs – More granular view into exactly which resources are accessed
  92. 92. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image security
  93. 93. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container generation and usage
  94. 94. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker within Docker! • By mounting Docker from the host to the container, we can run Docker and Docker Compose inside the container • -v /usr/bin/docker:/usr/bin/docker • Utilized in the CI system to test Docker images the same way the are deployed • Allows repeatable building of Docker images in ECS within immutable containers
  95. 95. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image creation and usage
  96. 96. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image creation and usage
  97. 97. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image creation and usage
  98. 98. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image creation and usage
  99. 99. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Image creation and usage • No repository code built into images • First step: pull down credentials (locked behind IAM roles) • Second step: clone code or artifacts to actually run onto images • Prevents leakage of information if images are stolen • Even if image stolen, running the image will fail if user does not have the right IAM role to get credentials
  100. 100. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Packages and registries • Images build off of internal base image configured to only pull from internal package repository • Packages in internal repository vetted by security team • Installed packages pinned to specific versions • Amazon ECR • More performant so utilize as storage for security approved images that are used on clusters • Artifactory • Storage for intermediate images and artifacts for longer-term storage
  101. 101. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Key takeaways
  102. 102. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Key takeaways • Protect from unauthorized access • VPC and security groups to limit approved network traffic • IAM roles to limit per task access to creds and services • Minimize surface area for problems • Limit packages installed • Be prepared to fix issues • Constant patching necessitates agile change infrastructure
  103. 103. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×