Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트

1,365 views

Published on

K8s, Amazon EKS

Published in: Technology
  • Be the first to comment

K8s, Amazon EKS - 유재석, AWS 솔루션즈 아키텍트

  1. 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Jaeseok Yoo K8s, Amazon EKS
  2. 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Time 9:30 - 10:30 Docker & Container Orchestration, k8s 10:30 – 10:45 Beak 10:45 - 12:00 K8s, Amazon EKS HoL: Launch EKS Cluster 12:00 – 13:00 Launch 13:00 – 13:40 HoL: Launch microservices 13:40 – 14:20 HoL: Helm 15:15 – 16:00 HoL: Monitoring with Prometheus and Grafana
  3. 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker
  4. 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 애플리케이션의 구성 런 타임 엔진 코드 디펜던시 구성
  5. 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • 다른 애플리케이션 스택 • 다른 하드웨어 배포 환경 • 다른 환경에서 애플리케이션을 실행하는 효율적인 방법은? • 다른 환경으로 쉽게 마이그레이션하는 방법은? 문제점
  6. 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 솔루션 - 도커 이식성 : 이미지 기반 배포 유연성 : 마이크로 서비스 모듈화 신속성 : 가벼운 도커 이미지 효율성 : OS kernel 공유
  7. 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VM과 컨테이너 비교 Server (Host) Host OS Hypervisor App 2 Guest OS Guest OS Guest OS Bins/Libs Bins/Libs Bins/Libs App 1 App 3 VM Server (Host) Host OS Docker Bins/Libs Bins/Libs Bins/Libs App 1 App 2 App 3 Container Hypervisor Guest OS
  8. 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker 이미지 구성 bootfs kernel Base image Image Image W ritable Container add nginx add nodejs U buntu References parent image Base Image : 템플릿으로 사용되는 읽기 전용 이미지 Base Image에서 시작해서 커스텀 Image 추가하는 방식 Dockerfile 활용하여 손쉽게 배포 관련 구성 설정 및 재배포에 용이함
  9. 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker 엔진 구조 및 Docker CLI 예 • docker build # Build an image from a Dockerfile • docker info # Display system-wide information • docker images # List all images on a Docker host • docker run # Run an image • docker ps # List all running and stopped instances • docker stop # Stop a running instances • docker rm # Remove an instance • docker rmi # Remove an image • docker pull # Download an image from registry • docker push # Upload an image to the registry
  10. 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dockerfile은 이미지를 빌드하기 위한 일련의 명령어 모음
  11. 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dockerfile # our base image FROM alpine:3.5 # Install python and pip RUN apk add --update py2-pip # install Python modules needed by the Python app COPY requirements.txt /usr/src/app/ RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt # copy files required for the app to run COPY app.py /usr/src/app/ COPY templates/index.html /usr/src/app/templates/ # tell the port number the container should expose EXPOSE 5000 # run the application CMD ["python", "/usr/src/app/app.py"] $ docker build -t <YOUR_USERNAME>/myfirstapp . Sending build context to Docker daemon 9.728 kB Step 1 : FROM alpine:latest ---> 0d81fc72e790 Step 2 : RUN apk add --update py-pip ---> 976a232ac4ad Removing intermediate container 8abd4091b5f5 Step 3 : COPY requirements.txt /usr/src/app/ ---> 65b4be05340c Step 4 : RUN pip install --no-cache-dir -r /usr/src/app/requirements.txt ---> 8de73b0730c2 Step 5 : COPY app.py /usr/src/app/ … Dockerfile은 컨테이너 내부 이미지 환경 및 구성 정의
  12. 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dockerfile best practice - 딱 필요한 Base 파일 선택 From the stock ubuntu image: ubuntu latest 2b1dc137b502 52 seconds ago 458 MB From python:2.7-alpine: alpine latest d3145c9ba1fa 2 minutes ago 86.8 MB
  13. 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FROM ubuntu:latest RUN apt-get update -y && apt-get install -y python-pip python-dev build-essential LABEL maintainer changsul@amazon.com COPY . /app WORKDIR /app RUN pip install ­r requirements.txt EXPOSE 5000 ENTRYPOINT ["python"] CMD ["application.py"] Dockerfile best practice - 딱 필요한 Base 파일 선택
  14. 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FROM python:2.7-alpine LABEL maintainer changsul@amazon.com COPY . /app WORKDIR /app RUN pip install ­r requirements.txt EXPOSE 5000 ENTRYPOINT ["python"] CMD ["application.py"] Dockerfile best practice - 딱 필요한 Base 파일 선택
  15. 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FROM python:2.7-alpine LABEL maintainer abbyfull@amazon.com COPY requirements.txt /app RUN pip install ­r /app/requirements.txt COPY . /app WORKDIR /app EXPOSE 5000 ENTRYPOINT ["python"] CMD ["application.py"] Dockerfile best practice - 캐쉬 무효화 최소화
  16. 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dockerfile best practice 빌드 이미지 크기 및 레이어 수 최소화 런타임시 필요한 것만 선택 각 빌드별 태깅 Semantic version (i.e. “1.3.2-9”) Build Number (i.e., “127”) Build Id (i.e. “511d5e51-b415-4cb2-b229-b3c8a46b7a2f”) 템프 파일 제거 RUN apt-get update && apt-get install -y bzr cvs git mercurial subversion && rm ­rf /var/lib/apt/lists/*
  17. 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 고객사례 - Nextdoor Base OS version Apt packages: OpenSSL libpq syslog-ng Datadog Python runtime PyPI packages: Boto Django Mapnik SendGrid Source code Static assets Images JS CSS
  18. 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Layer 별 각기 다른 업데이트 주기 Quarterly Weekly/ monthly Continuous
  19. 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AMI에서 Docker Container로 변경 Base OS layer System packages Python packages Nextdoor source
  20. 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Docker 이전에는 빌드 20분 소요 chroot sudo apt-get install sudo pip install git clone make install dpkg create
  21. 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Base image , system deps 추가 FROM hub.corp.nextdoor.com/nextdoor/nd_base:precise ADD app/docker/scripts/apt-fast app/docker/scripts/system-deps.sh /deps/ RUN /deps/system-deps.sh
  22. 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Python virtualenv 설정 업데이트 ADD app/docker/scripts/venv-deps.sh app/apps/nextdoor/etc/requirements*.txt app/apps/nextdoor/etc/nextdoor.yml app/services/scheduler/etc/scheduler.yml app/services/supervisor/etc/supervisor.yml /deps/ RUN /deps/venv-deps.sh
  23. 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. App 소스 업데이트 ADD app/static/nextdoorv2/images /app/static/nextdoorv2/images ADD app/thrift /deps/thrift ADD app/nd /deps/nd ADD app /app
  24. 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 빌드 시간 20분 -> 평균 2분 ECS에 최종 배포까지 평균 5분
  25. 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://docs.docker.com/ https://en.wikipedia.org/wiki/Docker_(software) https://en.wikipedia.org/wiki/LXC https://en.wikipedia.org/wiki/Linux_namespaces https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Chroot https://www.slideshare.net/Docker/creating-effective-images-abby-fuller-aws https://docs.docker.com/develop/develop-images/dockerfile_best-practices/ https://github.com/docker/labs/blob/master/beginner/chapters/webapps.md http://crosbymichael.com/dockerfile-best-practices.html References
  26. 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Common Questions • How do I deploy my containers to hosts? • How do I do zero downtime or blue green deployments? • How do I keep my containers alive? • How can my containers talk to each other? • Linking? Service Discovery? • How can I configure my containers at runtime? • What about secrets? • How do I best optimize my "pool of compute”?
  27. 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How do we make this work at scale?
  28. 28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. We need to • start, stop, and monitor lots of containers running on lots of hosts • decide when and where to start or stop containers • control our hosts and monitor their status • manage rollouts of new code (containers) to our hosts • manage how traffic flows to containers and how requests are routed
  29. 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  30. 30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance Instance Instance OS OS OS Container Runtime Container Runtime Container Runtime App Service App App Service Service Container Orchestration
  31. 31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration myJob: { Cpu: 10 Mem: 256 } Orchestrator Schedule Run “myJob”
  32. 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance/OS Instance/OS Instance/OS App Service App App Service Service Service Management Scheduling Resource Management OrchestrationService Management §Availability §Lifecycle §Discovery
  33. 33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance/OS Instance/OS Instance/OS App Service App App Service Service Service Management Scheduling Resource Management Orchestration Scheduling §Placement §Scaling §Upgrades §Rollbacks
  34. 34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Orchestration Instance/OS Instance/OS Instance/OS App Service App App Service Service Service Management Scheduling Resource Management Orchestration Resource Management § Memory § CPU § Ports
  35. 35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are container orchestration tools?
  36. 36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Container Services Landscape MANAGEMENT Deployment, Scheduling, Scaling & Management of containerized applications HOSTING Where the containers run Amazon Elastic Container Service Amazon Elastic Container Service for Kubernetes Amazon EC2 AWS Fargate IMAGE REGISTRY Container Image Repository Amazon Elastic Container Registry
  37. 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Run a (managed) container on AWS AMAZON CONTAINER SERVICES Choose your orchestration tool1 Choose your launch type2 ECS EKS EC2 Fargate EC2 Fargate
  38. 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  39. 39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is Kubernetes? Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications
  40. 40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes(K8s) Components Control Plane (Controller) Etcd Lightweight, open source Key-Value store containing the cluster API Server Serves the APIs required to manage the cluster Scheduler Determines where (on which nodes) pods will run in the cluster Controller Manager The “worker on the controller” that actually manages the cluster (e.g. replication) Kubernetes Node kubelet Runs the node, starts and stops containers kube-proxy Acts as a network proxy – routes traffic based upon IP and Port. Each service is assigned a unique port on the nodes it runs across, kube-proxy allows that port to be mapped to whatever the service expects. cAdvisor Agent that monitors node health and statistics
  41. 41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes(K8s) Architecture
  42. 42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes(K8s) Objects • kubectl • Pods • Labels • Deployments • Replication Controllers • Services
  43. 43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. kubectl • Command line interface for running commands against the k8s API • Intuitive familiar commands (get, create, describe, delete, etc.) that are simple to learn and easy to use ~/.kube/config k8s master kube-api scheduler
  44. 44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pods • A group of one or more containers • Shared: • Data volumes • cgroup • Namespace – network, IPC, etc. node pod1 pod2
  45. 45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Labels • Key/Value Pairs • Used to query specific resources within your cluster pod1 pod2 dev prod app001 app001
  46. 46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ReplicaSets • Ensure that a specified number of pod “replicas” exist in the cluster 23
  47. 47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployments • Declarative updates for Pods and ReplicaSets 23
  48. 48. Containers on Hosts Host 1 Host 2 Host 3 A host is a server – e.g. EC2 virtual machine. We run these hosts together as a cluster. Web App To start let’s run a 3 copies of our web app across our cluster of EC2 hosts. 3x Our simple example web application is already containerized. Cluster
  49. 49. Run n containers Host 1 Host 2 Host 3 We define a deployment and set the replicas to 3 for our container. deploymentkubectl rep = 3
  50. 50. Scale up! Host 1 Host 2 Host 3 Need more containers? Update the replication set! deploymentkubectl rep = 5 The new containers are started on the cluster.
  51. 51. Untimely termination Host 1 Host 2 Host 3 Oh no! Our host has died! Replication set rep = 5 Kubernetes notices only 3 of the 5 containers are running and starts 2 additional containers on the remaining hosts.
  52. 52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services • A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is (usually) determined by a Label Selector. • Let’s talk about what are the differences between LoadBalancer, NodePort and Ingress
  53. 53. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : ClusterIP • Exposes the service on a cluster- internal IP • Only reachable from within the cluster • Access possible via kube-proxy • Useful for debugging services, connecting from your laptop or displaying internal dashboards
  54. 54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : NodePort • Exposes the service on each Node’s IP at a static port. • Routes to a ClusterIP service, which is automatically created. • from outside the cluster: <NodeIP>:<NodePort> • 1 service per port • Uses ports 30000-32767
  55. 55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : LoadBalancer • Exposes the service externally using a cloud provider’s load balancer. • NodePort and ClusterIP services (to which LB will route) automatically created. • Each service exposed with a LoadBalancer (ELB or NLB) will get its own IP address • Exposes L4 (TCP) or L7 (HTTP) services
  56. 56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Service : LoadBalancer - Sample apiVersion: v1 kind: Service metadata: name: my-nginx-lb labels: app: nginx-lb spec: type: LoadBalancer ports: - port: 80 selector: app: nginx-lb apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-nginx-lb spec: replicas: 3 template: metadata: labels: app: nginx-lb spec: containers: - name: nginx-lb image: nginx:1.7.9 ports: - containerPort: 80
  57. 57. Services One of the ways traffic gets to your containers. • Internal IP addresses are assigned to each container • Services are connected to containers and use labels to reference which containers to route requests to IP IP IP Service IP
  58. 58. Deployments IP IP IP Service IPReplication set version = 1 count = 3 Deployment Services work with deployments to manage updating or adding new pods. Let’s say we want to deploy a new version of our web app as a ‘canary’ and see how it handles traffic.
  59. 59. Deployments IP IP IP Service IPReplication set version = 1 count = 3 The deployment creates a new replication set for our new pod version. Replication set version = 2 count = 1 IP Deployment
  60. 60. Deployments – Rolling Update IP IP IP Service IPReplication set version = 1 count = 3 Only after the new pod returns a healthy status to the service do we add more new pods and scale down the old. Replication set version = 2 count = 1 IP Deployment Replication set version = 1 count = 0 Replication set version = 2 count = 3
  61. 61. Deployments - Blue/Green Service app=nginx Version=1 IP Replication set app=nginx version=1 count=3 Deployment Replication set app=nginx version=2 count=3 Deployment Service app=nginx version=2
  62. 62. Deployments – Canary Service app=nginx Version=1 IP Replication set app=nginx version=1 count=3 Deployment Replication set app=nginx version=2 count=1 Deployment Service app=nginx Replication set app=nginx version=2 count=2 Replication set app=nginx version=1 count=2 Replication set app=nginx version=1 count=1 Replication set app=nginx version=2 count=3
  63. 63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : Ingress
  64. 64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services : Ingress • exposes HTTP/HTTPS routes to services within the cluster • Many implementations: ALB, Nginx, F5, HAProxy etc • Default Service Type: ClusterIP
  65. 65. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ALB Ingress Controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTP ListenerHTTPS Listener Rule: /cheesesRule: /charcuterie TargetGroup: Green (IP Mode) TargetGroup: Blue (Instance Mode) NodePort NodePort
  66. 66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ConfigMap and Secret
  67. 67. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ConfigMap & Secret • ConfigMap and Secret allow you to decouple configuration artifacts from image content to keep containerized applications portable. • You can pass the ConfigMap or Secret to the pod by environment variable or volume mount. • Secret uses Base64 encoding.
  68. 68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet
  69. 69. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Statefulset Properties • Network identifiers • Persistent Storage • Ordered graceful deployment and scaling • Ordered graceful termination • Ordered rolling updates • If none of these fit your portfolio, use Deployment or Replicaset
  70. 70. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet 1) Define headless service, statefulset and PVC 2) Control loop allocates PV based on PVC request StorageClass gp2 io1 sc1 encrypted io1 st1 3) Kubernetes creates statefulset MySQL Pods mysql-0 mysql-1 mysql-2 mysql-3 Network Identifiers Ordered Deployment
  71. 71. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StatefulSet 1) Define headless service, statefulset and PVC 2) Control loop allocates PV based on PVC request 3) Kubernetes creates statefulset MySQL Pods mysql-0 mysql-1 mysql-2 mysql-3 Ordered Scaling mysql-4
  72. 72. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage
  73. 73. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Lifecycle of a storage volume Provisioning Binding Using Reclaiming • Static • Dynamic* • Control loop watches for PVC requests and satisfies if PV is available. • For Dynamic, PVC will provision PV • PVC to PV binding is one-to-one mapping • Cluster mounts volume based on PVC • Retain (default) • Recycle • Delete
  74. 74. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage Class Persistent Volume Persistent Volume Claim Pod
  75. 75. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What if I need specific volume type? StorageClass gp2 io1 sc1 encrypted io1 st1 1) Admin pre-provisions StorageClass based on workload needs 2) End user requests for specific volume types (For ex, encrypted io1 volume) 3) Control loop watches PVC request and allocates volume if PV exists MySQL Pods 4) End user creates stateful workload
  76. 76. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  77. 77. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 57%of Kubernetes workloads run on AWS today — Cloud Native Computing Foundation
  78. 78. Containers options on AWS – over time Docker Host AWS Cloud AWSmanagedCustomermanaged
  79. 79. Containers options on AWS – over time Amazon ECS EC2 Container Instances Auto Scaling group 2015 ECS API Docker Host AWS Cloud AWSmanagedCustomermanaged
  80. 80. Containers options on AWS – over time AWS Fargate Amazon ECS EC2 Container Instances Auto Scaling group 2017 ECS API Docker Host AWS Cloud AWSmanagedCustomermanaged
  81. 81. Containers options on AWS – over time AWS Fargate Amazon ECS EC2 Container Instances Auto Scaling group Worker nodes Auto Scaling groupDIY K8S ECS API K8s API Docker Host AWS Cloud AWSmanagedCustomermanaged
  82. 82. Containers options on AWS – over time AWS Fargate Amazon ECSAmazon EKS EC2 Container Instances Auto Scaling group Worker nodes Auto Scaling groupDIY K8S 2018 K8s API ECS API K8s API Docker Host AWS Cloud AWSmanagedCustomermanaged
  83. 83. Management of the Kubernetes control plane Phase 1
  84. 84. Management of the Kubernetes control plane Phase 1 Phase 2 Management of the Kubernetes data plane
  85. 85. Containers options on AWS – over time AWS Fargate Amazon ECSAmazon EKS EC2 Container Instances Auto Scaling group Managed Node Groups Auto Scaling group Worker nodes Auto Scaling groupDIY K8S 2019 K8s API ECS API K8s API Docker Host AWS Cloud AWSmanagedCustomermanaged
  86. 86. Containers options on AWS – over time AWS Fargate Amazon ECSAmazon EKS EC2 Container Instances K8s API ECS API AWS Cloud Auto Scaling group Managed Node Groups Auto Scaling group Worker nodes Auto Scaling groupDIY K8S NEW Docker Host K8s API AWSmanagedCustomermanaged
  87. 87. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes on AWS Managed Kubernetes on AWS Highly available Automated version upgrades Integration with other AWS services Etcd Master Managed Kubernetes control plane CloudTrail, CloudWatch, ELB, IAM, VPC, PrivateLink
  88. 88. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes on AWS 3x Kubernetes masters for HA
  89. 89. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Availability Zone 1 Master Master Availability Zone 2 Availability Zone 3 Master Workers Workers Workers Customer Account AWS Managed
  90. 90. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Control Plane
  91. 91. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
  92. 92. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
  93. 93. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What happens when I run ‘kubectl create –f pods.yaml’?
  94. 94. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM Authentication Kubectl 3) Authorizes AWS Identity with RBAC K8s API 1) Passes AWS Identity 2) Verifies AWS Identity 4) K8s action allowed/denied AWS Auth
  95. 95. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Control Plane Master Node Scheduler Controller Manager Cloud Controller Manager API Server etcd Kubectl
  96. 96. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cluster Authentication and Authorization • User or IAM role who creates EKS cluster gains Admin privileges • This {“super”} user/role can then add additional users or IAM roles and configure RBAC permissions • To add, configure aws-auth Configmap kubectl edit -n kube-system configmap/aws-auth
  97. 97. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. aws-auth configuration apiVersion: v1 data: mapRoles: | - rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6 username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam::555555555555:user/admin username: admin groups: - system:masters - userarn: arn:aws:iam::555555555555:user/john username: john groups: - pod-admin # k8s RBAC group
  98. 98. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Data Plane
  99. 99. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Architecture EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
  100. 100. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Data Plane Worker Node kube-dnsKubelet aws- node Container runtime Control Plane API kube- proxy
  101. 101. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] ExecStartPre=/sbin/iptables -P FORWARD ACCEPT ExecStart=/usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --allow-privileged=true --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS Restart=on-failure RestartForceExitStatus=SIGPIPE RestartSec=5 KillMode=process [Install] WantedBy=multi-user.target
  102. 102. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS AMI Build Scripts https://github.com/awslabs/amazon-eks-ami Source of truth for EKS Optimized AMI Easily build your own EKS AMI Build assets for EKS AMI for each supported Kubernetes version
  103. 103. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Optimized AMI with GPU Support Easily run Tensorflow/Kubeflow on Amazon EKS Includes NVIDIA packages to support Amazon P2 and P3 instances Available on AWS Marketplace
  104. 104. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Managed Node Group • You can create, update, or terminate nodes for your cluster with a single operation. • Nodes run using the latest Amazon EKS-optimized AMIs in your AWS account while node updates and terminations gracefully drain nodes to ensure that your applications stay available. • All managed nodes are provisioned as part of an Amazon EC2 Auto Scaling group that is managed for you by Amazon EKS.
  105. 105. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Worker Node Setup – Bootstrapping /etc/eks/bootstrap.sh <cluster-name> [options] Uses UserData for configuring System resources and extra Kubelet config Reserve compute resources for System Daemons (Kubelet, Container runtime) and Pod eviction thresholds
  106. 106. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Upgrades
  107. 107. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes Version Versions supported: 1.12.10, 1.13.12, 1.14.9 EKS will support up to 3 versions of Kubernetes at once ”Deprecation” will prevent new cluster creation on old version
  108. 108. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Platform Version Platform Version revisions represent API server configuration changes or Kubernetes patches Platform Versions increment within a Kubernetes version only
  109. 109. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. EKS Kubernetes Version Updates New UpdateClusterVersion API – supports in place updates of Kubernetes version Introduces an ”update” EKS API object ListUpdates and DescribeUpdate APIs to provide visibility into the status of a given update
  110. 110. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Updating Worker Nodes Two options: 1) Create new node group with latest EKS AMI >> taint old nodes >> drain old nodes >> terminate old CFN template 2) Simply update AMI in CFN template; “rolling” replacement policy terminates nodes (Downsides: un-graceful termination of applications)
  111. 111. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Container Services Roadmap https://github.com/aws/containers-roadmap
  112. 112. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Get Started https://eksworkshop.com Modules: • Health Checks • Logging with Elasticsearch, Fluentd, and Kibana (EFK) • Monitoring using Prometheus and Grafana • Servicemesh with Istio • Stateful Containers using StatefulSets
  113. 113. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×