Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Running Kubernetes with Amazon EKS - AWS Online Tech Talks

3,888 views

Published on

Learning Objectives:
- What is Amazon EKS
- How Amazon EKS helps you run Kubernetes on AWS
- Timelines for availability and next steps

Running Kubernetes with Amazon EKS - AWS Online Tech Talks

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Nishi Davidson, Product Management, Amazon EKS Dan Wilson, Principal Development Architect, SAP Concur March 2018 Running Kubernetes on AWS Managing workloads on-prem vs Amazon EKS
  2. 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 63% of Kubernetes workloads run on AWS today — CNCF survey
  3. 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Cloud agnostic design: Common design in AWS and on-prem in your data center. • Self-healing hence Stable: Applications once created, are kept alive even if they failure. • Standardized hence Consistent: Inconsistencies isolated in the container while rest of the infrastructure tools are standardized. • Flexible hence Fast: Containerized applications can be deployed in seconds on any hardware. • Integrated with monitoring/logging/alerting: Frees engineering to innovate on application logic. • Scalable: Design scales easily to 10s of 1000s of machines. Why k8s helps your environment?
  4. 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Kubernetes @ SAP Concur So how have some AWS customers adopted Kubernetes in production?
  5. 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Polyglot – golang, javascript, java, .net, etc. • Mostly private data centers, some AWS • Global – US, Europe, Japan, China, etc. • Public Sector, Enterprise & Consumer Apps • Started w/ production clusters on k8s v1.1 • Need k8s multi-cluster day 1 for HA & blue/green cluster upgrades k8s @ SAP Concur – on-prem and in AWS Cloud
  6. 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. K8s Cluster Master Mastermaster CoreOS Master Masteretcd CoreOS worker1 CoreOS worker2 CoreOS workerN CoreOS flannel newRelic & prometheus Logstash forwarder svc ELK Monitoring Load Balancers API svc svc svc k8s @ SAP Concur – v1 multi-cluster
  7. 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur – v1 multi-cluster K8s 1P LBaaS API Load Balancers K8s 2 K8s 3 pool VIP P P deployment service deployment service deployment service
  8. 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. N clusters CI k8s @ SAP Concur
  9. 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Skipper Kubernetes Deployment API Curlable API to deploy microservice apps to multiple k8s clusters Open Source - Apache 2.0 https://github.com/concur/skipper
  10. 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. deployment API CI Deploy petshop:v1 to prod US14a, b & c EMEA14a, b & c APAC14a, b & c K8s API calls for service, deployment, etc.
  11. 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. K8s 1 Ingress Controller DNS replicas sets & pods Service Ingress internal k8s @ SAP Concur – v2 clusters
  12. 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur - auth K8s 1 Clients/Skipper Token per namespace and jurisdiction K8s 2 K8s 3
  13. 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur - oauth2 w/ dex K8s 1 Clients/Skipper Token per namespace and cluster token 1 token 2 token 3 K8s 2 K8s 3
  14. 14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur – dns & routing K8s 1Ingress Controller DNS replicas sets & pods Service 1 & 2 Ingress K8s 1 Ingress Controller replicas sets & pods Service 2 only Ingress ? Cluster 1 Cluster 2
  15. 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur – dns & routing K8s 1 Ingress Controller DNS replicas sets & pods Service 1 & 2 Ingress K8s 1 Ingress Controller replicas sets & pods Service 2 only Ingress CNAME service1.domain.com Cluster1.domain.com CNAME service2.domain.com Cluster1-2.domain.com Cluster 1 Cluster 2
  16. 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur – dns & routing - redis K8s 1 Ingress Controller DNS replicas sets & pods redis1 Ingress CNAME redis1.domain.com -> Cluster1.domain.com CNAME redis2.domain.com -> Cluster2.domain.com CNAME redis3.domain.com -> Cluster3.domain.com Cluster 1 K8s 1 Ingress Controller replicas sets & pods redis2 Ingress Cluster 2 K8s 1 Ingress Controller replicas sets & pods redis3 Ingress Cluster 3
  17. 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur – upgrades Location US EUROPE Zone A B C A B C Cluster v.1.7 us17a us17b us17c eur17a eur17b eur17c
  18. 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8s @ SAP Concur – upgrades Location US EUROPE Zone A B C A B C Cluster v.1.7 us17a us17b us17c eur17a eur17b eur17c Cluster v.1.8 us18a us18b us18c eur18a eur18b eur18c
  19. 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. In place upgrades
  20. 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. k8S @ SAP Concur – monitoring and logging • Add monitors for etcd and ingress • Tune Prometheus • monitor cluster capacity from the perspective of allocated resources instead of actual resource usage • configure alertmanager to route alerts appropriately • tuning of some alert thresholds to get rid of the noise • just added a monitor for error rate of kube2cnqrf5 • Many custom overrides on top of the 3rd party Prometheus operator
  21. 21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How will AWS make k8s adoption easier? Intro to Amazon EKS
  22. 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo – basic flow Subnet EKS Master EKS Node Group 2 EKS Node Group 1 aws clikubectl Pre-requisites: • Configure VPC and EKSServiceRole • Configure kubectl and awscli
  23. 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM ROLE User X IAM ROLE Service Account Y kubectl → K8s APIs → CRUD Operations on K8s aws-cli → EKS Service APIs → CRUD Operations on Infra K8s Master Nodes K8s Master Nodes K8s Master Nodes API Server Controller Mgr kubelet etcd Cloud Controller Mgr. Scheduler Authentication Webhook Tokens Authorization RBAC Mode Admission Control NamespaceLifecyle,LimitRanger ServiceAccount,DefaultStorageClass,Reso urceQuota Amazon EKS – access and auth AWS STS client side Heptio-aws-authenticator server side
  24. 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 172.16.0.0/16 User X Service Account Y Kubectl K8s Node 2K8s Node 1 kubelet kube-proxy kubelet kube-proxy VPC Subnet per AZ 172.16.0.1/24 ENI ENIPrimary Private IP: 172.16.1.118 Secondary IPs: 172.16.1.147, 172.16.1.224…. Service: Front end POD 2 POD 3 eth0 Service: Back end POD 1 POD 4 eth0 ec2.associateaddress() L3 RouteTable veth0 Bveth0 A eth0 172.16.1.147/32 eth0 172.16.1.224/32 CNI K8s Master NodesK8s Master NodesK8s Master Nodes API Server Controller Manager kubelet etcd Scheduler kube-proxy Cloud Controller Mgr. Amazon EKS – networking with CNI plugin
  25. 25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EKS – networking with CNI plugin 1. Simplify networking options for customers 2. Support high throughput, high availability, low latency and minimal jitter. 3. Allow customers to reuse AWS VPC networking and security best practices such as use of: • VPC flow logs for troubleshooting and compliance auditing, • VPC routing polices for traffic engineering, • Security groups for isolation and regulatory requirements 4. Setup Pod networking within seconds. 5. Support cluster scale to a min. of 5000+ (K8s scale).
  26. 26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S T A G E S E P A R A T I O N “ T E N A N T ” S E P A R A T I O N F I N E - G R A I N E D F I R E W A L L S C O M P L I A N C E E.g., typically use namespaces for different teams within a company—but without network policy, they are not network isolated Reduce attack surface within microservice-based applications Isolate dev, test, and prod E.g., PCI, HIPAA
  27. 27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 172.16.0.0/16 User X Service Account Y K8s Node 2K8s Node 1 kubelet kube-proxy kubelet kube-proxy VPC Subnet per AZ - 172.16.0.1/24 ENI ENI Service: Front end POD 2 POD 3 Service: Back end POD 1 POD 4 CNI K8s Master NodesK8s Master NodesK8s Master Nodes API Server Controller Manager kubelet etcd Scheduler kube-proxy Cloud Controller Mgr. Amazon EKS – dns, services and elb DNS kubedns dnsmasq healthz DNS Service – Static IP POD 2 POD 2 kind: Service type: LoadBalancer
  28. 28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EKS – monitoring and logging • Cluster-wide Cloudwatch metrics and Cloudtrail logs for Control plane – EKS GA • Cloudwatch metrics and Cloudtrail logs for Node groups – EKS Roadmap Infra. metrics agents.– collectd, cloudwatch Logs Sink – ElasticSearch, Cloudwatch, Cloudtrail Log Text / Graphs – Cloudwatch, Kibana Metrics Sink – Prometheus, InfluxDB, Graphite, Cloudwatch Infra. log agents or exporters – cloudwatch, fluentd, logstash Metric Graphs – Grafana, Cloudwatch • Application container metrics / logs – Customer configured K8sOptions
  29. 29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EKS – cluster management • Control Plane high-availability and scale • Control Plane auto-upgrades with version testing • In-built Cluster DNS and Service Discovery • Cluster-wide metrics and logs for control plane • AWS AMIs bundled with necessary tools • Packer scripts to run K8s with distros of your choice on AWS
  30. 30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Node Group management Master Auto-scale Fargate for EKS Add-Ons: UI, Helm, Plugins, Brokers…. Amazon EKS – key roadmap and open source elements Roadmap Items Open Source @ AWS • Heptio aws-authenticator • CNI Plugin https://github.com/heptiolabs/k ubernetes-aws-authenticator https://github.com/aws/amazon-vpc-cni- k8s/blob/master/proposals/cni- proposal.md Privatelink support
  31. 31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you d-nishi on github, dnishi@amazon.com k8s slack & gmail danwilson on github, k8s slack & gmail, https://github.com/concur/skipper

×