Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes on AWS @Zalando - Berlin AWS User Group 2017-05-09

16,829 views

Published on

In this talk we share our learnings from running Kubernetes on AWS in production and how we are migrating 200+ engineering teams from AWS/STUPS to Kubernetes.

This talk was given at the Berlin AWS User Group meetup on 2017-05-09 hosted by NewStore (https://www.meetup.com/aws-berlin/events/236795816/).

More information on http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html

Published in: Technology

Kubernetes on AWS @Zalando - Berlin AWS User Group 2017-05-09

  1. 1. Kubernetes on AWS @ZalandoTech AWS BERLIN USER GROUP 2017-05-09 HENNING JACOBS @try_except_
  2. 2. 2 ZALANDO 15 markets 6 fulfillment centers 20 million active customers 3.6 billion € net sales 2016 165 million visits per month 12,000 employees in Europe
  3. 3. 3 ZALANDO TECHNOLOGY HOME-BREWED, CUTTING-EDGE & SCALABLE technology solutions >1,600 employees from tech locations + HQs in Berlin6 77 nations help our brand to WIN ONLINE
  4. 4. 4 KUBERNETES ON AWS: CONTEXT 200 engineering teams 30 prod. clusters AWS Dockerized apps No manual operations Reliability Autoscaling Seamless migration
  5. 5. 5 ARCHITECTURE
  6. 6. 6 ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Product ABC Product XYZ EC2 LBLB
  7. 7. 7 KUBERNETES ON AWS
  8. 8. 8 CLUSTER PROVISIONING
  9. 9. 9 CLUSTER PROVISIONING • Two Cloud Formation stacks • Master & worker ASGs + etcd • Nodes w/ Container Linux • K8s manifests applied separately • kube-system Deployments • DaemonSets
  10. 10. 10 DEPLOYMENT
  11. 11. 11 DEPLOYMENT CONFIGURATION . ├── apply │ ├── credentials.yaml # K8s TPR │ ├── ingress.yaml # K8s Ingress │ ├── redis-deployment.yaml # K8s Deployment │ ├── redis-service.yaml # K8s Service │ └── service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # proprietary config
  12. 12. 12 JENKINS DEPLOY PIPELINE
  13. 13. 13 INGRESS
  14. 14. 14 INGRESS.YAML apiVersion: extensions/v1beta1 kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "{{ application }}" servicePort: 80
  15. 15. 15 INGRESS CONTROLLER https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
  16. 16. 16 AWS INTEGRATION
  17. 17. 17 CLOUD FORMATION VIA CI/CD . ├── apply │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml # K8s Ingress │ ├── kube-secret.yaml # K8s Secret │ └── kube-service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # CI/CD config
  18. 18. 18 ASSIGNING AWS IAM ROLE TO POD kind: Deployment spec: template: metadata: annotations: # annotation for kube2iam iam.amazonaws.com/role: "app-{{ application }}-1" spec: containers: - name: ... ... https://github.com/jtblin/kube2iam ⇒ AWS SDKs just work as expected
  19. 19. 19 CLUSTER AUTOSCALING
  20. 20. 20 CLUSTER AUTOSCALING Control # of worker nodes in ASG: • Satisfy all resource requests • One spare node per AZ • No manual config “tweaking” • Scale down, but not too fast
  21. 21. 21 CURRENT SETUP • https://github.com/hjacobs/kube-aws-autoscaler • Node draining via systemd unit Open topic: node “readiness” during scale out
  22. 22. 23 OAUTH / IAM INTEGRATION
  23. 23. 24 DECLARING NEEDED CREDENTIALS # apply/credentials.yaml apiVersion: "zalando.org/v1" kind: PlatformCredentialsSet metadata: name: "..." spec: tokens: # OAuth service tokens mytok: privileges: - com.zalando::foobar.write clients: # OAuth clients implicit: grant: implicit # grant type according to RFC-6749 realm: users redirectUri: https://myapp.foo.example.org/oauth ⇒ TPRs FTW
  24. 24. 25 MOUNTING THE OAUTH CREDENTIALS kind: Deployment spec: template: spec: containers: - name: ... ... volumeMounts: - name: "..." mountPath: /meta/credentials readOnly: true volumes: - name: "..." secret: secretName: "..."
  25. 25. 26 USING THE OAUTH CREDENTIALS #!/bin/bash type=$(cat /meta/credentials/mytok-token-type) secret=$(cat /meta/credentials/mytok-token-secret) curl -H "Authorization: $type $secret" https://resource-server.example.org/protected
  26. 26. 27 POSTGRESQL
  27. 27. 28 POSTGRESQL ON KUBERNETES • StatefulSet + PV (EBS) • Spilo/Patroni • PostgreSQL Operator • ThirdPartyResources https://github.com/zalando/spilo https://github.com/zalando/patroni
  28. 28. 29 WHY NOT RDS? • Logical Replication • Outbound Streaming Replication • Heterogeneous Replicas • Custom Extensions • Real Superuser • OAuth Login NB: we also use RDS and it works great!
  29. 29. 30 OPERATIONS
  30. 30. 31 CLUSTER UPDATES
  31. 31. 32 LIMIT RANGE kubectl describe limitrange Name: limits Namespace: default Type Resource Min Max Default Req Default Limit Max Limit/Request Ratio ---- -------- --- --- ----------- ------------- ----------------------- Container memory - 64Gi 100Mi 1Gi - Container cpu - 16 100m 3 - http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html#resources ⇒ Mitigate errors on OSI layer 8 ;-)
  32. 32. https://github.com/hjacobs/kube-ops-view
  33. 33. 34 KUBERNETES VS. AWS ECS
  34. 34. 35 AWS API Tasks, Services Static AWS API Blox Operating worker nodes Vendor community/support AWS only KUBERNETES VS. AWS ECS Declarative API (fast & no rate limits) High level abstractions (Ingress, CronJob) Extensible API (e.g. TPR) Batteries included (DaemonSet, StatefulSet) Operating etcd, master & worker nodes Huge community Run anywhere ⟺ ⟺ ⟺ ⟺ ⟺ ⟺ ⟺ disclaimer: incomplete and opinionated ;-)
  35. 35. 36 OPEN SOURCE
  36. 36. 37 LINKS Running Kubernetes in Production on AWS http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html Kube AWS Ingress Controller https://github.com/zalando-incubator/kube-ingress-aws-controller External DNS https://github.com/kubernetes-incubator/external-dns Zalando Cluster Configuration https://github.com/zalando-incubator/kubernetes-on-aws List of Organizations using Kubernetes on AWS https://github.com/hjacobs/kubernetes-on-aws-users
  37. 37. QUESTIONS? HENNING JACOBS TECH INFRASTRUCTURE CLOUD ENGINEER henning@zalando.de @try_except_ Illustrations by @01k

×