Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2017 07-19 automatic infrastructure for kubernetes ingress in aws

726 views

Published on

At Zalando we have 265 AWS accounts and 31 Kubernetes clusters running in AWS. We have to make sure that feature teams do not have to do more work than needed to deploy their application and publish endpoints.

Creating a Kubernetes ingress object is enough to provision an AWS Application Load Balancer (ALB) including automated SSL certificate lookup and TLS termination, HTTP routing based on our ingress implementation and a public resolvable DNS entry. This talk will explain the stack and deep dive into how this works for us.

Published in: Technology
  • Be the first to comment

2017 07-19 automatic infrastructure for kubernetes ingress in aws

  1. 1. DOCKER MEETUP BERLIN 2017-07-19 SANDOR SZÜCS @sszuecs Automatic infrastructure for Kubernetes ingress in AWS
  2. 2. 2 ZALANDO 15 markets 6 fulfillment centers 20 million active customers 3.6 billion € net sales 2016 165 million visits per month 12,000 employees in Europe
  3. 3. 3 ZALANDO TECHNOLOGY HOME-BREWED, CUTTING-EDGE & SCALABLE technology solutions >1,700 employees from tech locations + HQs in Berlin6 77 nations help our brand to WIN ONLINE
  4. 4. 4 ZALANDO TECH’S INFRASTRUCTURE
  5. 5. 5 FOUR ERAS AT ZALANDO TECH ZOMCATPHP STUPS KUBERNETES 2010 2015 2016 Data center WAR LXC AWS Docker Cloud Formation Low level (AWS API) AWS Docker Cloud Formation Kubernetes manifest Higher abstraction level Data center PHP files
  6. 6. 6 LARGE SCALE?
  7. 7. 9 KUBERNETES: ARCHITECTURE
  8. 8. 10 ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Product ABC Product XYZ EC2 LBLB
  9. 9. 11 KUBERNETES ON AWS
  10. 10. 12 DEPLOYMENT
  11. 11. 13 DEPLOYMENT CONFIGURATION . ├── apply │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml # K8s Ingress │ ├── kube-secret.yaml # K8s Secret │ └── kube-service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # CI/CD config
  12. 12. 14 INGRESS.YAML apiVersion: extensions/v1beta1 kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
  13. 13. 15 JENKINS DEPLOY PIPELINE
  14. 14. 16 CHALLENGES
  15. 15. 17 1. Getting Started 2. Ingress CHALLENGES
  16. 16. 18 CHALLENGE 1: GETTING STARTED
  17. 17. 19 GETTING STARTED https://github.com/hjacobs/kubernetes-on-aws-users
  18. 18. 20 GETTING STARTED https://github.com/hjacobs/kubernetes-on-aws-users
  19. 19. 21 CLUSTER PROVISIONING • Two Cloud Formation stacks • Master & worker ASGs + etcd • Nodes w/ Container Linux • K8s manifests applied separately • kube-system Deployments • DaemonSets
  20. 20. 22 CLUSTER PROVISIONING
  21. 21. 23 GETTING STARTED Other questions we asked ourselves.. • Single AZ vs. Multi AZ? ⇒ Multi AZ • Federation? ⇒ No, not ready yet • Overlay network? ⇒ Flannel, “rock solid” • Authnz? ⇒ OAuth, webhook
  22. 22. 24 CHALLENGE 2: Ingress
  23. 23. 25 Ingress • System view • Developer point of view • Enhancing Kubernetes with weighted traffic switching
  24. 24. 26 System View Goal: use Kubernetes API as primary interface • External DNS → Route53 • Kubernetes Ingress Controller for AWS → ALB+TLS • Skipper - http router → your dockerized app ⇒ we wrote new components to achieve our goal
  25. 25. 27 System View - traffic flow https://github.com/zalando/skipper ALB Node Skipper Node Skipper MyApp MyApp MyApp Service Service K8s network EC2 network TLS HTTP
  26. 26. 28 Kubernetes - ingress - configuration https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
  27. 27. 29 Kubernetes - service - configuration https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns Select PODs by label Target of Ingress definition Port of the my-app
  28. 28. 30 Kubernetes - POD - configuration https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns Target port Selected by service
  29. 29. 31 Skipper https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns • Skipper used as main shop router • Skipper can be used as ingress implementation
  30. 30. 32 Skipper https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
  31. 31. 33 System View - skipper - configuration https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns Skipper reads
  32. 32. 34 kube-ingress-aws-controller https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
  33. 33. 35 System View - controller - configuration https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns kube-ingress-aws-controller Optional read write
  34. 34. 36 System View - external DNS https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns
  35. 35. 37 System View - external-dns - configuration https://github.com/zalando/skipper https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns external-dns reads
  36. 36. 38 System View - summary ● DNS Name pointing to ALB ● ALB + TLS ● Skipper routes
  37. 37. 39 Developer point of view ● Define templates ● Execute pipelines
  38. 38. 40 Developer point of view - defines template Pipeline expands some variables from pipeline.yaml
  39. 39. 41 Developer point of view - executes pipeline
  40. 40. 42 WEIGHTED TRAFFIC SWITCHING • STUPS - former AWS platform toolkit • Kubernetes • supports rolling updates • enhancement to ingress required
  41. 41. 43 WEIGHTED TRAFFIC SWITCHING - STUPS • STUPS uses weighted Route53 DNS records • Allows canary, blue/green, slow ramp up • Problem: DNS caching
  42. 42. 44 WEIGHTED TRAFFIC SWITCHING - Kubernetes • Approach: add weights to Ingress backends https://github.com/zalando/skipper
  43. 43. 45 Traffic Switching - Ingress configuration https://github.com/zalando/skipper Skipper reads
  44. 44. 46 LINKS Running Kubernetes in Production on AWS http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html Kube AWS Ingress Controller https://github.com/zalando-incubator/kube-ingress-aws-controller Skipper HTTP Ingress Router https://github.com/zalando/skipper/ External DNS https://github.com/kubernetes-incubator/external-dns Zalando Cluster Configuration https://github.com/zalando-incubator/kubernetes-on-aws PostgreSQL Operator https://github.com/zalando-incubator/postgres-operator
  45. 45. QUESTIONS? SANDOR SZÜCS TECH INFRASTRUCTURE CLOUD ENGINEER sandor.szuecs@zalando.de @sszuecs Slides based on @try_except_ Illustrations by @01k

×