Download as PDF, PPTX
Uploaded byZalando adtech lab
12.07.2017 Docker Meetup - KUBERNETES ON AWS @ ZALANDO TECH
The document outlines Zalando's implementation of Kubernetes on AWS, detailing their technology infrastructure and deployment strategies across multiple engineering teams. It discusses key challenges faced, such as stability, onboarding, and user experience, while also presenting solutions and tools utilized for seamless operation. Additionally, it highlights various configuration files and resources for enhancing Kubernetes deployments and integration with AWS services.
More Related Content
Similar to 12.07.2017 Docker Meetup - KUBERNETES ON AWS @ ZALANDO TECH
12.07.2017 Docker Meetup - KUBERNETES ON AWS @ ZALANDO TECH
- 1. CONTAINER DAYS HAMBURG 2017-06-20 HENNINGJACOBS @try_except_ Kubernetes on AWS @ZalandoTech
- 2. 2 ZALANDO 15 markets 6 fulfillmentcenters 20 million active customers 3.6 billion € net sales 2016 165 million visits per month 12,000 employees in Europe
- 3. 3 ZALANDO TECHNOLOGY HOME-BREWED, CUTTING-EDGE & SCALABLE technologysolutions >1,700 employees from tech locations + HQs in Berlin6 77 nations help our brand to WIN ONLINE
- 4.
- 5. 5 FOUR ERAS ATZALANDO TECH ZOMCATPHP STUPS KUBERNETES 2010 2015 2016 Data center WAR AWS Docker Cloud Formation Low level (AWS API) AWS Docker Kubernetes manifest High abstraction level Data center PHP files
- 6.
- 8.
- 9. 9 KUBERNETES ON AWS:CONTEXT 200 engineering teams 30 prod. clusters AWS/STUPS Dockerized apps No manual operations Reliability Autoscaling Seamless migration
- 10. 10 ISOLATED AWS ACCOUNTS Internet *.abc.example.org*.xyz.example.org Product ABC Product XYZ EC2 LBLB
- 11.
- 12.
- 13. 13 DEPLOYMENT CONFIGURATION . ├── apply │├── credentials.yaml # K8s TPR │ ├── ingress.yaml # K8s Ingress │ ├── redis-deployment.yaml # K8s Deployment │ ├── redis-service.yaml # K8s Service │ └── service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # proprietary config
- 14. 14 INGRESS.YAML apiVersion: extensions/v1beta1 kind: Ingress metadata: name:"..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
- 15.
- 16.
- 17. 17 CLOUD FORMATION VIACI/CD . ├── apply │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml # K8s Ingress │ ├── kube-secret.yaml # K8s Secret │ └── kube-service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # CI/CD config
- 18. 18 ASSIGNING AWS IAMROLE TO POD kind: Deployment spec: template: metadata: annotations: # annotation for kube2iam iam.amazonaws.com/role: "app-myapp-role" spec: containers: - name: ... ... https://github.com/jtblin/kube2iam ⇒ AWS SDKs just work as expected
- 19.
- 20. 20 CLUSTER AUTOSCALING Control #of worker nodes in ASG: • Satisfy all resource requests • One spare node per AZ • No manual config “tweaking” • Scale down, but not too fast ⇒ we want to be “elastic” https://github.com/hjacobs/kube-aws-autoscaler
- 21.
- 22. 22 SERVICE TO SERVICEAUTHNZ Kubernetes Cluster https://resource-server.example.org/protected HTTP/1.1 401 Unauthorized { "message": "Authorization required" }
- 23.
- 24. 24 USING THE OAUTHCREDENTIALS #!/bin/bash secret=$(cat /creds/mytok-token-secret) curl -H "Authorization: Bearer $secret" https://resource-server.example.org/protected
- 25.
- 26. 26 1. Getting Started 2.Stability 3. Onboarding 4. User Experience 5. Operations CHALLENGES
- 27.
- 28.
- 29.
- 30.
- 31. 31 CLUSTER PROVISIONING • TwoCloud Formation stacks • Master & worker ASGs + etcd • Nodes w/ Container Linux • K8s manifests applied separately • kube-system Deployments • DaemonSets
- 32. 32 GETTING STARTED Goal: useKubernetes API as primary interface for AWS • Mate, External DNS • Kubernetes Ingress Controller for AWS • kube2iam ⇒ we wrote new components to achieve our goal
- 33. 33 INGRESS CONTROLLER https://github.com/zalando-incubator/kube-ingress-aws-controller /https://github.com/kubernetes-incubator/external-dns
- 34. 34 GETTING STARTED Other questionswe asked ourselves.. • Single AZ vs. Multi AZ? • Federation? • Overlay network? • Authnz?
- 35. 35 GETTING STARTED Other questionswe asked ourselves.. • Single AZ vs. Multi AZ? ⇒ Multi AZ • Federation? ⇒ No, not ready yet • Overlay network? ⇒ Flannel, “rock solid” • Authnz? ⇒ OAuth, webhook
- 36.
- 37. 37 STABILITY • Cluster Updates •Docker • AWS Rate Limits
- 38.
- 40. 40 STABILITY: AWS RATELIMITS • Ran into the same trap twice (Mate & Ingress Ctrl) • Kubernetes core causes many calls (e.g. EBS) • Monitoring (ZMON) needs to poll AWS ⇒ One of our biggest pain points with AWS (and all workarounds are hard and/or ugly)
- 41. 41 STABILITY: LIMIT RANGE kubectldescribe limitrange Name: limits Namespace: default Type Resource Min Max Default Req Default Limit Max Limit/Request Ratio ---- -------- --- --- ----------- ------------- ----------------------- Container memory - 64Gi 100Mi 1Gi - Container cpu - 16 100m 3 - http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html#resources ⇒ Mitigate errors on OSI layer 8 ;-)
- 43. Recommended: The 5Whys https://en.wikipedia.org/wiki/5_Whys
- 44.
- 45. 45 ONBOARDING • Many newconcepts to grasp vs. 200 teams • Kubernetes Training (2h) • Documentation • Recorded Friday Demos • Support Channels (chat, mail)
- 46.
- 47. 47 USER EXPERIENCE • Jenkinsdeployment only covers “happy case” • Juggling with YAMLs • Weighted traffic switching missing
- 48. 48 UX: WEIGHTED TRAFFICSWITCHING • STUPS uses weighted Route53 DNS records • Allows canary, blue/green, slow ramp up • Approach: add weights to Ingress backends https://github.com/zalando/skipper/issues/324
- 49. 49 UX: WEIGHTED TRAFFICSWITCHING https://github.com/zalando/skipper/issues/324
- 50.
- 51. 51 OPERATIONS • Team Autonomy? •Platform as a Service • Convergence • Emergency Operator Access ⇒ Hard challenges..
- 52.
- 53. 53 LINKS Running Kubernetes inProduction on AWS http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html Kube AWS Ingress Controller https://github.com/zalando-incubator/kube-ingress-aws-controller External DNS https://github.com/kubernetes-incubator/external-dns PostgreSQL Operator https://github.com/zalando-incubator/postgres-operator Zalando Cluster Configuration https://github.com/zalando-incubator/kubernetes-on-aws List of Organizations using Kubernetes on AWS https://github.com/hjacobs/kubernetes-on-aws-users
- 54. QUESTIONS? HENNING JACOBS TECH INFRASTRUCTURE CLOUDENGINEER henning@zalando.de @try_except_ Illustrations by @01k