Kubernetes at Zalando - CNCF End User Committee Presentation
•
2 likes•1,190 views
This document summarizes Zalando's use of Kubernetes on AWS. It discusses how Zalando uses Kubernetes across 200+ engineering teams, 30+ production clusters, and 8 non-production clusters. Key aspects covered include the architecture with isolated AWS accounts per product, cluster provisioning via CloudFormation, deployment via Jenkins pipelines, ingress controller, integration with AWS services, cluster autoscaling, OAuth integration, operations, monitoring with ZMON, and open sourcing of tools like the Kubernetes ingress controller and external DNS. Overall experience has been positive but missing best practices have been challenging and community/user discussions have been very helpful.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubernetes at Zalando - CNCF End User Committee Presentation
Similar to Kubernetes at Zalando - CNCF End User Committee Presentation (20)
More from Henning Jacobs
More from Henning Jacobs (14)
Recently uploaded
Recently uploaded (20)
Kubernetes at Zalando - CNCF End User Committee Presentation
- 1. Kubernetes at Zalando CNCF END USER COMMITTEE HENNING JACOBS @try_except_ 2017-04-13
- 2. 2 …HAS BECOME THE EUROPEAN ONLINE PLATFORM FOR FASHION
- 3. OUR VISION: CONNECTING PEOPLE AND FASHION
- 4. 4 WE OFFER A SUCCESSFUL AND CURATED ASSORTMENT ~200,000 articles from >1,500 international brands 17 private labels HIGHLY EXPERIENCED category management >350 designers & stylistsLOCALIZATION of the assortment CURATED SHOPPING with Zalon
- 5. 5 ZALANDO 15 markets 6 fulfillment centers 20 million active customers 3.6 billion € net sales 2016 165 million visits per month 12,000 employees in Europe
- 6. 6 OUR FOOTPRINT AROUND EUROPE as at Dec 2016 10 9 8 7 6 5 3 2 1 11 1 2 3 4 5 6 7 8 9 10 11 12 13 12 13 4 BERLIN HEADQUARTERS / TECH HUB BRIESELANG FULFILLMENT CENTER ERFURT FULFILLMENT CENTER MÖNCHENGLADBACH FULFILLMENT CENTER LAHR FULFILLMENT CENTER DORTMUND TECH HUB FRANKFURT OUTLET DUBLIN TECH HUB HELSINKI TECH HUB STRADELLA FULFILLMENT CENTER KÖLN OUTLET MOISSY-CRAMAYEL FULFILLMENT CENTER GRYFINO (start autumn 2017) FULFILLMENT CENTER
- 7. 7 ZALANDO TECHNOLOGY HOME-BREWED, CUTTING-EDGE & SCALABLE technology solutions >1,600 employees from tech locations + HQs in Berlin6 77 nations help our brand to WIN ONLINE
- 8. 8 KUBERNETES ON AWS: CONTEXT 200 engineering teams 30 prod. clusters AWS Dockerized apps No manual operations Reliability Autoscaling Seamless migration
- 9. 9 CURRENT STATUS • First service in prod. on Kubernetes since Nov 2016 • 8 production clusters • 8 non-production clusters • “Early Access” phase (onboarding of individual teams)
- 10. 10 ARCHITECTURE
- 11. 11 ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Product ABC Product XYZ EC2 LBLB
- 13. 13 ARCHITECTURE DECISIONS • One prod. cluster per AWS account / “product” • API server behind SSL ELB, OAuth webhook • Read only access to production • CI/CD for write access • etcd running separately on EC2 • Multi AZ clusters
- 15. 15 CLUSTER PROVISIONING • Two Cloud Formation stacks • Master & worker ASGs + etcd • Nodes w/ Container Linux • K8s manifests applied separately • kube-system Deployments • DaemonSets
- 16. 16 DEPLOYMENT
- 17. 17 DEPLOYMENT CONFIGURATION . ├── apply │ ├── credentials.yaml # K8s TPR │ ├── ingress.yaml # K8s Ingress │ ├── redis-deployment.yaml # K8s Deployment │ ├── redis-service.yaml # K8s Service │ └── service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # proprietary config
- 19. 19 INGRESS
- 22. 22 CLOUD FORMATION VIA CI/CD . ├── apply │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml # K8s Ingress │ ├── kube-secret.yaml # K8s Secret │ └── kube-service.yaml # K8s Service ├── deployment.yaml # K8s Deployment └── pipeline.yaml # CI/CD config
- 24. 24 CLUSTER AUTOSCALING Control # of worker nodes in ASG: • Satisfy all resource requests • One spare node per AZ • No manual config “tweaking” • Scale down, but not too fast https://github.com/hjacobs/kube-aws-autoscaler
- 26. 26 OAUTH INTEGRATION • App declares needed credentials via Kubernetes Third Party Resource • OAuth client/tokens are provisioned as Kubernetes secrets
- 28. 28 OPERATIONS • Cluster updates automatic via CLM • CronJob is great, but needs cleanup • Docker can be PITA
- 30. 30 MONITORING •
- 31. 31 MONITORING • Each cluster contains ZMON appliance • K8s resources are available as ZMON entities • Users can create app checks/alerts via UI https://zmon.io/
- 33. 33 OPEN SOURCE
- 34. 34 OPEN SOURCE Kube AWS Ingress Controller https://github.com/zalando-incubator/kube-ingress-aws-controller External DNS https://github.com/kubernetes-incubator/external-dns Zalando Cluster Config & Docs https://github.com/zalando-incubator/kubernetes-on-aws more to come...
- 35. 35 OUR EXPERIENCE SO FAR • Missing best practices are a big pain point • Community is great and welcoming • Talking with other users is essential • Slack channels are “compensating” lack of docs
- 36. 36 LIST OF KUBERNETES ON AWS USERS If you are using Kubernetes on AWS, please fill out the Google form: https://github.com/hjacobs/kubernetes-on-aws-users +
- 37. QUESTIONS? HENNING JACOBS TECH INFRASTRUCTURE CLOUD ENGINEER henning@zalando.de @try_except_