How do we use Kubernetes
•Download as PPTX, PDF•
0 likes•36 views
Zalando uses Kubernetes extensively to manage its technology infrastructure and platforms. It currently operates 99 clusters across 380 AWS accounts. Key aspects of Zalando's Kubernetes architecture include using one production cluster per product, running etcd separately on EC2 instances, implementing multi-AZ clusters, providing isolated live and test environments, and integrating Kubernetes deployments with its continuous delivery platform and AWS using tools like kube2iam. Zalando has also developed and contributed several open source projects related to Kubernetes operations.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How do we use Kubernetes
Similar to How do we use Kubernetes (20)
Recently uploaded
Recently uploaded (20)
How do we use Kubernetes
- 1. HOW DO WE USE KUBERNETES HELSINKI KUBERNETES MEETUP October 2018
- 3. 3 ZALANDO TECH PLATFORM THE COMPLETE HISTORY (ABRIDGED) ZOMCATPHP STUPS KUBERNETES 2010 2015 2016 Data center WAR LXC AWS Docker Cloud Formation AWS Docker Cloud Formation Kubernetes manifest Data center PHP files 2008 CDP 2017 … same … plus git-controlled deployments
- 4. 4 ISOLATED AWS ACCOUNTS Internet *.abc.zalando.net Product XYZ abc Account Load Balancer def Account Load Balancer *.def.zalando.net
- 5. 5 ZALANDO TECH ~ 2.000 Employees in Tech > 200 Delivery teams
- 6. 6 MOTIVATION FOR KUBERNETES Resource efficiency Cost efficiency Velocity Improved compliance THIS IS AN OPPORTUNITY FOR CHANGE!
- 8. 8 “PHILOSOPHY” No pet clusters We don’t want to tweak custom settings for dozens of clusters. Always provide the latest stable Kubernetes version Oldest clusters were upgraded from v1.4 through v1.11. Continuous and non-disruptive cluster updates No maintenance windows. “Fully” automated operations Operators should only need to manually merge PRs to initiate upgrade.
- 10. 10 ARCHITECTURE DECISIONS • One production cluster per “product” • API server behind SSL ELB, OAuth webhook • Read only access to production • CI/CD for write access • etcd running separately on EC2 • Multi AZ clusters
- 11. 11 etcd etcd CLUSTER CONTROL AT A GLANCE ... ... zkubectl Worker Node (3 AZs) Kubelet Pod Container Pod Container Container Master Node API Server Scheduler Controller Manager etcd USER
- 12. 12 ISOLATED LIVE AND TEST CLUSTERS *.abc.zalando.net *.def.zalando.net def Account Load Balancerabc Account Load Balancer abc-test Account Load Balancer *.abc-test.zalando.net def-test Account Load Balancer *.def-test.zalando.net Internet
- 15. 15 CDP: APPLY
- 17. 17 CLOUD FORMATION VIA CI/CD . ├── deploy/apply │ ├── deployment.yaml # K8s Deployment │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml # K8s Ingress │ ├── kube-secret.yaml # K8s Secret │ └── kube-service.yaml # K8s Service └── delivery.yaml # CI/CD config
- 18. 18 ASSIGNING AWS IAM ROLE TO POD kind: Deployment spec: template: metadata: annotations: # annotation for kube2iam iam.amazonaws.com/role: "app-myapp-role" spec: containers: - name: ... ... https://github.com/jtblin/kube2iam ⇒ AWS SDKs just work as expected
- 20. 20 INFRASTRUCTURE @ ZALANDO STUPS (toolset around AWS) Kubernetes AWS accounts per team. All instances must run the same AMI. PowerUser access to Production. Clusters per product (multiple teams). Instances are not managed by teams. Hands off approach. You build it, you run EVERYTHING. A lot of stuff out of the box.
- 23. 23 OPEN SOURCE
- 25. 25 • Kubernetes Custom Resource Controller • Creates and manages PostgreSQL cluster • Based on Patroni and Spilo projects POSTGRESQL OPERATOR
- 27. 27 When a new postgresql custom resource appears, the operator creates: 1. StatefulSet for PostgreSQL/Patroni cluster 2. Service for master node (ClusterIP or LB) 3. Service for replica nodes (ClusterIP or LB) 4. DNS names for the services if needed If the resource is modified, the operator applies the modification to the cluster. POSTGRESQL OPERATOR
- 28. 28 CONNECT TO THE POSTGRESQL DB
- 30. 30 OPEN SOURCE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws AWS ALB Ingress controller github.com/zalando-incubator/kube-ingress-aws-controller Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator Kubernetes Resource Report github.com/hjacobs/kube-resource-report Kubernetes Downscaler github.com/hjacobs/kube-downscaler
- 31. THANK YOU QUESTIONS? uri.savelchev@zalando.fi Thanks to Jannis Rake-Revelant, Rodrigo Reis, Dimitrij Holev, Henning Jacobs, etc. 2018-10-08 URI SAVELCHEV HELSINKI TECH HUB
Editor's Notes
- How we came to Kubernetes...
- AWS is our primary platform since 2015 Why Kubernetes:
- Zalando Radical Agility and Team Autonomy - every team decides what’s working for it Isolated AWS accounts ⇒ No VPN tunnels or VPC peering One single K8S Cluster per Account Various services live in the cluster, grouped by so called Product Communities (e.g. Fashion Store, Logistics) Several teams “live” in the same cluster Cluster to Cluster communication goes through public internet and require encrypted channels and authentication
- We are around 2.000 employees in tech and more than 200 tech teams.
- Resource Efficiency Application instances share nodes Scaling of worker nodes handled automatically Cost Efficiency - And reduce costs related with resource underutilization, like ASG of dedicated EC2 instances Velocity - No need to spin-up nodes Cloud Independence - Kubernetes available in GKE, Azure, ...
- With that in mind we had sort of a philosophy for how to run Kubernetes at our scale and provide most value for our users.
- We have Immutable nodes and use docker as deployment artifact. This perfectly matches our existing AWS deployment model
- This is a really simplified picture of Kubernetes’ setup; The ASG from the previous slide Another ASG for Worker Nodes One etcd cluster to store configuration Nodes are distributed throughout 3 AZs
- 3 >> Operations
- Cloud Formation YAML files in “apply” are treated similar to K8s manifests
- Kube2iam, mention that ECS uses the same trick!
- How is the transition for the teams? Quick context on how teams move from stups to K8S
- Currently we have X AWS Accounts and Y Kubernetes clusters We have one cluster per AWS account. We have many AWS accounts because we are transitioning from a model where each team have their own AWS Account.
- No manual operations: all cluster updates and operations need to be fully automated. No pet clusters: clusters should all look the same and not require any specific configurations/tweaking Reliability: the infrastructure should be rock-solid for our delivery teams to entrust our clusters with their most critical applications Autoscaling: clusters should automatically adapt to deployed workloads and hourly scaling events are expected Seamless migration: Dockerized twelve-factor apps currently deployed on AWS/STUPS should work without modifications on Kubernetes
- What is the problem we had to solve with Skipper and External DNS