Large Scale Kubernetes on AWS at Europe's Leading Online Fashion Platform - AWS Tech Community Days Cologne
Bootstrapping a Kubernetes cluster is easy, rolling it out to nearly 200 engineering teams and operating it at scale is a challenge.
In this talk, we are presenting our approach to Kubernetes provisioning on AWS, operations and developer experience for our growing Zalando Technology department. We will highlight in the context of Kubernetes: AWS service integrations, our IAM/OAuth infrastructure, cluster autoscaling, continuous delivery and general developer experience. The talk will cover our most important learnings and we will openly share failure stories.
Presented on 2017-09-28 at AWS Tech Community Days in Cologne.
FOUR ERAS AT ZALANDO TECH
ZOMCATPHP STUPS KUBERNETES
2010 2015 2016
Low level (AWS API)
High abstraction level
ASSIGNING AWS IAM ROLE TO POD
# annotation for kube2iam
- name: ...
⇒ AWS SDKs just work as expected
STABILITY: AWS RATE LIMITS
• Ran into the same trap twice (Mate & Ingress Ctrl)
• Kubernetes core causes many calls (e.g. EBS)
• Monitoring (ZMON) needs to poll AWS
⇒ One of our biggest pain points with AWS
(and all workarounds are hard and/or ugly)
STABILITY: LIMIT RANGE
kubectl describe limitrange
Type Resource Min Max Default Req Default Limit Max Limit/Request Ratio
---- -------- --- --- ----------- ------------- -----------------------
Container memory - 64Gi 100Mi 1Gi -
Container cpu - 16 100m 3 -
⇒ Mitigate errors on OSI layer 8 ;-)
Recommended: The 5 Whys
Running Kubernetes in Production on AWS
Kube AWS Ingress Controller
Zalando Cluster Configuration
List of Organizations using Kubernetes on AWS