In this talk we share our learnings from running Kubernetes on AWS in production and how we are migrating 200+ engineering teams from AWS/STUPS to Kubernetes. This talk was given at the Berlin AWS User Group meetup on 2017-05-09 hosted by NewStore (https://www.meetup.com/aws-berlin/events/236795816/). More information on http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html

1. Kubernetes on AWS
@ZalandoTech
AWS BERLIN USER GROUP
2017-05-09
HENNING JACOBS
@try_except_

2. 2
ZALANDO
15 markets
6 fulfillment centers
20 million active customers
3.6 billion € net sales 2016
165 million visits per month
12,000 employees in Europe

3. 3
ZALANDO TECHNOLOGY
HOME-BREWED,
CUTTING-EDGE
& SCALABLE
technology solutions
>1,600
employees from
tech locations
+ HQs in Berlin6
77
nations
help our brand to
WIN ONLINE

4. 4
KUBERNETES ON AWS: CONTEXT
200 engineering teams
30 prod. clusters
AWS
Dockerized apps
No manual operations
Reliability
Autoscaling
Seamless migration

5. 5
ARCHITECTURE

6. 6
ISOLATED AWS ACCOUNTS
Internet
*.abc.example.org *.xyz.example.org
Product ABC Product XYZ
EC2
LBLB

7. 7
KUBERNETES ON AWS

8. 8
CLUSTER PROVISIONING

9. 9
CLUSTER PROVISIONING
• Two Cloud Formation stacks
• Master & worker ASGs + etcd
• Nodes w/ Container Linux
• K8s manifests applied separately
• kube-system Deployments
• DaemonSets

10. 10
DEPLOYMENT

11. 11
DEPLOYMENT CONFIGURATION
.
├── apply
│ ├── credentials.yaml # K8s TPR
│ ├── ingress.yaml # K8s Ingress
│ ├── redis-deployment.yaml # K8s Deployment
│ ├── redis-service.yaml # K8s Service
│ └── service.yaml # K8s Service
├── deployment.yaml # K8s Deployment
└── pipeline.yaml # proprietary config

12. 12
JENKINS DEPLOY PIPELINE

13. 13
INGRESS

14. 14
INGRESS.YAML
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "myapp.foo.example.org"
http:
paths:
- backend:
serviceName: "{{ application }}"
servicePort: 80

15. 15
INGRESS CONTROLLER
https://github.com/zalando-incubator/kube-ingress-aws-controller / https://github.com/kubernetes-incubator/external-dns

16. 16
AWS INTEGRATION

17. 17
CLOUD FORMATION VIA CI/CD
.
├── apply
│ ├── cf-iam-role.yaml # AWS IAM Role
│ ├── cf-rds.yaml # AWS RDS Database
│ ├── kube-ingress.yaml # K8s Ingress
│ ├── kube-secret.yaml # K8s Secret
│ └── kube-service.yaml # K8s Service
├── deployment.yaml # K8s Deployment
└── pipeline.yaml # CI/CD config

18. 18
ASSIGNING AWS IAM ROLE TO POD
kind: Deployment
spec:
template:
metadata:
annotations:
# annotation for kube2iam
iam.amazonaws.com/role: "app-{{ application }}-1"
spec:
containers:
- name: ...
...
https://github.com/jtblin/kube2iam
⇒ AWS SDKs just work as expected

19. 19
CLUSTER
AUTOSCALING

20. 20
CLUSTER AUTOSCALING
Control # of worker nodes in ASG:
• Satisfy all resource requests
• One spare node per AZ
• No manual config “tweaking”
• Scale down, but not too fast

21. 21
CURRENT SETUP
• https://github.com/hjacobs/kube-aws-autoscaler
• Node draining via systemd unit
Open topic: node “readiness” during scale out

23. 23
OAUTH / IAM
INTEGRATION

24. 24
DECLARING NEEDED CREDENTIALS
# apply/credentials.yaml
apiVersion: "zalando.org/v1"
kind: PlatformCredentialsSet
metadata:
name: "..."
spec:
tokens: # OAuth service tokens
mytok:
privileges:
- com.zalando::foobar.write
clients: # OAuth clients
implicit:
grant: implicit # grant type according to RFC-6749
realm: users
redirectUri: https://myapp.foo.example.org/oauth
⇒ TPRs FTW

25. 25
MOUNTING THE OAUTH CREDENTIALS
kind: Deployment
spec:
template:
spec:
containers:
- name: ...
...
volumeMounts:
- name: "..."
mountPath: /meta/credentials
readOnly: true
volumes:
- name: "..."
secret:
secretName: "..."

26. 26
USING THE OAUTH CREDENTIALS
#!/bin/bash
type=$(cat /meta/credentials/mytok-token-type)
secret=$(cat /meta/credentials/mytok-token-secret)
curl -H "Authorization: $type $secret"
https://resource-server.example.org/protected

27. 27
POSTGRESQL

28. 28
POSTGRESQL ON KUBERNETES
• StatefulSet + PV (EBS)
• Spilo/Patroni
• PostgreSQL Operator
• ThirdPartyResources
https://github.com/zalando/spilo https://github.com/zalando/patroni

29. 29
WHY NOT RDS?
• Logical Replication
• Outbound Streaming Replication
• Heterogeneous Replicas
• Custom Extensions
• Real Superuser
• OAuth Login
NB: we also use RDS
and it works great!

30. 30
OPERATIONS

31. 31
CLUSTER
UPDATES

32. 32
LIMIT RANGE
kubectl describe limitrange
Name: limits
Namespace: default
Type Resource Min Max Default Req Default Limit Max Limit/Request Ratio
---- -------- --- --- ----------- ------------- -----------------------
Container memory - 64Gi 100Mi 1Gi -
Container cpu - 16 100m 3 -
http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html#resources
⇒ Mitigate errors on OSI layer 8 ;-)

33. https://github.com/hjacobs/kube-ops-view

34. 34
KUBERNETES
VS.
AWS ECS

35. 35
AWS API
Tasks, Services
Static AWS API
Blox
Operating worker nodes
Vendor community/support
AWS only
KUBERNETES VS. AWS ECS
Declarative API (fast & no rate limits)
High level abstractions (Ingress, CronJob)
Extensible API (e.g. TPR)
Batteries included (DaemonSet, StatefulSet)
Operating etcd, master & worker nodes
Huge community
Run anywhere
⟺
⟺
⟺
⟺
⟺
⟺
⟺
disclaimer: incomplete and opinionated ;-)

36. 36
OPEN SOURCE

37. 37
LINKS
Running Kubernetes in Production on AWS
http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html
Kube AWS Ingress Controller
https://github.com/zalando-incubator/kube-ingress-aws-controller
External DNS
https://github.com/kubernetes-incubator/external-dns
Zalando Cluster Configuration
https://github.com/zalando-incubator/kubernetes-on-aws
List of Organizations using Kubernetes on AWS
https://github.com/hjacobs/kubernetes-on-aws-users

38. QUESTIONS?
HENNING JACOBS
TECH INFRASTRUCTURE
CLOUD ENGINEER
henning@zalando.de
@try_except_
Illustrations by @01k