This document discusses Zalando's transition to using Kubernetes on AWS. It describes Zalando's motivation for adopting Kubernetes which was to improve resource efficiency, cost efficiency, velocity and cloud independence. It outlines some of the challenges in deploying Kubernetes at scale, such as compliance, deployment processes and ease of use. Zalando developed an in-house Continuous Delivery Platform (CDP) and practices to address these challenges by enabling hands-off, compliant-by-default and secure-by-default Kubernetes deployments.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NORDICS
Clarion Hotel Helsinki
March 21, 2018

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HOW ZALANDO INTEGRATES
KUBERNETES WITH AWS
| MARCH 2018RUBEN DIAZ
URI SAVELCHEV

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WE BRING FASHION
TO PEOPLE IN 15
EU COUNTRIES
+2 IN 2018
2008-2009
2010
2012-2013
2011

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
> 23
million
active customers
~ 2,000
brands
~4.5billion EUR
revenue 2017
> 210
million
visits
per
month
> 15,000
employees in
Europe
> 70%
of visits via
mobile devices
> 250,000
product choices
15
countries

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BUILDING OUR
ECOMMERCE
PLATFORM
AWS, Microservices, Scala,
Android and iOS
>110
employees
Autonomous delivery
teams working with
modern technologies
12
29
Nationalities
Our office is located in
KAMPPI

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TECH
INFRASTRUCTURE

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ZALANDO TECH PLATFORM - THE HISTORY
(ABRIDGED)
ZOMCATPHP STUPS KUBERNETES
2010 2015 2016
Data center
WAR
LXC
AWS
Docker
Cloud Formation
AWS
Docker
Cloud Formation
Kubernetes manifest
Data center
PHP files
2008
CDP
2017
… same …
plus
git-controlled
deployments

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ISOLATED AWS ACCOUNTS
Internet
*.abc.zalando.net
Product
XYZ
abc
Account
Load Balancer
def
Account
Load Balancer
*.def.zalando.net

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MOTIVATION FOR KUBERNETES
• Resource efficiency
• Cost efficiency
• Velocity
• Cloud independence
THIS IS AN OPPORTUNITY FOR CHANGE!

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SCALE?
66 Clusters 329 Accounts ~10,000 EC2 30 TB
(Docker images)

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
OUR KUBERNETES
ARCHITECTURE

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CLUSTER COMPONENTS

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DETAILED TRAFFIC FLOW
SKIPPER

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
etcd
etcd
...
...
zkubectl
Worker
Node (3 AZs)
Kubelet
Pod
Container
Pod
Container
Container
Master
Node
API
Server
Scheduler
Controller
Manager
etcdUSER

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHALLENGES
• Compliance
• How To Deploy
• Easy Of Use

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
COMPLIANCE CHALLENGES
Some of our compliance rules:
● Applications must run on certified (or whitelisted) AMIs
● All images must:
○ Come from an authorized Docker registry
○ Contain an SCM Source file to refer to a specific revision of the
code
○ Be versioned
● Code changes must be peer reviewed and approved (4 eyes
principle)

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHALLENGE - HOW TO DEPLOY
Options for CI/CD:
• Jenkins
• GoCD
• Concourse
• Spinnaker
• Travis Enterprise
• AWS CodeBuild,
CodePipeline
Problems:
• Non reproducible builds
• Not cloud ready
• No automatic setup
• Difficult to scale up/down
• Cumbersome build
configuration
• Manual credential configuration
• Lack of Kubernetes support

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EASE OF USE

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What if the developer doesn’t have to
worry about those steps?*
* coding not included

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A DIFFERENT
APPROACH
• Hands off
• Compliant by
default
• Secure by default

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A DIFFERENT APPROACH
• Hands off
• Compliant by
default
• Secure by default
 Disable manual access to Production*
 Automate Setup/Deployment steps
 Separate Test and Production
environments
* some exceptions apply

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A DIFFERENT
APPROACH
• Hands off
• Compliant by
default
• Secure by default

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HANDS OFF
Back to our options for CI/CD…
● Jenkins
● GoCD
● Concourse
● Spinnaker
● Travis Enterprise
● AWS CodeBuild / CodePipeline
✓ In-house developed (CDP)

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
HANDS OFF
Continuous Delivery Platform (CDP)
● Fully integrated with Kubernetes
● No need to manage CI infrastructure
● Fully integrated with GitHub Enterprise
● Triggered by code changes
● Can also deploy CloudFormation stacks

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DEPLOYMENT PIPELINE
CDP
GHE
abc
Account
deploytrigger buildpush code
push
build
trigger
depoy
...

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CONTINUOUS DELIVERY PLATFORM

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CONTINUOUS DELIVERY PLATFORM – VIEW LOGS

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS INTEGRATION

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CLOUDFORMATION VIA CI/CD
.
├── deploy/apply
│ ├── deployment.yaml # K8s Deployment
│ ├── cf-iam-role.yaml # AWS IAM Role
│ ├── cf-rds.yaml # AWS RDS Database
│ ├── kube-ingress.yaml # K8s Ingress
│ ├── kube-secret.yaml # K8s Secret
│ └── kube-service.yaml # K8s Service
└── delivery.yaml # CI/CD config

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASSIGNING AWS IAM ROLE TO A POD
kind: Deployment
spec:
template:
metadata:
annotations:
# annotation for kube2iam
iam.amazonaws.com/role: "app-myapp-role"
spec:
containers:
- name: ...
...

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A DIFFERENT
APPROACH
 Hands off
• Compliant by
default
• Secure by default

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
COMPLIANT BY DEFAULT
Kubernetes AMIs
● Developers don’t have to choose instance type or AMI
○ Deployments result in Pods running in existing Worker
Nodes
● All Kubernetes Nodes are based in compliant, whitelisted AMIs
Docker Registry
● CDP only pulls images from authorized repositories
● When pushing images after build CDP automatically includes:
○ SCM Source information
○ Version tagging

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
COMPLIANT BY DEFAULT
Our compliance rules are met automatically:
 Applications must run on certified (or whitelisted) AMIs
 All images must:
 Come from an authorized Docker registry
 Contain an SCM Source file to refer to a specific revision of the
code
 Be versioned
 Code changes must be peer reviewed and approved (4 eyes
principle)

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A DIFFERENT
APPROACH
 Hands off
 Compliant by
default
• Secure by default

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SECURE BY DEFAULT
● Test and Production environments are completely isolated
● A Kubernetes Test Cluster is also provisioned
○ Manual access is permitted
● OAuth Credentials are tied to either Test or Live environments
○ Different OAuth Provider in test environment
○ Communication between Test and Live services is
effectively disabled

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SO THIS SCHEMA TRANSFORMS INTO…
Internet
*.abc.zalando.net
Product
XYZ
abc
Account
Load Balancer
def
Account
Load Balancer
*.def.zalando.net

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
...EXPANDS TO THIS*.abc.zalando.net *.def.zalando.net
def
Account
Load Balancerabc
Account
Load Balancer
abc-test
Account
Load Balancer
*.abc-test.zalando.net
def-test
Account
Load Balancer
*.def-test.zalando.net
Internet

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PUTTING IT ALL TOGETHER
CDP
GHE
abc-test
Account
trigger
push
build
trigger
depoy
...
abc
Account
deploy
release

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A DIFFERENT
APPROACH
 Hands off
 Compliant by
default
 Secure by default

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CONCLUSION
• CDP enables hands off deployments to Kubernetes
• Compliance is automatically handled by CDP and Developer Console
• Test and Production are guaranteed to be separated through Credentials
Isolation
 Automation saves time
 New features go live faster
 Isolation secures environments
 AWS let us run smoothly and fast
 Developers focus on development

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LINKS Zalando Cluster Configuration
https://github.com/zalando-incubator/kubernetes-on-aws
Kubernetes on AWS Docs
http://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-
production.html
Skipper HTTP Ingress Router
https://github.com/zalando/skipper/
Kube AWS Ingress Controller
https://github.com/zalando-incubator/kube-ingress-aws-controller
External DNS
https://github.com/kubernetes-incubator/external-dns

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU! QUESTIONS?
RUBEN DIAZ, URI SAVELCHEV
HELSINKI TECH HUB
ruben.diaz@zalando.fi
uri.savelchev@zalando.fi
Also thanks to
Rodrigo Reis, Dimitrij Holev,
Henning Jacobs and others
MARCH 2018