Speakers: Joost & Milo (Albert Heijn) Genre & level: Backend, Junior Joost and Milo from Albert Heijn join us to talk about how they implemented Kubernetes.

1. Kubernetes
Love at first sight?
7, June 2018
Joost Hofman (Lead Developer @ Albert Heijn IT Online)
Milo van der zee (Senior Developer @Albert Heijn IT Online)

2. Agenda
How?
Kubernetes
Why at AH?
Relational problems Is it real love?
Questions
kubectl get

3. Kubernetes
is an open-source system for automating deployment, scaling,
and management of containerized applications.
kubectl get

4. Kubernetes - Searches
kubectl get

5. Kubernetes
kubectl get
Service
POD POD
1 … n1 … n

6. Kubernetes
kubectl get
Operator /
Developer
Kubernetes Master
API Server Controller Manager
Scheduler
ETCD
Kubernetes Node
Kubelet kube-proxy
Pod Pod Pod Pod…
Kubernetes Node
Kubelet kube-proxy
Pod Pod Pod Pod…Up to 5000
Users
Plugin Network - Calico

7. user@host $ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8snode2098 Ready,SchedulingDisabled master 12d v1.8.4+coreos.0
k8snode2099 Ready,SchedulingDisabled master 12d v1.8.4+coreos.0
k8snode2100 Ready,SchedulingDisabled master 12d v1.8.4+coreos.0
k8snode2101 Ready node 12d v1.8.4+coreos.0
k8snode2102 Ready node 12d v1.8.4+coreos.0
k8snode2103 Ready node 12d v1.8.4+coreos.0
k8snode2104 Ready node 12d v1.8.4+coreos.0
k8snode2105 Ready node 12d v1.8.4+coreos.0
k8snode2107 Ready node 12d v1.8.4+coreos.0
k8snode2108 Ready node 12d v1.8.4+coreos.0
k8snode2109 Ready node 12d v1.8.4+coreos.0
k8snode2110 Ready node 12d v1.8.4+coreos.0
k8snode2111 Ready node 12d v1.8.4+coreos.0
Kubernetes

8. user@host $ kubectl get pods -o wide
NAME READY STATUS IP NODE
shoppinglist-widget-3162246403-q7c1x 1/1 Running 10.233.106.55 k8snode1657
subscription-service-8cc4c97fb-dh9zz 1/1 Running 10.233.87.218 k8snode1656
subscription-service-8cc4c97fb-t7wrj 1/1 Running 10.233.73.169 k8snode1651
taxonomy-neo4j-neo4j-core-0 1/1 Running 10.233.124.123 k8snode1814
taxonomy-neo4j-neo4j-core-1 1/1 Running 10.233.73.147 k8snode1651
taxonomy-neo4j-neo4j-core-2 1/1 Running 10.233.79.109 k8snode1813
taxonomy-service-7b4fb7f8d5-c6mvb 1/1 Running 10.233.79.105 k8snode1813
taxonomy-service-7b4fb7f8d5-h2hjk 1/1 Running 10.233.68.145 k8snode1655
gateway-3060515939-57r22 1/1 Running 10.233.124.98 k8snode1814
gateway-3060515939-9lqzk 1/1 Running 10.233.68.185 k8snode1655
gateway-3060515939-fkt9k 1/1 Running 10.233.71.29 k8snode1654
gateway-3060515939-ls9pv 1/1 Running 10.233.79.101 k8snode1813
Kubernetes

9. # kubectl -n online-prd describe pod gateway-3060515939-57r22
Name: gateway-3060515939-57r22
Namespace: online-prd
Node: k8snode1814/150.83.153.243
Start Time: Wed, 14 Feb 2018 13:12:03 +0100
Labels: name=gateway
Status: Running
IP: 10.233.124.98
Containers:
gateway:
Image: regisry-docker.online.ah.nl:443/ah-open-api-gateway:0.1.2
Port: <none>
Pods – kubectl describe pod api gateway

10. # kubectl describe svc gateway
Name: gateway
Namespace: online-prd
Labels: run=gateway
Annotations: kubectl.kubernetes.io/last-applied-
configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"ru
n":"gateway"},"name":"gateway","namespace":"online-prd"},"spec":{"ports":...
Selector: run=gateway
Type: ClusterIP
IP: 10.233.52.234
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
Endpoints: 10.233.124.98:8080,10.233.68.185:8080,10.233.71.29:8080 + 1 more...
Session Affinity: None
Events: <none>
Service - kubectl describe svc api gateway

11. -A KUBE-SERVICES -d 10.233.52.234/32 -p tcp -m tcp
--dport 443 -j SVC-JFMNS
-A SVC-JFMNS --mode random --probability 0.25 -j KUBE-SEP-JPX2Q
-A SVC-JFMNS --mode random --probability 0.33 -j KUBE-SEP-KUJYT
-A SVC-JFMNS --mode random --probability 0.5 -j KUBE-SEP-HTGFR
-A SVC-JFMNS --mode random -j KUBE-SEP-JP5GT
-A SEP-JPX2Q -p tcp -m recent
-j DNAT --to-destination 143.54.22.4:6443
kubectl get
api service – iptables

12. Why @ Albert Heijn?
kubectl get
2015
Monolith
Binary coupling
Scalability problems
Growth issues
CI/CD impossible
Downtime
Scalable
Decoupling
Rolling updates
Services
CI/CD to the max
Isolation of code
Zero downtime
Technology agnostic
NOW and future

13. Why @ Albert Heijn?
kubectl get
… on a modern, scalable, automated platform
Scalable architecture and technology
Commodity
hardware
Virtualization
Virtual hardware
Container management platform
- Manual
- Within months
- Semi-automated
- Within weeks
- Fully automated
- Within minutes
Containers

14. On Premise VS Cloud
kubectl get
No cloud options in 2016 and 2017

15. How?
kubectl get

16. How?
kubectl get

17. A HTTP call to appietoday.nl
kubectl get
Users
Loadbalancer
Nginx - Ingress
Frontend (service)
Frontend (pod)
API Gateway (service)
API Gateway (pod)
API (service)
API (pod)
IDP (service)
IDP (pod)

18. Our setup?
kubectl get
PlatformServicesAPI GatewayFrontend
65+ services /
components
5 Clusters
50+ nodes
850+ Docker containers
Continuous delivery
Continuous delivery –
Automated from
development to
production
Authorization
Authentication
Throttling
Routing
Automate platform
deployment with Ansible

19. Relational problems: Communication and Storage.
kubectl get

20. Relational problems: Storage.
kubectl get
On premise Storage
vSphere volumes
Host path
NFS

21. Relational problems: Storage.
kubectl get
On premise Storage
GlusterFS

22. Relational problems: Postgres on Gluster.
kubectl get
pg_restore: [archiver (db)] Error from TOC entry 53398; 0 16503 TABLE
DATA l1aaux_sci sdmcleod
pg_restore: [archiver (db)] COPY failed for table "l1aaux_sci": ERROR:
unexpected data beyond EOF in block 9391 of relation base/16386/17043
HINT: This has been seen to occur with buggy kernels; consider
updating your system.
CONTEXT: COPY l1aaux_sci, line 319329: "1854661 N
1.05156717906094999 1378796678.44843268 2012-02-01
07:04:39.5+00 2012-02-01 07:04:38.4484..."
pg_restore: [archiver (db)] Error from TOC entry 53399; 0 16528 TABLE
DATA l1afts_dbl sdmcleod
pg_restore: [archiver (db)] COPY failed for table "l1afts_dbl": ERROR:
unexpected data beyond EOF in block 10097 of relation
base/16386/17068
HINT: This has been seen to occur with buggy kernels; consider
updating your system.

23. Relational problems: Postgres on Gluster.
postgres source code: src/backend/storage/buffer/bufmgr.c
kubectl get
/*
* We get here only in the corner case where we are trying to extend
* the relation but we found a pre-existing buffer marked BM_VALID.
* This can happen because mdread doesn't complain about reads beyond
* EOF (when zero_damaged_pages is ON) and so a previous attempt to
* read a block beyond EOF could have left a "valid" zero-filled
* buffer. Unfortunately, we have also seen this case occurring
* because of buggy Linux kernels that sometimes return an
* lseek(SEEK_END) result that doesn't account for a recent write. In
* that situation, the pre-existing buffer would contain valid data
* that we don't want to overwrite. Since the legitimate case should
* always have left a zero-filled buffer, complain if not PageIsNew.
*/
bufBlock = isLocalBuf ? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr);
if (!PageIsNew((Page) bufBlock))
ereport(ERROR,
(errmsg("unexpected data beyond EOF in block %u of relation %s",
blockNum, relpath(smgr->smgr_rnode, forkNum)),
errhint("This has been seen to occur with buggy kernels; consider updating your system.")));

24. Relation problems: Containers drop
kubectl get

25. Relation problems: Communication.
kubectl getKubernetes Node
Network
Test (Pod)
DS
Kube DNS (service)
Kubernetes Master
Network
Test (Pod)
DS
Kube DNS (pod)
Kubernetes Node
Network
Test (Pod)
DS
Kube DNS (pod)

26. Kubernetes gives more benefits than doubts on premise
kubectl get
A lot of open source tools around
Helm packages
Fast delivery of software
Auto healing
Very very stable (Only got called out of bed once at night in 2017)
Happy developers
Enabler for DevOps
Etc..

27. Open source tools that boosts our relationship
kubectl get

28. Projects that boosts our relationship
kubectl get
Kubespray saved months of work setting up Kubernetes on premise.
Easily deploying production-ready Kubernetes clusters.

29. Projects that boosts our relationship
kubectl get
KUBEADM

30. Projects that boosts our relationship
kubectl get
Helm makes upgrading and maintaining our applications
predictable and super easy.
Package manager for Kubernetes

31. Love
kubectl get
Joost Milo

32. Questions?
kubectl get