Lessons learned with kubernetes in productionat PlayPass, presented at the 6th Docker Birthday Meetup in Antwerpen. What went well and what are some open issues. Also, we discussed some security measures after the presentations.

1. Lessons learned with
Kubernetes in production
at PlayPass
Peter Vandenabeele
19 March 2019

2. PlayPass
● Antwerp scale-up
● Cashless payments
● Access control
● NFC wristband
● world-wide
● Music Festivals
● Sports Events

3. New apps: new tech stack
● existing apps: pets, Ruby/Rails, postgresql, classic stack
● Docker for development (docker-compose, ruby, elixir)
● Kubernetes for deployment, for HA
● on GKE (Google Kubernetes Engine)
● Gitlab for CI/CD
● terraform for base infra (clusters, networking, firewall)
● kubectl and helm for app deployment
● Vault for secrets management
● Stackdriver and DataDog for logging and monitoring

4. Environments (GCP projects + namespaces)
k8s-dev
(pre-empt)
edge
(pre-empt)
production
(fixed VM’s)
clusterapplications
(default)
play
experimental
apps
(default)
play
gitlab runners
dashboard
new apps ...
(default)
play
-gitlab runners
dashboard
new apps ...

5. Kubernetes => pet infrastructure firewalling
RMQ
firewall:
only
ingress
from
fixed
IP
node 1 (preemptible)
new public IP on preempt.
node 2 (preemptible)
new public IP on preempt.
node 3 (preemptible)
new public IP on preempt.
“cloud-native” “pet infra”

6. Use a NAT GW, but, not for all traffic

7. NAT
GW
Kubernetes => NAT GW => outside
node 1 (preemptible)
new public IP on preempt.
node 2 (preemptible)
new public IP on preempt.
node 3 (preemptible)
new public IP on preempt.
fixed IP
to outside
default
GW
GKE
master
ssh
jump
host
800
700

8. Node pre-emption and upgrades (planned events)
● Is GCP LB aware upfront of nodes going down ??
● Are nodes doing proper `drain` on pre-emption ??
● Are nodes hanging in `cordoned` state after upgrade ??
● Node pre-emption => short service interruptions
● Production => non-preemptible nodes $$$
● Node upgrade => short service interruption, manual
uncordon needed (see next slide)

9. Node upgrade
from GKE GUI
leaves most
nodes in
SchedulingDisabled
state
=> manual uncordon

10. Cloud SQL postgres performance issues
● Try to be “servicefull”, obviously :-)
● postgres service with HA, backups, auto scaling of disk
● basic performance test revealed that a workload with a
few INSERT’s per transaction, dropped to only 20
INSERT’s per second and started disconnecting cloudsql-
proxy
● => reverted 1 “fast aggregate” to self-hosted postgres,
using Docker (back-up, HA => rebuild the aggregate)

11. the good news: fast e2e deploy by developer

12. Extra security measures as discussed
● Kaniko for building images without “docker-in-docker” or privileged mode
● PodSecurityPolicy with non privileged and MustRunAsNonRoot
○ spec:
○ privileged: false
○ allowPrivilegeEscalation: false
○ runAsUser:
○ # Require the container to run without root privileges.
○ rule: 'MustRunAsNonRoot'
● “Private” cluster with NAT for external access
● RBAC on different namespaces
● tiller per namespace with limited roleBinding and `--listen=localhost:44134`
(ref https://engineering.bitnami.com/articles/helm-security.html)