Running kubernetes in production can be difficult here is what we have learned.

1. Running Kubernetes
Adam Hamsik
DevOps/Cloud Engineer

2. www.pixelfederation.com
Running Kubernetes
TL;DR Summary
- Introduction
- Kubernetes @ PixelFederation
• Architecture
• Deployment
• Addons
- Day 2 Kubernetes operations
• Monitoring
• Troubleshooting

3. www.pixelfederation.com
Running Kubernetes
Introduction
We are game studio based in Slovakia developing
mobile games.
● Trainstation
● Diggy’s Adventure
● Seaport
● Trainstation 2
● Steamcats
● Emporea

4. www.pixelfederation.com
Running Kubernetes
deployment on AWS
Kubernetes @ Pixel
1. We manage our own Kubernetes clusters
a. When we started EKS was not available
1. Our clusters are deployed with Kops and Terraform
a. Currently we are running version 1.11 and planning upgrade to 1.12
1. Our applications are managed with Helm
1. We plan to deploy Istio service mesh soon

5. www.pixelfederation.com
Running Kubernetes
deployment on AWS
Using Kops for Kubernetes Deployment
1. Cloud Architecture
a. Kops can create cloud components or it can generate Terraform templates
b. Cluster nodes are deployed as immutable components
1. Highly available deployment
a. Masters/Nodes running in multiple AZs
b. ASGs used to make sure we can detect/heal from server failure
1. Cluster Updates
a. Rolling update for cluster without outage
i. User-data script is used for kubernetes installation

6. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes AWS architecture

7. www.pixelfederation.com
Running Kubernetes
Kubernetes Gotchas
1. Do not try to deploy everything into Kubernetes. Use managed services.
a. Aurora
b. Elasticache
c. Elasticsearch service
d. Managed Kafka
1. Amazon EBS can cause you troubles
a. EBS Volumes don’t work cross AZ
1. Choose the right instances for your Application
1. Cluster Autoscaler doesn’t understand AZ architecture

8. www.pixelfederation.com
Addons
extend core Kubernetes functionality (CustomResources)
1. External-dns
a. Create route53 dns entries based on kubernetes services
1. Cluster-autoscaler
a. Scale cluster nodes based on scheduler events
1. Prometheus operator
a. Monitor kubernetes cluster and applications running in it
Running Kubernetes
Kubernetes Addons

9. www.pixelfederation.com
1. Trust your health-checks
a. liveness / readiness
1. Stateless applications
a. Sharing data between pods complicates cluster architecture
b. Sticky sessions possible, but problematic
1. One process per container
a. Use sidecar containers if needed
1. Scaling
a. Scale apps with HPA (Add pods based on cpu/mem metrics)
b. Scale pods vertically with VPA (Not compatible with HPA)
c. Scale cluster with cluster-autoscaler (Add nodes based on scheduler events)
Running Kubernetes
Running applications on Kubernetes

10. www.pixelfederation.com
1. Use ingress controllers (traefik, nginx, haproxy, AWS ALB)
a. Cost effective solution for traffic proxy
b. Can forward different paths to different services
c. Use multiple ingress controllers for different types of traffic
1. Plan your Application updates, Kubernetes has native support for
a. Recreate
b. Rolling Update
c. Blue/Green, Canary, A/B needs to be scripted
Running Kubernetes
deployment on AWS
Running applications on Kubernetes vol. 2

11. www.pixelfederation.com
1. Nodes on AWS fail for any number of reasons make sure you can detect
that.
a. Node-problem-detector
b. Draino
1. Make sure you reserve some resources for node OS itself
1. Use scheduler hints (affinities/taints/tolerations) to spread your PODs
around cluster
1. TODO: Chaos engineering
Running Kubernetes
Kubernetes resiliency

12. www.pixelfederation.com
1. Prometheus is perfect opensource monitoring solution for Kubernetes
a. Consider running it outside of a cluster
1. Gather Kubernetes Logs/Events and make them visible
a. We use [1] to gather kubernetes events and push them to ElasticSearch
Cluster
1. Use Vault to manage application secrets inside cluster.
a. If porting application to vault is too expensive use init containers to fetch
secrets from vault and load them into application env
[1] https://github.com/haad/event-exporter
Running Kubernetes
Kubernetes Operations

13. www.pixelfederation.com
Running Kubernetes
And one more thing,
We are hiring and always looking for skilled
PHP/Game/DevOps engineers so check out our open
positions at [1].
[1] https://portal.pixelfederation.com/en/career

14. www.pixelfederation.com
Running Kubernetes
Thanks !
ahamsik@pixelfederation.com