This document discusses Kubernetes day 2 operations at Pixel Federation. It introduces Pixel Federation and provides an overview of their infrastructure including Kubernetes clusters, AWS services, and self-managed services. It then covers various aspects of day 2 operations with Kubernetes including CI/CD, observability, and FinOps.

1. Kubernetes Day 2
Jozef Halgas
DevOps Engineer
1

2. www.pixelfederation.com
Kubernetes Day 2
Summary
● Pixel Federation introduction
● Infrastructure overview
● Day 2 operations
○ Kubernetes
○ CI/CD
○ Observability
○ FinOps
2

3. www.pixelfederation.com
Kubernetes Day 2
Pixel Federation
We develop and publish free-to-play games, which are available on Android
and iOS mobile devices, Facebook and on our PIXEL portal site.
● Trainstation
● Emporea
● Diggy’s Adventure
● Seaport
● Trainstation 2
● Puzzle Adventure
● Port City
3

4. www.pixelfederation.com
Kubernetes Day 2
App architecture
● CDN for high availability and
performance
● backend on self managed K8s
cluster
● AWS managed services
● self managed services
4

5. www.pixelfederation.com
Kubernetes Day 2
Infrastructure
5
● Kubernetes clusters
○ kops/EKS
○ sandbox/dev/prod
● AWS services
○ RDS, ECR, ES,
ElastiCache,MSK…
○ dev/prod
● self managed services
○ EC2/on-premise datacenter
○ dev/prod

6. www.pixelfederation.com
Kubernetes Day 2
Cluster utilization
6

7. www.pixelfederation.com
Kubernetes Day 2
Kubernetes
● self managed vs managed K8s (kops, EKS, AKS, GKE),
● managed services and K8s operators (CRD and Operator Controller,
Thanos, Jenkins),
● dedicated resources (instance groups, ingress etc.),
● resource management
(requests and limits),
● liveness, readiness and startup probes,
● right instances selection
(c,m,r, credit balance),
● volume types (gp2, gp3, efs),
● etcd keyspace size (storage size),
7

8. www.pixelfederation.com
Kubernetes Day 2
Kubernetes
● timeouts, retries, circuit breakers, exponential backoff,
● CA doesn’t understand AZ architecture, Safe-to-evict,
● PodDistributionBudget, podAntiAffinity,
● AWS ASG rebalance policy disabled,
● Amazon EBS can cause you troubles,
● EBS Volumes don’t work cross AZ,
● AWS organizations,
● container security, CDN,
● documentation for developers,
● unattended upgrades,
● registry as a pull through cache,
● load testing, test everything,
8

9. www.pixelfederation.com
Kubernetes Day 2
CI/CD
● git as single source of true,
● base images,
● dynamic environments,
● NFS,
● Helm, Helmfile, Terraform, asdf,
● rolling update, blue green,
canary
● possibility to rollback,
● small reusable pipelines,
● Jenkins shared library,
● maintenance mode,
9

10. www.pixelfederation.com
Kubernetes Day 2
Observability
● Prometheus/Thanos,
● Alert Noise reduction,
○ dedicated channels,
○ onCall reduction,
○ alerts check (labels
change),
● Grafana,
○ tell a story (deploy
annotation),
○ reusable dashboards,
○ links in dashboards,
● troubleshoot manual,
10
● ELK, Fluentbit, Fluentd/Vector
● HPA with Prometheus,
● tcpdump, strace…

11. www.pixelfederation.com
Kubernetes Day 2
FinOps
● reserved instances (EC2,DB)
(don't try save too much),
● saving plans,
● spot instances,
● resource tagging,
● cluster overprovisioner,
● FinOps tools,
11

12. www.pixelfederation.com
Kubernetes Day 2
Q & A
PIXEL EVENTS
Herná komunita chce teba
meetups.wtf
portal.pixelfederation.com/en/career
jhalgas@pixelfederation.com
linkedin.com/in/jozefhalgas
12