Kubernetes Day 2
Jozef Halgas
DevOps Engineer
1
www.pixelfederation.com
Kubernetes Day 2
Summary
● Pixel Federation introduction
● Infrastructure overview
● Day 2 operations
○ Kubernetes
○ CI/CD
○ Observability
○ FinOps
2
www.pixelfederation.com
Kubernetes Day 2
Pixel Federation
We develop and publish free-to-play games, which are available on Android
and iOS mobile devices, Facebook and on our PIXEL portal site.
● Trainstation
● Emporea
● Diggy’s Adventure
● Seaport
● Trainstation 2
● Puzzle Adventure
● Port City
3
www.pixelfederation.com
Kubernetes Day 2
App architecture
● CDN for high availability and
performance
● backend on self managed K8s
cluster
● AWS managed services
● self managed services
4
www.pixelfederation.com
Kubernetes Day 2
Infrastructure
5
● Kubernetes clusters
○ kops/EKS
○ sandbox/dev/prod
● AWS services
○ RDS, ECR, ES,
ElastiCache,MSK…
○ dev/prod
● self managed services
○ EC2/on-premise datacenter
○ dev/prod
www.pixelfederation.com
Kubernetes Day 2
Cluster utilization
6
www.pixelfederation.com
Kubernetes Day 2
Kubernetes
● self managed vs managed K8s (kops, EKS, AKS, GKE),
● managed services and K8s operators (CRD and Operator Controller,
Thanos, Jenkins),
● dedicated resources (instance groups, ingress etc.),
● resource management
(requests and limits),
● liveness, readiness and startup probes,
● right instances selection
(c,m,r, credit balance),
● volume types (gp2, gp3, efs),
● etcd keyspace size (storage size),
7
www.pixelfederation.com
Kubernetes Day 2
Kubernetes
● timeouts, retries, circuit breakers, exponential backoff,
● CA doesn’t understand AZ architecture, Safe-to-evict,
● PodDistributionBudget, podAntiAffinity,
● AWS ASG rebalance policy disabled,
● Amazon EBS can cause you troubles,
● EBS Volumes don’t work cross AZ,
● AWS organizations,
● container security, CDN,
● documentation for developers,
● unattended upgrades,
● registry as a pull through cache,
● load testing, test everything,
8
www.pixelfederation.com
Kubernetes Day 2
CI/CD
● git as single source of true,
● base images,
● dynamic environments,
● NFS,
● Helm, Helmfile, Terraform, asdf,
● rolling update, blue green,
canary
● possibility to rollback,
● small reusable pipelines,
● Jenkins shared library,
● maintenance mode,
9
www.pixelfederation.com
Kubernetes Day 2
Observability
● Prometheus/Thanos,
● Alert Noise reduction,
○ dedicated channels,
○ onCall reduction,
○ alerts check (labels
change),
● Grafana,
○ tell a story (deploy
annotation),
○ reusable dashboards,
○ links in dashboards,
● troubleshoot manual,
10
● ELK, Fluentbit, Fluentd/Vector
● HPA with Prometheus,
● tcpdump, strace…
www.pixelfederation.com
Kubernetes Day 2
FinOps
● reserved instances (EC2,DB)
(don't try save too much),
● saving plans,
● spot instances,
● resource tagging,
● cluster overprovisioner,
● FinOps tools,
11
www.pixelfederation.com
Kubernetes Day 2
Q & A
PIXEL EVENTS
Herná komunita chce teba
meetups.wtf
portal.pixelfederation.com/en/career
jhalgas@pixelfederation.com
linkedin.com/in/jozefhalgas
12

Kubernetes day 2_jozef_halgas_pf

  • 1.
    Kubernetes Day 2 JozefHalgas DevOps Engineer 1
  • 2.
    www.pixelfederation.com Kubernetes Day 2 Summary ●Pixel Federation introduction ● Infrastructure overview ● Day 2 operations ○ Kubernetes ○ CI/CD ○ Observability ○ FinOps 2
  • 3.
    www.pixelfederation.com Kubernetes Day 2 PixelFederation We develop and publish free-to-play games, which are available on Android and iOS mobile devices, Facebook and on our PIXEL portal site. ● Trainstation ● Emporea ● Diggy’s Adventure ● Seaport ● Trainstation 2 ● Puzzle Adventure ● Port City 3
  • 4.
    www.pixelfederation.com Kubernetes Day 2 Apparchitecture ● CDN for high availability and performance ● backend on self managed K8s cluster ● AWS managed services ● self managed services 4
  • 5.
    www.pixelfederation.com Kubernetes Day 2 Infrastructure 5 ●Kubernetes clusters ○ kops/EKS ○ sandbox/dev/prod ● AWS services ○ RDS, ECR, ES, ElastiCache,MSK… ○ dev/prod ● self managed services ○ EC2/on-premise datacenter ○ dev/prod
  • 6.
  • 7.
    www.pixelfederation.com Kubernetes Day 2 Kubernetes ●self managed vs managed K8s (kops, EKS, AKS, GKE), ● managed services and K8s operators (CRD and Operator Controller, Thanos, Jenkins), ● dedicated resources (instance groups, ingress etc.), ● resource management (requests and limits), ● liveness, readiness and startup probes, ● right instances selection (c,m,r, credit balance), ● volume types (gp2, gp3, efs), ● etcd keyspace size (storage size), 7
  • 8.
    www.pixelfederation.com Kubernetes Day 2 Kubernetes ●timeouts, retries, circuit breakers, exponential backoff, ● CA doesn’t understand AZ architecture, Safe-to-evict, ● PodDistributionBudget, podAntiAffinity, ● AWS ASG rebalance policy disabled, ● Amazon EBS can cause you troubles, ● EBS Volumes don’t work cross AZ, ● AWS organizations, ● container security, CDN, ● documentation for developers, ● unattended upgrades, ● registry as a pull through cache, ● load testing, test everything, 8
  • 9.
    www.pixelfederation.com Kubernetes Day 2 CI/CD ●git as single source of true, ● base images, ● dynamic environments, ● NFS, ● Helm, Helmfile, Terraform, asdf, ● rolling update, blue green, canary ● possibility to rollback, ● small reusable pipelines, ● Jenkins shared library, ● maintenance mode, 9
  • 10.
    www.pixelfederation.com Kubernetes Day 2 Observability ●Prometheus/Thanos, ● Alert Noise reduction, ○ dedicated channels, ○ onCall reduction, ○ alerts check (labels change), ● Grafana, ○ tell a story (deploy annotation), ○ reusable dashboards, ○ links in dashboards, ● troubleshoot manual, 10 ● ELK, Fluentbit, Fluentd/Vector ● HPA with Prometheus, ● tcpdump, strace…
  • 11.
    www.pixelfederation.com Kubernetes Day 2 FinOps ●reserved instances (EC2,DB) (don't try save too much), ● saving plans, ● spot instances, ● resource tagging, ● cluster overprovisioner, ● FinOps tools, 11
  • 12.
    www.pixelfederation.com Kubernetes Day 2 Q& A PIXEL EVENTS Herná komunita chce teba meetups.wtf portal.pixelfederation.com/en/career jhalgas@pixelfederation.com linkedin.com/in/jozefhalgas 12