Cicd pixelfederation

CI/CD with Jenkins in
Kubernetes
Jozef Halgas
Adam Hamsik
DevOps Engineers

www.pixelfederation.com
CI/CD with Jenkins in Kubernetes
Introduction
We are game studio based in Slovakia developing free
to play mobile games.
● Trainstation
● Diggy’s Adventure
● Seaport
● Trainstation 2
● AFK Cats
● Emporea

TL;DR Summary
● Introduction
● Infrastructure overview
● Jenkins overview
● Pipeline/environment overview
● TS2 workflow and architecture
● TS2 pipeline
● Dynamic environments
● Jenkins shared library
● Base images

Infrastructure
● Kubernetes clusters
○ kops
○ sandbox/dev/prod
● AWS services
○ RDS, ECR, ES, ElastiCache...
○ Terraform
○ dev/prod
● self managed services
○ EC2/on-premise datacenter
○ Terraform/Ansible
○ dev/prod

Kubernetes cluster

Jenkins architecture
● runs in K8s
● K8s plugin
● dedicated autoscaling group
● 1-4 nodes
● out of office hour just 1 node

Jenkins k8s pod templates
● containers:
○ jnlp-slave/dind/util/php...
● pod template inheritance
● empty dir volumes:
○ sharing data
● persistent volumes
○ cache idea
○ speed up builds

Jenkins configuration
● template’s containers pipelines
○ configuration in git
○ versioning
○ Jenkins job
● job per project and environment
○ DEVs/TEST/PROD/SANDBOX( Jenkinsfile testing)
○ dev/test/prod jobs are all prod from Jenkinsfile point of view
○ Jenkinsfile source branches
● jobs configuration in Jenkinsfiles
○ stored in project repo, 2 Jenkinsfiles (dev/prod)
● Jenkins shared library

Env requirements / Pipeline overview
● Docker base image
● Helm chart
● env values file
● Vault
● databases (SQL,
Redis, db users)
● env namespace
● Logstash
● ELK
● monitoring
● DNS name
● Jenkins job
● CDN

TS2 architecture
● client
● backend
● CDN (static files)
● separate pipeline for backend
and CDN
● customized base images
● Helm

TS2 workflow
● local development environment
● automated integration test
(postman collections)
● code reviews, tests with client
● feature branches/dev/master
● dynamic dev environments
● app versioning (prod vs dev)
● blue-green deployment
(integration
test,liveness/readiness probe)
● maintenance mode

TS2 build parameters

TS2 pipeline

Dynamic environments
Static dev environments
● create an env requires lot of
steps
● labor intensive to keep all
envs in sync
● switching features in few envs
● db migration issues

Dynamic environments
Dynamic dev environments
● env per feature/branch
● shared infrastructure
● resource, labor friendly
● makes workflow more transparent
● sandbox job

Dynamic environments requirements
● specific env vars set dynamically based on branch name
during Jenkins build (url, db, logger_app_name...)
● dynamic db creation
● keep env vars in Helm values file as dictionary
● Helm chart dict to list transformation
● cleanup procedure

Jenkins shared library
Before:
● At least one Jenkinsfile per
project with over 500 lines
● Hard to maintain
● With more projects making
changes becoming harder and
harder
● Maintaining at least 60
Pipelines
● Developers maintain their
Pipelines, but DevOps needs
to help them.
● Developing scripted pipelines
is not easy.
● Troubleshooting/Debuging
takes experience

After:
● One shared library used in all
Jenkins Pipeline files across
company
● We provide easy to use blocks to
build full Pipeline with
● Written in Jenkins Groovy
● Development/Release
management done by DevOps
Team
Design Patterns:
● Unifty repeatable code
● Declarative style
● Better readability
● Let developers deal with
application specific code and
hide infrastructure details
from them.

Example Pipeline

Docker Base images
Before:
● Every application had own docker
image.
● Hard to maintain
● With more projects making
changes becoming harder and
harder
● Huge variety of php versions/
container varietes
● Developers maintain their
application docker images
they are using them with local
docker-compose environment
● Deploying images built from
same source to productions
as we deploy to local env

Docker Base images
After:
● One shared docker image for every
kubernetes container
○ PHP
○ NGINX
○ FileBeat
● Development/Release
management and done by
DevOps Team
● Easier to push required changes to
development teams.
● Dockerfile syntax
● Runtime configuration done
with environment variables
and confd
● Release management done in
git tags
● Simplifies Application
Dockerfiles

Docker Base images
Once git tag is pushed we do following:
● Parse variables from tag
○ Upstream git tag
○ Our build version
● Generate required Dockerfiles
○ We have our own templating
engine
● Build images and tag them with
required versions
php-fpm (master*) $ git tag -l
7.0.33-fpm-stretch_18
7.3.9-fpm-stretch_3
7.4.0beta2-fpm-buster_16
7.4.0beta2-fpm-buster_17

Docker Base images

Questions?
jhalgas@pixelfederation.com
ahamsik@pixelfederation.com
linkedin.com/in/jozefhalgas
linkedin.com/in/adam-hamsik/

Cicd pixelfederation

More Related Content

What's hot

Similar to Cicd pixelfederation

More from Juraj Hantak

Recently uploaded

Cicd pixelfederation