How Vinted uses GitOps to deploy thousands of pods to Kubernetes

1. How Vinted uses GitOps to
deploy thousands of pods to
Kubernetes
Edgaras Apšega
Site Reliability Engineer / CNCF Ambassador
@apsega

2. Vinted infrastructure
3x growth over 2 years Kubernetes production stats
● All services running on Kubernetes*
● 2000 deployments per day
● 10k+ running pods
● 480+ physical nodes (70k CPU cores; 190TB memory)
*Almost

3. What’s GitOps, anyway?

4. GitOps
Coined in August 2017 by Weaveworks CEO Alexis Richardson
GitOps is a DevOps process characterized by:
Best practices of deployment, management and monitoring of containerized
applications
Experience for managing applications with fully automated pipelines/workflows using
Git for development and operations
Use of Git revision control system to track and approve changes to the infrastructure

5. Why GitOps?
Infrastructure as a Code (IaC)
Repeatability
Reliability
Efficiency
Visibility
Self-service / Internal Developer Platform (Backstage)
Code reviews

6. Imperative vs. Declarative
Imperative Declarative
$ kubectl run busybox --image=busybox:1.29
--restart=Never --command -- sleep 3600
apiVersion: v1
kind: Pod
metadata:
labels:
run: busybox
name: busybox
spec:
containers:
- command:
- sleep
- "3600"
image: busybox:1.29
imagePullPolicy: Always
name: busybox

7. Build
Docker
push
Test
Git commit
and push
Git clone
config repo
Update
manifests
kubectl
apply
Git clone
config repo
Discover
manifests
GitOps continuous integration
GitOps continuous deployment

8. Git strategies
Single branch (multiple directories) Multiple branches
Do use directories for GitOps environments
kubernetes-deployments
├── demo-app
│ ├── development
│ ├── production
│ └── staging
└── guestbook
├── development
├── production
└── staging
Don’t use long-running branches for GitOps
environments

9. Environment promotion
Telefonistka Kargo

10. Configuration management
Helm Kustomize
Package manager
Go templating language
Environments per values files
No parameters and templates - as
close as you can get to Kubernetes
manifests
No parameters and templates -
limiting in edge cases
Overlays per environment
More a language than a tool
JSON with comments and
templating
Not Kubernetes specific
Jsonnet

11. Vinted case study:
GitOps

12. ArgoCD App of Apps Pattern
App of Apps manifest ArgoCD custom plugins

13. Kubernetes deployments
Kubernetes deployments directory Application manifests
kubernetes-deployments
├── guestbook
│ ├── frontend
│ │ ├── development-values.yaml
│ │ ├── development-application.yaml
│ │ ├── staging-values.yaml
│ │ ├── staging-application.yaml
│ │ ├── production-values.yaml
│ │ └── production-application.yaml
│ ├── backend
│ │ ├── development-values.yaml
│ │ ├── development-application.yaml
│ │ ├── staging-values.yaml
│ │ ├── staging-application.yaml
│ │ ├── production-values.yaml
│ │ └── production-application.yaml
│ └── common-values.yaml
└── demo-app
├── development-values.yaml
├── development-application.yaml
├── staging-values.yaml
├── staging-application.yaml
├── production-values.yaml
└── production-application.yaml

14. Helm charts
Centralized Helm charts Helm chart values files

15. Helm charts (2)
Centralized Helm charts
Helm chart values files in Kubernetes
deployments repository

16. ArgoCD application view

17. Change image tag in deployments repo
Jenkinsfile in code repo
git clone
yq write
--inplace "${params.ENVIRONMENT}-values.yaml"
--tag '!!str' image.tag "${params.IMAGE_TAG}"
git push
stage('Build') {
when {
branch 'master'
}
steps {
echo 'Build and push Docker image'
DockerImageBuildAndPublish('frontend')
}
}
stage('Deploy App to Kubernetes') {
when {
branch 'master'
}
steps {
KubernetesDeploymentsApply('frontend',
'production')
}
}
Jenkins update image tag action

18. ArgoCD Sync waves
Supports application dependencies and defines
deployments order
Supports only definitions within same defined application
Prolongs deployments
Use when doing frequent deployments
argocd app sync $(APP_NAME)
--revision ${ARGOCD_APP_REVISION}
Sync wave example
Sync waves

19. Bonus: scaling ArgoCD
for high load

20. ArgoCD architecture
argocd-api-server
argocd-api-server
argocd-api-server
argocd-api-server
argocd-api-server
argocd-application-controller
argocd-api-server
argocd-api-server
argocd-repo-server Git
Redis
Visualize
Changes
Apply
changes
Retrieve
manifests

21. Application controller
Too many applications:
increase number of processes
Too many clusters:
shard application controller (increase replicas)

22. Typically running 3-4 servers is enough
reposerver.parallelism.limit
Cache with webhooks
Repo server

23. GitOps is awesome (while GitHub/GitLab works)
Use directories for GitOps environments
Separate code and configuration repositories
Use common values for repetitive configuration
Avoid using cross application dependencies

24. Thank you!
@apsega