Presentation about challenges with building a GitOps managed kubernetes platform for fleets of clusters.

1. No GitOps Pain, No Platform
Gain: Day 2 Challenges of
Managing Kubernetes Fleets
with GitOps
Cloud Native Rejekts
March 18th
2024

2. About me
Hi, I’m Łukasz Piątkowski, PhD
● Working as Platform Architect at
https://giantswarm.io
● 7+ years with Kubernetes in production
● Working with a team responsible for Developer
Experience and internal application
management platform
● Heavy GitOps user
● Tinkering with everything: electronics, home
automation, coffee machines and woodworking
Find me here (very limited ‘social media’ activity):
● https://tailored.cloud/
● https://twitter.com/piontec

3. Intro
Platform GitOps
Fleet
You are here

4. Intro
Platform GitOps
Fleet
You are here
PAIN

5. GitOps driven
platform
GitOps
Everything (configuration, deployments, environments,
…) has to come from a repository.
Platform
Make Dev and Ops folks lives easier by providing a set
of tools that allow them to build, configure, deploy and
manage applications easily™.
Focus on this

6. Fleet management

7. Fleet management

8. Git is cool

9. Not this. Let’s keep it out of scope…

10. What’s hard? And
why? Day 2 problems
● Repository layout
○ Security
○ Isolation
○ Extensibility - your customers want it too
○ DRY
● Migrations - moving stuff around
● Honorable mention
○ Automated upgrades
○ Understanding an impact of your change
○ Performance
○ Complexity
Core of your setup - has to be future-proof

11. How do we deal with it?

12. How do we deal with it?

13. Repository layout Three main approaches to configuration structure (for
example: ‘dev’, ‘staging’, ‘prod’)
● By directory
○ Good: 1 branch, consistent global changes
○ Bad: permissions, sharing with multiple tenants
● By branch
○ Good: the ‘dev’ branch is exactly what you have in
your ‘dev’ environment
○ Bad: permissions, doing the same change for
different environments (branches), change
propagation can be tricky
● By repo
○ Good: permissions, isolation
○ Bad: consistent changes

14. What’s our choice?

15. What’s our choice?

16. Why all? Our opinionated approach
● Multiple repos - to isolate tenants and to share a
common base (DRY)
● Multiple branches - to provide stuff like a feature
freeze
● Multiple directories - to manage clusters

17. Repos Structure
Can be public*

18. Benefits ● For us
○ DRY configuration
○ Isolation of customers’ installations
○ Exposing extension points
● For platform teams
○ Bases to start the platform with
○ Flexibility to add/modify/enforce own bases
● For developers
○ Easy onboarding and configuration of projects
across multiple clusters - the platform!
We can’t satisfy all the needs - customers’ platform team’s
knowledge is essential!
No need to start the platform from scratch

19. Tools Why we chose Flux?
● Solves some issues right away
○ Helps with security (secrets encryption)
○ Has auto update strategies
● Full native API
● Very well-designed
● Offers tools useful in structuring your repositories
○ Using remote repos
But let’s not focus on tools.

20. Example
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/giantswarm/management-cluster-bases/bases/provider/capa/flux-v2?ref=main
- https://github.com/giantswarm/management-cluster-bases/bases/provider/eks/helmrelease/?ref=main
- configmap-management-cluster-metadata.yaml
- age-secret-keys.yaml
patches:
- path:
https://raw.githubusercontent.com/giantswarm/management-cluster-bases/main/extras/vaultless/patch.yaml
- target: { kind: Kustomization, name: collection, namespace: flux-giantswarm }
path: ../../bases/patches/collection-patch-for-aac.yaml
- target: { kind: GitRepository, name: management-clusters-fleet }
patch: |-
- op: add
path: /metadata/annotations/repo-source
value: https://github.com/giantswarm/mcs
Repo: mcs; Branch: main; Path: management_clusters -> mc1 -> kustomization.yaml

21. What’s hard? And
why? Day 2 problems
● Repository layout
○ Security
○ Isolation
○ Extensibility - your customers want it too
○ DRY
● Migrations - moving stuff around
● Honorable mention
○ Automated upgrades
○ Understanding an impact of your change
○ Performance
○ Complexity
Core of your setup - has to be future-proof

22. More problems ● Refactoring and migrations
○ General approach
■ suspend target objects
■ if needed, suspend gitops controller
■ move stuff (across dirs, repos, branches)
■ resume everything in reverse order
○ Worked for us on multiple occasions

23. More problems ● Refactoring and migrations
○ General approach
■ suspend target objects
■ if needed, suspend gitops controller
■ move stuff (across dirs, repos, branches)
■ resume everything in reverse order
○ Worked for us on multiple occasions
● Pruning
○ Flux allows setting prune: True for K8s objects it
manages

24. Pruning
● Commit-1
● Commit-2
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://myrepo.github.com/awesome/bases/capi-cluster/capi-cluster.yaml
- https://myrepo.github.com/awesome/bases/apps/grafana.yaml
- ../bases/apps/external-secrets/operator
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://myrepo.github.com/awesome/bases/capi-cluster/capi-cluster.yml
- https://myrepo.github.com/awesome/bases/apps/grafana.yaml

25. Pruning

26. Pruning
● Commit-1
● Commit-2
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://myrepo.github.com/awesome/bases/capi-cluster/capi-cluster.yaml
- https://myrepo.github.com/awesome/bases/apps/grafana.yaml
- ../bases/apps/external-secrets/operator
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://myrepo.github.com/awesome/bases/capi-cluster/capi-cluster.yml
- https://myrepo.github.com/awesome/bases/apps/grafana.yaml

27. Actual footage of our test cluster named ‘grizzly’

28. What’s hard? And
why? Day 2 problems
● Repository layout
○ Security
○ Isolation
○ Extensibility - your customers want it too
○ DRY
● Migrations - moving stuff around
● Honorable mention
○ Automated upgrades
○ Understanding an impact of your change
○ Performance
○ Complexity
Core of your setup - has to be future-proof

29. Other problems ● It’s hard to tell what impact will a change in one of
the repositories have on the final configuration
○ You can render kustomizations locally
■ But remote references are a problem
○ We need better tools to preview and evaluate a
change
● Complexity can build up quickly: layers of
configuration, YAML over YAML patched with
YAML
● Performance: think twice before you set a really
low reconciliation period for hundreds of
resources
○ Overall, Flux’s performance is great, just don’t
overdo it

30. Summary ● Platforms are big and can be complex, but
remember: the end goal is to make (Dev)Ops
faster and happier
● GitOps is great
○ Your configuration git repo is your core: plan
carefully, think about your use cases
○ It’s easier to get started with a future-proof
structure than to do migrations on live production
systems
● Know your tools inside-out, so you know what’s
possible, what’s not, and how they really work

31. Links ● https://fluxcd.io/
● A set of examples for structuring more complex
GitOps repositories
○ https://github.com/giantswarm/gitops-template
● Our real bases used for managing the platform
○ https://github.com/giantswarm/management-clust
er-bases/
● Completely free Kubernetes cluster with flux to get
started with GitOps
○ https://github.com/piontec/free-oci-kubernetes

32. Any questions?