This document discusses managing add-ons across Kubernetes clusters. It defines add-ons as standardized services like monitoring, logging, and security that need to be available in every cluster. While GitOps is useful for cluster operations, it has limitations for add-on management across multiple clusters. The document proposes using Kustomize along with a central secrets store like Vault to fully automate add-on deployment and updates. It suggests GitOps controllers can further streamline add-on management by applying configurations per target cluster. A demo is provided to illustrate how Kustomization files can be used to reproduce YAML configurations for add-on deployment on any cluster.

1. nirmata
Managing add-ons across clusters
CNCF Webinar – April 2021

2. nirmata
2
About Us
Damien Toledo
Co-founder and VP of Engineering
Nirmata, @damien_toledo
Ritesh Patel
Co-founder and VP of Products
Nirmata, @riteshdp

3. About Nirmata
🗹 Creators of Kyverno (now a CNCF project)
🗹 Delivering Day 2 management for Kubernetes
clusters and workloads
🗹 Active member of the CNCF and Kubernetes
community
🗹 Global production deployments at multiple
Fortune 1000 enterprises
3

4. 4
Agenda
• Inside a Kubernetes Cluster
• What are Add-ons
• GitOps
• Automating Add-on Management
• Demo

5. 5
What’s Inside Your Kubernetes Cluster?
Kubernetes Cluster
Core Services
Add-ons
Applications
Custom or third-party
applications
CNI, CSI, DNS, Ingress etc.
Monitoring, Logging, Security etc.
e.g. Vault agent, Datadog agent,
Prisma cloud, etc.

6. 6
What are Add-ons?
• Set of standardize services that need to be available in every cluster
e.g. Security, Monitoring, Logging, Backup, Secrets Management etc.
• Different teams may own these services
• Used by developers or central teams
• Operations and compliance requirements
• Require on-going managent to ensure availability (monitoring,
upgrades, updates etc.)

7. 7
GitOps
Cluster Operations using Git
Ops managed and performed in a
declarative manner with Git as the
“source-of-truth” system.
Benefits:
• Single source of truth
• Developer self-service
• Observability
• Disaster Recovery
GitOps
Controller
git push

8. 8
Limitations of GitOps
• Not designed for automatic updates
• Proliferation of Git repositories/branches
• Doesn’t solve centralized
secret management
• Lack of visibility
• No validation
https://blog.container-solutions.com/gitops-limitations

9. 9
GitOps Controller
• Responsible to applying Git changes to the cluster
• Can address multi-cluster deployments by applying Kustomizations
per target
• Provides visibility and state
• Enables advanced progressive delivery workflows
o Approvals
o Rollbacks
o Etc…

10. 10
Automating Add-On Management
• Reducing the number of Git repositories/branches
o Using `kustomize`
o Selecting target-based kustomization
• Centralized secret management for ids, tokens, certificates, licenses
etc.
o Using Vault as central secrets store
o Using Vault agent injector to dynamically inject secrets
o Using target-based Kustomization to configure specific labels/annotations
• Easy to reproduce final YAMLs for any target using Kustomize

11. 11
Kustomization files

12. Demo

13. 13
Summary
• GitOps has become a preferred approach for continuous delivery of
Kubernetes workloads and add-ons
• GitOps has limitations especially when using it to deploy across
multiple clusters (e.g. deploying add-ons)
• Kustomize along with a central secret management solution can be
used to fully automate add-on management
• GitOps controllers like Nirmata, Fleet, etc. streamline add-on
management enabling Clusters-as-a-Service for enterprises

14. Thank you!