1
1
Implementing Flux for Scale
with Soft Multi-tenancy
Russ Parmer
Senior Engineer, Weaveworks
Priyanka Ravi
Developer Experience Engineer, Weaveworks
2
2
Weaveworks is founded on open source
● Flux & Flagger (CNCF): GitOps and Progressive Delivery for k8s
● Weave GitOps: A powerful extension & web UI for Flux
● Weave GitOps Terraform Controller: Flux controller for Terraform
resources
● GitOps Tools for Flux: VS Code extension
weave.works
3
3
● Operating model for cloud native applications such as Kubernetes
● Utilizes a version controlled system (Commonly Git) as the “single
source of truth”
● Enables continuous delivery through automated deployment,
monitoring, and management by a version controlled system
● Managing your infrastructure and applications declaratively
What is GitOps
Source: GitOps Working Group
https://opengitops.dev/
5
5
Individuals, teams, and organizations who implement GitOps experience
many benefits, including:
● Stronger Security Guarantees
● Increased Developer & Operational Productivity
● Enhanced Developer Experience
● Improved Stability
● Higher Reliability
● Consistency and Standardization
Why GitOps
6
6
● A git centric package manager for your applications
● A set of continuous and progressive delivery solutions for Kubernetes
What is Flux
fluxcd.io
7
7
🤝 Flux provides GitOps for both apps and infrastructure
🤖 Just push to Git and Flux does the rest
🔩 Flux works with your existing tools
☸ Flux works with any Kubernetes and all common Kubernetes tooling
🤹Flux does Multi-Tenancy (and “Multi-everything”)
📞 Flux alerts and notifies
👍 Users trust Flux
💖 Flux has a lovely community that is very easy to work with!
Flux in Short fluxcd.io
8
8
● Reduces developer burden
● Extensible
● Comes with out of the box support for Kustomize and Helm
● Designed For Kubernetes
Benefits of Flux
fluxcd.io
9
9
Overview of Flux
Source
controller
Notification
Controller
Image Reflector
& Automation
Controller
Flux
Flux is a set of
Kubernetes Controllers
fluxcd.io
Terraform
Controller
Helm
Controller
Kustomize
controller
VS Code
Extension
10
10
What Flux’s Controllers do
Source Controller
- Fetch resources and store as
artifacts
Kustomize Controller
- Apply manifests, Run
manifest generation using
kustomize
Helm Controller
- Deployment of Helm Charts
Notification Controller
- Notification Dispatch
Image Reflector Controller
- Reflects Image metadata for
Automation Controller
Image Automation Controller
- Updates YAML when new
container images are
available
fluxcd.io
11
11
● Helm
● Kustomize
● Prometheus
● Grafana
● Jenkins
● EKS
● AKS
● GCP
Flux Works with Other Tools
● Traefik
● Falco
● GitHub, GitLab, Bitbucket,
s3-compatible buckets
● Terraform
● …and more!!!
fluxcd.io
12
12
● Makes life easier
● Multi-tenancy
● DependsOn
● Helm integration
● Notifications and Alerts
● Bootstrap
● Flux CLI
Reasons I and Others Love Flux fluxcd.io
13
13
● There are two different forms of multi-tenancy
○ Hard multi-tenancy
■ Every tenant has their own cluster
○ Soft multi-tenancy
■ A cluster is shared across many different tenants
■ Tenants need to be isolated
What is Multi-Tenancy
14
14
● Multi-tenancy lock down*
○ Ensure relevant controllers have cross namespace references disabled via
`--no-cross-namespace-refs=true`
● Resource Isolation
○ Ensure additional Flux instances are deployed when mission critical
tenants/workloads must be assured.
● Node Isolation
○ Ensure worker nodes are not being shared across tenants and the Flux
components.
● Network Isolation
○ Ensure the Container Network Interface (CNI) being used in the cluster
supports Network Policies.
Best Practices for Multi-Tenancy
15
15
● Add `--no-cross-namespace-refs=true`
flag
○ Allow Flux to only reconcile Flux
resources that exist in the same
namespace
● Add `--default-service-account=default`
flag
○ Default Kustomization and HelmRelease
objects to not use the cluster scoped
service account
● Set the flux-system Kustomization
resource to use correct service account
○ This still needs cluster level access and
should not use the ‘default’ service
account
Enable Tenant Isolation for Flux
16
Confidential do not distribute 16
Demo Time!
17
17
Next Steps & Resources
● Try it yourself! Flux Docs: fluxcd.io/flux
○ Flux Multi-cluster setup:
■ https://fluxcd.io/flux/get-started/#multi-cluster-setup
■ https://github.com/fluxcd/flux2-kustomize-helm-example
○ Additional Best Practices for Shared Cluster Multi-tenancy:
https://fluxcd.io/flux/security/best-practices/#additional-best-practices-for
-shared-cluster-multi-tenancy
○ Flux Bootsrap Cheatsheet: https://fluxcd.io/flux/cheatsheets/bootstrap
● Kyverno Docs to Generate Flux Multi-tenant Resources
○ https://kyverno.io/policies/flux/generate-flux-multi-tenant-resources/gene
rate-flux-multi-tenant-resources/
18
Confidential do not distribute 18
weave.works
Thank you

Implementing Flux for Scale with Soft Multi-tenancy

  • 1.
    1 1 Implementing Flux forScale with Soft Multi-tenancy Russ Parmer Senior Engineer, Weaveworks Priyanka Ravi Developer Experience Engineer, Weaveworks
  • 2.
    2 2 Weaveworks is foundedon open source ● Flux & Flagger (CNCF): GitOps and Progressive Delivery for k8s ● Weave GitOps: A powerful extension & web UI for Flux ● Weave GitOps Terraform Controller: Flux controller for Terraform resources ● GitOps Tools for Flux: VS Code extension weave.works
  • 3.
    3 3 ● Operating modelfor cloud native applications such as Kubernetes ● Utilizes a version controlled system (Commonly Git) as the “single source of truth” ● Enables continuous delivery through automated deployment, monitoring, and management by a version controlled system ● Managing your infrastructure and applications declaratively What is GitOps
  • 4.
    Source: GitOps WorkingGroup https://opengitops.dev/
  • 5.
    5 5 Individuals, teams, andorganizations who implement GitOps experience many benefits, including: ● Stronger Security Guarantees ● Increased Developer & Operational Productivity ● Enhanced Developer Experience ● Improved Stability ● Higher Reliability ● Consistency and Standardization Why GitOps
  • 6.
    6 6 ● A gitcentric package manager for your applications ● A set of continuous and progressive delivery solutions for Kubernetes What is Flux fluxcd.io
  • 7.
    7 7 🤝 Flux providesGitOps for both apps and infrastructure 🤖 Just push to Git and Flux does the rest 🔩 Flux works with your existing tools ☸ Flux works with any Kubernetes and all common Kubernetes tooling 🤹Flux does Multi-Tenancy (and “Multi-everything”) 📞 Flux alerts and notifies 👍 Users trust Flux 💖 Flux has a lovely community that is very easy to work with! Flux in Short fluxcd.io
  • 8.
    8 8 ● Reduces developerburden ● Extensible ● Comes with out of the box support for Kustomize and Helm ● Designed For Kubernetes Benefits of Flux fluxcd.io
  • 9.
    9 9 Overview of Flux Source controller Notification Controller ImageReflector & Automation Controller Flux Flux is a set of Kubernetes Controllers fluxcd.io Terraform Controller Helm Controller Kustomize controller VS Code Extension
  • 10.
    10 10 What Flux’s Controllersdo Source Controller - Fetch resources and store as artifacts Kustomize Controller - Apply manifests, Run manifest generation using kustomize Helm Controller - Deployment of Helm Charts Notification Controller - Notification Dispatch Image Reflector Controller - Reflects Image metadata for Automation Controller Image Automation Controller - Updates YAML when new container images are available fluxcd.io
  • 11.
    11 11 ● Helm ● Kustomize ●Prometheus ● Grafana ● Jenkins ● EKS ● AKS ● GCP Flux Works with Other Tools ● Traefik ● Falco ● GitHub, GitLab, Bitbucket, s3-compatible buckets ● Terraform ● …and more!!! fluxcd.io
  • 12.
    12 12 ● Makes lifeeasier ● Multi-tenancy ● DependsOn ● Helm integration ● Notifications and Alerts ● Bootstrap ● Flux CLI Reasons I and Others Love Flux fluxcd.io
  • 13.
    13 13 ● There aretwo different forms of multi-tenancy ○ Hard multi-tenancy ■ Every tenant has their own cluster ○ Soft multi-tenancy ■ A cluster is shared across many different tenants ■ Tenants need to be isolated What is Multi-Tenancy
  • 14.
    14 14 ● Multi-tenancy lockdown* ○ Ensure relevant controllers have cross namespace references disabled via `--no-cross-namespace-refs=true` ● Resource Isolation ○ Ensure additional Flux instances are deployed when mission critical tenants/workloads must be assured. ● Node Isolation ○ Ensure worker nodes are not being shared across tenants and the Flux components. ● Network Isolation ○ Ensure the Container Network Interface (CNI) being used in the cluster supports Network Policies. Best Practices for Multi-Tenancy
  • 15.
    15 15 ● Add `--no-cross-namespace-refs=true` flag ○Allow Flux to only reconcile Flux resources that exist in the same namespace ● Add `--default-service-account=default` flag ○ Default Kustomization and HelmRelease objects to not use the cluster scoped service account ● Set the flux-system Kustomization resource to use correct service account ○ This still needs cluster level access and should not use the ‘default’ service account Enable Tenant Isolation for Flux
  • 16.
    16 Confidential do notdistribute 16 Demo Time!
  • 17.
    17 17 Next Steps &Resources ● Try it yourself! Flux Docs: fluxcd.io/flux ○ Flux Multi-cluster setup: ■ https://fluxcd.io/flux/get-started/#multi-cluster-setup ■ https://github.com/fluxcd/flux2-kustomize-helm-example ○ Additional Best Practices for Shared Cluster Multi-tenancy: https://fluxcd.io/flux/security/best-practices/#additional-best-practices-for -shared-cluster-multi-tenancy ○ Flux Bootsrap Cheatsheet: https://fluxcd.io/flux/cheatsheets/bootstrap ● Kyverno Docs to Generate Flux Multi-tenant Resources ○ https://kyverno.io/policies/flux/generate-flux-multi-tenant-resources/gene rate-flux-multi-tenant-resources/
  • 18.
    18 Confidential do notdistribute 18 weave.works Thank you