Slides about how we at Giant Swarm manage Kubernetes apps at scale. This was a talk I gave during DevOps Pro Europe '22 conference.

1. May 26, 2022
Łukasz Piątkowski
Kubernetes Platform Architect
Managing Apps at Scale
In Kubernetes

2. Agenda
Introduction
What’s the problem, what’s the solution?
How does it work?
Summary

3. Introduction

4. Problems
● Preparing applications for a Kubernetes cluster
○ How to prepare (package) an app?
○ How to ensure best practices?
○ How to make sure the application will run on your cluster?
● Deploying applications
○ How to manage life cycle?
○ How to manage configuration?
■ Extracting common patterns
■ Yet doing last mile overrides

5. What is Helm?
● Package manager for Kubernetes
● Allows you to pack different YAMLs (Deployment, Service,
anything) into a single package and run some templating on the
YAMLs
● Templating parameters act as Chart’s configuration
● Terminology
○ Chart - a package that can be installed on a cluster
○ Repository - a service exposing downloadable Charts
○ Release - a Chart and its configuration installed on a Kubernetes
cluster

8. More problems
● Shortcomings of Helm
○ No support for deploying to multiple clusters
○ Only installation time tests
○ No representation in K8s API
○ One configuration layer only
○ CRD management can be hard
○ Cluster live state detection is hard
○ Source command (intent) is On My Laptop Only (™)

9. Solution: App Platform

10. Why App Platform?
We manage fleets (hundreds) of Kubernetes clusters, so we need tools that can:
● Ensure quality at build and release time, with easy repeatable process
● Target many clusters from a single control point,
● Share, reuse, and also override configuration of applications across multiple clusters,
● Offer the same set of applications across all managed clusters,
● Offer a native Kubernetes API for application management.

11. What is App Platform?
A set of tools to help create, test, deliver and manage applications (Helm charts) on top of Kubernetes,
at scale.
● create - app-build-suite
○ Best practices about building and QA-ing Helm charts
○ Providing additional metadata about the app
● test - app-test-suite
○ Tools to help test the app before delivering to clusters
● deliver - chart repositories
○ Tools and practices about storing Helm charts
● manage - operators
○ Kubernetes native app life-cycle management API for fleets

12. App Platform 10,000 m view
app-build-suite
Operators Operators
app-test-suite
Workload Cluster Management Cluster
Helm chart sources
Metadata info
Helm chart
Test
Kubernetes
Cluster
Helm
repository
Tests
Metadata
CI/CD Process
scan
deploy

13. App Platform 1,000 m view
Operators Operators
app-test-suite
Workload Cluster Management Cluster
Test
Kubernetes
Cluster
Helm
repository
scan
deploy
app-build-suite
Helm chart sources
Metadata info
Helm chart
Tests
Metadata
CI/CD Process

15. Building an app
● App-build-suite
○ Opinionated and repeatable process to run on dev machines and in CI/CD
○ Docs: https://github.com/giantswarm/app-build-suite
○ The build process
■ App and chart versions in the Chart.yaml file are set using git info (if configured)
■ External linters and code quality tools are invoked
■ Helm creates a chart archive
■ Metadata is generated from the data collected during the build (if configured)
○ What is metadata?
■ We extend Helm with a side file that includes more non-standard metadata about the chart, like:
● Which cloud infrastructure provider is this app valid for?
● Is it safe to install it multiple times on a single cluster or in a single namespace?

16. App Platform 1,000 m view
Operators Operators
Workload Cluster Management Cluster
deploy
app-build-suite
Helm chart sources
Metadata info
Helm chart
Tests
Metadata
app-test-suite
Test
Kubernetes
Cluster
Helm
repository
scan
CI/CD Process

18. Testing an app
● App-test-suite
○ Repeatable process to test on dev machines and in CI/CD
○ Docs: https://github.com/giantswarm/app-test-suite
○ Runs scenarios, currently smoke, functional and upgrade tests
○ Takes care of bootstrapping target cluster
○ Allows to implement tests in python or go
○ Declarative matching between scenarios and test implementation
○ Can produce additional metadata
■ Upgrade tests save info on successfully tested upgrade path

19. Testing an app
Test matching and execution
Smoke
run tests marked @smoke
Functional
run tests marked @functional
Upgrade
• run tests marked @upgrade on stable App version
• upgrade the App version
• run tests marked @upgrade again on new App version
@pytest.mark.smoke
def test_app_installed(cluster):
@pytest.mark.functional
@pytest.mark.upgrade
def test_login_api_ok(cluster):
@pytest.mark.upgrade
def test_new_api_ok(cluster):
app-test-suite tests.py

20. Testing an app
● Python test helper - pytest-helm-charts
○ Pytest plugin
○ Delivers test information and cluster connection as a set of fixtures (dependency-injected objects)
○ Integrated with pykube-ng library

21. App Platform 10.000 m view
Operators Operators
Workload Cluster Management Cluster
deploy
app-build-suite
Helm chart sources
Metadata info
Helm chart
Tests
Metadata
app-test-suite
Test
Kubernetes
Cluster
Helm
repository
scan
CI/CD Process

23. Chart storage
● Currently, very simple
○ As a Helm repository available through HTTPS
○ Charts stored together with their metadata
○ The repository is periodically scanned by our life-cycle
management operators and reflected as Kubernetes objects

24. App Platform 10.000 m view
app-build-suite
Helm chart sources
Metadata info
Helm chart
Tests
Metadata
app-test-suite
Test
Kubernetes
Cluster
Helm
repository
CI/CD Process
Operators Operators
Workload Cluster Management Cluster
deploy
scan

26. Life-cycle management
● Tasks
○ Managing configuration - global defaults, last mile overrides
○ Native K8s style API - available the same way as any other
object in API server
○ Status reporting and monitoring
○ Configuration validation and defaulting
● 100% compatible with Helm charts and catalogs
○ With optional extensions like metadata

27. App life-cycle management API overview
Catalog CR
AppCatalogEntry CR
Shows which app described by
AppCatalogEntry should be
installed on which Workload Cluster.
Does that by creating Chart CR
there.
App CR
Local representation (on the
Workload Cluster) of an app that
should be installed on that cluster.
Creates local Helm Release.
Chart CR
Show what we have. Catalogs point
to remote Helm repositories.
AppCatalogEntries are created for
each app and its version present in
the catalog.
Management Cluster Workload Cluster

28. Main software components
● App-operator
○ Watches Catalog CRs
■ “Where is the catalog?”
○ Watches configured Catalog URLs to produce
AppCatalogEntries CRs
■ “What is in the catalog?”
■ Based on index.yaml and metadata files
○ Watches App CRs
■ “On which WC a specific app described by ACE should be deployed?”
○ Does 3-level config merge
■ Catalog level config and 2 App level configs (base and user configs)
○ Creates Chart CR on the target WC

29. Main software components
● Chart-operator
○ Runs on WC (applies to all CRs below)
○ Watches Chart [namespaced]
■ “Where should I install with Helm on this cluster?”
○ Manages local installation/update/removal requests using Helm
● App-admission-controller
○ Runs on MC
○ Validation and admission of App CRs
● App-exporter
○ Runs on MC
○ Prometheus metrics about the status of locally present App CRs

30. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller

31. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller
cafe1 namespace
ACE
ACE
Catalog
kubectl
app-operator

32. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller
cafe1 namespace
ACE
ACE
ACE
ACE
ACE
Catalog
kubectl
app-operator

33. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller
cafe1 namespace
ACE
ACE
ACE
ACE
ACE
App
ACE
ACE
Catalog
kubectl
app-operator

34. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller
cafe1 namespace
ACE
ACE
ACE
ACE
ACE
App
ACE
ACE
Catalog
kubectl
app-operator
cafe1 Cluster
giantswarm namespace
chart-operator
ACE
ACE
Chart

35. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller
cafe1 namespace
ACE
ACE
ACE
ACE
ACE
App
ACE
ACE
Catalog
kubectl
app-operator
cafe1 Cluster
giantswarm namespace
chart-operator
ACE
ACE
Chart
app1 namespace
Helm release
Deployment
Service
…

36. App Platform Deployment Process
Management Cluster
giantswarm namespace
app-operator chart-operator app-exporter
app-admission-controller
cafe1 namespace
ACE
ACE
ACE
ACE
ACE
App
ACE
ACE
Catalog
kubectl
app-operator
cafe2 namespace
ACE
ACE
ACE
ACE
ACE
App
ACE
ACE
Catalog
kubectl
app-operator
cafe1 Cluster
giantswarm namespace
chart-operator
ACE
ACE
Chart
app1 namespace
Helm release
Deployment
Service
…
cafe2 Cluster
giantswarm namespace
chart-operator
ACE
ACE
Chart
app2 namespace
Helm release
Deployment
Service
…

38. Integration with gitops tools
How to integrate app platform with gitops?
● The integration is natural - just keep your App CRs
definitions in the repo
● Remember to add configuration ConfigMaps and Secrets
○ Secrets need to be encrypted at rest in the repo, so use
tools like sops
● We recommend flux as gitops tool

39. Summary

41. Summary
● We need to deliver multiple apps to many clusters
○ We’re using Helm and are Helm compatible, but also extended it a lot
○ We’re addressing the delivery process from build, through test and then life-cycle management
○ We care about user experience
○ Nothing lives in void
■ We integrate well with gitops tools - we use Flux to manage our apps
● Future
○ Delivery pipeline security
○ More functionality in the metadata area
■ Kubernetes version compatibility testing
■ App dependencies

42. Questions?