I would like to share a little story with you about how we started testing our Helm charts, why we did it, and were the outcomes. Our journey went from no testing to snapshot testing with Terratest and then from Terratest to more granular approach towards the validation of the configuration. I would like to show our journey to you as well as the tools we met in the process and the lessons we learned.

2. ~$ whoami
● Platform Engineer
● Est. 2014
● CatOps
○ Telegram
○ Substack
○ YouTube
● Grem1.in
● HashiCorp User Group Kyiv
● DevOps Days Ukraine
● DOU Conference
● LinkedIn
● GitHub

3. Helm is boring
● There are more exciting ways of
deploying to Kubernetes:
○ CUE
○ Cdk8s
○ Pkl
● Yet, boring tech is worth talking
about
Casas y Carbo, Ramon - “Apres le bal”. Painting Montserrat (Catalonia),
Museo dela Abadia.

4. Norwegian Bokmål: Leiv Eiriksson discovers
North America

5. A journey begins…
● Company-wide migration to
Kubernetes
● Self-hosted cluster (on AWS)
● Dozens of plugins (CNI, service
mesh, observability, custom
operators, etc.) distributed in a form
of Helm charts *
● 3 Helm complex* Helm charts to
support ~ 200 applications
● A couple of library charts to
standardize certain logic in the
complex Helm charts
Margrethe II - Anduin River

6. The Crew
● At the same time we spit one big
“kitchen sink” team into multiple
specialized teams
● Each team had its own focus e.g.
infrastructure, observability, CI/CD,
etc.
● Each platform team has something
to contribute to the Helm charts
● Application charts (those 3) were
consumed by hundreds of product
developers
Washington Crossing the Delaware (1851). Metropolitan Museum of
Art, New York City

7. John Singleton Copley (American, 1738 – 1815 ),
Watson and the Shark, 1778, oil on canvas,
Ferdinand Lammot Belin Fund

8. Universal Charts in Production
● Centralizing the charts gave a lot of
control to platform teams
● Easy API (values.yaml) for
developers
● Multiple platform teams
contributed to the same charts
● Some shared functionality was
provided with Library charts
● 3 charts was enough to migrate
almost all the applications
Platform
Devs

9. Ivan Aivazovsky - Exploding Ship, oil, canvas, Aivazovsky National Art Gallery, Feodosiya,
Ukraine

10. Let’s Add Some Tests!

11. Helm Test
● A native command that first comes
to mind
● Requires a K8s cluster
● We can create a local KIND or
MiniKube, or even K3s/K0s cluster
in CI time
● But what do we do with external
dependencies? Think of Vault,
Consul, cloud storage, etc.
The Hay Wagon - central panel by Hieronymus Bosch

12. What do we even testing?
● Have a chunky test cluster and run
tests in CI against it = Money
● Create a test cluster before running
the tests = Time
● What do we even test when
deploying a generic Nginx into a
cluster?
Salvador Dali - The Persistence of Memory

13. “In the most cases, configuration management
can be tested using simple static code analysis”
- Jeff Smith @ DevOps Days Chicago
Kazimir Malevich - Suprematist Composition

14. Let’s validate manifests without a cluster
● The first thing we came up with is
so-called “snapshot testing”
● We could template a given Helm
chart and compare the results with
a “golden” file
● And we did it with Terratest
Georgia O’Keeffe - “Pelvis with Distance,” 1943. San Diego Museum of Art’s

15. Terratest
● The Good:
○ You can use Go to write the tests, which makes it very flexible
○ It’s fairly simple to use with a very minimal setup

16. Terratest
● The Good:
○ You can use Go to write the tests, which makes it very flexible
○ It’s fairly simple to use with a very minimal setup
● The Bad:
○ Not everyone is comfortable writing Go :(
○ Code duplication for various charts, so we moved the code that does the
heavy-lifting into a separate package
○ You still need to maintain the code

17. Terratest
● The Good:
○ You can use Go to write the tests, which makes it very flexible
○ It’s fairly simple to use with a very minimal setup
● The Bad:
○ Not everyone is comfortable writing Go :(
○ Code duplication for various charts, so we moved the code that does the
heavy-lifting into a separate package
○ You still need to maintain the code
● The Ugly:
○ Way to many false-positives, which make people discard the test results
○ Fixing those false-positives only resulted in more maintenance

18. Maria
Prymachenko -
"For the joy of
people"

19. Let’s only test what matters
● A chart can be rendered at all
● Charts themselves are following good practices
● Resulting manifestes are “correct”
● Resulting manifestes follow good practices (including security)
● Any logic inside charts (conditionals, includes, etc.)
● Tests are executed in reasonable amount of time
● Tests are simple to maintain
● Tests are reproducible

20. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema

21. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint

22. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform

23. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno

24. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
● Any logic inside charts (conditionals, includes, etc.)
○ Kyverno

25. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
● Any logic inside charts (conditionals, includes, etc.)
○ Kyverno
○ Helm Unittest

26. Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
○ Helm Unittest
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
○ Helm Unittest
● Any logic inside charts (conditionals, includes, etc.)
○ Kyverno
○ Helm Unittest

27. Tests Baseline
Custom tests of the logic (Helm Unittest)
Security practices (Kyverno)
Kubernetes good practices (Kubeconform)
Helm charts good practices (Helm Lint)
“Renderability” (Helm Template)

28. František Xaver
Sandmann -
Napoleon in exile
on St. Helena,
Watercolor, c.
1820.

29. Are these tests enough?

30. Are these tests enough?
● Helm Unittest has its own
limitations
● For example, it doesn’t work well
with nested lists
● In such cases we fallback to
snapshot testing (Helm Unittest
supports that)
Gustav Klimt - The Kiss, 1907–1908, oil on canvas

31. Are these tests enough?
● Although, we created the whole
“local” pyramid, we still wanted to
test something in a real cluster
● For this reason, we created a
“dummy” service that mimicked a
real one
● We also created stress-tests for
Kubernetes using Kube Burner and
E2E tests using Sonobuoy &
kubernetes-e2e-framework
● Yet, this is a story for another time
Claude Monet - Fishing Boats Leaving the Harbor, Le Havre, 1874

32. And one more thing
● You can use the same “static code”
analysis not only for Helm
● Conftest can achieve very similar
results for Terraform (but with
Rego)
Vincent Van Gogh - Almond Blossom

33. Thank you for your time!
See you in Q&A
Hokusai: The Breaking Wave off Kanagawa

34. Links
● Helm Unittest - https://github.com/helm-unittest/helm-unittest
● Terratest - https://terratest.gruntwork.io/
● Terratest-helm-testing-example -
https://github.com/gruntwork-io/terratest-helm-testing-example
● Automated Testing for Kubernetes and Helm Charts using Terratest -
https://blog.gruntwork.io/automated-testing-for-kubernetes-and-helm-chart
s-using-terratest-a4ddc4e67344
● Advanced Test Practices For Helm Charts -
https://medium.com/@zelldon91/advanced-test-practices-for-helm-charts-58
7caeeb4cb
● Kubeconform - https://github.com/yannh/kubeconform
● Kyverno - https://kyverno.io/
● Conftest (for Terraform) - https://www.conftest.dev/