SlideShare a Scribd company logo
1 of 34
Download to read offline
~$ whoami
● Platform Engineer
● Est. 2014
● CatOps
○ Telegram
○ Substack
○ YouTube
● Grem1.in
● HashiCorp User Group Kyiv
● DevOps Days Ukraine
● DOU Conference
● LinkedIn
● GitHub
Helm is boring
● There are more exciting ways of
deploying to Kubernetes:
○ CUE
○ Cdk8s
○ Pkl
● Yet, boring tech is worth talking
about
Casas y Carbo, Ramon - “Apres le bal”. Painting Montserrat (Catalonia),
Museo dela Abadia.
Norwegian Bokmål: Leiv Eiriksson discovers
North America
A journey begins…
● Company-wide migration to
Kubernetes
● Self-hosted cluster (on AWS)
● Dozens of plugins (CNI, service
mesh, observability, custom
operators, etc.) distributed in a form
of Helm charts *
● 3 Helm complex* Helm charts to
support ~ 200 applications
● A couple of library charts to
standardize certain logic in the
complex Helm charts
Margrethe II - Anduin River
The Crew
● At the same time we spit one big
“kitchen sink” team into multiple
specialized teams
● Each team had its own focus e.g.
infrastructure, observability, CI/CD,
etc.
● Each platform team has something
to contribute to the Helm charts
● Application charts (those 3) were
consumed by hundreds of product
developers
Washington Crossing the Delaware (1851). Metropolitan Museum of
Art, New York City
John Singleton Copley (American, 1738 – 1815 ),
Watson and the Shark, 1778, oil on canvas,
Ferdinand Lammot Belin Fund
Universal Charts in Production
● Centralizing the charts gave a lot of
control to platform teams
● Easy API (values.yaml) for
developers
● Multiple platform teams
contributed to the same charts
● Some shared functionality was
provided with Library charts
● 3 charts was enough to migrate
almost all the applications
Platform
Devs
Ivan Aivazovsky - Exploding Ship, oil, canvas, Aivazovsky National Art Gallery, Feodosiya,
Ukraine
Let’s Add Some Tests!
Helm Test
● A native command that first comes
to mind
● Requires a K8s cluster
● We can create a local KIND or
MiniKube, or even K3s/K0s cluster
in CI time
● But what do we do with external
dependencies? Think of Vault,
Consul, cloud storage, etc.
The Hay Wagon - central panel by Hieronymus Bosch
What do we even testing?
● Have a chunky test cluster and run
tests in CI against it = Money
● Create a test cluster before running
the tests = Time
● What do we even test when
deploying a generic Nginx into a
cluster?
Salvador Dali - The Persistence of Memory
“In the most cases, configuration management
can be tested using simple static code analysis”
- Jeff Smith @ DevOps Days Chicago
Kazimir Malevich - Suprematist Composition
Let’s validate manifests without a cluster
● The first thing we came up with is
so-called “snapshot testing”
● We could template a given Helm
chart and compare the results with
a “golden” file
● And we did it with Terratest
Georgia O’Keeffe - “Pelvis with Distance,” 1943. San Diego Museum of Art’s
Terratest
● The Good:
○ You can use Go to write the tests, which makes it very flexible
○ It’s fairly simple to use with a very minimal setup
Terratest
● The Good:
○ You can use Go to write the tests, which makes it very flexible
○ It’s fairly simple to use with a very minimal setup
● The Bad:
○ Not everyone is comfortable writing Go :(
○ Code duplication for various charts, so we moved the code that does the
heavy-lifting into a separate package
○ You still need to maintain the code
Terratest
● The Good:
○ You can use Go to write the tests, which makes it very flexible
○ It’s fairly simple to use with a very minimal setup
● The Bad:
○ Not everyone is comfortable writing Go :(
○ Code duplication for various charts, so we moved the code that does the
heavy-lifting into a separate package
○ You still need to maintain the code
● The Ugly:
○ Way to many false-positives, which make people discard the test results
○ Fixing those false-positives only resulted in more maintenance
Maria
Prymachenko -
"For the joy of
people"
Let’s only test what matters
● A chart can be rendered at all
● Charts themselves are following good practices
● Resulting manifestes are “correct”
● Resulting manifestes follow good practices (including security)
● Any logic inside charts (conditionals, includes, etc.)
● Tests are executed in reasonable amount of time
● Tests are simple to maintain
● Tests are reproducible
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
● Any logic inside charts (conditionals, includes, etc.)
○ Kyverno
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
● Any logic inside charts (conditionals, includes, etc.)
○ Kyverno
○ Helm Unittest
Tools
● A chart can be rendered at all
○ helm template & helm JSON schema
○ Helm Unittest
● Charts themselves are following good practices
○ helm lint
● Resulting manifestes are “correct”
○ Kubeconform
● Resulting manifestes follow good practices (including security)
○ Kyverno
○ Helm Unittest
● Any logic inside charts (conditionals, includes, etc.)
○ Kyverno
○ Helm Unittest
Tests Baseline
Custom tests of the logic (Helm Unittest)
Security practices (Kyverno)
Kubernetes good practices (Kubeconform)
Helm charts good practices (Helm Lint)
“Renderability” (Helm Template)
František Xaver
Sandmann -
Napoleon in exile
on St. Helena,
Watercolor, c.
1820.
Are these tests enough?
Are these tests enough?
● Helm Unittest has its own
limitations
● For example, it doesn’t work well
with nested lists
● In such cases we fallback to
snapshot testing (Helm Unittest
supports that)
Gustav Klimt - The Kiss, 1907–1908, oil on canvas
Are these tests enough?
● Although, we created the whole
“local” pyramid, we still wanted to
test something in a real cluster
● For this reason, we created a
“dummy” service that mimicked a
real one
● We also created stress-tests for
Kubernetes using Kube Burner and
E2E tests using Sonobuoy &
kubernetes-e2e-framework
● Yet, this is a story for another time
Claude Monet - Fishing Boats Leaving the Harbor, Le Havre, 1874
And one more thing
● You can use the same “static code”
analysis not only for Helm
● Conftest can achieve very similar
results for Terraform (but with
Rego)
Vincent Van Gogh - Almond Blossom
Thank you for your time!
See you in Q&A
Hokusai: The Breaking Wave off Kanagawa
Links
● Helm Unittest - https://github.com/helm-unittest/helm-unittest
● Terratest - https://terratest.gruntwork.io/
● Terratest-helm-testing-example -
https://github.com/gruntwork-io/terratest-helm-testing-example
● Automated Testing for Kubernetes and Helm Charts using Terratest -
https://blog.gruntwork.io/automated-testing-for-kubernetes-and-helm-chart
s-using-terratest-a4ddc4e67344
● Advanced Test Practices For Helm Charts -
https://medium.com/@zelldon91/advanced-test-practices-for-helm-charts-58
7caeeb4cb
● Kubeconform - https://github.com/yannh/kubeconform
● Kyverno - https://kyverno.io/
● Conftest (for Terraform) - https://www.conftest.dev/

More Related Content

Similar to "Testing of Helm Charts or There and Back Again", Yura Rochniak

Puppet Camp Dublin - 06/2012
Puppet Camp Dublin - 06/2012Puppet Camp Dublin - 06/2012
Puppet Camp Dublin - 06/2012
Roland Tritsch
 
Pycon 2012 What Python can learn from Java
Pycon 2012 What Python can learn from JavaPycon 2012 What Python can learn from Java
Pycon 2012 What Python can learn from Java
jbellis
 

Similar to "Testing of Helm Charts or There and Back Again", Yura Rochniak (20)

reBuy on Kubernetes
reBuy on KubernetesreBuy on Kubernetes
reBuy on Kubernetes
 
JVM Performance Tuning
JVM Performance TuningJVM Performance Tuning
JVM Performance Tuning
 
groovy & grails - lecture 6
groovy & grails - lecture 6groovy & grails - lecture 6
groovy & grails - lecture 6
 
Montreal OpenStack Q2 MeetUp - May 30th 2017
Montreal OpenStack Q2 MeetUp - May 30th 2017Montreal OpenStack Q2 MeetUp - May 30th 2017
Montreal OpenStack Q2 MeetUp - May 30th 2017
 
Kubernetes & Google Container Engine @ mabl
Kubernetes & Google Container Engine @ mablKubernetes & Google Container Engine @ mabl
Kubernetes & Google Container Engine @ mabl
 
Integrating microservices with apache camel on kubernetes
Integrating microservices with apache camel on kubernetesIntegrating microservices with apache camel on kubernetes
Integrating microservices with apache camel on kubernetes
 
OpenStack Toronto Q2 MeetUp - June 1st 2017
OpenStack Toronto Q2 MeetUp - June 1st 2017OpenStack Toronto Q2 MeetUp - June 1st 2017
OpenStack Toronto Q2 MeetUp - June 1st 2017
 
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On PremTo Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
To Russia with Love: Deploying Kubernetes in Exotic Locations On Prem
 
BBC's GraphDB (formerly Owlim) AWS Cloud Migration
BBC's GraphDB (formerly Owlim) AWS Cloud MigrationBBC's GraphDB (formerly Owlim) AWS Cloud Migration
BBC's GraphDB (formerly Owlim) AWS Cloud Migration
 
Новый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныНовый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоны
 
Introduction to Container Storage Interface (CSI)
Introduction to Container Storage Interface (CSI)Introduction to Container Storage Interface (CSI)
Introduction to Container Storage Interface (CSI)
 
Kubernetes - how to orchestrate containers
Kubernetes - how to orchestrate containersKubernetes - how to orchestrate containers
Kubernetes - how to orchestrate containers
 
The Return of the Dull Stack Engineer
The Return of the Dull Stack EngineerThe Return of the Dull Stack Engineer
The Return of the Dull Stack Engineer
 
ECMAScript 6 from an Attacker's Perspective - Breaking Frameworks, Sandboxes,...
ECMAScript 6 from an Attacker's Perspective - Breaking Frameworks, Sandboxes,...ECMAScript 6 from an Attacker's Perspective - Breaking Frameworks, Sandboxes,...
ECMAScript 6 from an Attacker's Perspective - Breaking Frameworks, Sandboxes,...
 
rsyslog meets docker
rsyslog meets dockerrsyslog meets docker
rsyslog meets docker
 
Puppet Camp Dublin - 06/2012
Puppet Camp Dublin - 06/2012Puppet Camp Dublin - 06/2012
Puppet Camp Dublin - 06/2012
 
Kubernetes CRI containerd integration by Lantao Liu (Google)
Kubernetes CRI containerd integration by Lantao Liu (Google)Kubernetes CRI containerd integration by Lantao Liu (Google)
Kubernetes CRI containerd integration by Lantao Liu (Google)
 
Pycon 2012 What Python can learn from Java
Pycon 2012 What Python can learn from JavaPycon 2012 What Python can learn from Java
Pycon 2012 What Python can learn from Java
 
Making Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with NovaMaking Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with Nova
 
Building a continuous delivery platform for the biggest spike in e-commerce -...
Building a continuous delivery platform for the biggest spike in e-commerce -...Building a continuous delivery platform for the biggest spike in e-commerce -...
Building a continuous delivery platform for the biggest spike in e-commerce -...
 

More from Fwdays

More from Fwdays (20)

"What I learned through reverse engineering", Yuri Artiukh
"What I learned through reverse engineering", Yuri Artiukh"What I learned through reverse engineering", Yuri Artiukh
"What I learned through reverse engineering", Yuri Artiukh
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
"Micro frontends: Unbelievably true life story", Dmytro Pavlov
"Micro frontends: Unbelievably true life story", Dmytro Pavlov"Micro frontends: Unbelievably true life story", Dmytro Pavlov
"Micro frontends: Unbelievably true life story", Dmytro Pavlov
 
"Objects validation and comparison using runtime types (io-ts)", Oleksandr Suhak
"Objects validation and comparison using runtime types (io-ts)", Oleksandr Suhak"Objects validation and comparison using runtime types (io-ts)", Oleksandr Suhak
"Objects validation and comparison using runtime types (io-ts)", Oleksandr Suhak
 
"JavaScript. Standard evolution, when nobody cares", Roman Savitskyi
"JavaScript. Standard evolution, when nobody cares", Roman Savitskyi"JavaScript. Standard evolution, when nobody cares", Roman Savitskyi
"JavaScript. Standard evolution, when nobody cares", Roman Savitskyi
 
"How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y...
"How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y..."How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y...
"How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y...
 
"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil Topchii
"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil Topchii"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil Topchii
"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil Topchii
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
"What is a RAG system and how to build it",Dmytro Spodarets
"What is a RAG system and how to build it",Dmytro Spodarets"What is a RAG system and how to build it",Dmytro Spodarets
"What is a RAG system and how to build it",Dmytro Spodarets
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
"Distributed graphs and microservices in Prom.ua", Maksym Kindritskyi
"Distributed graphs and microservices in Prom.ua",  Maksym Kindritskyi"Distributed graphs and microservices in Prom.ua",  Maksym Kindritskyi
"Distributed graphs and microservices in Prom.ua", Maksym Kindritskyi
 
"Rethinking the existing data loading and processing process as an ETL exampl...
"Rethinking the existing data loading and processing process as an ETL exampl..."Rethinking the existing data loading and processing process as an ETL exampl...
"Rethinking the existing data loading and processing process as an ETL exampl...
 
"How Ukrainian IT specialist can go on vacation abroad without crossing the T...
"How Ukrainian IT specialist can go on vacation abroad without crossing the T..."How Ukrainian IT specialist can go on vacation abroad without crossing the T...
"How Ukrainian IT specialist can go on vacation abroad without crossing the T...
 
"The Strength of Being Vulnerable: the experience from CIA, Tesla and Uber", ...
"The Strength of Being Vulnerable: the experience from CIA, Tesla and Uber", ..."The Strength of Being Vulnerable: the experience from CIA, Tesla and Uber", ...
"The Strength of Being Vulnerable: the experience from CIA, Tesla and Uber", ...
 
"[QUICK TALK] Radical candor: how to achieve results faster thanks to a cultu...
"[QUICK TALK] Radical candor: how to achieve results faster thanks to a cultu..."[QUICK TALK] Radical candor: how to achieve results faster thanks to a cultu...
"[QUICK TALK] Radical candor: how to achieve results faster thanks to a cultu...
 
"[QUICK TALK] PDP Plan, the only one door to raise your salary and boost care...
"[QUICK TALK] PDP Plan, the only one door to raise your salary and boost care..."[QUICK TALK] PDP Plan, the only one door to raise your salary and boost care...
"[QUICK TALK] PDP Plan, the only one door to raise your salary and boost care...
 
"4 horsemen of the apocalypse of working relationships (+ antidotes to them)"...
"4 horsemen of the apocalypse of working relationships (+ antidotes to them)"..."4 horsemen of the apocalypse of working relationships (+ antidotes to them)"...
"4 horsemen of the apocalypse of working relationships (+ antidotes to them)"...
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 

Recently uploaded (20)

The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 

"Testing of Helm Charts or There and Back Again", Yura Rochniak

  • 1.
  • 2. ~$ whoami ● Platform Engineer ● Est. 2014 ● CatOps ○ Telegram ○ Substack ○ YouTube ● Grem1.in ● HashiCorp User Group Kyiv ● DevOps Days Ukraine ● DOU Conference ● LinkedIn ● GitHub
  • 3. Helm is boring ● There are more exciting ways of deploying to Kubernetes: ○ CUE ○ Cdk8s ○ Pkl ● Yet, boring tech is worth talking about Casas y Carbo, Ramon - “Apres le bal”. Painting Montserrat (Catalonia), Museo dela Abadia.
  • 4. Norwegian Bokmål: Leiv Eiriksson discovers North America
  • 5. A journey begins… ● Company-wide migration to Kubernetes ● Self-hosted cluster (on AWS) ● Dozens of plugins (CNI, service mesh, observability, custom operators, etc.) distributed in a form of Helm charts * ● 3 Helm complex* Helm charts to support ~ 200 applications ● A couple of library charts to standardize certain logic in the complex Helm charts Margrethe II - Anduin River
  • 6. The Crew ● At the same time we spit one big “kitchen sink” team into multiple specialized teams ● Each team had its own focus e.g. infrastructure, observability, CI/CD, etc. ● Each platform team has something to contribute to the Helm charts ● Application charts (those 3) were consumed by hundreds of product developers Washington Crossing the Delaware (1851). Metropolitan Museum of Art, New York City
  • 7. John Singleton Copley (American, 1738 – 1815 ), Watson and the Shark, 1778, oil on canvas, Ferdinand Lammot Belin Fund
  • 8. Universal Charts in Production ● Centralizing the charts gave a lot of control to platform teams ● Easy API (values.yaml) for developers ● Multiple platform teams contributed to the same charts ● Some shared functionality was provided with Library charts ● 3 charts was enough to migrate almost all the applications Platform Devs
  • 9. Ivan Aivazovsky - Exploding Ship, oil, canvas, Aivazovsky National Art Gallery, Feodosiya, Ukraine
  • 11. Helm Test ● A native command that first comes to mind ● Requires a K8s cluster ● We can create a local KIND or MiniKube, or even K3s/K0s cluster in CI time ● But what do we do with external dependencies? Think of Vault, Consul, cloud storage, etc. The Hay Wagon - central panel by Hieronymus Bosch
  • 12. What do we even testing? ● Have a chunky test cluster and run tests in CI against it = Money ● Create a test cluster before running the tests = Time ● What do we even test when deploying a generic Nginx into a cluster? Salvador Dali - The Persistence of Memory
  • 13. “In the most cases, configuration management can be tested using simple static code analysis” - Jeff Smith @ DevOps Days Chicago Kazimir Malevich - Suprematist Composition
  • 14. Let’s validate manifests without a cluster ● The first thing we came up with is so-called “snapshot testing” ● We could template a given Helm chart and compare the results with a “golden” file ● And we did it with Terratest Georgia O’Keeffe - “Pelvis with Distance,” 1943. San Diego Museum of Art’s
  • 15. Terratest ● The Good: ○ You can use Go to write the tests, which makes it very flexible ○ It’s fairly simple to use with a very minimal setup
  • 16. Terratest ● The Good: ○ You can use Go to write the tests, which makes it very flexible ○ It’s fairly simple to use with a very minimal setup ● The Bad: ○ Not everyone is comfortable writing Go :( ○ Code duplication for various charts, so we moved the code that does the heavy-lifting into a separate package ○ You still need to maintain the code
  • 17. Terratest ● The Good: ○ You can use Go to write the tests, which makes it very flexible ○ It’s fairly simple to use with a very minimal setup ● The Bad: ○ Not everyone is comfortable writing Go :( ○ Code duplication for various charts, so we moved the code that does the heavy-lifting into a separate package ○ You still need to maintain the code ● The Ugly: ○ Way to many false-positives, which make people discard the test results ○ Fixing those false-positives only resulted in more maintenance
  • 19. Let’s only test what matters ● A chart can be rendered at all ● Charts themselves are following good practices ● Resulting manifestes are “correct” ● Resulting manifestes follow good practices (including security) ● Any logic inside charts (conditionals, includes, etc.) ● Tests are executed in reasonable amount of time ● Tests are simple to maintain ● Tests are reproducible
  • 20. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema
  • 21. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema ● Charts themselves are following good practices ○ helm lint
  • 22. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema ● Charts themselves are following good practices ○ helm lint ● Resulting manifestes are “correct” ○ Kubeconform
  • 23. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema ● Charts themselves are following good practices ○ helm lint ● Resulting manifestes are “correct” ○ Kubeconform ● Resulting manifestes follow good practices (including security) ○ Kyverno
  • 24. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema ● Charts themselves are following good practices ○ helm lint ● Resulting manifestes are “correct” ○ Kubeconform ● Resulting manifestes follow good practices (including security) ○ Kyverno ● Any logic inside charts (conditionals, includes, etc.) ○ Kyverno
  • 25. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema ● Charts themselves are following good practices ○ helm lint ● Resulting manifestes are “correct” ○ Kubeconform ● Resulting manifestes follow good practices (including security) ○ Kyverno ● Any logic inside charts (conditionals, includes, etc.) ○ Kyverno ○ Helm Unittest
  • 26. Tools ● A chart can be rendered at all ○ helm template & helm JSON schema ○ Helm Unittest ● Charts themselves are following good practices ○ helm lint ● Resulting manifestes are “correct” ○ Kubeconform ● Resulting manifestes follow good practices (including security) ○ Kyverno ○ Helm Unittest ● Any logic inside charts (conditionals, includes, etc.) ○ Kyverno ○ Helm Unittest
  • 27. Tests Baseline Custom tests of the logic (Helm Unittest) Security practices (Kyverno) Kubernetes good practices (Kubeconform) Helm charts good practices (Helm Lint) “Renderability” (Helm Template)
  • 28. František Xaver Sandmann - Napoleon in exile on St. Helena, Watercolor, c. 1820.
  • 29. Are these tests enough?
  • 30. Are these tests enough? ● Helm Unittest has its own limitations ● For example, it doesn’t work well with nested lists ● In such cases we fallback to snapshot testing (Helm Unittest supports that) Gustav Klimt - The Kiss, 1907–1908, oil on canvas
  • 31. Are these tests enough? ● Although, we created the whole “local” pyramid, we still wanted to test something in a real cluster ● For this reason, we created a “dummy” service that mimicked a real one ● We also created stress-tests for Kubernetes using Kube Burner and E2E tests using Sonobuoy & kubernetes-e2e-framework ● Yet, this is a story for another time Claude Monet - Fishing Boats Leaving the Harbor, Le Havre, 1874
  • 32. And one more thing ● You can use the same “static code” analysis not only for Helm ● Conftest can achieve very similar results for Terraform (but with Rego) Vincent Van Gogh - Almond Blossom
  • 33. Thank you for your time! See you in Q&A Hokusai: The Breaking Wave off Kanagawa
  • 34. Links ● Helm Unittest - https://github.com/helm-unittest/helm-unittest ● Terratest - https://terratest.gruntwork.io/ ● Terratest-helm-testing-example - https://github.com/gruntwork-io/terratest-helm-testing-example ● Automated Testing for Kubernetes and Helm Charts using Terratest - https://blog.gruntwork.io/automated-testing-for-kubernetes-and-helm-chart s-using-terratest-a4ddc4e67344 ● Advanced Test Practices For Helm Charts - https://medium.com/@zelldon91/advanced-test-practices-for-helm-charts-58 7caeeb4cb ● Kubeconform - https://github.com/yannh/kubeconform ● Kyverno - https://kyverno.io/ ● Conftest (for Terraform) - https://www.conftest.dev/