The document outlines a journey of developing a dynamic infrastructure using Kubernetes to create a scalable and maintainable pipeline for application deployment. It discusses initial challenges, the implementation of a proof-of-concept using branch-based infrastructure, and highlights issues with the existing system that necessitate improvement. Proposed enhancements include using Helm for service-specific charts, creating internal tools for easier infrastructure management, and implementing automatic maintenance and garbage collection features.

1. Dynamic infrastructure for
development
Balázs Rostás

2. About me

3. Motivation
● Startup At the beginning many attempts, chaotic unsystematic code and
workflows
● Later on the requirements stabilized It was time to create a scalable developer
friendly pipeline
● The road from idea to release was too long
● We needed an easy way to test the applications

4. What was at the beginning?
● New branch
● Local build
● (Build or pull the dependencies)
● Deploy it locally (with the dependencies)
● Deploy the database if needed and seed it with dummy data
Problem: Not scalable, a more complex service might not have enough resources to
run on a local machine, error-prone and hard to maintain.

5. Kubernetes to the rescue
● We moved all of our infrastructure to kubernetes
● The infrastructure became more scalable and maintainable
● It allowed us to easily implement zero downtime deployments
● Proper load balancing, health check and automatic restarts out of the box

6. Idea: Dynamic infrastructure
PoC:
● Branch name identifies the new infrastructure
● Multiple repositories can join the same environment by using the same branch
name
● The created infrastructure has a deterministic access point
○ branch.platform.avatao.build
● Every environment gets its own isolated database with proper mock data

7. ● Create branch on a repository (e.g.: test)
● A build is started using Google Cloud Build
● New kubernetes namespace is created (called test, derived from the branch)
● Branch is created on the centralized kubernetes manifest repository (test)
● The configuration is changed on this new branch (sed) so that it uses the new
namespace (and others like replica number, read-only filesystem etc...)
PoC Implementation

8. ● hack/deploy-platform.sh (test branch)
○ Copy kubernetes secrets from QA (default namespace in the dev cluster)
○ Generate kubernetes manifest files using envsubst
○ Run managers to provision databases or anything that is needed to deploy the platform
○ Deploy the platform using the QA docker images except for the service that is currently deployed
● Resources like etcd, redis, RabbitMQ are shared amongst namespaces to lower
costs and are separated using other techniques like prefixing
● The platform is now available here: test.platform.avatao.build
If we use the same branch name on any other repository it will realize that one is
already created with the same name so it would just deploy it here.
PoC Implementation

9. Success?
Yes.

10. Good as it is?
No.

11. What's the problem?
● Wasteful, we don't need a new infrastructure for every branch
● The cleaning up is manual and tedious
● The system using branch name as the identifier was easy to implement but it has
many flaws
○ I have to rename a branch if I'd like it to join a specific infrastructure
○ What if I accidentally make a branch on my service with an existing infrastructure?
○ One branch can only run in one infrastructure
● If I'd like to change something in the kubernetes manifest files or introduce new
microservices I am in trouble
● If anything changes on the QA the synchronization is not easy

12. Let's improve it
● Find the right the tool for the goal (CircleCI, Spinnaker)
● Helm instead of envsubst
● Instead of a centralized kubernetes manifest repository, each service is responsible
for their own helm chart
● Instead of using the branch as the identifier, develop an internal tool to create
infrastructures easily from any branch (it needs to store these information)

13. Example #1
● Developer is working on X and Y service using A and B branches respectively
● It is time to try it out in a feature infrastructure
● Using the internal tool a new feature infra is created where the X and Y services
use the A and B branches
● There is a push on either X or Y repository -> using the configuration of the
infrastructures we can deploy the app to the proper environment or
environments(!).

14. Example #2
● We'd like to deprecate a service or create new ones
● When creating the infrastructure we simply remove the service we'd like to
deprecate during creation
● For the new service we choose which repository we'd like to deploy (it needs to
have the proper helm chart and spinnaker pipeline ready)
● The infrastructure will be created with the old service removed and the new
service deployed

15. Garbage collection and maintenance
● If the infrastructure is not needed we can remove it with a click of a button, or it
can be automatically deleted after some time of inactivity.
○ Idea: Associate Github PRs to environments and on merged status delete them
● If the QA environment changes we can synchronize it using the internal tool
○ Must not be automatic to prevent breaking change
○ Only update the parts that are not worked on (master branch versions)

16. Thank you!