This document discusses scaling a Jenkins continuous integration (CI) infrastructure on AWS from a single virtual machine to using Kubernetes, Docker, and Terraform. The solution principles are immutable infrastructure, self-healing, and on-demand scaling. The technology stack includes AWS, Terraform for AWS automation, Docker for tool configuration, and Kubernetes with Helm for container orchestration and Jenkins. Job definitions are put in version control and reused. Kubernetes secrets and Git-crypt are used for credentials provisioning. Artifacts are stored in S3. Multiple Jenkins masters are deployed in separate namespaces with shared node pools to allow team-specific builds at scale.

1. Scaling a conventional CI
infrastructure in the public
cloud
Mikhail Advani

2. Who am I
● DevOps Lead Architect at EdCast
● IaC practitioner
● Security enthusiast

3. Historical architecture
● Single VM jenkins master in AWS
● Multiple slaves which due to lack of
maintenance were terminated - but
not cleaned up

4. Problems with historical architecture
● Resource starvation when multiple builds run in parallel
● Continuously running build agents even when jobs weren’t running
● UI driven job definitions without change tracking
● Risky upgrades
● Single point of failure
● Manual maintenance like disk cleanups

5. Solution principles
● Immutable infrastructure
● Self healing
● On demand scaling up & down

6. Technology stack
● AWS
● Terraform for AWS automation
● Docker for workspace tooling configuration
● Kubernetes with helm for container orchestration
● Jenkins

7. Architecture: Node distribution

8. Architecture: Isolation

9. Solution: Job definition & templatization
● Put job definitions in version control
● Define once & re-use multiple times
● Centralised control for job definitions

10. Solution: Master & slave management
● Helm chart to deploy master
● Multi-container pod templates as per build requirements
● Implicit slave sidecar responsible for master-slave communication

11. Solution: Master + plugin configuration
● Plugin installation & upgrade management

12. Solution: Credentials provisioning
● Kubernetes secrets
● Git-crypt

13. Solution: Artifact storage
● S3 backed artifact repository

14. Scaling to multiple masters
● Team specific jenkins masters
● One Kubernetes namespace per master
● Shared node-pools

15. Gotchas
● Exposing docker in docker
● Multi-AZ ASGs & cluster-autoscaler

16. Peripheral services
● Cluster autoscaling
● Traffic management
● DNS management
● TLS certificate management
● Monitoring

17. Thank you
https://linkedin.com/in/mikhailadvani
https://github.com/mikhailadvani