Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pipeline your pipelines!

devOpsDays Galway 18 Nov 2019

  • Be the first to comment

  • Be the first to like this

Pipeline your pipelines!

  1. 1. Pipeline your Pipelines!
  2. 2. Hardware spec: 1KB RAM (upg. 16KB) 4KB ROM First computer Past Companies Communities Giulio Vian Principal Engineer & DevOps Lead
  3. 3. Session framing Introductory level Presentation deck on SlideShare Blog series with examples GitHub Repository with examples Is my timer running?
  4. 4. There are no development environments
  5. 5. It all starts with a itch 90% of our builds were running on a physical server
  6. 6. What if I lose my build infrastructure?
  7. 7. I stop deploying fixes in production!
  8. 8. Principle #1: Development is Production
  9. 9. Infrastructure: Continuous Integration (build) Orchestrator Server Build Server Agent Build Server Agent Artifact Store
  10. 10. Did I say Agent?
  11. 11. Pick your favourites slave agent runner agent remote agent executor master server node server server server Jenkins Azure DevOps GitLab TeamCity Bamboo CircleCI orchestrator agent
  12. 12. Infrastructure: Continuous Delivery (deploy) Orchestrator Server Artifact StoreServer Server Agent Deploy Server Agent
  13. 13. Infrastructure: CI+CD (build & deploy) Orchestrator Server Build Server Agent Build Server Agent Artifact Store Server Server Agent Deploy Server Agent
  14. 14. Why not using Build as‐a‐Service?
  15. 15. Principle #2: Don’t share any secret
  16. 16. Secrets? Production passwords Keys SSL Certificates Code signing certificates
  17. 17. What is disposable?
  18. 18. Recycle scale Source code Keys Deploy data Testing data Build data Build infrastructure Precious Disposable
  19. 19. Principle #3: Automate everything
  20. 20. Pattern #1 – pure Docker Host Container Agent AgentOrchestrator Server 1 3 Pipelines: 1. Create the Host VM with Agent installed (require bootstrap) 2. Build and Push Docker images for Build Agents 3. Start Docker Containers hosting Build Agents Docker Registry 2
  21. 21. Design #2 – Kubernetes Kubernetes Cluster Container Agent AgentOrchestrator Server Docker Registry Pipelines: 1. Create the Kubernetes cluster (require bootstrap) 2. Build and Push Docker images for Build Agents (bootstrap) 3. Deploy Docker Containers hosting Build Agents to cluster 1 3 2
  22. 22. Step 0: Bootstrapping Aim to minimal requirements, e.g. • Cloud credentials • Cloud toolkit (e.g. AWS or Azure CLI) • Script that setup a VM • Installs the server • Restore backups
  23. 23. Step 1: Hosts Physical machine(s) VMs Kubernetes cluster Metal-as-a-Service Microsoft Deployment Toolkit Infrastructure-as-code Terraform, Pulumi CloudFormation, ARM Templates
  24. 24. Toolchain + Agent Update regularly Stateless Builds Step 2: docker-ize the agent
  25. 25. Step 3: pipeline the creation of agents Automate deployment of new / existing agents Agent on host launches new agents in containers Hurdle: manual admin grant
  26. 26. Step 4: pipeline the creation of hosts Script to create host Create VM Install Agent Run on master or hosted agent Autoscale?
  27. 27. Alternate path: K8S hosting Why bother managing machines? Still need a cluster… Illustration © SignMission 2018
  28. 28. Benefits and collaterals Documenting build and deploy infrastructure Re-create on any cloud provider Engineers familiarize with Docker & Kubernetes
  29. 29. Backup Files Databases Printed instructions
  30. 30. Monitoring
  31. 31. The more you automate, the more everything becomes production
  32. 32. Series of posts
  33. 33. Links @giulio_vian
  34. 34. End of transmissions 34