The AT&T team recently embarked on a journey with CloudStack and has since deployed a solution which encompasses multiple data-centers. This talk focuses on how they are using open source tools like CloudStack, FreeIPA, and Metal as a Service (MaaS) to support KVM-based VM provisioning at an enterprise scale within a GitOps model. ----------------------------------------- The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.

1. CloudStack and
GitOps at Enterprise
Scale
Rene Glover
Alex Dometrius

2. Who are we
• Application Infrastructure Platforms and Services Team
• Centrally-managed, Enterprise Kubernetes platform
• Hundreds of applications supported
• Thousands of nodes
• Tens of thousands of microService containers
• Open-Source first mindset

4. Table of Contents
• Prologue: How did we get here
• Chapter 1: The Journey
• Chapter 2: GitOps
• Chapter 3: Architecture of CloudStack GitOps
• Chapter 4: Success with CloudStack

5. How did we get here
• Present State:
• Ordering-based, waterfall model for on-premise capacity
• Delivery measured in weeks/months
• Supported by multiple home-grown applications/tools
• Large footprint of under-utilized resources
• Modernization Progress:
• Migrating applications to Public Cloud providers
• Adoption of IaC best practices: Terraform, Packer, etc.
• Gained experience with private-networking models
• Already using Agile development practices

6. Why CloudStack
• Tenancy through projects
• CloudStack Terraform Provider and Native APIs
• Project Quotas & Usage Server/Project Accounting
• Over-provisioning capabilities
• Ability to manage with GitOps

7. Gaps/Issues Solved
• FreeIPA – UAM solution
• Metal as a Service (MaaS) – KVM Host Provisioning Solution
• “Golden” Templates – Reusability and Compliance Solution
• Platform Monitoring – Observability Solution

8. FreeIPA
• CloudStack Project Role = FreeIPA Group
• Role Permission
• Instance Group = FreeIPA hostgroup
• Instance Group

9. MaaS
• PXE boot & install Operating System
• Configure OS, Networking, Storage, iDRAC/iLO
• Perform OS Patching of KVM hosts
• MaaS & Redfish Terraform Provider updates

10. “Golden” Templates
• Several Linux Flavors and Versions
• Golden Database Templates
• Reduce OS version disparity
• Baked-in metrics and log collection agents

11. Platform Monitoring
• Agents/Exporters
• Node-Exporter (systemd enabled)
• Fluent-Bit
• Blackbox Exporter
• MySQL & SQL Exporters
• Postgres Exporter
• IPMI Exporter
• Grafana
• Visualization & Alerting

17. GitOps
https://www.weave.works/blog/-6-ideas-shaping-gitops

22. The Fellowship

23. 10,000 foot view
• Multiple Zones/Data Centers
• 2 central control-planes
• Rest site control-planes
• Data-plane in each zone

24. Central Control-Plane

25. Site Control and Data Planes

26. CloudStack via GitOps

31. (PR) Created PR Approved
Reconciliation
Suspended
PR Merged
Reconciliation
Resumed;
Changes
Validated
Backout PR
Merged
Dev
1. Development Team creates Pull Request with changes.
a) A Backout PR is also created; only used if issues arise during deployment
2. Pull Request is Reviewed and Approved (Gate)
3. Flux Reconciliation is Suspended
a) flux suspend helmrelease {helmrelease} -n {cluster}
4. Pull Request is Merged
5. Flux Reconciliation is Resumed
a) flux resume helmrelease {helmrelease} -n {cluster}
6. Changes are Validated
1. If successful, deployment is complete
2. If Backout is required, Backout PR is merged reverting changes
DONE
Action Performed
(Git vs FluxCLI)
Ops

33. Success with CloudStack
• Deployed KVM hosts in multiple zones/data centers
• Dozens of VPCs/Guest Networks & 1000 VMs running in CloudStack
• 50%+ reduction in CPU/Memory Allocation*
• PR - Fiber Channel MultiPath for KVM
• GitOps management of platform

34. Thank You!
Rene Glover
Alex Dometrius