Learn how to manage and scale productive ArgoCD deployments in a multi cluster environment easily and securely using GitOps patterns. We will present the deployment of applications across multiple Kubernetes clusters on various public, private, on-premises and explore the best practices for scaling and managing highly available, reliable and secure applications. We will also demonstrate how you can structure your Git repositories to share and scale within your teams and organization in order to deploy your cloud native deployments in an automatic and predictive way.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling production grade
Kubernetes Multi-Cluster
environments using GitOps
Rodrigo Bersa
EKS Specialist Solutions Architect
AWS
Carlos Santana
EKS Specialist Solutions Architect
AWS
KCD Washington, DC

2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
I have chosen to use
Kubernetes – now what?

3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What customers are building
Apps and services
Mobile
IoT
Static websites
Complex web apps
.NET apps
Legacy homegrown
Linux apps
Monoliths
Autonomous vehicles
(object tracking,
sensor fusion)
Robotics
(vision, grasping,
motion control)
Modeling, training,
and inference
Real-time
MapReduce
Batch

4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enforce security standards and best practices
across clusters to automate deployments
Define boundaries between multiple teams
Provision multiple workloads at scale
Cluster management
Team management
Workload management
Install add-ons and their dependencies
Add-on management
Configuration management
Automate configuration and upgrade
lifecycle from a single source of truth
Challenges and Goals

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CNCF Cloud Map
“There is no shortage of amazing tooling
in the K8s ecosystem, but there is no
guide for how to put all the tools
together”

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes Journey
Choose an
orchestrator
Decisions

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes Journey
Choose an
orchestrator
Data Plane
Compute
Decisions

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes Journey
Choose an
orchestrator
Data Plane
Compute Cluster Addons
Decisions

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Kubernetes Journey
Choose an
orchestrator
Data Plane
Compute Cluster Addons
Decisions
Day 2
Operations

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cluster add-ons
Security
Cilium Gatekeeper
Kyverno
Observability
Prometheus Fluent Bit OTEL
Reliability
Karpenter Autoscaler Keda
Delivery
ArgoCD Flux Crossplane
Other

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Principle of GitOps
A system managed by
GitOps must have its
desired state expressed
declaratively
Desired state is stored in a
way that enforces
immutability, versioning
and retains a complete
version history
Software agents
continuously observe
actual system state and
attempt to apply the
desired state
Software agents
automatically pull the
desired state declarations
from the source
Enforces Consistency
Reduces Business Risk Enhances Auditability Boosts Security

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GitOps Continuously Reconcile
Build
Test
Scan
Operate/Fix
Deploy/Verify
Observe/Alert
Immutability Firewall
Git becomes the single
source of truth for the
system’s desired state,
enabling reproducible
automated deployments,
cluster management, and
monitoring.

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Secure environments

14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Private environments
Corporate datacenter
Kubernetes Cluster
Build
Infrastructure deployment
Corporate Network

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Public repositories
Corporate datacenter
Kubernetes Cluster
Build
Deploy
Corporate Network
Infrastructure deployment

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Private repositories
Corporate datacenter
Kubernetes Cluster
Repository
Scan
Scan Store
Deploy
Corporate Network
Build
Pull
Grype
Clair
Infrastructure deployment

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Trusted repositories
Corporate datacenter
Kubernetes Cluster
Repository
Scan
Scan Store
Deploy
Corporate Network
Build
Grype
Clair
Infrastructure deployment
Pull
Image replication

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud environment
AWS Cloud
Amazon Inspector
Amazon EKS
Amazon ECR
VPC
Private subnet
VPN connection
Scan
Deploy
AWS Direct Connect
Secured Network
Endpoints
AWS PrivateLink
Store
Build
Infrastructure
deployment
Image replication
AWS CodeCommit

20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Team management

21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Identity & Access
Management
Policy
Management
Namespace as a
Service
Multi-team considerations

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Separation of concerns
Platform engineers
Platform builders – build and integrate tools
that provision, manage and secure the cloud
computing infrastructure
Software engineers
Application builders – free to focus on
building applications that deliver business
value to customers

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Teams management (k8s)
Dev Team A
Dev Team B
Platform Team
Audit Team
Kubernetes Cluster
Control Plane
Data Plane
Instances Instances
Developer RBAC
Temp RBAC
Admin RBAC

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Teams management (cloud)
Developer Role
Admin Role
Dev Team A
Dev Team B
Platform Team
Audit Team
Temporary Role
VPC
Availability Zone 1 Availability Zone 2
Managed Node Group
Karpenter “Groupless”
Amazon EKS
AWS Cloud

25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dev/Test Cluster
Teams management (ArgoCD)
Dev Team A
Dev Team B
Platform Team
Audit Team
Apps
ArgoCD Projects
Apps
Repository
Audit
ArgoCD Project
Policy
Repository
Platform
ArgoCD Project
Platform
Repository
Production Cluster
workloads
policies
addons

26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automated cluster deployment

27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automated deployment (k8s)
Build
Git push
IaC
repository
Trigger
IaC
Pipeline
Control Plane
Data Plane
Instances Instances
Dev/Test Cluster
Control Plane
Data Plane
Instances Instances
Production Cluster
Dev Team A
Dev Team B
Platform Team
Audit Team
Corporate datacenter Teams
Repository
Scanning
Pull
Corporate Network
Push

28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automated deployment (Cloud)
Build
Git push
IaC
repository
Trigger
Dev Team A
Dev Team B
Platform Team
Audit Team
Teams
Scan
Pull Push
IaC
Pipeline
AWS Cloud
Production account
Dev/Test account
Availability Zone 2
Availability Zone 2
Amazon EKS
Managed Node Group
Karpenter “Groupless”
Amazon EKS
Managed Node Group
Karpenter “Groupless”
Amazon EKS
VPC
VPC
Availability Zone 2
Availability Zone 2
VPC
Managed Node Group
Karpenter “Groupless”
Development tools
Secured Network

29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Infrastructure as
Code with
Terraform and CDK
Based on AWS best
practices and
recommendations
Integrated with
popular K8s tools
and services
Fully extensible
and
customizable
Amazon EKS Blueprints
An open-source framework that allows you to configure and deploy complete
Amazon EKS clusters across accounts and Regions

30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cluster creation with Amazon EKS Blueprints
• Infrastructure as Code (IaC)
• AWS CDK
• HashiCorp Terraform
• Addons
• OSS
• AWS
• Partner
• GitOps
• ArgoCD - (New GitOps-Bridge)
EKS Blueprints

31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-Cluster Management
with GitOps (Topologies)

32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Standalone/distributed GitOps
Namespace
Tenant AWS account
Amazon EKS
Tenant AWS account
Amazon EKS
Namespace
On premises
Kubernetes
Namespace
On premises
Kubernetes
Namespace
Full ArgoCD
UI/CLI
API Server
Redis Server
Repo
Controllers

33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Centralized/Hub-Spoke (Push)
Central Amazon EKS cluster
Central AWS account
Namespace
Tenant AWS account On premises
Amazon EKS
Tenant AWS account
Kubernetes
On premises
Kubernetes
Amazon EKS
Full ArgoCD
UI/CLI
API Server
Redis Server
Repo
Controllers

34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Centralized/Hub-Spoke (Shared)
Central Amazon EKS cluster
Central AWS account
Namespace
Tenant AWS account On premises
Amazon EKS
Tenant AWS account
Kubernetes
On premises
Kubernetes
Amazon EKS
App-2 repo
App-1 repo
App-4 repo
Platform Config
App-3 repo
Full ArgoCD
UI/CLI
API Server
Redis Server
Repo
Controllers

35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Centralized/Hub-Spoke (Agent)
Central Amazon EKS cluster
Central AWS account
Namespace
Tenant AWS account On premises
Amazon EKS
Tenant AWS account
Kubernetes
On premises
Kubernetes
Amazon EKS
App-2 repo
App-1 repo
App-4 repo
Platform Config
App-3 repo
Core ArgoCD
UI/CLI
API Server
Redis Server
Repo
Controllers
https://argo-cd.readthedocs.io/en/stable/operator-manual/core
https://akuity.io/blog/reducing-argocd-operational-burden
https://github.com/open-cluster-management-io/argocd-pull-integration
Full ArgoCD
UI/CLI
API Server
Redis Server
Repo
Controllers

36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reliability and Performance

37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ArgoCD Scaling Challenges
37
Image: https://colocatedeventseu2023.sched.com/event/1JoAP/scaling-argo-security-and-multi-tenancy-in-aws-eks-at-the-new-york-times-david-grizzanti-luke-philips-the-new-york-times

38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scaling ArgoCD
38
• Enable HPA (API, Repo, Redis)
• Controller Replica shards (random, round-robin)
• Tunning (timeouts, processors queues)
• Reconcile Optimization
 ignoreResourceUpdates vs. ignoreDifferences
https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability
https://argo-cd.readthedocs.io/en/stable/operator-manual/reconcile

39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring ArgoCD
39
Prometheus
Use Operator or Setup service labels
OpenTelemetry/ADOT
Alerts
AMP (Amazon Managed Service for Prometheus)
Grafana
ArgoCD Dashboard (tweak)
Sync time
Work queue
AMG (Amazon Managed Grafana)
Logging
Find k8s resources properties to ignore

40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GitOps Bridge

41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GitOps Bridge: IaC and Addons https://github.com/gitops-bridge-dev
ack terraform kops
ansible pulumi cdk
capa crossplane
IaC
Virtual private cloud (VPC)
Subnet
Kubernetes
Amazon EKS
IAM
Role NAT gateway Instances
Infrastructure
Kubernetes CR (ArgoCD Cluster)
metadata:
annotations:
aws_alb_role_arn: arn….
labels:
enable_aws_alb: true
1
2
3 ./aws/aws-cloudwatch-metrics-appset.yaml
./aws/aws-csi-ebs-resources-appset.yaml
./aws/aws-csi-efs-driver-appset.yaml
./aws/aws-csi-fsx-driver-appset.yaml
./aws/aws-fluentbit-appset.yaml
./aws/aws-gateway-api-controller-appset.yaml
./aws/aws-load-balancer-controller-appset.yaml
./aws/aws-node-termination-handler-appset.yaml
./aws/aws-oss-cert-manager-appset.yaml
./aws/aws-oss-cluster-autoscaler-appset.yaml
./aws/aws-oss-crossplane-providers-appset.yaml
./aws/aws-oss-external-dns-appset.yaml
./aws/aws-oss-external-secrets-appset.yaml
./aws/aws-oss-karpenter-appset.yaml
./aws/aws-oss-privateca-issuer-appset.yaml
./aws/aws-oss-velero-appset.yaml
./aws/aws-secrets-store-csi-appset.yaml
./oss/argo-cd-appset.yaml
./oss/argo-events-appset.yaml
./oss/argo-rollouts-appset.yaml
./oss/argo-workflows-appset.yaml
GitOps

42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GitOps Bridge: ApplicationSet (Addon versions)
version in dev is 1.6.0
version in staging is 1.5.5
version in production is 1.5.4
Cluster opt-in for the addon
Chart name and repo in a single place
Merge generator
Prevent Outages

43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GitOps Bridge: ApplicationSet (overrides)
Metadata based on IaC
Namespace based on IaC
Override values files
Value files in git

44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GitOps Bridge Configuration
Platform
team
Platform Repo
Region
Kubernetes
Account
Control Plane
Addons
App Of AppSet
Addon-1
charts/
environments/
Addon-2
App Of
ApplicationSets
Addon-1
ApplicationSet
Addon-3
ApplicationSet
Addon-2
ApplicationSet
clusters/
Kubernetes CR (ArgoCD Cluste)
metadata:
annotations:
aws_alb_role_arn: arn….
labels:
enable_aws_alb: true
Addon-1
Application
Addon-3
Application
Addon-2
Application

45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Carlos Santana
carrlos@amazon.com
Rodrigo Bersa
bersr@amazon.com
@csantanapr
csantanapr bersa