AWS Community Day Kochi 2019 - Technical Session Turbocharge Developer productivity with platform build on K8S and AWS services by - Laks , Principal Engineer - Intuit

1. Intuit Modern SaaS Platform
Running Kubernetes Clusters at scale on AWS
Laks

2. ● Why we built it?
● What is the Modern SaaS platform?
● How we built it?
Agenda

3. Intuit Confidential and Proprietary 3
Measured
value
Expected
Improvement
Recoverability X 10X faster
Release cycle time (days)
(PR to deploy)
Y 1.4Y faster
Release frequency (days) Z 3Z faster
Goals

4. Intuit Confidential and Proprietary 4
Monolith vs Microservice

5. Intuit Confidential and Proprietary 5
Service Onboarding Prior to M-SaaS
I want to
develop
new micro
service
● Create Service on Service Portal
● Get an Cloud Account -
● Create all the basic resources in AWS
● If you have Web server do this...
● Monitoring Alerting Logs…..
● :
● Take care of AWS SG, EC2, Auto scaling, R53,
Ingress, Egress
● Create
○ Git Repo
○ Pipeline
○ Set up CD …
○ …….
Takes few weeks to set up and run the service
Steep learning curve
Multiple AWS Accounts
Take care of security patches periodically

6. Intuit Confidential and Proprietary 6
Monolith vs Microservice

7. Intuit Confidential and Proprietary 7
Service Onboarding Today…. with M-SaaS
VELOCITY OPERABILITY SECURITY
I want to
develop
new micro
service
AppD
Microservice
15 minutes

8. Gitops Flow

9. Intuit Confidential and Proprietary 9
The Modern SaaS Platform @ Intuit

10. Intuit Confidential and Proprietary 10
● Design and development started in Jan ‘18
● First application was running Kafka on Kubernetes
● Running clusters in dev/test, pre-prod and prod environments
since Apr ‘18.
● Over 150 Kubernetes clusters and 3000 namespaces today…
Journey so far ...

11. Intuit Confidential and Proprietary 11
Journey so far … Services on MSaaS

12. Intuit Confidential and Proprietary 12
Journey so far … Namespaces on MSaaS

13. Intuit Confidential and Proprietary 13
Journey so far … Clusters on MSaaS

14. Intuit Confidential and Proprietary 14
● Intuit Kubernetes Service
○ Using Kops today
○ Moving to EKS
● Intuit Kubernetes Service Manager (may open source)
● Custom Resources for cluster lifecycle management (aka. Keiko)
Modern SaaS platform today ...

15. Intuit Confidential and Proprietary 15
alb-ingress kube-dns fluentd metrics prometheus autoscaler
Addons
User namespace 1 User namespace 2 User namespace 3 User namespace n
Applications
kube-apiserver kube-proxy
K8s Control Plane
kube-scheduler kube-controlleretcd
Each Kubernetes cluster today ...

16. Intuit Confidential and Proprietary 16
Master Nodes
alb-ingress kiam eventrouter metrics kube-dns autoscaler
Addons
kube-apiserver kube-proxy
K8s Control Plane
kube-scheduler kube-controlleretcd
Each Kubernetes cluster today ...

17. Intuit Confidential and Proprietary 17
The Challenges

18. Intuit Confidential and Proprietary 18
Addons
- Common functionality needed by all apps on a cluster
- DNS, log forwarding, metrics, identity, etc.
- Integrate with other AWS services such as ALB.

19. Intuit Confidential and Proprietary 19
Multi-tenancy
- What does each tenant mean?
- Namespace?
- Kubernetes objects with the same label?
- Some CRD?
We decided to go with Kubernetes Namespaces

20. Intuit Confidential and Proprietary 20
More Multi-tenancy issues
- Noisy neighbour
- Customized setup
- Tenant specific AMIs
- Tenant specific instance types
- Cost accounting

21. Intuit Confidential and Proprietary 21
Resilience and hardening ...
- Pods stuck in terminating state ...
- EC2 instance networking broken …
- Bunch of 502s during upgrade...

22. Intuit Confidential and Proprietary 22
Deep monitoring
- Not enough to simply check if components are “up”
- Deep monitoring
- Actually exercise the functionality
- Periodically
- Preferably automatic remediation

23. Intuit Confidential and Proprietary 23
Cost efficiency
- How do we reduce costs?

24. Intuit Confidential and Proprietary 24
The Solutions

25. Intuit Confidential and Proprietary 25
Addon-Manager
Addons are critical components within a Kubernetes cluster that
provide basic services needed by applications like DNS,
Ingress, Metrics, Logging, etc. Addon Manager provides a CRD
for lifecycle management of such addons using Argo
Workflows.

26. Intuit Confidential and Proprietary 26
Addon-Manager

27. Intuit Confidential and Proprietary 27
Multi-tenancy solutions
- Instance Group per Namespace
- Customized labels
- Centralized upgrades
We decided to go with ...

28. Intuit Confidential and Proprietary 28
Instance-manager
- Declaratively provision and manage ASGs (nodes)
- Number and type of nodes
- Labels and taints
- Subnets and security groups
$ kubectl create -f /tmp/hello_world.yaml
instancegroup.instancemgr.keikoproj.io/hello-world created
$ kubectl get igs
NAME STATE MIN MAX GROUP NAME PROVISIONER STRATEGY
AGE
hello-world Ready 2 3 shri-east-2-instance-manager-hello-world-NodeGroup-16Y8ZA1ZJW8JK eks-cf crd 3m
nodes Ready 2 3 shri-east-2-instance-manager-nodes-NodeGroup-1K1T3YSXCCCK9 eks-cf crd 1d

29. Intuit Confidential and Proprietary 29
Upgrade-manager
- Upgrade Manager provides RollingUpgrade, a
Kubernetes native mechanism for doing rolling-
updates of instances in an AutoScaling group using a
CRD and a controller.

30. Intuit Confidential and Proprietary 30
Governor
Governor improves the stability of large Kubernetes
clusters by proactively terminating failed but stuck pods
and misbehaving nodes.

31. Intuit Confidential and Proprietary 31
Active-monitor
Active-Monitor is a Kubernetes custom
resource controller which uses Argo
Workflows for deep cluster monitoring.

32. Intuit Confidential and Proprietary 32
Minion-manager
Minion-manager enables the intelligent use of Spot
Instances in Kubernetes clusters on AWS. This is done
by factoring in on-demand prices, spot-instance prices
and current state of the AutoScalingGroups.

33. Intuit Confidential and Proprietary 33
Kube-forensics
Kube-forensics allows a cluster administrator to dump
the current state of a running pod and all its containers
so that security professionals can perform offline
forensic analysis.

34. Intuit Confidential and Proprietary 34
Keiko
“Keiko provides a set of independent open-source tools for
orchestration and management of multi-tenant, reliable,
secure and efficient Kubernetes clusters at scale.”
github.com/keikoproj
Instance manager Kube forensics
Upgrade
manager
Active monitor Addon manager Governor Minion manager

35. Intuit Confidential and Proprietary 35
Keiko
github.com/keikoproj
Instance manager Kube forensics
Upgrade
manager
Active monitor Addon manager Governor Minion manager
github.com/keikoproj
twitter.com/keikoproj

36. Intuit Confidential and Proprietary 36
Keiko
Orchestration
Instance-manager Upgrade-manager
Reliability
Governor
Cost Eff
Minion Manager
Addon Manager
Security
Kube-Forensics
Monitoring
Active-monitor

37. Intuit Confidential and Proprietary 37
Coming up ...
- Kubernetes control plane using EKS
- Multi-cluster Service Mesh using Istio
- OpenTelemetry
- GitOps for AWS resources
- Experimentation platform
- And more ...

38. Intuit Confidential and Proprietary 38
There’s a lot happening ...
<We are hiring />

39. Thank You
laks@intuit.com
https://www.linkedin.com/in/laks1/