The document discusses Intuit's journey to building a modern software-as-a-service (SaaS) platform using Kubernetes. It outlines the goals of increasing release cycle time, release frequency, and recoverability. It then describes challenges around multi-tenancy, common functionality, resilience, monitoring, and costs. Intuit developed the Keiko project to help solve these challenges, which includes tools like an addon manager, instance manager, upgrade manager, and active monitor. The solutions have helped Intuit improve velocity, operability, and security when developing and running microservices on Kubernetes.

1. December 14, 2019
Intuit Kubernetes Journey
- Ravi & Laks

2. ● Why we built Modern Saas Platform?
● What is it?
● How we built it?
Agenda

3. Intuit Confidential and Proprietary 3
Measured
value
Expected
Improvement
Recoverability X 10X faster
Release cycle time (days)
(PR to deploy)
Y 1.4Y faster
Release frequency (days) Z 3Z faster
Goals

4. Intuit Confidential and Proprietary 4
Monolith vs Microservice

5. Intuit Confidential and Proprietary 5
Service Onboarding Prior to M-SaaS
I want to
develop
new
micro
service
● Create Service on Service Portal
● Get an Cloud Account -
● Create all the basic resources in AWS
● If you have Web server do this...
● Monitoring Alerting Logs…..
● :
● Take care of AWS SG, EC2, Auto scaling, R53,
Ingress, Egress
● Create
○ Git Repo
○ Pipeline
○ Set up CD …
○ …….
Takes few weeks to set up and run the service
Steep learning curve
Multiple AWS Accounts
Take care of security patches periodically

6. Intuit Confidential and Proprietary 6
Monolith vs Microservice

7. Intuit Confidential and Proprietary 7
Service Onboarding Today…. with M-SaaS
VELOCITY OPERABILITY SECURITY
I want to
develop
new
micro
service
AppD
Microservice
15 minutes

8. Gitops Flow

9. Intuit Confidential and Proprietary 9
The Modern SaaS Platform @ Intuit

10. Intuit Confidential and Proprietary 10
Journey so far … Services on MSaaS

11. Intuit Confidential and Proprietary 11
alb-ingress kube-dns fluentd metrics prometheus autoscaler
Addons
User namespace 1 User namespace 2 User namespace 3 User namespace n
Applications
kube-apiserver kube-proxy
K8s Control Plane
kube-scheduler kube-controlleretcd
Each Kubernetes cluster today ...

12. Intuit Confidential and Proprietary 12
Master Nodes
alb-ingress kiam eventrouter metrics kube-dns autoscaler
Addons
kube-apiserver kube-proxy
K8s Control Plane
kube-scheduler kube-controlleretcd
Each Kubernetes cluster today ...

13. Intuit Confidential and Proprietary 13
The Challenges

14. Intuit Confidential and Proprietary 14
Multi-tenancy
- What does each tenant mean?
- Namespace?
- Kubernetes objects with the same label?
- Some CRD?
We decided to go with Kubernetes Namespaces

15. Intuit Confidential and Proprietary 15
More Multi-tenancy issues
- Noisy neighbour
- Customized setup
- Tenant specific AMIs
- Tenant specific instance types
- Cost accounting

16. Intuit Confidential and Proprietary 16
- Common functionality needed by all apps on a
cluster: DNS, Logs, Metrics etc.,
- Multi Tenancy - Noisy neighbour, Instance Types
- Resiliency and Hardening
- Deep Monitoring
- Cost Efficiency
The Challanges

17. Intuit Confidential and Proprietary 17
Addons
- Common functionality needed by all apps on a
cluster
- DNS, log forwarding, metrics, identity, etc.
- Integrate with other AWS services such as ALB.

18. Intuit Confidential and Proprietary 18
Resilience and hardening ...
- Pods stuck in terminating state ...
- EC2 instance networking broken …
- Bunch of 502s during upgrade...

19. Intuit Confidential and Proprietary 19
Deep monitoring
- Not enough to simply check if components are “up”
- Deep monitoring
- Actually exercise the functionality
- Periodically
- Preferably automatic remediation

20. Intuit Confidential and Proprietary 20
Cost efficiency
- How do we reduce costs?

21. Intuit Confidential and Proprietary 21
The Solutions

22. Intuit Confidential and Proprietary 22
Keiko
“Keiko provides a set of independent open-source tools
for orchestration and management of multi-tenant,
reliable, secure and efficient Kubernetes clusters at scale.”
github.com/keikoproj
Instance manager Kube forensics
Upgrade
manager
Active monitor Addon manager Governor Minion manager

23. Intuit Confidential and Proprietary 23
Keiko
Orchestration
Instance-manager Upgrade-manager
Reliability
Governor
Cost Eff
Minion Manager
Addon Manager
Security
Kube-Forensics
Monitoring
Active-monitor

24. Intuit Confidential and Proprietary 24
Addon-Manager
Addons are critical components within a Kubernetes cluster that
provide basic services needed by applications like DNS,
Ingress, Metrics, Logging, etc. Addon Manager provides a CRD
for lifecycle management of such addons using Argo
Workflows.

25. Intuit Confidential and Proprietary 25
Addon-Manager

26. Intuit Confidential and Proprietary 26
Multi-tenancy solutions
- Instance Group per Namespace
- Customized labels
- Centralized upgrades
We decided to go with ...

27. Intuit Confidential and Proprietary 27
Instance-manager
- Declaratively provision and manage ASGs (nodes)
- Number and type of nodes
- Labels and taints
- Subnets and security groups
$ kubectl create -f /tmp/hello_world.yaml
instancegroup.instancemgr.keikoproj.io/hello-world created
$ kubectl get igs
NAME STATE MIN MAX GROUP NAME PROVISIONER STRATEGY
AGE
hello-world Ready 2 3 shri-east-2-instance-manager-hello-world-NodeGroup-16Y8ZA1ZJW8JK eks-cf crd 3m
nodes Ready 2 3 shri-east-2-instance-manager-nodes-NodeGroup-1K1T3YSXCCCK9 eks-cf crd 1d

28. Intuit Confidential and Proprietary 28
Upgrade-manager
- Upgrade Manager provides RollingUpgrade, a
Kubernetes native mechanism for doing
rolling-updates of instances in an AutoScaling group
using a CRD and a controller.

29. Intuit Confidential and Proprietary 29
Governor
Governor improves the stability of large Kubernetes
clusters by proactively terminating failed but stuck pods
and misbehaving nodes.

30. Intuit Confidential and Proprietary 30
Active-monitor
Active-Monitor is a Kubernetes custom
resource controller which uses Argo
Workflows for deep cluster monitoring.

31. Intuit Confidential and Proprietary 31
Minion-manager
Minion-manager enables the intelligent use of Spot
Instances in Kubernetes clusters on AWS. This is done
by factoring in on-demand prices, spot-instance prices
and current state of the AutoScalingGroups.

32. Intuit Confidential and Proprietary 32
Kube-forensics
Kube-forensics allows a cluster administrator to dump
the current state of a running pod and all its containers
so that security professionals can perform offline
forensic analysis.

33. Intuit Confidential and Proprietary 33
Keiko
github.com/keikoproj
Instance manager Kube forensics
Upgrade
manager
Active monitor Addon manager Governor Minion manager
github.com/keikoproj
github.com/argoproj
twitter.com/keikoproj

34. Intuit Confidential and Proprietary 34
Coming up ...
- Kubernetes control plane using EKS
- Multi-cluster Service Mesh using Istio
- OpenTelemetry
- GitOps for AWS resources
- Experimentation platform
- And more ...

35. Intuit Confidential and Proprietary 35
There’s a lot happening ...
<We are hiring />
Ravi_Hari@intuit.com
https://www.linkedin.com/in/ravi-hari-46896a15/
laks@intuit.com
https://www.linkedin.com/in/laks1/

36. 36