Speaker : Sunny Hu

1. K8s Cluster Autoscaler
Docs didn’t tell you …
Sr. Devops of Ayla - Sunny Hu.

2. How it works?

3. Official Docs for deployment

4. In the real world
● Deploy by Kubespray
● Not set cloud-provider
● Pod with local-storage
● Deploy with node selector

5. Restriction & Limit
● Not support regional Instance Group
● Handle up to 1000 nodes running 30 pods each
● Scale time
- Scale Up : 30 seconds
- Down : 10 mins
● Scale UP
- Not scale up/down when pod has node selector
● Scale Down
- Bulk delete up to 10 nodes
- Pods with restrictive PodDisruptionBudget.
- Pods with local storage. *
- CPU usage > 20%

6. - Kube-system pods that:
○ are not run on the node by default, *
○ don't have PDB or their PDB is too restrictive (since CA 0.6).
- Pods that are not backed by a controller object (so not created by
deployment, replica set, job, stateful set etc). *
- Pods that cannot be moved elsewhere due to various constraints (lack of
resources, non-matching node selectors or affinity, matching
anti-affinity, etc)
- Pods that have the following annotation set:
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

7. STEPS

8. Step 1 Steup CA manager
● Setup IG/ASG ACL for GCE / EC2
● Setup /etc/gce.conf
[global]
project-id = <Your project name>
network-project-id =<Your project name>
network-name = ayla-kube-gcp
subnetwork-name = ayla-app
node-instance-prefix = ayla-minion-pert-extra

9. Step 2 Prepare Extra Minion
● Create Base Image :
- Setup system config
- Install kubelet, kubeadm, kubectl
● Create Instance Templete:
- Startup script for kubelet config (set Provider ID)
- kubeadm join ( Never expire token )
● Create Instance Group
- Multi zone, *ACL
- Do not enable auto scaling of instance group

10. Provider ID
Source code have already told you
kubectl describe node
Every nodes include master, ingresses, minions need Provider ID.

11. Hack Provider ID
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

12. GCE
gce://<project_id>/<zone>/<hostname>
curl http://metadata.google.internal/computeMetadata/v1/project/project-id
-H "Metadata-Flavor: Google"
curl http://metadata.google.internal/computeMetadata/v1/instance/zone -H
"Metadata-Flavor: Google" | cut -d "/" -f4
curl http://metadata.google.internal/computeMetadata/v1/instance/name -H
"Metadata-Flavor: Google"

13. EC2
aws:///<znoe>/<name>
aws:///$(/usr/bin/ec2metadata --availability-zone)/$(curl
http://169.254.169.254/latest/meta-data/hostname)

14. STEP 3 Protect your node from Scale Down

15. Step 4 HELM Install
helm install stable/cluster-autoscaler
--name gcp-scaler
--namespace kube-system
--set cloudProvider=gce
--set rbac.create=true
--set nodeSelector.aylanetwork="camanager"
--set regional=true
--set skip-nodes-with-local-storage=false
--set balance-similar-node-groups=false

16. --set
"autoscalingGroups[0].name=https://www.googleapis.com/
compute/v1/projects/xxx/zones/us-east1-c/instanceGroup
s/k8s-minion-perf-scaler,autoscalingGroups[0].maxSize=
3,autoscalingGroups[0].minSize=0"

17. CA Deployment

18. Advance Setup
● Control more than 1 Instance Grouop
● Debug level : 1 Less -10 More
● Expendar :
○ random
○ most-pods
○ Least-waste
○ price

19. Bug ?
● After draining node pod cann’t remove form k8s metadata
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/scale
_down.go
● How to fix it ?
*/10 * * * * /opt/extra/clean-CA-NotReadyNode.sh >>
/var/log/cleanCANode.log 2>&1
#!/bin/bash
/usr/local/bin/kubectl delete node $(/usr/local/bin/kubectl get nodes | grep
NotReady | grep "[0-9][h|m]" | awk '{print $1}' | xargs)

20. All Proerties
● Helm chart
https://github.com/helm/charts/tree/master/stable/cluster-auto
scaler#configuration
● Cluster Autoscaler container
https://github.com/kubernetes/autoscaler/blob/cluster-autosca
ler-1.13.5/cluster-autoscaler/FAQ.md#what-are-the-paramete
rs-to-ca

21. Reference
● Cluster Autoscaler FAQ
● Cluster Autoscaler Helm
● K8s Autoscaler 101
● Understand K8s Autoscaler
● Official K8s Cluster Autoscaling Document
● Pod Disruption Budget