Ever wondered how the K8s scheduler works, and how can you “help” it make the right decision for your application? In this session, we'll cover several different scheduling use-cases in K8s, what scheduling techniques are required in each and when to use them.

1. Advanced Scheduling
in Kubernetes
Tsahi Duek | Solutions Architect @ Spotinst

2. K8s Pod Scheduling
Deep Dive
Tsahi Duek | Solutions Architect @ Spotinst

3. Agenda
● Scheduling before K8s
● Why do we even care?
● K8s scheduling techniques
● Demo
● Resources

4. WHO AM I?

8. Pre K8s Architecture

10. Worker node
us-west-1a us-west-1b
Worker node Worker node Worker node Worker node Worker node
Pod
Pod
Pod
Pod Pod Pod
Pod
Pod
Pod
PodPod Pod
Pod
New Technology - New Challenges
Pod

11. Worker node
r5.4xlarge
Worker node
r5.4xlarge
Worker node
r5.8xlarge
Pod
10G of
Network
Bandwidth
Specific instance - instance=type: r5.8xlarge

12. apiVersion: v1
kind: Pod
metadata:
name: nodeselector-instance-type
spec:
nodeSelector:
beta.kubernetes.io/instance-type: r5.8xlarge
containers:
- name: nodeselector-linux
image: "gcr.io/google-samples/hello-go-gke:1.0"
...
nodeSelector

13. All instances with Network-Bandwidth of 10G:
● m5.8xlarge
● c5.9xlarge
● r5.8xlarge
● ...
Multiple instances - multiple instance-types

14. apiVersion: v1
kind: Pod
metadata:
name: nodeselector-in-nodeaffinity-depl
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/instance-type
operator: In
values:
- r5.8xlarge
- c5.9xlarge
- m5.8xlarge
containers:
- name: nodeselector
image: "gcr.io/google-samples/hello-go-gke:1.0"
...
nodeAffinity

15. Applications that go “together”

17. Worker node
c5d.4xlarge
Applications that are bounded together
Worker node Worker node
r5.4xlarge r5.8xlarge
Pod - a
Pod - b
Pod - x
Pod - y
Pod - 𝛼
Pod - 𝜷

18. apiVersion: v1
kind: Pod
metadata:
name: pod-affinity
service: core
spec:
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: service
operator: In
values:
- core
topologyKey: kubernetes.io/hostname
containers:
- name: nodeselector-linux
image: "gcr.io/google-samples/hello-go-gke:1.0"
...
podAffinity rule

19. Pod - a
Pods from different “family” are not allowed
Worker node
c5d.4xlarge
Worker node Worker node
r5.4xlarge r5.8xlarge
Pod - a
Pod - b
Pod - x
Pod - y
Pod - 𝛼
Pod - 𝜷

21. apiVersion: v1
kind: Pod
metadata:
name: pod-affinity
service: core
spec:
affinity:
podAffinity:
...
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: service
operator: In
values:
- core
topologyKey: kubernetes.io/hostname
containers:
...
podAffinity + podAntiAffinity

22. Multiple replicas/instances/zones - HA

23. Worker node
us-west-1a us-west-1b
Worker node Worker node Worker node Worker node Worker node
Pod
Pod
Pod
Pod
PodPod
Pod
Multiple replicas/instances/zones - HA

24. Worker node
us-west-1a us-west-1b
Worker node Worker node Worker node Worker node Worker node
Pod Pod Pod
Pod
Pod
Pod
Pod
PodPod
Pod
Multiple replicas/instances/zones - HA

25. apiVersion: apps/v1
kind: Deployment
metadata:
name: pod-ha-host-zone
spec:
replicas: 3
...
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: name
operator: In
values:
- pod-ha-host-zone
topologyKey: kubernetes.io/hostname
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 10
podAffinityTerm:
labelSelector:
matchExpressions:
- key: service
operator: In
values:
- pod-ha-host-zone
topologyKey: failure-domain.beta.kubernetes.io/zone
1
2
Pod antiAffinity - zone & host level

26. Taints & Tolerations
● Taints - rule for nodes to repel pods
● Tolerations - attribute of pods

27. Taints & Tollerations

28. Additional scheduling considerations
● PVC location - PVC usually exists in a single AZ
● GPU - expensive instances -> “taint” them to repel
unwanted pods
● Don’t let coincidence manage your application
It is your responsibility to take care of this

31. Resources
https://github.com/tsahiduek/k8s-examples
Medium blog - https://bit.ly/2msBdJY
https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
Kubernetes design-proposal directory -
https://github.com/kubernetes/community/tree/master/contributors/design-proposals
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/sch
eduling/nodeaffinity.md
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/sch
eduling/podaffinity.md