Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency - Highload++

Optimizing Kubernetes
Resource Requests/Limits
for Cost-Efficiency and Latency
Henning Jacobs (Zalando SE)
@try_except_

4
ZALANDO AT A GLANCE
~ 4.5billion EUR
revenue 2017
> 200
million
visits
per
month
> 15.000
employees in
Europe
> 70%
of visits via
mobile devices
> 24
million
active customers
> 300.000
product choices
~ 2.000
brands
17
countries

5
SCALE
100Clusters
373Accounts

6
KUBERNETES: IT'S ALL ABOUT RESOURCES
Node Node
Pods demand capacity
Nodes offer capacity
Scheduler

7
COMPUTE RESOURCE TYPES
● CPU
● Memory
● Local ephemeral storage (1.12+)
● Extended Resources
○ GPU
○ TPU?
Node

8
KUBERNETES RESOURCES
CPU
○ Base: 1 AWS vCPU (or GCP Core or ..)
○ Example: 100m (0.1 vCPU, "100 Millicores")
Memory
○ Base: 1 Byte
○ Example: 500Mi (500 MiB memory)

9
REQUESTS / LIMITS
Requests
○ Affect Scheduling Decision
○ Priority (CPU, OOM adjust)
Limits
○ Limit maximum container usage
resources:
requests:
cpu: 100m
memory: 300Mi
limits:
cpu: 1
memory: 300Mi

10
CPU
Memory
REQUESTS: POD SCHEDULING
CPU
Memory
CPU
Memory
Node 1
Node 2
Pod 1
Requests

11
POD SCHEDULING
CPU
Memory
CPU
Memory
Node 1
Node 2
Pod 2

12
POD SCHEDULING
CPU
Memory
CPU
Memory
Node 1
Node 2
Pod 3

13
POD SCHEDULING
CPU
Memory
CPU
Memory
Node 1
Node 2
Pod 4

14
POD SCHEDULING: TRY TO FIT
CPU
Memory
CPU
Memory
Node 1
Node 2

15
POD SCHEDULING: NO CAPACITY
CPU
Memory
CPU
Memory
Node 1
Node 2
Pod 4
"PENDING"

16
REQUESTS: CPU SHARES
docker run --requests=cpu=10m/5m ..sha512()..
cat /sys/fs/cgroup/cpu/kubepods/burstable/pod5d5..0d/cpu.shares
10 // relative share of CPU time
cat /sys/fs/cgroup/cpu/kubepods/burstable/pod6e0..0d/cpu.shares
5 // relative share of CPU time
cat /sys/fs/cgroup/cpuacct/kubepods/burstable/pod5d5..0d/cpuacct.usage
/sys/fs/cgroup/cpuacct/kubepods/burstable/pod6e0..0d/cpuacct.usage
13432815283 // total CPU time in nanoseconds
7528759332 // total CPU time in nanoseconds

17
LIMITS: COMPRESSIBLE RESOURCES
Can be taken away quickly,
"only" cause slowness
CPU Throttling
200m CPU limit ⇒
container can use
0.2s of CPU time per second

18
CPU THROTTLING
docker run --cpus CPUS -it python
python -m timeit -s 'import hashlib' -n 10000 -v
'hashlib.sha512().update(b"foo")'
CPUS=1.0 3.8 - 4ms
CPUS=0.5 3.8 - 52ms
CPUS=0.2 6.8 - 88ms
CPUS=0.1 5.7 - 190ms

19
LIMITS: NON-COMPRESSIBLE RESOURCES
Hold state,
are slower to take away.
Killing (OOMKill)

20
MEMORY LIMITS: OUT OF MEMORY
kubectl get pod
NAME READY STATUS RESTARTS AGE
kube-ops-view-7bc-tcwkt 0/1 CrashLoopBackOff 3 2m
kubectl describe pod kube-ops-view-7bc-tcwkt
...
Last State: Terminated
Reason: OOMKilled
Exit Code: 137

21
QOS
Guaranteed: all containers have limits == requests
Burstable: some containers have limits > requests
BestEffort: no requests/limits set
kubectl describe pod …
…
Limits:
memory: 100Mi
Requests:
cpu: 100m
memory: 100Mi
QoS Class: Burstable

22
OVERCOMMIT
Limits > Requests ⇒ Burstable QoS ⇒ Overcommit
Overcommit for CPU: fine, running into completely fair scheduling
Overcommit for memory: fine, as long as demand < node capacity
https://code.fb.com/production-engineering/oomd/
Might run into unpredictable OOM situations
when demand reaches node's memory
capacity (Kernel OOM Killer)

23
LIMITS: CGROUPS
docker run --cpus 1 -m 200m --rm -it busybox
cat /sys/fs/cgroup/cpu/docker/8ab25..1c/cpu.{shares,cfs_*}
1024 // cpu.shares (default value)
100000 // cpu.cfs_period_us (100ms period length)
100000 // cpu.cfs_quota_us (total CPU time in µs consumable per period)
cat /sys/fs/cgroup/memory/docker/8ab25..1c/memory.limit_in_bytes
209715200

24
LIMITS: PROBLEMS
1. CPU CFS Quota: Latency
2. Memory: accounting, OOM behavior

25
PROBLEMS: LATENCY
https://github.com/zalando-incubator/kubernetes-on-aws/pull/923

27
PROBLEMS: HARDCODED PERIOD
https://github.com/kubernetes/kubernetes/issues/51135

28
WHAT'S A GOOD VALUE FOR CFS PERIOD?
Latencyp95→

29
NOW IN KUBERNETES 1.12
https://github.com/kubernetes/kubernetes/pull/63437

31
OVERLY AGGRESSIVE CFS: EXPERIMENT #1
CPU Period: 100ms
CPU Quota: None
Burn 5ms and sleep 100ms
⇒ Quota disabled
⇒ No Throttling expected!
https://gist.github.com/bobrik/2030ff040fad360327a5fab7a09c4ff1

32
EXPERIMENT #1: NO QUOTA, NO THROTTLING
2018/11/03 13:04:02 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms

33
CPU Period: 100ms
CPU Quota: 20ms
⇒ No 100ms intervals where possibly 20ms is burned
⇒ No Throttling expected!

34
EXPERIMENT #2: OVERLY AGGRESSIVE CFS

35
CPU Period: 10ms
CPU Quota: 2ms
⇒ Same 20% CPU (200m) limit, but smaller period
⇒ Throttling expected!

36
SMALLER CPU PERIOD ⇒ BETTER LATENCY

37
LIMITS: VISIBILITY
docker run --cpus 1 -m 200m --rm -it busybox top
Mem: 7369128K used, 726072K free, 128164K shrd, 303924K buff, 1208132K cached
CPU0: 14.8% usr 8.4% sys 0.2% nic 67.6% idle 8.2% io 0.0% irq 0.6% sirq

38
• Container-aware memory configuration
• JVM MaxHeap
• Container-aware processor configuration
• Thread pools
• GOMAXPROCS
• node.js cluster module
LIMITS: VISIBILITY

39
ZALANDO: DEVELOPERS USING KUBERNETES

41
ZALANDO: DECISION
• Forbid Memory Overcommit
• Implement mutating admission webhook
• Set requests = limits
• Disable CPU CFS Quota in all clusters

42
ZALANDO: DISABLING CPU LIMITS

43
INGRESS LATENCY IMPROVEMENT

Example: Getting started
with Zalenium & UI Tests
Example: Step by step guide to the first UI test with Zalenium running in the
Continuous Delivery Platform. I was always afraid of UI tests because it looked too
difficult to get started, Zalenium solved this problem for me.

45
CLUSTER AUTOSCALER
Simulates the Kubernetes scheduler internally to find out..
• ..if any of the pods wouldn’t fit on existing nodes
⇒ upscale is needed
• ..if it’s possible to fit some of the pods on existing nodes
⇒ downscale is needed
⇒ Cluster size is determined by resource requests (+ constraints)
github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler

46
ALLOCATABLE
Reserve resources for
system components,
Kubelet, and container runtime:
--system-reserved=
cpu=100m,memory=164Mi
--kube-reserved=
cpu=100m,memory=282Mi

47
CPU/memory requests "block" resources on nodes.
Difference between actual usage and requests → Slack
SLACK
CPU
Memory
Node
"Slack"

48
STRANDED RESOURCES
Stranded
CPU
Memory
CPU
Memory
Node 1
Node 2
Some available capacity
can become unusable /
stranded.
⇒ Reschedule, bin packing

49
HOW TO FIND GOOD VALUES FOR REQUESTS?
resources:
limits:
memory: "2Gi"
requests:
cpu: "1.5"
memory: "2Gi"

50
TWO TOOLS
● Kubernetes Resource Report
○ Dashboard showing Kubernetes Resource Slack
● Kubernetes Application Dashboard
○ Grafana Dashboard showing historical CPU and Memory
usage for Kubernetes Applications

51
KUBERNETES RESOURCE REPORT
github.com/hjacobs/kube-resource-report

52
KUBERNETES RESOURCE REPORT
"Slack"

53
KUBERNETES APPLICATION DASHBOARD

54
VERTICAL POD AUTOSCALER (VPA)
"Some 2/3 of the Borg users use autopilot."
- Tim Hockin
VPA: Set resource requests
automatically based on usage.

55
VPA FOR PROMETHEUS
apiVersion: poc.autoscaling.k8s.io/v1alpha1
kind: VerticalPodAutoscaler
metadata:
name: prometheus-vpa
namespace: kube-system
spec:
selector:
matchLabels:
application: prometheus
updatePolicy:
updateMode: Auto
CPU/memory

56
AUTOSCALING BUFFER
• Cluster Autoscaler only triggers on Pending Pods
• Node provisioning is slow
⇒ Reserve extra capacity via low priority Pods
"Autoscaling Buffer Pods"

57
AUTOSCALING BUFFER
kubectl describe pod autoscaling-buffer-eu-central-1a-78-rzjq5 -n kube-system
...
Namespace: kube-system
Priority: -1000000
PriorityClassName: autoscaling-buffer
Containers:
pause:
Image: teapot/pause-amd64:3.1
Requests:
cpu: 1600m
memory: 6871947673
Evict if higher
priority (default)
Pod needs
capacity

58
HORIZONTAL POD AUTOSCALER
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: myapp
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 3
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 100
target: ~100% of
CPU requests
...

59
HORIZONTAL POD AUTOSCALING (CUSTOM METRICS)
Queue Length
Prometheus Query
Ingress Req/s
ZMON Check
github.com/zalando-incubator/kube-metrics-adapter

60
DOWNSCALING DURING OFF-HOURS
github.com/hjacobs/kube-downscaler
Weekend

61
DOWNSCALING DURING OFF-HOURS
DEFAULT_UPTIME="Mon-Fri 07:30-20:30 CET"
annotations:
downscaler/exclude: "true"

62
WHAT WORKED FOR US
● Disable CPU CFS Quota in all clusters
● Admission Controller:
○ Prevent memory overcommit
○ Set default requests
● Kubernetes Resource Report
● Downscaling during off-hours

63
WRAP UP
● Understanding Requests/Limits
● Deciding on quota period (or disable), overcommit
● Autoscaling (cluster-autoscaler, HPA, VPA, downscaler)
● Buffer capacity
● Optimizing slack

64
OPEN SOURCE
Kubernetes on AWS
github.com/zalando-incubator/kubernetes-on-aws
AWS ALB Ingress controller
github.com/zalando-incubator/kube-ingress-aws-controller
Skipper HTTP Router & Ingress controller
github.com/zalando/skipper
External DNS
github.com/kubernetes-incubator/external-dns
Postgres Operator
github.com/zalando-incubator/postgres-operator
Kubernetes Resource Report
github.com/hjacobs/kube-resource-report
Kubernetes Downscaler

https://github.com/hjacobs/kube-ops-view

66
OTHER TALKS
• Everything You Ever Wanted to Know About Resource Scheduling
• Inside Kubernetes Resource Management (QoS) - KubeCon 2018
• Setting Resource Requests and Limits in Kubernetes (Best Practices)

QUESTIONS?
HENNING JACOBS
HEAD OF
DEVELOPER PRODUCTIVITY
henning@zalando.de
@try_except_
Illustrations by @01k

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency - Highload++

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency - Highload++

Similar to Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency - Highload++ (20)

More from Henning Jacobs

More from Henning Jacobs (20)

Recently uploaded

Recently uploaded (20)

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency - Highload++